How to Troubleshoot a Camera in Shinobi

Sometimes camera setup isn't as easy for you as it is for others. Hopefully this article will help you deal with that pesky camera.

By : Moe Alam

Date Added : Tue Jun 12 2018 17:17:25 GMT+0000 (Coordinated Universal Time)
Last Updated : Wed Jun 30 2021 19:33:11 GMT+0000 (Coordinated Universal Time)

This article assumes your Shinobi is installed in the default location /home/Shinobi and using Ubuntu 19.10.

How to view the Log stream for System, Monitor, or User

Learn about this here

https://hub.shinobi.video/articles/view/bp4CWlGcXNHHHAL

A Stream to Test Shinobi's Internal Processing

You can test Shinobi by creating a Monitor with these settings changed. This is a static file to simulate a livestream.

"bind failed: Address already in use" for UDP

On June 26, 2020 a community member by the name of Tastiest#3464 in the Shinobi Community chat provided their solution to the problem.

  1. Open the Monitor Settings
  2. In your Connection section Set Automatic to No
  3. Then fill Path with
    • ?reuse=1
  4. Save and Enjoy!

Stream Delay (High Latency in Web Browser Stream)

One major factor is I-Frames (Keyframes) set in the camera's stream. If you choose not to encode (copy as video codec) then your camera will dictate this setting, usually from its web interface. In this instance set the I-Frame interval to match the FPS you are using. So if you are doing 15 FPS from the camera, set I-Frame to 15.

Although the latency is mainly because the dashboard you are looking at is in a web browser. There are multiple layers of processing upon the client-side CPU that occurs and that partially causes this delay, especially with H.264 based stream types.

So the best options for low latency are :

  • At the expense of slight delay but low CPU use. Set the Stream Type (blue section of Monitor Settings) to Poseidon over Websocket.
  • At the expense of CPU power on the server you can lower this delay. Set the Stream Type (blue section of Monitor Settings) to Base64 or MJPEG.

Things to note :

  • Poseidon over Websocket has possibility of stutter under poor network-to-server connectivity conditions.
  • Base64 has no stream limit, watch as many of them on the dashboard as you want.
  • MJPEG is limited by the browsers HTTP connection limit.
  • Both MJPEG and Base64 will use a large chunk of CPU per stream.

Smudgy or Broken Streams

As with many issue this one is characteristic of many different problems. Here are some possible solutions.

  • Using a Constant Bitrate is the most common solution. Learn more here.
  • Toggling RTSP Transport from TCP to UDP.
    • To try UDP; Open your Monitor Settings and set Automatic to No and set the RTSP Transport selector to UDP.
    • Shinobi defaults to TCP for RTSP Streams.
    • If using UDP does not help you should reset it to TCP.

Packet Loss or Black Stream - Disable Wall Clock Timestamps

Sometimes a camera that won't stream (appears black) correctly will better operate with this disabled.

  1. Open Monitor Settings for the pesky camera. The wrench icon.
  2. Set Use Camera Timestamps to Yes.

The error below is one of many that can be fixed by this solution. 

FFMPEG STDERRa few seconds ago
DTS 143301882296313, next:280367600000 st:0 invalid dropping PTS 143301882296313, next:280367600000 invalid dropping st:0

Black Stream - Analyzation Duration and Probe Size

You may fiddle with Analyzation Duration and Probe Size values to possibly acquire the stream. Try 1000000 for each field to start and work your way around.

Analyzation Duration defined in the FFmpeg Documentation. 

Specify how many microseconds are analyzed to probe the input. A higher value will enable detecting more accurate information, but will increase latency. It defaults to 5,000,000 microseconds = 5 seconds.

Probe Size defined in the FFmpeg Documentation. 

Set probing size in bytes, i.e. the size of the data to analyze to get stream information. A higher value will enable detecting more information in case it is dispersed into the stream, but will increase latency. Must be an integer not lesser than 32. It is 5000000 by default.

Black Stream - Incompatible Codec

In many cases the audio data from the camera is causing FFmpeg process to hang. You can avoid this by doing one of the following in the Monitor Settings in Shinobi.

  • Set Audio Codec in the Stream section to AAC
  • Set Audio Codec in the Stream section to No Audio

It's possible that the stream data coming from the camera isn't compatible with web browsers. Usually that is H.265. You have three choices when that occurs.

  • Change the camera's internal settings (easiest)
    • Open the camera's internal settings and change the encoding type to H.264. This option's labelling and location varies based on manufacturer.
  • Have Shinobi encode the data into a viewable format (bad for hardware, easy)
    • Just set the Video Codec in the Stream section to libx264
  • Use the Sub-stream from the camera to view in the browser with H.264 and still record in H.265.

Stuttering Streams, Dropping Streams

This can be a characteristic of different problems but usually just because of network connection issues from the client machine to the Shinobi server. The client machine is the machine you are using to view the Shinobi dashboard. Possible solutions to this are as follows.

  • Try setting the Stream Type to HLS.
    • This value is found in the Stream (blue) section of your Monitor Settings.
  • Try setting the Stream Type to MJPEG.
    • This will use a fair amount of CPU power and has many limitation on the client side, use with caution.
  • Check that no other applications have an active connection to your camera.
    • Some cameras only allow a certain number of connections at a time due to resource limits on the camera itself.
  • Check that your wiring is maintained and isn't causing any packets to drop.
    • To learn more about reading camera logs please review the upcoming section of this article.

Reolink Camera Stream Drop

Try disabling the Auto-Reboot feature from the camera's web panel. This option is not exposed in the mobile app.

How to save Camera Logs

Be careful not to leave this on indefinitely if your camera has connection issues even if appearing to be fine. You will just be saving a whole lot of needless information into the database.

  1. Open Monitor Settings for the camera. The wrench icon.
  2. Switch to Advanced view.
  3. Scroll down to Logging.
  4. Set to Warning and enable Save to SQL.
  5. Wait a few moments or until problem strikes.
  6. Click your Username in the top left then Logs.

Be careful not to forget that this is on after diagnosis. It may produce many log rows with many being useless information.

Getting logs from the Shinobi (camera.js) process

Open Terminal and run the following 

pm2 logs --lines 100

Does the camera stream work in other applications?

Let's assume you have the correct stream URL. Try to run the stream in one of the following applications.

  • VLC - media player by VideoLAN Organization - https://www.videolan.org
  • FFprobe - Stream analyzer included with FFmpeg and is meant to be used on command line. Shinobi has  GUI that can be accessed by clicking your username in the top left, then Probe.

The camera does not work in VLC or FFprobe

  1. Verify the network status of the camera.
    • You should be able to open the camera's IP Address in a web browser.
    • If your camera is connected by Ethernet then you should be able to see a light blinking (or solid if idle) on your internet router or network switch.
  2. Verify connection status by opening a Web Browser and navigating to the IP Address of your camera.
  3. Verify the type of stream coming from your camera by opening the internal settings.
    • Check for whether it is MJPEG or H.264. If possible to set H.264 then you should do for that.
    • If you are using H.265 you can set the Video Codec in the Stream section to Auto or libx264.
  4. Verify the stream URL of your camera.

The camera works in one or both of these applications but not in Shinobi

Please put the hammer away and try each of the following.

  • Turn off all other applications that are accessing the streams, like ZoneMinder or VLC.
  • Check the connection information. See below.
  • Check your version of FFmpeg or swap it for another build. See below.
  • Check your file system permissions. See below.

Checking the Connection Information

In the Connection section try a combination of the following suggested option changes.

  • Set Skip Ping to Yes, if still not working you should switch it to No.
  • Set Automatic to No and turn Force Port to Yes.
  • Try the Web Port of the Camera in the Port field if you are using the RTSP port. Web port is the port you use to open the camera's internal settings from a web browser. The default Web Port is 80. The default RTSP port is 554. Leave Protocol as it is.

Check FFmpeg and Swap it for a static version

Open Terminal and run the following to see the version of FFmpeg. The Version number will be at the top of the output. 

ffmpeg -version

If not at least version 3 or above then you need to update. There are three ways to achieve this.

  • Update your repositories to download the newest ffmpeg.
  • Install a static build with NPM.
    • Enter the Shinobi directory and run the following 
cd /home/Shinobi
npm install ffmpeg-static

To uninstall the NPM static build and use the build just run the following. 

cd /home/Shinobi
npm uninstall ffmpeg-static

Check file system permissions

Run the following commands to ensure your directories are accessible by Shinobi. 

sudo chmod -R 755 /dev/shm/streams
sudo chmod -R 755 /home/Shinobi/videos

More info you might want to look at