LiveStream Technology

What is video latency?

Video latency is the difference in time taken between capturing a real time image and displaying the image onto a display. This difference is sometimes called 'action to eyeball' or 'glass to glass' latency, the camera lens to the display screen. If a signal does not require additional processing, the signal can simply be routed to the output device directly. When additional processing of the signal is required, for example text overlay, edge blending, encoding etc, the signal must be sampled into a system capable of these actions. Datapath Vision capture technology is used in solutions that require additional processing of the signal prior to display, disk archive or network transport.

When displaying video, latency can be split into three distinct latency stages, camera, process and display. Synchronisation of the data transfer between thses three stages is critical to minimising latency. Synchronisation of the input capture rate to the output display refresh rate is the responsibility of the system developer. If the input and output rate are not synchronised, an asynchronous system exists. An asynchronous system draws frames to the display at an independent rate to the input signal, processed or not.

Video rate management

An interesting example of an asynchronous capture display system is a 60 Hz input signal displayed at 59.97 Hz. In this case, the input rate is faster than the output rate, if not managed correctly, the 'process' and 'display' sections could buffer additional video frames, increasing latency. For the case of input signal rate faster than output, frames of video must be discarded or dropped. If the input rate is 59.97 Hz and the output rate is 60 Hz, the input rate is slower than the output rate. For this case, the display will be starved of video frames and be forced to play the same buffer every so often.

Audio play-out

For synchronised audio and video play-out at low latency, audio must be buffered with a duration no greater than 100 ms. A two frame video latency at 60 Hz is 33.2 ms ( 16.7 ms x2 ), a 100 ms audio delay is the maximum difference tolerable by the human ear for live playback. Many audio playback methods exist for Windows and Linux operating systems. The Windows Audio Session API (WASAPI) provides best results for low latency live audio streaming. Sample2 of the RGBEasy SDK shows how to interface to the WASAP

How to measure latency

Video latency is measured in frames, multiply the frame duration in milliseconds by the frame count for total latency in milliseconds. For a 60 Hz video source, the frame duration is 16.7 ms, therefore a latency of 3 frames using a 60 Hz input signal is 50 ms (3x16.7 ms). In an asynchronous system, 'glass to glass' latency has a tolerance of 1 frame time. Therefore a best case and worst case latency exists.

A common incorrect test to determine latency it to split a signal at its source, one input into a capture system and the second input directly to a display. The source shows a clock or timer, then the user takes a photo of both displays to calculate the difference. Using this method many factors must be considered, camera shutter speed, source clock update rate, where in the rendering operation the display is in relation to the position of the clock on the screen and the asynchronous nature of the signal as described above.

To measure both latency and rate management correctly, the input and output signal must be monitored using an oscilloscope. Passively converting the input signal and output signal to VGA analogue RGB and using a ramped greyscale input signal, buffers can be tracked by a scope when reading any of the RGB wires. The two frames of video latency shown below uses this method of buffer tracking.

LiveStream

Datapath LiveStream capture card technology allows developers to achieve lower latency when capturing video signals.

LiveStream is available on all video capture cards in the Vision series. Using various APIs including RGBEasy, Kernel Streaming, DirectX and OpenGL, low latency sub frame peer-to-peer DMA operations are achieved using a proprietary technology allowing capture lines (not frames) to be transferred as they arrive into the frame store. This process reduces the minimum DMA transfer time from a full frame time to a single line time. Datapath LiveStream technology is available for selection if and when required, a single client per input can be considered for LiveStream. Using Datapath LiveStream technology, all other clients (a further 15) would operate as usual in a 'round robin' transfer schedule immediately after the LiveStream client transfer. If the LiveStream client is disabled, all clients will DMA after the completed frame has finished capturing into the frame store. The capture card transfers data to system memory or off-screen memory on the graphics device, depending on the system and how the software is written.

The delay on a third-party graphics system is dependent on several factors, for example, if the data can be sent to the GPU directly or if transferred via system memory, how many buffers are being used to queue the incoming video stream.

The diagrams below shows a "before and after" example capture system using a video camera source.

Typical latency with LiveStream disabled:

Reduced latency with LiveStream enabled:

Datapath LiveStream can be used to output any of the available nine colour space buffer formats: YUY2, YUVY, UYVY, RGB32, RGB24, RGB565, RGB555, RGB8, and NV12. LiveStream can also up- and down-scale as part of the DMA operation. Down-scaling will incur no additional latency, up-scaling could under some circumstances increase latency.

Timing Diagram

The following diagram shows a scope trace taken with and without LiveStream enabled. The trace shows the increase in processing time prior to the arrival of the next frame. In a display system, LiveStream enables users to process captured frames longer without dropping frames.

Capture Card Latency

The Datapath Vision range of video capture cards are responsible for delivering the video frame to a user supplied buffer. Any processing and rendering operations of the buffer are ultimately the responsibility of the developer. The supplied Vision application and SDK samples show numerous low latency rendering methods in more detail. With LiveStream enabled, Datapath capture latency is a single frame time plus 1.2ms for last line transfer and the user notification call to complete. The dgcdiag transform filter installed as part of the DirectShow framework can be used to validate latency timings. Using GraphEdt.exe, connect the user mode source filter 'Capture' pin to the diagtrans filter input pin. Enable LiveStream in the user mode source filter property page. Turn off the DirectShow default clock under 'Graph->Use Clock'. Start the graph and open the dgctrans 'Filter Properties' dialog. The latency value if non-zero shows the latency from the first pixel input to DMA completion.

Two Frame Latency Example

The Two Frame Video Latency Example Video shows a 1920x1080p 60 Hz video source signal passing through a PCIe based system with a 'glass to glass' latency of only 2 frames (33.3 ms).

The source signal carries nine 1080p greyscale images in a loop at 60 Hz. For measurement purposes, the digital input signal to the Vision capture card is passively converted to VGA (no buffering) and additionally split to the oscilloscope as the 'green' input seen on the scope. The Vision capture card is driven using the Sample5 binary included with the RGBEasy SDK. The Vision capture card performs a 1:1 RGB32 peer-to-peer DMA to an AMD FirePro graphics device. The output of the graphics device is then split to the display and to the oscilloscope as the 'yellow' input. The scope trace shows a two frame input to output latency of 33.3 ms.

Similar results can be seen on Intel GPU-s and NVIDIA graphic devices.

Common Use Cases

Datapath LiveStream technology can be used in a number of solutions.

• Live Events - edge blend, warp and overlay
• Medical - overlay and annotation
• Conferencing - H.264/5 slice based encoding

A whitepaper proposal for further reductions in end to end latency can be found in here. The method describes how the output buffer could be split into multiple DMA slices for parallel access. Note that this document is a proposal and requires a new development based on user feedback.

SDK Details

LiveStream functionality is disabled by default. As previously stated, a LiveStream input client can affect the throughput of a second client of the same input. Enabling LiveStream can be performed using DirectShow, Media Foundation and RGBEasy API's.

Programmatic COM interfaces exist for both the Media Foundation and DirectShow frameworks

For Kernel Mode source filters use the programmatic IVisionPin COM interface.

For User Mode source filters use the programmatic IVisionUserPin COM interface.

Third party software developers should query both interfaces to determine the functionality. Users will be installed as either User or Kernel Mode filters, never both..

Further function documentation exists in the Vision Streaming SDK type library (.TLB).pin properties

[propget, id(9), helpstring("Live Stream")]

HRESULT LiveStream( [out, retval] LIVESTREAM_CLIENT *pLiveStreamClient );

[propput, id(9), helpstring("Live Stream")]

HRESULT LiveStream( [in] LIVESTREAM_CLIENT LiveStreamClient );

In addition, with User Mode source filters, LiveStream can be configured via the 'Pin Properties' dialogue as shown on the right:

RGBEasy

Sample3c and Sample5 of the RGBEasy SDK show examples of the LiveStream API calls listed below:

Sample3c and Sample5 also show how to rate manage video as described above for asynchronous input and output rates.

The latest version of the Datapath capture SDK for RGBEasy and Streaming API's are available as an install option within the latest Vision driver and application installation program.

For technical support contact technical@datapath.co.uk