OTT testing: HTTP Live Streaming (HLS)

Description

A Netrounds Test Agent can measure user experience for adaptive video streaming using HTTP Live Streaming (HLS). On being told the URL of the video, the Test Agent will parse the manifest file and start downloading the video segments. The algorithm will adapt to current network conditions and select the highest possible quality (bit rate), while avoiding buffering.

OTT is an abbreviation of "Over The Top" and refers to the content being delivered on top of an ordinary Internet service, unlike IPTV, which runs as a separate service. HLS is one common OTT protocol, specified by Apple Inc. and supported by all Apple mobile devices as well as by the Safari browser.

The HLS protocol supports multiple qualities with different bit rates, called variants, for the same video, and the client can choose to download segments from the variant that best matches network and client performance.

More general information about OTT and HLS is available in Wikipedia:

Setting up a test or monitoring

To run HLS measurements you need to have at least one Netrounds Test Agent installed. If you haven't already done the installation, consult our quick start guides for various types of Test Agents in the section Netrounds Test Agents.

Create a new test or monitoring and fill in the mandatory parameters below:

Parameters

Only tests

  • Duration (seconds): The duration of this test step in seconds. Min: 30 s. Max: 604800 s. Default: 60 s.
  • Fail threshold (seconds): The maximum number of errored seconds (ES) that may occur without triggering a fail for this test step. Default: 0.
  • Wait for ready: Time to wait before starting the test. The purpose of inserting a wait is to allow all Test Agents time to come online and acquire good time sync. Min: 1 min. Max: 24 hours. Default: "Don't wait", i.e. zero wait time.

General

  • Clients: Specify the Test Agents you want to use in the test or monitoring.
  • URL: Specify the stream URL. This can be either a playlist linking to other playlists for the different variants, or a playlist containing the segments. The file extension is .m3u or .m3u8.

Thresholds for errored seconds (ES)

  • Playback rate (Mbit/s): An errored second is triggered if the playback rate drops below this threshold. Min: 0 Mbit/s. No default.
  • Download rate (Mbit/s): An errored second is triggered if the download rate drops below this threshold. Min: 0 Mbit/s. No default.
  • Selected rate (Mbit/s): An errored second is triggered if the rate of the variant selected by the Test Agent is below this threshold. Min: 0 Mbit/s. No default.
  • Buffer size (seconds): An errored second is triggered if the duration of the data segment buffered in the Test Agent drops below this threshold. Min: 0 s. No default.

Advanced

  • Buffer size (seconds): Target duration of buffered data. When the duration of the buffered data falls below this value, new segments will be downloaded. Min: 0 s. Default: 60 s.
  • Initial buffering (seconds): The duration of the initial buffered data required before the playback starts. Min: 0 s. Default: 10 s.
  • Loop: If set to Yes, the video stream will loop when the end of the playlist is reached. If set to No, the playback will stop, and errored seconds will be triggered. The "No" option is relevant mainly for live streams which you expect never to end, and for which you want to trigger an alarm if that happens. The default is Yes.
  • Delayed start (seconds): (Tests only) Time by which to delay the start of the test within a test step. Default: 0 s.

SLA thresholds (monitorings only)

  • SLA Good: Threshold for good fulfillment of service level agreement. Default: 99.95%.
  • SLA Acceptable: Threshold for acceptable fulfillment of service level agreement. Default: 99.5%.

Result metrics

  • Playback rate (Mbit/s): Actual data rate of the video stream.
  • Download rate (Mbit/s): Download rate for the segments. The download is started when the amount of buffered data falls below the configured buffer size.
  • Selected rate (Mbit/s): Data rate of the selected variant in the manifest file.
  • (Min) Buffer (s): Actual buffer length measured in seconds.
  • ES playback rate (%): Percentage of seconds during which the playback rate dropped below the Playback rate threshold.
  • ES download rate (%): Percentage of seconds during which the download rate dropped below the Download rate threshold.
  • ES selected rate (%): Percentage of seconds during which the selected rate dropped below the Selected rate threshold.
  • ES buffer underrun (%): Percentage of seconds during which the buffered data dropped below the Buffer size threshold.
  • ES buffering (%): Percentage of seconds during which the buffer became empty, so that rebuffering was needed.
  • ES total: Aggregated errored second percentage, taking into account all types of error.
  • SLA: Service level agreement fulfillment for the test or monitoring: equal to (100 – ES total) %.

Example

Below is an example of OTT testing.

The first graph shows how OTT data is downloaded in bursts (red), adding to the amount of data stored in the buffer (gray). The buffer is gradually emptied as content is played back; when the buffer level goes down to Buffer size, another batch of data is downloaded, etc.

The second graph shows the actual playback rate (green) compared to the playback rate currently selected by the client (blue).

Have more questions? Submit a request

Comments

Powered by Zendesk