Quality of Outcome (QoO)

4.1. Requirements

This section describes the requirements for an objective network quality framework and metric that is useful for end-users, application developers, and network operators/vendors alike. Specifically, this section outlines the three main requirements and their motivation.¶

In general, all stakeholders ultimately care about the performance of applications running over a network. Application performance does not only depend on bandwidth but also on the delay and delay variation of network links and computational steps involved in making the application function. These delays depend on how the application places load on the network, how the network is affected by environmental conditions, and the behavior of other users and applications sharing the network resources. Likewise, packet loss, e.g., caused by congestion, can also negatively impact application performance in different ways depending on the class of application.¶

Different applications may have different needs from a network and may impose different patterns of load. To determine whether an application will likely work well or fail, a network quality framework must compare measurements of network performance to a wide variety of application requirements. It is important that these measurements reflect the actual network service configuration that will handle the application flows, including any traffic prioritization, network slicing, VPN services, or other differentiated service mechanisms (see Section 8.1). Flexibility in describing application requirements and the ability to capture the delay and loss characteristics of a network with sufficient accuracy and precision are necessary to compute a meaningful QoO network quality score that can be used to better estimate application performance.¶

The framework must also support spatial composition [RFC6049], [RFC6390] to enable operators to take actions when measurements show that applications fail too often. In particular, spatial composition allows results to be divided into sub-results, each measuring the performance of a required sub-milestone that must be reached in time for the application to perform adequately.¶

To summarize, the QoO framework and the corresponding QoO score should have the following properties in order to be meaningful:¶

1. Capture a set of network performance metrics which provably correlate to the application performance of a set of different applications as perceived by users.¶

2. Compare meaningfully to different application requirements.¶

3. Be composable so operators can isolate and quantify the contributions of different sub-outcomes and sub-paths of the network.¶

Next, the document focuses on the requirements of each of the mentioned target groups.¶

5.1. Discussion of Other Network Quality Metrics

Numerous network quality metrics and associated frameworks have been proposed, adopted, and, at times, misapplied over the years. The following is a brief overview of several key network quality metrics.¶

Each metric is evaluated against the three criteria established in Section 4.1. Table 1 summarizes the properties of each of the surveyed metrics.¶

The column "Can Assess How Well Applications Are Expected to Work" indicates whether a metric can, in principle, capture relevant information to assess application performance, assuming that measurements cover the properties of the end-to-end network path that the application uses. "Easy to Articulate Application Requirements" refers to the ease with which application-specific requirements can be expressed using the respective metric. "Composable" indicates whether the metric supports mathematical composition to enable detailed network analysis.¶

6.1. Sampling Requirements

To ensure broad applicability across diverse use cases, the QoO framework deliberately avoids prescribing specific conditions for sampling, such as fixed time intervals or defined network load levels. This flexibility enables deployment in both controlled and production environments.¶

For a complete assessment, the QoO framework requires a latency distribution, packet loss measurements, and throughput estimates as described in Section 6.2. When measurements are taken during periods of network load, the result naturally includes latency under load. In scenarios such as passive monitoring of production traffic, capturing artificially loaded conditions may not always be feasible, whereas passively observing the actual network load may be possible.¶

Modeling the full latency distribution may be too complex to allow for easy adoption of the framework. Instead, reporting latency at selected percentiles offers a practical compromise between accuracy and deployment considerations. A commonly accepted set of percentiles spanning from the 0th to the 100th in a logarithmic-like progression has been suggested by others [BITAG] and is recommended here: [0th, 10th, 25th, 50th, 75th, 90th, 95th, 99th, 99.9th, 100th].¶

The framework is agnostic to traffic direction but mandates that measurements specify whether latency is one-way or round-trip.¶

Importantly, the framework does not enforce a minimum sample count. This means that even a small number of samples (e.g., 10) could technically constitute a distribution but such cases are clearly insufficient for statistical confidence. The intent is to balance rigor with practicality, recognizing that constraints vary across devices, applications, and deployment environments.¶

To support reproducibility and enable confidence analysis, each measurement must be accompanied by the following metadata:¶

• Description of the measurement path, including the endpoints (source and destination), network segments traversed, measurement points (if applicable), and direction (uplink, downlink, or bidirectional)¶

• Timestamp of first sample¶

• Total duration of the sampling period¶

• Number of samples collected¶

• Sampling method, including:¶

  • Cyclic: One sample every N milliseconds (specify N)¶

  • Burst: X samples every N milliseconds (specify X and N)¶

  • Passive: Opportunistic sampling of live traffic (non-uniform intervals)¶

These metadata elements are essential for interpreting the precision and reliability of the measurements. As demonstrated in [QoOSimStudy], low sampling frequencies and short measurement durations can lead to misleadingly optimistic or imprecise QoO scores.¶

To assess the precision of network performance measurements, implementers should consider:¶

• The repeatability of measurements under similar network conditions¶

• The impact of sampling frequency and duration on percentile estimates, particularly for high percentiles (e.g., 99th, 99.9th)¶

• The measurement uncertainty introduced by hardware/software timing jitter, clock synchronization errors, and other system-level noise sources¶

• The statistical confidence intervals for percentile estimates based on sample size¶

Acceptable levels of precision depend on the use case. Implementers should document their precision assessment methodology and report precision metrics alongside QoO scores when precision is critical for the use case.¶

6.2. Describing Network Performance Requirements

The QoO framework builds upon the work already proposed in the Broadband Forum standard called Quality of Experience Delivered (QED) [TR-452.1], which defines the Quality Attenuation metric. Correspondingly, QoO expresses network performance requirements as a set of percentile-latency tuples with corresponding packet loss thresholds and a minimum required throughput. For example, a requirement might state: at 4Mbps, 90% of packets must arrive within 100ms, and 100% within 200ms, implying 0% packet loss. This list can be minimal (e.g., 100% within 200ms) or extended as needed and different percentiles may be used to characterize different applications. Still, it might be beneficial for future standardization activities to converge on a fixed set of general percentiles or for specific applications/ application classes to make QoO measurements between different providers more comparable. For the sake of simplicity, this document only states that the latency percentiles in the requirements must match one or more of the percentiles defined in the measurements, i.e., one can set requirements at the [0th, 10th, 25th, 50th, 75th, 90th, 95th, 99th, 99.9th, 100th] percentiles. Packet loss rates and bandwidth must be reported as separate values.¶

Applications do of course have throughput requirements, and thus a complete framework for application-level network quality must also take capacity into account. Insufficient bandwidth may give unacceptable application outcomes without necessarily inducing a lot of latency or packet loss. Therefore, the network requirements must include a minimum throughput requirement. A fully specified requirement can be thought of as specifying the latency and loss requirements to be met while the end-to-end network path is loaded in a way that is at least as demanding of the network as the application itself. This may be achieved by running the actual application and measuring delay and loss alongside it, or by generating artificial traffic to a level at least equivalent to the application traffic load.¶

Whether the requirements are one-way or two-way must be specified. Where the requirement is one-way, the direction (user-to-network or network-to-user) must be specified. In case of a two-way requirement, a decomposition into uplink and downlink components may be specified.¶

Network performance requirements and measurements are already standardized in the QED framework [TR-452.1]. This document extends the QED framework with a method that translates the network performance requirements and measurements into a network quality score that quantifies how close the provided network conditions are to the optimal conditions specified by the requirements.¶

To that aim, first recall the key design goal of establishing a quantifiable distance between optimal and unacceptable network conditions, thereby enabling an objective assessment of relative quality. Accordingly, the requirements specification is extended to define both the network performance required for achieving optimal application performance and the lower network performance threshold below which the application performance is considered unacceptable.¶

The two ends of the distance measure correspond to the Requirements for Optimal Performance (ROP) and the Conditions at the Point of Unacceptable Performance (CPUP). For example, ROP could be defined as: at 4Mbps, 99% of packets need to arrive within 100ms, 99.9% within 200ms, and 0.1% packet loss is acceptable for the outcome to be as intended. Similarly, CPUP could be defined as: if 99% of the packets have not arrived after 200ms, or 99.9% within 300ms, the perceived service will be unacceptable.¶

If a latency percentile is included in the ROP, it must also be defined in the CPUP, and vice versa, i.e., neither specification should define a percentile that is not present in the other. For example, if the 99.9th percentile is part of the CPUP then the ROP must also include the 99.9th percentile.¶

The derivation of ROP and CPUP values requires standardized testing conditions to ensure consistency and accuracy. Application developers should publish their testing methodologies, including the network conditions, hardware configurations, and measurement procedures used to establish these thresholds. Without such standardization, the overall accuracy and precision of QoO scores may be reduced due to variations in testing approaches across different applications and developers.¶

Developers are encouraged to follow relevant standards for testing methodologies, such as ITU-T P-series recommendations for subjective quality assessment ([P.800], [P.910], [P.1401]) and IETF IPPM standards for network performance measurement ([RFC7679], [RFC7680], [RFC6673]). These standards provide guidance on test design, measurement procedures, and statistical analysis that can help ensure consistent and reproducible threshold definitions.¶

6.3. Creating Network Performance Requirement Specifications

This document does not define a standardized approach for creating a quality-focused network performance requirement specification. Instead, this section provides general guidance on and a rough outline for deriving an admittingly subjective requirement specification, aiming to create a basis for future standardization efforts focusing on developing a standardized, objective requirement creation framework. Additional information is provided in [QoOAppQualityReqs]. Direct use of the approach described below in production scenarios is discouraged.¶

When determining quality-focused network performance requirements for an application, the goal is to identify the network conditions where application performance is optimal and where it becomes unacceptable. There is no universally strict threshold at which network conditions render an application unacceptable. For optimal performance, some applications may have clear definitions, but for others, such as web browsing and gaming, lower latency and loss is always preferable.¶

One approach for deriving possible thresholds is to run the application over a controlled network segment with adjustable quality and then vary the network conditions while continuously observing the resulting application-level performance. The latter can be assessed manually by the entity performing the testing or using automated methods, such as recording video stall duration within a video player. Additionally, application developers could set thresholds for acceptable fps, animation fluidity, i/o latency (voice, video, actions), or other metrics that directly affect the user experience and measure these user-facing metrics during tests to correlate the metrics with the network conditions.¶

Using this scenario, one can first establish a baseline under excellent network conditions. Network conditions can then be gradually worsened by adding delay or packet loss or decreasing network capacity until the application no longer performs optimally. The corresponding network conditions identify the minimal requirements for optimal performance (ROP). Continuing to worsen the network conditions until the application fails completely eventually yields the network conditions at the point of unacceptable performance (CPUP).¶

Note that different users may have different tolerance levels for application degradation. Hence, tests conducted by a single entity likely result in highly subjective thresholds. The thresholds established should represent acceptable performance for the target user base, which may require user studies or market research to determine appropriate values.¶

As stated at the beginning of this section, this document does not define a standardized approach for creating a quality-focused network performance requirement specification and directly using the approach described above is discouraged.¶

7.1. Calculation

7.2. Example

The following example illustrates the QoO calculations.¶

Example requirements and measured data:¶

• ROP: 4Mbps {99%, 200ms}, {99.9%, 300ms} 1% loss¶

• CPUP: {99%, 500ms}, {99.9%, 600ms} 5% loss¶

• Measured Latency: 99% = 350ms, 99.9% = 375ms¶

• Measured Packet Loss: 2%¶

• Measured Minimum Bandwidth: 32Mbps / 28Mbps¶

QoO_latency = min(min(max((1 - (350ms - 200ms) / (500ms - 200ms)) * 100, 0), 100), min(max((1 - (375ms - 300ms) / (600ms - 300ms)) * 100, 0), 100)) = min(50.00, 75.00) = 50.00¶

QoO_loss = min(max((1 - (2% - 1%) / (5% - 1%)) * 100, 0), 100) = 75.00¶

QoO = min(QoO_latency, QoO_loss) = min(50.00, 75.00) = 50.00¶

In this example, the network scores 50% on the QoO assessment range between unacceptable and optimal for the given application when using the measured network and considering both latency and packet loss. The score implies that the latency impact dominates the packet loss impact and that the network overall provides conditions at the midway point of the performance range.¶

8.1. Deployment Considerations

The QoO framework assumes that measurements reflect the actual connectivity service that will be provided to application flows. However, networks may offer multiple connectivity service levels (e.g., VPN services, corporate customer tiers, network slicing configurations). In such deployments, it is important to ensure that:¶

• Measurements are taken using the same connectivity service level that will be used by the application¶

• The measurement methodology accounts for any traffic prioritization, differentiated services, or quality-of-service mechanisms that may affect application performance¶

• Network configurations and policies that will apply to application traffic are reflected in the measurement conditions¶

Failing to align measurements with the actual service delivery may result in QoO scores that do not accurately reflect the application's expected performance.¶

8.2. Adaptive Applications

Many modern applications are adaptive, meaning they can adjust their behavior based on network conditions. For example, video streaming applications may reduce bit rate when bandwidth is limited, or increase buffer size when latency is high.¶

For adaptive applications, there are typically different levels of optimal performance rather than a single absolute threshold. For example, a video streaming application might provide different available video resolutions, ranging from 4K to 480p resolution. Combined with different transmission latencies, each of these resolutions can induce varying levels of perceived usability.¶

The QoO framework can accommodate such applications by defining multiple ROP/CPUP thresholds corresponding to different quality levels. The framework can then assess how well the application will achieve each quality level, providing a more nuanced view of application performance than a simple binary pass/fail metric. Another, less complex approach at the cost of reduced fidelity in the QoO score, is to set the threshold for optimal performance at the highest rendition available for the video stream, and the threshold for unacceptability where the lowest rendition cannot be delivered without resulting in stalling events.¶

Application developers implementing adaptive applications should consider publishing quality profiles that define network performance requirements for different adaptation levels, enabling more accurate QoO assessment.¶

8.3. Sensitity to Sampling Accuracy

While the QoO framework itself places no strict requirement on sampling patterns or measurement technology, a simulation study [QoOSimStudy] conducted to inform the creation of this document examined the metric's real-world applicability under varying conditions and made the following conclusions:¶

1. Sampling Frequency: Slow sampling rates (e.g., <1Hz) risk missing rare, short-lived latency spikes, resulting in overly optimistic QoO scores.¶

2. Measurement Noise: Measurement errors on the same scale as the thresholds (ROP, CPUP) can distort high-percentile latencies and cause artificially lower QoO.¶

3. Requirement Specification: Slightly adjusting the latency thresholds or target percentiles can cause significant changes in QoO, especially when the measurement distribution is near a threshold.¶

4. Measurement Duration: Shorter tests with sparse sampling tend to underestimate worst-case behavior for heavy-tailed latency distributions, biasing QoO in a positive direction.¶

In summary, overly noisy or inaccurate latency samples can artificially inflate worst-case percentiles, thereby driving QoO scores lower than actual network conditions would warrant. Conversely, coarse measurement intervals can miss short-lived spikes entirely, resulting in an inflated QoO.¶

From these findings, we deduce the following guidelines for practical application:¶

• Calibrate the combination of sampling rate and total measurement period to capture fat-tailed distributions of latency with sufficient accuracy.¶

• Avoid or account for significant measurement noise where possible (e.g., by calibrating time sources, accounting for clock drift, considering hardware/software measurement jitter).¶

• Thoroughly test application requirement thresholds so that the resulting QoO scores accurately reflect application performance.¶

These guidelines are non-normative but reflect empirical evidence on how QoO performs.¶

8.4. Insights From User Testing

While subjective QoE testing as specified in the ITU-T P-series recommendations ([P.800], [P.910], [P.1401]) is out of scope of this document, a study involving 25 participants tested the QoO framework in real-world settings [QoOUserStudy]. Participants used specially equipped routers in their homes for ten days, providing both network performance data and feedback through pre- and post-trial surveys.¶

Participants found QoO scores more intuitive and actionable than traditional metrics (e.g., speed tests). QoO directly aligned with their self-reported experiences, increasing trust and engagement.¶

These results indicate that users find it easier to correlate QoO scores with real-world application performance than, for example, a speed test. As such, QoO is expected to help bridge technical metrics with application performance. However, the specific impact of QoO should be studied further, for example, via comparative studies with blinded methodologies that compare QoO to other QoS-type approaches or application-provided QoE ratings as the mentioned study's design might have introduced different forms of bias.¶

9.1. Volatile Networks

Volatile networks - in particular, mobile cellular networks - pose a challenge for network quality prediction, with the level of assurance of the prediction likely to decrease as session duration increases. Historic network conditions for a given cell may help indicate times of network load or reduced transmission power, and their effect on throughput/latency/loss. However, as terminals are mobile, the signal bandwidth available to a given terminal can change by an order of magnitude within seconds due to physical radio factors. These include whether the terminal is at the edge of a cell for a radio network, or undergoing cell handover, the radio interference and fading from the local environment, and any switch between radio bearers with differing signal bandwidth and transmission-time intervals (e.g., 3GPP 4G and 5G). This suggests a requirement for measuring Quality Attenuation to and from an individual terminal, as that can account for the factors described above. How that facility is provisioned onto individual terminals and how terminal-hosted applications can trigger a Quality Attenuation query, is an open question.¶

9.2. Missing Temporal Information in Distributions.

The two latency series (1,200,1,200,1,200,1,200,1,200) and (1,1,1,1,1,200,200,200,200,200) have identical distributions, but may have different application performance. Ignoring this information is a tradeoff between simplicity and precision. To capture all information necessary to adequately capture outcomes quickly gets into extreme levels of overhead and high computational complexity. An application's performance depends on reactions to varying network conditions, meaning nearly all different series of latencies may have different application outcomes.¶

9.3. Subsampling the Real Distribution

Additionally, it is not feasible to capture latency for every packet transmitted. Probing and sampling can be performed, but some aspects will always remain unknown. This introduces an element of uncertainty and perfect predictions cannot be achieved; rather than disregarding this reality, it is more practical to acknowledge it. Therefore, discussing the assessment of outcomes provides a more accurate and meaningful approach.¶

9.4. Assuming Linear Relationship Between Optimal Performance and Unusable

It has been shown that, for example, interactivity cannot be modeled by a linear scale [G.1051]. Thus, the linear modeling proposed here adds an error in estimating the perceived performance of interactive applications.¶

One can conjure up scenarios where 50ms latency is actually worse than 51ms latency as developers may have chosen 50ms as the threshold for changing quality, and the threshold may be imperfect. Taking these scenarios into account would add another magnitude of complexity to determining network performance requirements and finding a distance measure (between requirement and actual measured capability).¶

9.5. Binary Bandwidth Threshold

Choosing a binary bandwidth threshold is to reduce complexity, but it must be acknowledged that many applications are not that simple. Network requirements can be set up per quality level (resolution, frames per-second, etc.) for the application if necessary.¶

9.6. Arbitrary Selection of Percentiles

A selection of percentiles is necessary for simplicity, because more complex methods may slow adoption of the framework. The 0th (minimum) and 50th (median) percentiles are commonly used for their inherent significance. According to [BITAG], the 90th, 98th, and 99th percentiles are particularly important for certain applications. Generally, higher percentiles provide more insight for interactive applications, but only up to a certain threshold beyond which applications may treat excessive delays as packet loss and adapt accordingly. The choice between percentiles such as the 95th, 96th, 96.5th, or 97th is not universally prescribed and may vary between application types. Therefore, percentiles must be selected arbitrarily, based on the best available knowledge and the intended use case.¶

10.1. Measurement Integrity and Authenticity

QoO relies upon accurate and trustworthy measurements of network performance. If an attacker can manipulate these measurements, either by injecting falsified data or tampering with the measurement process, they could distort the resulting QoO scores. This could mislead users, operators, or regulators into making incorrect assessments of network quality.¶

To mitigate this risk:¶

• Measurement agents have to authenticate with the systems collecting or analyzing QoO data.¶

• Measurement data has to be transmitted over secure channels (e.g., (D)TLS) to ensure confidentiality and integrity.¶

• Digital signatures may be used to verify the authenticity of measurement reports.¶

10.2. Risk of Misuse and Gaming

As QoO scores may influence regulatory decisions, SLAs, or user trust, there is a risk that network operators or application developers might attempt to "game" the system. For example, they might optimize performance only for known test conditions or falsify requirement thresholds to inflate QoO scores.¶

Mitigations include:¶

• Independent verification of application requirements and measurement methodologies.¶

• Use of randomized testing procedures.¶

• Transparency in how QoO scores are derived and what assumptions are made.¶

10.3. Denial of Service (DoS) Risks

Active measurement techniques used to gather QoO data (e.g., TWAMP, STAMP, and synthetic traffic generation) can place additional load on a network. If not properly rate-limited, this may inadvertently degrade services offered by a network or be exploited by malicious actors to launch DoS attacks.¶

To mitigate these risks, the following is recommended:¶

• Implement rate limiting and access control for active measurement tools.¶

• Ensure that measurement traffic does not interfere with critical services.¶

• Monitor for abnormal measurement patterns that may indicate abuse.¶

10.4. Trust in Application Requirements

QoO depends on application developers to define ROP and CPUP. If these are defined inaccurately-either unintentionally or maliciously-the resulting QoO scores may be misleading.¶

To address such risks, the following recommendations are made:¶

• Encourage peer review and publication of application requirement profiles.¶

• Where QoO is used for regulatory or SLA enforcement, require independent validation of requirement definitions.¶

13.1. qoo-c

• Link to the open-source repository:¶

  https://github.com/getCUJO/qoo-c¶

• The organization responsible for the implementation:¶

  CUJO AI¶

• A brief general description:¶

  A C library for calculating Quality of Outcome¶

• The implementation's level of maturity:¶

  A complete implementation of the specification described in this document¶

• Coverage:¶

  The library is tested with unit tests¶

• Licensing:¶

  MIT¶

• Implementation experience:¶

  Tested by the author. Needs additional testing by third parties.¶

• Contact information:¶

  Bjørn Ivar Teigen Monclair: bjorn.monclair@cujo.com¶

• The date when information about this particular implementation was last updated:¶

  27th of May 2025¶

13.2. goresponsiveness

• Link to the open-source repository:¶

  https://github.com/network-quality/goresponsiveness¶

  The specific pull-request: https://github.com/network-quality/goresponsiveness/pull/56¶

• The organization responsible for the implementation:¶

  University of Cincinatti for goresponsiveness as a whole, Domos for the QoO part.¶

• A brief general description:¶

  A network quality test written in Go. Capable of measuring RPM and QoO.¶

• The implementation's level of maturity:¶

  Under active development; partial QoO support integrated.¶

• Coverage:¶

  The QoO part is tested with unit tests¶

• Licensing:¶

  GPL 2.0¶

• Implementation experience:¶

  Needs testing by third parties¶

• Contact information:¶

  Bjørn Ivar Teigen Monclair: bjorn.monclair@cujo.com¶

  William Hawkins III: hawkinwh@ucmail.uc.edu¶

• The date when information about this particular implementation was last updated:¶

  10th of January 2024¶

14.1. Normative References

[RFC6049]

Morton, A. and E. Stephan, "Spatial Composition of Metrics", RFC 6049, DOI 10.17487/RFC6049, January 2011, <https://www.rfc-editor.org/rfc/rfc6049>.

[RFC6390]

Clark, A. and B. Claise, "Guidelines for Considering New Performance Metric Development", BCP 170, RFC 6390, DOI 10.17487/RFC6390, October 2011, <https://www.rfc-editor.org/rfc/rfc6390>.

[TR-452.1]

Broadband Forum, "TR-452.1: Quality Attenuation Measurement Architecture and Requirements", September 2020, <https://www.broadband-forum.org/download/TR-452.1.pdf>.

14.2. Informative References

[BITAG]

BITAG, "Latency Explained", October 2022, <https://www.bitag.org/documents/BITAG_latency_explained.pdf>.

[Bufferbloat]

"Bufferbloat: Dark buffers in the Internet", n.d., <https://queue.acm.org/detail.cfm?id=2071893>.

[CSGO]

Xu, X., Liu, S., and M. Claypool, "The Effects of Network Latency on Counter-strike: Global Offensive Players", IEEE, 2022 14th International Conference on Quality of Multimedia Experience (QoMEX) pp. 1-6, DOI 10.1109/qomex55416.2022.9900915, September 2022, <https://doi.org/10.1109/qomex55416.2022.9900915>.

[G.1051]

ITU-T, "Latency measurement and interactivity scoring under real application data traffic patterns", ITU-T G.1051, March 2023, <https://www.itu.int/rec/T-REC-G.1051>.

[Haeri22]

"Mind Your Outcomes: The ΔQSD Paradigm for Quality-Centric Systems Development and Its Application to a Blockchain Case Study", n.d., <https://www.mdpi.com/2073-431X/11/3/45>.

[I-D.draft-ietf-opsawg-rfc5706bis]

Claise, B., Clarke, J., Farrel, A., Barguil, S., Pignataro, C., and R. Chen, "Guidelines for Considering Operations and Management in IETF Specifications", Work in Progress, Internet-Draft, draft-ietf-opsawg-rfc5706bis-01, 17 December 2025, <https://datatracker.ietf.org/doc/html/draft-ietf-opsawg-rfc5706bis-01>.

[IRTT]

"Isochronous Round-Trip Tester", n.d., <https://github.com/heistp/irtt>.

[ISO5725-1]

ISO, "Accuracy (trueness and precision) of measurement methods and results Part 1: General principles and definitions", ISO 5725-1:2023, July 2022, <https://www.iso.org/standard/69418.html>.

[JamKazam]

Wilson, D., "What is Latency and Why does it matter?", n.d., <https://jamkazam.freshdesk.com/support/solutions/articles/66000122532-what-is-latency-why-does-it-matter-?>.

[Kelly]

Kelly, F. P., "Networks of Queues", n.d., <https://www.cambridge.org/core/journals/advances-in-applied-probability/article/abs/networks-of-queues/38A1EA868A62B09C77A073BECA1A1B56>.

[P.10]

ITU-T, "Vocabulary for performance, quality of service and quality of experience", ITU-T P.10/G.100, November 2017, <https://www.itu.int/rec/T-REC-P.10>.

[P.1401]

ITU-T, "Methods, metrics and procedures for statistical evaluation, qualification and comparison of objective quality prediction models", ITU-T P.1401, January 2020, <https://www.itu.int/rec/T-REC-P.1401>.

[P.800]

ITU-T, "Methods for subjective determination of transmission quality", ITU-T P.800, August 1996, <https://www.itu.int/rec/T-REC-P.800>.

[P.800.1]

ITU-T, "Mean opinion score (MOS) terminology", ITU-T P.800.1, July 2016, <https://www.itu.int/rec/T-REC-P.800.1>.

[P.910]

ITU-T, "Subjective video quality assessment methods for multimedia applications", ITU-T P.910, October 2023, <https://www.itu.int/rec/T-REC-P.910>.

[QoOAppQualityReqs]

Østensen, T., "Performance Measurement of Web Applications", n.d., <https://domos.ai/storage/U6TlxIlbcl1dQfcNhnCleziJWF23P5w0xWzOARh8-published.pdf>.

[QoOSimStudy]

Monclair, B. I. T., "Quality of Outcome Simulation Study", n.d., <https://github.com/getCUJO/qoosim>.

[QoOUserStudy]

Monclair, B. I. T., "Application Outcome Aware Root Cause Analysis", n.d., <https://domos.ai/storage/LaiW4tJQ2kj4OOTiZbnf48MbS22rQHcZQmCriih9-published.pdf>.

[RFC2681]

Almes, G., Kalidindi, S., and M. Zekauskas, "A Round-trip Delay Metric for IPPM", RFC 2681, DOI 10.17487/RFC2681, September 1999, <https://www.rfc-editor.org/rfc/rfc2681>.

[RFC3393]

Demichelis, C. and P. Chimento, "IP Packet Delay Variation Metric for IP Performance Metrics (IPPM)", RFC 3393, DOI 10.17487/RFC3393, November 2002, <https://www.rfc-editor.org/rfc/rfc3393>.

[RFC5357]

Hedayat, K., Krzanowski, R., Morton, A., Yum, K., and J. Babiarz, "A Two-Way Active Measurement Protocol (TWAMP)", RFC 5357, DOI 10.17487/RFC5357, October 2008, <https://www.rfc-editor.org/rfc/rfc5357>.

[RFC5481]

Morton, A. and B. Claise, "Packet Delay Variation Applicability Statement", RFC 5481, DOI 10.17487/RFC5481, March 2009, <https://www.rfc-editor.org/rfc/rfc5481>.

[RFC6673]

Morton, A., "Round-Trip Packet Loss Metrics", RFC 6673, DOI 10.17487/RFC6673, August 2012, <https://www.rfc-editor.org/rfc/rfc6673>.

[RFC7679]

Almes, G., Kalidindi, S., Zekauskas, M., and A. Morton, Ed., "A One-Way Delay Metric for IP Performance Metrics (IPPM)", STD 81, RFC 7679, DOI 10.17487/RFC7679, January 2016, <https://www.rfc-editor.org/rfc/rfc7679>.

[RFC7680]

Almes, G., Kalidindi, S., Zekauskas, M., and A. Morton, Ed., "A One-Way Loss Metric for IP Performance Metrics (IPPM)", STD 82, RFC 7680, DOI 10.17487/RFC7680, January 2016, <https://www.rfc-editor.org/rfc/rfc7680>.

[RFC7799]

Morton, A., "Active and Passive Metrics and Methods (with Hybrid Types In-Between)", RFC 7799, DOI 10.17487/RFC7799, May 2016, <https://www.rfc-editor.org/rfc/rfc7799>.

[RFC7942]

Sheffer, Y. and A. Farrel, "Improving Awareness of Running Code: The Implementation Status Section", BCP 205, RFC 7942, DOI 10.17487/RFC7942, July 2016, <https://www.rfc-editor.org/rfc/rfc7942>.

[RFC8033]

Pan, R., Natarajan, P., Baker, F., and G. White, "Proportional Integral Controller Enhanced (PIE): A Lightweight Control Scheme to Address the Bufferbloat Problem", RFC 8033, DOI 10.17487/RFC8033, February 2017, <https://www.rfc-editor.org/rfc/rfc8033>.

[RFC8239]

Avramov, L. and J. Rapp, "Data Center Benchmarking Methodology", RFC 8239, DOI 10.17487/RFC8239, August 2017, <https://www.rfc-editor.org/rfc/rfc8239>.

[RFC8290]

Hoeiland-Joergensen, T., McKenney, P., Taht, D., Gettys, J., and E. Dumazet, "The Flow Queue CoDel Packet Scheduler and Active Queue Management Algorithm", RFC 8290, DOI 10.17487/RFC8290, January 2018, <https://www.rfc-editor.org/rfc/rfc8290>.

[RFC8517]

Dolson, D., Ed., Snellman, J., Boucadair, M., Ed., and C. Jacquenet, "An Inventory of Transport-Centric Functions Provided by Middleboxes: An Operator Perspective", RFC 8517, DOI 10.17487/RFC8517, February 2019, <https://www.rfc-editor.org/rfc/rfc8517>.

[RFC8762]

Mirsky, G., Jun, G., Nydell, H., and R. Foote, "Simple Two-Way Active Measurement Protocol", RFC 8762, DOI 10.17487/RFC8762, March 2020, <https://www.rfc-editor.org/rfc/rfc8762>.

[RFC9000]

Iyengar, J., Ed. and M. Thomson, Ed., "QUIC: A UDP-Based Multiplexed and Secure Transport", RFC 9000, DOI 10.17487/RFC9000, May 2021, <https://www.rfc-editor.org/rfc/rfc9000>.

[RFC9197]

Brockners, F., Ed., Bhandari, S., Ed., and T. Mizrahi, Ed., "Data Fields for In Situ Operations, Administration, and Maintenance (IOAM)", RFC 9197, DOI 10.17487/RFC9197, May 2022, <https://www.rfc-editor.org/rfc/rfc9197>.

[RFC9312]

Kühlewind, M. and B. Trammell, "Manageability of the QUIC Transport Protocol", RFC 9312, DOI 10.17487/RFC9312, September 2022, <https://www.rfc-editor.org/rfc/rfc9312>.

[RFC9318]

Hardaker, W. and O. Shapira, "IAB Workshop Report: Measuring Network Quality for End-Users", RFC 9318, DOI 10.17487/RFC9318, October 2022, <https://www.rfc-editor.org/rfc/rfc9318>.

[RFC9341]

Fioccola, G., Ed., Cociglio, M., Mirsky, G., Mizrahi, T., and T. Zhou, "Alternate-Marking Method", RFC 9341, DOI 10.17487/RFC9341, December 2022, <https://www.rfc-editor.org/rfc/rfc9341>.

[RFC9817]

Kunze, I., Wehrle, K., Trossen, D., Montpetit, M., de Foy, X., Griffin, D., and M. Rio, "Use Cases for In-Network Computing", RFC 9817, DOI 10.17487/RFC9817, August 2025, <https://www.rfc-editor.org/rfc/rfc9817>.

[RPM]

"Responsiveness under Working Conditions", July 2022, <https://datatracker.ietf.org/doc/html/draft-ietf-ippm-responsiveness>.

[RRUL]

"Real-time response under load test specification", n.d., <https://www.bufferbloat.net/projects/bloat/wiki/RRUL_Spec/>.

[XboxNetReqs]

Microsoft, "Understanding your remote play setup test results", n.d., <https://support.xbox.com/en-US/help/hardware-network/connect-network/console-streaming-test-results>.