Comparison studies in clinical medicine are typically carried out in one of two settings. In one, an accepted criterion standard exists and is available for comparison; in the other, no independently established criterion standard for comparison exists and a new test or method must be evaluated by comparison to an already established, though possibly inaccurate, technique. For the purposes of this study, we have considered multiwavelength spectrophotometry (eg, CO-Oximetry) as a criterion standard for the determination of hemoglobin oxygen saturation. The statistical methods that we have used to assess agreement between in vivo and in vitro oximetry are based on this assumption. However, for completeness, we have included a “difference versus mean” plot (Fig 1b) as recommended by Bland and Altman.
Statistical analysis of performance comparison data is further complicated by two problems. First, two statistical approaches are used to evaluate measurements for agreement. The first approach uses a traditional null hypothesis stating that there is no difference between the two measures, and a traditional alternative hypothesis of a nonzero difference. This null hypothesis is not rejected unless the measures are different with high probability. However, in measurement agreement studies, the goal of testing is to discern whether the measures are functionally equivalent.