To demonstrate the importance of evaluating overall quality indicator reliability, in addition to component or variable level reliability, a comparison of interrater agreement on four chart-abstracted pneumonia-related processes of care was conducted. The hospital medical records of 356 Medicare patients' recent discharges for pneumonia were independently abstracted by different abstractors. Kappa, prevalence and bias-adjusted kappa, P(pos), P(neg), and the Bias Index were used to assess reliability of composite quality indicators and their components. The adjusted kappas for the data elements used to determine eligibility to receive as well as to derive the pneumonia-related processes of care ranged from 0.68 to 1.0. The adjusted kappa associated with overall eligibility to receive the pneumonia-related processes of care was 0.63. The kappa statistics for determining if processes of care were provided ranged from 0.56 to 0.83 and increased to 0.65 and 0.85 upon adjustment for the prevalence effect. Kappas for the composite quality indicators were lower, but improved with adjustment for the prevalence effect. The composite quality indicator with the highest adjusted kappa value was oxygenation assessment (0.93); the composite quality indicator with the lowest adjusted kappa value was antibiotic administration within 8 hours of hospital arrival (0.74). This study establishes the reliability of pneumonia indicators and underscores the need for reliability assessment at the quality indicator level, as well as at the component level.