Estimation of measurement error in binary data in surveys

Document Type : Original Scientific Paper

Author

Department of Statistics‎, ‎Faculty of Mathematical Sciences‎, ‎Alzahra University‎, ‎Tehran‎, ‎Iran

Abstract

Measurement error is an inherent and unavoidable component of nonsampling error in surveys‎, ‎and its estimation is essential for assessing the quality of survey results‎. ‎This paper investigates two agreement criteria‎, ‎Cohen’s Kappa and Gwet’s AC1‎, ‎for quantifying measurement error in binary survey data‎. ‎A theoretical comparison is first presented to highlight the conceptual differences between the two coefficients‎. ‎This is followed by a simulation study in which true binary values are generated under varying prevalence levels‎, ‎and observed responses are obtained through controlled misclassification mechanisms characterized by specified levels of observed agreement‎. ‎The performance and empirical variances of both agreement measures are evaluated and the results demonstrate that Gwet’s AC1 provides more stable agreement estimates and variance behavior than Cohen’s Kappa‎, ‎particularly under conditions of extreme prevalence.

Keywords

Main Subjects

References

Alimohammadi, R. and Navvabpour, H. (2008). Response error modeling in face to face surveys. Research Journal of Science, Isfahan University, 33(4):1-14. https://sid.ir/paper/56025/en.
Alimohammadi, R. (2011). Comparison of variance components estimation methods of response error model in surveys. Applied Mathematical Sciences, 5(48):2405-2410.
Biemer, P.P. and Stokes, S.L. (1991). Approaches of modeling of measurement error. In: Lyberg, L.E., Kasprzyk, D., Biemer, P.P., Groves, R.M., Lyberg, L.E., Mathiowetz, N.A. and Sudman, S. (Eds.) Measurement Error in Surveys, New York: Wiley.
Biemer, P.P. and Trewin, D. (1997). A review of measurement error effects on the analysis of survey data. Survey Measurement and Process Quality, 601-632.
Burton, A., Altman, D.G., Royston, P. and Holder, R.L. (2006). The design of simulation studies in medical statistics. Statistics in Medicine, 25(24):4279-4292.
Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1):37-46.
Fliess, J.L., Cohen, J. and Everitt, B.S. (1969). Large sample standard errors of Kappa and weighted Kappa. Psychological Bulletin, 72(5):323-327.
Fliess, J.L. and Cohen, J. (1973). The equivalence of weighted Kappa and the intraclass correlation coefficient as measure of reliability. Educational and Psychological Measurement, 33(3):613-619.
Fliess, J.L., Levin, B. and Paik, M.C. (2003). Statistical Methods for Rates and Proportions. 3rd ed., New York: Wiley.
Gwet, K.L. (2001). Statistical Tables for Inter-Rater Reliability Studies. Gaithersburg, MD: STATAXIS.
Gwet, K.L. (2008). Computing inter-rater reliability and its variance in the presence of high agreement. British Journal of Mathematical and Statistical Psychology, 61(1):29-48.
He, H. and Garcia, E.A. (2009). Learning from imbalanced data. IEEE Transactions on Knowledge and Data Engineering, 21(9):1269-1284.
Kish, L. (1962). Studies of interviewer variance for attitudinal variables. Journal of the American Statistical Association, 57(297):92-115.
Kish, L. (1965). Survey Sampling. New York: Wiley.
Morris, T.P., White, I.R. and Crowther, M.J. (2019). Using simulation studies to evaluate statistical methods. Statistics in Medicine, 38(11):2074-2102.
Journal of Statistical Modelling: Theory and Applications
Volume 6, Issue 1
January 2025
Pages 185-192

Files

History

  • Receive Date: 18 August 2025
  • Revise Date: 24 January 2026
  • Accept Date: 27 January 2026

Share

How to cite

Statistics