Comparing performance of statistical models for individual’s ability index and ranking

Iqbal, Javed (2012): Comparing performance of statistical models for individual’s ability index and ranking.

Abstract

Efficient allocation of resources is the basic problem in economics. Firms, educational institutions, universities are faces problem of estimating true abilities and ranking of individuals to be selected for job, admissions and scholarship awards etc. This study will provide a guide line what technique should to be used for estimating true ability indices and ranking that reveals ability with maximum efficiency as well as it clearly has the advantage of differentiating among individuals having equal raw score. Two major theories Classical Testing Theory and Item Response Theory have been using in the literature. We design two different Monte Carlo studies to investigate which theory is better and which model perform more efficiently. By discussing the weaknesses of CTT this study proved that IRT is superior to CTT. Different IRT models have been used in literature; we measured the performance of these models and found that Logistic P2 model is best model. By using this best model we estimate the ability indices of the students on the basis of their entry test scores and then compare with their abilities obtained from final board examination result (used as proxy of true abilities). This is a reasonable because the final exam consists of various papers and chance variation in ability Index is a minimum. With real life application this study also proved that IRT estimate the true abilities more efficiently as compared to classical methodology.

• Item Type: MPRA Paper
• Language: English
• Keywords: Ability Index, Monte Carlo study, Logistic and Probit models, Item Response Theory, Classical Test Theory, Ranking of Students
• Subjects: C - Mathematical and Quantitative Methods > C1 - Econometric and Statistical Methods and Methodology: General > C13 - Estimation: General; C - Mathematical and Quantitative Methods > C8 - Data Collection and Data Estimation Methodology ; Computer Programs > C87 - Econometric Software; C - Mathematical and Quantitative Methods > C6 - Mathematical Methods ; Programming Models ; Mathematical and Simulation Modeling > C61 - Optimization Techniques ; Programming Models ; Dynamic Analysis; C - Mathematical and Quantitative Methods > C1 - Econometric and Statistical Methods and Methodology: General > C15 - Statistical Simulation Methods: General

References

Baker, F. B. (1992). Item response theory: Parameter estimation techniques. New York: Marcel Dekker Bock, R., & Aitkin, M. (1981). Marginal maximum likelihood estimation of item parameters: An application of the EM algorithm. Psychometrika , 46, 443-459 Bock, R., & Lieberman, M. (1970). Fitting a response model for n dichotomously scored items. Psychometrika , 35, 179-197 Cantrell, C.E. (1999). Item response theory: Understanding the one-parameter rasch model. In B. Thompson, Advances in social science methodology (Vol. 5, pp. 171-192). Stamford, CT: JAI Press Courville, T. (2004). An Empirical Comparison of Item Response Theory And Classical Test Theory Item/Person Statistics: PhD thesis. A & M University: Texas Hambleton R. K., & Swaminathan, H. (1985). Item Response Theory: Principles and Applications. Boston, MA: Kluwer Academic Publishers Hambleton R. K., Swaminathan H. & Rogers H. J. (1991). Fundamentals of item response theory. Newbury Park, CA: Sage Hambleton R. K. and Russel W. (1993). Comparison of classical test theory and item response theory and their applications to test development. Educational Measurement: Issues and Practice , 12, 38-47 Henson, R. (1999). Understanding the one-parameter Rasch model of item response theory. Paper presented at the annual meeting of the Southwest Educational Research Association, San Antonio, TX. (ERIC Document Reproduction Service No. ED 428 078) Jimelo L. Silvestre-Tipay. (2009). Item Response Theory and Classical Test Theory: An Empirical Comparison of Item/Person Statistics in a Biological Science Test. The International Journal of Educational and Psychological Assessment. Vol. 1, Issue 1 Lawley, D. (1943). On problems connected with item selection and test construction. Proceedings of the Royal Society of Edinburgh , 61A, 273-287 Linden, W. J., & Glas, C. A. (2000). Computerized adaptive testing: theory and practice. Dordrecht: Kluwer Academic Publisher Lord, F. (1952). A theory of test scores. Psychometric Monograph (7), Psychometric Society Lord, F. M. (1980). Applications of item response theory to practical testing problems. Hillsdale, NJ: Erlbaum Associates Lord F. M. and Novick M. R. (1968). Statistical theories of mentaltest scores. Reading, MA: Addison-Wesley MacDonald P. and Paunonen S. (2002). A Monte Carlo comparison of item and person statistics based on item response theory versus classical test theory. Educational and Psychological Measurement , 62, 921-943 Richard J. Patz and Brian W. Junker (1999). Applications and Extensions of MCMC in IRT: Multiple Item Types, Missing Data, and Rated Responses. Journal of Educational and Behavioral Statistics, Vol. 24, No. 4 Robert K. Tsutakawa and Michael J. Soltys (1988). Approximation for Bayesian Ability Estimation. Journal of Educational Statistics, Vol. 13, No. 2, pp. 117-130 Sheng, Y. (2005). Bayesian Analysis of Hierarchical IRT Models: Comparing and Combining the Unidimensional & Multi-unidimensional IRT Models” PhD Thesis. University of Missouri: Columbia Sheng, Y. (2008). Markov Chain Monte Carlo Estimation of Normal Ogive IRT Models in MATLAB. Journal of Statistical Software , 25 (8) Sotaridona, L. S., Pornel, J. B., & Vallejo, A. (2003). Some Applications of Item Response Theory to Testing. The Philippine Statistician , 52 (1-4), 81-92 Stage, C. (1999). A comparison between item analysis based on item response theory and classical test theory: A study of the SweSAT test READ. (Educational Measurement No31). Umea University: Department of Educational Measurement Terry A. Ackerman. (1991). Reviewed work(s): Item Response Theory: Parameter Estimation Techniques. by Frank B. Baker Source: Journal of the American Statistical Association, Vol. 88, No. 422 Thomas, D. R., & Andre Cyr. (2002). Applying Item Response Theory Methods to Complex Survey Data. SSC Annual Meeting: Proceedings of the Survey Methods Section Troy Gerard Courville. (2004). An empirical comparison of Item Response Theory and Classical Test Theory item/person statistics. Ph.D thesis. Submitted to the Office of Graduate Studies of Texas A&M University Xitao Fan. (1998). Item response theory and classical test theory: an empirical comparison of their item/person statistics. Educational and Psychological Measurem. Vol.58, No.3 Wiberg, M. (2004). Classical Test Theory vs Item Response Theory: An evaluation of the theory test in the Swedish driving-license test. Working paper: EM No 50, UMEA University Zaman, A., & Atiq-ur-Rehman. (2008). Students Ranking, based on their Abilities on Objective type test: Comparison of CTT and IRT. Accepted in the Confrence, “teaching and learning” Rothenburg, Germany