Essentials for Developing and Validating Psychological Scales: Guide to Best Practices

Document Type : Original Article

Authors

1 Tarbiat Modares University

2 Shahed University

Abstract

Although Scale development is a common work in behavioral and psychological research, many of them are not adequately exact and perfect. Designing and validating a scale is not only straightforward but also it is an onerous and unfamiliar process. This process requires accuracy and exactness. Therefore, the purpose of this paper was to concisely review the step by step process of scale development.  The required and ignored phases and steps of current scale development research, they can be used to all researchers especially the novice. This paper focused on both theoretical basis and practical steps.
The Authors tried to stay away from theoretical superficiality and practical ignorance
 

Keywords


1- Miller s. Ten Steps in Scale Development and Reporting: A Guide for Researchers. Communication Methods and Measures (2018) 12(1):25-44 DOI: 10.1080/19312458.2017.1396583
2- DeVellis RF. Scale Development: Theory and Application. Los Angeles, CA: Sage Publications (2012)
3- De Vellis, R. F. Scale Development: Theory and Applications (2nd ed., Vol. 26). Thousand Oaks, CA: Sage Publications (2003).
4- Boateng GO, Neilands TB, Frongillo EA, Melgar-Quiñonez HR and Young SL.Best Practices for Developing and Validating Scales for Health, Social, and Behavioral Research: A Primer. Front. Public Health (2018) 6:149. DOI: 10.3389/fpubh.2018.00149
5- Clarke LA, Watson D. Constructing validity: basic issues in objective scale development. Pyschol Assess. (1995) 7:309–19. DOI: 10.1037/1040-3590.7.3.309
6- Nunnally JC. Pyschometric Theory. New York, NY: McGraw-Hill (1978).
7- Hinkin TR. A review of scale development practices in the study of organizations. J Manag. (1995) 21:967–88. DOI: 10.1016/0149- 2063(95)90050-0
8- Haynes SN, Richard DCS, Kubany ES. Content validity in psychological assessment: a functional approach to concepts and methods. Pyschol Assess. (1995) 7:238–47. DOI: 10.1037/1040-3590.7.3.238
9- Morgado FFR, Meireles JFF, Neves CM, Amaral ACS, Ferreira MEC. Scale development: ten main limitations and recommendations to improve future research practices. Psicol Reflex E Crítica (2018) 30:3. DOI: 10.1186/s41155-016-0057-1
10- Glanz K, Rimer BK, Viswanath K. Health Behavior: Theory, Research, and Practice. San Francisco, CA: John Wiley & Sons, Inc (2015).
11- Kline P. A Handbook of Psychological Testing. 2nd Edn. London: Routledge; Taylor & Francis Group (1993).
12- Hinkin, T. R. A review of scale development practices in the study of organizations', Journal of Management (1995) 21:  5, 967-988.
13- Hunt SD. Modern Marketing Theory. Cincinnati: South-Western Publishing (1991).
14- Hinkin TR. A review of scale development practices in the study of organizations. J Manag. (1995) 21:967–88. DOI: 10.1016/0149-2063(95)90050-0.
15- Hunt SD. Modern Marketing Theory. Cincinnati: South-Western Publishing (1991).
 16- Barzegar  B , Azadfallah P , Farahani H , Fathi_Ashtiani,A. Development of the Persian Version of the Integrative Hope Scale: A Review of Psychometric Indices. Int J Behav Sci. (2018) 12(1): 1-8
17- Loevinger J. Objective tests as instruments of psychological theory. Psychol Rep. (1957) 3:635–94. DOI: 10.2466/pr0.1957.3.3.635.
18- Clarke LA, Watson D. Constructing validity: basic issues in objective scale development. Pyschol Assess. (1995) 7:309–19. DOI: 10.1037/1040-3590.7.3.309
19- Kline P. A Handbook of Psychological Testing. 2nd Edn. London: Routledge; Taylor & Francis Group (1993).
20- Schinka JA, Velicer WF, Weiner IR. Handbook of Psychology, Vol. 2, Research Methods in Psychology. Hoboken, NJ: John Wiley & Sons, Inc. (2012).
21- Krosnick JA, Presser S. Question and questionnaire design. In: Wright JD, Marsden PV, editors. Handbook of Survey Research. San Diego, CA: Elsevier (2009), pp. 263–314.
22- MacKenzie SB, Podsakoff PM, Podsakoff NP. Construct measurement and validation procedures in MIS and behavioral research: integrating new and existing techniques. MIS Q. (2011) 35:293. DOI: 10.2307/23044045
23- Osborne, J. W .Best practices in exploratory factor analysis. Scotts Valley, CA: CreateSpace Independent Publishing. (2014).
24- McPhail SM. Alternative Validation Strategies: Developing New and Leveraging Existing Validity Evidence. San Francisco, CA: John Wiley & Sons, Inc (2007).
25- Guion R. Content validity - the source of my discontent. Appl Psychol Meas. (1977) 1:1–10. DOI: 10.1177/014662167700100103
26- Haynes SN, Richard DCS, Kubany ES. Content validity in psychological assessment: a functional approach to concepts and methods. Pyschol Assess. (1995) 7:238–47. DOI: 10.1037/1040-3590.7.3.238.
27- Dennis C-L, Faux S. Development and psychometric testing of the Breastfeeding Self-Efficacy Scale. Res Nurs Health (1999) 22:399–409. DOI: 10. 1002/(SICI)1098-240X(199910)22:53.0.CO;2-4
28- Tilden V, Nelson, C, May, B. Use of qualitative methods to enhance content validity. Nursing Research, 39(3), 172-175
29- Gregory E, Gilbert, E, Prion S. Making Sense of Methods and Measurement: Lawshe’s Content Validity Index .Clinical Simulation in Nursing (2016) 12, 530-531.
30- Haynes SN, Richard DCS, Kubany ES. Content validity in psychological assessment: a functional approach to concepts and methods. Pyschol Assess. (1995) 7:238–47. DOI: 10.1037/1040-3590.7.3.238.
31- Clarke LA, Watson D. Constructing validity: basic issues in objective scale development. Pyschol Assess. (1995) 7:309–19. DOI: 10.1037/1040-3590.7.3.309
32- Cooper, R., Potter, W. J., & Dupagne, M. (1994). A status report on methods used in mass communication research. Journalism Educator, 48(4), 54–61.
33- Drennan, J.Cognitive interviewing: Verbal data in the design and pretesting of questionnaires. Journal of Advancing Nursing (2003)42(1), 57–63.
34- Lewis, B. R., Templeton, G. F., & Byrd, T. A. A methodology for construct development in MIS research. European Journal of Information Systems (2005) 14(4), 388–400.
35- Reynolds N., Diamantopoulos A., & Schlegelmilch. Pretesting in questionnaire design: A review of the literature and suggestions for further research. Journal of Market Research Society (1993) 35(2), 171–182.
36- Comrey AL. Factor-analytic methods of scale development in personality and clinical psychology. Am Psychol Assoc. (1988) 56:754-61.
37- Hair, J., Jr., Black, W. C., Babin, B. J., & Anderson, R. EMultivariate data analysis (7th ed.). Upper Saddle River, NJ: Prentice Hall (2010).
38- Tabachnick, B. G., & Fidell, L. S. (2007). Using multivariate statistics (5th Ed.). Boston, MA: Allyn and Bacon
39- Beavers, A. S., Lounsbury, J. W., Richards, J. K., Huck, S. W., Skolits, G. J., & Esquivel, S. L. (2013). Practical considerations for using exploratory factor analysis in education research. Practical Assessment, Research & Evaluation, 18(6), 1–13.
40- Gottschall AC, West SG, Enders CK. A Comparison of item-level and scalelevel multiple imputation for questionnaire batteries. Multivar Behav Res. (2012) 47:1–25. DOI: 10.1080/00273171.2012.640589
41- Raykov T. Scale Construction and Development. Lecture Notes. Measurement and Quantitative Methods. East Lansing, MI: Michigan State University (2015).
42- Piedmont RL. Inter-item correlations. In Encyclopedia of Quality of Life and Well-Being Research. Dordrecht: Springer (2014). p. 3303–4. DOI: 10.1007/978-94-007-0753-5_1493.
43- Norris M., & Lecavalier L. Evaluating the use of exploratory factor analysis in developmental disability psychological research. Journal of Autism Development and Disorders (2010) 40, 8–20.
44- Costello, A. B. & Osborne J. W. Best practices in exploratory factor analysis: Four recommendations for getting the most from your analysis. Practical Assessment, Research & Evaluation (2005) 10(7), 1–9.
45- Ford, J. K., MacCullum, R. C., & Tait, M. The application of exploratory factor analysis in applied psychology: A critical review and analysis. Personnel Psychology) 1986) 39, 291–314
46- Haig, B. D. Exploratory factor analysis, theory generation, and scientific method. Multivariate Behavioral Research (2005), 40(3), 303–32.
47- Kline, R. B.Exploratory and confirmatory factor analysis. In Y. Petscher, C. Schatschneider, & D. L. Compton (Eds.), Applied quantitative analysis education and the social sciences (pp. 171–207). New York, NY, USA: Routledge (2013).
48- Pett, M. A., Lackey, N. R., & Sullivan, J. L. Factor analysis. The use of factor analysis for instrument development in health care research. Thousand Oaks, CA: Sage Publications, Inc. (2005).
49- Preacher, K. J., & MacCallum, R. C.Repairing Tom Swift’s electric factor analysis machine. Understanding Statistics (2003)2(1), 13–43.
50- Worthington R. L. & Whittaker, T. A. Scale development research. A content analysis for recommendations for best practices. The Counseling Psychologist (2006) 34(6), 806–838.پ
51- McCrosky, J. C., & Young, T. J. The use and abuse of factor analysis in communication research. Human Communication Research (1979(5, 375–82.
52- Pett, M. A., Lackey, N. R., & Sullivan, J. L. Making sense of factor analysis. The use of factor analysis for instrument development in health care research. Thousand Oaks, CA: Sage Publications, Inc. (2003).
53- Reise SP, Morizot J, Hays RD. The role of the bifactor model in resolving dimensionality issues in health outcomes measures. Qual Life Res. (2007) 16:19–31. DOI: 10.1007/s11136-007-9183-7
54- Conway, J. M., & Huffcutt, A. IA review and evaluation of exploratory factor analysis practices in organizational research. Organizational Research Methods (2003) 6(2), 147–168.
55- Snook, S. C., & Gorsuch, R. L.Common factor analysis vs. component analysis. Psychological Bulletin (1989(106, 148–154.
56- Osborne, J. WBest practices in exploratory factor analysis. Scotts Valley, CA: CreateSpace Independent Publishing (2014).
57- Fabrigar, L. R., Wegener, D. T., MacCallum, R. C., & Strahan, E. J. Evaluating the use of exploratory factor analysis in psychological research. Psychological Methods (1999(, 4, 272–299.
58- Nunnally, J. C., & Bernstein, I. Psychometric theory (3rd Ed.). New York, NY, USA: McGraw Hill (1994).
59- Ford, J. K., MacCullum, R. C., & Tait, M. The application of exploratory factor analysis in applied psychology: A critical review and analysis. Personnel Psychology (1986) 39, 291–314.
60- Henson, R. K., & Roberts, J. K. Use of exploratory factor analysis in published research. Common errors and some comment on improved practice. Education and Psychological Measurement (2006) 66(3), 393–416.
61- Morrison, J. T. Evaluating factor analysis decisions for scale design in communication research. Communication Methods and Measures (2009) 3(4), 195–215.
62- Russell, D. W. In search of underlying dimensions: The use (and abuse) of factor analysis and scale revision. Psychological Assessment (2002) 12(3), 1629–1646.
63- Humphreys, L. G., & Montanelli, R. G., Jr.An investigation of the parallel criterion for determining the number of common factors. Multivariate Behavioral Research (1975) 10, 193–205.
64- Velicer, W. F., Eaton, C. A., & Fava, J. L. Construct explication through factor or component analysis: A review and evaluation of alternative procedures for determining the number of factors or components. In R. D. Goffin, & E. Helmes (Eds.), Problems and solutions in human assessment: Honoring Douglas Jackson at seventy (pp. 41–71). Boston, MA: Kluwer. (2000).
65- Watkins, M. W. Determining parallel analysis criteria. Journal of Modern Applied Statistical Methods. (2006) 5(2), 344–346.
66- Zwick, W. R., & Velicer, W. F. Factors influencing four rules determining the number of components to retain. Multivariate Behavioral Research (1982)17, 253–269.
67- Borgotta, E. F., Kercher, K., & Stull, D. E. A cautionary note on use of principal components analysis. Sociological Methods and Research (1986) 15, 160–168.
68- Conway, J. M., & Huffcutt, A. I. A review and evaluation of exploratory factor analysis practices in organizational research. Organizational Research Methods (2003) 6(2), 147–168.
69- Norris, M., & Lecavalier, L. Evaluating the use of exploratory factor analysis in developmental disability psychological research. Journal of Autism Development and Disorders (2010) 40, 8–20.
70- Gorsuch, R. L. (1997). Exploratory factor analysis: Its role in item analysis. Journal of Personality Assessment, 68, 532– 560.
71- Hendrickson, A. E., & White, P. O. Promax: A quick method for rotation to oblique simple structure. British Journal of Mathematical Psychology (1964) 17, 65–70.
72- Kachigan, S. K.Statistical analysis: An interdisciplinary introduction to univariate and multivariate methods. New York: Radius Press. (1986).
73- Meyers, L. S., Gamst, G., & Guarino, A. J.Applied multivariate research: Design and interpretation. Thousand Oaks, CA: Sage. (2006).
74- Reinard, J. C. Communication research statistics. Sage Publications, Thousand Oaks, CA. (2006).
75- Mertler, C. A., & Vannatta, R. A. Advanced and multivariate statistical methods: Practical applications and interpretation. Los Angeles, CA: Pyrczak Publishin. (2001).
76- Bond TG, Fox C. Applying the Rasch Model: Fundamental Measurement in the Human Sciences. Mahwah, NJ: Erlbaum (2013)
77- Brown T. Confirmatory Factor Analysis for Applied Research. New York, NY: Guildford Press (2014).
78- Morin AJS, Arens AK, Marsh HW. A bifactor exploratory structural equation modeling framework for the identification of distinct sources of constructrelevant psychometric multidimensionality. Struct Equ Model Multidiscip J. (2016) 23:116–39. DOI: 10.1080/10705511.2014.961800
79- Hunsley J, Meyer, G. The Incremental Validity of Psychological Testing and Assessment: Conceptual, Methodological, and Statistical Issues. Psychological Assessment (2003), 15, 4, 446–455.
80- Haynes, S. N., & Lench, H. C. Incremental validity of new clinical assessment measures. Psychological Assessment (2003). 15, 456–466.
81- Byrne B. M., & Stewart S. M.The MACS approach to testing for multi-group invariance of a second-order factor structure: A walk through the process. Structural Equation Modeling. (2006) 13, 287–321. http://dx.doi.org/ 10.1207/s15328007sem1302 7.
82- Chen F. F. Sensitivity of goodness of fit indices to lack of measurement invariance. Structural Equation Modeling (2007) 14, 464–504. http://dx.doi.org/10.1080/10705510701301834.
83- Tylka, T. L. Evidence for the Body Appreciation Scale’s measurement equivalence/invariance between U.S. college women and men. Body Image (2013) 10, 415–418. http://dx.doi.org/10.1016/j.bodyim.2013.02.006.
84- Porta M. A Dictionary of Epidemiology. New York, NY: Oxford University Press (2008).
85- Raykov T, Marcoulides GA. Introduction to Psychometric Theory. New York, NY: Routledge, Taylor & Francis Group (2011).
86- Brennan RL. A generalized upper-lower item discrimination index. Educ Psychol Meas. (1972) 32:289–303. DOI: 10.1177/0013164472032 00206.
87- Sexton JB, Helmreich RL, Neilands TB, Rowan K, Vella K, Boyden J, et al. The Safety Attitudes Questionnaire: psychometric properties, benchmarking data, and emerging research. BMC Health Serv Res. (2006) 6:44. DOI: 10.1186/1472-6963-6-44
88- Rashid, Kazhal, Farahani, Hojjaollah and Roshan chesli, Rasol (2020) .Developing and valididating sexual health literacy Scale in Iranian Sample .American Journal of Sexuality Education (submitted).