Tests of Reliability and Validity - Term Paper Example

RUNNING HEAD Reliability and Validity Tests of Reliability and Validity Here Here Tests of Reliability and Validity This paper tackles the application and appropriateness of various tests of reliability and validity, together with their respective strengths and weaknesses. Under reliability, the following tests are discussed: internal consistency, split-half, test/retest, and parallel and alternate forms. Meanwhile, under tests of validity, the following are included in the discussion: face validity, content validity, criterion-related validity, and construct validity. Internal consistency. The test for internal consistency reliability deals with the “homogeneity of the items within a scale” (DeVellis, 2003, p.28). It is used to measure just one single phenomenon Items are said to be internally consistent if these items are highly correlated (DeVellis, 2003). Coefficients of internal consistency deals with variance as a result of variation in item content as a primary source of error variance. At the same time, coefficients “show how much the variance is systematically based on a common concept measured by the test as a whole” (Guion and Highhouse, 2006, p. 129). The main strengths of internal consistency validity is its applicability in education when teachers can not use alternate forms of the same test and its convenience, since only one administration and one test form is required (Oermann and Gaberson, 2006; Guion and Highhouse, 2006). The main weakness of internal consistency is associated with the complex field of human temperament and dynamics where high internal consistency has been found to be an antithesis to high validity, especially when the variables being measured is broad (Kline, 1986). Another weakness is that even if it is used to measure only one variable, estimators of internal consistency such as Cronbach’s alpha are functions of two parameters (Robins, Fraley and Krueger, 2007). It was also revealed in a research study by Puhan, Bryant, Guyatt, Heels-Ansdell and Schϋnemann (2005) that internal consistency is a poor predictor of responsiveness. Split-half. The test for split-half reliability measures the internal consistency of items with a high degree of fluctuation. As the name implies, the items are divided in half, usually on an odd even basis, and the two halves are correlated. The main strengths of split-half reliability is the impossibility of the effects of time to intervene in the results, since the test is administered only once; and that the respondents are exposed to the same physical, mental and environmental influences. Meanwhile, there are also a few limitations : (1) the fewer items resulting from the splitting into halves results in a wider dispersion of the responses; (2) as a result of the variability, the responses do not easily stabilize around the mean; (3) the need to determine if the two halves of the test actually measure the same thing (Groth-Marnat, 2003; Portney and Watkins, 2009) Test/retest. The test/retest reliability measures the degree with which a given test generates consistent results on retesting. In other words, this measure tests reliability from time to time. Test/retest reliability is best applied when the variables being measured are relatively stable. Some limitations of test/retest reliability are: (1) the tendency for the time interval between the administration of the test and retest to affect the reliability; (2) its inapplicability for speeded and memory tests; (3) its tendency to be affected by random, fluctuations in the respondents or by variations in the testing conditions (Groth-Marnat, 2003). The main strength of test/retest reliability is its ability to provide information on the degree of confidence that a particular measure reflects the construct and that this measure may be generalized to other assessment occasions (Haynes et al., in Netemeyer, Bearden and Sharma, 2003). Parallel and alternate forms. In this test of reliability, the variable being investigated is measured using parallel forms of the test with the expectation that for reliability to be high, the different measurements are to yield the same results. The main strength of parallel and alternate forms is the elimination of carry-over effects, such as recall of responses in the previous tests. The main limitation in this type of reliability testing is the determination whether the parallel test are, indeed, equivalent. A delay from one administration to the next is also susceptible to short-term fluctuations in the respondents’ condition which affects the results of the test (Groth-Marnat, 2003). Tests of Validity Face validity. Together with content validity, face validity reveals the degree by which a construct is translated as it is operationalized. However, face validity is different from content validity. The proof of face validity is made after a post hoc evaluation that items in a scale are adequate measures of a given construct. There are some advantages in using face validity, which may be considered its strengths, as well, such as its use in practical situations, and its utility when no other form of validation is possible (Netemeyer, et al., 2003; Portney and Watkins, 2009). Thus, the greatest limitation of face validity is that it only offers an incomplete or partial evidence of validity. It is also in this respect that face validity is considered the weakest form of measurement validity. Moreover, use of face validity offers no safeguards against potential challenges since validation is based only on an opinion (Portney and Watkins, 2009). Content validity. Content validity signifies the extent to which elements of a measurement instrument, such as a test or a questionnaire, are relevant to the construct and that these elements embody that construct for a specific evaluation purpose (Haynes et al., in Netemeyer, et al, 2003). Content validity is used when one wishes to know, whether or not a sample of items truly represents the totality of items pertaining to a given topic. The most common application of content validity is in achievement tests, and is most useful in questionnaires and inventories (Salkind, 2007; Portney and Watkins, 2009). The greatest strength of content validity is its applicability to constructs which are ambiguous or complex (Netemeyer, et al, 2003). The three primary weaknesses of content validity is that: (1) determination of content validity is fundamentally a subjective procedure; (2) no statistical index is available to evaluate content validity; and (3) has limited utility for evaluating the validity of empirical measures of theoretical concepts employed in the social sciences (Portney and Watkins, 2009; Carmines and Zeller, 1979). Criterion-related validity. Criterion validity is utilized when one needs to know whether or not test scores are systematically related to some other criteria, where the other criteria indicates that the person/s taking the test is competent in a certain area (Salkind, 2007). Criterion-related validity may also be as predictive validity or concurrent validity or post-dictive validity (Portney and Watkins, 2009). One of the most notable strengths of criterion-related validity is its being temporally neutral (DeVellis, 2003). However, like content validity, criterion-related validity also has limited utility for evaluating the validity of empirical measures of theoretical concepts employed in the social sciences (Carmines and Zeller, 1979). Construct validity. Construct validity is concerned with the theoretical relationship of one variable to another variable. DeVellis (2003) describes construct validity as “the extent to which a measure ‘behaves’ the way that the construct it purports to measure should behave with regards to established measures of their constructs” (p. 53). The main strength of construct validation is its generalized application in the social sciences ((Carmines and Zeller, 1979). The main limitations of construct validity are: (1) constructs are “typically multi-dimensional and it is not easy to determine if an instrument is actually measuring the variable of interest”; (2) the meaning of a construct is based on assumptions about how a trait would behave under given conditions (Portney and Watkins, 2009, p. 105). References Carmines, E. G. & Zeller, R. A. (1979). Reliability and Validity Assessment. Thousand Oaks, CA: Sage Publications, Inc. DeVellis, R. F. (2003). Scale development: Theory and applications. 2nd ed. Thousand Oaks, CA: Sage Publications, Inc. Groth-Marnat, G. (2003). Handbook of psychological assessment. Hoboken, NJ: John Wiley & Sons, Inc. Guion, R. M. & Highhouse S. (2006). Essentials of personnel assessment and selection. Mahwah, NJ: Lawrence Erlbaum Associates, Inc. Kline, P. (1986). A handbook of test construction: Introduction to psychometric design. New York: Methuen & Co. Ltd. Netemeyer, R. G., Bearden, W. O., & Sharma, S. (2003). Scaling procedures: Issues and applications. Thousand Oaks, CA: Sage Publications Inc. Oermann, M. H. & Gaberson, K. B. (2006). Evaluation and testing in nursing education. 2nd ed. New York: Springer Publishing Company, Inc. Portney, L. G. & Watkins, M. P. (2009). Foundations of clinical research: Applications to practice. 3rd ed. Upper Saddle River, NJ: Pearson Prentice Hall. Puhan, M. A., Bryant, D., Guyatt, G. H., Heels-ansdell, D. & Schϋnemann, H. J. (2005). Internal consistency reliability is a poor predictor of responsiveness. Health Quality of Life Outcomes, 3, p. 33. Retrieved June 29, 2009, from http://www.pubmedcentral.nih.gov/articlerender.fcgi?artid=1156927. Robins, R W., Fraley, R. C. & Krueger, R. F. (2007). Handbook of research methods in personality psychology. New York: Guilford Press. Salkind, N. J. (2007). Statistics for people who (think they) hate statistics. Excel ed. Thousand Oaks, CA: Sage Publications, Inc. Read More