You are here

UCF Digital Collections ยป ETD Collection

The Effects of Scoring Technique on Situational Judgment Test Validity

Download pdf | Full Screen View

Date Issued:
2011
Abstract/Description:
Situational Judgment Tests (SJTs) are frequently used by organizations as a face-valid selection measure with low adverse impact and a relatively strong relationship with relevant criteria. Despite their common use, there remain several research questions regarding the theoretical foundations and characteristics of SJTs. Additionally, developments in SJT scoring provide fertile ground for research to validate new scoring techniques to better predict criteria of interest. Motowidlo and his colleagues (2006) recently developed a scoring technique for SJTs based on the principle of Implicit Trait Policies (ITPs) which are implicit beliefs concerning the effectiveness of different behavioral choices that demonstrate varying levels of targeted traits. Individuals high in these targeted traits will rate item responses that demonstrate high levels of that particular trait as more effective. Taking into consideration this new method, and also considering the multitude of scoring methods already available to test developers, it logically follows that these different scoring methods will have different correlations with constructs of interest, and that by using this new method it may be possible to achieve a much higher correlation with personality. The effects of scoring technique on relationships between SJT scores and constructs of interest such as personality will in turn have effects on the criterion validity of the SJT. This research explored how scoring methods affected the relationship SJT scores have with general mental ability, personality traits, typical performance, and maximum performance. Results indicated significant differential validity as a function of the respondents' race. For minority participants, SJT scores predicted (")maximum performance ratings(") in a simulation exercise but not (")typical performance ratings(") provided by familiar peers. However, the reverse was true for Caucasian participants. The two scoring methods demonstrated differential validity. However, the nature of these differences varied as a function of the performance dimension in question (i.e., agreeableness, extraversion). Implications for future research will be discussed as well as the practical implications of these findings.
Title: The Effects of Scoring Technique on Situational Judgment Test Validity.
43 views
23 downloads
Name(s): Miller, Daniel, Author
Jentsch, Kimberly, Committee Chair
Jentsch, Florian, Committee Member
Fritzsche, Barbara, Committee Member
Burke, Shawn, Committee Member
University of Central Florida, Degree Grantor
Type of Resource: text
Date Issued: 2011
Publisher: University of Central Florida
Language(s): English
Abstract/Description: Situational Judgment Tests (SJTs) are frequently used by organizations as a face-valid selection measure with low adverse impact and a relatively strong relationship with relevant criteria. Despite their common use, there remain several research questions regarding the theoretical foundations and characteristics of SJTs. Additionally, developments in SJT scoring provide fertile ground for research to validate new scoring techniques to better predict criteria of interest. Motowidlo and his colleagues (2006) recently developed a scoring technique for SJTs based on the principle of Implicit Trait Policies (ITPs) which are implicit beliefs concerning the effectiveness of different behavioral choices that demonstrate varying levels of targeted traits. Individuals high in these targeted traits will rate item responses that demonstrate high levels of that particular trait as more effective. Taking into consideration this new method, and also considering the multitude of scoring methods already available to test developers, it logically follows that these different scoring methods will have different correlations with constructs of interest, and that by using this new method it may be possible to achieve a much higher correlation with personality. The effects of scoring technique on relationships between SJT scores and constructs of interest such as personality will in turn have effects on the criterion validity of the SJT. This research explored how scoring methods affected the relationship SJT scores have with general mental ability, personality traits, typical performance, and maximum performance. Results indicated significant differential validity as a function of the respondents' race. For minority participants, SJT scores predicted (")maximum performance ratings(") in a simulation exercise but not (")typical performance ratings(") provided by familiar peers. However, the reverse was true for Caucasian participants. The two scoring methods demonstrated differential validity. However, the nature of these differences varied as a function of the performance dimension in question (i.e., agreeableness, extraversion). Implications for future research will be discussed as well as the practical implications of these findings.
Identifier: CFE0004484 (IID), ucf:49314 (fedora)
Note(s): 2011-12-01
Ph.D.
Sciences, Psychology
Doctoral
This record was generated from author submitted information.
Subject(s): SJT -- Implicit Trait Policies -- Scoring Technique -- Adverse Impact -- High Fidelity -- Scoring Technique
Persistent Link to This Record: http://purl.flvc.org/ucf/fd/CFE0004484
Restrictions on Access: public 2012-06-15
Host Institution: UCF

In Collections