Current Search:  Knowledge Structure Evaluations (x)

View All Items

  • CSV Spreadsheet
(1 - 2 of 2)
INVESTIGATING THE RELIABILITY AND VALIDITY OF KNOWLEDGE STRUCTURE EVALUATIONS: THE INFLUENCE OF RATER ERROR AND RATER LIMITATIONS
Title
INVESTIGATING THE RELIABILITY AND VALIDITY OF KNOWLEDGE STRUCTURE EVALUATIONS: THE INFLUENCE OF RATER ERROR AND RATER LIMITATIONS.
Creator
Harper-Sciarini, Michelle, Jentsch, Florian, University of Central Florida
Abstract / Description
The likelihood of conducting safe operations increases when operators ave effectively integrated their knowledge of the operation into meaningful relationships, referred to as knowledge structures (KSs). Unlike knowing isolated facts about an operation, well integrated KSs reflect a deeper understanding. It is, however, only the isolated facts that are often evaluated in training environments. To know whether an operator has formed well integrated KSs, KS evaluation methods must be employed....
Show moreThe likelihood of conducting safe operations increases when operators ave effectively integrated their knowledge of the operation into meaningful relationships, referred to as knowledge structures (KSs). Unlike knowing isolated facts about an operation, well integrated KSs reflect a deeper understanding. It is, however, only the isolated facts that are often evaluated in training environments. To know whether an operator has formed well integrated KSs, KS evaluation methods must be employed. Many of these methods, however, require subjective, human-rated evaluations. These ratings are often prone to the negative influence of a raterÂÂÂÂÂÂ's limitations such as rater biases and cognitive limitations; therefore, the extent to which KS evaluations are beneficial is dependent on the degree to which the raterÂÂÂÂÂÂ's limitations can be mitigated. The main objective of this study was to identify factors that will mitigate rater limitations and test their influence on the reliability and validity of KS evaluations. These factors were identified through the delineation of a framework that represents how a raterÂÂÂÂÂÂ's limitations will influence the cognitive processes that occur during the evaluation process. From this framework, one factor (i.e., operation knowledge), and three mitigation techniques (i.e., frame-of-reference training, reducing the complexity of the KSs, and providing referent material) were identified. Ninety-two participants rated the accuracy of eight KSs over a period of two days. Results indicated that reliability was higher after training. Furthermore, several interactions indicated that the benefits of domain knowledge, referent material, and reduced complexity existed within subsets of the participants. For example, reduced complexity only increased reliability among evaluators with less knowledge of the operation. Also, referent material increased reliability only for those who scored less complex KSs. Both the practical and theoretical implications of these results are provided.
Show less
Date Issued
2010
Identifier
CFE0002973, ucf:47950
Format
Document (PDF)
PURL
http://purl.flvc.org/ucf/fd/CFE0002973
Identifying Type of Expertise as a Means to Measure CRM Knowledge Structures
Title
Identifying Type of Expertise as a Means to Measure CRM Knowledge Structures.
Creator
Jimenez, Camilo, Jentsch, Florian, Shumaker, Randall, Burke, Shawn, Sims, Valerie, University of Central Florida
Abstract / Description
Crew Resource Management (CRM) training has evolved since its inception in the 1980s to better accommodate the operational needs of flight crews. However, even as the aviation and research communities have pointed to the potential benefit of providing CRM training, some criticism continues to emerge periodically which claims that there is no concrete evidence of its impact on flight deck performance and safety. Therefore, it is imperative to develop tools that allow researchers and, more...
Show moreCrew Resource Management (CRM) training has evolved since its inception in the 1980s to better accommodate the operational needs of flight crews. However, even as the aviation and research communities have pointed to the potential benefit of providing CRM training, some criticism continues to emerge periodically which claims that there is no concrete evidence of its impact on flight deck performance and safety. Therefore, it is imperative to develop tools that allow researchers and, more importantly, practitioners, to more effectively and objectively assess training effectiveness and identify whether or not desired CRM behaviors are being put to practice during line operations.This study focused on evaluating pilots' CRM schemas and identifying differences in CRM knowledge structures among pilots. Differences in CRM knowledge and opinions about training could be an indication of the existence of what Hatano and Inagaki (1986) have described as two distinct types of expertise, namely, routine and adaptive expertise. The study sought to identify differences among routine and adaptive expert pilots in CRM knowledge structures (schemas), their perceptions on the value and efficacy of current CRM training evaluation, along with their opinion on how CRM training effectiveness could be more accurately assessed. Results from over 250 pilots showed that, in general, participants had a positive view of CRM training and training evaluation, regardless of their type of expertise. Some evidence of potential differences in the structural knowledge of CRM between routine and adaptive experts, as well as, differences in their opinions about CRM training, evaluation, and automation on the flight deck was also identified. Additionally, analysis of survey scores and free response items indicate the existence of a third category of experts, between routine and adaptive expertise (whom I call transitional experts). The study results provide evidence that assessment of CRM schemas could potentially be used as a way to evaluate CRM training effectiveness. The results of the study also indicate that identification of specific training needs for each group of expert may be possible through the assessment of CRM schemas and type of expertise. Implications for practice and theory, limitations of the study, and suggestions for future research are also provided.
Show less
Date Issued
2018
Identifier
CFE0007200, ucf:52264
Format
Document (PDF)
PURL
http://purl.flvc.org/ucf/fd/CFE0007200