情境判断测验的信度与效度研究:以我国公务员内隐知识测验为例

Evaluation of Reliability and Validity of Situational Judgment Tests: A Study of Implicit Knowledge of Chinese Civil Servants

作为一种有效的人事选拔评价方法,情境判断测试一直受到研究者和企事业单位的广泛关注。但由于受到目前情境判断测试的信度和效度研究有限、国内公开发表的数据少等的制约,该测试的有效运用受到了一定影响。本文通过《我国公务员内隐知识情境判断测验》量表对151名在岗公务员和61名在读本科生施测,检验了情境判断测验量表的信度和效度及其部分影响因素,丰富了相关研究的实证证据,能更好地帮助理解SJT的价值及其在中国情境中的运用规范。研究结果表明公务员内隐知识情境判断测验量表具有较好的构思效度,能够测到公务员认知能力和工作经验;量表也具有较好的效标关联效度,能够预测与公务员工作相关的指标。通过对不同赋分方式下情境判断测验效度以及不同性别的被测对象的量表得分进行比较,我们发现赋分方式是公务员情境判断测验效度的影响因素之一,性别因素对公务员内隐知识情境判断测验效度没有显著影响。

In personnel selection, situational judgment tests （SJT） can identify unique individual characteristics, which other assessment tools or indicators are unable to discover. SJT has emerged as an important and useful complement to traditional cognitively oriented tests. It has been paid extensive attention since being introduced to China. However, two aspects limit the effective implementation of SJTs in China. The first constraint concerns the reliability and validity of SJTs, namely, whether and to what extent an SJT is able to predict individuals＇ future work performance. In the last two decades, through studying the reliability and validity of SJTs as well as the factors which can influence the predictiveness of SJTs, scholars have gained considerable knowledge of SJTs＇ construct validity and criterion-related validity. However, there is a lack of an effective theoretical framework to examine these findings. We also need to further study the characteristics of SJTs, as well as their influence on the validity of SJTs. The second constraint arises from the research context. The existent literature is mainly provided by American scholars, and the current empirical evidence is dominantly derived from American context. Studies that concentrated on Chinese context have only been developed lately, yet very little data has been published. The limited context-related empirical evidence did not provide strong support for organizations to adopt SJTs when choosing from multiple methods to select and assess their employees. The current paper discusses the reliability and validity of SJTs, based on an empirical test of an SJT carried on a group of Chinese civil servants. It is composed of three sets of studies. The first study discusses the construct validity of SJTs, that is, whether SJT can measure cognitive ability and work experience of individuals, and whether SJT can measure a multidimensional construct, such as tacit knowledge of Chinese civil servants. The second study examines criteria validity of SJTs, that is, whether the SJT can predict work-related performance. The last study discusses whether two factors, including scoring methods and gender differences, would influence the validity of SJTs. In a previous study, we developed a model of implicit knowledge of Chinese civil servants. We proposed that Chinese civil servants＇ implicit knowledge includes seven dimensions： cultivating morality, loyalty, serving others, influencing others, fair use of power, processing official business, and attentive to details. Based on this model, we developed the ＂Situational Judgment Tests Inventories of/mplicit Knowledge of Chinese Civil Servants＂. In May 2011, we distributed our questionnaires to some civil servants and business students in the western part of China. Eventually we collected a valid sample of 151 on-the-job civil servants and 61 business undergraduate students. We examined our data by using multiple methods including descriptive statistics analysis, variance analysis, and confirmatory factor analysis. Our results supported the previous conclusion that SJTs are able to measure individuals＇ cognitive ability and work experience. We obtained a satisfactory fit between our empirical data and the proposed 7-dimensional model. This finding indicates that SJTs are able to measure a multinational construct. Furthermore, our inventories demonstrated satisfactory criteria validity, indicating that SJT inventories are able to predict civil servants＇ work-related performance. At last, through comparing the validity of SJTs by using different scoring methods, we found that scoring method could influence the validity of SJTs. Through comparing the scores of different gender groups, we found that there was no significant variance between males and females. This indicates that gender differences did not have significance influence on the validity of our SJT. To sum up, our study collected empirical data in Chinese context and thus provided strong evidence for the values of SJTs. Through developing and verifying SJT inventories of implicit knowledge of Chinese civil servants, our study formed a valid tool for selecting civil servants and other similar personnel. The study can also be considered a reliable reference for future SJT-related studies and practices, in particular for those who are interested in designing and implementing SJTs.