基于作答时间数据的改变点分析在检测加速作答中的探索——已知和未知项目参数

Exploration of change point analysis in detecting speededness based on response time data with known/unknown item parameters

相对于传统的离散作答数据,作答时间作为连续数据,可以提供更多信息。改变点分析(changepoint analysis)技术在心理和教育领域是一个比较新的技术。本文一方面对改变点分析在心理测量领域的应用进行了一个综合的总结和分析;另一方面,将基于作答数据的两种改变点分析统计量推广到作答时间数据,将改变点分析技术应用到测验异常作答模式:加速作答speededness的检测上。采用两种检验方法:似然比检验和Wald检验,分别在已知和未知项目参数的条件下,实现异常作答模式的检测。结果表明,所采用的方法对于加速作答行为的检测具有很高的检验力,同时能够很好的控制I类错误率。实证数据分析进一步表明本文中所使用的方法具有应用价值。

In recent years,response time has received a rapidly growing amount of attention in psychometric research,likely due to the increasing availability of(item-level)response time data through computer-based testing and online survey data collection.Compared to the conventional item response data that are often dichotomous or polytomous,the response time is continuous and can provide much more information.Aberrant response behaviors are frequently encountered during testing.It could cause various negative effects.Change point analysis(CPA)is a well-established statistical process control method to detect changes in a sequence,and it has provided testing professionals a new lens through to understand test-taking behavior at both the examinee and item levels.In this paper,we took test speededness as an example to illustrate how the CPA method can be used to detect aberrant behavior using item response time data.Response time under speededness was simulated using the gradual-change log-normal model for response time.Two CPA-based test statistics,the Likelihood Ratio Test and Wald Test,were used to detect aberrant response behaviors.The critical values were obtained through Monte Carlo simulations and compared with the approximate critical values in a previous study.Based on the chosen critical values,we examined the performance of the likelihood ratio test and Wald test in detecting speeded responses,specifically in terms of power and empirical Type-I error.On the one hand,the critical values are almost identical for Wald and the likelihood ratio test.They vary substantially at different nominalαlevels,but do not differ much across different test lengths.On the other hand,compared to approximate critical values,the critical values are not too far away from them but are different.That may be because the approximate critical values are suitable for situations where the change point appears in the middle of the test.Results indicate that the proposed method is much more powerful based on the critical values than conventional methods that use item response data.The power was close to 1 for most of the conditions while keeping the type-I error rate well-controlled.Real data analysis also demonstrates the performance of the method.This study uses CPA with response time data and offers a very promising approach to detecting aberrant response behavior.Through the simulation study,we demonstrated that it is possible to use fixed critical values in different test lengths,which makes the application of the method straightforward.It also means that it is unnecessary to reconduct the simulation to update critical values when small changes occur in the test.CPA is very flexible.This study assumed that the log-normal model fits the response time data,but the method is not bounded by that assumption.