Norms - a crucial issuse in testing

Test norms need to be specific to the user and the test context. This should be obvious, still is often ignored, perhaps due to the expenses involved. What happens if norms are not specific?

1. A very important aspect is that of faking. Faking is abundant in job applicants. If norms are collected from incumbents or, even worse, the population at large, test scores can be grossly misleading. The reason is that many applicants fake and the distribution of their test scores is shifted towards a higher mean than for incumbents who fake very little or not at all. As a consequence, test scores for applicants will be systematically overestimated. In a stanine scale, the error could easily be 2 or 3 steps. This problem could be greatly mitigated by using a correction procedure using one or several scales for measuring the tendency to respond in a socially desirable manner. In our data, about 95 % of the effect is eliminated this way. Note, however, that the correction model must be scale specific since scales are usually not equally vulnerable to distortion.

2. Test scores may be strongly dependent on the organizational context. In some contexts, independences is not a desired trait and people will on the average have low scores on this trait. Another example is perseverance in the face of failure. If failure is rarely obvious, test takers will report low perseverance. For reasons such as these, norms need to be specific to the organizations.

It is not excessively demanding to construct specific norms, given modern IT technology, and the sample size need to be only as small as 300, or even in some cases 120. The first step is to realize the importance of specific norms, of norms corrected for impression management if they are based on incumbents or the population at large, and the fact that the sample size can be fairly small. In our practice we work with such norms, but many Swedish test providers seem unaware of the issue and that the problems can be solved with relatively modest resources.

Normgrupper

Hur stor måste en normgrupp för ett personlighetstest vara? Många testproducenter rapporterar mycket stora, kanske t o m representativa för populationen, grupper. En noggrann diskussion av frågan kom först nyligen med ett arbete av Tett et al. De finner att normgrupper knappast behöver vara större än N=300 och att även ett så lågt värde som N=100 ger tillfredsställande precision.

Andra överväganden än storleken är lika viktiga, eller viktigare. Normdata bör komma från en relevant jämförelsegrupp, t ex chefer eller chefskandidater. Vid testanvändning för urval ska normdata samlas in i skarpt läge, vid rådgivning och utveckling i oskarpt läge. Normer kan t o m behöva vara helt lokala, t ex för att visst företag. I normalfallet bör man utveckla nya och mera precisa normdata allt eftersom erfarenheten av ett test växer när det tillämpas i praktisk verksamhet.

Tett et al. har gjort en viktig insats. Alla testutvecklare har givetvis känt till att man måste ha en normgrupp, men hur stor den måste vara har varit oklart. Kravet på stora grupper har troligen verkat hämmande på utvecklingen och användningen av nya, specialanpassade normer.


Referens

Tett, R. P., Fitzke, J. R., Wadlington, P. L., Davies, S. A., Anderson, M. G., & Foster, J. (2009). The use of personality test norms in work settings: Effects of sample size and relevance. Journal of Occupational and Organizational Psychology, 82(3), 639-659.