A major project in the US Army involves a new personality test, Tapas. This is a Big Five test with a combination of ipsative and normative formats which should make it less vulnerable to faking, although this has yet to be proven. A very interesting finding was that mental ability, or g, was an important predictor for "can do" criteria, while personality was important for "will do" criteria (around 0.2, probably not corrected for measurement error and restriction of range). The debate on how to weight g and personality must clearly take into account what is to be predicted. It is striking how much more important personality, even constrained to the ineffective Big Five framework, is with regard to "will do" criteria. Just what personailty dimenions are important is a matter of concern. Traditional military psychology, based as it was on WW II experience, said emotional stability, if combat effectriveness was a criterion and studied in real-world applications (war). Current work emphasizes conscientiousness as it is a dominating dimension in civilian and and peacetime applications, and perhaps even peaceful, settings. Will conscientiousness really help in high-stakes and threatening situations?
The Swedish Government has recently decided to create a professional army, where soldiers will get a small but decent salary. (SEK 16 500 per month). So far, the program is hugely popular with some 22 000 applicants for about 2000 openings. This means a selection ratio of 10 % which should make screening testing very feasible and effective. Values, held to be very important and measured in another US Arny project, can be measured by the proxy dimensional of emotional inteligence (EI) (self report).We have a wealth of data showing this. The second Army project uses another new personality test, GAT, (see earlier blog entry) but it sems to lack a reltionship to Tapas. No such studies were mentioned (and nobody asked). The GOT test is, by the way, kept secret and item formats and content are not disclosed, measring such things a spiritutal value sand justice seems to be a real challenge.
Check out these for the promise of a proxy measure of values:
Engelberg, E., & Sjöberg, L. (2005). Emotional intelligence and interpersonal skills. In R. D. Roberts & R. Schulze (Eds.), International handbook of emotional intelligence (pp. 289-308). Cambridge MA: Hogrefe.
Click here.
Engelberg, E., & Sjöberg, L. (2006). Money attitudes and emotional intelligence. Journal of Applied Social Psychology, 36(8), 2027-2047. Click here.
Engelberg, E., & Sjöberg, L. (2007). Money obsession, social adjustment, and economic risk perception. Journal of Socio-Economics, 36(5), 689-697. Click here.
Alternatively, write an e-mail to get reprints, write to lennartsjoberg@gmail.com
Friday, April 22, 2011
Saturday, April 16, 2011
Comments on SIOP 2011: Narrow vs broad traits
The tremendous interest in Big Five factors has not resulted in improvements of personality tests in the sense of increased validity. The reason is that the five factors all have no or only very modest validity in relation to job performance. Very extensive research, summarized in dozens of meta analyses, document this fact. There is now growing consensus about the need to develop and use measures of "narrow traits" in order to increase validity.
Some of these narrow traits are found among the facets of the Big Five, others not. The principle guiding the search for an improved basis of testing are seldom found in theory, but in common-sense thinking about what traits one could expected to be of importance with regard to important criteria, such as psychopathy in the case of counter-productive behavior. Other examples are emotional intelligence and affect. The Big Five cannot account for more than a minor share of the variance of any of the narrow traits.
In order to get improved personality tests there is a need for tests which complement the Big Five with a number of narrow trait scales which are focused on job functionality. Some can be found among the facets, others not. There is a need to limit the narrow trait scales in order to make test validation and test interpretation practically manageable. The illusory richness of tests having 30-40 subscales creates merely a feeling of understanding the tested person, a feeling which is a reflection of a well-known tendency to over-estimate the value of information, if there is more information.
The feeling that the Big Five somehow constitute a final answer to personality and personality testing is fading. The sooner that belief is abandoned, the better it is.
Some of these narrow traits are found among the facets of the Big Five, others not. The principle guiding the search for an improved basis of testing are seldom found in theory, but in common-sense thinking about what traits one could expected to be of importance with regard to important criteria, such as psychopathy in the case of counter-productive behavior. Other examples are emotional intelligence and affect. The Big Five cannot account for more than a minor share of the variance of any of the narrow traits.
In order to get improved personality tests there is a need for tests which complement the Big Five with a number of narrow trait scales which are focused on job functionality. Some can be found among the facets, others not. There is a need to limit the narrow trait scales in order to make test validation and test interpretation practically manageable. The illusory richness of tests having 30-40 subscales creates merely a feeling of understanding the tested person, a feeling which is a reflection of a well-known tendency to over-estimate the value of information, if there is more information.
The feeling that the Big Five somehow constitute a final answer to personality and personality testing is fading. The sooner that belief is abandoned, the better it is.
Comments on SIOP 2011: Faking on personality tests
The issue of faking is alive and well. Several sessions at the 2011 SIOP are devoted to it. Nobody or very few deny that faking occurs and that it can affect the outcome of a test, sometimes severely so. It is also realized that faking greatly hurts the credibility of personality testing. Non-experts test users simply are convinced that the test takers often fake good in a high-stakes situation, such as when they apply for a very desirable job or admittance to a prestige school.
It is clear that faking reduces the validity of personality tests, if left uncorrected. The effect can be very substantial. Meta analyses of the validity of personality tests tend to be based on data from incumbents, since job performance (criterion) data cannot normally be obtained from all applicants, and applicant scores are only correlated about 0.5 with incumbent scores. Hence, faking makes the data used in meta analyses of doubtful relevance to the question of test validity.
The most important of the Big Five factors, conscientiousness, is the one most affected by faking. It is also clear that the group of fakers, while heterogeneous, may contain some people who are risky to hire. Ignoring faking comes with great risks for the test users.
What can be done?A powerful alternative is to measure social desirability (SD) and use and SD scale to correct other scales for faking, to the extent hat they correlate with SD (not all scales do and correlations vary strongly in the typical case).The procedure has been validated both in experimental and field work.
There are a few objections, however.
1. SD scales are said to measure "personality". It is somewhat unclear what this means and why it is an argument. SD scales to have correlates with many other dimensions and they also have a certain amount of consistency over time and situations. So what? They can still measure faking at any given time.
2. There are several SD scales and they do not measure the same thing. The best known scales do have high intercorrelations, however.
3. You cannot detect who is a faker. Well, you can to some extent, albeit not perfectly, but who said that psychometrics ever comes up with perfect solutions?
4. Some people fake bad. This can be detected, but is not a major problem. Few people fake bad in a high-situations where they have applied for a desirable job.
Some commercial test suppliers and their agents try to solve the problem of faking by denying that it exists. This is not a credible statement. Since the future of personality testing is probably dependent on there being a solution to the faking problem - why not use the solution described here? It works.
It is clear that faking reduces the validity of personality tests, if left uncorrected. The effect can be very substantial. Meta analyses of the validity of personality tests tend to be based on data from incumbents, since job performance (criterion) data cannot normally be obtained from all applicants, and applicant scores are only correlated about 0.5 with incumbent scores. Hence, faking makes the data used in meta analyses of doubtful relevance to the question of test validity.
The most important of the Big Five factors, conscientiousness, is the one most affected by faking. It is also clear that the group of fakers, while heterogeneous, may contain some people who are risky to hire. Ignoring faking comes with great risks for the test users.
What can be done?A powerful alternative is to measure social desirability (SD) and use and SD scale to correct other scales for faking, to the extent hat they correlate with SD (not all scales do and correlations vary strongly in the typical case).The procedure has been validated both in experimental and field work.
There are a few objections, however.
1. SD scales are said to measure "personality". It is somewhat unclear what this means and why it is an argument. SD scales to have correlates with many other dimensions and they also have a certain amount of consistency over time and situations. So what? They can still measure faking at any given time.
2. There are several SD scales and they do not measure the same thing. The best known scales do have high intercorrelations, however.
3. You cannot detect who is a faker. Well, you can to some extent, albeit not perfectly, but who said that psychometrics ever comes up with perfect solutions?
4. Some people fake bad. This can be detected, but is not a major problem. Few people fake bad in a high-situations where they have applied for a desirable job.
Some commercial test suppliers and their agents try to solve the problem of faking by denying that it exists. This is not a credible statement. Since the future of personality testing is probably dependent on there being a solution to the faking problem - why not use the solution described here? It works.
Friday, April 15, 2011
Comments on SIOP 2011: Emotional intelligence
Yesterday's panel discussion of EI attracted a huge crowd, which certainly confirmed the statement of one of the panelists, that interest in EI is strong and steadily increasing. The panelists, all well-known researchers in the field, gave a good over-view of the field and agreed that the term should only be used for performance-based measures, not for so-called mixed models (read the Bar-on test). However, the major test of performance EI is the MSCEIT which has NO incremental validity in accounting for job performance, while mixed-model approaches do have such utility. So, why bother about performance EI, for other than theoretical purposes? It can be said that "we do not know what mixed-model tests of EI measure". This is true for such tests as the Bar-On (often used in Sweden) which indeed seem not to measure anything beyond the Big Five, or very little.
Tests can be devised which measure self-assessed EI, and they do make important contributions, admittedly for unclear reasons. Here is a promising topic for research. Meanwhile, they can be used in practical work. Worry about "faking" need not be a concern since scores can be adjusted for this bias factor with the help of one or more measures of social desirability - such adjustment is clearly necessary, by the way.
It is also interesting that performance and self-assessed EI have similar correlates such as age and gender so there is SOME evidence for a relationship, in spite for the very low within-group correlation.
In addition, work in my group has shown clear relationships between self-report EI and values: people with low EI in this sense tend to be materialistic, egoistic, and perhaps even manipulative. It is, reverse scored, a "dark side" measure of such attitudes and motivations, and could function as a good proxy measurement of them.
As a final comment, why continue work on performance EI if it offers so little of practical value? Sure, there are logical reasons it should have priority to the term EI, but that seems a remote advantage.
Tests can be devised which measure self-assessed EI, and they do make important contributions, admittedly for unclear reasons. Here is a promising topic for research. Meanwhile, they can be used in practical work. Worry about "faking" need not be a concern since scores can be adjusted for this bias factor with the help of one or more measures of social desirability - such adjustment is clearly necessary, by the way.
It is also interesting that performance and self-assessed EI have similar correlates such as age and gender so there is SOME evidence for a relationship, in spite for the very low within-group correlation.
In addition, work in my group has shown clear relationships between self-report EI and values: people with low EI in this sense tend to be materialistic, egoistic, and perhaps even manipulative. It is, reverse scored, a "dark side" measure of such attitudes and motivations, and could function as a good proxy measurement of them.
As a final comment, why continue work on performance EI if it offers so little of practical value? Sure, there are logical reasons it should have priority to the term EI, but that seems a remote advantage.
Monday, March 14, 2011
Ny proxyvalidering
Idén med proxyvalidering är att man relaterar sina testskalor till data på proxykriterier, som i sin tur är dimensioner av typ arbetsmotivation. Om de senare vet man att de har stabila och relativt höga samband med arbetsresultat eller andra viktiga dimensioner. Proxyvaliditeten är information om validitet hos ett test som är snabb att uppskatta och har man mätning av proxykriterierna inbakade i sitt test (vilket UPP har, men inga andra såvitt bekant) kan man rutinmässigt och utan extra kostnader göra studier av validiteten hos de test man använder, alltså även för test som INTE har proxykriterierna inbakade.
Jag har genomfört ett antal studier där jag DESSUTOM har haft tillgång till ett konventionellt externt kriterium i form av mått på arbetsresultat. Då har jag alltså kunnat uppskatta också validiteten av konventionellt slag. Resultatet har varit att proxyvaliditeten gett ungefär samma bild av testskalornas validitet som sambandet mellan dessa och det externa kriteriet.
I en pågående studie har jag undersökt "livsframgång" som kriterium. Denna har jag mätt i 12 olika avseenden som slagits ihop till ett sammansatt mått och relaterats till UPP-testets skalor. Dessa skalor har även relaterats till ett sedvanligt proxykriterium. De två serierna av korrelationer har relaterats i nedanstående figur.
Bilden visar ett starkt samband mellan de två serierna av korrelationer, vilket ger ytterligare starkt stöd för idén om proxyvaliditet. Hittills har alla studier av frågan gett ett sådan stöd. Vi kan nu vara ganska säkra på att proxyvalidering ger en hyfsad bild av validiteten hops ett test eller, för den delen, av vilken information som helst. Det är ett snabbt, enkelt och billigt sätt att validera, något som annars tar lång tid och är svårt och dyrbart att genomföra.
Jag har genomfört ett antal studier där jag DESSUTOM har haft tillgång till ett konventionellt externt kriterium i form av mått på arbetsresultat. Då har jag alltså kunnat uppskatta också validiteten av konventionellt slag. Resultatet har varit att proxyvaliditeten gett ungefär samma bild av testskalornas validitet som sambandet mellan dessa och det externa kriteriet.
I en pågående studie har jag undersökt "livsframgång" som kriterium. Denna har jag mätt i 12 olika avseenden som slagits ihop till ett sammansatt mått och relaterats till UPP-testets skalor. Dessa skalor har även relaterats till ett sedvanligt proxykriterium. De två serierna av korrelationer har relaterats i nedanstående figur.
Bilden visar ett starkt samband mellan de två serierna av korrelationer, vilket ger ytterligare starkt stöd för idén om proxyvaliditet. Hittills har alla studier av frågan gett ett sådan stöd. Vi kan nu vara ganska säkra på att proxyvalidering ger en hyfsad bild av validiteten hops ett test eller, för den delen, av vilken information som helst. Det är ett snabbt, enkelt och billigt sätt att validera, något som annars tar lång tid och är svårt och dyrbart att genomföra.
Friday, March 4, 2011
Intervjuernas validitet
Vid anställningar spelar intervjuer en stor roll, helt naturligt. Men frågan är i vilken mån man kan lita på bedömningar på den grunden. De vanligaste intervjuerna är troligen av typen "djupintervjuer". Sambandet mellan bedömningar utifrån sådana ostrukturerade och psykologiskt spekulativa intervjuer är 0.15 enligt en stor meta-analys, se referens nedan. Härtill kommer att intervjuer är dyrbara, åtminstone jämfört med psykologiska tester. På sina håll har man en mycket stark tilltro till "djupintervjuer", men forskningen ger som sagt inget stöd för den tilltron.
Strukturerade intervjuer fungerar betydligt bättre, kanske för att alla intervjuade får samma frågor och svaren bedöms på ett systematiskt sätt. Men ändå kan man inte vänta sig att personbedömningar på den grunden är högt korrelerade med annan information, som testdata. Detta är bra på det sättet att intervjuerna ger ett oberoende tillskott, men oroande eftersom de två typerna av information inte begreppsvaliderar varandra.
Referens
McDaniel, M. A., Whetzel, D. L., Schmidt, F. L., & Maurer, S. D. (1994). The validity of employment interviews: A comprehensive review and meta-analysis. [doi:10.1037/0021-9010.79.4.599]. Journal of Applied Psychology, 79(4), 599-616.
Strukturerade intervjuer fungerar betydligt bättre, kanske för att alla intervjuade får samma frågor och svaren bedöms på ett systematiskt sätt. Men ändå kan man inte vänta sig att personbedömningar på den grunden är högt korrelerade med annan information, som testdata. Detta är bra på det sättet att intervjuerna ger ett oberoende tillskott, men oroande eftersom de två typerna av information inte begreppsvaliderar varandra.
Referens
McDaniel, M. A., Whetzel, D. L., Schmidt, F. L., & Maurer, S. D. (1994). The validity of employment interviews: A comprehensive review and meta-analysis. [doi:10.1037/0021-9010.79.4.599]. Journal of Applied Psychology, 79(4), 599-616.
Wednesday, March 2, 2011
Värderingar och emotionell intelligens
Emotionell intelligens (EI) mäts på två sätt: självrapportskalor och prestation. Den förstnämnda metodiken har visat sig ge intressanta tillskott till andra personlighetsskalor och det är om sådana data som jag skriver här.
EI ansågs en tid vara en mycket viktig dimension i de flesta sammanhang i arbetslivet. Numera vet vi att den visserligen är av betydelse men också att den inte har så stort genomslag som man först trodde när det gäller arbetsresultat. Det finns emellertid en annan aspekt som är av vikt, nämligen sambandet EI - värderingar. Vi har funnit starka negativa samband mellan EI och värderingar som man gärna vill undvika i arbetslivet, nämligen en cynisk, manipulativ och materialistisk attityd.
Att mäta värderingar direkt kan vara svårt eftersom syftet genomskådas. Men har man en bra EI-skala kan den användas som ersättning (proxy) för värdeskalor. Värderingar är viktiga i arbetslivet. Därför kan man med fördel använda sig av EI i personlighetstest.
Referenser
Engelberg, E., & Sjöberg, L. (2005). Emotional intelligence and interpersonal skills. In R. D. Roberts & R. Schulze (Eds.), International handbook of emotional intelligence (pp. 289-308). Cambridge MA: Hogrefe. Klicka här.
Engelberg, E., & Sjöberg, L. (2006). Money attitudes and emotional intelligence. Journal of Applied Social Psychology, 36(8), 2027-2047. Klicka här.
Engelberg, E., & Sjöberg, L. (2007). Money obsession, social adjustment, and economic risk perception. Journal of Socio-Economics, 36(5), 689-697. Klicka här.
Sjöberg, L. (2001). Emotional intelligence: A psychometric analysis. European Psychologist, 6, 79-95. Klicka här.
EI ansågs en tid vara en mycket viktig dimension i de flesta sammanhang i arbetslivet. Numera vet vi att den visserligen är av betydelse men också att den inte har så stort genomslag som man först trodde när det gäller arbetsresultat. Det finns emellertid en annan aspekt som är av vikt, nämligen sambandet EI - värderingar. Vi har funnit starka negativa samband mellan EI och värderingar som man gärna vill undvika i arbetslivet, nämligen en cynisk, manipulativ och materialistisk attityd.
Att mäta värderingar direkt kan vara svårt eftersom syftet genomskådas. Men har man en bra EI-skala kan den användas som ersättning (proxy) för värdeskalor. Värderingar är viktiga i arbetslivet. Därför kan man med fördel använda sig av EI i personlighetstest.
Referenser
Engelberg, E., & Sjöberg, L. (2005). Emotional intelligence and interpersonal skills. In R. D. Roberts & R. Schulze (Eds.), International handbook of emotional intelligence (pp. 289-308). Cambridge MA: Hogrefe. Klicka här.
Engelberg, E., & Sjöberg, L. (2006). Money attitudes and emotional intelligence. Journal of Applied Social Psychology, 36(8), 2027-2047. Klicka här.
Engelberg, E., & Sjöberg, L. (2007). Money obsession, social adjustment, and economic risk perception. Journal of Socio-Economics, 36(5), 689-697. Klicka här.
Sjöberg, L. (2001). Emotional intelligence: A psychometric analysis. European Psychologist, 6, 79-95. Klicka här.
Subscribe to:
Posts (Atom)
