Dr Anna Brown

We introduce a new comparative response format, suitable for assessing personality and similar constructs. In this "graded-block" format, items measuring different constructs are first organized in blocks of 2 or more; then, pairs are formed from items within blocks. The pairs are presented one at a time, to enable respondents expressing the extent of preference for one item or the other using several graded categories. We model such data using confirmatory factor analysis (CFA) for ordinal outcomes. We derive Fisher information matrices for the graded pairs, and supply R code to enable computation of standard errors of trait scores. An empirical example illustrates the approach in low-stakes personality assessments and shows that similar results are obtained when using graded blocks of size 3 and a standard Likert format. However, graded-block designs may be superior when insufficient differentiation between items is expected (due to acquiescence, halo or social desirability).

INTRODUCTION: Family carers of people with dementia are their most important support in practical, personal and economic terms. Carers are vital to maintaining the quality of life (QOL) of people with dementia. This review aims to identify factors related to the QOL of family carers of people with dementia. METHODS: Searches on terms including 'carers', 'dementia', 'family' and 'quality of life' in research databases. Findings were synthesised inductively, grouping factors associated with carer QOL into themes. RESULTS: 909 abstracts were identified. Following screening, lateral searches and quality appraisal, 41 studies (n=5,539) were included for synthesis. Ten themes were identified: demographics; carer-patient relationship; dementia characteristics; demands of caring; carer health; carer emotional wellbeing; support received; carer independence; carer self-efficacy; and future. DISCUSSION: The quality and level of evidence supporting each theme varied. We need further research on what factors predict carer QOL in dementia and how to measure it.

Are recent cohorts of college students more narcissistic than their predecessors? To address debates about the so-called "narcissism epidemic," we used data from three cohorts of students (N1990s = 1,166; N2000s = 33,647; N2010s = 25,412) to test whether narcissism levels (overall and specific facets) have increased across generations. We also tested whether our measure, the Narcissistic Personality Inventory (NPI), showed measurement equivalence across the three cohorts, a critical analysis that had been overlooked in prior research. We found that several NPI items were not equivalent across cohorts. Models accounting for nonequivalence of these items indicated a small decline in overall narcissism levels from the 1990s to the 2010s (d = −0.27). At the facet-level, leadership (d = −0.20), vanity (d = –0.16), and entitlement (d = –0.28) all showed decreases. Our results contradict the claim that recent cohorts of college students are more narcissistic than earlier generations of college students.

Objective: Neurodegenerative diseases, such as dementia, have a profound impact on those with the conditions and their family carers. Consequently, the accurate measurement of family carers' quality of life (QOL) is important. Generic measures may miss key elements of the impact of these conditions so using disease-specific instruments has been advocated. This systematic review aimed to identify and examine the psychometric properties of disease-specific outcome measures of QOL of family carers of people with neurodegenerative diseases (Alzheimer's disease, other dementias; Huntington's disease; Parkinson's disease; Multiple Sclerosis; and Motor Neurone Disease). Design: Systematic review. Methods: Instruments were identified using five electronic databases (PubMed, PsycINFO, Web of Science, Scopus and IBSS) and lateral search techniques. Only studies which reported the development and/or validation of a disease-specific measure for adult family carers, and which were written in English, were eligible for inclusion. The methodological quality of the included studies was evaluated using the COSMIN checklist. The psychometric properties of each instrument were examined. Results: Six hundred and seventy six articles were identified. Following screening and lateral searches, a total of eight articles were included; these reported seven disease-specific carer QOL measures. Limited evidence was available for the psychometric properties of the seven instruments. Psychometric analyses were mainly focused on internal consistency, reliability and construct validity. None of the measures assessed either criterion validity or responsiveness to change. Conclusions: There are very few measures of carer QOL that are specific to particular neurodegenerative diseases. The findings of this review emphasise the importance of developing and validating psychometrically robust disease-specific measures of carer QOL.

Background: Early identification of children with potential development delay is essential to ensure access to care. The Ages & Stages Questionnaires (ASQ) are used as population outcome indicators in England as part of the 2.5 year review. Method: The aim of this article was to systematically review the worldwide evidence for the psychometric properties of the ASQ third edition (ASQ-3TM) and the Ages & Stages Questionnaires®: Social-Emotional (ASQ:SE). Eight electronic databases and grey literature were searched for original research studies available in English language, which reported reliability, validity, or responsiveness of the ASQ-3TM or ASQ:SE for children aged between 2 and 2.5 years. Twenty studies were included. Eligible studies used either the ASQ-3TM or the ASQ:SE and reported at least one measurement property of the ASQ-3TM and/or ASQ:SE. Data were extracted from all papers identified for final inclusion, drawing on Cochrane guidelines. Results: Using 'positive', 'intermediate', and 'negative' criteria for evaluating psychometric properties, results showed 'positive' reliability values in 11/18 instances reported, 'positive' sensitivity values in 13/18 instances reported, and 'positive' specificity values in 19/19 instances reported. Conclusions: Variations in age or language versions used, quality of psychometric properties, and quality of papers resulted in heterogeneous evidence. It is important to consider differences in cultural and contextual factors when measuring child development using these indicators. Further research is very likely to have an important impact on the interpretation of the ASQ-3TM and ASQ:SE psychometric evidence.

This article describes an investigation of whether Thurstonian item response modeling is a viable method for assessment of maladaptive traits. Forced-choice responses from 420 working adults to a broad-range personality inventory assessing six maladaptive traits were considered. The Thurstonian item response model's fit to the forced-choice data was adequate, while the fit of a counterpart item response model to responses to the same items but arranged in a single-stimulus design was poor. Mono-trait hetero-method correlations indicated corresponding traits in the two formats overlapped substantially, although they did not measure equivalent constructs. A better goodness of fit and higher factor loadings for the Thurstonian item response model, coupled with a clearer conceptual alignment to the theoretical trait definitions, suggested that the single-stimulus item responses were influenced by biases that the independent clusters measurement model did not account for. Researchers may wish to consider forced-choice designs and appropriate item response modeling techniques such as Thurstonian item response modeling for personality questionnaire applications in industrial psychology, especially when assessing maladaptive traits. We recommend further investigation of this approach in actual selection situations and with different assessment instruments.

Purpose There is a need to determine whether health-related quality-of-life (HRQL) assessments in dementia capture what is important, to form a coherent basis for guiding research and clinical and policy decisions. This study investigated structural validity of HRQL assessments made using the DEMQOL system, with particular interest in studying domains that might be central to HRQL, and the external validity of these HRQL measurements. Methods HRQL of people with dementia was evaluated by 868 self-reports (DEMQOL) and 909 proxy reports (DEMQOL-Proxy) at a community memory service. Exploratory and confirmatory factor analyses (EFA and CFA) were conducted using bifactor models to investigate domains that might be central to general HRQL. Reliability of the general and specific factors measured by the bifactor models was examined using omega (ω) and omega hierarchical (ω h) coefficients. Multiple-indicators multiple-causes models were used to explore the external validity of these HRQL measurements in terms of their associations with other clinical assessments. Results Bifactor models showed adequate goodness of fit, supporting HRQL in dementia as a general construct that underlies a diverse range of health indicators. At the same time, additional factors were necessary to explain residual covariation of items within specific health domains identified from the literature. Based on these models, DEMQOL and DEMQOL-Proxy overall total scores showed excellent reliability (ω h > 0.8). After accounting for common variance due to a general factor, subscale scores were less reliable (ω h < 0.7) for informing on individual differences in specific HRQL domains. Depression was more strongly associated with general HRQL based on DEMQOL than on DEMQOL-Proxy (−0.55 vs −0.22). Cognitive impairment had no reliable association with general HRQL based on DEMQOL or DEMQOL-Proxy. Conclusions The tenability of a bifactor model of HRQL in dementia suggests that it is possible to retain theoretical focus on the assessment of a general phenomenon, while exploring variation in specific HRQL domains for insights on what may lie at the 'heart' of HRQL for people with dementia. These data suggest that DEMQOL and DEMQOL-Proxy total scores are likely to be accurate measures of individual differences in HRQL, but that subscale scores should not be used. No specific domain was solely responsible for general HRQL at dementia diagnosis. Better HRQL was moderately associated with less depressive symptoms, but this was less apparent based on informant reports. HRQL was not associated with severity of cognitive impairment.

We examined the effects of response biases on 360-degree feedback using a large sample (N=4,675) of organizational appraisal data. Sixteen competencies were assessed by peers, bosses and subordinates of 922 managers, as well as self-assessed, using the Inventory of Management Competencies (IMC) administered in two formats – Likert scale and multidimensional forced choice. Likert ratings were subject to strong response biases, making even theoretically unrelated competencies correlate highly. Modeling a latent common method factor, which represented non-uniform distortions similar to those of "ideal-employee" factor in both self- and other assessments, improved validity of competency scores as evidenced by meaningful second-order factor structures, better inter-rater agreement, and better convergent correlations with an external personality measure. Forced-choice rankings modelled with Thurstonian IRT yielded as good construct and convergent validities as the bias-controlled Likert ratings, and slightly better rater agreement. We suggest that the mechanism for these enhancements is finer differentiation between behaviors in comparative judgements, and advocate the operational use of the multidimensional forced-choice response format as an effective bias prevention method.

In forced-choice questionnaires, respondents have to make choices between two or more items presented at the same time. Several IRT models have been developed to link respondent choices to underlying psychological attributes, including the recent MUPP (Stark, Chernyshenko & Drasgow, 2005) and Thurstonian IRT (Brown & Maydeu-Olivares, 2011) models. In the present article, a common framework is proposed that describes forced-choice models along three axes: 1) the forced-choice format used; 2) the measurement model for the relationships between items and psychological attributes they measure; and 3) the decision model for choice behavior. Using the framework, fundamental properties of forced-choice measurement of individual differences are considered. It is shown that the scale origin for the attributes is generally identified in questionnaires using either unidimensional or multidimensional comparisons. Both dominance and ideal point models can be used to provide accurate forced-choice measurement; and the rules governing accurate person score estimation with these models are remarkably similar.

To prevent response biases, personality questionnaires may use comparative response formats. These include forced choice, where respondents choose among a number of items, and quantitative comparisons, where respondents indicate the extent to which items are preferred to each other. The present article extends Thurstonian modeling of binary choice data (Brown & Maydeu-Olivares, 2011a) to "proportion-of-total" (compositional) formats. Following Aitchison (1982), compositional item data are transformed into log-ratios, conceptualized as differences of latent item utilities. The mean and covariance structure of the log-ratios is modelled using Confirmatory Factor Analysis (CFA), where the item utilities are first-order factors, and personal attributes measured by a questionnaire are second-order factors. A simulation study with two sample sizes, N=300 and N=1000, shows that the method provides very good recovery of true parameters and near-nominal rejection rates. The approach is illustrated with empirical data from N=317 students, comparing model parameters obtained with compositional and Likert scale versions of a Big Five measure. The results show that the proposed model successfully captures the latent structures and person scores on the measured traits.

A fundamental assumption in computerized adaptive testing (CAT) is that item parameters are invariant with respect to context – items surrounding the administered item. This assumption, however, may not hold in forced-choice (FC) assessments, where explicit comparisons are made between items included in the same block. We empirically examined the influence of context on item parameters by comparing parameter estimates from two FC instruments. The first instrument was compiled of blocks of three items, whereas in the second, the context was manipulated by adding one item to each block, resulting in blocks of four. The item parameter estimates were highly similar. However, a small number of significant deviations were observed, confirming the importance of context when designing adaptive FC assessments. Two patterns of such deviations were identified, and methods to reduce their occurrences in a FC CAT setting were proposed. It was shown that with a small proportion of violations of the parameter invariance assumption, score estimation remained stable.

The prevalent scoring practice for the Narcissistic Personality Inventory (NPI) ignores the forced-choice nature of the items. The aim of this study was to investigate whether findings based on NPI scores reported in previous research can be confirmed when the forced-choice nature of the NPI's original response format is appropriately modeled, and when NPI items are presented in different response formats (true/false or rating scale). The relationships between NPI facets and various criteria were robust across scoring approaches (mean score vs. model-based), but were only partly robust across response formats. In addition, the scoring approaches and response formats achieved equivalent measurements of the vanity facet and in part of the leadership facet, but differed with respect to the entitlement facet.

At a fundamental level, taxonomy of behavior and behavioral tendencies can be described in terms of approach, avoid, or equivocate (i.e., neither approach nor avoid). While there are numerous theories of personality, temperament, and character, few seem to take advantage of parsimonious taxonomy. The present study sought to implement this taxonomy by creating a questionnaire based on a categorization of behavioral temperaments/tendencies first identified in Buddhist accounts over fifteen hundred years ago. Items were developed using historical and contemporary texts of the behavioral temperaments, described as "Greedy/Faithful", "Aversive/Discerning", and "Deluded/Speculative". To both maintain this categorical typology and benefit from the advantageous properties of forced-choice response format (e.g., reduction of response biases), binary pairwise preferences for items were modeled using Latent Class Analysis (LCA). One sample (n1 = 394) was used to estimate the item parameters, and the second sample (n2 = 504) was used to classify the participants using the established parameters and cross-validate the classification against multiple other measures. The cross-validated measure exhibited good nomothetic span (construct-consistent relationships with related measures) that seemed to corroborate the ideas present in the original Buddhist source documents. The final 13-block questionnaire created from the best performing items (the Behavioral Tendencies Questionnaire or BTQ) is a psychometrically valid questionnaire that is historically consistent, based in behavioral tendencies, and promises practical and clinical utility particularly in settings that teach and study meditation practices such as Mindfulness Based Stress Reduction (MBSR).

Purpose: The Psychological Inventory of Criminal Thinking Styles (PICTS) has been applied extensively to the study of criminal behaviour and cognition. This study aimed to explore the psychometric characteristics (factorial structure, reliability and external validity) of an Arabic version of the PICTS, to explore cross-cultural differences between a sample of Middle-Eastern (Egyptian) prisoners and Western prison samples, and to examine the influence of type of crime on criminal thinking styles. Method: A group of 130 Egyptian male prisoners who had been sentenced for theft, drug dealing or murder completed the PICTS. Their scores were compared with the reported data of American, British, and Dutch prisoners. Results: The Arabic PICTS showed scale reliabilities estimated by coefficient alpha comparable to the English version, and reliabilities estimated as test-retest correlations were high. Confirmatory factor analysis showed that the PICTS subscale scores of Egyptian prisoners best fitted a two-factor model, in which one dimension comprised mollification, entitlement, superoptimism, sentimentality and discontinuity, and the second dimension reflected the thinking styles of power orientation, cut-off and cognitive indolence. Observed levels of thinking styles varied by type of crime, specifically between prisoners sentenced for theft, drug dealing, and murder. Cultural differences in criminal thinking styles were also found, whereby the Egyptian prisoners recorded the highest scores in most thinking styles, while American, Dutch and English prisoners were more comparable to each other. Conclusions: This study provides one of the first investigations of criminal thinking styles in a non-Western sample and suggests that cross-cultural differences in the structure of these thinking styles exist. In addition, the results indicate that criminal thinking styles need to be understood by the type of crime for which a person has been sentenced.

The present research addressed gaps in our current understanding of validity and quality of measurement provided by Patient Reported Experience Measures (PREM). We established the psychometric properties of a freely available Experience of Service Questionnaire (ESQ), based on responses from 7,067 families of patients across 41 UK providers of Child and Adolescent Mental Health Services (CAMHS), using the two-level latent trait modeling. Responses to the ESQ were subject to strong 'halo' effects, which were thought to represent the overall positive or negative affect towards one's treatment. Two strongly related constructs measured by the ESQ were interpreted as specific aspects of global satisfaction, namely Satisfaction with Care, and with Environment. The Care construct was sensitive to differences between less satisfied patients, facilitating individual and service-level problem evaluation. The effects of nesting within service providers were strong, with parental reports being the most reliable source of data for the between-provider comparisons. We provide a scoring protocol for converting the hand-scored ESQ to the model-based population-referenced scores with supplied standard errors, which can be used for benchmarking services as well as individual evaluations.

AIMS: To investigate pathways through which momentary negative affect and depressive symptoms affect risk of lapse during smoking cessation attempts. DESIGN: Ecological Momentary Assessment was carried out during two weeks after an unassisted smoking cessation attempt. A three-month follow-up measured smoking frequency. SETTING:Data were collected via mobile devices in German-speaking Switzerland. PARTICIPANTS: A total of 242 individuals (age 20-40, 67% men) reported 7,112 observations. MEASUREMENTS: Online surveys assessed baseline depressive symptoms and nicotine dependence. Real-time data on negative affect, physical withdrawal symptoms, urge to smoke, abstinence-related self-efficacy, and lapses. FINDINGS: Two-level structural equation model suggested that on the situational level, negative affect increased the urge to smoke and decreased self-efficacy (β = .20; β = -.12, respectively), but had no direct effect on lapse risk. A higher urge to smoke (β = .09) and lower self-efficacy (β = -.11) were confirmed as situational antecedents of lapses. Depressive symptoms at baseline were a strong predictor of a person's average negative affect (β = .35, all p <.001). However, the baseline characteristics influenced smoking frequency three months later only indirectly, through influences of average states on the number of lapses during the quit attempt. CONCLUSIONS: Controlling for nicotine dependence, higher depressive symptoms at baseline were strongly associated with a worse longer-term outcome. Negative affect experienced during the quit attempt was the only pathway through which the baseline depressive symptoms were associated with a reduced self-efficacy and increased urges to smoke, all leading to the increased probability of lapses.

Perfectionism cognitions capture automatic perfectionistic thoughts and have explained variance in psychological adjustment and maladjustment beyond trait perfectionism. The aim of the present research was to investigate whether a multidimensional assessment of perfectionism cognitions has advantages over a unidimensional assessment. To this aim, we examined in a sample of 324 university students how the Perfectionism Cognitions Inventory (PCI) and the Multidimensional Perfectionism Cognitions Inventory (MPCI) explained variance in positive affect, negative affect, and depressive symptoms when factor or subscale scores were used as predictors compared to total scores. Results showed that a multidimensional assessment (PCI factor scores, MPCI subscale scores) explained more variance than a unidimensional assessment (PCI and MPCI total scores) because, when the different dimensions were entered simultaneously as predictors, perfectionistic strivings cognitions and perfectionistic concerns cognitions acted as mutual suppressors thereby increasing each others' predictive validity. With this, the present findings provide evidence that―regardless of whether the PCI or the MPCI is used―a multidimensional assessment of perfectionism cognitions has advantages over a unidimensional assessment in explaining variance in psychological adjustment and maladjustment.

We reply to Flett and Hewitt's (2014) commentary on our findings (Stoeber, Kobori, & Brown, 2014) focusing on the multidimensionality of the Perfectionism Cognitions Inventory (PCI) and the question of whether the Multidimensional Perfectionism Cognitions Inventory (MPCI) represents an alternative to the PCI. In addition, we reiterate the importance of considering suppression effects when examining different dimensions of perfectionism and, in concluding, invite researchers to join forces to further advance the assessment of multidimensional perfectionism cognitions.

Perfectionism is a personality characteristic that has been found to predict sports performance in athletes. To date, however, research has exclusively examined this relationship at an individual level (i.e., athletes' perfectionism predicting their personal performance). The current study extends this research to team sports by examining whether, when manifested at team level, perfectionism predicts team performance. A sample of 231 competitive rowers from 36 boats completed measures of self-oriented, team-oriented, and team-prescribed perfectionism prior to competing against one another in a 4-day rowing competition. Strong within-boat similarities in the levels of team members' team-oriented perfectionism supported the existence of collective team-oriented perfectionism at the boat level. Two-level latent growth curve modeling of day-by-day boat performance showed that team-oriented perfectionism positively predicted the position of the boat in mid-competition and the linear improvement in position. The findings suggest that imposing perfectionistic standards on team members may drive teams to greater levels of performance.

We report a Monte Carlo study examining the effects of two strategies for handling measurement non-invariance – modeling and ignoring non-invariant items – on structural regression coefficients between latent variables measured with item response theory models for categorical indicators. These strategies were examined across four levels and three types of non-invariance – non-invariant loadings, non-invariant thresholds, and combined non-invariance on loadings and thresholds – in simple, partial, mediated and moderated regression models where the non-invariant latent variable occupied predictor, mediator, and criterion positions in the structural regression models. When non-invariance is ignored in the latent predictor, the focal group regression parameters are biased in the opposite direction to the difference in loadings and thresholds relative to the referent group (i.e., lower loadings and thresholds for the focal group lead to overestimated regression parameters). With criterion non-invariance, the focal group regression parameters are biased in the same direction as the difference in loadings and thresholds relative to the referent group. While unacceptable levels of parameter bias were confined to the focal group, bias occurred at considerably lower levels of ignored non-invariance than was previously recognized in referent and focal groups.

In multidimensional forced-choice (MFC) questionnaires, items measuring different attributes are presented in blocks, and participants have to rank-order the items within each block (fully or partially). Such comparative formats can reduce the impact of numerous response biases often affecting single-stimulus items (aka, rating or Likert scales). However, if scored with traditional methodology, MFC instruments produce ipsative data, whereby all individuals have a common total test score. Ipsative scoring distorts individual profiles (it is impossible to achieve all high or all low scale scores), construct validity (covariances between scales must sum to zero), criterion related validity (validity coefficients must sum to zero), and reliability estimates. We argue that these problems are caused by inadequate scoring of forced-choice items, and advocate the use of item response theory (IRT) models based on an appropriate response process for comparative data, such as Thurstone's Law of Comparative Judgment. We show that by applying Thurstonian IRT modeling (Brown & Maydeu-Olivares, 2011), even existing forced-choice questionnaires with challenging features can be scored adequately and that the IRT-estimated scores are free from the problems of ipsative data.

Early detection of child mental health problems in schools is critical for implementing strategies for prevention and intervention. The development of an effective measure of mental health and well-being for this context must be both empirically sound and practically feasible. This study reports the initial validation of a brief self-report measure for child mental health suitable for use with children as young as eight ("Me and My School" (M&MS)). After factor analysis, and studies of measurement invariance, two subscales emerged: emotional difficulties and behavioral difficulties. These two subscales were highly correlated with corresponding constructs of the Strengths and Difficulties Questionnaire (SDQ) and showed correlations with attainment, deprivation and educational needs similar to ones obtained between these demographic measures and the SDQ. Results suggest that this school-based self-report measure is psychometrically sound, and has the potential of contributing to school mental health surveys, evaluation of interventions, and recognition of mental health problems within schools.

To counter response distortions associated with the use of rating scales (a. k. a. Likert scales), items can be presented in a comparative fashion, so that respondents are asked to rank the items within blocks (forced-choice format). However, classical scoring procedures for these forced-choice designs lead to ipsative data, which presents psychometric challenges that are well described in the literature. Recently, Brown and Maydeu-Olivares (Educational and Psychological Measurement 71: 460-502, 2011a) introduced a model based on Thurstone's law of comparative judgment, which overcomes the problems of ipsative data. Here, we provide a step-by-step tutorial for coding forced-choice responses, specifying a Thurstonian item response theory model that is appropriate for the design used, assessing the model's fit, and scoring individuals on psychological attributes. Estimation and scoring is performed using Mplus, and a very straightforward Excel macro is provided that writes full Mplus input files for any forced-choice design. Armed with these tools, using a forced-choice design is now as easy as using ratings.

Risk behaviors such as substance use or deviance are often limited to the early stages of the life course. Whereas the onset of risk behavior is well studied, less is currently known about the decline and timing of cessation of risk behaviors of different domains during young adulthood. Prevalence and longitudinal developmental patterning of alcohol use, drinking to the point of drunkenness, smoking, cannabis use, deviance, and HIV-related sexual risk behavior were compared in a Swiss community sample (N = 2,843). Using a longitudinal cohort-sequential approach to link multiple assessments with 3 waves of data for each individual, the studied period spanned the ages of 16 to 29 years. Although smoking had a higher prevalence, both smoking and drinking up to the point of drunkenness followed an inverted U-shaped curve. Alcohol consumption was also best described by a quadratic model, though largely stable at a high level through the late 20s. Sexual risk behavior increased slowly from age 16 to age 22 and then remained largely stable. In contrast, cannabis use and deviance linearly declined from age 16 to age 29. Young men were at higher risk for all behaviors than were young women, but apart from deviance, patterning over time was similar for both sexes. Results about the timing of increase and decline as well as differences between risk behaviors may inform tailored prevention programs during the transition from late adolescence to adulthood.

Multidimensional forced-choice formats can significantly reduce the impact of numerous response biases typically associated with rating scales. However, if scored with classical methodology, these questionnaires produce ipsative data, which lead to distorted scale relationships and make comparisons between individuals problematic. This research demonstrates how item response theory (IRT) modeling may be applied to overcome these problems. A multidimensional IRT model based on Thurstone's framework for comparative data is introduced, which is suitable for use with any forced-choice questionnaire composed of items fitting the dominance response model, with any number of measured traits, and any block sizes (i.e., pairs, triplets, quads, etc.). Thurstonian IRT models are normal ogive models with structured factor loadings, structured uniquenesses, and structured local dependencies. These models can be straightforwardly estimated using structural equation modeling (SEM) software Mplus. A number of simulation studies are performed to investigate how latent traits are recovered under various forced-choice designs and provide guidelines for optimal questionnaire design. An empirical application is given to illustrate how the model may be applied in practice. It is concluded that when the recommended design guidelines are met, scores estimated from forced-choice questionnaires with the proposed methodology reproduce the latent traits well.

Although rating differences among incumbents of the same occupation have traditionally been viewed as error variance in the work analysis domain, such differences might often capture substantive discrepancies in how incumbents approach their work. This study draws from job crafting, creativity, and role theories to uncover situational factors (i.e., occupational activities, context, and complexity) related to differences among competency ratings of the same occupation. The sample consisted of 192 incumbents from 64 occupations. Results showed that 25% of the variance associated with differences in competency ratings of the same occupation was related to the complexity, the context, and primarily the nature of the occupation's work activities. Consensus was highest for occupations involving equipment-related activities and direct contact with the public.

The comparative format used in ranking and paired comparisons tasks can significantly reduce the impact of uniform response biases typically associated with rating scales. Thurstone's (1927, 1931) model provides a powerful framework for modeling comparative data such as paired comparisons and rankings. Although Thurstonian models are generally presented as scaling models, that is, stimuli-centered models, they can also be used as person-centered models. In this article, we discuss how Thurstone's model for comparative data can be formulated as item response theory models so that respondents' scores on underlying dimensions can be estimated. Item parameters and latent trait scores can be readily estimated using a widely used statistical modeling program. Simulation studies show that item characteristic curves can be accurately estimated with as few as 200 observations and that latent trait scores can be recovered to a high precision. Empirical examples are given to illustrate how the model may be applied in practice and to recommend guidelines for designing ranking and paired comparisons tasks in the future.

In this two part series, James Bywater and Anna Brown summarise some of the issues involved in determining the correct length of assessment in a personality questionnaire (PQ). In the first instalment they discuss the general issues that test designers face, and in the second they cover some more modern solutions to these, with associated disadvantages. It is aimed at practitioners rather than hard core psychometricians and can not be exhaustive. However wherever possible it attempts to distil out practical messages for the audience.

Forced-choice tests, despite being resistant to response biases and showing good operational validities, have psychometric problems if scored traditionally. These questionnaires are generally longer than their normative counterparts, and more cognitively challenging. The OPQ32i was shortened and re-scored using the latest advances in IRT. One item was removed out of each block, making the completion quicker and less cognitively complex. The shortened version (OPQ32r) shows good reliability, equivalent or better validity than the full ipsative version, and produces scale scores with normative properties. Results suggest that the IRT methodology can significantly improve efficiency of existing forced-choice measures so that test takers can do less (complete shorter and easier questionnaire) and test users can get more (bias-resistant instrument of superior psychometric quality).

In this two part series, James Bywater and Anna Brown summarise some of the issues involved in determining the correct length of assessment in a personality questionnaire (PQ). In the last edition of Assessment & Development Matters they discussed the general issues that test designers face, and in this one they cover some more modern solutions to these. It is aimed at practitioners rather than hard core psychometricians and can not be exhaustive. However wherever possible it attempts to distil out practical messages for the audience.

Correlations between Big Five personality factors and other variables have been examined in three different ways: direct scoring of items within a factor, application of a composite score formula, and taking the average of single-scale correlations. Those methods were shown to yield consistently different outcomes in four sets of data from sales-people and managers. Factor correlations with job performance were greatest for direct scoring, and were reduced by half when scale correlations were averaged. The insertion of previously suggested estimates into the composite score formula yielded intermediate correlations with performance. It is necessary to interpret summary accounts of correlations with a compound construct in the light of the aggregation method employed.

This study explores the equivalence of web-based administration with no local supervision and traditional paper-and-pencil supervised versions of OPQ32i (the ipsative format version of the Occupational Personality Questionnaire). Samples of data were collected from a range of client projects and matched in terms of industry sector, assessment purpose (selection or development) and candidate category (graduate or managerial/professional). The analysis indicates that lack of local supervision in high-stakes situations has little if any impact on scale scores. At worst, some scales appear to show shifts of less than quarter of an SD, with most scales showing little change, if any. Analyses in terms of the Big Five show differences of less than .2 of an SD. Scale reliabilities and scale covariances appear to be unaffected by the differences between the supervised and unsupervised administration conditions.

Instead of responding to questionnaire items one at a time, respondents may be forced to make a choice between two or more items measuring the same or different traits. The forced-choice format eliminates uniform response biases, although the research on its effectiveness in reducing the effects of impression management is inconclusive. Until recently, forced-choice questionnaires were scaled in relation to person means (ipsative data), providing information for intra-individual assessments only. Item response modeling enabled proper scaling of forced-choice data, so that inter-individual comparisons may be made. New forced-choice applications in personality assessment and directions for future research are discussed.

The ultimate goal of measurement is to produce a score by which individuals can be assessed and differentiated. Item response theory (IRT) modeling views responses to test items as indicators of a respondent's standing on some underlying psychological attributes (van der Linden & Hambleton, 1997) – we often call them latent traits – and devises special algorithms for estimating this standing. This chapter gives an overview of methods for estimating person attribute scores using one-dimensional and multi-dimensional IRT models, focusing on those that are particularly useful with patient-reported outcome (PRO) measures. To be useful in applications, a test score has to approximate the latent trait well, and importantly, the precision level must be known in order to produce information for decision-making purposes. Unlike classical test theory (CTT), which assumes the precision with which a test measures the same for all trait levels, IRT methods assess the precision with which a test measures at different trait levels. In the context of patient-reported outcomes measurement, this enables assessment of the measurement precision for an individual patient. Knowing error bands around the patient's score is important for informing clinical judgments, such as deciding upon significance of any change, for instance in response to treatment etc. (Reise & Haviland, 2005). At the same time, summary indices are often needed to summarize the overall precision of measurement in a research sample, population group, or in the population as a whole. Much of this chapter is devoted to methods for estimating measurement precision, including the score-dependent standard error of measurement and appropriate sample-level or population-level marginal reliability coefficients. Patient-reported outcome measures often capture several related constructs, the feature that may make the use of multi-dimensional IRT models appropriate and beneficial (Gibbons, Immekus & Bock, 2007). Several such models are described, including a model with multiple correlated constructs, a model where multiple constructs are underlain by a general common factor (second-order model), and a model where each item is influenced by one general and one group factor (bifactor model). To make the use of these models more easily accessible for applied researchers, we provide specialized formulae for computing test information, standard errors and reliability. We show how to translate a multitude of numbers and graphs conditioned on several dimensions into easy-to-use indices that can be understood by applied researchers and test users alike. All described methods and techniques are illustrated with a single data analysis example involving a popular PRO measure, the 28-item version of the General Health Questionnaire (GHQ28; Goldberg & Williams, 1988), completed in mid-life by a large community sample as a part of a major UK cohort study.

A new SHL approach to designing and scoring forced-choice questionnaires using Item Response Theory (IRT) has enabled a revolutionary improvement in efficiency, accuracy and scaling properties of OPQ32 trait scores, leading to the new OPQ32r. This manual presents technical information relating the IRT scoring model and the new OPQ32r instrument. It should be read in conjunction with the more detailed and extensive OPQ32 Technical Manual (SHL, 2006) which covers the design, development and technical characteristics of the OPQ32i and OPQ32n.

This Technical Manual is intended to be read in conjunction with the OPQ32 User Manual. The content of the latter focus on administration, scoring, norming and interpretation issues, and is intended to cover all the matters one needs to refer to when using the OPQ32. The technical manual is intended for reference purposes and provides all the technical backup needed when evaluating the OPQ32 in terms of its suitability for use.

Asking people to assess themselves or others on a set of psychological characteristics is by far the most popular method of gathering data in our field. We use this method either because it is the cheapest, or the best there currently exists for measuring the target characteristic. However, respondent-reported data are commonly affected by conscious and unconscious response distortions. Examples include individual styles in using rating options, inattention or cognitive difficulties in responding to reversed items, tendency to present self in positive light, halo effects, distortions driven by political pressures etc. The extent to which respondents engage in such behaviors varies, and if not controlled, the biases alter the true ordering of respondents on traits of interest. Response distortions, therefore, should concern everyone who uses respondent-reported measures. This talk provides an overview of research on biasing factors evoked by responding to questionnaire items with different features and in different contexts, discussing the evolution of views on the problem. I will discuss the emerging methods of statistical control, which explicitly incorporate biases in the models of item-level response processes (e.g. Böckenholt, 2012; Johnson & Bolt, 2010). These methods offer a great promise as well as natural limitations in their applicability and scope. Alternatives to statistical control include prevention, and there have been advances in this area too. Special response formats are one of the bias prevention methods, with the forced-choice format being particularly promising. During the past 10-15 years we have acquired methodology that enables modelling forced-choice data. This enables comparing the effectiveness in bias control of the two methods – statistical control versus prevention. I will report latest findings in this regard and share some of my own views and recommendations for the use of these methods depending on the context and stakes of assessments. I will argue that despite some significant progress, we are still far off bias-proof assessments. In order to create a breakthrough in this area, we must invest in research of test taker cognitions, mixing qualitative and quantitative methods. Few available studies of test taker behavior (e.g. Kuncel & Tellegen, 2009; Robie, Brown, & Beaty, 2007) show that the test takers have conflicting motives, and complex cognitions when it comes to sitting our assessments. Only when we understand these factors, can we hope to create better assessments.

Multidimensional forced-choice (MFC) questionnaires typically show good validities and are resistant to impression management effects. However, they yield ipsative data, which distorts scale relationships and makes comparisons between people problematic. Depressed reliability estimates also led developers to create tests of potentially excessive length. We apply an IRT Preference Model to make more efficient use of information in existing MFC questionnaires. OPQ32i used for selection and assessment internationally is examined using this approach. The latent scores recovered from a much reduced number of MFC items are superior to the full test‟s ipsative scores, and comparable to unbiased normative scores.

The Big Five Questionnaire (Brown & Maydeu-Olivares, 2011) was developed in the context of a study researching item response theory (IRT) modeling of forced-choice questionnaires. The purpose of the questionnaire is to measure the Big Five personality factor markers. Items were drawn from the 100 items of the International Personality Item Pool. The authors selected 60 items so that 12 items would measure each of the five marker traits, with 8 positively and 4 negatively keyed items per trait combined in a way that equal number of pairwise comparisons occur between items keyed in the same direction and items keyed in opposite directions. Each block of the questionnaire was presented in two formats. First, participants rated the three items using a 5-point rating scale, ranging from "very accurate" to "very inaccurate." This single-stimulus presentation was immediately followed by the forced-choice presentation, where the participants were asked to select one "most like me" item and one "least like me" out of the same block of three items. A total of 438 volunteers from the United Kingdom completed the questionnaire online. The reliability estimates ranged from .775 to .844 for the single-stimulus data and from .601 to .766 for the forced-choice data. The maximum a posteriori estimated trait scores for individuals based on single-stimulus and forced-choice responses correlated strongly, with correlations ranging from .69 for Agreeableness to .82 for Extraversion.

To counter response distortions associated with the use of rating scales in personality and similar assessments, test items may be presented in so-called 'forced-choice' formats. Respondents may be asked to rank-order a number of items, or distribute a fixed number of points between several items – therefore they are forced to make a choice. Until recently, basic classical scoring methods were applied to such formats, leading to scores relative to the person's mean (ipsative scores). While interpretable in intra-individual assessments, ipsative scores are problematic when used for inter-individual comparisons. Recent advances in estimation methods enabled rapid development of item response models for comparative data, including the Thurstonian IRT model (Brown & Maydeu-Olivares, 2011a), the Multi-Unidimensional Pairwise Preference model (Stark, Chernyshenko & Drasgow, 2005), and others. Appropriate item response modeling enables estimation of person scores that are directly interpretable for inter-individual comparisons, without the distortions and artifacts produced by ipsative scoring.

The ultimate goal of psychometric testing is to produce a score by which people can be differentiated. Item Response Theory (IRT) devises methods for estimating person's score on one or more psychological constructs (traits) from his/her responses to test items. This chapter gives an overview of scoring methods applicable to situations when the test items indicate one trait only; or a set of related traits but each item contributes to measurement of one trait; or when each item indicates multiple traits. We consider scoring methods based on item responses only, as well as Bayesian methods, which use prior knowledge of the trait distribution. Much of this chapter is devoted to methods for assessing measurement precision provided by individual items, the whole test, and the prior distribution. In IRT, this precision can be evaluated for each individual response pattern. All described methods are illustrated with a single empirical example.