New ways to tackle diversity The Power of SJQs


Designing a situational judgement tool for diversity


The Civil Service Fast Stream programme strives to attract graduates of the highest ability, equipped to meet the demands of a hugely varied central government environment. Additionally, there is a continuous aspiration to identify the widest blend of diverse applicants, to build a representative and talented staff profile from every background, explains Phil Wilson, Chief Assessor and Chief Psychologist at Civil Service Fast Stream.


The questionnaire


integrated carefully constructed and piloted workplace scenarios, each with a series of response options.


The aim Fast Stream has worked with Cubiks, the psychometric publishers, for over 10 years. During that period, a portfolio of innovative assessment tools has been developed. Most recently, a new bespoke situational judgement questionnaire (SJQ) was executed to supplement the existing psychometric offer. Based on evidence from existing research, an SJQ was an opportunity to achieve a positive diversity/quality mix.


The USP The questionnaire integrated carefully constructed and piloted workplace scenarios, each with a series of response options. Whilst SJQs have obviously been widely used in many graduate assessment packages, the new Fast Stream design was based on an approach that was distinctive for a number of reasons: • An unusually extensive trialling phase, involving multiple pilot items.


• An extended analysis process which compared and contrasted a number of different scoring methods.


• A focus on diversity metrics to support a wider equality agenda - drawing on the best performing trial items.


• Utilisation of on-line open access (pre- application) practice to support familiarity.


• Provision of skills coaching to target groups, supporting effective SJQ candidate performance.


The design To illustrate the extent of the analysis: • Over 70 SJQ scenarios/items were


20 Graduate Recruiter | www.agr.org.uk


developed, incorporating different candidate response styles (ranking and rating), drawing on the insights of Civil Service subject matter experts to construct the situations, steered towards assessment competency areas.


• The scenarios were trialled in seven different test formats, on over 11,000 Fast Stream candidates, for Generalist, European and Digital roles. 45 Fast Stream managers also completed the SJQ to bring in their perspective.


• Potential group performance differences were analysed based on gender, ethnicity, socio-economic status, Oxbridge attendance and Russell Group attendance.


• Correlation analysis was conducted on SJQ item scores against assessment centre performance.


The trial analysis suggested that using SJQ assessment had positive outcomes and findings, in particular: • Performance on the SJQ assessment successfully predicted final stage assessment centre performance (which in turn correlates with workplace performance).


• The SJQ significantly correlated with other assessment exercises completed by Fast Stream applicants.


• The most successful response style was to rank answer options (i.e. ‘most’ and ‘least’) with strong correlations against assessment centre performance, compared to rated options (i.e. ‘strongly agree’, ‘agree’, ‘disagree’, etc.)


• Limiting the response format to ranking in this way lessens the cognitive load, to improve candidate experience.


• No gender or ethnicity adverse impact was identified with the ranking approach.


• The subject matter expert score was a better predictor of assessment centre performance than the candidate ranking during trialling.


• The SJQ worked better with no time limit – statistical review showed that the largest correlation was where all items/scenarios were completed.


The outcome Moving from the trial to live format has delivered promising metrics and has demonstrated the benefit of careful construction – at the time of writing, highlights include: • Diversity outcomes for the largest (Generalist) Fast Stream scheme, comprising - so far – 4,500 candidates, showed no group differences for any category – gender, ethnicity, disability, Oxbridge and socio-economic status.


• A bespoke open access practice SJQ has been undertaken around 4,000 times in the initial few weeks of the application window.


• SJQ coaching sessions with diverse groups have encompassed well received workshops.


The tool will continue to be rolled out and initial outcomes have been encouraging, justifying the rigorous and carefully structured design approach. n


Page 1  |  Page 2  |  Page 3  |  Page 4  |  Page 5  |  Page 6  |  Page 7  |  Page 8  |  Page 9  |  Page 10  |  Page 11  |  Page 12  |  Page 13  |  Page 14  |  Page 15  |  Page 16  |  Page 17  |  Page 18  |  Page 19  |  Page 20  |  Page 21  |  Page 22  |  Page 23  |  Page 24  |  Page 25  |  Page 26  |  Page 27  |  Page 28  |  Page 29  |  Page 30  |  Page 31  |  Page 32