Business Intelligence

for Test Administration instructions click here

Introduction - Why another general reasoning test?

The simple response to this question is, as consultants who use assessment tools for selection and development, we were not particularly happy with the choice of tests available to us as practitioners, and we were confident others would share our misgivings about the test materials available.

What were our misgivings? Firstly, whilst many general reasoning tests have excellent technical merits, they are essentially rooted in an abstract world rather than an occupational one. Typically these tests involve looking at abstract patterns and trying to identify what is common to these shapes or what shape is next in the series. In our experience, individuals become less motivated when confronted with psychometric tests which they cannot easily relate to an occupational setting.

For psychometric tests to be useful, individuals need to be sufficiently motivated to give their best performance. Unfortunately, the effectiveness of many commercially available general reasoning tests is undermined due to this abstract, context free non-specific content and form. Candidates do not readily understand the reason why they are being asked to take the test, or the value to themselves or those testing them. And yet one of the most robust findings of occupational psychology is the utility of measures of general intelligence- particularly where the job is non -routine and involves complex forms of decision making.

Consequently, we believe there is a huge need for a general reasoning test which is focussed on the intellectual and commercial demands encountered in an occupational setting. In other words sampling general intelligence but in a socially acceptable way.

To provide an illustration of our point, consider the Differential Aptitude Test (DAT) series. We would suggest that this is the benchmark general reasoning test, superbly constructed and standardised, and rightly long-lived. However, despite its excellent psychometric properties, it can be a problematic tool to use in some environments. We have often found in our experience as practitioners individuals find it difficult to relate the abstract demands of this general reasoning test to their current job, or the job they are applying for.

Also, from the large number of existing tests explicitly designed to assess general reasoning ability, it is often hard to find an instrument to match the level of difficulty required by the assessment situation. For example, although the Graduate and Managerial Assessment (GMA) series incorporates an excellent abstract reasoning test, many candidates, even those applying for senior level positions, have found the test difficult and obtuse. The test does not appear to many candidates to fit easily with the analytical demands of their current job, or the job they are applying for, and so candidates often find it hard to motivate themselves sufficiently when taking this test.

In contrast, the Watson-Glazer Critical Thinking Appraisal provides an excellent example of how the test publishers have attempted to meet the requirements of a general reasoning test whilst making the content of the test more accessible to those taking the test. However, the ability sampled by critical reasoning tests such as the Watson Glaser represents more of a skill than a form of intelligence. It is relatively independent from intelligence . Thus, as practitioners, we wanted to design a general reasoning test which would fully simulate the analytical, intellectual and commercial demands frequently encountered in a business environment.

A lively discussion in the psychometric literature over the years has been concerned with the competing requirements of validity and fidelity. A valid assessment of general reasoning B whether abstract or specific B is one that provides an accurate assessment of an individual's relative status on the trait. Put succinctly, the test measures what it claims to measure. However, in some contexts validity is achieved at the expense of fidelity, that is, a test fails to appear to be related to the analytical demands of real-life situations. Conversely, it is possible to construct, as we are sometimes asked to do by clients, highly specific exercises using a particular occupational framework, which therefore achieves very high fidelity. However, in these situations organisational knowledge and familiarity with process and market conventions intrude on the assessment. And an individual's performance on the exercise is no longer a straightforward measure of his or her raw reasoning ability.

The goal of psychometric testing, therefore, must be to optimise both validity and fidelity. Our main objective has been to achieve this goal by constructing a simulation of the analytical demands which confront individuals in organisations while adhering closely to the principles of good metrics in psychological assessment. We believe that such a test has the benefit of making the assessment both relevant and defensible.

In summary, the rationale for Business Intelligence was to develop a general reasoning tool which would:

  • Optimise validity and fidelity
  • Assess a spread of occupationally relevant analytical demands
  • Use a variety of real-life formats for presenting problems
  • But yield a score which was internally consistent and thus readily interpretable
  • Be demanding but nonetheless interesting for the test taker
  • Discriminate meaningfully within fairly selected groups


When should I use Business Intelligence?

We consider an assessment tool like Business Intelligence B focussed on the analytical demands of real-life organisations situations B widely applicable, providing individuals and organisations with useful data in a variety of contexts.

Selection - In today's competitive employment market, organisations are seeking candidates with a high level of reasoning skills to fill a wide range of positions. The ability to analyse complex and weighty business information is considered to be a key skill in an increasingly broad range of both management and technical positions and is commonly the target of psychometric assessment during selection in many occupational fields. In short, Business Intelligence can be used as part of the selection process if the job requires individuals to identify discrepancies in quantitative and qualitative information, analyse trends in data, probe variances to infer possible cause and effect and ultimately make decisions on the basis of this analysis. Clearly, Business Intelligence will yield the most relevant information when the analytical demands on the job are high. However, Business Intelligence can also provide a useful estimate of the candidate's commercial intelligence.

If in doubt, job analysis questionnaires can be used to assess the relative importance of reasoning ability. Alternatively, qualitative interviews with jobholders or experts using protocols such as Critical Incidence Technique (see Principles of Organisational Behaviour@, Fincham and Rhodes, Oxford University Press 1999) can be used to assess the potential impact of differences in reasoning ability on job performance.

The question of when to use Business Intelligence to some extent depends on the likely selection ratios i.e. the number of applicants expected to apply for a job. If the selection ratio is high then there is a case for using Business Intelligence as a screening device. Screening means using the test at the start of the selection procedure. Progress onto the next stage B usually an interview B depends on performance on the test. Some bodies dispute the use of psychometric devices such as Business Intelligence as suitable for screening. The British Psychological Society (BPS), for example, argues this potentially puts too much weight on a test score. Furthermore, it has been argued rejecting candidates solely on the basis of psychometric test results can lead people to feel very negative about objective assessment procedures in the future. However, in the real world, screening has to occur if selection ratios are high. And we believe the data generated by Business Intelligence provides information of a much higher status and utility on which to base a decision than many traditionally used screening procedures.

Promotion - Business Intelligence can be particularly helpful when promoting individuals into jobs in which problem solving and analysis becomes a greater component of the new role compared to their existing one. For example, if the new role involves identifying discrepancies in data, analysing trends over a period of time and probing to identify the possible causes and effects of variances between what was expected and what is actually observed within the data.

Redeployment - Jobs can disappear. And if there are no opportunities elsewhere in an organisation an individual might be suited to, Business Intelligence might help in assessing how difficult a transition might be if the move involves moving to a role in which the analytical demands are greater.

Succession and Career Planning - The focus of succession planning is a particular job grade. And Business Intelligence can be used here in order to establish which individuals can potentially move into this job grade now or who might need to develop their ability for the future.

Reliability and Validity

How good is Business Intelligence? - Any psychometric test, and indeed any other assessment method, can be judged according to two related criteria: reliability and validity. To be useful a test has to achieve acceptable levels of both.

Reliability - Although there are about eight or nine different types of reliability, they are all essentially about the extent to which the test score represents a good estimate of an individual's ability. Reliability essentially addresses the issue of accuracy. Tests are not perfect. The technology is more similar to opinion polling than to, say, measuring someone's height. A test is essentially a behaviour sample. Business Intelligence samples an individual's ability to demonstrate general reasoning ability thirty eight times. But this sample contains error. In the same way that an opinion poll can be undermined by asking a question in the wrong way, some of the error comes from the questions themselves. Some of the error can also come from the administrator. Some test administrators are better than others at on the one hand handling any anxieties an individual might have about the test, and on the other hand motivating and interesting people. In addition, test-takers themselves may not be functioning fully. A bad night's sleep, some unlucky guesses, hay fever B a whole list of things might mean the test-taker is not able to give the test his or her best shot.

Because Business Intelligence is a sample, and as we have seen contains error, we need to treat the data it generates as probabilistic B providing an estimate of general reasoning ability within certain confidence limits rather than data which is set in stone.

There are a number of specific ways of quantifying the degree of error present in a test score. All of these use a statistical technique B correlation B to estimate the impact of error. Correlation is a statistical method of demonstrating the extent to which one thing goes with another. If two things or variables are perfectly related B a change in one variable produces a directly proportionate change in the second variable, the correlation coefficient between these two variables would be 1.0. It is possible to have a negative relationship between two variables, i.e. an increase in one variable produces a directly proportional decrease in the second variable. Where there is no relationship between two variables, the coefficient will be 0.

We use correlation to identify the extent to which test questions are related to one another. If responses to questions are affected by a number of factors in addition to the attribute being samples then the correlation between questions will be reduced. The more the responses to questions are determined by one attribute the higher the correlation will be. Obviously, given that psychometric tests are not perfect, their reliability coefficient will always be less than 1.0. But whilst a test can never be perfect, a good test should always have a reliability coefficient of 0.7 or more.

With a psychometric test it is possible to create very high internal consistencies by asking essentially the same question. And some personality tests have scales that essentially do this. But this level of internal consistency can only be achieved at the expense of measuring more broadly based psychological attributes. With Business Intelligence we were concerned to include a diversity of items which covered a range of analytical skills. But whilst on the surface, content and method of deduction vary; the hope was to provide items essentially measuring the same attribute.

The trial version of Business Intelligence contained 125 items and was piloted on a group of volunteers (N= 135) who met the criteria of the group for whom the test was intended ie those who had completed high school education, had reasonable levels of attainment and were likely to go on into junior management positions. A second item trialing on individuals applying for junior management positions also enabled the internal consistency of the test to be estimated.

In the first sample the internal consistency of the test was estimated at .88. In the second sample the internal consistency of the test was estimated at .82.

Validity

Validity addresses the issue of relevance. The current definition of validity embraces any evidence which helps in the interpretation of a test score and in its relevance. The simplest form of validity is face validity. Assessing validity is essentially a question of subjective judgement. Does the test seem to be measuring a relevant attribute? Since Business intelligence was devised to overcome issue of low face validity with tests of general intelligence currently available it should possess this form of validity. The questions clearly involve complex forms of analysis and judgement. The questions are also located in the world of work. Possessing face validity is important. Candidates respond better to test which possess facet validity-it increases the perception of relevance.

Content validity - a more complex form of validity, is about the extent to which the test samples a relevant domain. In constructing business intelligence we identified three domains- identifying, probing and judging. The first involved being able to identify discrepancies between observed and predicted. The second related to the ability to explore this variance- to unravel cause and effect. The final domain involved making judgements about the best response to the variance. Items in the test closely map onto this model of business thinking.

Construct validity - perhaps the most complex form of validity - involves collecting data which helps refine our understanding of what a test actually measures. This can involve testing specific hypotheses. And to test the hypothesis business intelligence measures general intelligence a ample of individuals (N= 71) was administered Business intelligence and the DAT abstract reasoning test. The uncorrected correlation was .65 (.74 when corrected for restriction of range). This supports the hypothesis that Business Intelligence measures general reasoning. Other test administered to the same group were the language usage test of the DAT where the correlation did not differ significantly from zero. Since Language usage essentially measures attainment this suggest Business Intelligence is not impacted by educational differences. The DAT Verbal Reasoning test had a relationship of .4 below the point where they are essentially measuring the same thing. This level of correlation is presumably the result of the VR test having an element of general intelligence in it.

Criterion validity - is what most individuals would intuitively regard validity as being about. This is the ability of the test to predict real world outcomes. Well constructed ability tests should have validities in the .4 -.5 Region. And in a study of merchandisers in a major high street retailer scores on the Business Intelligence correlated .42 with supervisory ratings of performance (N= 29). Although based on relatively few people this is encouraging- particularly as this was a selected group already functioning in the role.

Business Intelligence Scoring

Norms - When interpreting scores on Business Intelligence it is important to use a normative group that closely resembles the assessment context and occupational group for which the test is being used. Comparison with an inappropriate norm group may give a distorted picture of candidate's true level of general reasoning.

The norms used for businesses intelligence are based on a large sample of candidates applying for junior management positions with a large retail outlet in the UK. Because the norm group consists of a considerably large sample of individuals, the norms used can be considered a widely applicable standard reference point for any organisation. However, users of Business Intelligence who carry out large numbers of administrations may prefer to develop their own in-house or local norms in order to provide a more suitable reference point for their own organisation. In-house norms are useful because they reflect the specific circumstances of individual organisations and the particular characteristics of people within different occupational groups. In a selection context, in-house data can also be used to provide an indication of the range of raw scores

Interpreting Business Intelligence Reports

The Tests Direct system will automatically generate both a graphic representation of the candidate's performance as well as a narrative report.

Although statistics provide test-users with quantitative information about a candidate's performance on Business Intelligence, it is often necessary to examine the narrative interpretation of a candidate's score as this will provide a more detailed explanation of the candidate' performance. The system will also generate some feedback for the. So what does a score on Business Intelligence really mean? When reading a Business Intelligence score and report it is important to remember the following points:

  • Business Intelligence is designed to measure the ability to reason in a business or commercial occupational setting rather than demonstrating mental and intellectual agility alone.
  • Emphasis is placed on problem-solving strategies and critical analysis based on the information provided.
  • Candidates may have 'pockets' of ability which are masked by their overall score on the test. For example, a candidate may have managed to spot all of the discrepancies present in an exercise but struggled to perform a strong enough analysis to probe deeply enough to identify the underlying reason for the discrepancy. This is especially important in a development context where more detailed examination of candidates= performance on the test will often focus training on specific areas of analytical ability.


for Test Administration instructions click here