Critical Reasoning Test
for Test Administration instructions click here
Who should use this test?
A higher-level test, suitable for graduates and senior managers. As critical reasoning is a developable ability it is also of particular utility in training and development programs for high potential performers.
Psychological Rationale
As there are already several critical reasoning tests on the market why would we want to write another one? Firstly, the ability to think critically is not a single ability. The intention of authors of critical thinking tests is to capture a range of skills which affect an individual's ability to deal with information dispassionately.
The critical reasoning construct originally operationalized by Watson and Glaser in 1937 included a mixture of attitudes, knowledge and skill.
- Attitudes of enquiry that involve an ability to recognize the existence of problems and an acceptance of the general need for evidence in support for what is asserted to be true.
- Knowledge of the nature of valid inferences, abstractions and generalizations in which the weight of accuracy of different kinds of evidence is determined.
- Skills in employing and applying the attitudes and knowledge outlined above.
The individual who scores highly on critical reasoning tests is therefore capable of checking the coherence, strength and validity of an argument. To do this he or she is capable of suspending emotions, holding the information at arms length, looking 'through' the content of the material and examining the structure of the argument.
The skills and abilities sampled in critical thinking tests simulate situations where managers are presented with reports prepared by people who may know more about the subject matter than they do. When this occurs, to be effective, the manager has to fall back not on his or her understanding of the subject matter, but on the ability to examine the structure and coherence of the argument by asking questions such as:
- Do the conclusions follow from the evidence?
- Is the argument internally consistent?
- Are there hidden assumptions?
- Are the inferences valid?
- In short is this a good or bad argument?
High scorers are assets to organisations since they are capable of contributing effectively in management teams. Their critical ability cuts across management specialisations and means they can sharpen the arguments of colleagues and eliminate inferior contributions.
Studies demonstrate the discreteness of critical thinking. This has important implications in personnel. We have often found line managers perform extremely well on traditional verbal reasoning tests but fall down on critical reasoning tests. The implication is that they can absorb verbal material quickly but can do very little with it. This indicates a development need. Typically, this occurs where line managers have received little beyond compulsory education; critical reasoning, as a composite of attitude and skills, is normally acquired through higher education. At the most general level, inculcating critical reasoning is the objective of higher education. The subject matter may be forgotten but the analytic, critical, dispassionate approach to evidence is assumed to be more enduring.
The Critical Reasoning Test is an assessment of what is termed critical reasoning and is based on the original concept of critical reasoning comprising five areas:
- Inferences - the ability to evaluate the validity of inferences drawn from factual statements
- Deduction - the ability to determine whether certain conclusions necessarily follow from the information in given statements or premises
- Assumptions - the ability to recognise understated assumptions or presuppositions in a series of assertive statements
- Interpretation - the ability to weigh up evidence and determine whether conclusions or generalisations based on the given data are warranted
- Evaluation of arguments - distinguishing between arguments that are strong and relevant, or weak and irrelevant, to a particular question at issue.
However the Tests Direct Critical Reasoning Test represents a radical departure from the original model. In standard assessments of critical reasoning these facets are measured separately. Our view is, in organisational life, these abilities need to be demonstrated on the same information piece. So candidates are presented with two complex scenarios and assessed on their ability to demonstrate the above facets of critical reasoning. Thus, we have justified the construction of our test: at last we have a test which combines all the popular notions of what is critical reasoning in one neat online assessment.
When should I use the test?
The critical reasoning test can be used in a range of settings, for example, selection, development, succession planning, career planning etc.
Selection
The critical reasoning test is best applied to candidates who are applying for management positions junior, middle and senior management. Additionally, graduates who are applying for positions into management programs could be usefully given the test.
Candidates performance will give employers an indication of what level of critical reasoning they currently have, and whether their skills require further development.
Development
Critical reasoning tests can be particularly helpful when identifying management potential. Managers operate autonomously, for the most part, and they should have the intellectual and practical skills required to function as effective managers. Often it is assumed managers are born not made and academic intelligence is often associated with managerial ability. However, whilst it may be argued that individuals who demonstrate good academic ability will be able to meet the intellectual demands of managerial positions, it does not necessarily follow that they will have managerial skills such as diagnostic information seeking, planning/causal thinking and conceptualization/synthetic thinking (Klemp & McLelland 1986).
When organisations are considering promoting or developing staff into management positions they will rely on historical data to identify those potential candidates. By applying a test such as critical reasoning they will establish any areas of skill which have potential for development. Such development may be achieved by a programme such as DMS, MBA, CIPD qualification, discrete critical reasoning courses such as those held at Roffey Park, and the use of texts such as Critical Reasoning a practical introduction Anne Thomson.
About the Critical Reasoning Test
The test is in two parts, both of which are completed consecutively without interruption or prompting. Candidates are presented with two scenarios and 45 questions for each.
Five discrete critical reasoning dimensions are measured in the test Inference; Assumption; Deduction; Interpretation and Evaluation. This assesses how well candidates cope with situations which require different information processing methods and highlights particular areas of strength and weakness.
Both parts of the test require candidates to assimilate the information in the scenario and answer the questions which follow. They are also required to identify where there is insufficient information to reach a definitive answer.
The test requires no expert knowledge or previous experience.
Technical Features of The Critical Reasoning Test
Reliability - The question of reliability addresses the issue of accuracy. And reliability is characteristically assessed in two ways in psychometric tests: internal consistency and test retest reliability. The former assesses the extent to which each question essentially measures the same attribute. The assumption is each question is comprised of core and error. Core represents the extent to which the question is genuinely sampling the target attribute. Error represents the extent to which a response might be produced for reasons other than the individual's standing on the attribute. For example, in personality assessment in measuring an individuals extraversion we might ask the question 'Would you rather go to a party than read a book?' To some extent the response will be determined by the amount of stimulation in the environment the individual is seeking. However, extraneous issues may also come into play and which might determine the response to such a question. For example, living in an isolated rural area may mean few, if any, parties occur and so the party going habit may well have eroded. Although, of course, it might be argued a question such as this is less prone to this kind of error since extraverts are less likely to live in socially secluded areas. But error is nonetheless inevitable. And the question of internal consistency provides an indication of how much the questionnaire incorporates this kind of error.
Test-retest reliability looks at the extent to which a set of scores is likely to be repeated over time. This kind of reliability then incorporates a somewhat broader range of errors. For example, the effects of differences in administration, fluctuations in mood and motivation and, in the case of personality measures, the extent to which the trait itself is stable.
Internal consistency is assessed by estimating the extent to which each question is correlated with every other question. The internal consistency coefficient is therefore the average correlation between questions. If the coefficient is 1 we are in effect asking the same question over and over again. If the coefficient is 0 the questions have no relationship with one another. To estimate the test-retest reliability it is necessary to bring a group back, retest them and establish the relationship between scores at time one and scores at time two.
Since the main concern with the Critical Reasoning Test is establishing an individual's score at one point in time the internal consistency of the scales rather than test-retest has been calculated. The benchmark figure is .7. This is not an arbitrary figure. At this point the amount of error associated with the test is reasonable. Although the .7 criterion is not to be interpreted in an overly rigid and mechanistic way. Studies of internal consistency also of course only provide estimates and these may vary somewhat between samples. In a very homogenous group of individuals the internal consistency of the scale may be somewhat lower than in a sample of individuals to have much more varied personalities.
In ability testing, internal consistencies tend to be very impressive. This is because the tests are measuring one attribute usually and can take up to an hour to complete. And the Critical Reasoning Test is no exception. With just a small sample of candidates we were able to obtain an internal consistency of .73. So in the Critical Reasoning Test we have a reliable test.
Validity - If reliability addresses the accuracy question, validity addresses the question of relevance. And there are many ways of answering the rather more elusive issue of the extent to which the test is relevant in a particular context. For example, one very simple way of answering the validity question - known as face validity - is simply to look to see whether the questions appear relevant.
The Critical Reasoning Test easily passes the first test. The scenarios clearly require individuals to utilise their critical thinking abilities. In fact, the beauty of this critical reasoning test is that it clearly requires individuals to invoke several abilities (induction, deduction, assumptions, interpretation and evaluation of arguments) which, when taken as a whole, form the construct known as critical reasoning.
And this brings us to another type of validity - construct validity. Data about construct validity is essentially any data which helps refine our understanding of what the test actually measures. Happily, it is clear what the critical reasoning test is measuring. Critical reasoning is an extremely well established construct. As has already been mentioned in this manual, the first critical reasoning test was produced in 1937. And this test had excellent construct validity even at its inception. For example, one of the components of the construct of critical reasoning is that it is assumed to be a developable ability, often through higher education. Fogg and Calia (1967) noted that critical reasoning improved in liberal arts students over a period of two years. Many other studies demonstrate the clear existence of critical reasoning as a valid construct.
Content Validity. This is evidence that a test has the correct content - that it samples the relevant range of knowledge, ability or preferences (if an interest or personality test). For example, in a clerical test do the tasks such as checking copy, error location, spelling, language usage, mental arithmetic and problem-solving, represent the range of skills and abilities required to perform effectively in a modern clerical environment? The critical reasoning test has good content validity, as it is comprised of two scenarios, each with 45 questions that force individuals to draw upon a range of skills, all of which are components of critical reasoning. Thus, in the Critical Reasoning Test we have a test where the content provides us with a comprehensive coverage of the facets of critical reasoning.
Adverse Impact - The question of validity is, as suggested above, essentially the question of relevance. However, consideration of validity also includes the issue of fairness - does the test predict an outcome fairly or in a way which is systematically related to an individual status in terms of his or her age, gender or ethnicity? So, the issue of validity includes consideration of to what extent does the use of a test involve adverse impact - a group having a systematic disadvantage?
One easy way of assessing this is to explore test data in order to establish whether any significant differences in mean scores exist. Similarly, do validity coefficients appear to be different for groups? In other words, does the test tend to under predict an outcome for one group when compared to another?
As critical reasoning is a higher-level ability it is not thought to discriminate between ethnic groups. This is because it is assumed that if one has reached a sufficient level to be taking a critical reasoning test then one must have a good command of the English language. With regard to gender differences, the table below demonstrates that there is no significant difference in performance on critical reasoning tests for males and females.
|
Percentile
|
Raw Score for Females
|
Raw Score for Males |
|
99
|
73-80
|
75-80
|
|
75
|
67
|
69
|
|
25
|
59-60
|
61
|
|
1
|
0-50
|
0-38
|
(Percentile ranks are a sample taken from Hoogstraten and Christiaans 1975)
Norms
Preset norms are utilized within the Test Direct generic system. However, in-house benchmarks can be easily set up if it is anticipated usage will extend into three figures.
Getting the Most from Critical Reasoning Reports
One of the main concerns about testing via the Internet is security. Who is actually completing the questionnaire? In a way, these questions are based on a faulty assumption about the nature of the data. Even if administered in a traditional, exam room setting, the data may still contain numerous sources of error. For example from the test administrator, or the environment in which the test was taken (noise, heat etc.). So, even if collected in the traditional way ability data does not provide a 100% reliable picture of an individuals ability. And, anyone assuming it does has misunderstood the nature of the evidence. By testing online we are able to achieve greater standardization of test administration than ever before as we now have a way of ensuring that all individuals receive the standardized instructions and are timed correctly.
for Test Administration instructions click here
|
