A. E. DUGDALE

where p = probability of each event and all the p values add up

to 1. log2(1/p) is the logarithm to the base 2 of 1/p.

Most assessments based on objective questions are designed

so that the average student will score about 70% correct an-

swers. This built-in bias lessens the amount of information we

gain. An extreme example of this effect is to start a question-

naire by asking a random sample of people “Are you male or

female”. If the subjects were random users of a shopping mall,

then the response would help in the analysis of later answers so

provides useful information. However if we took that same

questionnaire to a nunnery, it would be no surprise that 100%

answered “Female”. We knew this in advance, so gained no

new information from the question. The formula above gives us

a numerical measure of the information gained, allowing for

prior knowledge.

I shall use this basic formula to calculate the information in

the different types of objective question used in assessment of

medical education. To make the different types comparable, I

shall give each type a choice of four items, but this is not nec-

essary or even desirable.

Information Content of Different Types of

Objective Questions

Information in standard MCQ style question:

1) MCQ Type A: The standard MCQ style question

In a healthy person the common colour of urine is (mark

one box)

1 yellow [ ]

2 blue [ ]

3 red [ ]

4 brown [ ]

(the correct answer is yellow)

The candidate must pick one out of the four alternatives. If

the question is designed so that the candidate is equally likely

to choose any option, then the information content of this deci-

sion is 2 bits. In practice, the candidate knows that one option is

correct and the others wrong. From results of past examinations

the examiners know that about 70% of students will pick the

correct option and the others will be evenly divided among the

other three options. The student can make only one response

and the likelihoods have been knowingly weighted by the ex-

aminer. Using the formula above, we can show that the infor-

mation content of the student’s response is 1.36 bits. We have

effectively limited the candidate’s options, so we learn less

about his/her knowledge

2) MTFQ Type A: Multiple true/false question

The student could be asked this question in a different way

In a person with organic disease the urine may be (mark

one box in each line)

1 yellow true[ ] false[ ]

2 blue true[ ] false[ ]

3 red true[ ] false[ ]

4 brown true[ ] false[ ]

(the correct answers are yellow, red, brown)

In this form of objective question, the wording of the stem is

slightly different to allow multiple true responses. The answer

to each line must be independent of other lines. The work in-

volved for the examiner to set the question is almost the same

as MCQ Type A, but the student must answer four independent

true/false items. If the student were equally likely to answer

true or false to each line, than each response would contain 1

bit of information and the total information gained would be 4

bits. However, the student's knowledge of the subject makes the

correct answer more likely, say he has a 70% chance of giving

the correct answer. This limits his choices so the information

content of each answer drops from 1 to 0.88. The student an-

swers four independent questions, so the total information is

3.52 bits. By changing the format of the question so as to de-

mand an answer to each alternative, we have gained 3.52 bits of

information compared with 1.36 for the standard MCQ, more

than double the information (260%).

3) MTFQ with DK alternative

We can include a formal don’t know option in the MTFQ

In a person with organic disease the urine may be (mark

one box in each line)

1 yellow true[ ] false[ ] don’t know[ ]

2 blue true[ ] false[ ] don’t know[ ]

3 red true[ ] false[ ] don’t know[ ]

4 brown true[ ] false[ ] don’t know[ ]

(the correct answers are yellow, red, brown)

As in the previous format, each line tests a separate item of

knowledge, but there are now three alternatives. The don’t

know option gives the student a wider choice. Let us assume

that, on the average, students mark the correct box 70% of the

time, the wrong box 15% of the time and the don’t know box

15% of the time. The answer to each line will give 1.16 bits of

information, giving a total gain of 4.72 bits of information.

Quite apart from any philosophical value of giving the student a

“Don’t know” alternative, we have increased the information

gained about the student’s knowledge to 4.72 bits, compared

with 1.36 bits for the standard MCQ, a factor of 347% in the

return for the same time and effort by the examiner.

Sensitivity Analysis

In the descriptions so far, I have assumed that there is prior

knowledge that in standard MCQs the student will choose the

correct response rate of 70%, with other responses distributed

equally among the false options. Papers could be set with dif-

ferent expectations of student responses. The main changes in

information yielded come from the format of the question and

the expected number of correct responses. These are shown in

Table 1 where the number of expected correct responses is

given and the remainder of the answers evenly divided among

other options.

In all types, the greatest information yield is when the prior

expectation is for 50% correct responses. The information

gained decreases in all types of objective questions as the ex-

pected level of correct responses rises, but the relative advan-

tage of the MTFQ and MTFQ + DK increases. Under all levels

of expected correct answers, the MTFQ yields at least twice as

much information as the standard MCQ and the MTFQ+DK

has at least three times the yield.

Discussion and Conclusion

Objective questions have many advantages and are widely

used for feedback to teacherssessment of student and as

Copyright © 2013 SciRes.

40