Creative Education
2013. Vol.4, No.6A, 39-41
Published Online June 2013 in SciRes (
Copyright © 2013 SciRes. 39
Towards More Efficient Assessments: Increasing Information
from Objective Examinations*
Alan E. Dugdale
Department of Paediatrics and Child Health, University of Queensland, St. Lucia, Australia
Email: dugdalea@gm ai l .com
Received April 12th, 2013; revis e d M a y 14th, 2013; accepted May 21st, 2013
Copyright © 2013 Alan E. Dugdale. This is an open access article distributed under the Creative Commons At-
tribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the
original work is properly cited.
Objective examination questions are widely used to assess students’ knowledge, but the standard MCQ
with a stem followed by 4 - 6 possible answers one of which is chosen as correct is very inefficient. Sim-
ple changes to the format can treble the information gained. Information theory is the tool for assessing
the information content of electronic and other communications. The “bit” is the unit of information and
equals on true/false choice. I have applied basic information theory to objective questions. The standard
MCQ with a stem, a choice of 4 possible answers (mark one answer true) and an expected correct answer
rate of 70% yields 1.36 bits. A MTFQ with 4 choices where the student must answer true/false to every
possible answer gives 3.52 bits of information. By adding a “don’t know” option the same MTFQ gives
4.72 bits of information, 350% of the standard MCQ. Thirty MTFQ with don’t know give the same in-
formation about students’ knowledge as 100 standard MCQs. The effort needed to set, sit and mark is the
same for both. Small changes to the format of objective questions give large gains in efficiency. We
should balance these gains against possible disadvantages.
Keywords: Objective Assessment; Information
Education is the transfer of relevant information from teacher
to student. In medical education, information is broadly classi-
fied into knowledge, skills and attitudes (KSA). Each needs
different pathways; in this paper I shall consider only knowl-
edge. In any transfer of information, there is a source (teacher,
book etc.), the channel (speech, vision, electronic) and the re-
cipient (student). Perfect transfer of information is not possible
in theory or in practice, so all channels transmitting important
information have feedback from recipient to source to check
levels of completeness and accuracy of the data transmitted. In
face-to-face teaching, we have immediate feedback; we can see
signs of comprehension, confusion or boredom in our students.
In other forms of learning, we lack such feedback, so we need
formal methods to ensure that the student has received the in-
formation and has incorporated it into his/her knowledge base.
In education we call this “assessment” and it can be done at the
end of each session or at a later date. In a crowded curriculum
where a few teachers are trying to transmit large amounts of
information to many students, we should have channels that
transmit information efficiently so that the times spent by
teachers and students are minimised. This applies equally to the
reverse process of feedback to check that the information has
been transmitted, received and absorbed without major errors or
omissions. Such feedback is vital for educational and regulatory
reasons, but it does not add to the student’s knowledge and is
therefore often seen as a burden to be minimized. There is
every incentive to make the feedback process as quick and
painless as possible compatible with the needs and goals of
such assessments.
The information we transmit to students is diverse, multi-
layered and complex, but we have largely simplified the feed-
back from the students by using “objective questions” with a
narrow range of formats and responses. Unlike to complex
information transferred from teacher to student, this simplified
feedback can be tested for accuracy and completeness. Basic
information theory shows great differences in the information
content of various forms of objective question. Minor changes
to the style of question can yield information up to 350% of the
standard MCQ.
Information Theory—A Very Brief Overview
Advanced information theory is complex and highly mathe-
matical, but the basic ideas are simple and need only high
school algebra. The basis was published in a trade journal by
Shannon (Shannon, 1948). . Many of the results derived from
information theory are numerical results for ideas that are intui-
tively obvious. Information is measured in bits, one bit is the
amount of information gained by choosing one of two equally
likely alternatives such as Yes/No or True/False. If there are
three equally likely choices, such as True/False/Don’t Know
then we can see that gain more information and this is con-
firmed by information theory. The basic equation is (Moulton,
Bits of informationplog1p
*Competing interests: The author has no competing int erests.
where p = probability of each event and all the p values add up
to 1. log2(1/p) is the logarithm to the base 2 of 1/p.
Most assessments based on objective questions are designed
so that the average student will score about 70% correct an-
swers. This built-in bias lessens the amount of information we
gain. An extreme example of this effect is to start a question-
naire by asking a random sample of people “Are you male or
female”. If the subjects were random users of a shopping mall,
then the response would help in the analysis of later answers so
provides useful information. However if we took that same
questionnaire to a nunnery, it would be no surprise that 100%
answered “Female”. We knew this in advance, so gained no
new information from the question. The formula above gives us
a numerical measure of the information gained, allowing for
prior knowledge.
I shall use this basic formula to calculate the information in
the different types of objective question used in assessment of
medical education. To make the different types comparable, I
shall give each type a choice of four items, but this is not nec-
essary or even desirable.
Information Content of Different Types of
Objective Questions
Information in standard MCQ style question:
1) MCQ Type A: The standard MCQ style question
In a healthy person the common colour of urine is (mark
one box)
1 yellow [ ]
2 blue [ ]
3 red [ ]
4 brown [ ]
(the correct answer is yellow)
The candidate must pick one out of the four alternatives. If
the question is designed so that the candidate is equally likely
to choose any option, then the information content of this deci-
sion is 2 bits. In practice, the candidate knows that one option is
correct and the others wrong. From results of past examinations
the examiners know that about 70% of students will pick the
correct option and the others will be evenly divided among the
other three options. The student can make only one response
and the likelihoods have been knowingly weighted by the ex-
aminer. Using the formula above, we can show that the infor-
mation content of the student’s response is 1.36 bits. We have
effectively limited the candidate’s options, so we learn less
about his/her knowledge
2) MTFQ Type A: Multiple true/false question
The student could be asked this question in a different way
In a person with organic disease the urine may be (mark
one box in each line)
1 yellow true[ ] false[ ]
2 blue true[ ] false[ ]
3 red true[ ] false[ ]
4 brown true[ ] false[ ]
(the correct answers are yellow, red, brown)
In this form of objective question, the wording of the stem is
slightly different to allow multiple true responses. The answer
to each line must be independent of other lines. The work in-
volved for the examiner to set the question is almost the same
as MCQ Type A, but the student must answer four independent
true/false items. If the student were equally likely to answer
true or false to each line, than each response would contain 1
bit of information and the total information gained would be 4
bits. However, the student's knowledge of the subject makes the
correct answer more likely, say he has a 70% chance of giving
the correct answer. This limits his choices so the information
content of each answer drops from 1 to 0.88. The student an-
swers four independent questions, so the total information is
3.52 bits. By changing the format of the question so as to de-
mand an answer to each alternative, we have gained 3.52 bits of
information compared with 1.36 for the standard MCQ, more
than double the information (260%).
3) MTFQ with DK alternative
We can include a formal don’t know option in the MTFQ
In a person with organic disease the urine may be (mark
one box in each line)
1 yellow true[ ] false[ ] don’t know[ ]
2 blue true[ ] false[ ] don’t know[ ]
3 red true[ ] false[ ] don’t know[ ]
4 brown true[ ] false[ ] don’t know[ ]
(the correct answers are yellow, red, brown)
As in the previous format, each line tests a separate item of
knowledge, but there are now three alternatives. The don’t
know option gives the student a wider choice. Let us assume
that, on the average, students mark the correct box 70% of the
time, the wrong box 15% of the time and the don’t know box
15% of the time. The answer to each line will give 1.16 bits of
information, giving a total gain of 4.72 bits of information.
Quite apart from any philosophical value of giving the student a
“Don’t know” alternative, we have increased the information
gained about the student’s knowledge to 4.72 bits, compared
with 1.36 bits for the standard MCQ, a factor of 347% in the
return for the same time and effort by the examiner.
Sensitivity Analysis
In the descriptions so far, I have assumed that there is prior
knowledge that in standard MCQs the student will choose the
correct response rate of 70%, with other responses distributed
equally among the false options. Papers could be set with dif-
ferent expectations of student responses. The main changes in
information yielded come from the format of the question and
the expected number of correct responses. These are shown in
Table 1 where the number of expected correct responses is
given and the remainder of the answers evenly divided among
other options.
In all types, the greatest information yield is when the prior
expectation is for 50% correct responses. The information
gained decreases in all types of objective questions as the ex-
pected level of correct responses rises, but the relative advan-
tage of the MTFQ and MTFQ + DK increases. Under all levels
of expected correct answers, the MTFQ yields at least twice as
much information as the standard MCQ and the MTFQ+DK
has at least three times the yield.
Discussion and Conclusion
Objective questions have many advantages and are widely
used for feedback to teacherssessment of student and as
Copyright © 2013 SciRes.
Copyright © 2013 SciRes. 41
Table 1.
The information yield for various types of objective que s tio ns s howing the effect of varying the expected level of correct responses.
% Correct Responses Expected 50 60 70 80 90
MCQ Bits 2.00 1.60 1.36 1.04 0.63
MTFQ Bits 4.00 3.88 3.52 2.88 1.82
MTFQ + Don’t
Know Bits 6.00 5.48 4.72 3.69 2.28
knowledge for regulatory and licensing bodies. Knowledge is
reduced to fragments and selected so that a True/False or Yes/
No answer is possible and meaningful. This allows the mathe-
matical measurement of the information content. This paper is
about the information contained in each question, but not with
the method of scoring the student’s responses nor the subject
matter tested, although I note that such testing is limited to
material which is definitely True or False. This excludes newer
material which may be important but is not yet fully established
and a wide range of material where an answer will depend on
surrounding circumstances.
Replacing the standard MCQ with the MTFQ involves no
philosophical or educational change or innovation. It is a simple
change of format, but one which doubles the information gained
at no cost in time for examiner, the system or the students. The
use of the MTFQ + DK involves an extension of the relevant
definition of knowledge. Admitted ignorance is a recognised
and valid state of knowledge, By omitting that formal option,
we force the student to guess, which is not only logically un-
sound, it is undesirable and probably dangerous in later profes-
sional life. Guessing also complicates the scoring system of the
examination (Harden et al., 1976; Ben-Simon et al., 1997). We
all have areas of ignorance, it is a sign of wisdom to acknowl-
edge ignorance and that should be encouraged by an appropri-
ate marking scheme. A scoring that allots marks to Correct >
Don’t Know > Wrong is logically sound and will discourage
The standard MCQ format has lasted well. Many pre-tested
and validated question banks are available. With no change in
the philosophy of objective questions, but only a change from
the MCQ format to the MTFQ format, we can increase the in-
formation content by a factor of two or more with no extra
work for the examiner, thus gaining necessary data about the
student more quickly, reducing the number of questions needed,
and so easing the load for both examiners and students. If we
acknowledge that admitted ignorance is a valid form of knowl-
edge, then the MTFQ + DK will further increase the efficiency
and information yielded by objective questions. The number of
questions still needed will ensure that the assessment covers a
wide range of the curriculum. Examiners must consider factors
other than information content, but the efficiency, ease and
reliability of collecting data about the students are important
factors when preparing assessments.
Author’s Contributions
The author has generated the hypotheses, written the needed
computer program, done all calculations and written the paper.
Author’s Background
Alan Dugdale MBChB MD FRACP is a retired academic
paediatrician who has been a teacher, clinician and examiner.
He now teaches in an honorary capacity.
Acknowledgements and Funding
The author received no funding from any organisation but
had access to the libraries of the University of Queensland,
Ben-Simon, A., Budescu, D. V., & Nevo, B. (1997). A comparative
study of measures of partial knowledge in multiple-choice questions.
Applied Psychological Me a s ur e m en t , 21, 65-88.
Harden, R. M. G., Brown, R. A., Biran, L. A., Dallas Ross, W. P., &
Wakeford, R. E. (1976). Multiple choice questions, to guess or not to
guess. Medical Education, 10, 27-32.
Moulton, R. (2010). A short simple introduction to information theory.
Shannon, C. E. (1948). A mathematical theory of communication. The
Bell System Technical Journal, 27, 379-423, 623-656.