Examiner Express

Multiple-choice test items are without a doubt the most popular test item.  While it is very easy to write a bad item, it’s a lot more difficult to write a good one.  Coming up with the correct alternative is usually pretty easy.  Coming up with decent distractors is a lot more difficult.  First off, let’s look at what a multiple-choice item is supposed to do.

“It’s supposed to show me what the examinee knows, right?” I hear you say.  True enough.  But if that’s all your item is doing you are losing out on the power of all your distractors.

“I just make up something confusing, add ‘None of the above’, and call it done”.  As my step-son says on occasion “I’m going to have to hurt you now…”

Never, ever, ever use “None of the above” and “All of the above”.  They tell you zip about what your examinee knows.  All they do is give you a lazy way out of of having to think about the item.

An item should not only show you what your examinee knows, it should show you what they don’t know.  I’d actually argue that knowing that is in many ways more important.

So, how do you make a good distractor?  There are a couple of good rules that I learned from my old buddy Stan Trollip.  (He’s a mystery writer now.):

• all your alternatives should be about the same length
• make sure your alternatives grammatically match your stem
• never use “give-away” alternatives just to fill out the item
• make your alternatives logically possible

What about that last one?  “Logically possible”?  The way I read that is that the alternative would look plausible to a person that didn’t know the correct answer, but obviously wrong to one who did.   It’s probably the most important guideline, and one of the hardest to meet.  There is, however, an easy way to create those “logical” distractors: have your examinees write them for you!

Here’s what you do:

1. Create an open-ended question that is pretty much identical to the stem of the item you want to use.
2. Put it in your assessment.
3. Gather and rank by frequency all the responses your examinees wrote in.
4. Take the top 4 (for a 5-alternative multiple-choice item) wrong responses and make them your distractors.

You can, of course, create an alternative by synthesizing something from a couple of examinee responses.  In any case, you will end up with something that represents common misconceptions about whatever it is you are asking about.

We came up with this idea back in the late 70’s on the old PLATO system (See: PLATO on Wikipedia and Cyber1).  It worked like a champ back then, and it’ll work now for you.