Examiner Express

I may have mentioned it before, but a modification of an old adage applies quite well to developing assessments:

No assessment ever survived first contact with an examinee.

You may think you’ve tested things out, but until you complete two very important steps you’re leaving yourself open to all sorts of problems.  Here’s the first thing you need to do:

Verify Your Key

Take your answer key and answer the test with 100%, 50%, and 25% of the answers marked in correctly (no, I didn’t say “Incorrectly”).  Score the test.  Look at your test results and make sure that the records system you’re using shows 100%, 50%, and 25%.  If it doesn’t you need to check your key because something is obviously amiss.

Most folks do this, fix up the problems they’ve seen, and call it a day.  This is not a good idea.  Not doing this step can even cost you big bucks.

Verify Your Assessment

This is different from validating your key.  All that does is take what you think is the correct answer and make sure that the assessment is being scored to match that correct answer.  This ignores one very important issue: what if what you have marked as the correct answer isn’t the correct answer!

Here’s what to do: give a copy of the assessment to someone who should know all the answers.  Give a 4th grade arithmetic standards assessment to a 7th grade teacher.  Score the assessment and look at the results.  Unless the 7th grade teacher is a complete dolt, that assessment better come back with a score of 100%  If it doesn’t you had best go back and figure out what went wrong.  Either the teacher just made a mistake and marked something wrong or, and more critical, you don’t have the correct answer marked.

OK, I hear you say…  I know I made a decent assessment…why do I have to go through that step?  Let me tell you a little story about a company that paid Big Time for not doing that final, simple, step.

First off, check out this article in the NY Times:   http://www.nytimes.com/2006/03/10/education/10sat.html

The company doing the assessment didn’t do that last step and it cost them a lot.  They could have spent two or three hours running a knowledgeable person through the assessment to make sure the answers were OK, but they didn’t.  And, as you can see in the article, they paid the price.

So, bottom line, make sure your answer key is ok but also make sure that the assessment that cranked out that answer key is ok too.  Failing to do that one simple step can come back to bite you!