Reverse Engineering and the Test Question

Thank you for joining me once more as we discuss the PANCE—the test that all PAs must take. As I explain in my PANCE Masterclass, I’ve made it one of my personal missions to ensure that students have the tools they need to prepare successfully and pass on their first attempt. 

Last week, I introduced two essential strategies for reducing the relatively strong sway of test anxiety. The first was familiarization, using practice tests (and not just one, either, but several) and practicing error analysis, which allows students to deduce what questions are tripping them up and why.

Now, I’d like to discuss my second strategy, which is comprehension of the stem questions and their possible answers. This goes beyond simple reading comprehension. In a testing environment, test-takers need to take in a great deal of information quickly while picking out what is essential for the answer and mentally side-stepping many distractors. 

Question dissection

I wrote test items for the NCCPA for four years. If you’ve never tackled writing high-level test questions, I assure you, it’s often more difficult than answering them. Test questions require highly specific construction that will allow them to be validated through multiple administrations. This challenging experience taught me how to reverse-engineer questions, a trick that will help with test-taking almost immediately.

Multiple choice questions at higher levels are usually presented in vignette form and organized in a particular manner.

•  Subjective data  (this includes patient information)

• Objective data  (the content within the question, laboratory data)

• Question (what the question wants to know from you)

• Distractors (wrong answers) and the correct answer.

Most multiple-choice questions are written within three levels of knowledge: 

1.      Declarative knowledge (1-step questions). Readers are given some information. From the answer options, one will stand out as the clear choice. These are relatively easy and don’t require deep analysis, and unfortunately, they’re not used often on the PANCE. If they were, many test-takers wouldn’t need my help!

2.     Application knowledge (1+ step questions). These questions require the reader to apply knowledge, while the vignette skips a step, requiring the reader to come to some conclusion before an answer is apparent. For example, this might include a clinical presentation without giving a diagnosis, asking, “How do you treat this?” These questions comprise a fair portion of the PANCE.

3.     Analysis / evaluation (2-step questions). These are the most challenging. Often, these questions force the test-taker to think differently. Test-takers who can get the first two levels of questions correct still struggle with this third level. There are enough of these questions on the PANCE that the percentage can make the difference between passing and failing.

Question hacks

I also offer these guidelines on coping with multiple-choice questions that demand more than one-step knowledge.

Always read the last line of the question first. 

I’ve heard many anxious test-takers lament, “I had no idea what the question was asking!” Look to the last line of the question for common key phrases that specify what information will be important.  

Test-takers should do this before reading any of the vignette because they can:

• Determine the question type

• Create a “focus lens” when reading

• Avoid “prepping” the wrong answer or focusing on the wrong details

• Limit the need to reread the question (essential when a test is timed!)

Be aware of distractor complexity. 

On many multiple-choice questions, two answers will seem like the possible “right answer,” and one must choose between them. This is a good way to narrow down answers to more challenging questions. 

However, in the highest-level questions, the concept of “correct” or “incorrect” falls on a spectrum, and none of the answers are egregiously incorrect. We are searching for the “most right” answer rather than eliminating “wrong” answers. Knowing this may help with selecting the correct solution. If all the answers sound very similar, one is likely dealing with a high-level question.

Textbook answers, not personal experience. 

I’ve heard from many students that test answers did not coincide with what they witnessed in their clinical rotations. I always remind my test-preppers that the PANCE is testing textbook literature (best practices). They must be cautious not to rationalize an incorrect answer based on their experience.

Final words

The PANCE (or any formative exam, for that matter) has a broad spectrum of question difficulty. For some basic questions, a test-taker will immediately know the answer or easily pick it from a lineup of options. The difference between scoring a 375 and a 500 on the PANCE is how the test-taker handles those questions for which they do not immediately know the answer.

That being said, it is almost inevitable that there will be a question that the test-taker doesn’t know. I’ve taken the entry-level PANCE six times to get a good feel for the difficulty. I also taught clinical medicine for 15 years. There are still questions on the PANCE that I do not know the answers to! On this challenging test, there is no shame in saying, “Oh well!” to a difficult question and moving on without losing confidence.

Well, that’s enough test analysis for now! I hope you’ll join me next week as we continue our journey through PA education. Our consultant, Dr. Medea Valdez, has recently presented a webinar on consultants to improve accreditation. If you didn’t get to attend, I’m excited to share some of the high points with you. Until then!

Mark your calendars!

If you would like to participate in the next presentation of PASS AND EXCEL, we’ll be announcing the dates and times for the October session soon. You can sign up now by clicking the button below to reserve your seat