We just closed our first pair of update quizzes, so I wanted to share with you the aggregated stats about the quiz and to talk about a few questions from it. You can look forward to an article like this at the end of every quiz season (including the upcoming Murders at Karlov Manor, where quizzes will be open soon!).
First, a little philosophy:
The update quizzes are designed to keep judges up to date on new or changed rules and policy. As a part of the maintenance process, these quizzes are a tool to help judges keep their skills current and relevant. That’s why we have a relaxed passing score and give judges two attempts to pass each one – our goal is that by the time you’ve finished taking the quiz, you’ve become familiar with the new rules and policy, even if you weren’t acquainted with them already.
The update quizzes also include questions that may be added to our evaluation exams – questions from the standard quiz could be added to the Level Two exams, and questions from the advanced quiz could be added to the Advanced Rules and Policy exam administered for Level Four and some Level Five candidates. Because these quizzes are part of maintenance, the data we get on pass rates and the comments we receive help us see which questions are performing appropriately for judges at this level, and lets us fine-tune them before adding them to advancement exams.
Stats
The quiz was five questions long (four rules, one policy), and judges needed a 60% or better score to pass. The question pool was big enough for each judge to make two attempts, each with unique questions.
The standard quiz was attempted 155 times by 138 judges. A little over 61% of our L2s and L3s took the quiz (note that this percentage is probably an undercount- some folks were promoted to L2 or joined Judge Foundry after the quiz closed, and a couple L2s were involved in writing the quiz). Here’s a chart on how folks did:
| Score | Overall | L2 | L3 |
|---|---|---|---|
| 5/5 | 39% | 39% | 40% |
| 4/5 | 46% | 46% | 40% |
| 3/5 | 10% | 9% | 13% |
| 2/5 | 5% | 5% | 7% |
| 1/5 | 0% | 0% | 0% |
| 0/5 | 0% | 0% | 0% |
Almost everyone who took the quiz passed on their first attempt, and most of those judges who didn’t pass their first attempts passed on the second.
The advanced quiz was attempted 21 times by 20 judges. That’s almost every L4 and L5 (except the ones who were involved in writing the exam)! Here’s the chart on how folks did:
| Score | Overall | L4 | L5 |
|---|---|---|---|
| 5/5 | 29% | 29% | 29% |
| 4/5 | 57% | 57% | 57% |
| 3/5 | 14% | 14% | 14% |
| 2/5 | 0% | 0% | 0% |
| 1/5 | 0% | 0% | 0% |
| 0/5 | 0% | 0% | 0% |
This isn’t a typo – L4s and L5s had the same percentage in each score bracket this time. All the test-takers passed this exam on their first attempt.
A few questions
I’m going to walk through a couple questions from the exams. For various reasons that I’ll get into, these questions won’t be added to our evaluation exams, so I’m comfortable sharing them here.
How big is that Raptor?
First, Q2245, a rules question written by Nathan Long for the advanced quiz. 67% of L4s and L5s answered it correctly:
Anthony activates the craft ability of Saheeli’s Lattice, choosing to exile a Pathfinding Axejaw with a +1/+1 counter on it and his Commander, Gishath, Sun’s Avatar, to the craft ability. Anthony chooses to move his commander to the command zone. What is the power of Mastercraft Raptor once the craft ability resolves?
- 11
- 5
- 4
- 12
Ready for the answer?
The correct answer is C. Here’s the explanation:
Mastercraft Raptor looks at the power of the cards in exile to determine what its own power is. Pathfinding Axejaw lost the +1/+1 counter when it left the battlefield, so its power is 4 in exile. And since Gishath, Sun’s Avatar is not in exile, the static ability of Mastercraft Raptor cannot get any information about it, even if it was briefly in exile [CR 702.167c]. The only card currently in exile that was used to craft it is Pathfinding Axejaw, and its power is 4 in exile, so Mastercraft Raptor’s power is 4.
This question is likely destined for a hard rules practice test, because it’s not a good candidate for any other exam. We want good questions involving Commander-specific rules for the L2 exam. But this question is probably too hard for that exam, and we generally try to avoid Commander-specific questions on the Advanced Rules and Policy exam.
Order of Operations
Next, let’s take a policy question from the standard quiz. Q2200 was written by me, and it has a pass rate of 34%.
During the first game of a match against Natasha at a Regional Championship, Angela discovers that she forgot to de-sideboard from a previous game, because she drew a sideboard card. Angela says that she has two sideboard cards in her deck and one in her hand. In what order is the remedy applied?
- Locate the incorrect cards in the library and hand and the cards in the sideboard that should be in the deck. Reveal all those cards to Natasha. Natasha decides if the cards are replaced now, or the next time a player would get priority. Replace each card, shuffling the library cards into the random portion of the library.
- Locate the incorrect cards in the library and hand. Reveal them to Natasha. Natasha decides if the cards are replaced now, or the next time a player would get priority. Locate the cards in the sideboard that should be in the deck. Reveal those cards to Natasha. Replace each card, shuffling the library cards into the random portion of the library.
- Locate the cards in the sideboard that should be in the deck. Reveal them to Natasha. Natasha decides if the cards are replaced now, or the next time a player would get priority. Locate the incorrect cards in the library and hand. Reveal those cards to Natasha. Replace each card, shuffling the library cards into the random portion of the library.
- Natasha decides if the cards are replaced now, or the next time a player would get priority. Locate the incorrect cards in the library and hand and the cards in the sideboard that should be in the deck. Reveal all those cards to Natasha. Replace each card, shuffling the library cards into the random portion of the library.
- Natasha decides if the cards are replaced now, or the next time a player would get priority. Locate the cards from the deck that should be in the sideboard, reveal them, then move them to the sideboard. Reveal one-by-one the cards in the sideboard that should be in the hand and library, as each card is revealed, Natasha decides where it should go.
- Locate the cards from the deck that should be in the sideboard, reveal them, then move them to the sideboard. Reveal one-by-one the cards in the sideboard that should be in the hand and library, as each card is revealed, Natasha decides where it should go. Natasha decides if the cards are replaced now, or the next time a player would get priority.
Ready for the answer?
The correct answer here is A. Here’s the explanation:
These steps are taken in the order that best allows Natasha to negate any possibility of advantage. First, all cards are revealed. Then Natasha decides whether to fix the problem now or the next time a player would receive priority. Then Natasha decides where each card goes.[IPG 3.5]
I expect us to discard this question, because it has a few problems. First, a pass rate that low means that either there’s an issue with the question or we wildly misjudged the difficulty. I think the former is probably more likely – sorry! As one judge who provided feedback pointed out:
The correct answer does not include that [NAP] gets to decide where each card goes. Only an incorrect answer does.
Because of this, judges had to pick what they felt was the least-incorrect answer – not really what we want! Policy questions often tend to get a little wordy, and cutting “[NAP] decides what goes where” in the interest of wordiness reduced clarity unacceptably.
In addition to the wordiness problem, this is a fairly question that tests a fairly esoteric change. We wanted to include it because it’s a good update quiz question (we want judges to realize that this was updated!), but the exact order of operations here isn’t something we need every L2 and L3 to know from memory. Overall, this question would just require too many changes to be appropriate for another exam.
We also had feedback from judges who felt that the answer was incorrect. This was most likely the result of citing an outdated version of the IPG. Make sure you’re reading the latest version of the IPG when answering policy questions on update quizzes, because the quiz is probably about the changes! For reference, here’s the text from the IPG 3.5:
Locate any cards missing from the deck and any incorrect cards in any game zone. Reveal those cards to the opponent. If the game has started, the opponent chooses whether to fix the problem now or when a player would next get priority, and which of the missing cards replaces each incorrect card; any extras are shuffled into the random portion of the library. If more cards are being removed than added, prioritize ones not in the library first. If there are still additional cards not in the library that need replacing, they are replaced by cards from the random portion of the library. If the missing card(s) are with the sideboard and it isn’t the first game, choose the ones to be returned to the deck at random from main deck cards in the sideboard.
Fail to Find
Last up, I’ll share Q2243, another rules question by Nathan Long from the standard quiz:
Andre casts Geological Appraiser, and when the discover trigger resolves, he realizes that he does not have any nonland cards with a mana value of 3 or less in his library. What happens next?
- Once he’s out of cards in his library, since there are no more cards to exile, we move on to the next part of resolving the discover ability. He will not cast anything nor put anything into his hand, and the cards he just exiled go on the bottom of his library in the order of Andre’s choosing.
- Once he’s out of cards in his library, since there are no more cards to exile, he will lose the game since he tried to exile cards from an empty library.
- Once he’s out of cards in his library, since there are no more cards to exile, we move on to the next part of resolving the discover ability. He will not cast anything nor put anything into his hand, and the cards he just exiled will stay in exile.
- Once he’s out of cards in his library, since there are no more cards to exile, we move on to the next part of resolving the discover ability. He will not cast anything nor put anything into his hand, and the cards he just exiled go on the bottom of his library in a random order.
Ready for the answer?
The correct answer is D. Here’s the explanation:
Once Andre is out of cards in the library, since we can no longer exile cards from the library, we move onto the next step of resolving the ability [CR 609.3]. Andre did not exile a card with a low enough mana value, so he won’t cast anything or put anything into his hand, and the rest of the exiled cards go on the bottom of the empty library in a random order [CR 701.55a].
This question will probably either go into Easy Rules Practice or be discarded. Why? Well, if you picked the correct choice, you’re not alone – all 64 L2s and L3s who saw this question picked the right answer!
We thought this was another good thing on which to test, since we really didn’t want judges ruling incorrectly and doing A, B, or C, since these are fairly catastrophic (I’ll leave it to the reader to decide which is worse: having Andre lose the game or letting him stack his entire library). But it turns out that everyone already knew how this should work – great!
While we could move this to the L1 exam and see how it performs there, we generally want the L1 exam to test evergreen or deciduous keywords over set-specific keywords. So while we’re pleased to see that everyone got this question right, you probably won’t see it again.
Conclusion
That’s it for the The Lost Caverns of Ixalan update quiz update. I hope it was a helpful look at how the update quizzes work.
Starting with Murders at Karlov Manor, our new Exam Manager for the update quizzes, Steven Zwanger, will be writing these articles. That quiz will be going live soon, so keep an eye out for it.
