We’ve just closed another quiz, which means it’s time for a quiz wrapup. As always, these wrapups are designed to help our members understand our exams, to better learn rules and policy, and to use puns themed around the set one last time before we move on to the next set.
Let me start by thanking the people who made this quiz happen. We couldn’t have these quizzes without our fantastic volunteers, so please join me in expressing gratitude to them.
Steven Zwanger is the Update Quiz Manager. Questions were written by Nathan Long, Mark Mason, Dan Milavitz, and Tobias Vyseri, and me, John Brian McCarthy. Testing wouldn’t be possible without the JudgeApps development work by Paul Baranay and Dan Collins.
Test like the animals
Each quiz was five questions long (four rules, one policy), and judges needed a 60% or better score to pass. The advanced question pool had 10 questions (enough for two attempts), while the standard pool had 18 questions (enough for three attempts).
The standard quiz was attempted 144 times by 113 judges. Here’s a chart on how folks did:
| Score | Overall | L2 | L3 |
|---|---|---|---|
| 5/5 | 35% | 30% | 70% |
| 4/5 | 40% | 42% | 25% |
| 3/5 | 19% | 21% | 5% |
| 2/5 | 4% | 5% | 0% |
| 1/5 | 1% | 2% | 0% |
| 0/5 | 1% | 1% | 0% |
93% of test-takers passed on their first attempt, and 98% passed within two tries. This was our toughest quiz yet, especially for our Level Two Judges! While almost everyone still passed, hopefully everyone learned something from the experience.
The advanced quiz was taken 19 times by 17 judges. Here’s the chart for the advanced quiz:
| Score | Overall | L4 | L5 |
|---|---|---|---|
| 5/5 | 16% | 12% | 18% |
| 4/5 | 68% | 62% | 73% |
| 3/5 | 11% | 12% | 9% |
| 2/5 | 0% | 0% | 0% |
| 1/5 | 5% | 12% | 0% |
| 0/5 | 0% | 0% | 0% |
94% of test-takers passed this quiz. Much like the standard quiz, this quiz proved challenging for many of our members.
Gathering two of every question type
Part of the purpose of the update quizzes is to generate new questions for our evaluation exams. While we always have plenty of mechanic-specific questions, our question authors also write a lot of great questions that are a good fit for testing general principles of rules and policy. Of these questions, we look at the ones that have a reasonable pass rate – too high or too low and it’s probably not a good fit, but a pass rate ± 10% relative to the exam pass rate is often a good candidate.
Questions that don’t make it to the evaluation exams aren’t necessarily bad – many are still great questions, they just aren’t broad enough that we want to make this knowledge essential to advancement. Many of these go onto our practice quizzes, which are available to all judges as a way of enhancing and testing their knowledge.
On this exam, here’s how relegation broke down:
| Destination | Questions |
|---|---|
| Advanced Rules and Policy | 5 |
| L2 Rules | 5 |
| L2 Policy | 2 |
| L1 Rules and Policy | 2 |
| Easy Rules Practice | 7 |
| Hard Rules Practice | 5 |
Paws for Applause
A number of questions had interesting results, but among them, three stood out as both worth discussing in depth and unlikely to be promoted to an exam.
Standard Stumper
Our first question today is Q2925. This was a policy question on the Standard test, with a pass rate of 49%… yikes! Let’s see what’s going on here:
As of the prerelease of Bloomburrow, which of the following sets are no longer legal in a Standard tournament? (Select all that apply.)
- Innistrad: Midnight Hunt.
- Streets of New Capenna.
- Dominaria United.
- March of the Machine: The Aftermath.
- Standard format rotation happens when a set is officially released, not at prerelease weekend.
Ready for the answer?
The correct answers are A and B. Here’s why:
With the prerelease of Bloomburrow, Standard rotates, and the four oldest sets in Standard (Innistrad: Midnight Hunt, Innistrad: Crimson Vow, Kamigawa: Neon Dynasty, and Streets of New Capenna) rotate out. Those sets are no longer legal in Standard. [MTR 6.3]
(note that the MTR has been updated with the release of Duskmourn, so you’ll see something different now than people saw in the cite while taking this quiz)
This is the kind of question that we like to ask every once in a while to see how well judges are able to answer players’ questions at an LGS. Knowing what’s going to rotate and when is something that players often misunderstand, and it can come up at events held around prerelease/release weekend.
So how did judges get this question wrong?
Let’s look at the stats (hidden so as not to spoil the answer for people who glance down while reading the question):
| Count | A | B | C | D | E |
|---|---|---|---|---|---|
| 25 | A | B | |||
| 18 | E | ||||
| 4 | A | B | E | ||
| 2 | A | B | C | ||
| 1 | A | ||||
| 1 | C | D |
For those who aren’t aren’t familiar with this chart, it’s a chart that appears on JudgeApps whenever we have a multi-select question, that helps the exam teams understand how people got it right or wrong. This chart can be useful to diagnose if a wrong answer is misleading or too close to the correct answer, or if there’s a wrong answer that is just never being selected and should probably be eliminated.
As you can see, the most common issue was that judges picked E instead of A and B. This answer was once correct! When I started judging, and I suspect when many of our members who answered this question incorrectly started judging, E was the correct answer. This meant that on a Standard event on the Wednesday night after prerelease, judges had to remind players that they couldn’t play with any of the exciting new cards they opened in their prerelease kits.
This was un-fun for many people. Players built decks about which they were excited, stores wanted people excited about the new product, judges didn’t like having to give bad news.
So it changed! With the release of Phyrexia: All Will Be One, Wizards updated when new sets became legal in Standard.
We didn’t receive any feedback on this question from people who answered it, which leads me to believe that all the people who got it wrong said “Huh. Learned something today.”
This question is getting discarded, and not going onto another exam, partially because I don’t want to keep updating it and want even less to have people have to figure out what was legal at a certain point of time, years later. But don’t be surprised if we ask a very similar question next summer to see if we get similar results…
And you won’t believe what happened next…
Our next question is Q2979, a rules question from the standard quiz. This question had a pass rate of 88%. That’s pretty good – see how you do:
Anita controls Blacksmith’s Talent at Level 3, Flowerfoot Swordmaster, Heartfire Hero, and a Sword Equipment artifact token created by Blacksmith’s Talent. The Sword is equipped to Heartfire Hero, which has no counters on it. She casts Valley Rally promising a gift, targeting the Flowerfoot Swordmaster. At the beginning of combat, Anita resolves the triggered ability of Blacksmith’s Talent, targeting the Sword token and Heartfire Hero. Then she attacks with all her creatures. During Anita’s turn, all triggered abilities resolve, and Nedra does not block. How much damage does Anita deal this turn in each combat damage step?
- In the first combat damage step, Flowerfoot Swordmaster deals 4 damage. In the second combat damage step, Heartfire Hero deals 6 damage.
- In the first combat damage step, Heartfire Hero deals 7 damage, and Flowerfoot Swordmaster deals 4 damage. In the second combat damage step, Heartfire Hero deals 7 damage.
- In the first combat damage step, Heartfire Hero deals 6 damage, and Flowerfoot Swordmaster deals 3 damage. In the second combat damage step, Heartfire Hero deals 6 damage.
- In the first combat damage step, Heartfire Hero deals 6 damage, and Flowerfoot Swordmaster deals 4 damage. In the second combat damage step, Heartfire Hero deals 6 damage.
- In the first combat damage step, Heartfire Hero deals 6 damage, and Flowerfoot Swordmaster deals 4 damage. In the second combat damage step, no creature deals damage.
Ready for the answer?
The correct answer is D. Here’s why:
At the start of the turn, Flowerfoot Swordmaster is a 1/2 and Heartfire Hero is a 2/2 with double strike and haste. When Valley Rally is cast, it triggers Flowerfoot Swordmaster’s valiant ability, giving +1/0 to both Mouse creatures. When Valley Rally resolves, both creatures gain +2/0 and Flowerfoot Swordmaster gains first strike. At the beginning of combat, Blacksmith’s Skill targets Heartfire Hero, which causes its valiant ability to trigger, adding a +1/+1 counter to it. [CR 608.2c]
This is a pretty good question – it tests both basic rules knowledge and the set mechanic, and the splits are well-designed to not give away the answer. So is it moving to an evaluation exam?
It’s not, and not through any fault of the question’s author. This question is in a weird spot where it’s probably the right difficulty for the L2 rules test, but it’s also a bit wordy for that test. We try to limit both the number of cards and the word count for questions and answers, because taking an exam can be exhausting enough without adding a ton of reading and re-reading. We’re more likely to bend this rule on higher-level exams (you kinda just need to raise the word cap to get enough questions with higher complexity) and on policy questions (which generally don’t have a lot of cards to read).
The question received a number of edits both before and after the quiz went live, but there’s not really a way to ask this question that doesn’t just use a lot of words. So while the pass rate is where we want, we think it’s still probably best to move this to the Hard Rules practice quiz.
A Cavern of Souls question… but not the Cavern of Souls question you think
Our last question is a policy question from the advanced quiz, Q2975. This question had a pass rate of 43%, or 3/7.
At a Regional Championship, Alfred plays Cavern of Souls and doesn’t name a creature type. Two turns later, the players realize the error and call a judge, who determines that Alfred has committed Game Play Error–Game Rule Violation, and Nadia has committed Game Play Error–Failure to Maintain Game State. The judge issues a Warning to each player. What best describes the appropriate remedy?
- Have Alfred name a creature type now.
- Leave the game state as is.
- Back up the game to the point where Alfred played Cavern of Souls.
- Have Nadia decide whether Alfred names a creature type now or at the beginning of the next phase.
Ready for the answer?
The correct answer is B. Here’s why:
The IPG prescribes a partial fix when a player made an illegal choice (including no choice where required) for a static ability generating a continuous effect still on the battlefield. However, Cavern of Souls has no static ability which generates a continuous effect, so the partial fix does not apply. Therefore, the options are to back up the game to the point of the error, or leave the game state as is. [IPG 2.5] Since backing up two full turns is not feasible, the appropriate fix is to leave the game state as is.
At some point over the winter, we decided to declare a moratorium on new Tishana’s Tidebinder questions, because we were approaching the point where we could make an entire rules exam out of them (we joked about doing this as an April Fool’s joke). Cavern of Souls is getting to a similar point – it’s a card that sees a ton of play, and which causes all manner of rules and policy problems.
For this question, we’re charting new ground with a fairly specific wording of policy – does the partial fix for a GRV cover this case? And I think that it’s a question where there’s a strict reading of policy that gives an answer, but that many L4s and L5s have a different interpretation of policy philosophy that would imply a different answer (everyone who gave an “incorrect” answer selected “Have [AP] name a creature type now.”
This wouldn’t be a good question for an evaluation exam, but it does make a good discussion question, and I’ve heard a lot of debate on the issue between judges over the last few months. Including the question on this exam helped make L4s and L5s aware of the debate, and I expect we’ll continue to discuss it and may even raise it with Toby.
The end of Rabbit Season Bird Season Rabbit Season
That’s it for this season – I hope it was a useful look into the Bloomburrow update quiz. The quiz for Duskmourn is open now – if you’re a Level Two+ Judge, make sure you take it in the exam hub. And for everyone, I’ll be migrating questions to the practice pools in a few days so keep an eye out for more chances to improve your knowledge.
