Judge Foundry had a unique milestone in the past month. If you have been following our board meetings you will know that one of my goals has been to go through our first panel process to make a L4 or L5 judge. That milestone has arrived – we received our first application!
Now that we get to move forward with that process, I have a strong desire for additional transparency, so I’m going to be writing articles, perhaps more in the style of blog posts, to share the goings on of advanced testing management. This first article serves a couple of purposes. First, I want to explain a brief history of panels, and give all judges some advice on etiquette around panels, the process, and expectations. Then I also want to give an update on the current status of what we’re doing.
What’s a Panel?
I guess we should start here. Judges who have joined the community lately, or haven’t been close with a judge who advanced to L3 prior to Judge Foundry, might not even know what a panel is. From the L4 level definition, we get this:
The panel is a group interview where the panel asks questions of the candidate to assess them in each quality. The questions will vary, depending on the candidate’s known strengths and weaknesses.
For a judge to advance to L4 or L5, they need to sit down with an L5 and have an extensive interview evaluation. While we do one on one interviews for Levels One through Three, the process differs for Levels Four and Five due to the additional scrutiny needed and a larger body of work to review. For L4 and L5, a group of high level judges will spend time together crafting an interview . That panel interview might involve questions about how the candidate might handle specific situations, policy philosophy, or even role-playing scenarios.
Panel Ground Rules and Etiquette
First and foremost, the L4/L5 process can be very stressful. Many candidates won’t tell anyone that they are testing or paneling at a specific event. If you learn about a panel, you might be excited to share that information with your colleagues. My advice is to remember that the candidate might not want to talk about something that is stressful while working an event, and that it’s their information to share. We should respect their privacy and ability to consent to releasing the information as they like.
Likewise, panelists are advised to not reveal they are on a panel for that reason. The panels manager might ask the candidate if they want to know who their panel lead is, or who is on the panel, but if they don’t want to know and don’t want it shared, that’s up to them. Many panels have been revealed by nosy judges spreading rumors, and sometimes they are wrong. You should be careful to respect privacy about this information as well.
The last thing that’s private about panels is their content. As you have likely read from our testing team, exam content is incredibly valuable. It takes a lot of time to develop good exam questions. It takes even more time to develop strong evaluation techniques for panels. Senior judges learn from being on and observing panels, but the expectation for candidates, panelists and leads is to not publicly share the questions or content used during a panel. As with violating exam content confidentiality, unauthorized sharing of the testing material used in a panel may result in sanctions.
Finally, and most importantly for the candidate, is that when the panel is over that they have support. Sometimes it’s a couple of close friends or mentors to console them if they don’t pass, and take a quiet dinner to bring down the stress and talk. Other times the candidate will achieve their new level, and want to celebrate. Those same close friends are encouraged to be there for that support, sometimes celebration also means a much more happy quiet dinner. Sometimes celebration means a victory lap around the staff party. Either way, the journey isn’t over after a panel, and the candidate will have feedback to help them improve as a judge.
Where we are now
You can review the full panel process on the L4 and L5 level description pages, but the tl;dr is that these steps need to happen for the candidate.
- Identify a panel lead, and have them approve the application
- The candidate takes the Advanced Rules and Policy exam
- If they score high enough, the panel lead will complete any necessary pre-panel activities
- Selecting the other panelists
- Select date and location of the panel
- Pre-panel interview questions (optional)
- Pre-panel practical observation (optional)
- Prepare panel content and evaluation plans
- The panel occurs, and the candidate may be advanced to their new level
- The panel lead writes a panel review for the candidate
What we need to do
To move forward, the Panels project has two key goals for the next couple of weeks.
- Identify all L4s and L5s interested in being on a panel or leading, and choose appropriate panel leads
- Finish preparing the Advanced Rules and Policy exam
That’s where we are now, and where we’re aiming for the next couple of months. If you’re an L3 or L4 and think you’re ready to apply, check out the requirements on the levels pages. You apply by emailing your checklist and self review and/or recommendations to joe.klopchic@judgefoundry.org.
If you’re an L4 or L5 and want to be involved in panels, please let me know via the same email.
