The FIRST® LEGO® League (FLL®) tournament experience includes 3 judged components in addition to the points-based Robot Game: Project, Robot Design, and Core Values. The judging process is designed both to evaluate each team’s accomplishments throughout the season and to provide meaningful feedback through interaction with the judges.
Awards at official FLL events recognize those teams that demonstrate extraordinary achievement in key
areas central to the FLL program and mission. An important part of your role as the Coach is to help each team member to understand that the real reward is what they discover and learn, both from one another and through the FLL experience.
Judging Roles
Judges at FLL events are Volunteers just like many Coaches. Judges may be veterans or rookies, and receive training from FLL or the local FLL Partner prior to the event. Judges may be educators, experts in an area related to the Challenge, or perhaps a community member or industry leader who cares about inspiring children.
The judging process at FLL events is overseen by a key Volunteer known as the Judge Advisor. The Judge Advisor leads the judging team and works with the tournament organizers to ensure that the event meets FLL judging standards. Just as the Head Referee determines the recorded score at the Robot Game table, the Judge Advisor’s word is final when it comes to any judging questions or decisions at an event.
Judging Session
Your team will participate in 30 minutes judging session at an official event. While the session may take many forms, It is designed to create an opportunity for judges to observe and interact with the team. During the judging session, the judges will ask your team questions about the work and season. Some questions may be standard ones, but judges are always free to use any questions they feel will help them assess and differentiate teams. Note that judges will address questions to team members. In regions where Coaches may join teams in their judging sessions, your job as a Coach is to sit quietly and allow the team to respond.
Awards at official FLL events recognize those teams that demonstrate extraordinary achievement in key
areas central to the FLL program and mission. An important part of your role as the Coach is to help each team member to understand that the real reward is what they discover and learn, both from one another and through the FLL experience.
Judging Roles
Judges at FLL events are Volunteers just like many Coaches. Judges may be veterans or rookies, and receive training from FLL or the local FLL Partner prior to the event. Judges may be educators, experts in an area related to the Challenge, or perhaps a community member or industry leader who cares about inspiring children.
The judging process at FLL events is overseen by a key Volunteer known as the Judge Advisor. The Judge Advisor leads the judging team and works with the tournament organizers to ensure that the event meets FLL judging standards. Just as the Head Referee determines the recorded score at the Robot Game table, the Judge Advisor’s word is final when it comes to any judging questions or decisions at an event.
Judging Session
Your team will participate in 30 minutes judging session at an official event. While the session may take many forms, It is designed to create an opportunity for judges to observe and interact with the team. During the judging session, the judges will ask your team questions about the work and season. Some questions may be standard ones, but judges are always free to use any questions they feel will help them assess and differentiate teams. Note that judges will address questions to team members. In regions where Coaches may join teams in their judging sessions, your job as a Coach is to sit quietly and allow the team to respond.
Core Values Judging
FLL Core Values are central to our mission. Throughout the tournament and particularly during any scheduled sessions, Core Values judges will assess how well your team understands and integrates these values into their tournament experience and daily life.
Judges will observe if the team has found a fun way to express a team spirit that is all their own. They will look for a balanced emphasis on all elements of the FLL competition, not just the part that comes easiest. They will ask about how the team sets and achieves its goals, manages conflict, creates different roles for team members, and manages time constraints.
Some events will rely on interviews, while others will utilize a hands-on teamwork activity, to help the judges assess how independently the team is able to operate and the level of respect accorded everyone’s ideas and contributions. They will look for signs that Gracious Professionalism® is alive and well, on and off the playing field.
Robot Design Judging
During the tournament day and any scheduled sessions, Robot Design judges will observe each robot and ask teams about their mechanical design and the programs they wrote. Judges want to hear why the team made certain choices in the development process to enhance durability, efficiency, or programming quality. They will want to see and hear about any innovative techniques or strategies the team came up with to solve problems and complete missions. The judges want to know about the overall design process and what the team considers the best parts of the robot. Importantly, through their questions they will ensure that the children completed and understand all work associated with building their robot.
Be aware that some events require formal robot presentations, so pay attention to any information provided by your tournament organizer about what your team should expect. Even when formal presentations are not required, some judges simply prefer to start interviews with a general, “Tell us about your robot.” Others will ask very specific questions. Help your team prepare for different scenarios and make sure the children are comfortable demonstrating the robot. We also recommend that your team bring a printout of its best programming to leave with the judges.
Project Judging
During their 5 minute Project presentation, the team must clearly demonstrate to the judges that they completed all required Project steps: they identified a real-world problem, created an innovative solution, and shared their findings with others. In addition, they must show that any other Challenge- specific requirements from this season’s Project description were met.
Remember, the judges can only evaluate what they hear; in other words, team members should tell the judges how they completed each required Project step during the presentation in order to receive credit.
The second half of each Project judging session is devoted to a question and answer interview. Judges will ask teams about how they selected and defined the problem they chose to study. They will want to hear about the research process and sources your team used to analyze the problem and existing solutions. Judges will want to learn about the team’s innovative solution and how it has the potential to make life better. They will want to know that the team considered what would be needed to make their idea a reality. In addition, they will want to understand how the team shared their ideas with others prior to the tournament and how those people might benefit from the team’s idea.
Judging Rubrics
Each FLL judging session is supported by a rubric developed to help judges record their feedback. The rubrics guide the judges through key criteria that reflect what we believe is most important about the FLL experience. Rubrics also create a consistent way to differentiate between teams at different levels of achievement. Your team will be assessed as Beginning, Developing, Accomplished, or Exemplary in each category.
The rubrics are not just a tool for the judges, however. After the event, completed rubrics should be returned to teams to help them learn their strengths and areas for improvement. In addition, teams are encouraged to use the rubrics as a roadmap throughout the season. Refer to them often and use them to your advantage. Pay close attention to the higher levels of achievement to understand the criteria that will help define a reliable, well-programmed robot design, a thoroughly researched and effective Project, and a high-functioning team that embraces the FLL Core Values.
FLL Core Values are central to our mission. Throughout the tournament and particularly during any scheduled sessions, Core Values judges will assess how well your team understands and integrates these values into their tournament experience and daily life.
Judges will observe if the team has found a fun way to express a team spirit that is all their own. They will look for a balanced emphasis on all elements of the FLL competition, not just the part that comes easiest. They will ask about how the team sets and achieves its goals, manages conflict, creates different roles for team members, and manages time constraints.
Some events will rely on interviews, while others will utilize a hands-on teamwork activity, to help the judges assess how independently the team is able to operate and the level of respect accorded everyone’s ideas and contributions. They will look for signs that Gracious Professionalism® is alive and well, on and off the playing field.
Robot Design Judging
During the tournament day and any scheduled sessions, Robot Design judges will observe each robot and ask teams about their mechanical design and the programs they wrote. Judges want to hear why the team made certain choices in the development process to enhance durability, efficiency, or programming quality. They will want to see and hear about any innovative techniques or strategies the team came up with to solve problems and complete missions. The judges want to know about the overall design process and what the team considers the best parts of the robot. Importantly, through their questions they will ensure that the children completed and understand all work associated with building their robot.
Be aware that some events require formal robot presentations, so pay attention to any information provided by your tournament organizer about what your team should expect. Even when formal presentations are not required, some judges simply prefer to start interviews with a general, “Tell us about your robot.” Others will ask very specific questions. Help your team prepare for different scenarios and make sure the children are comfortable demonstrating the robot. We also recommend that your team bring a printout of its best programming to leave with the judges.
Project Judging
During their 5 minute Project presentation, the team must clearly demonstrate to the judges that they completed all required Project steps: they identified a real-world problem, created an innovative solution, and shared their findings with others. In addition, they must show that any other Challenge- specific requirements from this season’s Project description were met.
Remember, the judges can only evaluate what they hear; in other words, team members should tell the judges how they completed each required Project step during the presentation in order to receive credit.
The second half of each Project judging session is devoted to a question and answer interview. Judges will ask teams about how they selected and defined the problem they chose to study. They will want to hear about the research process and sources your team used to analyze the problem and existing solutions. Judges will want to learn about the team’s innovative solution and how it has the potential to make life better. They will want to know that the team considered what would be needed to make their idea a reality. In addition, they will want to understand how the team shared their ideas with others prior to the tournament and how those people might benefit from the team’s idea.
Judging Rubrics
Each FLL judging session is supported by a rubric developed to help judges record their feedback. The rubrics guide the judges through key criteria that reflect what we believe is most important about the FLL experience. Rubrics also create a consistent way to differentiate between teams at different levels of achievement. Your team will be assessed as Beginning, Developing, Accomplished, or Exemplary in each category.
The rubrics are not just a tool for the judges, however. After the event, completed rubrics should be returned to teams to help them learn their strengths and areas for improvement. In addition, teams are encouraged to use the rubrics as a roadmap throughout the season. Refer to them often and use them to your advantage. Pay close attention to the higher levels of achievement to understand the criteria that will help define a reliable, well-programmed robot design, a thoroughly researched and effective Project, and a high-functioning team that embraces the FLL Core Values.
|
|
|
Judges Are All Around
In addition to evaluating teams using the rubrics during scheduled interview sessions, judges may also use less formal conversations and observations throughout the event to learn more about teams.
Remind your team that judges may not openly demonstrate who they are, but their ears and eyes are wide open. Lending a helping hand to a team that forgot to bring something speaks loudly about your team’s understanding of Gracious Professionalism.
Judges will also consider any input provided throughout the day by referees, event Volunteers and others who interact with the team. Help your team members to understand the process and encourage them to feel comfortable speaking with judges and other event Volunteers.
Call-backs
When time allows, some events incorporate a second round of judging sessions referred to as “call-backs.” The length and format for these sessions is not prescribed and can vary significantly due to event and schedule constraints.
Call-backs may be requested for many reasons, ranging from a judge wanting additional clarification about a team, project, or robot to multiple judges needing to see closely ranked teams that may be in consideration for an award. Please be careful not to make assumptions about your team’s chances based on a call-back request. Many awards are decided without the need for them and a call-back never guarantees an award will be given. If your team is asked to participate in a call-back, be sure to find out whether or not the team needs to bring their project, robot, or other materials to the session.
Judging Subjectivity
With the exception of Robot Performance, which is objectively determined by scores earned on the competition table, team achievement in all other FLL award categories is subjectively judged. Even the most experienced and skilled judges will not assess every team in exactly the same way. We have judges work in pairs or small groups so they create more balance in the way they review teams. In addition, FLL tournament organizers work hard to train their judges and utilize other tools to create a level playing field for all teams.
FLL continues to improve tools and training for our judges, and the processes we have developed yield fair, strong, and consistent results. However, the fact remains that judged awards are inherently subjective and this is important for all teams to understand. To account for this, awards are determined through a normalization process, which includes deliberations and discussions among the judges to help reduce the effect of subjectivity as much as possible.
FLL Deliberations
Led by the Judge Advisor, FLL deliberations rely on an in-depth discussion of all teams eligible for awards. Using observations and evaluations captured by the rubrics as one form of input, judges consider any and all additional team information gathered throughout the day.
Team achievements are reviewed and contrasted as the judges engage in an often intense discussion to decide which teams will be recognized with awards. Judges work together as a team to create an initial ranking of award candidates based on a team’s relative strengths and weaknesses compared to other teams and the award criteria as defined in the rubrics.
Once these initial rankings are complete, the judges enter the final phase of deliberations. As a first step, each Champion’s Award candidate is discussed and then a vote is taken by all judges to determine the winner. After the Champion’s award winner is known, the other awards are determined utilizing the rankings from initial deliberations following the distribution policy outlined in the next section.
Awards Distribution Policy
The goal of the FLL awards distribution process is to recognize (remember the “R” in FIRST®!) the whole group of teams who most deserve recognition at the tournament. With the exception of Robot Performance (which any team may receive based on the points they earn), teams may only win 1 Core Award per tournament.
Judges must consider how to recognize the best set of award candidates. Sometimes this means an award may not go to the individual team with the highest ranking in a category if that team is being recognized with another award. For example, if a team receives the Teamwork Award they will not receive the Presentation Award even if they were initially ranked highest for Presentation. They will receive the award in the area where they excelled the most.
The goal of the FLL deliberations process is to recognize the entire field of teams in the most appropriate way possible, to celebrate the achievements of all teams.
In addition to evaluating teams using the rubrics during scheduled interview sessions, judges may also use less formal conversations and observations throughout the event to learn more about teams.
Remind your team that judges may not openly demonstrate who they are, but their ears and eyes are wide open. Lending a helping hand to a team that forgot to bring something speaks loudly about your team’s understanding of Gracious Professionalism.
Judges will also consider any input provided throughout the day by referees, event Volunteers and others who interact with the team. Help your team members to understand the process and encourage them to feel comfortable speaking with judges and other event Volunteers.
Call-backs
When time allows, some events incorporate a second round of judging sessions referred to as “call-backs.” The length and format for these sessions is not prescribed and can vary significantly due to event and schedule constraints.
Call-backs may be requested for many reasons, ranging from a judge wanting additional clarification about a team, project, or robot to multiple judges needing to see closely ranked teams that may be in consideration for an award. Please be careful not to make assumptions about your team’s chances based on a call-back request. Many awards are decided without the need for them and a call-back never guarantees an award will be given. If your team is asked to participate in a call-back, be sure to find out whether or not the team needs to bring their project, robot, or other materials to the session.
Judging Subjectivity
With the exception of Robot Performance, which is objectively determined by scores earned on the competition table, team achievement in all other FLL award categories is subjectively judged. Even the most experienced and skilled judges will not assess every team in exactly the same way. We have judges work in pairs or small groups so they create more balance in the way they review teams. In addition, FLL tournament organizers work hard to train their judges and utilize other tools to create a level playing field for all teams.
FLL continues to improve tools and training for our judges, and the processes we have developed yield fair, strong, and consistent results. However, the fact remains that judged awards are inherently subjective and this is important for all teams to understand. To account for this, awards are determined through a normalization process, which includes deliberations and discussions among the judges to help reduce the effect of subjectivity as much as possible.
FLL Deliberations
Led by the Judge Advisor, FLL deliberations rely on an in-depth discussion of all teams eligible for awards. Using observations and evaluations captured by the rubrics as one form of input, judges consider any and all additional team information gathered throughout the day.
Team achievements are reviewed and contrasted as the judges engage in an often intense discussion to decide which teams will be recognized with awards. Judges work together as a team to create an initial ranking of award candidates based on a team’s relative strengths and weaknesses compared to other teams and the award criteria as defined in the rubrics.
Once these initial rankings are complete, the judges enter the final phase of deliberations. As a first step, each Champion’s Award candidate is discussed and then a vote is taken by all judges to determine the winner. After the Champion’s award winner is known, the other awards are determined utilizing the rankings from initial deliberations following the distribution policy outlined in the next section.
Awards Distribution Policy
The goal of the FLL awards distribution process is to recognize (remember the “R” in FIRST®!) the whole group of teams who most deserve recognition at the tournament. With the exception of Robot Performance (which any team may receive based on the points they earn), teams may only win 1 Core Award per tournament.
Judges must consider how to recognize the best set of award candidates. Sometimes this means an award may not go to the individual team with the highest ranking in a category if that team is being recognized with another award. For example, if a team receives the Teamwork Award they will not receive the Presentation Award even if they were initially ranked highest for Presentation. They will receive the award in the area where they excelled the most.
The goal of the FLL deliberations process is to recognize the entire field of teams in the most appropriate way possible, to celebrate the achievements of all teams.
