1.1. Returning Applicants and Readers

You will need to complete an application for the 2025 Scoring Season to be considered for upcoming Projects. You will be sent an invitation to reapply for the 2025 Scoring Season. It will include the information/link to the application process so that you can complete the questionnaire and be included in the pool of readers for the 2025 Scoring Season.

Measurement Incorporated’s HelpSpot Knowledge Books (https://measurementinc.helpspot.com/index.php?pg=kb) are your avenues for researching most questions you may have regarding the Reader/Evaluator position.

HelpSpot ticket submissions (https://measurementinc.helpspot.com/index.php?pg=request) are your avenues for communication if you cannot find the information you need.

To receive all correspondence needed during any phase of your experience with Measurement Incorporated, you should have these email addresses in your safe sender’s list. If you do not, you may miss important information, notifications, and updates.

• measinc.com
• mystaffingpro.com
• workbright.com
• hrservicesinc.com

Measurement Incorporated’s Application, Hiring, Training, and Scoring processes are unlike other companies that do similar work. We expect you to read and follow all the instructions and status updates provided to you for every scoring season you participate in as well as every project assignment you work on.

1.2. Enhancements to Reader Monitoring

Measurement Incorporated employs automated enhancements to our Reader monitoring capabilities.  These enhancements are designed to provide you with quick feedback to help improve your performance on items where you may be having some difficulty.  You will continue to have access to a Team Leader to answer your scoring questions throughout the project; these enhancements are intended to be a supplement to the type of monitoring and feedback MI has provided in the past, and they will consist of responses for you to either score or review.

Because these enhancements are auto-generated by our scoring system (MIRA), some of our typical feedback tools will be disabled when you are asked to score and/or review these responses.  For instance, if you attempt to send a note to leadership on any of the auto-generated sets of responses, you will receive a message indicating that this option is not available.

The most prominent enhancement we have rolled out is an Automated Reader Recalibration set (ARR).  Each of these sets will contain up to 10 responses focused on a single item.  The first few responses in this set will have been recently scored by you. Your scores will differ from the scores given by the second Reader.  On these, you will see the two discrepant scores.  Please review the responses carefully, as they have been selected to assist you in the areas where you are having some difficulty agreeing on the score.

The last five responses in this set will have been scored independently by two other Readers whose scores agreed.  You will not have scored these responses yourself; our system will select them and send them to you as helpful examples to assist you in the areas where you are having some difficulty.  On these, you will see the final, agreed-upon score.  Please review these responses carefully.

Should you encounter similar responses in live scoring and have any questions about them, please send them to your leadership with a note.  Your leadership will be happy to discuss them with you.

NOTE: ARR sets will be generated automatically by the MIRA system and sent only to you.  Your Team Leader and Scoring Director do not have access to these sets and cannot answer questions about responses included in these sets.

MI hopes you will enjoy the opportunity to receive timely feedback designed to improve your performance on each item to which you are assigned.  We are confident that these enhancements will boost each Reader’s individual performance and, in so doing, improve the overall quality of our work.  We sincerely appreciate your cooperation and attentiveness as we continually strive to improve.

1.3. Reader Monitoring: My Accuracy Is Unacceptably Low

Measurement Incorporated (MI) uses several processes to monitor Reader accuracy and production.  Our system is constantly reviewing data for each Reader in real time, and each morning the system summarizes Reader performance and determines whether each Reader’s accuracy and production are above average, average, below average, or unacceptably low.  A Reader whose accuracy is determined to be ‘unacceptably low’ is encouraged to review their training materials and to send scoring questions to their leadership for feedback.  Additionally, a Reader may receive unsolicited feedback from their leadership team in an attempt to help improve their accuracy.  Our goal is to help all our Readers improve and to have a successful, rewarding experience while scoring for MI, and we use all tools at our disposal to assist anyone whom the system identifies as a potential beneficiary of targeted feedback and recalibration.

1.4. How MI Uses Validity Responses

At Measurement Incorporated, we use multiple metrics to evaluate scoring accuracy at the group, item, and individual reader levels. One important tool for collecting these metrics is the use of validity responses. An industry-standard tool for evaluating reader accuracy, validity responses have been a key part of MI’s scoring process since the 1980s.

Validity responses are pre-scored by leadership, managers, and/or MI’s automated scoring engine, Project Essay Grade (PEG), to determine the true score for each response. These responses are included in operational scoring sets and are indistinguishable from operational student responses, providing a reliable benchmark for evaluating your ability to apply the scoring criteria accurately and consistently.

Validity reports, used by scoring leadership and managers, reflect:

• Your overall agreement with validity true scores.
  
  
• Any scoring trends you exhibit, such as a consistent tendency to assign lower (severe) or higher (lenient) scores than warranted.
  
  
• The difficulty of each validity response. Some responses clearly represent a specific score point and are easier to score accurately, while others contain characteristics of multiple score points and are more challenging. This variation is intentional, as it reflects the range of responses seen in operational scoring.

By analyzing scoring patterns across a large pool of validity responses, MI provides tailored feedback to help you monitor your scoring quality and adjust any trends that may impact accuracy. If your validity results indicate inaccurate or biased scoring, you will receive targeted feedback—both automated and from scoring leadership—to help you refine your approach.

We encourage you to use this feedback to improve your scoring by:

• Reviewing your training materials.
  
  
• Submitting line-call responses to leadership for guidance.
  
  
• Making the necessary adjustments to align with scoring expectations.

Accurate and bias-free scoring is essential to maintaining the integrity of the assessment process and ensuring that every student receives the score they have earned. Your role as a reader has a direct impact on students and their educational outcomes. We appreciate your commitment to fairness, consistency, and high-quality scoring, and we are here to support you in achieving those goals.

Thank you for your dedication to this important work.

1.5. Smarter 2025 Beta Project FAQ

This year, in our efforts to help everyone on the project improve the quality and accuracy of their scoring, we will be implementing some new tools that will give you all a chance to hone your skills.  We hope this FAQ will address any questions you may have, but if it doesn’t, you can direct any additional questions to your scoring leadership. 

Q:  What is this “Beta” project?  

A:  The Beta project is a way for all Smarter Balanced readers to practice their scoring after training, but before beginning to score actual student responses. You will score a selection of practice responses in this partition, during which time you can refine your skills. Even though this partition is specifically for practice, your accuracy in the Beta project will have a direct impact on which items you will be able to score, and which you will not.  

Q: Will my Team Leader be available to help me while I practice?   

A:  Yes! Your Team Leader(s) will be available to you through chat and, after you’ve begun to accumulate some scoring time, they may send you feedback on responses you scored.

 
Q:  Can I send questions up using the Notes function? 

A:  Yes! However, as part of this process, you may not hear back from your Team Leader immediately.  Part of this process involves you spending several days scoring on your own so that we can get an idea of how you’re scoring without your Team Leader’s guidance. Then, after a few days, your Team Leader will begin responding to notes.  But your Team Leader will always be available through chat if you need them. 

Q: When will I start scoring actual student responses? 

A: Once you’ve had some time to practice, we will evaluate your accuracy.  If we determine that you are scoring accurately, you’ll receive an automated chat message that will instruct you when to begin scoring in the Operational Smarter project.  You should then return to your project selection screen and select the Operational project and begin work there.  

Q: Once I start working in the Operational project, will I ever go back to the Beta project? 

A:  There are some instances in which you might return to working in the Beta project. Those include: 
1. You may still have additional items to practice.  While you might be ready to score some items outside of Beta, you might still need practice on others. If you run out of work in the main project, you should return to Beta to practice your other items. 
2. You may be assigned additional items or trainings, at which point your scoring leadership will ask you to return to the Beta project in order to do your new training and/or practice your new item(s).  
If you’re ever not sure where you should be working, you can chat with your Team Leader for 
clarification. 

Q:  How long will I practice in the Beta project? 

A: That will depend on several things. The first thing is how many items you’re learning to score. If you have only a few items to practice, you won’t work in the Beta project as long as someone with lots of items to practice.  It will also depend on if there are items where you need additional practice to help refine your scoring. Don’t try to compare your time in the Beta project to others. It will be different for everyone.  

2.1. All Projects

Once you accept a specific project job offer, you will receive a projected start date and a projected end date. The start date will be confirmed via an official training assignment that has all information included to report to work for your project.

There may be times during a project when there is insufficient work available. Also, the final day of a project can be hard to predict until there are only a few days of work to complete. Your scoring system dashboard as well as other resources provide information about work availability and "Out of Work" messages. It is important to access these resources and follow the instructions provided.

Please see the following section 2.2, Reporting to Work After a Hiatus and System Announcements for specific updates on reporting to work based on your assignment.

NOTICE: Employees are only paid for time spent working in MIRA. You must be logged into MIRA to be considered actively working. There are rare exceptions to this, such as meetings for leadership. Exceptions such as this will be communicated to you by Scoring Leadership prior to or during the project. Inability to access MIRA during System Maintenance will not be considered as time spent working in MIRA.

2.2. Reporting to Work After a Hiatus and System Announcements

There may be insufficient work available. You can find additional information regarding Out of Work messages by checking in Tab 1 of the Resource Library located on your MIRA dashboard.

If you have been told not to report due to "out of work" messages or a notice that your group has been put on hold, please do not log in to score. This time is not authorized.

Check your project status from the list of projects below. If your project status indicates that you should check for work, do so only once at the beginning of your shift. Before beginning your next shift, check your assignment status below. Do not log in more than once a day to check work availability.

This page will confirm the day you are to report back, by project and assignment. We appreciate your flexibility!

__________________________________________________________________________________________________________________________

Beta Smarter (Full Writes, Brief Writes, Research)

3/28 Update

• Full Writes: There will be scoring for Full Writes in the Beta partition this weekend.

__________________________________________________________________________________________________________________________

ClearSight Rescore

2/24 Update

• Phase 1 of this project is wrapping up, and at this time, we do not need anyone to score anything else until we get a decision on Phase 2.
• Thank you all for your hard work and dedication to getting us to this point. We will be in touch with more information as we receive it.

__________________________________________________________________________________________________________________________

ELPA 25 Summative-Speaking 

• The following ELPA Summative-Speaking grades are currently out of responses:
  • Grade K
  • Grade 1
  • Grade 2-3
  • Grade 4-5
  • Grade 6-8
  • Grade 9-12
• Please stop scoring once you run out of work.
• Return dates vary: 
• Grade K Speaking's readers have run out of work for the ELPA Summative (ELPA 25 - Speaking OP 25).
  • Readers should stop scoring once you run out of work.
  • You will be contacted if more work becomes available.
• Grade 1 Speaking's readers have run out of work for the ELPA Summative (ELPA 25 - Speaking OP 25).
  • Readers should stop scoring once you run out of work.
  • You will be contacted if more work becomes available.  
• Grade 2-3 Speaking's readers have run out of work for the ELPA Summative (ELPA 25 - Speaking OP 25).
  • Readers should stop scoring once you run out of work. 
  • You will be contacted if more work becomes available.
• Grade 4-5 Speaking's readers have run out of work for the ELPA Summative (ELPA 25 - Speaking OP 25).
  • Readers should stop scoring once you run out of work. 
  • You will be contacted if more work becomes available.
• Grade 6-8 Speaking's readers have run out of work for the ELPA Summative (ELPA 25 - Speaking OP 25).
  • Readers should stop scoring once you run out of work. 
  • You will be contacted if more work becomes available.
• Grade 9-12 Speaking's readers have run out of work for the ELPA Summative (ELPA 25 - Speaking OP 25).
  • Readers should stop scoring once you run out of work. 
  • You will be contacted if more work becomes available.

__________________________________________________________________________________________________________________________

ELPA 25 Summative-Writing 

• The following ELPA Summative-Writing grades are currently out of responses:
  • Grade 2-3 
  • Grade 4-5
  • Grade 9-12
• Please stop scoring once you run out of work.
• Return dates vary:
• Grade 2-3 Writing's readers have run out of work for the ELPA Summative (ELPA 25 - Writing OP 25).
  • Readers should stop scoring once you run out of work.
  • You will be contacted if more work becomes available.
• Grade 4-5 Writing's readers have run out of work for the ELPA Summative (ELPA 25 - Writing OP 25).
  • Readers should stop scoring once you run out of work.
  • You will be contacted if more work becomes available.
• Grade 9-12 Writing's readers have run out of work for the ELPA Summative (ELPA 25 - Writing OP 25).
  • Readers should stop scoring once you run out of work.
  • You will be contacted if more work becomes available.   
• Grade 6-8 Writing's readers have run out of work for the ELPA Summative (ELPA 25 - Writing OP 25).
  •     Readers should stop scoring once they run out of work and will be contacted if more work becomes available. 

__________________________________________________________________________________________________________________________

ERB March Session

3/20 Update

• Scoring for the stimulus responses has concluded. Thank you for all your hard work and for making this a successful administration.
• Scoring for the non-stimulus (NS) responses has concluded. Thank you for all your hard work and for making this a successful administration.

__________________________________________________________________________________________________________________________

MCAS Spring Operational

3/27 Update

• ELA Readers: G10 Readers please check back on Monday morning, March 31, to start scoring. Thank you for your patience.

__________________________________________________________________________________________________________________________

MCAS Winter Retest 

3/27 Update

• Math Readers: Scoring for the project has concluded. Thank you all for your hard work and for making this a successful administration. 
• ELA Readers: Scoring for the project has concluded. Thank you all for your hard work and for making this a successful administration.
  

__________________________________________________________________________________________________________________________

MCAS Feb Science Retest 

2/24 Update

• The Biology project has concluded. Thank you for all your hard work and for making this a successful administration.
• The Physics project has concluded. Thank you for all your hard work and for making this a successful administration.

__________________________________________________________________________________________________________________________

New Mexico NM-MSSA

3/31 Update

• Writing: All Teams have NO WORK available and are currently ON HOLD.  We expect that there will be additional responses to score starting Thursday, 4/3. Please plan to return to work at that time.  
• Math: All Teams have NO WORK available and are currently ON HOLD.  We expect that there will be additional responses to score starting Thursday, 4/3. Please plan to return to work at that time.   

___________________________________________________________________________________________

Smarter Math 

3/24 Update

• Math Grades 9-11: If you receive an Out of Work message please wait 10 minutes and check for more responses; if there are still no responses please log out and check back again tomorrow. 

__________________________________________________________________________________________________________________________

All Projects

Employees are only paid for time spent working in MIRA. You must be logged in to MIRA to be considered actively working. There are rare exceptions to this, such as meetings for leadership. Exceptions such as this will be communicated to you by Scoring Leadership prior to or during the project. Inability to access MIRA during System Maintenance will not be considered as time spent working in MIRA.

__________________________________________________________________________________________________________________________

System Announcements

When system maintenance occurs, please follow this protocol:

• None at this time

2.3. Scoring System Status

When system maintenance occurs, please follow this protocol:

Nothing to report at this time.

Reminder: Employees are only paid for time spent working in MIRA. You must be logged in to MIRA to be considered actively working. There are rare exceptions to this, such as meetings for leadership. Exceptions such as this will be communicated to you by Scoring Leadership prior to or during the project. Inability to access MIRA during System Maintenance will not be considered as time spent working in MIRA.