Is my Assessment Questionnaire (AQ) appropriate for use with a passing score?
You can potentially use an existing AQ or develop a new one prior to working through the process of determining what the passing score should be. In any case, it is very important to have a valid assessment that is developed based on job analysis in coordination with subject matter experts and that measures job-related competencies required at entry into the position. You will want to consider the type of rating scales (e.g., default vs. custom, multiple choice vs. select-all that-apply) and scoring protocol you are using. The more complex the scoring (e.g., with regard to weighting and response option values), the less straightforward the process for establishing a passing score. Also consider how many items are in your questionnaire (a general rule of thumb is a minimum of 3-8 items per competency if using competency-level scoring or 10-40 total items if using item-level or response-level scoring).
Can a given AQ passing score be used across positions?
A given AQ passing score is tied to a specific AQ. A passing score is based on the minimum proficiency required to be minimally competent in a given position or occupational series and grade. To the extent that the AQs and minimum required proficiency are similar across other positions or occupational series and grades, the passing scores may be similar. That said, agencies will want to think carefully about the implications of setting an AQ passing score on one position or series/grade in relation to other grades or similar positions in the organization.
If my agency uses an AQ with a passing score for a position that is repeatedly announced or includes the AQ in an Assessment Package Template, how often should we update the passing score once established?
If you modify the AQ at all (i.e., add, delete, or modify rating item content), the passing score should be re-evaluated. Also, if the position changes over time, it is necessary to re-evaluate the passing score as minimum proficiency requirements may change. Even if the position has not changed, it is recommended to re-evaluate the passing score on a regular schedule (e.g., every 3 - 5 years).
How is an AQ passing score set up in USA Staffing?
The Passing Score option is available only for AQs using the Weight-Based rating method. The ability to set/apply a passing score is permissioned and defaulted to disabled for all USA Staffing users except Office Administrators. HR users with appropriate permissions can set up an AQ passing score by:
- Turning on the Passing Score setting on the Assessment Information page, and then
- Setting the Passing Score Criteria based on the overall raw score that applicants receive on the competencies being assessed. The passing score can be set at different values for different rating combinations (e.g., GS-12 vs. GS-13).
Users can upload documentation supporting the development of the AQ passing score in the Documents section of USA Staffing. HR users are responsible for testing their AQ using the Test Plan scenarios within USA Staffing to confirm the actual outcomes associated with possible applicant response patterns will match expected outcomes.
Can I add/remove a Passing Score to an AQ after the announcement is open?
The passing score functionality cannot be turned on for an AQ once the announcement is released and opened. In addition, the passing score cannot be modified once the announcement is released. However, if an error is identified with the passing score criteria, an HR user with appropriate permissions can un-complete the Assessment Package, and then turn off the Passing Score setting for that AQ. Following these steps results in the minimum raw score reverting to 0, in effect removing the passing score requirement. The AQ then needs to be re-tested. Once the revised Assessment Package is completed, the system re-rates all applicants using the new scoring approach. This type of action will be tracked automatically in the AQ History. The HR user is responsible for documenting the rationale for removing the passing score through the Assessment Package notes and/or attached documents.
What if we discover an error with a particular question (poorly written, applicants had a hard time understanding it, mis-keyed, etc.) for an AQ with a passing score, when and how can we fix it?
If the error is discovered before the announcement is posted to USAJOBS, the item can be rewritten or removed. If it is removed, the passing score criteria should be re-evaluated and adjusted as needed.
If the error is discovered after the announcement has been released and opened on USAJOBS, the passing score criteria cannot be adjusted at that point. One course of action would be to update the Rating Criteria for the item to give all response options the same maximum score for the item (e.g., if answer E has the highest response value with 5 points, then update the rating criteria to assign the same 5-point value to each of the A through D response options as well). This would ensure that regardless of which response they selected, applicants will receive the maximum number of points for that item.
An alternate course of action to consider, particularly if multiple questions have errors, is whether removing the passing score requirement may be a more appropriate course of action, as a passing score based on minimum required proficiency could lose its meaning if applicants are being passed due to an HR error.
Will use of the AQ passing score functionality extend the time required for the assessment and hiring process?
Once set up in the system, application of the passing score is immediate. No additional time is needed. Rather, use of this functionality may reduce the size of the applicant pool and thus the time needed for the qualifications review. Deciding on the most appropriate assessment strategy for a given position or set of positions involves a host of considerations, including time required, the applicant pool, and the assessments used.
Do I need to take any additional steps to set up a job opportunity announcement (JOA) when there is an AQ passing score?
HR users will need to set up the Passing Score Criteria in the Assessment Questionnaire and upload appropriate documentation. HR users also will need to ensure appropriate verbiage about passing scores is in the JOA. Sample verbiage is below.
You will be evaluated based on how well you meet the qualifications listed in this vacancy announcement. Your qualifications will be evaluated based on your application materials (e.g., resume, supporting documents) and the responses you provide on the application questionnaire [add additional assessments as appropriate]. You will be assessed on the following competencies (knowledge, skills, abilities, and other characteristics): [add competencies]
The application questionnaire includes a passing score based on the minimum level of required proficiency across these critical competencies. You must meet or exceed the passing score to be considered. You will not be considered for the position if you score below the passing score.
How/when will applicants be notified whether they met the AQ passing score?
Applicants will not know how they scored and whether they met the AQ passing score until the HR user sends them a touchpoint notification with their Notice of Results (NOR). For most case examining recruitments, the HR user will send this notification after the announcement closes or will send a combined NOR/Notice of Referral after the qualification reviews are complete and the certificates are issued. However, for standing registers or open-continuous announcements, it may be standard practice to send NOR notifications at regular intervals throughout the announcement open period. If you are planning to send NORs for an AQ that uses a passing score while the announcement is still open, we recommend checking the Vacancy setting for “Prevent update of assessment responses after application submission” to ensure applicants cannot get a second chance to pass the assessment.
In what order is the AQ passing score applied relative to other steps in the assessment process?
The AQ passing score is applied after the self-evaluation of minimum qualifications but prior to an HR professional conducting a minimum qualifications review or applying any criteria for category rating or for competitive merit promotion. If USA Hire is being used, applicants who do not obtain the passing score on the AQ are not invited to complete the USA Hire assessments. Similarly, if a Manual Assessment is being used, applicants who do not meet the AQ passing score would be ineligible and therefore not move forward to the Manual Assessment.
Do I need to conduct a qualification review on applicants who were rated ineligible because they did not attain the AQ passing score?
No. Applicants rated ineligible because they did not attain the passing score on the AQ do not need any further review.
What Notice of Results (NOR) code will applicants receive if they do not attain the AQ passing score?
IQAQ - You are ineligible to proceed in the hiring process because you did not attain the overall passing score on the application questionnaire for the position.
Can an AQ that uses a passing score be combined with other assessments, including USA Hire assessments with a cut score?
Yes. AQs with a passing score requirement can be used independently or in combination with USA Hire (including USA Hire assessments with a cut score) and/or Manual Assessments. Scores are combined using the same or similar procedures as combining assessments without passing scores. The only restriction is for applicants who do not obtain the passing score on the AQ. These applicants are ineligible, so they would not be invited to complete a subsequent USA Hire battery and/or Manual Assessment. If you have questions about combining assessments, please contact your USA Staffing Account Manager.
Can I use the AQ passing score functionality with an ACWA assessment questionnaire?
The AQ passing score functionality cannot be applied on the ACWA assessments themselves because the scoring protocol for these assessments cannot be modified by HR users. However, agencies developing their own AQ using the Weight-Based rating method for a position covered by an ACWA questionnaire can set a passing score for it.
Can I use the AQ passing score functionality within USA Staffing with a wage grade position?
No. The AQ passing score functionality can be used only for AQs using the Weight-Based rating method. The Job Element rating method used to score AQs for wage grade positions already has passing score criteria set up ‘behind the scenes’ (i.e., as specified in the wage grade qualifications, all applicants must meet an average of two points across all job elements in order to be minimally qualified).