Geeking Out: Evaluating Job Performance

When evaluating job performance, one of the most important challenges organizations face is establishing that the performance measure used is truly meaningful. An organization must have confidence that the selected performance measure accurately differentiates varying levels of performance so that related decisions may be made fairly. Therefore, it is highly important to demonstrate that a given performance measure (e.g., performance appraisal form) reflects an accurate picture of what it takes to be successful in a given job. In other words, you must show that your performance measure is valid.

What is validity?

According to the Society for Industrial and Organizational Psychology, validity is the degree to which the inferences made upon a measure are justified or supported by evidence¹. To paraphrase, validity tells us whether or not a tool measures what it is supposed to measure. For example, let’s consider the use of a specific form to appraise a manager’s job performance. A valid performance appraisal form for the manager should accurately assess the actual job performance of this manager. Extraneous behaviors, tasks, skills, etc. that are not important to successful job performance for a manager should not be included as part of the performance appraisal form. Validity evidence lets us know when we are measuring the right things.

How can validity be established?

Technically speaking, it is not the performance appraisal form itself that is valid.  It is the inference or inferences that are made from the form that are valid. For example, a performance appraisal form may be shown to be a valid indicator of the job performance of a retail manager. However, this very same form may not be a valid indicator of the job performance of firemen. It is the specific inference made concerning the form that is valid. However, for the sake of the present paper, let’s not quibble over technical semantics. Simply keep in mind, when we say a performance appraisal form is valid, we are referring to the specific inferences made regarding that form. Now, how do we go about showing that the inferences made regarding a performance appraisal form are valid? We do so through the process of validation.

Validation is the method through which both the appropriateness and meaningfulness of inferences made about a measure (such as a performance appraisal form) can be estimated^1,2. The bottom-line is that a measure can be shown to be either valid or not^2,3. That’s it. There are no in-betweens, categories or degrees of validity. However, there are many approaches used to provide evidence of a measure’s validity.

Using a content validation approach to establish performance evaluation form appropriateness:

Demonstrating content validity involves ensuring that the content included on the measure of interest is representative of the actual content within the job domain in question. For example, a managerial performance appraisal form is demonstrated to include those behaviors, tasks, skills, etc. that are actually necessary to be performed by managers in the job setting. One can use a content validation approach to establish performance appraisal validity evidence. That is, simply match the content included on the appraisal form to the content of the actual job. Subject matter experts, direct supervisors, incumbents, job analysis results, job descriptions, templates of success, and other performance appraisal tools can all be effectively utilized to precisely determine the actual content of the job. Further, specifically identifying the content found to be most critical to effective job performance can be done through this manner. The result is a performance evaluation form developed so that its content is highly representative of the content of actual performance on the job. More importantly, we can now confidently say that we are measuring the “right things” and not just what is readily available or easy to assess at the time.

References:

¹ Society for Industrial and Organizational Psychology, Inc. (1987). Principles for the Validation and Use of Personnel Selection Procedures. (Third Edition) College Park, MD.

² Landy, F. J. (1986). Stamp collecting versus science: Validation as hypothesis testing. American Psychologist, 41(11), 1183-1192.

³ Standards for Educational and Psychological Testing (1999). Washington, D.C. American Educational Research Association (AERA), American Psychological Association (APA), National Council on Measurement in Education (NCME).