Here is my response to the required post where I needed to consider how I would combine the statistical scores.
Post by Day 4, an explanation of how you might combine multiple raters’ evaluations of an individual on a measure of job performance. Provide a specific example of this use. Then explain the psychometric implications of using multiple raters. Finally, explain steps you could take to improve the reliability of a multi-rater assessment. Support your response using the Learning Resources and the current literature.
Combine Multiple Raters’ Evaluations of an Individual on a Measure of Job Performance and Example.
About 34% of all organizations are using multiple-rater feedback evaluations with their employees (Aamodt, 2016). Many are using multi-rater evaluations to help develop their leaders (Van Velsor, Leslie, & Fleenor, 1997). Multi-rater evaluations have its benefits in that you get input from different sources (Lauby, 2016). Instead of having one point of view, the individual has feedback from many people that can provide varying insights based on their experiences with that person (Lauby, 2016). One example of a combined approach would be one that a past employer used. As a global insurance company that had a four-year leadership development program, they invested in their leaders. Year one was called Management Essentials, and year two was Avanza. Year three of the program was called the Management Development Program (MDP), and year four was the International Management Development Program. Everyone wanted to make it to year four since you got to go to Spain for two weeks of training. Each of these programs required a year-long commitment and two weeks of residency. During MDP, a combined self-report DiSC and multi-rater 360 evaluation were done. The assessments were completed in parts. Part one was a DiSC assessment, and part two was a 360-degree review. This allowed for the individual to self-assess their personality type and get feedback on how others saw them. The goal was to have the leaders make personalized development plans to work on their strengths and weaknesses and understanding how they like to work with others. DiSC (n.d.) helped the leaders identify their personality type, which could be a combination of the four categories:
Dominance - Person places emphasis on accomplishing results, the bottom line, confidence
Influence - Person places emphasis on influencing or persuading others, openness, relationships
Steadiness - Person places emphasis on cooperation, sincerity, dependability
Conscientiousness - Person places emphasis on quality and accuracy, expertise, competency
Moreover, it aligned this information with behaviors. For example, I am a Di, meaning I am a high dominant and influencer. Some of the behaviors associated with this are seeing the big picture, can be blunt, accepts challenges, gets straight to the point, shows enthusiasm, is optimistic and likes to collaborate.
McMahon (2019) defined psychometrics as a "field of study devoted to tests and measurements." Psychometric tests like the ones we discussed above are used to evaluate an individual's capabilities (McMahon, 2019). Some implications of psychometrics are the wether they are single rater or multi-rater assessments. McCrae (1994) stated that test scores could only be considered or understood when the trait and the method are taken into consideration. So, in other words, psychometric implications can be the trait and the method used to evaluate that trait. An important consideration is that measuring traits can not be done via self-reports (McCrae, 1994).
Improving the Reliability of a Multi-rater Assessment
Reliability is the degree a test score or evolution is free of errors and is consistent (Aamodt, 2016). In multi-rater assessments, reliability can be reached when raters are familiar with what is being measured and aggregated (McCrae, 1994). When considering raters, we should look at their familiarity with the person being rated and the role that they fill. How often do they interact with the person being rated? This is why it is recommended that the person being rated ask people that work with them regularly and know them to give them feedback. Another consideration may be time. If I have not worked with a person in the last year, I would not ask them to evaluate me. In a study by Rothstein (1990), they found that reliability is changed with time. The increase in reliability by having time to observe will improve the accuracy of the ratings given (Rothstein, 1990).
Aamodt, M. (2016). Industrial/Organizational Psychology: An Applied Approach.Eighth Edition. Belmont, CA: Wadsworth, ©2010.
DiSC Overview. (n.d.). Retrieved from https://www.discprofile.com/what-is-disc/overview/
Lauby, S. (2016). Multi-Rater Reviews: Advantages and Disadvantages. The SHRM Blog. Retrieved from https://blog.shrm.org/blog/multi-rater-reviews-adv...
McCrae, R. R. (1994). The counterpoint of personality assessment: Self-reports and observer ratings. Assessment, 1(2), 159–172.
McMahon, M. (2019). Psychometrics. Salem Press Encyclopedia. Retrieved from https://eds-a-ebscohost-com.ezp.waldenulibrary.org...
Rothstein, H. R. (1990). Interrater reliability of job performance ratings: Growth to asymptote level with increasing opportunity to observe. Journal of Applied Psychology, 75(3), 322–327.
Retrieved from the Walden Library databases.
Van Velsor, E., Leslie, J. B., & Fleenor, J. (1997). Choosing 360: A guide to evaluating multi-rater feedback instruments for management development. Center for Creative Leadership.