Copyright © 2013 Pearson Education, Inc. publishing as Prentice Hall Chapter 9 Evaluation of Training Copyright © 2013 Pearson Education, Inc. publishing as Prentice Hall Evaluation = 2 Types ➢ Process Evaluation ➢ Outcome Evaluation Copyright © 2013 Pearson Education, Inc. publishing as Prentice Hall Purpose of Process Evaluation ➢Identify areas where training process can be improved ➢Improved process leads to improved outcomes Copyright © 2013 Pearson Education, Inc. publishing as Prentice Hall Types of Questions to answer in a Process Analysis (Before Implementation) ➢ Were needs diagnosed correctly? ➢ Were needs translated into training objectives? ➢ Was an evaluation system designed to measure objectives? ➢ Was the training program designed to meet all the training objectives? ➢ Are the training methods appropriate for the learning objectives? ➢ Were the guidelines developed in the design phase used to develop the training content, materials and processes? Copyright © 2013 Pearson Education, Inc. publishing as Prentice Hall Types of Questions to answer in a Process Analysis (During Implementation) ➢ Was the trainer and techniques well matched to the objectives? ➢ Were lecture portions of the training effective? ▪ Was involvement encouraged/solicited? ▪ Were questions used effectively? ➢ Did the trainer conduct the various training methodologies (case study, role-play, etc.) appropriately? ▪ Was enough time allotted and was it used as intended? ▪ Did trainees follow instructions? ▪ Was there effective debriefing following exercises? ▪ Was time allowed for questions? Copyright © 2013 Pearson Education, Inc. publishing as Prentice Hall Who Is Interested in the Process Data Training Department Trainer: Yes, to determine what works well and what does not. Other trainers: Yes, to the extent that process is generalizable Training manager Only if training is not successful or a problem is shown for a particular trainer. Customers Trainees: No Trainees’ supervisor No Upper management: No Copyright © 2013 Pearson Education, Inc. publishing as Prentice Hall Who Is Interested in the Outcome Data Outcome Data Reaction Learning Behavior Results Yes Yes Yes No Perhaps Perhaps Perhaps No Yes Yes Yes Yes Yes Yes Yes Perhaps Only if no transfer Yes Yes No Perhaps Yes Training Department Trainer Other Trainers Training Manager Customers Trainees Trainees’ Supervisor Not really Management No Copyright © 2013 Pearson Education, Inc. publishing as Prentice Hall Outcome Evaluation What do you evaluate at the end of training??? Your Training Objectives Copyright © 2013 Pearson Education, Inc. publishing as Prentice Hall Outcome Evaluation The KEY outcomes achieved by your training Trainee Reactions Learning Transfer to the job Organizational Results The same as your objectives ☺ Copyright © 2013 Pearson Education, Inc. publishing as Prentice Hall Outcome Evaluation Design Alignment Between TNA and Outcomes TNA Training Objectives Measured the same way Copyright © 2013 Pearson Education, Inc. publishing as Prentice Hall Evaluation Outcomes Design Determines Evaluation Design Phase Activities Develop Training Objectives Evaluation Plan & Design Copyright © 2013 Pearson Education, Inc. publishing as Prentice Hall Evaluation Phase Reactions Learning Performance • Person • Unit • Org. Reaction Questionnaires Training reaction questionnaires do not assess learning, but DO assess the trainee’s attitudes about and perception of the training. Copyright © 2013 Pearson Education, Inc. publishing as Prentice Hall Reaction Questionnaires continued At a minimum, the following categories should be assessed in a reaction questionnaire: ➢Relevance/value of training (to trainees) ➢Training content and materials ➢Trainer’s behavior/effectiveness ➢Facilities, activities and procedures Copyright © 2013 Pearson Education, Inc. publishing as Prentice Hall Reaction Questionnaires continued It should also provide an overall evaluation of the training experience. Copyright © 2013 Pearson Education, Inc. publishing as Prentice Hall Steps to Consider in Developing a Reaction Questionnaire – Part 1 of 2 1. Determine what you want to find out (consider training objectives). 2. Develop a written set of questions to obtain the information. 3. Develop a scale to quantify respondents’ data. 4. Make forms anonymous so participants will feel free to respond honestly. 5. Ask for subgroup information (e.g., young vs. old; minority vs. nonminority). • This could be valuable in determining effectiveness of training for different cultures, for example, which may be lost in an overall assessment. Copyright © 2013 Pearson Education, Inc. publishing as Prentice Hall Steps to Consider in Developing a Reaction Questionnaire – Part 2 of 2 1. Allow space for “Additional Comments” in order to allow participants the opportunity to mention things you might not have considered. 2. Decide the best time to give the questionnaire to get the information you want. ▪ If right after training, ask someone other than the instructor to administer and pick up the information. ▪ If some time later, develop a mechanism to obtain a high response rate (e.g., encourage the supervisor to allow trainees to complete the questionnaire on company time). Copyright © 2013 Pearson Education, Inc. publishing as Prentice Hall Types of Outcomes and Examples of Factors Affecting those Outcomes Organizational Results Job Behavior External environment of the organization Internal environment of the organization Employee performance, KSAs, and needs Training’s transfer of training effectiveness Motivational forces in the job setting Opportunity to apply learned KSAs on the job Learning Trainee readiness for the course Trainee motivation to learn Design, materials, and content Reactions Trainee characteristics Trainer behavior Copyright © 2013 Pearson Education, Inc. publishing as Prentice Hall Examples of assessing objectives ➢Training objectives are set in the Design Phase based on “Gaps” identified. ➢Training objectives are assessed in the Evaluation Phase Let’s look at some examples of how this is done for reaction objectives. Copyright © 2013 Pearson Education, Inc. publishing as Prentice Hall Reaction Objectives -example At least 75% of participants will rate the following aspects of training a 4 or higher on a 5 point scale (with 5 being the highest rating): 1. Quality of the training content 2. Relevance of the training content 3. Effectiveness of each trainer 4. Overall effectiveness Copyright © 2013 Pearson Education, Inc. publishing as Prentice Hall Table of Reaction Results Training Content Strongly Disagree - Strongly Agree 2 0% 0% 13% 40% 33% 73% Relevance 0% 0% 13% 53% 33% 86% 0% 27% 47% 27% 74% Quality Overall Rating 0% 3 4 5 Rating of Ave % 4 or 5 1 Copyright © 2013 Pearson Education, Inc. publishing as Prentice Hall 78% Trainer Overall Effectiveness Ratings 1 = Low 2 3 5 = High 4 5 Rating of Average 4 or 5 Rating Trainer 1 A 0% 0% 27% 41% 32% 73% 4.05 B 0% 13% 19% 36% 32% 68% 3.87 C 0% 0% 100% 4.60 0% 40% 60% Copyright © 2013 Pearson Education, Inc. publishing as Prentice Hall Financial Evaluation ➢ Actual savings based on outcomes Cost of performance prior to training - Cost of performance after training = Performance savings - Cost of Training = Cost Savings ➢ Return on Investment ratio (ROI) = Cost Savings/Cost of Training Copyright © 2013 Pearson Education, Inc. publishing as Prentice Hall Cost Savings for Grievance Reduction Training Pretraining Post training Management time 10 hours/grievance 10 hrs. X 63 grievances = 630 hrs. 10 hrs. X 8 grievances = 80 hrs. Union Rep’s time 7.5 hrs/grievance 7.5 X 63 grievances = 472 ½ hrs 7.5 X 8 grievances = 60 hrs. Total Cost Management time Union rep’s time Total 630 hrs X $50 = $31,500.00 472 ½ hrs X $.= $11,812.50 $43,312.50 80 hrs. X $50 = $4,000.00 60 hrs. X $25 = $1,500.00 $5,500.00 Reduction in cost of grievances: 43,312.50 – 5,500.00 = $37,812.50 Cost of training: -32,430.00 Cost saving for the first year $ 5,382.50 Copyright © 2013 Pearson Education, Inc. publishing as Prentice Hall CHAPTER 9 CASE QUESTIONS Case 1 – Tricky Nicky Tips: This is a relatively straight forward assessment of the financial value of the training provided to the carpet cleaners. Questions 1-4 take the student step by step through the assessment. Question 5 gets the students to focus on the factors that affect performance, other than KSA deficiencies. 1. How much does the re-cleaning cost Nicky per year? Show all mathematical calculations. Number of offices cleaned per year = 6/day X 100 people X 250 days = 150,000 “Bad” cleanings = 1/6, so 150,000/6 = 25,000 “bad” cleanings/year Cost per “bad” cleaning = $20.00 Yearly cost = 25,000 X $20.00 = $ 500,000 Since the training program will reduce bad cleanings to 1 in 12, it will save Nicky $250,000 a year if everyone is trained. (If you wanted to figure in “opportunity” costs as well, the total cost of bad cleanings is $40.00. The additional $20 in cost is because time spent re-cleaning means time not spent cleaning and earning a $20.00 profit. If opportunity cost is built into subsequent calculations, it will be based on a $40 cost rather than the $20 figure we use.) 2. If everyone is trained, how much will the training cost? How much will it cost if only the group with the most errors is trained? Show all mathematical calculations. The costs associated with training that are listed on page 390 of the text add up to $17,900.00. This is the cost if everyone is trained (i.e., five sessions of 20 employees each). As stated in the case, these costs are based on conducting five sessions. Thus, if only the 50 with the most errors are trained, only three training sessions will be needed (no more than 20 employees/session). The value of this question is to determine if the students are able to see that some costs will be the same (development), some will be reduced by half (e.g., cost of employee wages while at training), and some will only be reduced by 40 percent (per session costs). If only half of the cleaners were trained (those averaging two bad cleanings out of every nine), the $4,000 in development costs would remain the same. Trainer cost would be reduced to $240.00 because three sessions would be required instead of five. The reduction in sessions also reduces the cost of the facility and equipment to $300.00. For factors associated with number of employees trained, the cost is cut in half. Thus, the cost of materials is $1,000.00, refreshments are $300.00, and employee costs would be $4,000.00. Indirect costs (evaluation) would also be cut by 50 percent to $1,200. The rationale is that, for the most part, these costs are based on the number of people being evaluated. Since most of the evaluation measures are developed in the TNA stage, the vast majority of trainer time is spent in collecting and analyzing post training data. Thus, total costs for training only 50 employees would be $11,040.00. Different answers could occur based on different assumptions in the area of indirect costs. 3. If everyone is trained, what is the cost savings for the first year? If only the group with the highest recleaning requirements is trained, what is the cost savings? Show all mathematical calculations. Since the training will reduce the number of “bad cleanings” by 50 percent, the yearly cost of “bad cleanings” after training = ½ cost prior to training ($500,000) = $250,000. Thus, the cost savings is $250,000 - $17,900 (cost of training) = $232,100.00. The group averaging 2/9 re-cleanings generates 2/3 of the re-cleanings, costing the company $333,333/yr. If this group reduces its re-cleaning to 1/12 it will still cost the company $125,000/yr. in “bad cleanings.” The gross savings will be $333,333-$125,000= $208,333/yr. The cost savings is $208,333-$11,040 (the cost of training) = $197,293. 4. What is your recommendation based on the expected return on investment? Should both groups be trained or just the group with the 2/9 recleaning ratio? Provide a rationale that includes both financial and other factors. Show any mathematical calculations. Since the gross savings if everyone is trained is $250,000, this means that training the group averaging 1/9 recleanings will result in a gross savings of $41,667.00 (i.e., $250,000$208,333=$41,667). The cost of training only the 2/9 group is $11,040.00. The additional cost to train the 1/9 group is $6,860. The percent return on investment for the 2/9 group is [($208,333-$11,040)/$11,040] x 100, or a 1787.07 percent. For the 1/9 group the return on investment is [($41,667 $6,860)/$6860] x 100, or 507.39 percent. Thus, from a financial perspective, both groups should be trained, even though one’s ROI is considerably higher than the other. From an employee relations standpoint, training both groups has the advantage of giving both groups the increased KSAs and equal treatment. 5. Let’s back up and assume we’re still at the needs analysis stage. Assume that employees had the KSAs needed to clean the offices effectively. What other factors might you look at as potential causes of the recleaning problem? Motivational factors such as pay, working conditions, group norms, etc. Opportunity factors such as equipment, procedures, etc. EXERCISES 1. Examine the reaction questionnaire your school uses. Is it designed to rate the course content and/or to rate instructors? Does it meet the requirements of a sound reaction questionnaire? Why or why not? Explain how you would improve it (if possible). Tips: Explain to students that the “reaction questionnaire” is commonly known as “class evaluations” in academic settings. It might be useful to have a copy of the reaction questionnaire or class evaluation form handy. Many schools do not have very good reaction questionnaires. This will make for some interesting discussions. Even if you have a good one, chances are that students will have some suggestions for improvement. You should also be prepared to answer questions about the use of the reaction questionnaire by the administration and/or different faculties, as this is a usual outcome of this exercise or you could add that assignment to the exercise (see what different faculties use them for, as well as the administration of the university). 2. Break into small groups, in which at least one member has received some type of training in an organization. Interview that person on what the training was designed to teach and how it was evaluated. Did the evaluation cover all the levels of outcomes? How did the trainee feel about that? Devise your own methods for evaluating each of the levels based on the person’s description of the training. Tips: This can be a long exercise and can be shortened by simply ignoring the last question. The advantage to this exercise is it provides a good framework about what is going on in organizations regarding training evaluation and this is a good backdrop to discussion about this issue. Generally students report minimal evaluation of their training. This can be a segue into the discussion on the arguments against doing an evaluation and the counter arguments for not doing one. 3. Go to the role-play for active listening in the Fabrics Inc. example. In groups of five or six choose someone to be the initiator and someone to be the trainee. Have them go through the role-play while the rest evaluate the trainee’s response on a one to seven scale (1 being poor, and 7 being excellent). Now share your scores. Were they all exactly the same? If not how could you make the instrument more reliable? If they were all the same, why was that? Is there anything you would suggest to make the evaluation process easier? Tips: This helps students understand the difficulty of the evaluation process for behavioral measures. It is likely that not all scores were the same, and refinement of the instrument can take many directions, such as more examples and longer, clearer definitions. It is useful to connect back to the idea of developing learning objectives and the kind of language used to describe objectives. Point out that similar language should be used in evaluation processes, to make outcomes more observable and measurable. If they are the same it is likely that the response was very bad or very good. Another issue that should come up is training in how to do it with multiple examples for practice. QUESTIONS FOR REVIEW 1. What is the relationship among the four levels of evaluation? Would you argue for examining all four levels if your boss suggested you should look only at the last one (results) and that if it improved, you would know that training had an impact? Kirkpatrick’s four levels of evaluation are ordered. The four outcome measures are reaction, learning, behavior and organizational results, in that order. Reaction comes first and will influence how much can be learned; learning outcomes determine how much behavior can change back on the job, behavior on the job determines how much organizational impact the training can have, and organizational results are the changes in the bottom line related to the reason for the training in the first place. If you only look at the last level (organizational results) and there was not any impact you will not know if it was the failure of the training or some other cause. Similarly, if there is a positive impact on organizational results and you do not evaluate at any other level (than at the organizational results level), you cannot be sure it was the training that caused the change. 2. What is the difference between cost/benefit analysis and cost/effectiveness analysis? Which would you use and why? Cost-benefit evaluation compares the monetary costs of training to non-monetary benefits such as attitudes and relationships. Cost-effectiveness evaluation compares the monetary costs of training to the financial benefits accrued from training. Both are important, however, strategic decision makers are interested in seeing that HR and HRD efforts lead to monetary gains when competing for funds; therefore, cost-effectiveness evaluation should be used as much as possible when proposing a training program. 3. What is the difference between cost-effectiveness analysis and utility analysis? When, if ever, would you use utility rather than cost effectiveness? Why? Note to professor: An error in the typing of this question occurred. It should have read: What is the difference between cost-savings analysis and utility analysis? When, if ever, would you use utility rather than cost savings? Why? You should make this correction before assigning this question to your students. In the text example it is shown that utility analysis allows us to estimate the increased results of training whereas cost-savings evaluation calculates the cost savings of the training. The research is not definitive about whether becoming more quantitative in the assessment and description of training outcomes leads to more acceptance of training as a unit that adds value to the bottom line. The textbook indicates that utility analysis can put the training manager on equal footing with the other mangers in the organization. However, this will depend on the culture and values of the organization. 4. Assume you were the training manager in the Westcan case (in chapter four). How would you suggest evaluating the training, assuming they were about to conduct it as suggested in the case? Be as specific as you can. As the training was to focus on skills development (how to run an effective meeting) the focus needs to be on behavior. Data from the case suggests they have the knowledge. Sitting in on some meetings and documenting what is actually done would provide a baseline for assessing skill development after training. Once this baseline was obtained, the test of learning could be a “mock meeting.” At this meeting the team could have an agenda of their choosing (hold an actual meeting) and you would have designed into the meeting specific problems to determine how the team deals with them. This could be done by having a “confederate” be a part of the meeting and interject specific problems at certain times to see how they are handled by the team (e.g. interrupting, changing the subject, pretending to not pay attention, etc.). The exact problems would be a function of what was actually trained. To assess “behavioral” change you could have assessors attend actual meetings with the similar rating scale they used for assessing learning and in this way assess the transfer. If transfer occurred, assessment of results could be done a number of ways. Results measures are often a reflection of what was the reason for the training in the first place. In the case it was the boss, but when he told managers about it they were also eager to have the training. So, one method of “”results” could be manager satisfaction with their meetings or the whole teams satisfaction with the meetings. If satisfaction measures were obtained pre training and again some time interval after the transfer had occurred, hopefully the satisfaction with meetings measure would be higher. Other important results could be “effective use of time in meetings” on particular topics (measured pre and post training). Better attendance at more effective meetings is likely, so attendance might also be an important measure. 5. Of all the designs presented in the appendix, which one would you consider most effective while also being practical enough to convince an organization to adopt it? If your design involved random assignment, how would you accomplish this? If your design included representative sampling, how would you accomplish it? There is no real right or wrong answer to this question. Students must demonstrate they know the internal and external validity issues associated with the design they select. Students must also demonstrate an understanding of the term representative sampling and the issues associated with it.
Purchase answer to see full attachment