Designing and Preparing to Implement an Evaluation
Using the text material from Russ-Eft and Preskill (2001) create a document reflecting an evaluation plan for a program of your choice that impacts learning, performance, and change interventions. You are instructed to utilize an imaginary workplace training program and follow-up evaluation regarding the program and learner outcomes for a population of 950 diverse employees. Please use the following information about the population: 50 employees are upper level managers, 150 are supervisors, and 750 are hourly workers. Refer to page 131 and 133 of Russ-Eft and Preskill (2001) for examples that will help you develop your imaginary scenario. I recommend that you carefully review chapters 3 and 5-12 as you develop your evaluation plan for a program that impacts learner performance. Please ensure that you address each of the following criteria shown below within a single-spaced document, including your cover page. Paper length for undergraduates is six to seven pages; eight to ten pages for graduate students. When appropriate, cite scholarly sources to substantiate your writing. List each of the sources in a reference list at the end of the last text material in your paper.
> Focus the evaluation (chapter 5)
1. Develop an evaluation plan for a program utilized to improve learner performance.
2. Develop the evaluation’s rationale and purpose.
3. Identify the evaluation’s stakeholders.
4. Develop key evaluation questions that focus on the allocation of resources.
> Choose an evaluation design (chapter 6)
1. If possible, select an evaluation design that utilizes both quantitative and qualitative data.
2. Describe how you will ensure internal and external validity. This item will require several paragraphs.
> Choose a data collection method (chapter 7)
1. Describe the data collection methods that you considered or proposed in your plan and offer rationale for the method(s) you selected.
2. Offer an explanation for other data collection methods that you considered but did not utilize.
3. Remember, it is often appropriate and highly recommended that evaluators utilize multiple methods to collect data for an evaluation. See the possibilities in chapters 8-10 shown below:
a. Observation and archival data (chapter 8).
b. Surveys and Questionnaires (chapter 9).
c. Individual and Focus Group Interviews (Chapter 10).
> Drawing a sample (Chapter 12)
1. Inform the reader of your Evaluation Plan why you are drawing a sample (if you propose to sample).
2. Offer rationale for why you drew a random or nonrandom sample or elected not to sample.
3. Explain your method for selecting either individuals or groups in a sample (if appropriate).
4. Clarify the desired level of representativeness and the size of the sample for your population (if appropriate).
> Analyzing evaluation data (Chapter 13)
1. Based on your key evaluation questions, and your perception of the level of analytic sophistication of your stakeholders, state your proposed approach for quantitative and qualitative data analysis. In other words, for quantitative data will you use frequencies, percentages, mean, mode, median, and standard deviation or will you utilize a more sophisticated approach to data analysis such as Analysis of Variance (ANOVA), Correlations, or perhaps Multiple Regression. For qualitative data analysis will you use “content analysis” or “thematic analysis” or some other method? Be sure to clarify your method for data analysis in a paragraph or two. Note: you may want to create a table showing how you intend to analyze the data.
Continue to page 2 for the final criterion…
> Calculating Estimating/Forecasting return on investment (see examples below).
1. Describe how you plan to conduct an economic analysis of the learning, performance, and change intervention (the program you selected). That is, how will you determine the costs of the program versus the net benefits and/or return on investment.
Please note the following example for a three-course sales training program conducted by an external consultant. The development costs for the course were assumed by the consultant.
Annualize Program Benefits
Average weekly sales for trained groups $12, 075
Average weekly sales for untrained groups 10, 449
Profit contribution 2% 32.50
Total weekly improvement (x46*) 1,495
Total Annual Benefits (x 48 weeks) $71,760
*Forty-six participants were still in job after 3 months
Facilitation Fees: 3 courses @ $3,750 $11,250
Program Materials: 48 @ $35/participant 1,680
Meals/Refreshments: 3 days @ $28/participant/day 4,032
Facilities: 9 days @ $120 1,080
Participant salaries plus benefits 12,442
Total Costs $32,984
Cost Benefit Ratio (CBR) = $71,760 = 2.2:1
Return on Investment (%):
(ROI) = $71,760-$32,984 x 100 = 118%
> Final instructions: Ensure that you have addressed each category and criterion shown in this document. Use headings (in bold) such as Rationale and Purpose of the Evaluation Plan or Stakeholders, or Evaluation Questions to inform the reader and to create a reader-friendly document. This also will help you determine if you have addressed each of the criteria. Finally, proof your paper carefully and submit on or prior to the deadline. You may consult/work with one other student on your paper but you MUST NOT submit papers that are similar or duplicate in manner. The latter will constitute a violation of ISU Academic Integrity Policy.