Week 4: Evaluation Design Introduction Why do you need evaluation design? Now that you have been introduced to evaluations and the use of logic models as a tool in evaluation, you may structure your evaluation in a way that will allow you to get the answers to what you and your stakeholders want to know. Perhaps you want to know whether your program is making a difference—whether it is causing an intended change in outcomes for its recipients, or whether it is changing physical and economic conditions in your target environment. You may also want to know if the change is sustained over time. Maybe you also want to know whether there were both intended and unintended consequences of the program to your beneficiaries or the target environment. If you want reliable answers to evaluation questions like these, you have to structure them in a way that will show you whether you actually got results, and whether those results were in fact due to your program or to other factors. In other words, you have to develop a design for your evaluation, to give you clear and reliable answers to your questions. This week, you will be introduced to validity threats and how to recognize these threats in different design scenarios. You will also start developing your own evaluation design. Learning Objectives Students will: Evaluate the validity of causal claimsAnalyze features of an evaluation designExplain implementation of evaluation designAnalyze validity threats with designs Week 4 Discussion Designs: Experimental Designs, Quasi Experiments, and Non-Experimental Designs Validity issues of causal claims are always at the forefront of evaluation design. It is critical for the evaluator to understand issues of internal, external, measurement, and statistical validity. Though the classical experiment remains the standard for evaluating statistical and causal rigor, many evaluations continue to use quasi experiments and non-experimental designs. For this Discussion, review this week’s Learning Resources and pick two of the following scenarios of causal claim: Field Experiment: Minimum wage legislation will lead to higher unemploymentCross Sectional Study: Capital punishment deters crimeTime Series Study: Privatization of government-owned industries will increase economic growthTime Series Study: Airline deregulation will result in lower airfares Post by Day 3 an evaluation of the most important validity issues of each causal claim chosen. Explain the variety of validity issues possible in each scenario and provide a rationale for what qualifies as “most important” to you. Your evaluation should also include identifying illustrative and supportive examples related to the scenarios, as well as relevant and credible references from your Learning Resources and Walden Library. Respond by Day 5 to two colleagues in the following way: In what ways do you agree or disagree that the validity issues in the scenarios of causal claim selected by your colleagues are the most important? Provide a rationale, including examples or references. Wee4 Assignment Initial Evaluation Design At the heart of any evaluation is the evaluation design. It is now common consensus in policy evaluation that no amount of sophisticated analysis can repair a poorly designed study. There is no such thing as a perfect evaluation design and there are always tradeoffs. One of the most important tradeoffs has to do with internal and external validity. This is true whether you are doing a quantitative, qualitative, or mixed methods evaluation. For this Assignment, review this week’s Learning Resources and begin the process of developing your evaluation design by outlining its main features. Submit by Day 7 a 2- to 3-page paper that addresses the following: Explain how you would implement your evaluation design within an official policy evaluation. Be sure to include specific examples from the course readings, academic research and professional experience. Provide a rationale for your implementation choices, including examples and references.Explain how you would address validity threats, particularly internal validity (plausible rival hypothesis). Provide a rationale for these plans, including any expected outcomes, using examples and references. Week 5 Measurement and Data Collection Data collection requires evaluators to consider a wide and diverse variety of data sources. These include both primary and secondary data sources. Data collection can significantly impact your evaluation project. Evaluators sometimes overlook problems where secondary data is involved and overlook the fact that data for comparison purposes can affect validity. The types of data in the Learning Resources are not meant to be exhaustive but are the most common types in program evaluation. To prepare for this Discussion, choose one of the cases provided, review the specified Learning Resources, and complete the corresponding prompts. Case 1 Steven Levitt, the author of Freakonomics, is exemplary in his ability to make data tell “stories.” Students of public administration and organizational behavior are familiar with the famous Hawthorne effect. Levitt and List’s (2009) reanalysis of the original Hawthorne study (which uses incomplete, archived, and secondary data) shows how management programs that were developed to increase worker productivity can be evaluated or reevaluated, even though the experiments were conducted almost a century ago. Post by Day 3 a description of at least two data collection problems that the reanalysis of Levitt and List might bring to the Hawthorne effect. Respond by Day 5 to two colleagues and explain whether the problems have to do with design or data collection. What other data collection problems could, in your view, also be important? Explain. Case 2 Barbara Geddes (1990) points out the pitfalls of selecting cases, units, or observations purely on the basis of the dependent variable. Post by Day 3 a brief explanation of the relevance of Barbara Geddes’ argument for program evaluation. Then, explain when a scenario of choosing cases solely on the dependent variable might be permissible. Provide a rationale for your explanations. Respond by Day 5 to two colleagues and explain in what ways you agree or disagree with the permissibility of the identified scenarios. Justify your reaction. Note: Within the Week 5 Forum, use the appropriate thread for your selected case: For Case 1, use the “Week 5 Discussion - Case 1” thread.For Case 2, use the “Week 5 Discussion - Case 2” thread. Week 5 Assignment Indicator Construction Now that you have thought through a logical model or framework for your Final Project, it is time to develop preliminary input, output, and outcome indicators. For this Assignment, use the guidelines from the Urban Institute resource and consult relevant Optional Resources from this week. Submit by Day 7 a 2- to 3-page paper which describes your input, output, and outcome program indicators, including the following: Describe the variables and the data you will be using.Provide a realistic discussion of the availability of research data.Provide an analysis of intended data collection strategies.If a sample or sample survey will be used, discuss the sampling frame or the sampling strategy you intend to use. Week 4 Grad writing Discussion: Reflecting on Your Writing: Paragraphing A well-organized paragraph is a central component to any piece of scholarly writing. This Week’s Learning Resources provide a useful framework to assess your first-draft paragraph for revisions in the Assignment, as well as to consider when drafting your next paragraph! In addition, you will review some general guidelines of effective revision practice. For this Discussion, you will reflect on your own writing in relation to revision and paragraph development. Based on what you have learned from the week’s Learning Resources, you will share your thoughts and experiences on revision and paragraph development. To prepare for this Discussion Review the Learning Resources on revision and the MEAL plan.Reflect on your writing process in relation to revision and paragraphing. You may even choose to consider how you approached this specific Assignment in relation to revision and paragraphing. By Day 3 Post a 1- to 2-paragraph reflection on your revision and paragraphing skills, including any strengths as well as weaknesses. (Note: You may choose to reflect specifically on how you will apply revision and effective paragraphing concepts in this week’s Assignment.) Read a selection of your colleagues’ postings. By Day 5 Respond to at least two of your colleagues’ postings in one or more of the following ways: Ask a probing question.Share an insight from having read your colleague’s posting.Offer and support an opinion.Validate an idea with your own experience.Make a suggestionExpand on your colleague’s post. Week 4 graduate writing assignment Are you ready to strengthen your paragraph in this first round of revisions? For this Assignment, you will review the Learning Resources on revision and paragraph development. With the principles outlined in the Learning Resources in mind, try your hand at revising your first-draft paragraph from Week 3’s Assignment. Look for the four components of main idea, evidence, analysis, and lead-in. To prepare for this Assignment: Review the Learning Resources related to revision and paragraph development.Review your first-draft paragraph from Week 3 Assignment against the MEAL plan model.Review the Infographic on MEAL plan guidelines The Assignment: Revise the paragraph from your Week 3 Assignment, to align with MEAL plan guidelines, so that your paragraph includes a main idea, supporting evidence, analysis, and lead-out. Your submission should be 1 paragraph long. Use the following questions to guide the writing of your paragraph: Does your paragraph begin with a clearly stated main idea?Does your paragraph include evidence supporting the main idea?Does your paragraph provide analysis and/or counterarguments?Does your paragraph end with a lead-out sentence? By Day 7 Submit your Assignment using Walden’s APA Course Paper Template. Assignments that are submitted without using Walden’s APA Course Paper Template will not be reviewed by your Instructor. Submission and Grading Information To submit your completed Assignment for review and grading, do the following: Please save your Assignment using the naming convention “WK4Assgn+last name+first initial.(extension)” as the name.Click the Week 4 Assignment Rubric to review the Grading Criteria for the Assignment.Click the Week 4 Assignment link. You will also be able to “View Rubric” for grading criteria from this area.Next, from the Attach File area, click on the Browse My Computer button. Find the document you saved as “WK4Assgn+last name+first initial.(extension)” and click Open.If applicable: From the Plagiarism Tools area, click the checkbox for I agree to submit my paper(s) to the Global Reference Database.Click on the Submit button to complete your submission. Week 5 Grad Writing Discussion: Reflecting on Your Writing: Scholarly Voice As a scholar, you want to write with objectivity and an appropriate tone. Once you have your first draft and have revised for strong paragraph development, you will want to take a further look at your writing, to ensure you are communicating in a scholarly voice. In addition, you will begin to examine how you are linking main ideas (also known as transitions). This week’s Learning Resources will help you examine your work for effective transitions and achieving scholarly voice. For this Discussion, you will reflect on your own writing in relation to scholarly voice Based on what you have learned from the week’s Learning Resources, you will share your thoughts and experiences on achieving scholarly voice. You may also choose to reflect on your practice of transitioning. To prepare for this Discussion Review the Learning Resources on scholarly voice and transitions.Reflect on your writing process in relation to scholarly voice and transitions. You may even choose to consider how you approached this specific Assignment in relation to scholarly voice and transitions. By Day 3 Post a 1- to 2-paragraph reflection on your use of scholarly voice. (Note: You may choose to reflect specifically on how you will apply scholarly voice concepts in this week’s Assignment.) You may also choose to address your practice of applying transitions. Read a selection of your colleagues’ postings. By Day 5 Respond to at least two of your colleagues’ postings in one or more of the following ways: Ask a probing question.Share an insight from having read your colleague’s posting.Offer and support an opinion.Validate an idea with your own experience.Make a suggestion.Expand on your colleague’s posting. By Day 7 Return to this Discussion to read the responses to your initial posting. Submission and Grading Information Assignment: Order and Transitions How are you linking main ideas in your writing? Might changing your paragraph’s order of content improve cohesion or flow of ideas? Are you writing in an objective, scholarly tone? For this Assignment, you will review the Learning Resources on scholarly tone and transitions. With the principle outlined in the Learning Resources in mind, try your hand at further revising your paragraph from Week 4 Assignment. You will also submit a first-draft second paragraph, focusing on a different main point from Week 2 Assignment Critical Reading and Outlining To prepare for this Assignment Review the Learning Resources on scholarly voice, transitions and, if necessary, the MEAL plan from Week 4.Review the content of your selected journal article.Choose another main point from Week 2 Assignment Critical Reading and Outlining. The Assignment Revise your paragraph from Week 4’s Assignment, ensuring you are effectively linking ideas and writing in a scholarly tone. Additionally, compose a first-draft paragraph summarizing another one of the article’s main points that you identified in Week 2 Assignment Critical Reading and Outlining. By Day 7 Submit your Assignment using Walden’s APA Course Paper Template. Your submission should be 2 paragraphs long. Assignments that are submitted without using Walden’s APA Course Paper Template will not be reviewed by your Instructor. Submission and Grading Information To submit your completed Assignment for review and grading, do the following: Please save your Assignment using the naming convention “WK5Assgn+last name+first initial.(extension)” as the name.Click the Week 5 Assignment Rubric to review the Grading Criteria for the Assignment.Click the Week 5 Assignment link. You will also be able to “View Rubric” for grading criteria from this area.Next, from the Attach File area, click on the Browse My Computer button. Find the document you saved as “WK5Assgn+last name+first initial.(extension)” and click Open.If applicable: From the Plagiarism Tools area, click the checkbox for I agree to submit my paper(s) to the Global Reference Database.Click on the Submit button to complete your submission.