Assessment Instrument Overview Activity DISCLAIMER: We require and expect students in the Communication program on the ASU’s West campus (where this course is housed) to be conversant in research methods. They take two research courses and are asked to regularly read research articles in their other courses for the major. If your program does not ask that of you then this assignment could prove particularly challenging. However, all of the correct information that your are asked to find, access, and report is contained within the documents you are instructed to read for this unit. Furthermore, the nature of the information you will be asked to find is covered in this unit or the preceding units. Follow the steps below to be successful on this activity. STEP 1: Review the Measurement PowerPoint from Unit 4. That is where you will find information related to Unit of Measurement. STEP 2: Review the Measurement Reliability PowerPoint in this unit (Unit 6). That is where you will find the information related to the reliability estimate or reliability coefficient, measurement items, and measurement dimensions. STEP 3: Read the two articles in this unit that describe the development of assessment instruments. These include: Kassing, J. K. (1998). Development and validation of the Organizational Dissent Scale. Management Communication Quarterly, 12, 183-229. Wood, J. A., & Winston, B. E. (2007). Development of three scales to measure leader accountability. Leadership & Organization Development Journal, 28, 167-185. STEP 4: Look at the final version of the assessment instrument, understanding that instruments are administered and tweaked (items added to or taken away) several times before a final version is determined. This is the case in both of the articles you will be consulting. For example, the final version of the Organizational Dissent Scale appears in the Study 2 portion of the article. STEP 5: Know and understand what it is you are looking for (review STEPS 1-2) when you consult the articles, otherwise you will get lost and overwhelmed quickly. This includes: What an assessment instrument is and does. What a measurement item is (the individual items that constitute an assessment instrument). What a dimension is (the difference concepts measured by an assessment instrument). What the reliability coefficient is (a statistical indicator of how well the measure performs). What the level of measurement is (the kind of data the measure provides). STEP 6: Consult the Assessment Instruments Summary Overview Table Template provided below to know where to place the appropriate information. Consult the Activity Guide to know where specifically you will find the information required. STEP 7: Complete the activity for each instrument by compiling the required information in the table on the final page. Assessment Instruments Summary Overview Table (Template) Use this template to complete the actual form on the next page. Name of Assessment Instrument Describe in brief above what the instrument assesses. Dimensions Description List each of the dimensions here. Use one cell per dimension. Describe in brief what each dimension assesses. Place your responses in the appropriate cell so that they correspond with the matching dimension in the cell to the left. Number of Items in the Dimension What is the level of measurement? What is the reliability of the dimension? Provide the number of items in each dimension in the appropriate cell. List what the level of measurement is for each dimension in the appropriate cell (see the Measurement PowerPoint). List what the reliability is for the final version of the dimension that is reported in the study. Assessment Instrument Overview Activity (Guide) While under development, most assessment instruments begin with more items than needed, so only some will be retained. Thus, it is important for you to look for the final version of the instruments produced in the reports, not the initial versions as the two can be quite different. There are two versions of each of the instruments created in the reports used in this activity. Assessment Instrument Dimensions Description Number of Items in the Dimension What is the level of measurement? What is the reliability of the dimension? Organizational Dissent Scale See pages 197-198. See page 192. See pages 206-207. See page 202. See pages 206-207. (consult the PDF named Organizational Dissent Scale) Leader Accountability Scale (Consult the PDF named Leader Accountability Scale) Match the format used (e.g., 1-3) with one of the levels of measurement from Unit 4: (i.e., nominal, ordinal, interval, or ratio) See pages 172. See page 172. See pages 177, 178, and 181. See page 281. Match the format used (e.g., 1-3) with one of the levels of measurement from Unit 4: (i.e., nominal, ordinal, interval, or ratio) See pages 177, 179, and 181. Assessment Instruments Summary Overview Table Assessment Instrument Organizational Dissent Scale Dimension Description Articulated Articulated dissent involves expressing dissent directly and openly to management, supervisor, and corporate officers. Antagonistic dissent occurs when employees believe they will be perceived as adversarial but also feel they have some safeguard against retaliation. Displaced dissent entails disagreeing without confronting or challenging. Antagonistic Displaced Responsibility scale Leader Accountability Scale Openness Scale Answerability Scale Measuring the leader’s acceptance of the inherent responsibilities incumbent in his/her role; Gauging the leader’s public disclosure of communication and action; Measuring the leader’s answerability for his/her actions and decisions following the literature review and expert review. Number of Items in the Dimension 10 What is the level of measurement? Interval What is the reliability of the dimension? 0.85 7 Interval 0.73 6 Interval 0.87 10 ratio 0.97 25 ratio 0.99 16 ratio 0.98 Unit 6: ASSESSMENT INSTRUMENTS Assessment instruments are used in training and development to determine where an organization or employees might have a need for training. That is, they are a fundamental part of conducting a needs assessment. Assessment instruments are the tools used to collect data that will then be analyzed to determine the training goals and objectives. Thus, they are a key step in the process. When it comes to measurement it is important that measures be both reliable and valid. Reliability refers to a measures ability to perform consistently across time without error, whereas validity refers to its ability to actually measure what it is intended to measure. In the studies assigned as reading for this unit you will see that the instruments developed are tested for reliability and validity in several ways. To better familiarize you with measurement reliability and validity consult the Measurement Reliability and Measurement Validity power points, which are located in the supplemental materials for this unit. Measurement Reliability Measurement Validity Unit 6 Unit 6 There are numerous assessment instruments available for training and development. They vary with regard to what they examine ranging from organizational commitment to team functioning. While many are privately held and require you to pay for their use, there are ample instruments available in the academic literature that can be accessed for no fee. See the Assessment Instrument Inventory for a sampling of some that will be relevant to communication training and development. As the final step in this unit, read the following two example articles (PDFs linked below, double click to access) to develop a better sense of how assessment instruments are developed and how they function. Kassing, J. K. (1998). Development and validation of the Organizational Dissent Scale. Management Communication Quarterly, 12, 183-229. Wood, J. A., & Winston, B. E. (2007). Development of three scales to measure leader accountability. Leadership & Organization Development Journal, 28, 167-185. Once you have carefully read about these two assessment instruments complete the Assessment Instrument Overview Activity. Before completing this activity you will need to read the document below (or available under the activity tab for this unit) — How to Read a Factor Analysis. It will help you make sense of the statistical reporting used in the assessment instrument articles. How to Read a Factor Analysis When completing the Assessment Instrument Overview Activity you will have to read through a section that details how the instrument was constructed. It will involve a statistical technique called factor analysis, which is the statistical technique used to develop measurement scales. To develop a scale, researchers will administer more items than they expect to keep in order to ensure that they retain the best group of items to measure the concept of interest. Thus, they need a statistical technique to indicate which items should be retained. This is where factor analysis comes in. Factor analysis tells us which times “fit” best together based upon how people responded to them. That is, factor analysis tells us which items to keep and which to discard. Thus, it is not unusual to see reports where a smaller number of items were retained for the scale than the amount used in the initial administration. In the example provided later one item was in fact discarded as not being usable. So, how do we determine which items to keep? Well, that depends on how they load on the factors revealed in the analysis. Factors are a statistical output that indicates a set of items that belong together. Once we have identified those items we call them a dimension. For example, there are 9 items in the Articulated Dissent dimension of the Organizational Dissent Scale (see the example instrument profile assignment). These items all loaded on the same factor, which was subsequently named Articulated Dissent. To be included in a factor/dimension items need to load high enough (with a numerical value) on one dimension to be included with the other items in that dimension. Loadings can range from .00 to .99, with higher numbers representing a better fit with the other items comprising the dimension. But items always produce values or loadings for each dimension. Thus, it is possible that an item can have equally strong loadings on two dimensions. This indicates that some people responded to the item differently than others. That it did not have the same meaning for all people. This is a problem in measurement because we want items to load clearly on one dimension and not on any other dimensions. Thus, there are guidelines adopted by most researchers to create “cut off” points for accepting or rejecting items. The two most common are the 60/40 and 50/30 rules. In both cases the higher and first number represents the minimum value an item must achieve on a factor in order to be considered part of that dimension. The second number represents a value that the item must not exceed on any other factor. Otherwise it is thought to be loading too heavily on more than one dimension (the problem noted above). With a 60/40 rule, then, an item that loads at .70 on Dimension 1 and .25 on Dimension 2 — clearly belongs in Dimension 1. In contrast, an item that loads at .55 on Dimension 1 and .24 on Dimension 2 would need to be rejected because it does not have a strong enough relationship with Dimension 1. And an item that loads at .55 on one dimension and .45 on the other would need to be rejected because it loads too heavily on two dimensions. Let’s have a look at an actual factor analysis conducted to determine the fit of items on a scale developed to measure Supervisors’ Reactions to Employee Dissent (Kassing, 2009). See the following page while we work through this discussion. Here you have a typical factor analysis table. It lists the items down the side and the dimensions across the top. Factor loadings appear in the respective columns. Notice that each item has a factor loading for each dimension. These factor loadings indicate the degree to which each item loads on the three dimensions. Items tend to be listed in the order of strength with which they load on the first dimension, then the second, and so on. Thus, larger numbers appear toward the top of the first column, in the middle of the second column, and toward the bottom of later columns. Using a 50/30 convention for accepting an item, we can see that the first 6 items all load above .50 on the Delaying factor/dimension, but not higher than .30 on any other factor/dimension. Thus, they are all included in this dimension. Beginning with the seventh item, we can see three items that have high values in the Decreasingly Favorable column (.88, .81, and .79) and low scores in all other columns (nothing higher than .12). These three items then comprise the Decreasingly Favorable dimension. Finally, if we look to the Increasingly Favorable column we can see that five items load higher than .50, but no higher than .30 on any other dimension. So this dimension is made up of these 5 items. The final item listed loads almost identically on the Delaying and Increasingly Favorable dimensions (.45 and .46 respectively), indicating that it should be rejected. It is in fact the only item that should be rejected. You can see however that if we used the 60/40 rule we would have to reject several additional items that loaded above.50 but not above.60. For this reason the 60/40 rule is thought to be more conservative and restrictive, whereas the 50/30 is rule is more liberal — usually leading to the inclusion of more items. Table 1. Factor Loadings for Supervisory Reactions to Employee Dissent ______________________________________________________________________________ Item Delaying Decreasingly Favorable Increasingly Favorable My supervisor waited too long to address the issue. .80 .00 .12 My supervisor failed to address the issue. .63 .00 -.29 My supervisor gave excuses for not addressing the issue. .62 .26 .12 My supervisor repeatedly dismissed the issue. .59 .07 -.22 My supervisor continued to ignore the issue. .56 .07 -.28 My supervisor made a point to address the issue quickly.R .54 .08 -.07 My supervisor became irritated with me over time. -.02 .88 -.08 My supervisor became aggravated with me over time. .08 .81 -.06 My supervisor became annoyed with me over time. .12 .79 -.05 My supervisor became more sympathetic to my concern. .03 -.15 .73 My supervisor became more responsive to me over time. -.04 -.08 .72 My supervisor became more receptive to me over time. .01 -.17 .71 My supervisor eventually resolved the issue. -.27 .12 .54 My supervisor slowly attempted to address the issue. .05 -.05 .50 -.45 .04 .46 My supervisor addressed the issue satisfactorily. _____________________________________________________________________________________________ R Reverse coded item. Assessment Instrument Overview Activity DISCLAIMER: We require and expect students in the Communication program on the ASU’s West campus (where this course is housed) to be conversant in research methods. They take two research courses and are asked to regularly read research articles in their other courses for the major. If your program does not ask that of you then this assignment could prove particularly challenging. However, all of the correct information that your are asked to find, access, and report is contained within the documents you are instructed to read for this unit. Furthermore, the nature of the information you will be asked to find is covered in this unit or the preceding units. Follow the steps below to be successful on this activity. STEP 1: Review the Measurement PowerPoint from Unit 4. That is where you will find information related to Unit of Measurement. STEP 2: Review the Measurement Reliability PowerPoint in this unit (Unit 6). That is where you will find the information related to the reliability estimate or reliability coefficient, measurement items, and measurement dimensions. STEP 3: Read the two articles in this unit that describe the development of assessment instruments. These include: Kassing, J. K. (1998). Development and validation of the Organizational Dissent Scale. Management Communication Quarterly, 12, 183-229. Wood, J. A., & Winston, B. E. (2007). Development of three scales to measure leader accountability. Leadership & Organization Development Journal, 28, 167-185. STEP 4: Look at the final version of the assessment instrument, understanding that instruments are administered and tweaked (items added to or taken away) several times before a final version is determined. This is the case in both of the articles you will be consulting. For example, the final version of the Organizational Dissent Scale appears in the Study 2 portion of the article. STEP 5: Know and understand what it is you are looking for (review STEPS 1-2) when you consult the articles, otherwise you will get lost and overwhelmed quickly. This includes: What an assessment instrument is and does. What a measurement item is (the individual items that constitute an assessment instrument). What a dimension is (the difference concepts measured by an assessment instrument). What the reliability coefficient is (a statistical indicator of how well the measure performs). What the level of measurement is (the kind of data the measure provides). STEP 6: Consult the Assessment Instruments Summary Overview Table Template provided below to know where to place the appropriate information. Consult the Activity Guide to know where specifically you will find the information required. STEP 7: Complete the activity for each instrument by compiling the required information in the table on the final page. Assessment Instruments Summary Overview Table (Template) Use this template to complete the actual form on the next page. Name of Assessment Instrument Describe in brief above what the instrument assesses. Dimensions Description List each of the dimensions here. Use one cell per dimension. Describe in brief what each dimension assesses. Place your responses in the appropriate cell so that they correspond with the matching dimension in the cell to the left. Number of Items in the Dimension What is the level of measurement? What is the reliability of the dimension? Provide the number of items in each dimension in the appropriate cell. List what the level of measurement is for each dimension in the appropriate cell (see the Measurement PowerPoint). List what the reliability is for the final version of the dimension that is reported in the study. Assessment Instrument Overview Activity (Guide) While under development, most assessment instruments begin with more items than needed, so only some will be retained. Thus, it is important for you to look for the final version of the instruments produced in the reports, not the initial versions as the two can be quite different. There are two versions of each of the instruments created in the reports used in this activity. Assessment Instrument Dimensions Description Number of Items in the Dimension What is the level of measurement? What is the reliability of the dimension? Organizational Dissent Scale See pages 197-198. See page 192. See pages 206-207. See page 202. See pages 206-207. (consult the PDF named Organizational Dissent Scale) Leader Accountability Scale (Consult the PDF named Leader Accountability Scale) Match the format used (e.g., 1-3) with one of the levels of measurement from Unit 4: (i.e., nominal, ordinal, interval, or ratio) See pages 172. See page 172. See pages 177, 178, and 181. See page 281. Match the format used (e.g., 1-3) with one of the levels of measurement from Unit 4: (i.e., nominal, ordinal, interval, or ratio) See pages 177, 179, and 181. Assessment Instruments Summary Overview Table Assessment Instrument Organizational Dissent Scale Leader Accountability Scale Dimension Description Number of Items in the Dimension What is the level of measurement? What is the reliability of the dimension?
Purchase answer to see full attachment