Assessment Instrument Overview Activity
DISCLAIMER: We require and expect students in the Communication program on the ASU’s West campus
(where this course is housed) to be conversant in research methods. They take two research courses and are asked
to regularly read research articles in their other courses for the major. If your program does not ask that of you
then this assignment could prove particularly challenging. However, all of the correct information that your are
asked to find, access, and report is contained within the documents you are instructed to read for this unit.
Furthermore, the nature of the information you will be asked to find is covered in this unit or the preceding units.
Follow the steps below to be successful on this activity.
STEP 1:
Review the Measurement PowerPoint from Unit 4. That is where you will find information related to
Unit of Measurement.
STEP 2:
Review the Measurement Reliability PowerPoint in this unit (Unit 6). That is where you will find the
information related to the reliability estimate or reliability coefficient, measurement items, and
measurement dimensions.
STEP 3:
Read the two articles in this unit that describe the development of assessment instruments. These
include:
Kassing, J. K. (1998). Development and validation of the Organizational Dissent Scale. Management
Communication Quarterly, 12, 183-229.
Wood, J. A., & Winston, B. E. (2007). Development of three scales to measure leader accountability.
Leadership & Organization Development Journal, 28, 167-185.
STEP 4:
Look at the final version of the assessment instrument, understanding that instruments are
administered and tweaked (items added to or taken away) several times before a final version is
determined. This is the case in both of the articles you will be consulting. For example, the final
version of the Organizational Dissent Scale appears in the Study 2 portion of the article.
STEP 5:
Know and understand what it is you are looking for (review STEPS 1-2) when you consult the
articles, otherwise you will get lost and overwhelmed quickly.
This includes:
What an assessment instrument is and does.
What a measurement item is (the individual items that constitute an assessment instrument).
What a dimension is (the difference concepts measured by an assessment instrument).
What the reliability coefficient is (a statistical indicator of how well the measure performs).
What the level of measurement is (the kind of data the measure provides).
STEP 6:
Consult the Assessment Instruments Summary Overview Table Template provided below to know
where to place the appropriate information. Consult the Activity Guide to know where
specifically you will find the information required.
STEP 7:
Complete the activity for each instrument by compiling the required information in the table on the
final page.
Assessment Instruments Summary Overview Table (Template)
Use this template to complete the actual form on the next page.
Name of Assessment
Instrument
Describe in brief above
what the instrument
assesses.
Dimensions
Description
List each of the
dimensions here.
Use one cell per
dimension.
Describe in brief what each dimension assesses.
Place your responses in the appropriate cell so that they
correspond with the matching dimension in the
cell to the left.
Number of
Items in the
Dimension
What is the
level of
measurement?
What is the
reliability of
the dimension?
Provide the
number of
items in each
dimension
in the
appropriate
cell.
List what the
level of
measurement
is for each
dimension in
the appropriate
cell (see the
Measurement
PowerPoint).
List what the
reliability is
for the final
version of the
dimension that
is reported in
the study.
Assessment Instrument Overview Activity (Guide)
While under development, most assessment instruments begin with more items than needed, so only some will be retained. Thus, it is
important for you to look for the final version of the instruments produced in the reports, not the initial versions as the two can be
quite different. There are two versions of each of the instruments created in the reports used in this activity.
Assessment
Instrument
Dimensions
Description
Number of Items
in the Dimension
What is the level of
measurement?
What is the reliability of
the dimension?
Organizational
Dissent
Scale
See
pages
197-198.
See
page
192.
See
pages
206-207.
See
page
202.
See
pages
206-207.
(consult the
PDF named
Organizational
Dissent Scale)
Leader
Accountability
Scale
(Consult the
PDF named
Leader
Accountability
Scale)
Match the format used
(e.g., 1-3) with one of the
levels of measurement
from Unit 4: (i.e., nominal,
ordinal, interval, or ratio)
See
pages
172.
See
page
172.
See
pages
177,
178,
and
181.
See
page
281.
Match the format used
(e.g., 1-3) with one of the
levels of measurement
from Unit 4: (i.e., nominal,
ordinal, interval, or ratio)
See
pages
177,
179,
and
181.
Assessment Instruments Summary Overview Table
Assessment
Instrument
Organizational Dissent
Scale
Dimension
Description
Articulated
Articulated dissent involves expressing dissent directly and
openly to management, supervisor, and corporate officers.
Antagonistic dissent occurs when employees believe they
will be perceived as adversarial but also feel they have
some safeguard against retaliation.
Displaced dissent entails disagreeing without confronting or
challenging.
Antagonistic
Displaced
Responsibility scale
Leader Accountability Scale
Openness Scale
Answerability Scale
Measuring the leader’s acceptance of the inherent
responsibilities incumbent in his/her role;
Gauging the leader’s public disclosure of communication
and action;
Measuring the leader’s answerability for his/her actions
and decisions following the literature review and expert
review.
Number of
Items in the
Dimension
10
What is the
level of
measurement?
Interval
What is the
reliability of
the dimension?
0.85
7
Interval
0.73
6
Interval
0.87
10
ratio
0.97
25
ratio
0.99
16
ratio
0.98
Unit 6:
ASSESSMENT INSTRUMENTS
Assessment instruments are used in training and development to determine
where an organization or employees might have a need for training. That is, they
are a fundamental part of conducting a needs assessment. Assessment instruments
are the tools used to collect data that will then be analyzed to determine the
training goals and objectives. Thus, they are a key step in the process.
When it comes to measurement it is important that measures be both reliable
and valid. Reliability refers to a measures ability to perform consistently across
time without error, whereas validity refers to its ability to actually measure what it
is intended to measure. In the studies assigned as reading for this unit you will see
that the instruments developed are tested for reliability and validity in several
ways. To better familiarize you with measurement reliability and validity consult
the Measurement Reliability and Measurement Validity power points, which
are located in the supplemental materials for this unit.
Measurement
Reliability
Measurement
Validity
Unit 6
Unit 6
There are numerous assessment instruments available for training and
development. They vary with regard to what they examine ranging from
organizational commitment to team functioning.
While many are privately held and require you to pay for their use, there are
ample instruments available in the academic literature that can be accessed for no
fee. See the Assessment Instrument Inventory for a sampling of some that will
be relevant to communication training and development.
As the final step in this unit, read the following two example articles (PDFs
linked below, double click to access) to develop a better sense of how assessment
instruments are developed and how they function.
Kassing, J. K. (1998). Development and validation of the Organizational
Dissent Scale. Management Communication Quarterly, 12, 183-229.
Wood, J. A., & Winston, B. E. (2007). Development of three scales to
measure leader accountability. Leadership & Organization
Development Journal, 28, 167-185.
Once you have carefully read about these two assessment instruments
complete the Assessment Instrument Overview Activity. Before completing this
activity you will need to read the document below (or available under the activity
tab for this unit) — How to Read a Factor Analysis. It will help you make sense
of the statistical reporting used in the assessment instrument articles.
How to Read a Factor Analysis
When completing the Assessment Instrument Overview Activity you will
have to read through a section that details how the instrument was constructed. It
will involve a statistical technique called factor analysis, which is the statistical
technique used to develop measurement scales.
To develop a scale, researchers will administer more items than they expect
to keep in order to ensure that they retain the best group of items to measure the
concept of interest. Thus, they need a statistical technique to indicate which items
should be retained. This is where factor analysis comes in. Factor analysis tells us
which times “fit” best together based upon how people responded to them. That is,
factor analysis tells us which items to keep and which to discard. Thus, it is not
unusual to see reports where a smaller number of items were retained for the scale
than the amount used in the initial administration. In the example provided later
one item was in fact discarded as not being usable.
So, how do we determine which items to keep? Well, that depends on how
they load on the factors revealed in the analysis. Factors are a statistical output that
indicates a set of items that belong together. Once we have identified those items
we call them a dimension. For example, there are 9 items in the Articulated Dissent
dimension of the Organizational Dissent Scale (see the example instrument profile
assignment). These items all loaded on the same factor, which was subsequently
named Articulated Dissent.
To be included in a factor/dimension items need to load high enough (with a
numerical value) on one dimension to be included with the other items in that
dimension. Loadings can range from .00 to .99, with higher numbers representing a
better fit with the other items comprising the dimension. But items always produce
values or loadings for each dimension. Thus, it is possible that an item can have
equally strong loadings on two dimensions. This indicates that some people
responded to the item differently than others. That it did not have the same
meaning for all people. This is a problem in measurement because we want items
to load clearly on one dimension and not on any other dimensions.
Thus, there are guidelines adopted by most researchers to create “cut off”
points for accepting or rejecting items. The two most common are the 60/40 and
50/30 rules. In both cases the higher and first number represents the minimum
value an item must achieve on a factor in order to be considered part of that
dimension. The second number represents a value that the item must not exceed on
any other factor. Otherwise it is thought to be loading too heavily on more than one
dimension (the problem noted above).
With a 60/40 rule, then, an item that loads at .70 on Dimension 1 and .25 on
Dimension 2 — clearly belongs in Dimension 1. In contrast, an item that loads at
.55 on Dimension 1 and .24 on Dimension 2 would need to be rejected because it
does not have a strong enough relationship with Dimension 1. And an item that
loads at .55 on one dimension and .45 on the other would need to be rejected
because it loads too heavily on two dimensions.
Let’s have a look at an actual factor analysis conducted to determine the fit
of items on a scale developed to measure Supervisors’ Reactions to Employee
Dissent (Kassing, 2009).
See the following page while we work through this discussion. Here you
have a typical factor analysis table. It lists the items down the side and the
dimensions across the top. Factor loadings appear in the respective columns.
Notice that each item has a factor loading for each dimension. These factor
loadings indicate the degree to which each item loads on the three dimensions.
Items tend to be listed in the order of strength with which they load on the first
dimension, then the second, and so on. Thus, larger numbers appear toward the top
of the first column, in the middle of the second column, and toward the bottom of
later columns.
Using a 50/30 convention for accepting an item, we can see that the first 6
items all load above .50 on the Delaying factor/dimension, but not higher than .30
on any other factor/dimension. Thus, they are all included in this dimension.
Beginning with the seventh item, we can see three items that have high values in
the Decreasingly Favorable column (.88, .81, and .79) and low scores in all other
columns (nothing higher than .12). These three items then comprise the
Decreasingly Favorable dimension. Finally, if we look to the Increasingly
Favorable column we can see that five items load higher than .50, but no higher
than .30 on any other dimension. So this dimension is made up of these 5 items.
The final item listed loads almost identically on the Delaying and Increasingly
Favorable dimensions (.45 and .46 respectively), indicating that it should be
rejected. It is in fact the only item that should be rejected.
You can see however that if we used the 60/40 rule we would have to reject
several additional items that loaded above.50 but not above.60. For this reason the
60/40 rule is thought to be more conservative and restrictive, whereas the 50/30 is
rule is more liberal — usually leading to the inclusion of more items.
Table 1.
Factor Loadings for Supervisory Reactions to Employee Dissent
______________________________________________________________________________
Item
Delaying
Decreasingly
Favorable
Increasingly
Favorable
My supervisor waited too long to address the issue.
.80
.00
.12
My supervisor failed to address the issue.
.63
.00
-.29
My supervisor gave excuses for not addressing the issue.
.62
.26
.12
My supervisor repeatedly dismissed the issue.
.59
.07
-.22
My supervisor continued to ignore the issue.
.56
.07
-.28
My supervisor made a point to address the issue quickly.R .54
.08
-.07
My supervisor became irritated with me over time.
-.02
.88
-.08
My supervisor became aggravated with me over time.
.08
.81
-.06
My supervisor became annoyed with me over time.
.12
.79
-.05
My supervisor became more sympathetic to my concern.
.03
-.15
.73
My supervisor became more responsive to me over time.
-.04
-.08
.72
My supervisor became more receptive to me over time.
.01
-.17
.71
My supervisor eventually resolved the issue.
-.27
.12
.54
My supervisor slowly attempted to address the issue.
.05
-.05
.50
-.45
.04
.46
My supervisor addressed the issue satisfactorily.
_____________________________________________________________________________________________
R
Reverse coded item.
Assessment Instrument Overview Activity
DISCLAIMER: We require and expect students in the Communication program on the ASU’s West campus
(where this course is housed) to be conversant in research methods. They take two research courses and are asked
to regularly read research articles in their other courses for the major. If your program does not ask that of you
then this assignment could prove particularly challenging. However, all of the correct information that your are
asked to find, access, and report is contained within the documents you are instructed to read for this unit.
Furthermore, the nature of the information you will be asked to find is covered in this unit or the preceding units.
Follow the steps below to be successful on this activity.
STEP 1:
Review the Measurement PowerPoint from Unit 4. That is where you will find information related to
Unit of Measurement.
STEP 2:
Review the Measurement Reliability PowerPoint in this unit (Unit 6). That is where you will find the
information related to the reliability estimate or reliability coefficient, measurement items, and
measurement dimensions.
STEP 3:
Read the two articles in this unit that describe the development of assessment instruments. These
include:
Kassing, J. K. (1998). Development and validation of the Organizational Dissent Scale. Management
Communication Quarterly, 12, 183-229.
Wood, J. A., & Winston, B. E. (2007). Development of three scales to measure leader accountability.
Leadership & Organization Development Journal, 28, 167-185.
STEP 4:
Look at the final version of the assessment instrument, understanding that instruments are
administered and tweaked (items added to or taken away) several times before a final version is
determined. This is the case in both of the articles you will be consulting. For example, the final
version of the Organizational Dissent Scale appears in the Study 2 portion of the article.
STEP 5:
Know and understand what it is you are looking for (review STEPS 1-2) when you consult the
articles, otherwise you will get lost and overwhelmed quickly.
This includes:
What an assessment instrument is and does.
What a measurement item is (the individual items that constitute an assessment instrument).
What a dimension is (the difference concepts measured by an assessment instrument).
What the reliability coefficient is (a statistical indicator of how well the measure performs).
What the level of measurement is (the kind of data the measure provides).
STEP 6:
Consult the Assessment Instruments Summary Overview Table Template provided below to know
where to place the appropriate information. Consult the Activity Guide to know where
specifically you will find the information required.
STEP 7:
Complete the activity for each instrument by compiling the required information in the table on the
final page.
Assessment Instruments Summary Overview Table (Template)
Use this template to complete the actual form on the next page.
Name of Assessment
Instrument
Describe in brief above
what the instrument
assesses.
Dimensions
Description
List each of the
dimensions here.
Use one cell per
dimension.
Describe in brief what each dimension assesses.
Place your responses in the appropriate cell so that they
correspond with the matching dimension in the
cell to the left.
Number of
Items in the
Dimension
What is the
level of
measurement?
What is the
reliability of
the dimension?
Provide the
number of
items in each
dimension
in the
appropriate
cell.
List what the
level of
measurement
is for each
dimension in
the appropriate
cell (see the
Measurement
PowerPoint).
List what the
reliability is
for the final
version of the
dimension that
is reported in
the study.
Assessment Instrument Overview Activity (Guide)
While under development, most assessment instruments begin with more items than needed, so only some will be retained. Thus, it is
important for you to look for the final version of the instruments produced in the reports, not the initial versions as the two can be
quite different. There are two versions of each of the instruments created in the reports used in this activity.
Assessment
Instrument
Dimensions
Description
Number of Items
in the Dimension
What is the level of
measurement?
What is the reliability of
the dimension?
Organizational
Dissent
Scale
See
pages
197-198.
See
page
192.
See
pages
206-207.
See
page
202.
See
pages
206-207.
(consult the
PDF named
Organizational
Dissent Scale)
Leader
Accountability
Scale
(Consult the
PDF named
Leader
Accountability
Scale)
Match the format used
(e.g., 1-3) with one of the
levels of measurement
from Unit 4: (i.e., nominal,
ordinal, interval, or ratio)
See
pages
172.
See
page
172.
See
pages
177,
178,
and
181.
See
page
281.
Match the format used
(e.g., 1-3) with one of the
levels of measurement
from Unit 4: (i.e., nominal,
ordinal, interval, or ratio)
See
pages
177,
179,
and
181.
Assessment Instruments Summary Overview Table
Assessment
Instrument
Organizational Dissent
Scale
Leader Accountability Scale
Dimension
Description
Number of
Items in the
Dimension
What is the
level of
measurement?
What is the
reliability of
the dimension?
Purchase answer to see full
attachment