Methods in Program Design

Anonymous
timer Asked: Mar 10th, 2019

Question Description

P8640 Methodsin Program Evaluation

Session 7: In-Class Exercise Part A

Review the following article (ATTACHED):

Ager, A., Akesson, B., Stark, L., Flouri, E., Okot, B., McCollister, F., & Boothby, N. (2011). The impact of the school‐based Psychosocial Structured Activities (PSSA) program on conflict‐affected children in northern Uganda. Journal of child psychology and psychiatry, 52(11), 1124-1133.

Answer the following questions:

  1. What were the original program goals for the intervention?
  2. Sketch out a bare-bones logic model summarizing the inputs, activities, and outcomes. For the purposes of this exercise, skip the outputs and impacts.
  3. How did the authors measure and operationalize their primary outcome? What were the indicators that were used?
  4. What is the study design used?
  5. What are the threats to internal validity common in this design?
  6. Complete the table of participants.

Baseline

Follow-up

% LTF

Children

Intervention

Comparison

Parents

Intervention

Comparison

Session 7: In-Class Exercise Part B

Read the case study:

WVI Child-Friendly Spaces (CFS) Intervention at Buramino Camp, Dolo Ado, Ethiopia:

Buramino refugee camp, located along the southern Somali-Ethiopian border, was selected in cooperation with World Vision Ethiopia as the first evaluation site. At the time of the evaluation, the Ethiopian Administration for Refugee and Returnee Affairs (ARRA), the main implementing partner of UNHCR, had organised five camps within two hours of the Dollo Ado transit centre to address the growing needs of Somali refugees who had migrated in response to drought and ongoing conflict in the region. In January 2012, World Vision Ethiopia’s first Child Learning Centre (subsequently referred to through this report as a CFS site) was established, offering a range of services to children aged six to 17 years. The second CFS site started activities later that year, in May 2012. Within the wide continuum of CFS programming approaches, these two sites emphasised functional literacy and numeracy skills. Based on discussions with ARRA and parents of children attending these centres, psychosocial activities, such as drawing, singing, and recreational play, were included, but to a modest degree. On-site counselling and a feeding programme were also offered. To help meet the overwhelming demand in the camp, each centre provided two three-hour sessions for children: a morning session for younger children aged six to 11 and an afternoon session for older children aged 12 to 17. The first CFS provided services for 784 children (500 aged between six and 11 and 284 aged between 12 and 17). The second CFS site provided services for 787 children (506 aged between six and 11 and 281 aged between 12 and 17).

Source: http://www.cpcnetwork.org/wp-content/uploads/2015/04/WV-CU-CFS-Ethiopia-Field-Study-Summary-Report.pdf

Answer the following questions:

  • Develop one evaluation question
  • Consider what study design would best answer your question
  • Decide what data you need to collect

4.Recommend one random and one non-random sampling scheme. (circle your choices)

Types of random sampling

Types of non-random sampling

Simple random sample

Systematic

Proportionate stratified

Disproportionate stratified (Cluster sampling)

Quota sampling

Extreme case sampling

Maximum variation sampling

Homogeneous sampling

Typical case sampling

Snowball sampling

Criterion sampling

5.Discuss the advantages and disadvantages of each scheme that you choose.

6.Why did you choose the random sampling scheme you did?

7.What aspect(s) of the evaluation question would this scheme help you answer?

8.Why did you choose the non-random sampling scheme?

9.What aspect(s) of the evaluation question would this scheme help you answer?

Journal of Child Psychology and Psychiatry 52:11 (2011), pp 1124–1133 doi:10.1111/j.1469-7610.2011.02407.x The impact of the school-based Psychosocial Structured Activities (PSSA) program on conflict-affected children in northern Uganda Alastair Ager,1 Bree Akesson,2 Lindsay Stark,3 Eirini Flouri,4 Braxton Okot,5 Faith McCollister,3 and Neil Boothby1 1 Mailman School of Public Health, Columbia University, USA; 2Care and Protection of Children in Crisis-Affected Countries Initiative; 3Program on Forced Migration & Health, Mailman School of Public Health, Columbia University, USA; 4Department of Psychology and Human Development, Institute of Education, University of London, UK; 5Child Resilience Program, Save the Children in Uganda Background: Children in northern Uganda have undergone significant psychosocial stress during the region’s lengthy conflict. A Psychosocial Structured Activities (PSSA) program was implemented in 21 schools identified as amongst those most severely affected by conflict-induced displacement across Gulu and Amuru Districts. The PSSA intervention comprised a series of 15 class sessions designed to progressively increase children’s resilience through structured activities involving drama, movement, music and art (with additional components addressing parental support and community involvement). Method: Eight schools were selected by random quota sampling from those schools receiving the PSSA intervention. Two hundred and three children were identified in these schools as being scheduled to receive intervention, and were followed up 12 months later following engagement with PSSA activities. A comparison group comprised 200 children selected from schools that had met inclusion criteria for receipt of intervention, but were not scheduled for intervention coverage until later. Preliminary research used participatory focus group methodology to determine local indicators of child well-being as viewed by parents, teachers, and children respectively. Pre- and post- assessments focused on ratings for each child – by parents, teachers and children – with respect to these indicators. Results: Significant increases in ratings of child well-being were observed in both intervention and comparison groups over a 12-month period. However, the well-being of children who had received the PSSA intervention increased significantly more than for children in the comparison group, as judged by child and parent (but not teacher) report. This effect was evident despite considerable loss-to-follow-up at post-testing as a result of return of many households to communities of origin. Conclusion: General improvement in child well-being over a 12-month period suggests that recovery and reconstruction efforts in Northern Uganda following the onset of peace had a substantive impact on the lives of children. However, exposure to the PSSA program had an additional positive impact on child well-being, suggesting its value in post-conflict recovery contexts. Keywords: Psychosocial, structured activities, children, schools, Northern Uganda, evaluation. Abbreviations: PSSA: Psychosocial Structured Activities. Northern Uganda is in a state of transition after 20 years of conflict between the Lord’s Resistance Army (LRA) and Ugandan government forces. Despite continuing LRA activity in the region and General Joseph Kony’s refusal to sign a Final Peace Agreement in March 2008, the war drew to an effective close in northern Uganda with an ‘Agreement on Cessation of Hostilities’ signed in 2006 between the Government of Uganda and the LRA (Government of Uganda & Lord’s Resistance Army/ Movement, 2006; Machar, 2008). During the conflict between 1.8 and 2 million people were displaced to over 200 camps, located primarily in Gulu, Amuru, Kitgum and Pader districts. These Internally Displaced Persons (IDP) camps were often cramped and lacking in sanitation, and left most families unable Conflict of interest statement: At the time of the study Braxton Okot was serving as Program Manager of the Child Resilience Program. Other authors declare no conflict of interest with respect to the reported study. to support themselves through traditional subsistence agricultural practices (Civil Society Organizations for Peace in Northern Uganda (CSOPNU), 2006). Excess mortality rates exceeded emergency thresholds with estimates as high as 1,000 deaths per week in 2005 among IDPs in the region (World Health Organization, 2005). The conflict disrupted the social and economic development of northern Uganda for a generation. Children were particularly harshly affected by this conflict. One of the defining tactics of the LRA has been its use of children as combatants, porters, and sex slaves, often abducting them from their homes and schools. While many of these formerly abducted children eventually returned to their communities, many returned with significant physical and emotional after-effects of their experience. Amongst abducted youth in the region Annan, Blattman, and Horton (2006) showed that ‘high levels of violence committed and violence experienced are associated  2011 The Authors. Journal of Child Psychology and Psychiatry  2011 Association for Child and Adolescent Mental Health. Published by Blackwell Publishing, 9600 Garsington Road, Oxford OX4 2DQ, UK and 350 Main St, Malden, MA 02148, USA Psychosocial structured activities with conflict-affected children in northern Uganda with substantial increases in emotional distress’ (p. 67). Those who were not abducted were also found to have experienced significant violence, with male youths between 14 and 30 typically reporting having experienced nine traumatic events over their lifetime. The educational system – and thus children’s schooling – has also been significantly affected by the conflict. A 2006 survey indicated that 60% of schools in the affected regions were not functioning, depriving over 250,000 children of an education (CSOPNU, 2006). The Uganda Ministry of Education, in cooperation with a number of international nongovernmental agencies, responded to this situation by expanding places at schools adjacent to IDP camps (and deploying displaced teachers across such schools). In Gulu and Amuru Districts, for example, 40 schools were identified as having major increases in student enrollment, and received assistance in infrastructure development to assist them to respond to such demands (Wilson, 2007). Accommodating large numbers of displaced children, many of whom had been abducted or experienced other extreme conflict-related trauma and virtually all of whom had experienced personal and familial threat and loss linked to the conflict, was acknowledged to have significant impact on the learning environment of the schools. It was in this context that Save the Children proposed to work in a number of these schools with the ‘Child Resilience’ curriculum project, which involved deployment of the Psychosocial Structured Activities (PSSA) intervention. The PSSA intervention builds upon the work of Robert Macy’s Classroom-Based Intervention (CBI) Program, which has been used in a number of settings, including Palestine (Khamis, Macy, & Coignez, 2004) and Sri Lanka (Somasundaram, 2007).1 It is a school-based, multi-phased approach designed to use children’s natural resilience to help them recover from trauma. Its core focus is 15 progressively structured sessions leading from themes of safety and control, through those of awareness and selfesteem, to personal narratives, coping skills, and future planning. These sessions incorporate play therapy, drama, art and movement in an effort to enhance children’s resilience and feelings of stability and security after trauma as they progress emotionally and cognitively over the course of the program. Each session opens with an activity designed to prepare children for the session and then continues into a central activity and cooperative game designed around the session’s theme. The PSSA 1 In some contexts the CBI program is designated as a CampBased or Community-Based Intervention. The CBI Manual was adapted for use in the Uganda context by Susan Garland. Copies of the CBI manual can be obtained by contacting Robert Macy. http://www.traumacenter.org/initiatives/psychosocial. php. 1125 program also seeks to incorporate a community service component, and encourages parental involvement through periodic meetings during which issues related to the children’s experience and the needs of the local community are discussed. There is considerable value in determining a robust methodology to establish the impact of interventions such as PSSA. The PSSA program was implemented by Save the Children in the United States on the Gulf Coast after Hurricane Katrina and in tsunami-affected areas of Indonesia. Few published, rigorous evaluations exist measuring the impact of the program, though reports from the field suggested that it was a popular intervention that garnered the support of parents and school officials (see Jaycox, Morse, Tanielian, & Stein, 2006; Lauten & Lietz, 2008). Encouraged by such reports, Save the Children in Uganda implemented the PSSA program among children attending government schools in northern Uganda beginning in the 2006/2007 school year. Positive anecdotal reports led to the commissioning of an evaluation (Wilson, 2007) that suggested that the program had indeed a positive impact on the lives of children and their caregivers. The intervention was reported to have had a positive impact on attendance rates, student grades, and teacher–student interactions in the classroom. Increases in children’s resilience (defined through consolidation of items elicited through a free-listing exercise with participants in a fashion built upon in the current study) were reported by both children and parents. However, in the absence of baseline measures, the study relied on retrospective judgments of changes since receiving the intervention. It also had no form of control condition, only collecting data from individuals exposed to the intervention. The current study represents an attempt to establish a more rigorous evaluation of the impact of the PSSA program on children’s well-being in this context. In general, there have been few rigorous studies of the implementation of school-based psychosocial activities in conflict and post-conflict environments. An evaluation of a secondary-school-based mental health intervention for children exposed to political violence in Indonesia found some improvement in post-traumatic stress disorder symptoms and hope but did not reduce a variety of other traumatic-stress symptoms (Tol et al., 2008). A study of structured activities in child-focused interventions in Palestine designed to increase children’s resilience found improved emotional and behavioral well-being and parental support, but no increase in hopefulness (Loughry et al., 2006). A cluster randomized trial of a version of the CBI program in Nepal found no main effect of the intervention on mental health but significant impact on social well-being indicators among sub-groups (such as girls’ prosocial behavior and hope amongst older children; Jordans et al., 2010). A recent literature review by Peltonen and Punamaki (2010) analyzed the effectiveness and  2011 The Authors. Journal of Child Psychology and Psychiatry  2011 Association for Child and Adolescent Mental Health. 1126 Alastair Ager et al. theoretical basis of 16 interventions for children exposed to trauma associated with armed conflict, a number of which involved school-based programming on conflict or post-conflict settings (Berger, Pat-Horenczyk, & Gelkopf, 2007; Chase et al., 1999; Layne et al., 2001, 2008; Woodside, Santa Barbara, & Benner, 1999). The studies by Berger et al. (2007) and Layne et al. (2008) were two of only four studies identified as meeting formal criteria for meta-analysis of intervention effectiveness (an analysis that was suggestive of the effectiveness of cognitive-behavioural and resilience-enhancing approaches on relief of post-traumatic symptoms). Otherwise, although reporting some promising findings, Peltonen and Punamaki (2010) suggest that little can be concluded overall from these studies in terms of intervention effectiveness as a result of methodological weaknesses, including the lack of randomization and control conditions. It is clear that implementation in conflict and postconflict settings means that such studies have had to deal with significant methodological challenges in relation to design and inference. For example, although the validity of deploying psychometric measures of well-being developed in other cultures and circumstances is frequently questioned, there are clear challenges for developing rigorous measures de novo in such settings (Ager, Stark, Akesson, & Boothby, 2010a). Promising approaches appear to be characterized by robust specification of replicable data collection and analytical methods which, through effective fieldwork engagement, nonetheless flexibly accommodate to local construction of concepts (e.g., Annan et al., 2006; Bolton et al., 2007). Study designs that provide some form of control or comparison condition – generally utilizing a population ‘waited listed’ for receipt of an intervention as a result of resource constraints – are also now increasingly seen as both feasible and appropriate in such settings (Ager, Ager, Stavrou, & Boothby, 2011). The study reported here sought to build upon such principles in constructing a rigorous methodology for evaluating the impact of the PSSA intervention as its implementation was scaled up in Gulu and Amuru Districts through 2007 to 2009. Methods Participants and setting As noted earlier, education authorities in Gulu and Amuru districts, in collaboration with Save the Children, had identified 40 schools significantly impacted by displacement of children following the conflict as eligible for physical infrastructure upgrading and potential curriculum support through programs such as PSSA. All 40 schools were physically upgraded, but funding and logistical concerns limited initial implementation of PSSA to 21 schools located across Gulu and Amuru’s four sub-districts. Eight intervention schools were selected for inclusion in the study – with a quota of two schools per sub-district – by toss of a coin. Comparison schools were selected from the remaining 19 schools meeting eligibility criteria for support (for all of which PSSA implementation was provisionally scheduled for the following year). Comparison schools were selected and matched with intervention schools on the basis of geographical location, which was seen as appropriate proxy for both exposure to conflict and local resource conditions. On this basis, one school was selected to serve as a comparison for two intervention schools (though different grades were drawn upon, see below). School names are not reported in this article to ensure confidentiality of study participants. From each of the eight intervention schools, the first group of 25 students selected to undergo the PSSA program in the school year 2007/08 was the focus of the evaluation. The program was targeted toward all children who had grown up in the context of stress and conflict, not only formerly abducted children or those with identified psychological difficulties. At each school a ‘rolling program’ of implementation by successive ‘batches’ of student groups was conceived, with all students potentially benefiting. However, in selecting the initial grade level for intervention, teachers were encouraged to consider the number of children in that year group who met certain criteria, such as those who isolated themselves, were frequently absent, seemed particularly stressed, had low self-esteem and/or a violent family background. These criteria were additionally used by teachers to select from the chosen grade the first ‘batch’ of students to receive the intervention. In the matched schools, children from the same grades were chosen for the comparison group, with similar criteria adopted to select specific participants (i.e., following intervention group selection criteria). In total, 403 primary school students took part in the study, 202 girls and 201 boys, with 203 in the intervention group and 200 in the comparison group. At baseline children ranged in age from 7 to 12 years with a mean age of 10.23 (sd = 1.61). Students from primary school year groups 1, 3, 4, 5, and 6 had been selected by schools to receive the intervention, which contributed to the wide range in ages participating in the evaluation. Intervention The intervention was structured according to the three PSSA components noted in the introduction. The classroom component involved 15 highly structured one-hour sessions delivered over the course of five weeks by two regular school teachers trained in the methodology in a residential workshop before the start of the school year. Sessions followed a regular structure of opening, thematic and closing activities, with thematic activities addressing issues of safety and control, self-esteem, thoughts and reactions during danger, resource identification and coping skills. Methods included didactic presentations, reflective exercises, drama and games. The community service component was facilitated by the same teachers, and included helping the sick and elderly, digging boreholes and planting trees. The parental engagement component  2011 The Authors. Journal of Child Psychology and Psychiatry  2011 Association for Child and Adolescent Mental Health. Psychosocial structured activities with conflict-affected children in northern Uganda was principally implemented through periodic discussion meetings with parents facilitated by Save the Children in Uganda (SCiUg) Child Resilience Project staff who made regular supervisory visits to implementation schools. In original program documentation these components had related to distinct program goals of ‘enhancing school performance’, ‘increasing participation in social activities and self-help community initiatives’ and ‘improving care and emotional support received in families’. However, through inception the program elements were increasingly seen as connected to the core goal of enhancing child well-being and ‘resilience’. With this construct the central focus of change, the study focused on operationalization of this concept. Measures Researchers adopted a modified form of brief ethnographic interviewing (Hubbard, 2008) to determine local understandings of child-well-being and resilience. Prior to the evaluation, each of the eight intervention schools was visited, and discussions held with children, parents (and other main caregivers) and teachers. In each school, children who had engaged in the first ‘batch’ of PSSA activities at the school (in academic year 2006/7) engaged in a ‘free-listing’ exercise, identifying characteristics of a resilient child who demonstrated good well-being (after the methodology outlined in Ager et al., 2011). Responses from the 120 children engaged in this manner were pooled and independently grouped by seven raters with local language and cultural knowledge. The preliminary categories of response were then, through discussion, consolidated into the six indicators listed in Table 1. An analogous procedure was followed with 136 parents from across the eight schools – and with 42 teachers during the course of their training and supervision in implementation of PSSA – to yield the list of indicators listed in Table 1. Data collection 1127 day for data collection at each school, the research team gathered the selected children outside or in an empty classroom. The purpose of the evaluation was discussed and the meaning of each of the indicators reviewed in Acholi. Children were reminded that participation in the study was voluntary, and asked if they were happy to proceed with the exercise. No children declined participation at this stage. To facilitate rating on the identified well-being indicators, a long roll of paper was used to represent a continuum for their scoring, marked with numbers from one (lowest) to ten (highest). The focus of each indicator was reviewed with raters before rating began (using pictorial prompts and ‘acting out’ of the indicator, as appropriate). Children were asked to rate themselves on that indicator by standing upon – or placing an object upon – the appropriate point of the continuum. Children did so in turn, in an area away from the group, so that ratings were made in private. A similar procedure was then followed with teachers, who completed the rating assessment for each of the selected children who were in their charge. Towards the end of the school day, the procedure was completed with parents (or other caregivers acting in the place of parents) of selected children. All responses were marked on a pre-coded response sheet. Participants were identified by school-based codes, with no names recorded on the response sheet. Participant codes and names were linked only by a master code list retained by the research team for use at follow-up. Follow-up data collection proceeded in an analogous fashion in all selected schools approximately 12 months later. Coding lists were used to match responses for participants from baseline to follow-up. Table 2 indicates the number of children and parents from whom data were collected at follow-up. Return of families to home communities with the gradual closure of IDP camps was a major factor influencing tracing of participants. Results Seven local researchers who lived in the Gulu area and spoke Acholi were hired and trained in participatory interview methodology and data collection. A schedule of visits to schools was confirmed with headmasters and other relevant local authorities. Informed consent was sought and received from parents, children and teachers participating in the evaluation by sharing a statement indicating that attending all meetings related to the assessment was voluntary and that there would be no penalty for non-participation. On the appointed Ratings on each of the six indicators were summed to produce composite scale scores of well-being by children, teachers and parents respectively (with individual scale scores ranging from a minimum possible score of 6 to a maximum possible score of 60). Internal consistency of scales was deemed adequate to consider scores as valid overall measures of well-being (see Shrout & Fleiss, 1979), with Cronbach a-coefficients ranging from a strong .79 for Table 1 Child, parent, and teacher-derived measures of child well-being Children Playful and social Interested in school, intelligent Happy Respectful and non-violent Responsible and hardworking Healthy Parents Teachers Strong and healthy Responsible and self-directed Intelligent Open and seeks support Happy and unstressed Playful and social Attends regularly Cooperative and sociable Interested in school, makes academic progress Speaks freely Obedient and respectful Responsible, helps in school and community  2011 The Authors. Journal of Child Psychology and Psychiatry  2011 Association for Child and Adolescent Mental Health. 1128 Alastair Ager et al. Table 2 Number of children and parents participating at baseline and follow-up Children: intervention Children: comparison Parents: intervention Parents: comparison Baseline Follow-up 187 180 118 117 105 106 84 86 teachers, through .70 for parents, to a marginal .67 for children ratings at baseline. At baseline there was no significant difference between intervention and comparison groups on age (F(1,401) = 1.79, p > .05) or gender (chisquare = .48, df: 1, p > .05). Children’s, parents’ and teachers’ ratings of children’s well-being at baseline are summarized in Table 3. This indicates no significant difference at baseline between intervention and comparison groups on parent-reported wellbeing (F(1,227) = .74, p > .05) but trends for the intervention group scoring lower on self-reported well-being (F(1,356) = 26.37, p < .001) and higher on teacher-reported well-being (F(1,347) = 9.81, p < .01). There was an upward trend between baseline and follow-up in both intervention and comparison schools, though larger changes in the former (see Table 3 and Figure 1). Analysis of the change in scale scores over time was completed using a fixed-effects linear model that included time, group assignment (i.e., intervention vs. comparison) and interaction terms between time and group. The model also accounted for the design effect of the place-based sampling in schools, and adjusted for age and gender. We fitted this linear mixed model with random intercepts for child and school to well-being as reported by children, parents and teachers. Full models (i.e., models including the interaction between group and time) were consistent in showing significant improvements in well-being over time. Well-being scores improved with time whether child well-being was self-reported (b = 1.7, se = .07, p < .05), teacher-reported (b = 3.0, se = .08, p < .001) or parent-reported (b = 4.4, se = 1.1, p < .001). For child-reported well-being, the model showed group assignment as a significant predictor (b = )9.5, se = 1.8, p < .001), as was age (b = .08, se = .04, p < .05), but not gender (b = .04, se = .06). For parent-reported child well-being, condition was not a significant predictor (b = –3.9, se = 2.4, p > .05), but both age (b = .7, se = .3, p < .05) and gender (b = 1.9, se = .8, p < .05) were. For teacher-reported child-well-being, neither condition (b = .5, se = 2.2, p > .05) nor age (b = ).4, se = .5, p > .05) were significant predictors, but gender was (b = 1.2, se = .6, p < .05). Overall this suggests a general trend for recovery and enhanced well-being over the period of the study, with some evidence (from parent and Table 3 Means and standard deviations of scale scores across conditions and time Baseline (T1) Child Scale Parent Scale Teacher Scale Intervention Control Intervention Control Intervention Control n Mean Std dev. n Mean Std dev. 180 178 115 114 176 173 36.8 40.8 37.2 36.8 35.4 33.0 7.3 7.4 7.7 8.6 7.0 7.3 105 106 83 86 98 107 44.1 43.0 45.3 41.3 40.6 35.7 5.6 6.5 7.1 9.0 7.4 7.8 50 44.1 45 40 40.8 Follow-up (T2) 45.3 43.0 41.3 37.2 36.8 36.8 40.6 35.4 35 35.7 33.0 30 25 20 15 10 5 0 Child Scale: PSSA Child Scale: Comp Parent Scale: PSSA Baseline Parent Scale: Comp Teacher Scale: PSSA Teacher Scale: Comp Followup Figure 1 Comparison of scale scores at baseline and follow-up  2011 The Authors. Journal of Child Psychology and Psychiatry  2011 Association for Child and Adolescent Mental Health. Psychosocial structured activities with conflict-affected children in northern Uganda teacher reports) of girls making greater progress than boys and (from self- and parent reports) of older children making greater progress than younger children. The interaction term between group and time in the models provides the clearest evidence of the influence of the intervention in this context of such recovery. This term was significant for both self-reported and parent-reported child well-being, indicating greater increases on these measures over time for children receiving the intervention (b = 5.4, se = 1.1, p < .001; (a) and b = 4.0, se = 1.5, p = .01, respectively). This interaction effect was not, however, found to be significant for teacher-reported child well-being (b = 2.0, se = 1.2, p > .1). These trends are presented in Figures 2a, 2b and 2c respectively. Given the large number of participants unavailable at follow-up (principally due to return migration), a bias analysis was conducted comparing those retained and those lost to the study at T2 on baseline scores at T1 (see Table 4). With respect to parent- and teacher-reported well-being there were Comparison 50 40 Child scale score 20 2 1 2 Time (Baseline vs. Follow-Up) Time (Baseline vs. Follow-Up) Intervention Comparison 40 30 20 20 30 40 Parent scale score 50 50 60 60 (b) 1 1 2 Time (Baseline vs. Follow-Up) Intervention Comparison 40 10 10 20 20 30 40 Teacher scale score 50 50 (c) 2 Time (Baseline vs. Follow-Up) 30 Parent scale score 30 40 30 20 Child scale score 50 Intervention 1 Teacher scale score 1129 1 2 Time (Baseline vs. Follow-Up) 1 2 Time (Baseline vs. Follow-Up) Figure 2 (a) Child scale scores from baseline (T1) to follow-up (T2) (b) Parent scale scores from baseline (T1) to follow-up (T2) (c) Teacher scale scores from baseline (T1) to follow-up (T2)  2011 The Authors. Journal of Child Psychology and Psychiatry  2011 Association for Child and Adolescent Mental Health. 1130 Alastair Ager et al. Table 4 Comparison of baseline (T1) scores for those retained and lost at follow-up (T2) Child Scale Parent Scale Teacher Scale Age Lost Retained Lost Retained Lost Retained Lost Retained Mean Std dev. 37.71 39.66 36.50 37.60 34.54 33.92 9.94 10.50 7.86 7.28 7.99 8.20 7.14 7.41 1.68 1.51 F(df1, df2), p-value 5.92 (1,356), p < .05 1.050 (1,227), p > .05 .611 (1,347), p > .05 12.69 (1,401), p < .001 no differences between those retained and those lost to the study. However, those retained were significantly older and had higher self-report well-being scores at baseline. There was no difference in the proportion of boys and girls retained at follow-up (chi-square = .948, df = 1, p = .33). Discussion Before the above findings are discussed in detail, limitations of study design and consequent constraints on interpretation need to be considered. Limitations and constraints were very much determined by the operational context of working in a post-conflict environment. First, the follow-up period of 12 months combined with teacher deployment policy in schools (teachers generally staying with the same year grade each year, rather than following a class through the school) resulted in it generally being a different teacher reporting on a child’s well-being at the two data collection points. It seems more likely that this issue reduced sensitivity of the study design to real impacts of the intervention, rather than biased findings. The failure for the trend (change in teacher-reported wellbeing over time) to reach statistical significance seems plausibly related to such insensitivity. Second, as noted in Table 2, over 150 children were not available at the time of follow-up data collection. Reports from teachers suggested a wide range of reasons for this, including temporary absence, marriage, and taking on work because of harsh economic conditions. Most students were reported not to be available, however, because of significant household return to home villages (with children potentially re-registering at their local school). The study had taken this expected return migration into account, with parental interviews at baseline including information to help trace the family in the event of a move, including the parent’s and child’s full name, the mobile phone number of a friend or relative, and their place of residence and village of origin. However, at the time of follow-up data collection, time and financial resources for tracing students were limited, and researchers were unable to follow up with many of the students who had moved or dropped out of school. The sample retained at follow-up was of the same gender balance as at baseline, and those retained were indistinguishable from those lost with regard to parent and teacher ratings of well-being. However, those retained were somewhat older and had higher selfreported well-being at baseline than those lost to the study. Along with potential variation on other characteristics not measured, this finding urges caution in the interpretation of results, although, with the design adopted, these relate more plausibly to issues of insensitivity rather than bias. A third major constraint concerns the lack of documentation regarding the reliability and quality of intervention. Schools in northern Ugandan districts at the time of the evaluation were challenging educational settings, with severe restrictions in funding, learning materials and human resources. In this context those who had been trained in PSSA techniques may not have been reliably able throughout the school year to deliver sessions as planned. Supervision from staff of the Child Resilience Project was not intensive, and although records of supervisory meetings, community activities, and parents’ meetings were kept, there was little reliable means of determining the compliance of PSSA sessions with the manual and training provided. Such factors would be a major threat to study validity if no impact of intervention was suggested (on the premise that a more robustly implemented intervention might have shown an impact). However, here a significant intervention effect was noted even with such constraints on implementation. Fourthly, impacts of intervention (and broader adjustment of children over time) are judged in this study purely on the basis of parents’, teachers’, and children’s ratings of well-being. The original study design had sought to map impact and adjustment also with respect to school attendance records and records of academic achievement (term grades). While such information was held in some schools, this was not reliably the case, and there was no uniform approach to documenting either attendance or achievement across schools. Notwithstanding these issues, our findings clearly support two contentions of major significance. Firstly, over the course of one year, during which significant efforts had been made to support processes of recovery and reconstruction in the area, there is evidence that parents, teachers and children themselves all considered that children’s well-being – as measured with respect to culturally salient indicators – had improved over this period. There was evidence on some indicators of older children and girls demonstrating greatest gains. Secondly, on self-report and parent measures children receiving interventions through the PSSA program showed significantly greater increases in well-being than those in comparison schools that did not receive this (or, indeed, any other) form of curriculum enrichment program.  2011 The Authors. Journal of Child Psychology and Psychiatry  2011 Association for Child and Adolescent Mental Health. Psychosocial structured activities with conflict-affected children in northern Uganda The former finding is coherent with the growing literature on resilience and recovery, which indicates that children’s well-being is strongly supported through the normalization of social conditions. This is a key construct of the IASC Guidelines on Mental Health and Psychosocial Support in Emergency Settings (IASC, 2007), and of conceptual analyses of recovery mechanisms from the Psychosocial Working Group (PWG, 2002, 2003; Boothby, Wessells, & Strang, 2006) from which such guidelines heavily draw. Increasingly, field studies are documenting this dynamic between social reconstruction and personal well-being (Ager, Strang, & Abebe, 2005; Ager, Stark, Olsen, Wessells, & Boothby, 2010b). Such literature has a major impact on our necessary understanding of the role and value of interventions in post-conflict contexts. If broader social reconstruction and adaptation predicts incremental recovery in personal well-being of children and youth, interventions need to be justified on the basis of their acceleration of such processes. In this study there is indeed evidence that exposure to the PSSA intervention had an additional benefit for children, which accelerated further the processes of recovery associated with broader social and economic reconstruction. From the design of the current study it is not possible to say what components of the PSSA intervention were most active in promoting such acceleration. The classroom-based element of programming appeared to receive the greatest emphasis in PSSA implementation in the Child Resilience Project in Gulu and Amuru, but the impact of non-specific features of the intervention (such as the provision of training, materials and supervision) cannot be ruled out as key forces in driving change in such resource-poor educational environments. Future evaluations of the PSSA program might usefully examine the potential contribution and impact of different components of the intervention (e.g., the classroom-based elements with respect to the parent- and community-focused ones). Such designs would usefully incorporate stronger monitoring procedures regarding the reliability of delivery of specific elements of the intervention than adopted here. Finally, the study highlights both the opportunities and challenges for methodological development in post-conflict contexts. The use of free-listing as a basis for deriving local indicators that proved a reliable and sensitive basis for changes in perceived well-being over time is clearly promising, and is consistently emerging as robust basis for evaluation of program outcomes in diverse cultural contexts (Bolton et al., 2007; Ager, Stark, & Potts, 2010c). The use of a ‘wait list’ control, as in the current study, also offers the potential for more rigorous compara- 1131 tive designs and analysis than is typically available through standard pre- post- evaluation designs (Bamberger, Rugh, & Mabry, 2006; Jordans et al., 2010). However, the fact that in the circumstances described here difficulties in securing resources led to the indefinite postponement of the ‘roll-out’ of the PSSA intervention to comparison communities alerts to ethical challenges in this approach in unstable contexts. Conclusion Increases in child well-being as reported by parents, teachers and children themselves indicate that children in Gulu and Amuru districts are returning to a stronger sense of well-being and resilience. In the context of this general recovery, exposure to the PSSA program appears to have accelerated increases in reported well-being in recipients of the intervention, as noted by both parents and children themselves. The research has demonstrated the potential for rigorous approaches to impact evaluation – including use of locally defined indicators – in unstable post-conflict contexts. Author note Braxton Okot is currently Protection Officer, Southern Sudan, World Vision International. Acknowledgements The researchers would like to thank the Child Resilience team of Save the Children in Uganda’s Gulu office for their support in conducting this evaluation. Jimmy Duong, Biostatistician with the Irving Institute for Clinical and Translational Research, Columbia University (supported by grant UL1 RR024156 from the National Center for Research Resources) assisted with statistical analysis. The assistance of Carolyn Bancroft and Wendy Ager in preparation and review of the manuscript is also gratefully acknowledged. The Program on Forced Migration and Health acknowledges the financial support of USAID (Displaced Children and Orphans Fund), the Oak Foundation and the United States Institute for Peace for its work on child care and protection in crisis-settings. Statements made in this paper are the views of the authors alone, and do not constitute the policy of the above-listed funding bodies. CUMC IRB approval reference: AAAB7134. Correspondence to Alastair Ager, Program on Forced Migration and Health, Mailman School of Public Health, Columbia University, 60 Haven Avenue, B-4, Suite 432, New York, NY 10032, USA; Tel: 212-342-5215; Fax: 212-342-5222; Email: aa2468@columbia.edu  2011 The Authors. Journal of Child Psychology and Psychiatry  2011 Association for Child and Adolescent Mental Health. 1132 Alastair Ager et al. Key points • Post-conflict recovery processes appear to be supporting an improved sense of well-being in children in northern Uganda. • The Psychosocial Structured Activities program had a positive effect on the well-being of children in northern Uganda additional to such general recovery processes. • Participatory focus group methodology using free-listing and thematic analysis was effective in developing local measures of child well-being of acceptable internal consistency. • Despite the constraints of post-conflict settings, it is possible to collect valuable data on the impact of programs targeting improvements in children’s well-being. References Ager, A., Ager, W., Stavrou, V., & Boothby, N. (2011). Interagency guide to the evaluation of psychosocial programming in emergencies. New York: UNICEF. Ager, A., Stark, L., Akesson, B., & Boothby, N. (2010a). Defining best practice in care and protection of children in crisis-affected settings: A Delphi study. Child Development, 81, 1271–1286. Ager, A., Stark, L., Olsen, J., Wessells, M., & Boothby, N. (2010b). Sealing the past, facing the future: An evaluation of a program to support the reintegration of girls and young women formerly associated with armed groups and forces in Sierra Leone. Girlhood Studies, 3, 70–93. Ager, A., Stark, S., & Potts, A. (2010c). Participative ranking methodology: A brief guide: Version 1.1. Program on Forced Migration & Health, Mailman School of Public Health, Columbia University. Ager, A., Strang, A., & Abebe, B. (2005). Conceptualising community development in war-affected populations: Illustrations from Tigray. Community Development Journal, 40, 158–168. Annan, J., Blattman, C., & Horton, R.. (2006). The state of youth and youth protection in northern Uganda: Findings from the Survey for War Affected Youth. Survey for War Affected Youth, Kampala, UNICEF Uganda and AVSI Uganda. Bamberger, M., Rugh, J., & Mabry, L. (2006). RealWorld evaluation: Working under budget, time, data, and political constraints. Thousand Oaks, CA: Sage. Berger, R., Pat-Horenczyk, R., & Gelkopf, M. (2007). Schoolbased intervention for prevention and treatment of elementary students’ terror-related distress in Israel: A quasi-randomized controlled trial. Journal of Traumatic Stress, 20, 541–551. Bolton, P., Bass, J., Betancourt, T., Speelman, L., Onyango, G., Clougherty, K., Neugebauer, R., Murray, L., & Verdeli, H. (2007). Interventions for depression symptoms among adolescent survivors of war and displacement in northern Uganda: A randomized controlled trial. Journal of the American Medical Association, 298, 519–527. Boothby, N., Wessells, M., & Strang, A. (Eds.). (2006). A world turned upside down: The social ecology of children and war. Bloomfield, CT: Kumarian Press. Chase, R., Doney, A., Sivayogan, S., Ariyaratne, V., Satkunanayagam, P., & Swaminathan, A. (1999). Mental health initiatives as peace initiatives in Sri Lankan school children affected by armed conflict. Medicine, Conflict and Survival, 15, 379–390. Civil Society Organizations for Peace in Northern Uganda (CSOPNU). (2006). Counting the cost: Twenty years of war in northern Uganda. Kampala: Civil Society Organizations for Peace in Northern Uganda. Government of Uganda & Lord’s Resistance Army/Movement. (2006). Agreement on cessation of hostilities between the Government of the Republic of Uganda and the Lord’s Resistance Army/Movement. August 26. Juba, Sudan. Retrieved April 26, 2010 from http://northernuganda.usvpp.gov/uploads/images/hUgm3ncZ_oIetYG7u_CjFQ/ agreement.pdf. Hubbard, J.. (2008). Using brief ethnographic interviewing as a method for understanding an issue, problem or idea from a local perspective. Unpublished manuscript. Minnesota: Center for the Victims of Torture. Inter-Agency Standing Committee (IASC). (2007). IASC guidelines on mental health and psychosocial support in emergency settings. Geneva: IASC. Jaycox, L.H., Morse, L.K., Tanielian, T., & Stein, B.D.. (2006). How schools can help students recover from traumatic experiences: A tool kit for supporting long-term recovery. Technical Report, Gulf States Policy Institute, Rand Corporation, Santa Monica, CA. Jordans, M.J.D., Komproe, I.H., Tol, W.A., Kohrt, B.A., Luitel, N.P., Macy, R.D., & de Jong, J.T.V.M. (2010). Evaluation of a classroom-based psychosocial intervention in conflict-affected Nepal: A cluster randomized controlled trial. Journal of Child Psychology and Psychiatry, 51, 818–826. Khamis, V., Macy, R., & Coignez, V. (2004). The impact of the Classroom/Community/Camp- Based Intervention (CBI) program on Palestinian children. Jerusalem: Save the Children USA. Lauten, A.W., & Lietz, K. (2008). A look at the standards gap: Comparing child protection responses in the aftermath of Hurricane Katrina and the Indian Ocean tsunami. Children, Youth and Environments, 18, 158–201. Layne, C.M., Pynoos, R.S., Saltzman, W.R., Arslanagiz, B., Savjak, N., Popovic, T., Durakovic, E., Music, M., Campara, N., Djapo, N., & Houston, R. (2001). Trauma/grief-focused group psychotherapy: School-based postwar intervention with traumatized Bosnian adolescents. Group Dynamics: Theory, Research and Practice, 5, 277–290. Layne, C.M., Saltzman, W.R., Poppleton, L., Burlingame, G.M., Pasalic, A., Durakovic, E., Music, M., Campara, N., Dapo, N., Arslanagić, B., Steinberg, A.M., & Pynoos, R.S. (2008). Effectiveness of a school-based group psychotherapy program for war-exposed adolescents: A randomized controlled trial. Journal of the American Academy of Child and Adolescent Psychiatry, 47, 1048–1062. Loughry, M., Ager, A., Flouri, E., Khamis, V., Afana, A., & Qouta, S. (2006). The impact of structured activities among Palestinian children in a time of conflict. Journal of Child Psychology and Psychiatry, 47, 1211–1218. Machar, R.. (2008). Report and recommendations of the chief mediator of the peace process between the Government of the Republic of Uganda and the Lord’s Resistance Army. June 16. Available from: http://www.beyondjuba.org/ peace_agreements/Machar_Report_1.pdf. Peltonen, K., & Punamaki, R. (2010). Preventive interventions among children exposed to trauma of armed conflict: A literature review. Aggressive Behavior, 36, 95–116. PWG. (2002). Psychosocial intervention in complex emergencies: A conceptual framework. Psychosocial Working Group  2011 The Authors. Journal of Child Psychology and Psychiatry  2011 Association for Child and Adolescent Mental Health. Psychosocial structured activities with conflict-affected children in northern Uganda Working Paper. Queen Margaret University College, Edinburgh. Available from: http://www.forcedmigration.org/ psychosocial/papers. PWG. (2003). The vision of the PWG: Learning, practice and policy development to meet psychosocial needs in complex emergencies. Psychosocial Working Group. Queen Margaret University College, Edinburgh. Available from: http:// www.forcedmigration.org/ psychosocial/papers. Shrout, P.E., & Fleiss, J.L. (1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86, 420–428. Somasundaram, D. (2007). Collective trauma in northern Sri Lanka: A qualitative psychosocial- ecological study. International Journal of Mental Health Systems, 1, 5. Tol, W., Komproe, I., Susanty, D., Jordans, M., Macy, R., & de Jong, J. (2008). School-based mental health intervention for children affected by political violence in Indonesia: A cluster 1133 randomized trial. Journal of the American Medical Association, 300, 655–662. Wilson, A.K. (2007). Evaluation of Save the Children in Uganda’s pilot phase of the Child Resilience Program in Schools in northern Uganda. New York: Program on Forced Migration and Health, Columbia University. Woodside, D., Santa Barbara, J., & Benner, D.G. (1999). Psychological trauma and social healing in Croatia. Medicine, Conflict and Survival, 15, 355–367. World Health Organization. (2005). Health and mortality survey among internally displaced persons in Gulu, Kitgum and Pader districts, northern Uganda. Geneva: Author. Accepted for publication: 5 January 2011 Published online: 26 May 2011  2011 The Authors. Journal of Child Psychology and Psychiatry  2011 Association for Child and Adolescent Mental Health.
P8640 Methods in Program Evaluation Session 7: In-Class Exercise Part A Review the following article: Ager, A., Akesson, B., Stark, L., Flouri, E., Okot, B., McCollister, F., & Boothby, N. (2011). The impact of the school‐based Psychosocial Structured Activities (PSSA) program on conflict‐ affected children in northern Uganda. Journal of child psychology and psychiatry, 52(11), 11241133. Answer the following questions: 1. What were the original program goals for the intervention? 2. Sketch out a bare-bones logic model summarizing the inputs, activities, and outcomes. For the purposes of this exercise, skip the outputs and impacts. 3. How did the authors measure and operationalize their primary outcome? What were the indicators that were used? 4. What is the study design used? 5. What are the threats to internal validity common in this design? 6. Complete the table of participants. Baseline Children Follow-up % LTF Intervention 1 Comparison Parents Intervention Comparison 2 Session 7: In-Class Exercise Part B Read the case study: WVI Child-Friendly Spaces (CFS) Intervention at Buramino Camp, Dolo Ado, Ethiopia: Buramino refugee camp, located along the southern Somali-Ethiopian border, was selected in cooperation with World Vision Ethiopia as the first evaluation site. At the time of the evaluation, the Ethiopian Administration for Refugee and Returnee Affairs (ARRA), the main implementing partner of UNHCR, had organised five camps within two hours of the Dollo Ado transit centre to address the growing needs of Somali refugees who had migrated in response to drought and ongoing conflict in the region. In January 2012, World Vision Ethiopia’s first Child Learning Centre (subsequently referred to through this report as a CFS site) was established, offering a range of services to children aged six to 17 years. The second CFS site started activities later that year, in May 2012. Within the wide continuum of CFS programming approaches, these two sites emphasised functional literacy and numeracy skills. Based on discussions with ARRA and parents of children attending these centres, psychosocial activities, such as drawing, singing, and recreational play, were included, but to a modest degree. On-site counselling and a feeding programme were also offered. To help meet the overwhelming demand in the camp, each centre provided two three-hour sessions for children: a morning session for younger children aged six to 11 and an afternoon session for older children aged 12 to 17. The first CFS provided services for 784 children (500 aged between six and 11 and 284 aged between 12 and 17). The second CFS site provided services for 787 children (506 aged between six and 11 and 281 aged between 12 and 17). Source: http://www.cpcnetwork.org/wp-content/uploads/2015/04/WV-CU-CFS-Ethiopia-FieldStudy-Summary-Report.pdf Answer the following questions: 1. Develop one evaluation question 2. Consider what study design would best answer your question 3. Decide what data you need to collect 4. Recommend one random and one non-random sampling scheme. (circle your choices) Types of random sampling Types of non-random sampling Simple random sample Quota sampling Systematic Extreme case sampling Proportionate stratified Maximum variation sampling 3 Disproportionate stratified (Cluster sampling) Homogeneous sampling Typical case sampling Snowball sampling Criterion sampling 5. Discuss the advantages and disadvantages of each scheme that you choose. 6. Why did you choose the random sampling scheme you did? 7. What aspect(s) of the evaluation question would this scheme help you answer? 8. Why did you choose the non-random sampling scheme? 9. What aspect(s) of the evaluation question would this scheme help you answer? 4

This question has not been answered.

Create a free account to get help with this and any other question!

Similar Questions
Related Tags

Brown University





1271 Tutors

California Institute of Technology




2131 Tutors

Carnegie Mellon University




982 Tutors

Columbia University





1256 Tutors

Dartmouth University





2113 Tutors

Emory University





2279 Tutors

Harvard University





599 Tutors

Massachusetts Institute of Technology



2319 Tutors

New York University





1645 Tutors

Notre Dam University





1911 Tutors

Oklahoma University





2122 Tutors

Pennsylvania State University





932 Tutors

Princeton University





1211 Tutors

Stanford University





983 Tutors

University of California





1282 Tutors

Oxford University





123 Tutors

Yale University





2325 Tutors