Designing Program Evaluations - Essay Example

DESIGNING PROGRAM EVALUATIONS There are 2 types of program evaluations 1 formative 2 summative evaluations Formative: focus on description than on causality the interpretive approach is more appropriate Summative -determines the extent to which the goals of a program were met A positivist approach is often adopted Causality demands 3 conditions are met: 1 the cause precedes the effect 2 the cause and effect are related to each other 3 this relationship cannot be accounted by other relationships It is usually demanded by funding organizations. Certain aspects of summative evaluations establish this causality e.g. timing of data collection and comparison groups. EXAMPLE OF CASE to be used We will assume that a program that offers high-risk adolescents a series of birth control classes to increase knowledge of birth control practices. The evaluation is to evaluate the effectiveness of the program. Different types of designs will have to be considered. FORMATIVE EVALUATIONS These are in general very descriptive and so can offer more details about the program strengths and weaknesses. The interpretive approach (using qualitative data) is useful for this type of evaluation. Example: Weissman and Larue (1998) evaluating a program serving students whose are in prison. They looked at the outreach and programmatic approaches that were put in place to handle the special needs of the population of the program. They found that the holistic, multifaceted, and open approach used in the program to identify, assess, and meet the needs of the students was beneficial (87). In the birth control program a formative evaluation would be used to find out how the adolescents experienced the program. What did they perceive as its limitations and strengths? It can also find out how the parenting classes were run, the syllabus developed and if it is followed. In formative evaluations: no attempt is made to establish causality no attempt is made to generalize findings ( no dependent and independent variables) are focused on in-depth description and analysis as a means of improving and strengthening the program. Emphasis is on quality assessment Formative evaluations can be strengthened by comparing a number of factors e.g. male/female, ethnic groups, socioeconomic groups etc. They are useful at the beginning of a program like in the first year and the feedback can lead o improvement and growth and lay ground for summative evaluations. SUMMATIVE EVALUATIONS These are primarily concerned with causality and so validity is a central issue. There are 2 issues of validity considered: 1. Internal 2. And external validity Internal validity: The extent to which changes in the dependent variables are a result of the introduction of the independent variables and no others. E.g. was the knowledge of birth control a result of the adolescents’ participation in the classes or were other factors responsible for the increase in knowledge? To ensure internal validity in this case the 3 conditions of causality have to be confirmed? Fig 6.1 It is important to ensure equivalency in the groups being studied i.e. in the experimental group (receiving the intervention) and the control group (receiving no intervention). The participants with the same qualities are chosen then distributed randomly (random assignment). This means any resulting difference will be due to the intervention but not differences in the participants in either group. NOTE: Random assignment is different from random sampling which is the creation of a sample from a sample and has nothing with assigning experimental and control groups. Both, may or not be used in the study. External validity Refers to the extent to which results can be generalized to the wider population. Relates to the effectiveness with other similar client systems. Generalizability has 2 other conditions it depends on ensure 1. The equivalency of the groups 2. Ensuring that nothing happens to change the equivalency in the groups Ensure equivalency: To you decided to assign adolescents not eligible for classes into the comparison groups then it means that the results will lack internal validity and would be generalized only to a small group i.e. those taking or are eligible for the classes. No interference (treatment diffusion). This can occur when 1. The boys discuss the class with their peers some may be from the control group. 2. the program is not clearly defined leading to difficulties distinguishing the program group from the comparison group 3. reactivity effects occur because participants are aware they are in a study TYPES OF SUMMATIVE PROGRAM EVALAUTION DESIGNS Pre-experimental designs: it favors the use of comparison groups than control groups has limited internal and external validity A. ONE –GROUP POST-TEST ONLY DESIGN (ONE SHOT CASE STUDY) Has only one group with one point of data collection (after the intervention). It is also known as one-shot study. Fig 6.2 Program/Intervention post-test Data collection can be done through behavior observation or questionnaire. It can be useful in studies to see if a program is functioning. Often used for client satisfaction surveys where clients say how they perceived the program. It is limited for establishing that a program caused an outcome and if the results can be generalized other population. EXAMPLE: (1999) Threats to Internal validity With the post –test design threats are 1. Selection 2. History 3. Mortality Selection: e.g. the clients that enroll may be highly motivated to learn about birth control. If no pre-test is done to check this predisposition then internal validity is threatened. History Involves those events that could have an effect on the outcome e.g. high levels of knowledge about birth control could be from classes from school or other factors. A comparison group can be used to assess this. Mortality: This happens when participants drop out of their groups e.g. some adolescents may take one class and then drop out. The group that undergoes post-test is biased and may show higher success ha could be with those that dropped. Threats to external validity 1. selection- treatment interaction 2. History treatment interaction Selection- treatment interaction: It occurs when the ability to generalize is limited because the sample is not randomly selected or there is no pre-test e.g. in the example, the adolescents may all be highly motivated to learn before enrolling so it is irrelevant whether they complete classes or not. History treatment interaction: This happens when other factors are the cause of the outcome; this means the results may not be easily generalized e.g. in the example a positive result could be sue to a media campaign on pregnancy prevention and not the program. B.ONE GROUP PRE-TEST/POST-TEST DESIGN The same as the one group post test design except that it adds pretest e.g. in our example, all the clients will be given a questionnaire on their knowledge of birth control before the classes. Designing the evaluation in this way helps measure how well the clients are functioning at the end if minimum standards of results are met how participants change during the program It is useful to use this method when no comparison group is possible. It can enhance validity because the pre-test helps to rule out selection as an explanation of the outcome. Pre-test program/ intervention post-test Hurd (1998) studied strengths-based teaching in social work. To find changes in students’ perceptions of their professional strengths, a pre-test was given on the first day of class, and a post-test on the last day of class. The result was that students felt more competent at the end of the classes and that the results were statistically relevant (93). Threats to Internal validity 1 History 2. Maturation 3. Testing 4. Instrumentation 5. Regression to the mean interaction of selection and other threats History: The lack of a comparison group means there is no way to tell if other factors influenced the outcome. Maturation: Changes may be attributable to lifelong learning rather than effects of the program. This is especially so with adolescents. Maturation is an important factor if the clients are young or if the time between the pre-test and the post test is long. Testing When participants are exposed to a measuring instrument more than once e.g. if the pre-test could increase the adolescents’ knowledge of birth control, an effect which cannot be separated from that of the classes. A comparison group can help to control for this kind of effect. Instrumentation This refers to any changes in the way that the variables are measured. E.g. the context in which a questionnaire is administered can change as well as the person administering it. Regression to the mean In our example, if eligibility for birth control classes was through a test, then a post-test after may show regression to the mean. Regression to the mean means the results is close to the average. Therefore any change would not be from the program but regression to the mean. Interaction of selection and other threats. Even if none of the above threats are applicable to the general population, the threats may be relevant to for those subjects selected to participate in the study .e.g. adolescents who express a desire to receive more information through counseling, however may also be more likely to become more knowledgeable because of their age. This is the interaction of selection and other threats – which here is maturation. Threats to external validity 1. History –treatment interaction 2. Reactive effects Reactive effects refer to when participants change their behaviors because they know that they are in a study. The results will be distorted and won’t be generalized to the wider population. It is hard to avoid because consent must be sought. C.STATIC –GROUP COMPARISON DESIGN It is an extension of the post-test only design and it includes a comparison group .some strategies can help to get equivalency for the comparison group. E.g. Baseline comparison, matching, cohort groups and overflow comparison. Itzhaky (1995) compared two types of community organizations in Israel. One was a community centre supervised by an interdisciplinary committee and social worker and focused on the participation of the clients. The other had decisions made by the staff and management with no social worker role. The clients from the two centers were compared (95). Baseline comparison: Used when the comparison group is composed of cases handles before the program. One problem with this is that it is difficult to determine is cases seen as eligible in absence of a program would have been referred to the program. Program/intervention Post-test Comparison group Post test Matching: This is the selection of characteristics considered to have important impact on outcomes – like gender and ethnicity and to ensure they are represented in each group. It is equivalent to quota sampling and can also be combined with random assignment. This combination is equal to stratified random sampling. But matching has the same problems as stratified quota sampling. It is important also to ensure that the variables being chosen are key, which is difficult if there is lack of previous research or other information sources. Example: Manion, Firestone, Cloutier, Ligezinska, McIntyre, and Ensom (1998) matched case families with comparison families on the sex and age of the child (within 6 months) and, where possible, the family constellation ( single-/two-parent family) to find out the emotional and behavioral adjustments of parents and children within 3 months and 1 year of the exposure of extra familial sexual abuse. The children in the comparison families had never experienced sexual abuse according to parents and /or the child. The results showed that both parents and children of case families felt significant effects both initially and 1 year after exposure. Children had self-blamed and guilt for the abuse and the degree of traumatization predicated their symptomology 3 months and 1 year post exposure. Child age and gender also affected the prediction of child outcome measures. For mothers, satisfaction in the parenting role perceived support, and intrusive predictors of emotional functioning was predicted by avoidant symptoms, child’s internalizing behavior and mother’s initial emotional functioning. (96) Cohort groups: It is another strategy for compiling comparison groups. These are individuals who move through an organization at the same time as those in the program but do not receive programs services. E.g. you can compare adolescents in the same class at school- some are enrolled in birth control class and some are not. Overflow comparison: People referred to a program that are put on a waiting list (because it is full) form this group and can serve as a comparison group compared to single- group designs the threat from history does not exist as the effect will occur in both groups. Threats to internal validity 1. Selection 2. Mortality Selection: Arises from not randomly assigning the groups and having no pretest. It is then hard to tell how similar the groups are to each other. Mortality This is because of the lack of a pretest and of a randomly assigned comparison group. Threats to external validity 1. selection-treatment interaction 2. Reactive effects D.QUASI-EXPERIMENTAL DESIGNS These designs eliminate most of the threats to internal and external validity than pre-experimental designs. They use comparison groups rather than control groups and so will have limited external and internal validity. E. TIME SERIES DESIGNS Measures several times before and after the intervention so can overcome the problem of the designs above. So can overcome the problem of the designs above e.g. in our example we can test the adolescents several times in several moths before the classes and the same test is given several times after or even during the classes. One advantage is the ability to show trends in the data before and after the intervention. And this helps to take care of such problems as maturation, testing and instrumentation found in single pretest/ posttest design. Example: Nugent, Bruley, and Allen (1999) set to test the effectiveness of the Aggression Replacement Training (ART) on male and female antisocial behavior at a runway shelter. The checked the records of 522 adolescents who stayed at the shelter were assessed through antisocial behavior measures for 310 days before starting the program and then for 209 days after the program. The results showed that ART can be useful used with other approaches in reducing juvenile antisocial behavior in a short term residential setting (98). Pretest pre-test pre-test Program/intervention Posttest post-test posttest Threats to internal validity 1. History History is a threat because of lack of a comparison group. Threats to External validity 1. history- treatment interaction 2. reactive effects History-treatment interaction: From history reacting with the classes Reactive effects From repeated testing F. PRETEST/POSTTEST COMPARISON –GROUP DESIGN A combination of the static- group comparison and the one-group pre-test/post-test design. Pretest program /intervention posttest Pretest comparison group posttest G. PRETEST/POST TEST COMAPRAISON GROUP DESIGN Kramer (1998) compared two groups of students in order to find out the level of their death acceptance and sense of preparedness to respond to personal and professional losses. One group was enrolled in a Grief, Death, Loss and Life course and the other in other courses. He gave a pretest the first day of class to both groups and a posttest on the last day. The group ha took the grief course felt more competent in their knowledge, skills, and sense of preparation for working with grieving clients. This group also showed increased cognitive and affective dimensions of death acceptance (99). Because it combines static-group comparison and one group pretest/posttest, this design becomes less problematic than wither of hem. History is not a problem because of the comparison group and differences and similarities between the groups are identified through the pretest. Threats to internal validity 1. Selection and maturation interaction: E.g. in our example increased knowledge in the treatment group may be due to maturation and not to the classes. Threats to external validity: 1. selection-treatment interaction This will affect the generalizability of the results. 2. maturation-treatment interaction 3. reactive effects H. EXPERIMENTAL DESIGNS These results have the strongest causality claims. I. PRETEST/POSTTEST CONTROL- GROUP DESIGN Different from pretest/posttest comparison group design because the comparison group and experimental group are randomly assigned. This results in a control group. These kinds of selections eliminate threats to internal validity like history, maturation, mortality, selection, regression to the mean testing and instrumentation. There is only one threat to external validity left: the reactive effect of the pretest. However, this design faces problems of implementation like treatment diffusion and lack of a list of clients for the random assignment. Pretest experimental group posttest Random assignment Pretest control group posttest Example Children with learning disabilities area t risk for poor peer relationships even in mainstreamed classrooms. The program to be evaluated was designed to work with fifth grade learning disabled (LD) children to improve their acceptance by their non learning disabled fifth grade peers (NLD). Hepler (1997) compared pretest and posttest of both LD and NLD children randomly assigned to a program group and to a no- treatment (control) group. The results were a good feedback for the implementation of programs to help LD children increase their social skills and acceptance by their peers (101). J. POSTTEST-ONLY CONTROL- GROUP DESIGN By eliminating the pretest, the threat to external validity seen in the last design can be eliminated. The 2 groups are randomly assigned and they are assumed to be equivalent so no need for pretests. Some researchers are not comfortable skipping this step which is a safety measures. Example: Reid and Bailey-Dempsey (1995) examined the effect of monetary incentives on school performance. Teenage girls at risk of school failures wee randomly assigned to a control group or t one of two experimental groups: one experimental group offering monetary incentives and the other a case management program. Posttest measures included GPAs and absences (101). Experimental group posttest Random assignment Control group posttest K.THE SOLOMON FOUR-GROUP DESIGN It combines the last two designs and is very valid. It is difficult to find enough subjects to assign randomly into the groups and the costs are too high for most social work program evolutions; so it rarely used in social work. Pretest experimental group posttest Pretest control group posttest Random assignment Experimental group posttest Control group posttest Type of design Threats to internal validity Threats to external validity One –group posttest -only Selection, history, mortality Selection-treatment interaction, history- treatment interaction One-group pretest/ posttest History, maturation, testing, instrumentation, regression to mean, interaction of selection, and other threats History-treatment interaction, reactive effects Static-group comparison Selection and mortality Selection-treatment interaction Time series History History-treatment interaction, reactive effects Pretest/posttest comparison Selection and maturation Selection-treatment interaction, maturation-treatment interaction, reactive effects Pretest/protest control group None Reactive effects Posttest-only control group None None Solomon four-group None None THE AGENCY AND PROGRAM EVALUATION DESIGN Although experimental designs with randomly assigned control groups are better for establishing if program intervention are causes of the outcome, it may not always be practical for generalist social workers. Other designs may give better answers. It is important to develop designs that are feasible and appropriate for the research question. For example: 1 alternative ways of setting up comparison groups 2 uses of unbalance designs with fewer subjects to the comparison/ control group so clients referred are more likely to receive services 3. Formative evaluations can be used for new programs 4. To consider design drawbacks and include them in evaluation report. ETHICAL ISSUES IN PROGRAM EVALUATION DESIGNS There are 2 issues 1. Are clients denied services to set the control or comparison group? 2. Should the subjects consent be obtained before setting the comparison group? Assignment to the comparison/control group: The NASW Code of Ethics (1997) social workers should ensure that participants in evaluation and research have access to supportive services. Comparison/ control groups do not always cause an ethical problem because 1. Decision on who receives the services is arbitrary and political 2. If it is suggested that clients are denied services then it is assumed that the intervention is effective though there may be no empirical basis. Other factors as values and individual judgments may play a role. Decisions may be influenced by the seriousness of the problem e.g. it is hard to justify a control group of emotionally disturbed children involved in self- destructive behaviors. Using waiting lists, cohort groups, baseline comparison and assignment to other types of intervention or program can help diminish the bad effects of being in the control group. Informed consent: Subjects have to know their role and the consequences of participating in the research and their consent should be obtained. The NASW Code of Ethics (1997) 1. Social workers should get voluntary and written informed consent from participants without implied or actual deprivation, with no inducement, and with regards for participants’ wellbeing, privacy, and dignity. The informed consent should include the nature, extent and duration of the participation requested and disclosure of the risks and benefits of participation in the research. 2. If research participants cannot give consent it should be obtained (written) from a proxy after explaining to the participant all the details. 3. Participants should know of their right to withdraw at anytime without penalty. 2 Issues: A) Informed consent is difficult with comparison groups because it is difficult to explain to participants that they will be missing out on the service. It is not a problem with control groups where subjects are randomly assigned to the control or experimental group so that they can be informed as to whether they will be getting the service or not. However, consent should be obtained at all times. By adopting alternative strategies like waiting lists, other programs, the researchers will feel free to inform subjects of their position and to obtain their consent. B) By informing the subjects of the details of the study may impact on the validity o f the findings because when subjects are aware which group they are in, they ay react to the expectations of the group e.g. the experimental group may show improvement which may not come from the program. Since consent must still be obtained one way is to explain the study but not to tell the subjects which group they will be in but that they will be in either a control/comparison group or experimental group. But this is only useful if the control/comparison group is getting some intervention. If intervention to the control/comparison is not possible then reactive effects need to be accounted for. HUMAN DIVERSITY ISSUES IN PROGRAM EVALUATION To ensure that certain groups are not exploited for the purpose of forming the comparison groups. It is common in social science research to use disadvantaged groups in comparison groups: women, minorities, poor etc. Parlee (1981) points out that in psychology research; the choice of a comparison group shows the “implicit theoretical framework” of the researcher. The same can be said of social research. She says these frameworks are biased against women. She says by “knowing” what variables to include means biases favoring some groups over others and in turn influencing the findings. It is also a problem when matching. It is important to involve diverse people in the planning of research if the findings are to have impacts on diverse populations. So, alternative viewpoints and perspectives should be included in the design. The Parlee (1981) study in which matched comparison groups of women was developed for a 20 year old men only study of aging. The women could be matched by intelligence, education, and occupation; or by physiological similarities. The first choice is a social science perspective and the second biomedical science. Read More