Designing and Preparing the Evaluation - Assignment Example

 Designing and Preparing the Evaluation Chapter 5-Focus on evaluation Developing an evaluation plan The marketing training program is meant to improve the performance of employees and hence increase productivity. In developing a plan for this program, the following steps must be followed: Preparation This step lays a basis for the strategic planning procedure by determining the aim of the program. It involves identification of stakeholders, establishing the kind of information necessary, the resources, and roles needed for the process, and creating the timeline for this. The products of this step are the modeling of a tactical planning workforce and identifying data necessary to tell the procedure of strategic planning. Assessment The “asses” step is the procedure through which the strategic workforce evaluates and examines program-related data so that the program can assign sources and services in the most appropriate way. In this step, one determines where the program presently is. The result of the Assessment step is an analysis of the SWOTs (Strengths, Weaknesses, Opportunities, and Threats) as per the data analysis. Create the plan Here, a five-year strategic plan should be developed and written. The strategic planning workforce analyses the SWOTs and uses the results to make out and prioritize policies that the program aims to implement during a stated work frame agreement. I will then revise my plan logic model and line up my yearly work plan with the prioritized policies and the timeline to be implemented. The key product of this step is the documented strategic plan. Of the most important sections of the “create” step are the executive summary, SWOT analysis, stakeholder list, program strategies, data sources list, aligned yearly work plan, implementation process, communication process, and revised five-year strategy. Communicate the strategic plan The step involves sharing out data concerning the plan in a manner that make the plan easily understood and essential for stakeholders. The results of the “communicate” step include the communication notes and products that I distribute annually about the strategic plan, such as evaluation, creation, and implementation. Implementation step The strategies in the plan are practiced as illustrated in the strategic plan execution timeline. This step produces completion of actions in yearly work plans, as mirrored in the accomplishment of SMART aims. The implementation is recorded in my strategic plan in progress documents, using the indicators of the marketing training programs. Evaluation In the evaluation step, I will evaluate the implementation of the plan and my program activities. I can develop questions and gather data to notify the yearly work plan for the following year. Data used in evaluation are utilized in monitoring the progress of the five-year strategic plan. The “evaluate” step produces evaluation results, summaries of the progress, and a description of any change that was made in program actions based on the findings of the evaluation. The rationale for developing the evaluation The marketing training program is organized on the logic of helping the diverse employees such as the supervisors to understand, organize, and act in response to workforce training strategies and an the effects of those strategies on the learners and the program. The marketing training program is meant to respond to any requirement for the supervisors to realize the measures that are utilized to weigh the performance of their respective programs and get to know of how the measures operate and the impacts they have on the programs. Another key reason for evaluating the program is to conduct and spot modifications that could enhance the effectiveness of the workforce in achieving its results and the necessities and anticipations of the targeted consumers. The products of the evaluation will be regarded based on the training’s costs to establish if the training ought to be repeated. In every organization, there is a desire to convey a valuable and effectual training. This is the most important rationale behind the evaluation process of this marketing training. A faculty will appreciate the fact that the training results in a successful endeavor. It is most probable that the value designed to the training will have a positive impact in the marketing field as well as influencing the employees to take part in several other meaningful programs offered in future. It is therefore very significant that all the diverse employees attend the training fully. When a positive experience is realized, then the employees will have a desire to attend the training and put what is learned in action. Purpose of the evaluation Most specifically, the purpose of evaluating the program is to establish the degree to which the training meets the desires and the prospects of the trainees and leads to the intended results. From the outcome of the program, the findings will be used to establish if the training would continue being offered, and if held again, to recognize the modifications of what is to be done to increase the achievements of such results and fluffing the targeted consumers’ desires and preferences. The evaluation’s stakeholders These are people who are interested in the evaluation process and the evaluation results can be of importance to them. In the marketing training program, top level managers can serve as primary stakeholders. They are the program participants as well as facilitators. They are responsible for initiating the training in corporation with other stakeholders to respond to a specified need among employees for further knowledge regarding the performance, events and similar concepts. The evaluation results are meaningful to top level managers in maximizing the product of the time and effort they invested in the development and liberation of the training. The supervisors are the secondary stakeholders. They supervise the hourly workers to ensure that professional growth tasks benefit the employees and enhance program performance. The supervisors solely motivate and sustain employee involvement in training on the grounds that the evaluation indicates that this training is meeting employee needs and anticipations and its producing positive results. Hourly workers are important too in the evaluation process. They are tertiary stakeholders. What is examined in the evaluation is established by their stated goals, anticipated results of the program, and the available resources to their evaluators. Since the hourly employees may have varying, likely conflicting objectives, it is vital that the objectives and goals be classified accordingly and meant to attain a consensus beforehand as to which specific objectives are most significant to examine as per the availed resources. The data collected from the stakeholder interviews and focus groups ought to be documented and accessed to the evaluator’s program. Key evaluation questions that focus on the allocation of resources What are the appropriate resources required to excel in performance? What is the optimum timeframe recognized by the organization, required to evaluate a set of dimensions? What information technology resources are needed to support information gathering, analysis, and documentation? What is the right number of employees and roles to assign to safety and quality? How much money or resources should be budgeted for employee training? Chapter 6-Choosing an evaluation design Retrospective Pretest Design Here, the evaluators will collect information from employees after learning, acting, or changing intervention. The learners will be expected to respond retrospectively about individual skills, knowledge, or attitudes. After this, the evaluator can compare and contrast their retrospective pre-evaluations with the assessments there after. However, this design is highly dependent on the accuracy of the learners to remember most of the events as well as their desire and will to give truthful information. The retrospective pre-test design may also use a survey measuring attitude on the employees after a new flextime training program. In the survey, the learners will be asked about their attitudes prior to the training and how they feel currently. They can also be asked on the impact the training had on their intents to continue training and to go on working in the organization. The most outstanding feature of the retrospective design is that it is simple and easy to collect data. Further, the evaluators can have a chance to compare retrospective pre-information and post-test information. In fact, the experience of the pretest does not contaminate posttest data. How to ensure internal and external validity To make sure that the internal and external data is valid, one should cautiously check the precision in data taping and coding to be sure all the information is ready and can be accessed for analysis purposes. Recurred and persistent observations are highly encouraged to build affinity and trust with the learners so that the probability of them offering valid information is increased. A longer time on site may build more faith and also increases the likelihood of a complete observation. Internal and external validity can be ensured through employing member checking techniques to test categories, interferences, and conclusions. Data should be checked continuously with the trainees or other stakeholders after several data gathering activities, or following drafting of a final report. Data collected from various methods, and sources can be compared and contrasted through triangulation. Multiple references can be used to check the validity of external and internal data since more information from different sources increase the likelihood of trustworthiness. For instance, data triangulation, methodological triangulation, and investigator triangulation can be utilized. Investigator triangulation will involve more than one evaluator in inquiring the validity of the data. If these evaluators plan, enforce, debrief, and analyze the results, internal and external data will most probably be richer and valid. The use of an audit trail will permit an elevator to establish the validity of the study. He must retain and make accessible interview notes, peer briefing messages, documents, journals, and other writings that are relevant in assessing the validity. Lastly, pilot testing will be a significant technique as it will help try out each of the information gathering tools with some employee samples. This allows the evaluator to know if the queries may elicit the values and sort of data being looked for. Before implementing the tools with the whole sample or the whole population, revisions should be made to ensure accuracy. Chapter7-Data collection methods The collection methods I propose for my plan are questionnaires, archival data, and surveys. Questionnaires These are forms presented by the evaluators to the learners. In the evaluation of this program, questionnaires will be used for the diverse employees to collect the information regarding their needs and anticipations in the organization as well as what they think of the training. This method is relevant for using in the training program since all the workers are literate and can respond to questions appropriately. Archival data Preexisting documents and sources may be a good method of collecting data, based on the reason that they are easy to use and may not call for many expenses for the evaluators as they are already in existence. Another benefit is that they can offer quantitative information that may address various key evaluation queries. Archival data may also act as a good alternative for the evaluators as it may provide some knowledge on the historical context of the marketing training program and a sequence of events. Despite the fact that archival data method of collection may have flaws, it may be not very much susceptible to distortions, and thus may proof more credible. Given that information is already there, assembling data with this method needs less demand on evaluation for the organizers. One cannot exhaust the rationales for using archival data method, as it is significant even in longitudinal evaluations. If the supervisors use an evaluation based on questions that look at changes or trends over time, they can gain largely from this method of data collection. Survey method This may be another relevant method, given that the number of employees to train adds up to 950. This is not a small number, and thus using a survey method may give accurate results. Surveys will help the elevators to obtain new information from this large number, at a low cost. The rationale for using this method is that, it permits easy tabulation of information, especially when the evaluators use closed-ended questions. Even though it is a complex method that if not used well, it can fail to materialize, the evaluators only need to use clear worded tools and test the method to ensure the usefulness and validity of the collected data. I thought of interviews and observations as well, but they were not used in this context for the following reasons: Observations This can involve habitual counting of a number of events or writing stories to describe what is going on. Extensive observations help evaluators to get a clear understanding of the context of the program and the way the employees interact. This would help them know if the trainees are pleased with the training or enjoy learning, or if they don’t seem so interested. Observers can be from within the organization or from external organizations. In this context, they can be either supervisors or managers. Such personnel posses much information about the trainees and can be very helpful in collecting data. However, this method may not materialize if the observers lack adequate skills. It is also not the best method of collection especially if the employees participating in the training can manipulate their behavior. Some can even pretend to show some kinds of traits or feelings that are not real, thus an observer can collect the wrong information. Knowledge of the trainees can also be a drawback for using this method, since they can be biased to give information in favor or against the participants. Conducting interviews on individuals or groups Interviews, either on a group or groups or on an individual will also be a good method to collect data. With this, the evaluator will interact directly with the learners and this may result to new viewpoints concerning the raining. For instance, an evaluator can make use of relevant examples, narratives, and critical occurrences that are motivating to the trainees and help them get a deeper understanding of how the training can be of help to an individual, a group, or the organizing firm. It is also a more practical way of getting close to the learners and get to know how they really feel about everything going around during the workshop. One advantage of interviews is that the participants may answer the questions according to personal characteristics of the evaluator, thus affecting the results of the interview. At the same time, some employees may have personal opinions, which cannot be perfectly based on to analyze or generalize the view of the rest of the population. This is because interviews can only be preformed on a small number of the total population as it is impossible to interview a total of 950 employees in a short period of time. Chapter 12-Drawing a sample Evaluation plan and the reason for sampling Evaluation data collected at the start of the program will help establish a foundation from which alterations in the learners or the effects on the organization can be tracked. This will involve a pretest of collecting foundation data on chosen indicators. The evaluation will be planned and implemented to help make immediate adjustments and changes in the training program in preparation for summative evaluation. Evaluating an extensive program will involve first hand gathering of information from the trainees. Sampling is an important aspect in evaluating since not all units in the employee population can be reached. Thus to determine their response regarding the training, a sample is taken and the results used to analyze the whole population. Also, even if all the employees could be identified, the process may be too costly, and may consume much time than expected. Rationale for choosing random sampling Even though sampling non-randomly is generally preferred in most surveys, only a few surveyors utilize it due to the prohibitive expenses; meaning that it requires numbering every participant of the survey populace, while sampling randomly calls for taking each nth participant. Research shows that so long as the subject to be sampled is randomly allocated, these two methods can produce fundamentally different findings. Sometimes, the non random technique results in much better deductions on the population, but in most cases, its inferences are discouraging. In this evaluation, random sampling will be the best option. This is for the rationale that random sampling gets rid of bias by giving all the participants an equal opportunity to be selected. The method is also relevant especially when the evaluators have little knowledge of mathematical background. A method for selecting groups in a sample Stratified random sampling will be the best in marketing training program since the population is heterogeneous. The employees will be divided into sub-groups, with measurements varying among the diverse sub-groups. However, this will be accounted for in the selection of a sample from the rest so that a sample can be obtained to precisely represent all the participants. The rationale for utilizing stratified sampling is that it reduces the cost per observation in the entire survey. In addition, accuracy at particular costs is increased and it is relevant when estimates of the total population parameters are required for every sub-group. Sampling the trainees into sub-groups allows the evaluators to have enough data so that a meaningful statistical analysis can be carried out regarding the attitudes of the employees in every selected sub-group. Nonetheless, tocomment anything on the attitudes of the entire group in the session, the analyst will have to relate weights to the results for every sub-group, relative to its presence in the whole workforce. Chapter 13-Analyzing evaluation data Based on the key evaluation questions and the way the employees are organized, I propose using a quantitative method expressed in numbers. Since most of the data collected is in numeric forms, statistical tests will be applied in analyzing them. Descriptive statics including median, mean, mode, and standard deviation will be the most preferred. The best method here will be the standard deviation method, as it will represent the distribution of the answers centered on the mean. This will indicate the level of consistency in all responses. Further, it can be combined with the mean to offer a better understanding of the collected data. For qualitative data analysis, I will use content analysis. This is for the logic that it is an orderly, replicable method to compress the massive data collected into less categories basing on the clear rules of coding. It will also enable the evaluators to sieve through large contents of data with much ease in a systematic way. Most importantly, it will enable them to discover and explain the focus of the groups, single trainees, or the institution. Content analysis gives a chance for inferences to be achieved, which can then be supported using other data collection methods. The final criterion Calculation of return on investment Determination of costs incurred on the three months marketing training program for 50 upper level managers, 150 supervisors and 750 hourly employees. Table 1 Fifty upper level managers Annual program benefits Standard monthly marketing for trained managers $50000 Standard monthly marketing for untrained managers $42000 Increment $8000 Profit contribution 3% $240 Total monthly improvement (50*240) $12000 Total annual Benefits (12000*48weeks) $576000 Cost summary Facilitation fees: 4 courses @ $30000 $120000 Materials for the program: 50 employees @ $45 per participant $2250 Meals: 4 weeks @ $ 30 per participant per week $6000 Facilities: 12 weeks @ $1000 $12000 Participants salaries and benefits $25000 Evaluation process $150000 Total costs $315250 Cost benefits ratio = Total annual benefits / Total cost = 576000 / 315250 = 1.83 Return on investment = (Total annual benefits – Total cost) / Total cost = (576000 – 315250) / 315250 = 0.83 * 100 = 83% Table 2 One hundred and fifty Supervisors Annual program benefits Standard monthly marketing for trained supervisors $145000 Standard monthly marketing for untrained supervisors $123000 Increment $22000 Profit contribution 3% $660 Total monthly improvement (150*660) $99000 Total annual Benefits (99000*48weeks) $4752000 Cost summary Facilitation fees: 4 courses @ $30000 $220000 Materials for the program: 150 employees @ $45 per participant $6750 Meals: 4 weeks @ $ 30 per participant per week $18000 Facilities: 12 weeks @ $1000 $12000 Participants salaries and benefits $198000 Evaluation process $250000 Total costs $704750 Cost benefits ratio = Total annual benefits / Total cost = 4752000 / 704750 = 6.74 Return on investment = (Total annual benefits – Total cost) / Total cost = (4752000 – 704750) / 704750 = 5.74 * 100 =574% Table 3 seven hundred and fifty hourly employees Annual program benefits Standard monthly marketing for trained hourly employees $55000 Standard monthly marketing for untrained hourly employee $42000 Increment $13000 Profit contribution 3% $390 Total monthly improvement (750*390) $292500 Total annual Benefits (292500*48weeks) $14040000 Cost summary Facilitation fees: 3 courses @ $25000 $675000 Materials for the program: 750 employees @ $45 per participant $33750 Meals: 4 weeks @ $ 20 per week per participant $60000 Facilities: 12 weeks @ $20000 $108000 Participants salaries and benefits $550000 Evaluation process $660000 Total costs $2086750 Cost benefits ratio = Total annual benefits / Total cost = 14040000 / 2086750 =6.73 Return on investment = (Total annual benefits – Total cost) / Total cost = (14040000 – 2086750) / 2086750 =5.728 * 100 = 573% Reference Preskill, H., and Russ-Eft, D. (2001). Evaluation in Organization: A Systematic Approach to Enhancing Learning, Performance, and Change. New York, NY: Basic Books. Read More