Information Systems Evaluation - Essay Example

INFORMATION SYSTEMS EVALUATION: Introduction The IT needs in the business and personal lives of people worldwide have madeIT one of the areas where many resources are invested. The amount of resources invested in acquisition and maintenance of technology makes it necessary for the users to periodically evaluate its effectiveness. For example, in the business world, a manager would invest in IT infrastructure to boost efficiency and profits from operations. It is necessary to evaluate the achievement of these goals to justify the investment to stakeholders. The question of what to evaluate can be approached from two perspectives; evaluation of the system in use and evaluation of the basic system. To identify what to evaluate and how to evaluate it, we have to rely on past literature. There are three primary techniques of evaluating systems. They are goal based techniques, goal free techniques and criteria based techniques. The main differentiating factors for the techniques are the motivation behind the evaluation process. Goal based evaluation When using the goal-based approach, the organization elicits specific goals that drive the system evaluation process. The IT system is compared with the organization’s goals to determine if they have been achieved. One of the main shortcomings of this approach is that it misses the human and social aspect of the system and instead concentrates of measurable concepts of economics and technology ( Hirschheim and Smithson, 1988). The technique could therefore capture important metrics, such as user satisfaction and usability of the system. The goal based approach can be hard to carry out, especially on qualitative measures of the system. Instead, it uses a quantitative approach seeking to establish if various goals have been achieved and what these goals are. What it leaves out is the extent to which various goals have been achieved. The nature of the goals that the organization has set determines the outcome of the process. Goal free approach On the other hand, when taking a goal free approach to the evaluation process, there are no goals elicited by system users. It relies on the situation arising to require application of the system’s capability. It views IT infrastructure as social infrastructure with various aspects of IT embedded into it. It seeks a deeper understanding of the system to be evaluated and the evaluation criteria that will be used on it. Several groups of stakeholders have to be involved in the evaluation of these systems. However, this can result in delays on the process, especially when the time limits to carry out the process are tight. To evaluate a system using this approach, the one carrying out the evaluation has to collect a wide variety of data and effects of the system. Then, he/she has to evaluate the significance of the information in determining the effectiveness and usability of the system. Stakeholders, such as users of the system and people who have contextual knowledge of its functionality, have to be left out of the evaluation process. The approach has several goals: The analyzer missing unanticipated outcomes of the system by restricting the analysis and study on narrowly defined objectives. Elimination of the negativity associated with unanticipated effects of a system. Elimination of analyzer’s bias that would be introduced by prior knowledge of the system work. Insurance of the analyzer’s independence and objectivity Criteria based approach This approach is quite common and practiced in the form of several common approaches. It includes approaches like checklist and heuristic. In most IT systems, heuristics are employed in evaluation of the user interface. These heuristics have a set of predefined criteria that is used in evaluation of how the system functions. The criteria used in this approach are formulated from various scientific premises and common knowledge. Computer science and cognitive theories have been used in formulation of criteria for various systems. Usability theories have also played an important role in determination of the choices that people make in the field. The criteria set for the process is the main determinant of the conclusions that the evaluator will draw about the system. Unlike in goal based approach, the criteria used in this process are not derived from specially chosen organizational contexts. Because of this, they can be taken as generalizations of the system and assumptions about them hold for the whole system. What to evaluate As previously outlined, there are two general approaches to evaluation. They are: Evaluation of the system in use and’ Evaluation of the system as is. Evaluation as is In this approach, the system is evaluated without inclusion of users in the process. Here, the only participants are the system and the evaluator. The evaluator relies on the system and its documentation as the data sources for the evaluation process. The strategy chosen for the analysis process determines how the analysis is conducted. In each of the approaches of what to evaluate, all the strategies of how to evaluate are applied. This leaves us with 6 possible strategies of system evaluation. The understanding of the evaluator on how the system helps the organization will be important in determining the results of the evaluation process. Exclusion of the user means that the outcome is free from errors in judgment that the users may make. The context within which the system is used and the effects and benefits the users realize from using it have no part to play in the analysis. The implementation functionality of the system is one of the main focuses of the study. Evaluating the IT system in use This mainly depends on the observational character of the evaluator. The process is much more complex than analyzing the system as is. It includes the user into the analyses process and presents a more comprehensive picture of the system. Some of the main data sources for the process are interviews and observations of the system as it is being used. The qualities and functionality of the system is another main source of data used to conduct the system analysis. The quality of data presented in this case is high enough to support complex research requirements. Depending on various factors, usability of a system is a highly subjective metric. It may vary by large extent over a period of time. Like the process of evaluating the system as is, evaluating the system in use can use any of the “how” strategies (goal based, goal free and criteria based). The strategy chosen determines the outcomes of the evaluation process. Unlike the evaluation of the system as is, the opinion and understanding of the users determines the outcomes of evaluating the system in use. The users are a key source of data used for evaluating the system. The six generic evaluation techniques Combining the three “how to evaluate” techniques with the two “what to evaluate techniques”, there are 6 generic evaluation strategies that could be applied. Below is a table showing the 6 combinations: Type 1 (Evaluating an IT system as such using the goal free approach) This strategy favors examination of various aspects of the IT systems. It is an inductive process with most of the data used for it being sourced from the system and its descriptions (documentation). The results are usually limited and therefore it is often chosen when introducing the system for the first time. It is essential to give the evaluator an idea of the context within which the system operates. The evaluator begins by establishing a description of the system’s functionality and then taking an explorative walk through the system. Besides the system and its evaluation, interviewing the system owner could be another source of information and data. There should be an evaluation of the system’s strengths and weaknesses for the process to be complete. The main result from this process should be recommendations of further system evaluations. Type 2 (evaluating an IT system as such using the goal based approach) In this approach, the system is evaluated according to some predefined business goals. The subject of the evaluation is the IT system itself rather than the users and its environment. Taking a goal oriented approach means that the evaluation may concentrate on the economic and technical perspectives. Goals also have a character which plays an important role in determining the outcome of the evaluation process. The main aim of the evaluator is to determine if the system’s goals have been achieved. Since the evaluator uses the goals to prove hypothesis, the process can be seen as employing deductive reasoning. Besides the system’s descriptions, the other main source of data for the evaluation is the goal descriptions from the organization using the system. The evaluator should begin by describing the business goals and the functionality of the IT system. Business goals are quite simple to describe, especially from sources like strategy statements. One could also interview the managers of the organization that the system belongs to. Next, the functionality of the IT system has to be established and documented adequately. This information can be obtained through observation of the system in use. Documentation of the system can also help in determination of the system goals and functionality. This way, the evaluator can determine if the system’s functionality meets the goals of the organization. Type 3 (Evaluating a system as such using the criteria based approach) The approach uses some predefined criteria to evaluate the system. Since it is an evaluation of the system as such, the system is the main focus of the process. The perspective of the evaluation depends on the criteria used for the process. The objective of the process is to determine if the quality of the IT system meets the criteria for the evaluation. It is a deductive process seeking to establish if the system meets the operational criteria. The description of the IT system, the criteria and the system’s documentation are the main sources of data for the evaluation process. The users participate in the process through their participation is at a limited level. When there is need for focused evaluation corresponding to certain criteria, this approach is the most suited. Typically, the process starts by establishing appropriate criteria for the evaluation. The desired evaluation perspective determines the criteria chosen for the process. Next, the evaluator has to describe the system’s functionality. For this, he/she depends on the system’s documentation. A study of the IT system can also help in revealing the system’s functionality and what it is intended for. As soon as the criteria are well defined, the evaluation process can begin. By comparing the criteria with the systems functionality, it can be decided if it meets the criteria and therefore, if the system is good enough for its intended function. Type 4 (Evaluation of an IT system in use using the goal free approach) The object of the evaluation is the system itself. The evaluator is not concerned with the system’s existing goals. The recommended approach is where the evaluator is not aware of any functional parameters of the system. He/she discovers various aspects of the system as he/she continues with the evaluation process. It is an inductive process since the evaluator ignores any existing preconditions of the system. There are more data sources in this approach, because the evaluator uses information obtained from the system’s users to understand the functionality of the systems. Several techniques of gathering data from users can be used for interaction. These include observation of the system in use, interviews and others. Unspoken and unwritten information about the system can also be acquired using this approach. The evaluator has the chance to use his/her deductive skills to determine the attitudes of the system’s users. It is well suited when there is need for a deeper understanding of the system. To perform this evaluation, the evaluator begins by establishing a description of the system’s functionality. There should also be a description of the users and their interactions with the system. To describe the system’s functionality, one starts by studying its documentation and its operational parameters. One could also use interviews to obtain data from the system’s owner. Observation of the users and how they interact with the system can be carried out using techniques such as thinking aloud on typical use cases. The evaluator can also interview a regular system user to understand basic flow of activities in using the system. Type 5 (Evaluation of IT systems in use using a goal based approach) In this technique, there are predefined business goals focusing the evaluation on the IT system. The goals have character, which is the main determinant of the evaluation perspective. This technique seeks to establish if the system’s goals have been fulfilled as required. This makes it a deductive strategy. It differs from type 2 evaluation in that there are more data sources for the activity. The additional sources come from the interaction between the users and system. Some of the interactions between the users and the system include their understanding of the system and their attitude towards it. This understanding can be gained through interviews and questionnaires. The approach is best suited for cases where there are abundant resources available to be used for the research. It yields a focused result based on set goals. Typically, the evaluation process begins by describing the goals of the system and its functionality. To determine business goals, the evaluator relies on various documents such as the strategy documents. He/she relies on observations and induction to obtain data about the system. Type 6 (Evaluating IT systems in use by a criteria based approach) There is a predefined criterion to be used in the evaluation process. It focuses on both the system and the environment within which it is used. Like in type 5, the perspective of the evaluation depends on the characteristics of the chosen criteria. Deductive thinking enables the evaluator to determine if the system meets criteria set. The evaluator has more access to a larger pool of data about the system. This is obtained from the users and analysis of their perception of the system. The evaluation is not dependent on any factors and no limitations are placed on it. Why to evaluate IT systems cost money. In most cases, business managers are responsible for making the decision to invest in these systems. They have to justify their spending for the system to a board of directors or to other stakeholders. A business would want to increase the yields in terms of profits. An evaluation has to be conducted to determine the extent to which they have achieved the intended goals for the system. For those using IT systems for personal use, they also have goals and objectives wanting to invest financial resources in acquisition of the system. To determine the worthiness of the investment, an audit has to be carried out by a professional evaluator. Conclusion The strategies outlined above are helpful in enabling people to justify their investments in IT systems. Businesses are also able to discover disparities in their requirements and what they are able to achieve with the systems they purchase. The techniques described above are quite comprehensive. Reference list Bosselaers A., & Preneel, B. (1995). Integrity primitives for secure information systems: final report of RACE Integrity Primitives Evaluation RIPE RACE (1040). Berlin, Springer-Verlag. Cross-Language Evaluation Forum, & Peters, C. (2004). Comparative evaluation of multilingual information access systems: 4th Workshop of the Cross-Language Evaluation Forum, CLEF 2003, Trondheim, Norway, August 21-22, 2003 : revised papers. Berlin, Springer. Currie, W., & Galliers, R. (1999). Rethinking management information systems an interdisciplinary perspective. New York, Oxford University Press. [online] Available at: http://www.books24x7.com/marc.asp?bookid=2148. (Accessed 03 March 2013) Hirschheim, R. A. (1986). A critical analysis of information systems evaluation. Oxford [Eng.], Oxford Institute of Information Management, Templeton College. King, D. W., & Bryant, E. C. (1971). The evaluation of information services and products. Washington, Information Resources Press. Kourouthanassis, P. E., & Giaglis, G. M. (2008). Pervasive information systems. Armonk, N.Y., M.E. Sharpe. [online] Available at: http://site.ebrary.com/id/10292198. (Accessed 03 March 2013) Levy, M., & Powell, P. (2005). Strategies for growth in SMEs the role of information and information systems. Oxford, Elsevier Butterworth-Heinemann. [online] Available at: http://search.ebscohost.com/login.aspx?direct=true&scope=site&db=nlebk&db=nlabk&AN=123427. (Accessed 03 March 2013) Syed, M. R. (2001). Design and management of multimedia information systems opportunities and challenges. Hershey, Pa, Idea Group Pub. [online] Available at: http://search.ebscohost.com/login.aspx?direct=true&scope=site&db=nlebk&db=nlabk&AN=60700. (Accessed 03 March 2013) Van Grembergen, W. (2002). Information systems evaluation management. Hershey, PA, IRM Press. [online] Available at: http://search.ebscohost.com/login.aspx?direct=true&scope=site&db=nlebk&db=nlabk&AN=66471. (Accessed 03 March 2013) Wolstenholme, E. F., Henderson, S., & Gavine, A. (1993). The evaluation of management information systems a dynamic and holistic approach. Chichester, J. Wiley. [online] Available at: http://search.ebscohost.com/login.aspx?direct=true&scope=site&db=nlebk&db=nlabk&AN=17792. (Accessed 03 March 2013) Read More