Assessing the Quality of Qualitative Research - Essay Example

The Core Principles of Conducting & Reporting Qualitative Research Assessing Quality of Qualitative Research Research is making inquiry intohuman affairs and the phenomena around in order to have better understanding of the world around us. There are two research paradigms which have two different views of reality. Quantitative research assumes reality to be well defined and can be measurable in quantitative terms. While on the hand Qualitative research “is a form of social inquiry that focuses on the way people interpret and make sense of their experiences and the world they live” (Holloway, 1997, p1). The end of this research method is to understand the social reality from the perspective of the people. This research aims at interpretation rather than verification of the theories. This paradigm is mostly popular in social sciences research and is the favorite method of anthropologists. Qualitative researchers typically rely on the following methods for gathering information: Participant Observation, Non-participant Observation, Field Notes, Reflexive Journals, Structured Interview, Unstructured Interview, Analysis of documents and materials.( Marshall& Rossman ,1998, page no not known). ‘Qualitative research’ is a broad term encompassing a number of approaches, paradigms and schools. Among these different approaches and traditions of qualitative research, ‘there are widespread concerns for quality.’; ‘the need for principles of practice to be made manifest’ (Spencer et al (2003. p 4) .To assess the quality in qualitative research, a framework has been designed by Spencer et al (2003). This framework consists of four guiding principles which branch out into appraisal questions’ and ‘quality indicators’. Let us try to assess the quality of a qualitative work on the basis of this framework. FRAMEWORK FOR ASSESSING QUALITATIVE EVALUATIONS Study being appraised: ‘Stress in Elite Sport Performers: A Comparative Study of Competitive and Organizational Stressors’, Hanton et al, 2005. a) Appraisal questions b) Quality indicators (possible features for consideration) c) Notes on study being appraised 1 How credible are the findings? Findings/conclusions are supported by data/study evidence (i.e. the reader can see how the researcher arrived at his/her conclusions; the ‘building blocks’ of analysis and interpretation are evident) Findings/conclusions ‘make sense’/have a coherent logic Findings/conclusions are resonant with other knowledge and experience (this might include peer or member review) Use of corroborating evidence to support or refine findings (i.e. other data sources have been used to examine phenomena; other research evidence has been evaluated: see also Q14) The findings have been supported by not only the data collected in this study but also in the previous studies. The reader is made aware of all the stages of the study. The findings do make sense in the sense that the study compares organizational and competitive stressors, finding out organizational issues causing greater anxiety. Yes, The study claims that its findings are resonant with the findings of the previous non-empirical studies—(Woodman & Hardy (2001) and then Fletcher and Hanton’s (2003) 2 How has knowledge/ understanding been extended by the research? Literature review (where appropriate) summarising knowledge to date/key issues raised by previous research Aims and design of study set in the context of existing knowledge/ understanding; identifies new areas for investigation (for example, in relation to policy/practice/substantive theory) Credible/clear discussion of how findings have contributed to knowledge and understanding (e.g. of the policy, programme or theory being reviewed); might be applied to new policy developments, practice or theory Findings presented or conceptualized in a way that offers new insights/alternative ways of thinking Discussion of limitations of evidence and what remains unknown/unclear or what further information/research is needed Through the literature review the study claims to be unique in the sense that previous studies focused on identification of stressors while this study undertakes comparison of the identified stressors. Since, no study undertook such comparison, it investigates a new aspect of previously identified area. At the same time, it identifies a new area for investigation-- ‘‘safety’’, the demands associated with terrorist threats. The findings are of use for stress management programmes and can help scientists understand and tackle the stress process. Definitely it offers new ways of thinking about organizational stressor and safety stressor with facts and figure. However, there seems to be a little discussion about the limitations of evidence and what remains unknown or unclear. 3 tyyttytyutytyty a) Appraisal questions b) Quality indicators (possible features for consideration) c) Notes on study being appraised 3 How well does the evaluation address its original aims and purpose? Clear statement of study aims and objectives; reasons for any changes in objectives Findings clearly linked to the purposes of the study – and to the initiative or policy being studied Summary or conclusions directed towards aims of study Discussion of limitations of study in meeting aims (e.g. are there limitations because of restricted access to study settings or participants, gaps in the sample coverage, missed or unresolved areas of questioning; incomplete analysis; time constraints?) 1. As it was meant to be a comparative study, the researchers seem to have provided enough evidence for the establishment of their point of view. Yes, the findings seem to be clearly linked to the purpose. The study does provide a summary/conclusion directed towards the aim of the study—comparison of identified stressors. Little discussion about the limitations of study in meeting aims or any unresolved areas or incomplete analysis or time constraints. 4 Scope for drawing wider inference – how well is this explained? Discussion of what can be generalized to wider population from which sample is drawn/case selection has been made Detailed description of the contexts in which the study was conducted to allow applicability to other settings/contextual generalities to be assessed Discussion of how hypotheses/ propositions/findings may relate to wider theory; consideration of rival explanations Evidence supplied to support claims for wider inference (either from study or from corroborating sources) Discussion of limitations on drawing wider inference (e.g. re-examination of sample and any missing constituencies: analysis of restrictions of study settings for drawing wider inference) It does give a detailed description of the contexts of study elite UK athletes from a wide range of sports but no mention of generalization to other settings. However, it claims that it is useful for sports scientists. Little discussion about limitations on drawing wider inference. This approach provides sports scientists with an integrated framework for understanding and tackling the complexities of the stress process as a whole. It does not seem to claim wider inference. 5 How clear is the basis of evaluative appraisal? Discussion of how assessments of effectiveness/evaluative judgements have been reached (i.e. whose judgements are they and on what basis have they been reached?) Description of any formalised appraisal criteria used, when generated and how and by whom they have been applied Discussion of the nature and source of any divergence in evaluative appraisals Discussion of any unintended consequences of intervention, their impact and why they arose The study develops amid references to previous theories; studies on the topic, their findings, and methods employed for data collection and analysis especially those made by Fletcher and colleagues 2003. It begins by referring to such studies and the emphasis on the relational meaning of the stress process which forms the premise of this study. Little discussion about any such consequences or intervention a) Appraisal questions b) Quality indicators (possible features for consideration) c) Notes on study being appraised 9 How well was the data collection carried out? Discussion of: • who conducted data collection • procedures/documents used for collection/recording • checks on origin/status/authorship of documents Audio or video recording of interviews/discussions/conversations (if not recorded, were justifiable reasons given?) Description of conventions for taking fieldnotes (e.g. to identify what form of observations were required/to distinguish description from researcher commentary/analysis) Discussion of how fieldwork methods or settings may have influenced data collected Demonstration, through portrayal and use of data, that depth, detail and richness were achieved in collection Researchers’ profiles not mentioned. A pilot study on recently retired elite Performers was conducted which led to refinement of questions. The copies of interview were sent a week before the interview. Written informed consent was obtained before the data collection and anonymity was assured. This enhanced the trustworthiness of the data. Face to face interviews. The interviews were taperecorded and transcribed verbatim. The interviews were conducted away from the environment of competitive performance and sport organization. This helped minimize situational influences. The participants were allowed to go with the flow of discussion, it helped collect rich data about the issues. Clarification, elaboration and general probes were used to investigate issues in greater depth. At the end, the interviewers were asked whether they wanted to add anything to what had already been discussed. 1:As the researchers have used interviews as a data collection tool which is considered to be an effective data collection tool in qualitative research. 2: But the researchers should have used other data to strengthen their research as validity to qualitative research comes from varied sources of data collection. 3: The data collection has been done quite carefully. 10 How well has the approach to, and formulation of, the analysis been conveyed? Description of form of original data (e.g. use of verbatim transcripts, observation or interview notes, documents, etc.) Clear rationale for choice of data management method/tool/package Evidence of how descriptive analytic categories, classes, labels etc. have been generated and used (i.e. either through explicit discussion or portrayal in the commentary) Discussion, with examples, of how any constructed analytic concepts/typologies etc. have been devised and applied QSR N5 (2000) computer software package Was used which is a software for unstructured data is indexing. A combination of inductive and deductive content analysis was employed to analyze the data—a method used in previous studies mentioned in the research. 1: Cross validation or triangulation of data lends qualitative research a kind of authenticity which is not found in the given research . 11 Contexts of data sources – how well are they retained and portrayed? Description of background or historical developments and social/organizational characteristics of study sites or settings Participants’ perspectives/observations placed in personal context (e.g. use of case studies/vignettes/individual profiles, textual extracts annotated with details of contributors) Explanation of origins/history of written documents Use of data management methods that preserve context (i.e. facilitate within case description and analysis) Participants’ perspectives were not only preserved in data collection and analysis but also mentioned in findings as well in the form of quotations containing participants’ own expression. However, no case studies were undertaken. a) Appraisal questions b) Quality indicators (possible features for consideration) c) Notes on study being appraised 12 How well has diversity of perspective and content been explored? Discussion of contribution of sample design/ case selection in generating diversity Description and illumination of diversity/multiple perspectives/alternative positions in the evidence displayed Evidence of attention to negative cases, outliers or exceptions Typologies/models of variation derived and discussed Examination of origins/influences on opposing or differing positions Identification of patterns of association/linkages with divergent positions/groups The small sample design helped in in-depth analysis and in achieving the aim of this study and that sample characteristics inevitably limit the generalization of the findings to other populations such as female performers. Little discussion about typologies and models of variation or examination of differing groups. 13 How well has detail, depth and complexity (i.e. richness) of the data been conveyed? Use and exploration of contributors’ terms, concepts and meanings Unpacking and portrayal of nuance/subtlety/intricacy within data Discussion of explicit and implicit explanations Detection of underlying factors/influences Identification and discussion of patterns of association/conceptual linkages within data Presentation of illuminating textual extracts/observations Yes, definitely there has been a long description of the phrases and concepts of the contributors and their analysis. There is discussion of conceptual linkages within the data. Yes , there are illuminating textual extracts or observations 14 How clear are the links between data, interpretation and conclusions – i.e. how well can the route to any conclusions be seen? Clear conceptual links between analytic commentary and presentations of original data (i.e. commentary and cited data relate; there is an analytic context to cited data, not simply repeated description) Discussion of how/why particular interpretation/significance is assigned to specific aspects of data – with illustrative extracts of original data Discussion of how explanations/ theories/conclusions were derived – and how they relate to interpretations and content of original data (i.e. how warranted); whether alternative explanations explored Display of negative cases and how they lie outside main proposition/theory/ hypothesis etc.; or how proposition etc. revised to include them The findings are reported in the form of thick descriptive quotes to enable the reader gain a feel. Data has been collected to compare effect of different stressors on the performance of the elite players. The numeric evidence gives credence to the researcher’s hypothesis that environmental stress has more impact on the performance of the elite players. a) Appraisal questions b) Quality indicators (possible features for consideration) c) Notes on study being appraised 15 How clear and coherent is the reporting? Demonstrates link to aims of study/research questions Provides a narrative/story or clearly constructed thematic account Has structure and signposting that usefully guide reader through the commentary Provides accessible information for intended target audience(s) Key messages highlighted or summarized A narrative tale style has been Adopted. A ‘writer-persuader’ style has been adopted. By citing participants’’ responses, they have attempted to allow readers the freedom to interpret and evaluate the text from their unique vantage points. The reader easily moves through various sections of the study. Hierarchical content trees and sample representative quotes make reporting easy to comprehend. Definitely, through tables audience can easily have a comparative survey of the stressors. 16 How clear are the assumptions/theoretical perspectives/values that have shaped the form and output of the evaluation? Discussion/evidence of the main assumptions/hypotheses/theoretical ideas on which the evaluation was based and how these affected the form, coverage or output of the evaluation (the assumption here is that no research is undertaken without some underlying assumptions or theoretical ideas) Discussion/evidence of the ideological perspectives/values/philosophies of research team and their impact on the methodological or substantive content of the evaluation (again, may not be explicitly stated) Evidence of openness to new/alternative ways of viewing subject/theories/ assumptions (e.g. discussion of learning/concepts/ constructions that have emerged from the data; refinement restatement of hypotheses/theories in light of emergent findings; evidence that alternative claims have been examined) Discussion of how error or bias may have arisen in design/data collection/analysis and how addressed, if at all Reflections on the impact of the researcher on the research process Fletcher and Hanton’s distinction 2003 between major stressors forms the conceptual backdrop of the research. There are two theoretical approaches behind the research. One is homothetic and other is ideographic techniques. The researchers’ use of interviews as a data collection tool falls into the category of ideographic technique which entails individuals understanding of their own reality. Little discussion about the philosophies of research team. There is certainly evidence of openness in the fact that the interviews were semi-structured open fuller coverage of the problem. Moreover, a procedural link was allowed to develop between the data collection phase and early stages of the inductive analysis. This involved modifying existing probes and introducing new probes to further investigate issues. This flexibility and investigative approach allowed for a deeper engagement and more effective examination of the study. Little discussion of dealing with bias in design and data collection. However, in analysis, measures were taken that researchers’ bias does not color participants’ responses. a) Appraisal questions b) Quality indicators (possible features for consideration) c) Notes on study being appraised 17 What evidence is there of attention to ethical issues? Evidence of thoughtfulness/sensitivity about research contexts and participants Documentation of how research was presented in study settings/to participants (including, where relevant, any possible consequences of taking part) Documentation of consent procedures and information provided to participants Discussion of confidentiality of data and procedures for protecting Discussion of how anonymity of participants/sources was protected Discussion of any measures to offer information/advice/services etc. at end of study (i.e. where participation exposed the need for these) Discussion of potential harm or difficulty through participation, and how avoided Anonymity of the participants at their own request was preserved. Written consent was sought from the participants. In addition to the informed written consent, an interview guide was also used. The names of the participants have been changed. The participants’ sports organizations were not mentioned. Preserving the anonymity of the participants helped in enhancing the trustworthiness of the data. Potential would have been there if the participants identities had been revealed. 18 How adequately has the research process been documented? Discussion of strengths and weaknesses of data sources and methods Documentation of changes made to design and reasons; implications for study coverage Documentation and reasons for changes in sample coverage/data collection/analytic approach; implications Reproduction of main study documents (e.g. letters of approach, topic guides, observation templates, data management frameworks etc.) Yes there is discussion of strengths and weaknesses of the study. Its strengths lie in its investigative and open approach towards data collection. But the sample design defines its limitations. The researchers have conducted the research following nearly all the parameters of qualitative research. But the process of documentation of research needs further elaboration. The descriptive data should have been complemented by numeric data. In the end we can say that all the features of the framework are not applicable to this study. The strengths of this study are hierarchical content trees, sample representative quotes, investigative approach in data collection and the fact that the participants were allowed to go with the discussion which helped in producing more data. However, the limitation of the study lies in its sample. The sample characteristics limit the generalization of findings to other populations. This study has extended our knowledge by comparing competitive and organizational stressors among elite sport performers. It draws our attention to new areas of organizational stressors and safety stressors which had not been focused in most of earlier studies. However, literature review has not been given in proper order and there are no elaborate details. With regard to data collection, the paper does not answer the crucial question of who conducted the interviews. It is also not stated why other means of qualitative research were not used. Moreover, there seems to be a little discussion about the limitations of evidence and what remains unknown or unclear. References Hanton, Sheldon, Fletcher, David and Coughlan, Guy.,2005 Stress in elite sport performers: A comparative study of competitive and organizational stressors, Journal of Sports Sciences, 23: 10, 1129 — 1141n Available at http://www.informaworld.com/smpp/title~content=t713721847 [Accessed 2 Jan 2009] Immy Holloway .2007. Basic concepts for Qualitative Research. London: Blackwell Science. Marshall, Catherine & Rossman, Gretchen B. 1998 . Designing Qualitative Research. Thousand Oaks, CA: Sage Spencer,L., Ritchie, J., Lewis.J .m, Dillon,L. 2003.Quality in Qualitative Evaluation: A framework for assessing research evidence. National Centre for Social Research. Available at http://www.gsr.gov.uk/downloads/evaluating_policy/a_quality_framework.pdf. [Accessed 2 Jan 2009] Read More