Data Warehousing and Business Intelligence - Essay Example

Topic: Data Warehousing and Business Intelligence Introduction Information, data, knowledge etc. are the most significant part of virtual world and almost every organization has either already shifted or planning to shift his knowledge base to virtual world. Maintenance of such enormous amounts of information poses a challenging task on the part of organizations, which regularly come with more advanced and developed technologies for dealing with this bulk of knowledge. However, more often than not, most solutions fail either due to lack of proper planning, lack of foresight or due to sheer quantity of the data that needs to be dealt. This is where a planned, technical and structured method for data processing becomes inevitable and concepts solely concentrating on Information management come in picture. For all organizations, Data Warehousing and Business Intelligence have become key research areas. Both concepts are central for proper data management, particularly if extremely large amount of data is concerned. Data Warehouses are central to the Business Intelligence of an organization, which basically represents the Knowledge Reach of an organization. A large number of methods have developed for design of efficient warehouses and other peripheral tools meant to access or update the data warehouses. Definitions The key terms related to Data Warehousing and Business Intelligence have been summarized below: 1) Data Warehouses: Data Warehouse is generally defined as subject-oriented, time-variant, static and consolidated storage of information. Data Warehousing, in terms of Decision Support systems, could be defined as collections of decision support technologies, trying to enable strategy makers of an organization take speedy decisions. 2) Decision Support Systems (DSS): Decision Support systems are generally knowledge based systems pooling data from different softwares and web services and providing the users with mechanisms to utilize the data for developing better tools and strategies and forecasting market and consumer related events more accurately. 3) Data Mart:  A highly specialized independent data warehouse separated from central repository and customized to concentrate on some particular aspects of data management is called a data mart. 4) Metadata: It is the data about data present in warehouses defining source of data, Warehouse structure, queries about queries and other data used for data processing. 4) Business Intelligence: Business Intelligence is a wide concept consisting of the tools, methodologies and data sources contributing to enhancement of information management system in an efficient, user friendly and integrated manner for an organization so as to increase productivity and Return on Investment (ROI) and decrease development and maintenance costs.  5) Enterprise Performance Management (EPM): Enterprise Performance management is generally defined as measure of efficiency in execution of business strategies and processing of data. 6) Enterprise Information management (EIM): Enterprise Information Management is the management of end-to-end data of an organization through application of various services and technologies, and proper representation of information using established and well defined semantics. They also ensure proper integration of data obtained through various channels, and speedy execution of various data-processing related transactions. Data Warehouses: Usually data warehouses contain extremely large (and complex) bulk of data ranging from gigabytes to terabytes. Equally complex mechanism is required to access the data from Warehouses, often resulting into formation of extremely large queries compounded by several levels of joints between different tables. While operational databases are usually normalized to increase the performance of transactions and maintain the data consistency, the priorities for data warehouses are quiet different. The information in Data warehouses is basically used for querying and analysis. Data is generally not overwritten and multiple copies of the same data may be present on different granular levels for facilitating querying of data. Data queried from warehouses in generally used to generate reports and analyze different trends in organizations e.g. sales of products, recurring complaints against products, people or organizations. Warehouses thus form an important part of Customer Relationships management (CRM). Unlike OLTP which contain minimum data and keep on deleting or overwriting historical data, Warehouses usually contain maintain historical data in consolidated form. A warehouse can support multiple decision support systems and a decision support system can utilize the data from multiple warehouses to enhance the business intelligence of an organization.   Data warehouses generally store the data after segregating them as facts and dimensions whose most popular structure is star schema. Facts are generally the values of a quantity while dimensions are the descriptions of facts. Care is taken so as to keep the data in dimensional tables independent and simple and usually natural keys are replaced by surrogate keys. Proper segregation of facts from dimensions is evidently a core issue in design of warehouses. Depending upon the complexity of data present, the segregation can still be broken down to several hierarchical orders to facilitate querying of data. Data Marts are either partially or completely independent of the central Warehouse due to various reasons (security, performance etc.) and have become particularly popular due to ease of handling a smaller portion of data. The data present in data marts is usually accessed through web applications and load performance issues may arise with significant increase in volume of data. Data Marts can also be starting point for development of larger warehouses for many organizations. Activities concerned with Warehousing can be loosely summarized as below: 1) Load, Clean and Refresh Data: Obtaining data and correcting them, either by transforming (by string-replacement, format change, etc ) 2) Take necessary backups: Archiving of data present in Warehouses 3) Upgrade Data: Deals with upgrade of data warehouse structures (tables, indexes etc) and the resulting change in queries. 4) Metadata Management: This can be a complex process, even though comparatively very small amount of data needs to be dealt with. With Data Structure upgrade, Metadata also needs to be updated. Also included generation of higher level queries. Multiple tools and interfaces either directly or indirectly connect to Warehouses for gathering data for querying and analysis. OLAP and Data-Mining tools also frequently use data warehouses. Business Intelligence: Business Intelligence tends to increase the performance of various organizational business processes by utilizing powerful decision support systems. In the mean time, they also contribute to improving the support systems. Business Intelligence processes attempt to increase the Knowledge Reach of an organization by putting various Information Systems into perspective and increase the Strategic Flexibility of an organization, which may be defined as the capability of an organization for changing or developing new the marketing and development strategies without affecting their market presence. They analyze the information on current performance of organization, compare it with the past performance based on the past methods of information management and try to develop a decision support system (DSS), which can make accurate predictions for realizable productivity on employment of current resources, and possible fallacies in the present system which need to be eliminated or upgraded. Business Intelligence heavily depends upon technological advancements and hence organizations trying to increase their business intelligence significantly employ latest and most efficient technologies. The integration and uniformity of data is a very important challenge faced by enterprises and Business Intelligence prioritizes it so as to make uniform and correct data available for usage to all the users of the system. However, BI should not be confused with the term Business Process Management (BPM), a recently developed term, which increases the scope of Business Intelligence from reactive to proactive level. BPM is an approach for formulation of management strategies and their effective execution, depending heavily on IT services and technologies. Generally, the state of progress of an organization is measured and compared through some key performance indicators (KPI). Depending upon the nature and key business of the organization, the key performance indicators may differ. However, the basic indicators are usually revenue (current month, last month), variance, turn over etc. The key performance indicators can be used to analyze the current progress of an organization compared to other organizations as well as the growth achieved with respect to the past month (or year). Key performance indicators can be accessed through various web applications, dashboards or queries. Enterprise Performance management (EPM) is the key objective of Business Intelligence. Measure of Enterprise Performance management usually involves measure and analysis of some key performance indicators, depending upon the domain of organization. Enterprise Information Management (EIM) tools ensure high speed transactions and real-time management of data to accelerate the process of gathering and integration of data for increasing the Business Intelligence of an organization. Role of data Warehousing and Business Intelligence in Enterprise Information Management: Data warehousing evidently plays a central role in increasing Business Intelligence of an organization by smoothening the process of Data consolidation, analysis, integration and presentation. The separation of facts from dimensions while storage in Data warehouses can greatly simplify the process of querying key performance indicators for an organization. Both Data warehousing and Business Intelligence have become the subjects of extensive study and research. Major software companies like Oracle and IBM have developed numerable tools for developing stable warehouses and other data processing tools and mechanisms to grow decision support systems. Along with stable storage of data, a successful enterprise requires an agile information system with minimum inconsistencies, less maintenance costs and easy upgrade options.  Let us review the characteristics of a successful organization and estimate the role that can be played by Data Warehousing:   1) Agility: It is the ability of the organization to anticipate possible changes in their business environment, technology advancements, market fluctuations, changes in customer requirements etc. Along with the capability to anticipate the changes, an organization should be able to respond to a situation effectively by integrating its resources, initiating technological advancements in advance and changing market propositions.  Usually Data Warehouses contain large amount of data obtained through different feds and legacy systems. Agility requires that an organization can sense and respond to a situation in the smallest amount of time. However, due to its very nature of development, the structure of warehouses becomes very unsusceptible to change. It is inevitable for an organization to develop the warehouses in manner which facilitates later addition of different sources of data. If the architecture of Data Warehouses is proper, addition of proper feeds and trigger mechanisms based on trends in data or abrupt changes in type or quantity of information reaching the Warehouse can substantially contribute to the development of an agile enterprise.   2)    Dynamism: It represents the ability and extent of an organization to adapt to new scenarios in order to face challenges and achieve sustainable growth. Dynamic capabilities of a firm can be defined in terms of its ability of resource management, which can include procuring new resources, updating existing resources and releasing unused or effete resources. Capability of a firm to sense the necessity for manipulation of resources symbolizes its dynamism. Resources can denote technologies, manpower (employees), machines or processes. Significant alterations in business strategies may be required by an organization to cope with the ever changing environment and customer demands and for maintaining a competitive edge in the market. Such kind of strategy shift may require important upgrades in the way data is stored and processed. In order for complex solutions to evolve and adapt to other evolving challenges, entire system needs to be in place. 3)    Simplicity: Almost in every kind of organization, the success of depends upon the amount of complexity faced by strategy makers in assembling and analyzing information and then disseminating the information to end users (usually customers). Data Warehouses are always customized to ensure simplicity and maintainability of data presentation. This simplicity enables policy makers to understand the information easily and clearly and develop market oriented strategies as to utilize current market trends. Specialized Data Marts can exponentially increase the quality of analysis by giving more power and simplified structure for independent holders of such data units Challenges faced by developers and solutions:  Research in BI basically comprises of exploring new software technologies and increasing the level of intelligence of both customers and sellers towards developments in market. The shift towards automation of various businesses has been taking place at astronomical pace in all organizations. Most of the market transactions (buying, selling, auctioning) takes place on web. To maintain a dominant edge in competition, it is inevitable for every organization to have a technological know-how of leading web technologies and customer behavior. Most of the customers generally prefer simplicity and clarity of software processes related to market processes and this simplicity usually places an arduous task on the part of developers. Information systems should be seen as collection of interpersonal facts and mechanisms trying to improve knowledge management systems. Business Intelligence plays a central role in making the developers tasks easier by bringing highly advanced software tools, highly customized for database development. The crucial problems faced in development of data warehouses depend upon the kind of information that needs to be stored. While simple information management can be done using star shaped schema, storage of denser information requires complicate schemas like Snowflake schema, Starflake schema, Fact Constellations etc. Snowflake schema generally refines the dimensional hierarchy of star schema whereas Fact Constellations allow one table to contain dimensional data corresponding to more than one fact tables. Higher management, who usually deals with Metadata, the data about data (Neil Foshay, 2007) of warehouses remains completely ignorant about intricacies of inner details of Warehouses and hence unable to detect inconsistencies in the warehouses.Development and optimization of efficient and simpler queries is yet another challenge with data warehouses. Under utilization of warehouses is another problem hard to cope with. Even though terabytes of data are present in warehouses, utilization of data remains less than a fraction. Ownership related issues can arise when data flow form one warehouses to data-mart or vice-versa. Many Warehouse architectures have failed completely in the market under the ever-changing environment and continuous demand for upgrade and uniformity of data. Warehouses are infamous for being costly, complex and inclined to fail. More than 60% of warehouse projects result in failures. Conclusion: With rapidly changing knowledge industry and customer intelligence, enterprises are falling short of understanding the customer demands and requirements. Many data warehousing and Business Intelligence projects have been unsuccessful as a result of improper integration or usage of same vocabulary to clearly divided groups with different preferences and choice of data presentation. With the increasing awareness of significance of data warehouses, many third party softwares and tools form managing and improving data warehouses have come in the market. Warehouse designers are continuously trying to come with better methods of query optimization, data transfer and representation. Many warehouse designers use ontological semantics for getting a better representation of data while preparing the warehouse structure. With this pace of research and enthusiasm, it is not very difficult to imagine well-structured and comprehensive data warehouse being developed in near future, being able to process much larger and complex data for all kind of organizations. However, with the increasing competition in market for development of tools, many organizations, particular smaller firms, are still coping with lack of enough technical know-how for development of intelligent Information Management tools. Larger firms exercise strong monopoly in development of tools for maintenance of Warehouses, data processing and presentation. HP, Oracle, IBM, SAP etc are some of the major organizations possessing most successful warehouse design technologies and peripheral tools. Bibliography: Surajit Chaudhuri, Umeshwar Dayal (n.d.) An Overview of Data Warehousing and OLAP Technology Catherine L. Wang, Pervaiz K. Ahmed (2007) Dynamic capabilities: A review and research Agenda Eisenhardt, K.M., Martin, J.A. (2000). Dynamic capabilities: what are they? Strategic Management Journal, 21, 1105–1121 Eric Overby, Anandhi Bharadwaj, V. Sambamurthy (n.d.) Enterprise agility and the enabling role of information technology , Neil Foshay, Avinandan Mukherjee, Andrew Taylor (n.d.) DOES DATA WAREHOUSE END-USER METADATA ADD VALUE? Michael Uschold, Michael Gruninger (n.d.) Ontologies and Semantics for Seamless Connectivity Mark N. Frolick, Thilini R. Ariyachandra (2006) Buisness Performance Management : one truth Anderson, Philip (1999) Complexity theory and organization science Organization Science; May/Jun 1999; 10, 3; ABI/INFORM Research pg. 216 Ilkka Tuomi (2000) Data is more than knowledge: Implications of the reversed knowledge hierarch... Journal of Management Information Systems; 16, 3; ABI/INFORM Research pg. 103 Business Intelligence, 2008 (http://www.businessobjects.com/businessintelligence) Ricky Whiting, 2003, The Data-Warehouse Advantage (http://www.informationweek.com/story/showArticle.jhtml?articleID=12802974) Read More