P A R I T Introduction to Analytics and AI 1 CHAPTER 1 Overview of Business Intelligence, Analytics, Data Science, and Artificial Intelligence: Systems for Decision Support LEARNING OBJECTIVES Understand the need for computerized support of managerial decision making ■■ Understand the development of systems for providing decision-making support ■■ Recognize the evolution of such computerized support to the current state of analytics/data science and artificial intelligence ■■ Describe the business intelligence (BI) methodology and concepts ■■ T Understand the different types of analytics and review selected applications ■■ Understand the basic concepts of artificial intelligence (AI) and see selected applications ■■ Understand the analytics ecosystem to identify various key players and career opportunities ■■ he business environment (climate) is constantly changing, and it is becoming more and more complex. Organizations, both private and public, are under pressures that force them to respond quickly to changing conditions and to be innovative in the way they operate. Such activities require organizations to be agile and to make frequent and quick strategic, tactical, and operational decisions, some of which are very complex. Making such decisions may require considerable amounts of relevant data, information, and knowledge. Processing these in the framework of the needed decisions must be done quickly, frequently in real time, and usually requires some computerized support. As technologies are evolving, many decisions are being automated, leading to a major impact on knowledge work and workers in many ways. This book is about using business analytics and artificial intelligence (AI) as a computerized support portfolio for managerial decision making. It concentrates on the 2 Chapter 1 • Overview of Business Intelligence, Analytics, Data Science, and Artificial Intelligence theoretical and conceptual foundations of decision support as well as on the commercial tools and techniques that are available. The book presents the fundamentals of the techniques and the manner in which these systems are constructed and used. We follow an EEE (exposure, experience, and exploration) approach to introducing these topics. The book primarily provides exposure to various analytics/AI techniques and their applications. The idea is that students will be inspired to learn from how various organizations have employed these technologies to make decisions or to gain a competitive edge. We believe that such exposure to what is being accomplished with analytics and that how it can be achieved is the key component of learning about analytics. In describing the techniques, we also give examples of specific software tools that can be used for developing such applications. However, the book is not limited to any one software tool, so students can experience these techniques using any number of available software tools. We hope that this exposure and experience enable and motivate readers to explore the potential of these techniques in their own domain. To facilitate such exploration, we include exercises that direct the reader to Teradata University Network (TUN) and other sites that include team-oriented exercises where appropriate. In our own teaching experience, projects undertaken in the class facilitate such exploration after students have been exposed to the myriad of applications and concepts in the book and they have experienced specific software introduced by the professor. This introductory chapter provides an introduction to analytics and artificial intelligence as well as an overview of the book. The chapter has the following sections: 1.1 Opening Vignette: How Intelligent Systems Work for KONE Elevators and Escalators Company 3 1.2 Changing Business Environments and Evolving Needs for Decision Support and Analytics 5 1.3 Decision-Making Processes and Computer Decision Support Framework 9 1.4 Evolution of Computerized Decision Support to Business Intelligence/ Analytics/Data Science 22 1.5 Analytics Overview 30 1.6 Analytics Examples in Selected Domains 38 1.7 Artificial Intelligence Overview 52 1.8 Convergence of Analytics and AI 59 1.9 Overview of the Analytics Ecosystem 63 1.10 Plan of the Book 65 1.11 Resources, Links, and the Teradata University Network Connection 66 1.1 OPENING VIGNETTE: How Intelligent Systems Work for KONE Elevators and Escalators Company KONE is a global industrial company (based in Finland) that manufactures mostly elevators and escalators and also services over 1.1 million elevators, escalators, and related equipment in several countries. The company employs over 50,000 people. THE PROBLEM Over 1 billion people use the elevators and escalators manufactured and serviced by KONE every day. If equipment does not work properly, people may be late to work, cannot get home in time, and may miss important meetings and events. So, KONE’s objective is to minimize the downtime and users’ suffering. 3 4 Part I • Introduction to Analytics and AI The company has over 20,000 technicians who are dispatched to deal with the elevators anytime a problem occurs. As buildings are getting higher (the trend in many places), more people are using elevators, and there is more pressure on elevators to handle the growing amount of traffic. KONE faced the responsibility to serve users smoothly and safely. THE SOLUTION KONE decided to use IBM Watson IoT Cloud platform. As we will see in Chapter 6, IBM installed cognitive abilities in buildings that make it possible to recognize situations and behavior of both people and equipment. The Internet of Things (IoT), as we will see in Chapter 13, is a platform that can connect millions of “things” together and to a central command that can manipulate the connected things. Also, the IoT connects sensors that are attached to KONE’s elevators and escalators. The sensors collect information and data about the elevators (such as noise level) and other equipment in real time. Then, the IoT transfers to information centers via the collected data “cloud.” There, analytic systems (IBM Advanced Analytic Engine) and AI process the collected data and predict things such as potential failures. The systems also identify the likely causes of problems and suggest potential remedies. Note the predictive power of IBM Watson Analytics (using machine learning, an AI technology described in Chapters 4–6) for finding problems before they occur. The KONE system collects a significant amount of data that are analyzed for other purposes so that future design of equipment can be improved. This is because Watson Analytics offers a convenient environment for communication of and collaboration around the data. In addition, the analysis suggests how to optimize buildings and equipment operations. Finally, KONE and its customers can get insights regarding the financial aspects of managing the elevators. KONE also integrates the Watson capabilities with Salesforce’s service tools (Service Cloud Lightning and Field Service Lightning). This combination helps KONE to immediately respond to emergencies or soon-to-occur failures as quickly as possible, dispatching some of its 20,000 technicians to the problems’ sites. Salesforce also provides superb customer relationship management (CRM). The people–machine communication, query, and collaboration in the system are in a natural language (an AI capability of Watson Analytics; see Chapter 6). Note that IBM Watson analytics includes two types of analytics: predictive, which predicts when failures may occur, and prescriptive, which recommends actions (e.g., preventive maintenance). THE RESULTS KONE has minimized downtime and shortened the repair time. Obviously, elevators/ escalators users are much happier if they do not have problems because of equipment downtime, so they enjoy trouble-free rides. The prediction of “soon-to-happen” can save many problems for the equipment owners. The owners can also optimize the schedule of their own employees (e.g., cleaners and maintenance workers). All in all, the decision makers at both KONE and the buildings can make informed and better decisions. Some day in the future, robots may perform maintenance and repairs of elevators and escalators. Note: This case is a sample of IBM Watson’s success using its cognitive buildings capability. To learn more, we suggest you view the following YouTube videos: (1) youtube.com/watch?v=6UPJHyiJft0 (1:31 min.) (2017); (2) youtube.com/watch?v=EVbd3ejEXus (2:49 min.) (2017). Sources: Compiled from J. Fernandez. (2017, April). “A Billion People a Day. Millions of Elevators. No Room for Downtime.” IBM developer Works Blog. developer.ibm.com/dwblog/2017/kone-watson-video/ (accessed September 2018); H. Srikanthan. “KONE Improves ‘People Flow’ in 1.1 Million Elevators with IBM Watson IoT.” Generis. https://generisgp.com/2018/01/08/ibm-case-study-kone-corp/ (accessed September 2018); L. Slowey. (2017, February 16). “Look Who’s Talking: KONE Makes Elevator Services Truly Intelligent with Watson IoT.” IBM Internet of Things Blog. ibm.com/blogs/internet-of-things/kone/ (accessed September 2018). Chapter 1 • Overview of Business Intelligence, Analytics, Data Science, and Artificial Intelligence u QUESTIONS FOR THE OPENING VIGNETTE 1. It is said that KONE is embedding intelligence across its supply chain and enables smarter buildings. Explain. 2. Describe the role of IoT in this case. 3. What makes IBM Watson a necessity in this case? 4. Check IBM Advanced Analytics. What tools were included that relate to this case? 5. Check IBM cognitive buildings. How do they relate to this case? WHAT CAN WE LEARN FROM THIS VIGNETTE? Today, intelligent technologies can embark on large-scale complex projects when they include AI combined with IoT. The capabilities of integrated intelligent platforms, such as IBM Watson, make it possible to solve problems that were economically and technologically unsolvable just a few years ago. The case introduces the reader to several of the technologies, including advanced analytics, sensors, IoT, and AI that are covered in this book. The case also points to the use of “cloud.” The cloud is used to centrally process large amounts of information using analytics and AI algorithms, involving “things” in different locations. This vignette also introduces us to two major types of analytics: predictive analytics (Chapters 4–6) and prescriptive analytics (Chapter 8). Several AI technologies are discussed: machine learning, natural language processing, computer vision, and prescriptive analysis. The case is an example of augmented intelligence in which people and machines work together. The case illustrates the benefits to the vendor, the implementing companies, and their employees and to the users of the elevators and escalators. 1.2 CHANGING BUSINESS ENVIRONMENTS AND EVOLVING NEEDS FOR DECISION SUPPORT AND ANALYTICS Decision making is one of the most important activities in organizations of all kind— probably the most important one. Decision making leads to the success or failure of organizations and how well they perform. Making decisions is getting difficult due to internal and external factors. The rewards of making appropriate decisions can be very high and so can the loss of inappropriate ones. Unfortunately, it is not simple to make decisions. To begin with, there are several types of decisions, each of which requires a different decision-making approach. For example, De Smet et al. (2017) of McKinsey & Company management consultants classify organizational decision into the following four groups: • Big-bet, high-risk decisions. • Cross-cutting decisions, which are repetitive but high risk that require group work (Chapter 11). • Ad hoc decisions that arise episodically. • Delegated decisions to individuals or small groups. Therefore, it is necessary first to understand the nature of decision making. For a comprehensive discussion, see (De Smet et al. 2017). Modern business is full of uncertainties and rapid changes. To deal with these, organizational decision makers need to deal with ever-increasing and changing data. This book is about the technologies that can assist decision makers in their jobs. 5 6 Part I • Introduction to Analytics and AI Decision-Making Process For years, managers considered decision making purely an art—a talent acquired over a long period through experience (i.e., learning by trial and error) and by using intuition. Management was considered an art because a variety of individual styles could be used in approaching and successfully solving the same types of managerial problems. These styles were often based on creativity, judgment, intuition, and experience rather than on systematic quantitative methods grounded in a scientific approach. However, recent research suggests that companies with top managers who are more focused on persistent work tend to outperform those with leaders whose main strengths are interpersonal communication skills. It is more important to emphasize methodical, thoughtful, analytical decision making rather than flashiness and interpersonal communication skills. Managers usually make decisions by following a four-step process (we learn more about these in the next section): 1. Define the problem (i.e., a decision situation that may deal with some difficulty or with an opportunity). 2. Construct a model that describes the real-world problem. 3. Identify possible solutions to the modeled problem and evaluate the solutions. 4. Compare, choose, and recommend a potential solution to the problem. A more detailed process is offered by Quain (2018), who suggests the following steps: 1. 2. 3. 4. 5. 6. 7. Understand the decision you have to make. Collect all the information. Identify the alternatives. Evaluate the pros and cons. Select the best alternative. Make the decision. Evaluate the impact of your decision. We will return to this process in Section 1.3. The Influence of the External and Internal Environments on the Process To follow these decision-making processes, one must make sure that sufficient alternative solutions, including good ones, are being considered, that the consequences of using these alternatives can be reasonably predicted, and that comparisons are done properly. However, rapid changes in internal and external environments make such an evaluation process difficult for the following reasons: • Technology, information systems, advanced search engines, and globalization result in more and more alternatives from which to choose. • Government regulations and the need for compliance, political instability and terrorism, competition, and changing consumer demands produce more uncertainty, making it more difficult to predict consequences and the future. • Political factors. Major decisions may be influenced by both external and internal politics. An example is the 2018 trade war on tariffs. • Economic factors. These range from competition to the genera and state of the economy. These factors, both in the short and long run, need to be considered. Chapter 1 • Overview of Business Intelligence, Analytics, Data Science, and Artificial Intelligence • Sociological and psychological factors regarding employees and customers. These need to be considered when changes are being made. • Environment factors. The impact on the physical environment must be assessed in many decision-making situations. Other factors include the need to make rapid decisions, the frequent and unpredictable changes that make trial-and-error learning difficult, and the potential costs of making mistakes that may be large. These environments are growing more complex every day. Therefore, making decisions today is indeed a complex task. For further discussion, see Charles (2018). For how to make effective decisions under uncertainty and pressure, see Zane (2016). Because of these trends and changes, it is nearly impossible to rely on a trialand-error approach to management. Managers must be more sophisticated; they must use the new tools and techniques of their fields. Most of those tools and techniques are discussed in this book. Using them to support decision making can be extremely rewarding in making effective decisions. Further, many tools that are evolving impact even the very existence of several decision-making tasks that are being automated. This impacts future demand for knowledge workers and begs many legal and societal impact questions. Data and Its Analysis in Decision Making We will see several times in this book how an entire industry can employ analytics to develop reports on what is happening, predict what is likely to happen, and then make decisions to make the best use of the situation at hand. These steps require an organization to collect and analyze vast stores of data. In general, the amount of data doubles every two years. From traditional uses in payroll and bookkeeping functions, computerized systems are now used for complex managerial areas ranging from the design and management of automated factories to the application of analytical methods for the evaluation of proposed mergers and acquisitions. Nearly all executives know that information technology is vital to their business and extensively use these technologies. Computer applications have moved from transaction-processing and monitoring activities to problem analysis and solution applications, and much of the activity is done with cloud-based technologies, in many cases accessed through mobile devices. Analytics and BI tools such as data warehousing, data mining, online analytical processing (OLAP), dashboards, and the use of cloud-based systems for decision support are the cornerstones of today’s modern management. Managers must have high-speed, networked information systems (wired or wireless) to assist them with their most important task: making decisions. In many cases, such decisions are routinely being fully automated (see Chapter 2), eliminating the need for any managerial intervention. Technologies for Data Analysis and Decision Support Besides the obvious growth in hardware, software, and network capacities, some developments have clearly contributed to facilitating the growth of decision support and analytics technologies in a number of ways: • Group communication and collaboration. Many decisions are made today by groups whose members may be in different locations. Groups can collaborate and communicate readily by using collaboration tools as well as the ubiquitous smartphones. Collaboration is especially important along the supply chain, where partners—all the way from vendors to customers—must share information. Assembling a group of decision makers, especially experts, in one place can be 7 8 Part I • Introduction to Analytics and AI costly. Information systems can improve the collaboration process of a group and enable its members to be at different locations (saving travel costs). More critically, such supply chain collaboration permits manufacturers to know about the changing patterns of demand in near real time and thus react to marketplace changes faster. For a comprehensive coverage and the impact of AI, see Chapters 2, 10, and 14. • Improved data management. Many decisions involve complex computations. Data for these can be stored in different databases anywhere in the organization and even possibly outside the organization. The data may include text, sound, graphics, and video, and these can be in different languages. Many times it is necessary to transmit data quickly from distant locations. Systems today can search, store, and transmit needed data quickly, economically, securely, and transparently. See Chapters 3 and 9 and the online chapter for details. • Managing giant data warehouses and Big Data. Large data warehouses (DWs), like the ones operated by Walmart, contain huge amounts of data. Special methods, including parallel computing and Hadoop/Spark, are available to organize, search, and mine the data. The costs related to data storage and mining are declining rapidly. Technologies that fall under the broad category of Big Data have enabled massive data coming from a variety of sources and in many different forms, which allows a very different view of organizational performance that was not possible in the past. See Chapter 9 for details. • Analytical support. With more data and analysis technologies, more alternatives can be evaluated, forecasts can be improved, risk analysis can be performed quickly, and the views of experts (some of whom may be in remote locations) can be collected quickly and at a reduced cost. Expertise can even be derived directly from analytical systems. With such tools, decision makers can perform complex simulations, check many possible scenarios, and assess diverse impacts quickly and economically.This, of course, is the focus of several chapters in the book. See Chapters 4–7. • Overcoming cognitive limits in processing and storing information. The human mind has only a limited ability to process and store information. People sometimes find it difficult to recall and use information in an error-free fashion due to their cognitive limits. The term cognitive limits indicates that an individual’s problem-solving capability is limited when a wide range of diverse information and knowledge is required. Computerized systems enable people to overcome their cognitive limits by quickly accessing and processing vast amounts of stored information. One way to overcome humans’ cognitive limitations is to use AI support. For coverage of cognitive aspects, see Chapter 6. • Knowledge management. Organizations have gathered vast stores of information about their own operations, customers, internal procedures, employee interactions, and so forth through the unstructured and structured communications taking place among various stakeholders. Knowledge management systems (KMS) have become sources of formal and informal support for decision making to managers, although sometimes they may not even be called KMS. Technologies such as text analytics and IBM Watson are making it possible to generate value from such knowledge stores. (See Chapters 6 and 12 for details. • Anywhere, anytime support. Using wireless technology, managers can access information anytime and from any place, analyze and interpret it, and communicate with those using it. This perhaps is the biggest change that has occurred in the last few years. The speed at which information needs to be processed and converted into decisions has truly changed expectations for both consumers and businesses. These and other capabilities have been driving the use of computerized decision support since the late 1960s, especially since the mid-1990s. The growth of mobile technologies, social media platforms, and analytical tools has enabled a different level of information systems (IS) to support managers. This growth in providing Chapter 1 • Overview of Business Intelligence, Analytics, Data Science, and Artificial Intelligence data-driven support for any decision extends not just to managers but also to consumers. We will first study an overview of technologies that have been broadly referred to as BI. From there we will broaden our horizons to introduce various types of analytics. • Innovation and artificial intelligence. Because of the complexities in the decision-making process discussed earlier and the environment surrounding the process, a more innovative approach is frequently need. A major facilitation of innovation is provided by AI. Almost every step in the decision-making process can be influenced by AI. AI is also integrated with analytics, creating synergy in making decisions (Section 1.8). u SECTION 1.2 REVIEW QUESTIONS 1. 2. 3. 4. Why is it difficult to make organizational decisions? Describe the major steps in the decision-making process. Describe the major external environments that can impact decision making. What are some of the key system-oriented trends that have fostered IS-supported decision making to a new level? 5. List some capabilities of information technologies that can facilitate managerial decision making. 1.3 DECISION-MAKING PROCESSES AND COMPUTERIZED DECISION SUPPORT FRAMEWORK In this section, we focus on some classical decision-making fundamentals and in more detail on the decision-making process. These two concepts will help us ground much of what we will learn in terms of analytics, data science, and artificial intelligence. Decision making is a process of choosing among two or more alternative courses of action for the purpose of attaining one or more goals. According to Simon (1977), managerial decision making is synonymous with the entire management process. Consider the important managerial function of planning. Planning involves a series of decisions: What should be done? When? Where? Why? How? By whom? Managers set goals, or plan; hence, planning implies decision making. Other managerial functions, such as organizing and controlling, also involve decision making. Simon’s Process: Intelligence, Design, and Choice It is advisable to follow a systematic decision-making process. Simon (1977) said that this involves three major phases: intelligence, design, and choice. He later added a fourth phase: implementation. Monitoring can be considered a fifth phase—a form of feedback. However, we view monitoring as the intelligence phase applied to the implementation phase. Simon’s model is the most concise and yet complete characterization of rational decision making. A conceptual picture of the decision-making process is shown in Figure 1.1. It is also illustrated as a decision support approach using modeling. There is a continuous flow of activity from intelligence to design to choice (see the solid lines in Figure 1.1), but at any phase, there may be a return to a previous phase (feedback). Modeling is an essential part of this process. The seemingly chaotic nature of following a haphazard path from problem discovery to solution via decision making can be explained by these feedback loops. The decision-making process starts with the intelligence phase; in this phase, the decision maker examines reality and identifies and defines the problem. Problem ownership is established as well. In the design phase, a model that represents the system is constructed. This is done by making assumptions that simplify reality and by writing down 9 10 Part I • Introduction to Analytics and AI Intelligence Reality Simplification Assumptions Organization objectives Search and scanning procedures Data collection Problem identification Problem ownership Problem classification Problem statement Problem Statement Design Validation of the Model Formulate a model Set criteria for choice Search for alternatives Predict and measure outcomes Alternatives Success Choice Verification, Testing of the Proposed Solution Solution to the model Sensitivity analysis Selection to the best (good) alternative(s) Plan for implementation Implementation of the solution Failure FIGURE 1.1 The Decision-Making/Modeling Process. the relationships among all the variables. The model is then validated, and criteria are determined in a principle of choice for evaluation of the alternative courses of action that are identified. Often, the process of model development identifies alternative solutions and vice versa. The choice phase includes the selection of a proposed solution to the model (not necessarily to the problem it represents). This solution is tested to determine its viability. When the proposed solution seems reasonable, we are ready for the last phase: implementation of the decision (not necessarily of a system). Successful implementation results in solving the real problem. Failure leads to a return to an earlier phase of the process. In fact, we can return to an earlier phase during any of the latter three phases. The decisionmaking situations described in the opening vignette follow Simon’s four-phase model, as do almost all other decision-making situations. The Intelligence Phase: Problem (or Opportunity) Identification The intelligence phase begins with the identification of organizational goals and objectives related to an issue of concern (e.g., inventory management, job selection, lack of or incorrect Web presence) and determination of whether they are being met. Problems occur because of dissatisfaction with the status quo. Dissatisfaction is the result of a difference between what people desire (or expect) and what is occurring. In this first phase, a decision maker attempts to determine whether a problem exists, identify its symptoms, determine its magnitude, and Chapter 1 • Overview of Business Intelligence, Analytics, Data Science, and Artificial Intelligence explicitly define it. Often, what is described as a problem (e.g., excessive costs) may be only a symptom (i.e., measure) of a problem (e.g., improper inventory levels). Because real-world problems are usually complicated by many interrelated factors, it is sometimes difficult to distinguish between the symptoms and the real problem. New opportunities and problems certainly may be uncovered while investigating the causes of symptoms. The existence of a problem can be determined by monitoring and analyzing the organization’s productivity level. The measurement of productivity and the construction of a model are based on real data. The collection of data and the estimation of future data are among the most difficult steps in the analysis. ISSUES IN DATA COLLECTION The following are some issues that may arise during data collection and estimation and thus plague decision makers: • Data are not available. As a result, the model is made with and relies on potentially inaccurate estimates. • Obtaining data may be expensive. • Data may not be accurate or precise enough. • Data estimation is often subjective. • Data may be insecure. • Important data that influence the results may be qualitative (soft). • There may be too many data (i.e., information overload). • Outcomes (or results) may occur over an extended period. As a result, revenues, expenses, and profits will be recorded at different points in time. To overcome this difficulty, a present-value approach can be used if the results are quantifiable. • It is assumed that future data will be similar to historical data. If this is not the case, the nature of the change has to be predicted and included in the analysis. When the preliminary investigation is completed, it is possible to determine whether a problem really exists, where it is located, and how significant it is. A key issue is whether an information system is reporting a problem or only the symptoms of a problem. For example, if reports indicate that sales are down, there is a problem, but the situation, no doubt, is symptomatic of the problem. It is critical to know the real problem. Sometimes it may be a problem of perception, incentive mismatch, or organizational processes rather than a poor decision model. To illustrate why it is important to identify the problem correctly, we provide a classical example in Application Case 1.1. Application Case 1.1 Making Elevators Go Faster! This story has been reported in numerous places and has almost become a classic example to explain the need for problem identification. Ackoff (as cited in Larson, 1987) described the problem of managing complaints about slow elevators in a tall hotel tower. After trying many solutions for reducing the complaint—staggering elevators to go to different floors, adding operators, and so on—the management determined that the real problem was not about the actual waiting time but rather the perceived waiting time. So the solution was to install full-length mirrors on elevator doors on each floor. As Hesse and Woolsey (1975) put it, “The women would look at themselves in the mirrors and make adjustments, while the men would look at the women, and before they knew it, the elevator was there.” By reducing the perceived waiting time, the problem went away. Baker and Cameron (1996) (Continued ) 11 12 Part I • Introduction to Analytics and AI Application Case 1.1 (Continued) give several other examples of distractions, including lighting and displays, that organizations use to reduce perceived waiting time. If the real problem is identified as perceived waiting time, it can make a big difference in the proposed solutions and their costs. For example, full-length mirrors probably cost a whole lot less than adding an elevator! Sources: Based on J. Baker and M. Cameron. (1996, September). “The Effects of the Service Environment on Affect and Consumer Perception of Waiting Time: An Integrative Review and Research Propositions,” Journal of the Academy of Marketing Science, 24, pp. 338–349; R. Hesse and G. Woolsey (1975). Applied Management Science: A Quick and Dirty Approach. Chicago, IL: SRA Inc; R. C. Larson. (1987, November/December). “Perspectives on Queues: Social Justice and the Psychology of Queuing.” Operations Research, 35(6), pp. 895–905. Questions for Case 1.1 1. Why this is an example relevant to decision making? 2. Relate this situation to the intelligence phase of decision making. PROBLEM CLASSIFICATION Problem classification is the conceptualization of a problem in an attempt to place it in a definable category, possibly leading to a standard solution approach. An important approach classifies problems according to the degree of structuredness evident in them. This ranges from totally structured (i.e., programmed) to totally unstructured (i.e., unprogrammed). Many complex problems can be divided into subproblems. Solving the simpler subproblems may help in solving a complex problem. Also, seemingly poorly structured problems sometimes have highly structured subproblems. Just as a semistructured problem results when some phases of decision making are structured whereas other phases are unstructured, and when some subproblems of a decision-making problem are structured with others unstructured, the problem itself is semistructured. As a decision support system is developed and the decision maker and development staff learn more about the problem, it gains structure. PROBLEM DECOMPOSITION In the intelligence phase, it is important to establish problem ownership. A problem exists in an organization only if someone or some group takes the responsibility for attacking it and if the organization has the ability to solve it. The assignment of authority to solve the problem is called problem ownership. For example, a manager may feel that he or she has a problem because interest rates are too high. Because interest rate levels are determined at the national and international levels and most managers can do nothing about them, high interest rates are the problem of the government, not a problem for a specific company to solve. The problem that companies actually face is how to operate in a high interest-rate environment. For an individual company, the interest rate level should be handled as an uncontrollable (environmental) factor to be predicted. When problem ownership is not established, either someone is not doing his or her job or the problem at hand has yet to be identified as belonging to anyone. It is then important for someone to either volunteer to own it or assign it to someone. The intelligence phase ends with a formal problem statement. PROBLEM OWNERSHIP The Design Phase The design phase involves finding or developing and analyzing possible courses of action. These include understanding the problem and testing solutions for feasibility. A model of the decision-making problem is constructed, tested, and validated. Let us first define a model. Chapter 1 • Overview of Business Intelligence, Analytics, Data Science, and Artificial Intelligence MODELS A major characteristic of computerized decision support and many BI tools (notably those of business analytics) is the inclusion of at least one model. The basic idea is to perform the analysis on a model of reality rather than on the real system. A model is a simplified representation or abstraction of reality. It is usually simplified because reality is too complex to describe exactly and because much of the complexity is actually irrelevant in solving a specific problem. Modeling involves conceptualizing a problem and abstracting it to quantitative and/ or qualitative form. For a mathematical model, the variables are identified and their mutual relationships are established. Simplifications are made, whenever necessary, through assumptions. For example, a relationship between two variables may be assumed to be linear even though in reality there may be some nonlinear effects. A proper balance between the level of model simplification and the representation of reality must be obtained because of the cost–benefit trade-off. A simpler model leads to lower development costs, easier manipulation, and a faster solution but is less representative of the real problem and can produce inaccurate results. However, a simpler model generally requires fewer data, or the data are aggregated and easier to obtain. The Choice Phase Choice is the critical act of decision making. The choice phase is the one in which the actual decision and the commitment to follow a certain course of action are made. The boundary between the design and choice phases is often unclear because certain activities can be performed during both of them and because the decision maker can return frequently from choice activities to design activities (e.g., generate new alternatives while performing an evaluation of existing ones). The choice phase includes the search for, evaluation of, and recommendation of an appropriate solution to a model. A solution to a model is a specific set of values for the decision variables in a selected alternative. Choices can be evaluated as to their viability and profitability. Each alternative must be evaluated. If an alternative has multiple goals, they must all be examined and balanced against each other. Sensitivity analysis is used to determine the robustness of any given alternative; slight changes in the parameters should ideally lead to slight or no changes in the alternative chosen. What-if analysis is used to explore major changes in the parameters. Goal seeking helps a manager determine values of the decision variables to meet a specific objective. These topics are addressed in Chapter 8. The Implementation Phase In The Prince, Machiavelli astutely noted some 500 years ago that there was “nothing more difficult to carry out, nor more doubtful of success, nor more dangerous to handle, than to initiate a new order of things.” The implementation of a proposed solution to a problem is, in effect, the initiation of a new order of things or the introduction of change. And change must be managed. User expectations must be managed as part of change management. The definition of implementation is somewhat complicated because implementation is a long, involved process with vague boundaries. Simplistically, the implementation phase involves putting a recommended solution to work, not necessarily implementing a computer system. Many generic implementation issues, such as resistance to change, degree of support of top management, and user training, are important in dealing with information system–supported decision making. Indeed, many previous technologyrelated waves (e.g., business process reengineering [BPR] and knowledge management) have faced mixed results mainly because of change management challenges and issues. Management of change is almost an entire discipline in itself, so we recognize its importance and encourage readers to focus on it independently. Implementation also includes 13 14 Part I • Introduction to Analytics and AI a thorough understanding of project management. The importance of project management goes far beyond analytics, so the last few years have witnessed a major growth in certification programs for project managers. A very popular certification now is the Project Management Professional (PMP). See pmi.org for more details. Implementation must also involve collecting and analyzing data to learn from the previous decisions and improve the next decision. Although analysis of data is usually conducted to identify the problem and/or the solution, analytics should also be employed in the feedback process. This is especially true for any public policy decisions. We need to be sure that the data being used for problem identification is valid. Sometimes people find this out only after the implementation phase. The decision-making process, though conducted by people, can be improved with computer support, which is introduced next. The Classical Decision Support System Framework The early definitions of decision support system (DSS) identified it as a system intended to support managerial decision makers in semistructured and unstructured decision situations. DSS was meant to be an adjunct to decision makers, extending their capabilities but not replacing their judgment. DSS was aimed at decisions that required judgment or at decisions that could not be completely supported by algorithms. Not specifically stated but implied in the early definitions was the notion that the system would be computer based, would operate interactively online, and preferably would have graphical output capabilities, now simplified via browsers and mobile devices. An early framework for computerized decision support includes several major concepts that are used in forthcoming sections and chapters of this book. Gorry and ScottMorton created and used this framework in the early 1970s, and the framework then evolved into a new technology called DSS. Gorry and Scott-Morton (1971) proposed a framework that is a 3-by-3 matrix, as shown in Figure 1.2. The two dimensions are the degree of structuredness and the types of control. The left side of Figure 1.2 is based on Simon’s (1977) idea that decision-making processes fall along a continuum that ranges from highly structured (sometimes called programmed) to highly unstructured (i.e., non-programmed) decisions. Structured processes are routine and typically repetitive problems for which standard solution methods exist. Unstructured processes are fuzzy, complex problems for which there are no cut-and-dried solution methods. An unstructured problem is one where the articulation of the problem or the solution approach may be unstructured in itself. In a structured problem, the procedures for obtaining the best (or at least a good enough) solution are known. Whether the problem involves finding an appropriate inventory level or choosing an optimal investment strategy, the objectives are clearly defined. Common objectives are cost minimization and profit maximization. Semistructured problems fall between structured and unstructured problems, having some structured elements and some unstructured elements. Keen and Scott-Morton (1978) mentioned trading bonds, setting marketing budgets for consumer products, and performing capital acquisition analysis as semistructured problems. DEGREE OF STRUCTUREDNESS The second half of the Gorry and Scott-Morton (1971) framework (refer to Figure 1.2) is based on Anthony’s (1965) taxonomy, which defines three broad categories that encompass all managerial activities: strategic planning, which involves defining long-range goals and policies for resource allocation; management control, the TYPES OF CONTROL Chapter 1 • Overview of Business Intelligence, Analytics, Data Science, and Artificial Intelligence Type of Control Type of Decision Operational Control Managerial Control 1 Structured 2 3 Managing finances Analyzing budget Forecasting short-term Monitoring investment portfolio Reporting on personnel Locating warehouse Making or buying Monitoring distribution systems Monitoring accounts receivable Monitoring accounts payable Placing order entries 4 Scheduling production Controlling inventory Semistructured 5 Evaluating credit Preparing budget Laying out plant Scheduling project Designing reward system Categorizing inventory 7 Unstructured Strategic Planning Buying software Approving loans Operating a help desk Selecting a cover for a magazine 8 6 Building a new plant Planning mergers and acquisitions Planning new products Planning compensation Providing quality assurance Establishing human resources policies Planning inventory 9 Planning research and Negotiating development Recruiting an executive Buying hardware Developing new Lobbying technologies Planning social responsibility FIGURE 1.2 Decision Support Frameworks. acquisition and efficient use of resources in the accomplishment of organizational goals; and operational control, the efficient and effective execution of specific tasks. THE DECISION SUPPORT MATRIX Anthony’s (1965) and Simon’s (1977) taxonomies are combined in the nine-cell decision support matrix shown in Figure 1.2. The initial purpose of this matrix was to suggest different types of computerized support to different cells in the matrix. Gorry and Scott-Morton (1971) suggested, for example, that for making semistructured decisions and unstructured decisions, conventional management information systems (MIS) and management science (MS) tools are insufficient. Human intellect and a different approach to computer technologies are necessary. They proposed the use of a supportive information system, which they called a DSS. Note that the more structured and operational control-oriented tasks (such as those in cells 1, 2, and 4 of Figure 1.2) are usually performed by lower-level managers, whereas the tasks in cells 6, 8, and 9 are the responsibility of top executives or highly trained specialists. Since the 1960s, computers have historically supported structured and some semistructured decisions, especially those that involve operational and managerial control. Operational and managerial control decisions are made in all functional areas, especially in finance and production (i.e., operations) management. COMPUTER SUPPORT FOR STRUCTURED DECISIONS 15 16 Part I • Introduction to Analytics and AI Structured problems, which are encountered repeatedly, have a high level of structure, as their name suggests. It is therefore possible to abstract, analyze, and classify them into specific categories. For example, a make-or-buy decision is one category. Other examples of categories are capital budgeting, allocation of resources, distribution, procurement, planning, and inventory control decisions. For each category of decision, an easy-to-apply prescribed model and solution approach have been developed, generally as quantitative formulas. Therefore, it is possible to use a scientific approach for automating portions of managerial decision making. Solutions to many structured problems can be fully automated (see Chapters 2 and 12). COMPUTER SUPPORT FOR UNSTRUCTURED DECISIONS Unstructured problems can be only partially supported by standard computerized quantitative methods. It is usually necessary to develop customized solutions. However, such solutions may benefit from data and information generated from corporate or external data sources. Intuition and judgment may play a large role in these types of decisions, as may computerized communication and collaboration technologies, as well as cognitive computing (Chapter 6) and deep learning (Chapter 5). Solving semistructured problems may involve a combination of standard solution procedures and human judgment. Management science can provide models for the portion of a decision-making problem that is structured. For the unstructured portion, a DSS can improve the quality of the information on which the decision is based by providing, for example, not only a single solution, but also a range of alternative solutions along with their potential impacts. These capabilities help managers to better understand the nature of problems and, thus, to make better decisions. COMPUTER SUPPORT FOR SEMISTRUCTURED PROBLEMS The early definitions of DSS identified it as a system intended to support managerial decision makers in semistructured and unstructured decision situations. DSS was meant to be an adjunct to decision makers, extending their capabilities but not replacing their judgment. It was aimed at decisions that required judgment or at decisions that could not be completely supported by algorithms. Not specifically stated but implied in the early definitions was the notion that the system would be computer based, would operate interactively online, and preferably would have graphical output capabilities, now simplified via browsers and mobile devices. DECISION SUPPORT SYSTEM: CAPABILITIES A DSS Application A DSS is typically built to support the solution of a certain problem or to evaluate an opportunity. This is a key difference between DSS and BI applications. In a very strict sense, business intelligence (BI) systems monitor situations and identify problems and/or opportunities using analytic methods. Reporting plays a major role in BI; the user generally must identify whether a particular situation warrants attention and then can apply analytical methods. Again, although models and data access (generally through a data warehouse) are included in BI, a DSS may have its own databases and is developed to solve a specific problem or set of problems and are therefore called DSS applications. Formally, a DSS is an approach (or methodology) for supporting decision making. It uses an interactive, flexible, adaptable computer-based information system (CBIS) especially developed for supporting the solution to a specific unstructured management problem. It uses data, provides an easy user interface, and can incorporate the decision maker’s own insights. In addition, a DSS includes models and is developed (possibly by Chapter 1 • Overview of Business Intelligence, Analytics, Data Science, and Artificial Intelligence 14 Can be standalone, integrated, and Webbased tool 13 1 Provides support for semistructured or unstructured problems 2 Supports managers at all levels Provides data access 3 Provides models and analysis 4 Supports individuals and groups 12 11 Supports interdependent or sequential decisions Decision Support System (DSS) Provides ease of development by end users 5 Supports intelligence, design, choice, and implementation 10 Provides complete human control of the process 6 9 Improves effectiveness and efficiency 8 7 Provides interactivity, ease of use Support variety of decision processes and styles Is adaptable and flexible FIGURE 1.3 Key Characteristics and Capabilities of DSS. end users) through an interactive and iterative process. It can support all phases of decision making and may include a knowledge component. Finally, a DSS can be used by a single user or can be Web based for use by many people at several locations. THE CHARACTERISTICS AND CAPABILITIES OF DSS Because there is no consensus on exactly what a DSS is, there is obviously no agreement on the standard characteristics and capabilities of DSS. The capabilities in Figure 1.3 constitute an ideal set, some members of which are described in the definitions of DSS and illustrated in the application cases. The key characteristics and capabilities of DSS (as shown in Figure 1.3) are as follows: 1. Supports decision makers, mainly in semistructured and unstructured situations, by bringing together human judgment and computerized information. Such problems cannot be solved (or cannot be solved conveniently) by other computerized systems or through use of standard quantitative methods or tools. Generally, these problems gain structure as the DSS is developed. Even some structured problems have been solved by DSS. 2. Supports all managerial levels, ranging from top executives to line managers. 3. Supports individuals as well as groups. Less-structured problems often require the involvement of individuals from different departments and organizational levels or even from different organizations. DSS supports virtual teams through collaborative Web tools. DSS has been developed to support individual and group work as well 17 18 Part I • Introduction to Analytics and AI 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. as to support individual decision making and groups of decision makers working somewhat independently. Supports interdependent and/or sequential decisions. The decisions may be made once, several times, or repeatedly. Supports all phases of the decision-making process: intelligence, design, choice, and implementation. Supports a variety of decision-making processes and styles. Is flexible, so users can add, delete, combine, change, or rearrange basic elements. The decision maker should be reactive, able to confront changing conditions quickly, and able to adapt the DSS to meet these changes. It is also flexible in that it can be readily modified to solve other, similar problems. Is user-friendly, has strong graphical capabilities, and a natural language interactive human-machine interface can greatly increase the effectiveness of DSS. Most new DSS applications use Web-based interfaces or mobile platform interfaces. Improves the effectiveness of decision making (e.g., accuracy, timeliness, quality) rather than its efficiency (e.g., the cost of making decisions). When DSS is deployed, decision making often takes longer, but the decisions are better. Provides complete control by the decision maker over all steps of the decisionmaking process in solving a problem. A DSS specifically aims to support, not to replace, the decision maker. Enables end users to develop and modify simple systems by themselves. Larger systems can be built with assistance from IS specialists. Spreadsheet packages have been utilized in developing simpler systems. OLAP and data mining software in conjunction with data warehouses enable users to build fairly large, complex DSS. Provides models that are generally utilized to analyze decision-making situations. The modeling capability enables experimentation with different strategies under different configurations. Provides access to a variety of data sources, formats, and types, including GIS, multimedia, and object-oriented data. Can be employed as a stand-alone tool used by an individual decision maker in one location or distributed throughout an organization and in several organizations along the supply chain. It can be integrated with other DSS and/or applications, and it can be distributed internally and externally, using networking and Web technologies. These key DSS characteristics and capabilities allow decision makers to make better, more consistent decisions in a timely manner, and they are provided by major DSS components, Components of a Decision Support System A DSS application can be composed of a data management subsystem, a model management subsystem, a user interface subsystem, and a knowledge-based management subsystem. We show these in Figure 1.4. The Data Management Subsystem The data management subsystem includes a database that contains relevant data for the situation and is managed by software called the database management system (DBMS). DBMS is used as both singular and plural (system and systems) terms, as are many other acronyms in this text. The data management subsystem can be interconnected with the corporate data warehouse, a repository for corporate relevant decision-making data. Chapter 1 • Overview of Business Intelligence, Analytics, Data Science, and Artificial Intelligence Data: internal and/or external ERP/POS Other Computer-Based Systems Data Management Legacy Internet, Intranet, Extranet Model Management External Models Knowledge-Based Subsystems Web, etc. User Interface Organizational Knowledgebase Manager (user) FIGURE 1.4 Schematic View of DSS. Usually, the data are stored or accessed via a database Web server. The data management subsystem is composed of the following elements: • DSS database • Database management system • Data directory • Query facility Many of the BI or descriptive analytics applications derive their strength from the data management side of the subsystems. The Model Management Subsystem The model management subsystem is the component that includes financial, statistical, management science, or other quantitative models that provide the system’s analytical capabilities and appropriate software management. Modeling languages for building custom models are also included. This software is often called a model base management system (MBMS). This component can be connected to corporate or external storage of models. Model solution methods and management systems are implemented in Web development systems (such as Java) to run on application servers. The model management subsystem of a DSS is composed of the following elements: • Model base • MBMS • Modeling language • Model directory • Model execution, integration, and command processor Because DSS deals with semistructured or unstructured problems, it is often necessary to customize models, using programming tools and languages. Some examples of these are .NET Framework languages, C++, and Java. OLAP software may also be used to work with models in data analysis. Even languages for simulations such as Arena and 19 20 Part I • Introduction to Analytics and AI Application Case 1.2 SNAP DSS Helps OneNet Make Telecommunications Rate Decisions Telecommunications network services to educational institutions and government entities are typically provided by a mix of private and public organizations. Many states in the United States have one or more state agencies that are responsible for providing network services to schools, colleges, and other state agencies. One example of such an agency is OneNet in Oklahoma. OneNet is a division of the Oklahoma State Regents for Higher Education and operated in cooperation with the Office of State Finance. Usually agencies such as OneNet operate as an enterprise-type fund. They must recover their costs through billing their clients and/or by justifying appropriations directly from the state legislatures. This cost recovery should occur through a pricing mechanism that is efficient, simple to implement, and equitable. This pricing model typically needs to recognize many factors: convergence of voice, data, and video traffic on the same infrastructure; diversity of user base in terms of educational institutions and state agencies; diversity of applications in use by state clients from e-mail to videoconferences, IP telephoning, and distance learning; recovery of current costs as well as planning for upgrades and future developments; and leverage of the shared infrastructure to enable further economic development and collaborative work across the state that leads to innovative uses of OneNet. These considerations led to the development of a spreadsheet-based model. The system, SNAP-DSS, or Service Network Application and Pricing (SNAP)based DSS, was developed in Microsoft Excel 2007 and used the VBA programming language. The SNAP-DSS offers OneNet the ability to select the rate card options that best fit the preferred pricing strategies by providing a real-time, userfriendly, graphical user interface (GUI). In addition, the SNAP-DSS not only illustrates the influence of the changes in the pricing factors on each rate card option but also allows the user to analyze various rate card options in different scenarios using different parameters. This model has been used by OneNet financial planners to gain insights into their customers and analyze many what-if scenarios of different rate plan options. Source: Based on J. Chongwatpol and R. Sharda. (2010, December). “SNAP: A DSS to Analyze Network Service Pricing for State Networks.” Decision Support Systems, 50(1), pp. 347–359. statistical packages such as those of SPSS offer modeling tools developed through the use of a proprietary programming language. For small- and medium-sized DSS or for less complex ones, a spreadsheet (e.g., Excel) is usually used. We use Excel for several examples in this book. Application Case 1.2 describes a spreadsheet-based DSS. The User Interface Subsystem The user communicates with and commands the DSS through the user interface subsystem. The user is considered part of the system. Researchers assert that some of the unique contributions of DSS are derived from the intensive interaction between the computer and the decision maker. A difficult user interface is one of the major reasons that managers do not use computers and quantitative analyses as much as they could, given the availability of these technologies. The Web browser provided a familiar, consistent GUI structure for many DSS in the 2000s. For locally used DSS, a spreadsheet also provides a familiar user interface. The Web browser has been recognized as an effective DSS GUI because it is flexible, user-friendly, and a gateway to almost all sources of necessary information and data. Essentially, Web browsers have led to the development of portals and dashboards, which front end many DSS. Explosive growth in portable devices, including smartphones and tablets, has changed the DSS user interfaces as well. These devices allow either handwritten input or Chapter 1 • Overview of Business Intelligence, Analytics, Data Science, and Artificial Intelligence typed input from internal or external keyboards. Some DSS user interfaces utilize natural language input (i.e., text in a human language) so that the users can easily express themselves in a meaningful way. Cell phone inputs through short message service (SMS) or chatbots are becoming more common for at least some consumer DSS-type applications. For example, one can send an SMS request for search on any topic to GOOGL (46645). Such capabilities are most useful in locating nearby businesses, addresses, or phone numbers, but it can also be used for many other decision support tasks. For example, users can find definitions of words by entering the word “define” followed by a word, such as “define extenuate.” Some of the other capabilities include • Price lookups: “Price 64GB iPhone X.” • Currency conversions: “10 US dollars in euros.” • Sports scores and game times: Just enter the name of a team (“NYC Giants”), and Google SMS will send the most recent game’s score and the date and time of the next match. This type of SMS-based search capability is also available for other search engines such as Microsoft’s search engine Bing. With the emergence of smartphones such as Apple’s iPhone and Android smartphones from many vendors, many companies are developing apps to provide purchasing-decision support. For example, Amazon’s app allows a user to take a picture of any item in a store (or wherever) and send it to Amazon.com. Amazon.com’s graphics-understanding algorithm tries to match the image to a real product in its databases and sends the user a page similar to Amazon.com’s product info pages, allowing users to perform price comparisons in real time. Millions of other apps have been developed that provide consumers support for decision making on finding and selecting stores/restaurants/service providers on the basis of location, recommendations from others, and especially from your own social circles. Search activities noted in the previous paragraph are also largely accomplished now through apps provided by each search provider. Voice input for these devices and the new smart speakers such as Amazon Echo (Alexa) and Google Home is common and fairly accurate (but not perfect). When voice input with accompanying speech-recognition software (and readily available text-tospeech software) is used, verbal instructions with accompanied actions and outputs can be invoked. These are readily available for DSS and are incorporated into the portable devices described earlier. An example of voice inputs that can be used for a generalpurpose DSS is Apple’s Siri application and Google’s Google Now service. For example, a user can give her or his zip code and say “pizza delivery.” These devices provide the search results and can even place a call to a business. The Knowledge-Based Management Subsystem Many of the user interface developments are closely tied to the major new advances in their knowledge-based systems. The knowledge-based management subsystem can support any of the other subsystems or act as an independent component. It provides intelligence to augment the decision maker’s own or to help understand a user’s query so as to provide a consistent answer. It can be interconnected with the organization’s knowledge repository (part of a KMS), which is sometimes called the organizational knowledge base, or connect to thousands of external knowledge sources. Many artificial intelligence methods have been implemented in the current generation of learning systems and are easy to integrate into the other DSS components. One of the most widely publicized knowledge-based DSS is IBM’s Watson, which was introduced in the opening vignette and will be described in more detail later. This section has covered the history and progression of Decision Support Systems in brief. In the next section we discuss evolution of this support to business intelligence, analytics, and data science. 21 22 Part I • Introduction to Analytics and AI u SECTION 1.3 REVIEW QUESTIONS 1. 2. 3. 4. 5. 6. 7. 8. 9. 1.4 List and briefly describe Simon’s four phases of decision making. What is the difference between a problem and its symptoms? Why is it important to classify a problem? Define implementation. What are structured, unstructured, and semistructured decisions? Provide two examples of each. Define operational control, managerial control, and strategic planning. Provide two examples of each. What are the nine cells of the decision framework? Explain what each is for. How can computers provide support for making structured decisions? How can computers provide support for making semistructured and unstructured decisions? EVOLUTION OF COMPUTERIZED DECISION SUPPORT TO BUSINESS INTELLIGENCE/ANALYTICS/DATA SCIENCE The timeline in Figure 1.5 shows the terminology used to describe analytics since the 1970s. During the 1970s, the primary focus of information systems support for decision making focused on providing structured, periodic reports that a manager could use for decision making (or ignore them). Businesses began to create routine reports to inform decision makers (managers) about what had happened in the previous period (e.g., day, week, month, quarter). Although it was useful to know what had happened in the past, managers needed more than this: They needed a variety of reports at different levels of granularity to better understand and address changing needs and challenges of the business. These were usually called management information systems (MIS). In the early 1970s, Scott-Morton first articulated the major concepts of DSS. He defined DSS as “interactive computer-based systems, which help decision makers utilize data and models to solve unstructured problems” (Gorry and Scott-Morton, 1971). The following is another classic DSS definition provided by Keen and Scott-Morton (1978): Decision support systems couple the intellectual resources of individuals with the capabilities of the computer to improve the quality of decisions. It is a computer-based support system for management decision makers who deal with semistructured problems. R ob ot l En ec / i D a n B D M lN ut -D te ee cs, us C D ec e ive lo p em rp S e a m i n is ud So D t sh e o io A Lea ma an rise , B ry wo bo Inf ft ata ss ut r D r d / o r w R AI/ n S R R om ni t R ig In- k/M r at ar ou E ar /Te Int u el St es ng ds ma D o a e D x e a tin xp pp at at o e l a at te , l boW , S tio as t/W lige ta dia io ic ur e ert ort o a d a A n c b n T n R re co e a s a al Re e A A ep Sy Sy na se Ana na /Se sis ho re Sys Se b M ce, D po Pla or st st n c / t l l l u B t B M r y y y r a i s n e e e si vic nin P tin m m tic tic o an M tin n PP tic ng rds ms S e g M s s s rs ts g g ing s s Ex O 1970s Decision Support Systems 1980s Enterprise/Executive IS In 1990s So ci 2000s Business Intelligence A 2010s Analytics Big Data FIGURE 1.5 Evolution of Decision Support, Business Intelligence, Analytics, and AI. 2020s Automation Chapter 1 • Overview of Business Intelligence, Analytics, Data Science, and Artificial Intelligence Note that the term decision support system, like management information system and several other terms in the field of IT, is a content-free expression (i.e., it means different things to different people). Therefore, there is no universally accepted definition of DSS. During the early days of analytics, data were often obtained from the domain experts using manual processes (i.e., interviews and surveys) to build mathematical or knowledge-based models to solve constrained optimization problems. The idea was to do the best with limited resources. Such decision support models were typically called operations research (OR). The problems that were too complex to solve optimally (using linear or nonlinear mathematical programming techniques) were tackled using heuristic methods such as simulation models. (We will introduce these as prescriptive analytics later in this chapter). In the late 1970s and early 1980s, in addition to the mature OR models that were being used in many industries and government systems, a new and exciting line of models had emerged: rule-based expert systems (ESs). These systems promised to capture experts’ knowledge in a format that computers could process (via a collection of if-then-else rules or heuristics) so that these could be used for consultation much the same way that one would use domain experts to identify a structured problem and to prescribe the most probable solution. ESs allowed scarce expertise to be made available where and when needed, using an “intelligent” DSS. The 1980s saw a significant change in the way organizations captured businessrelated data. The old practice had been to have multiple disjointed information systems tailored to capture transactional data of different organizational units or functions (e.g., accounting, marketing and sales, finance, manufacturing). In the 1980s, these systems were integrated as enterprise-level information systems that we now commonly call enterprise resource planning (ERP) systems. The old mostly sequential and nonstandardized data representation schemas were replaced by relational database management (RDBM) systems. These systems made it possible to improve the capture and storage of data as well as the relationships between organizational data fields while significantly reducing the replication of information. The need for RDBM and ERP systems emerged when data integrity and consistency became an issue, significantly hindering the effectiveness of business practices. With ERP, all the data from every corner of the enterprise is collected and integrated into a consistent schema so that every part of the organization has access to the single version of the truth when and where needed. In addition to the emergence of ERP systems, or perhaps because of these systems, business reporting became an ondemand, as-needed business practice. Decision makers could decide when they needed to or wanted to create specialized reports to investigate organizational problems and opportunities. In the 1990s, the need for more versatile reporting led to the development of executive information systems (EISs; DSS designed and developed specifically for executives and their decision-making needs). These systems were designed as graphical dashboards and scorecards so that they could serve as visually appealing displays while focusing on the most important factors for decision makers to keep track of the key performance indicators. To make this highly versatile reporting possible while keeping the transactional integrity of the business information systems intact, it was necessary to create a middle data tier known as a DW as a repository to specifically support business reporting and decision making. In a very short time, most large- to medium-sized businesses adopted data warehousing as their platform for enterprise-wide decision making. The dashboards and scorecards got their data from a DW, and by doing so, they were not hindering the efficiency of the business transaction systems mostly referred to as ERP systems. In the 2000s, the DW-driven DSS began to be called BI systems. As the amount of longitudinal data accumulated in the DWs increased, so did the capabilities of hardware 23 24 Part I • Introduction to Analytics and AI and software to keep up with the rapidly changing and evolving needs of the decision makers. Because of the globalized competitive marketplace, decision makers needed current information in a very digestible format to address business problems and to take advantage of market opportunities in a timely manner. Because the data in a DW are updated periodically, they do not reflect the latest information. To elevate this information latency problem, DW vendors developed a system to update the data more frequently, which led to the terms real-time data warehousing and, more realistically, right-time data warehousing, which differs from the former by adopting a data-refreshing policy based on the needed freshness of the data items (i.e., not all data items need to be refreshed in real time). DWs are very large and feature rich, and it became necessary to “mine” the corporate data to “discover” new and useful knowledge nuggets to improve business processes and practices, hence, the terms data mining and text mining. With the increasing volumes and varieties of data, the needs for more storage and more processing power emerged. Although large corporations had the means to tackle this problem, small- to medium-sized companies needed more financially manageable business models. This need led to service-oriented architecture and software and infrastructure-as-a-service analytics business models. Smaller companies, therefore, gained access to analytics capabilities on an as-needed basis and paid only for what they used, as opposed to investing in financially prohibitive hardware and software resources. In the 2010s, we are seeing yet another paradigm shift in the way that data are captured and used. Largely because of the widespread use of the Internet, new data generation mediums have emerged. Of all the new data sources (e.g., radio-frequency identification [RFID] tags, digital energy meters, clickstream Web logs, smart home devices, wearable health monitoring equipment), perhaps the most interesting and challenging is social networking/social media. These unstructured data are rich in information content, but analysis of such data sources poses significant challenges to computational systems from both software and hardware perspectives. Recently, the term Big Data has been coined to highlight the challenges that these new data streams have brought on us. Many advancements in both hardware (e.g., massively parallel processing with very large computational memory and highly parallel multiprocessor computing systems) and software/ algorithms (e.g., Hadoop with MapReduce and NoSQL, Spark) have been developed to address the challenges of Big Data. The last few years and the upcoming decade are bringing massive growth in many exciting dimensions. For example, streaming analytics and the sensor technologies have enabled the IoT. Artificial Intelligence is changing the shape of BI by enabling new ways of analyzing images through deep learning, not just traditional visualization of data. Deep learning and AI are also helping grow voice recognition and speech synthesis, leading to new interfaces in interacting with technologies. Almost half of U.S. households already have a smart speaker such as Amazon Echo or Google Home and have begun to interact with data and systems using voice interfaces. Growth in video interfaces will eventually enable gesture-based interaction with systems. All of these are being enabled due to massive cloudbased data storage and amazingly fast processing capabilities. And more is yet to come. It is hard to predict what the next decade will bring and what the new analytics-related terms will be. The time between new paradigm shifts in information systems and particularly in analytics has been shrinking, and this trend will continue for the foreseeable future. Even though analytics is not new, the explosion in its popularity is very new. Thanks to the recent explosion in Big Data, ways to collect and store these data and intuitive software tools, datadriven insights are more accessible to business professionals than ever before. Therefore, in the midst of global competition, there is a huge opportunity to make better managerial decisions by using data and analytics to increase revenue while decreasing costs by building better products, improving customer experience, and catching fraud before it happens, improving customer engagement through targeting and customization, and developing entirely Chapter 1 • Overview of Business Intelligence, Analytics, Data Science, and Artificial Intelligence new lines of business, all with the power of analytics and data. More and more companies are now preparing their employees with the know-how of business analytics to drive effectiveness and efficiency in their day-to-day decision-making processes. The next section focuses on a framework for BI. Although most people would agree that BI has evolved into analytics and data science, many vendors and researchers still use that term. So the next few paragraphs pay homage to that history by specifically focusing on what has been called BI. Following the next section, we introduce analytics and use that as the label for classifying all related concepts. A Framework for Business Intelligence The decision support concepts presented in Sections 1.2 and 1.3 have been implemented incrementally, under different names, by many vendors that have created tools and methodologies for decision support. As noted in Section 1.2, as the enterprise-wide systems grew, managers were able to access user-friendly reports that enabled them to make decisions quickly. These systems, which were generally called EISs, then began to offer additional visualization, alerts, and performance measurement capabilities. By 2006, the major commercial products and services appeared under the term business intelligence (BI). DEFINITIONS OF BI Business intelligence (BI) is an umbrella term that combines architectures, tools, databases, analytical tools, applications, and methodologies. It is, like DSS, a content-free expression, so it means different things to different people. Part of the confusion about BI lies in the flurry of acronyms and buzzwords that are associated with it (e.g., business performance management [BPM]). BI’s major objective is to enable interactive access (sometimes in real time) to data, to enable manipulation of data, and to give business managers and analysts the ability to conduct appropriate analyses. By analyzing historical and current data, situations, and performances, decision makers get valuable insights that enable them to make more informed and better decisions. The process of BI is based on the transformation of data to information, then to decisions, and finally to actions. A BRIEF HISTORY OF BI The term BI was coined by the Gartner Group in the mid-1990s. However, as the history in the previous section points out, the concept is much older; it has its roots in the MIS reporting systems of the 1970s. During that period, reporting systems were static, were two dimensional, and had no analytical capabilities. In the early 1980s, the concept of EISs emerged. This concept expanded the computerized support to top-level managers and executives. Some of the capabilities introduced were dynamic multidimensional (ad hoc or on-demand) reporting, forecasting and prediction, trend analysis, drill-down to details, status access, and critical success factors. These features appeared in dozens of commercial products until the mid-1990s. Then the same capabilities and some new ones appeared under the name BI. Today, a good BI-based enterprise information system contains all the information that executives need. So, the original concept of EIS was transformed into BI. By 2005, BI systems started to include artificial intelligence capabilities as well as powerful analytical capabilities. Figure 1.6 illustrates the various tools and techniques that may be included in a BI system. It illustrates the evolution of BI as well. The tools shown in Figure 1.6 provide the capabilities of BI. The most sophisticated BI products include most of these capabilities; others specialize in only some of them. The Architecture of BI A BI system has four major components: a DW, with its source data; business analytics, a collection of tools for manipulating, mining, and analyzing the data in the DW; BPM for monitoring and analyzing performance; and a user interface (e.g., a dashboard). The relationship among these components is illustrated in Figure 1.7. 25 26 Part I • Introduction to Analytics and AI Querying and Reporting ETL Metadata Data Warehouse DSS EIS/ESS Data Marts Financial Reporting Spreadsheets (MS Excel) OLAP Digital Cockpits and Dashboards Business Intelligence Scorecards and Dashboards Workflow Alerts and Notifications Data and Text Mining Broadcasting Tools Predictive Analytics Portals FIGURE 1.6 Evolution of Business Intelligence (BI). The Origins and Drivers of BI Where did modern approaches to DW and BI come from? What are their roots, and how do those roots affect the way organizations are managing these initiatives today? Today’s investments in information technology are under increased scrutiny in terms of their bottom-line impact and potential. The same is true of DW and the BI applications that make these initiatives possible. Data Warehouse Environment Data Sources Business Analytics Environment Technical staff Business users Build the data warehouse - Organizing - Summarizing - Standardizing Access Data warehouse Performance and Strategy Managers/executives BPM strategies Manipulation, results User interface Future component: Intelligent systems - Browser - Portal - Dashboard FIGURE 1.7 A High-Level Architecture of BI. Source: Based on W. Eckerson. (2003). Smart Companies in the 21st Century: The Secrets of Creating Successful Business Intelligent Solutions. Seattle, WA: The Data Warehousing Institute, p. 32, Illustration 5. Chapter 1 • Overview of Business Intelligence, Analytics, Data Science, and Artificial Intelligence Organizations are being compelled to capture, understand, and harness their data to support decision making to improve business operations. Legislation and regulation (e.g., the Sarbanes-Oxley Act of 2002) now require business leaders to document their business processes and to sign off on the legitimacy of the information they rely on and report to stakeholders. Moreover, business cycle times are now extremely compressed; faster, more informed, and better decision making is, therefore, a competitive imperative. Managers need the right information at the right time and in the right place. This is the mantra for modern approaches to BI. Organizations have to work smart. Paying careful attention to the management of BI initiatives is a necessary aspect of doing business. It is no surprise, then, that organizations are increasingly championing BI and under its new incarnation as analytics. Data Warehouse as a Foundation for Business Intelligence BI systems rely on a DW as the information source for creating insight and supporting managerial decisions. A multitude of organizational and external data is captured, transformed, and stored in a DW to support timely and accurate decisions through enriched business insight. In simple terms, a DW is a pool of data produced to support decision making; it is also a repository of current and historical data of potential interest to managers throughout the organization. Data are usually structured to be available in a form ready for analytical processing activities (i.e., OLAP, data mining, querying, reporting, and other decision support applications). A DW is a subject-oriented, integrated, time-variant, nonvolatile collection of data in support of management’s decision-making process. Whereas a DW is a repository of data, data warehousing is literally the entire process. Data warehousing is a discipline that results in applications that provide decision support capability, allows ready access to business information, and creates business insight. The three main types of data warehouses are data marts (DMs), operational data stores (ODS), and enterprise data warehouses (EDW). Whereas a DW combines databases across an entire enterprise, a DM is usually smaller and focuses on a particular subject or department. A DM is a subset of a data warehouse, typically consisting of a single subject area (e.g., marketing, operations). An operational data store (ODS) provides a fairly recent form of customer information file. This type of database is often used as an interim staging area for a DW. Unlike the static contents of a DW, the contents of an ODS are updated throughout the course of business operations. An EDW is a large-scale data warehouse that is used across the enterprise for decision support. The large-scale nature of an EDW provides integration of data from many sources into a standard format for effective BI and decision support applications. EDWs are used to provide data for many types of DSS, including CRM, supply chain management (SCM), BPM, business activity monitoring, product lifecycle management, revenue management, and sometimes even KMS. In Figure 1.8, we show the DW concept. Data from many different sources can be extracted, transformed, and loaded into a DW for further access and analytics for decision support. Further details of DW are available in an online chapter on the book’s Web site. Transaction Processing versus Analytic Processing To illustrate the major characteristics of BI, first we will show what BI is not—namely, transaction processing. We are all familiar with the information systems that support our transactions, like ATM withdrawals, bank deposits, and cash register scans at the grocery store. These transaction processing systems are constantly involved in handling updates to what we might call operational databases. For example, in an ATM withdrawal transaction, we need to reduce our bank balance accordingly; a bank deposit adds to an account; and a grocery store purchase is likely reflected in the store’s calculation of total sales for the day, and it should reflect an appropriate reduction in the store’s inventory for the items we bought, and so on. These online transaction processing (OLTP) systems handle a 27 Part I • Introduction to Analytics and AI No data mart options Data Sources ERP Legacy Data Marts ETL Process Select Metadata Other OLTP/Web Transform Data mart (Operations) Enterprise Data Warehouse Integrate Data mart (Finance) Data mart (...) Load External Data Routine Business Reporting Data mart (Marketing) Extract POS Applications (Visualization) API/Middleware 28 Data/Text Mining OLAP, Dashboard, Web Custom-Built Applications Replication FIGURE 1.8 Data Warehouse Framework and Views. company’s routine ongoing business. In contrast, a DW is typically a distinct system that provides storage for data that will be used for analysis. The intent of that analysis is to give management the ability to scour data for information about the business, and it can be used to provide tactical or operational decision support whereby, for example, line personnel can make quicker and/or more informed decisions. DWs are intended to work with informational data used for online analytical processing (OLAP) systems. Most operational data in ERP systems—and in their complementary siblings like SCM or CRM—are stored in an OLTP system, which is a type of computer processing where the computer responds immediately to user requests. Each request is considered to be a transaction, which is a computerized record of a discrete event, such as the receipt of inventory or a customer order. In other words, a transaction requires a set of two or more database updates that must be completed in an all-or-nothing fashion. The very design that makes an OLTP system efficient for transaction processing makes it inefficient for end-user ad hoc reports, queries, and analysis. In the 1980s, many business users referred to their mainframes as “black holes” because all the information went into them, but none ever came back. All requests for reports had to be programmed by the IT staff, whereas only “precanned” reports could be generated on a scheduled basis, and ad hoc real-time querying was virtually impossible. Although the client/server-based ERP systems of the 1990s were somewhat more report friendly, they have still been a far cry from a desired usability by regular, nontechnical end users for things such as operational reporting and interactive analysis. To resolve these issues, the notions of DW and BI were created. DWs contain a wide variety of data that present a coherent picture of business conditions at a single point in time. The idea was to create a database infrastructure that was always online and contained all the information from the OLTP systems, including historical data, but reorganized and structured in such a way that it was fast and efficient for querying, analysis, and decision support. Separating the OLTP from analysis and decision support enables the benefits of BI that were described earlier. A Multimedia Exercise in Business Intelligence TUN includes videos (similar to the television show CSI) to illustrate concepts of analytics in different industries. These are called “BSI Videos (Business Scenario Investigations).” Not only are these entertaining, but they also provide the class with some questions for discussion. For starters, please go to https://www.teradatauniversitynetwork.com/ Library/Items/BSI-The-Case-of-the-Misconnecting-Passengers/ or www.youtube. Chapter 1 • Overview of Business Intelligence, Analytics, Data Science, and Artificial Intelligence com/watch?v=NXEL5F4_aKA. Watch the video that appears on YouTube. Essentially, you have to assume the role of a customer service center professional. An incoming flight is running late, and several passengers are likely to miss their connecting flights. There are seats on one outgoing flight that can accommodate two of the four passengers. Which two passengers should be given priority? You are given information about customers’ profiles and relationships with the airline. Your decisions might change as you learn more about those customers’ profiles. Watch the video, pause it as appropriate, and answer the questions on which passengers should be given priority. Then resume the video to get more information. After the video is complete, you can see the slides related to this video and how the analysis was prepared on a slide set at www.slideshare.net/teradata/bsi-how-we-did-itthe-case-of-the-misconnecting-passengers. This multimedia excursion provides an example of how additional available information through an enterprise DW can assist in decision making. Although some people equate DSS with BI, these systems are not, at present, the same. It is interesting to note that some people believe that DSS is a part of BI—one of its analytical tools. Others think that BI is a special case of DSS that deals mostly with reporting, communication, and collaboration (a form of data-oriented DSS). Another explanation (Watson, 2005) is that BI is a result of a continuous revolution, and as such, DSS is one of BI’s original elements. Further, as noted in the next section onward, in many circles, BI has been subsumed by the new terms analytics or data science. First and foremost, the fundamental reasons for investing in BI must be aligned with the company’s business strategy. BI cannot simply be a technical exercise for the information systems department. It has to serve as a way to change the manner in which the company conducts business by improving its business processes and transforming decision-making processes to be more data driven. Many BI consultants and practitioners involved in successful BI initiatives advise that a framework for planning is a necessary precondition. One framework, proposed by Gartner, Inc. (2004), decomposed planning and execution into business, organization, functionality, and infrastructure components. At the business and organizational levels, strategic and operational objectives must be defined while considering the available organizational skills to achieve those objectives. Issues of organizational culture surrounding BI initiatives and building enthusiasm for those initiatives and procedures for the intra-organizational sharing of BI best practices must be considered by upper management—with plans in place to prepare the organization for change. One of the first steps in that process is to assess the IS organization, the skill sets of the potential classes of users, and whether the culture is amenable to change. From this assessment, and assuming there are justification and the need to move ahead, a company can prepare a detailed action plan. Another critical issue for BI implementation success is the integration of several BI projects (most enterprises use several BI projects) among themselves and with the other IT systems in the organization and its business partners. Gartner and many other analytics consulting organizations promoted the concept of a BI competence center that would serve the following functions: APPROPRIATE PLANNING AND ALIGNMENT WITH THE BUSINESS STRATEGY • A center can demonstrate how BI is clearly linked to strategy and execution of strategy. • A center can serve to encourage interaction between the potential business user communities and the IS organization. • A center can serve as a repository and disseminator of best BI practices between and among the different lines of business. • Standards of excellence in BI practices can be advocated and encouraged throughout the company. • The IS organization can learn a great deal through interaction with the user communities, such as knowledge about the variety of types of analytical tools that are needed. 29 30 Part I • Introduction to Analytics and AI • The business user community and IS organization can better understand why the DW platform must be flexible enough to provide for changing business requirements. • The center can help important stakeholders like high-level executives see how BI can play an important role. Over the last 10 years, the idea of a BI competence center has been abandoned because many advanced technologies covered in this book have reduced the need for a central group to organize many of these functions. Basic BI has now evolved to a point where much of it can be done in “self-service” mode by the end users. For example, many data visualizations are easily accomplished by end users using the latest visualization packages (Chapter 3 will introduce some of these). As noted by Duncan (2016), the BI team would now be more focused on producing curated data sets to enable self-service BI. Because analytics is now permeating across the whole organization, the BI competency center could evolve into an analytics community of excellence to promote best practices and ensure overall alignment of analytics initiatives with organizational strategy. BI tools sometimes needed to be integrated among themselves, creating synergy. The need for integration pushed software vendors to continuously add capabilities to their products. Customers who buy an all-in-one software package deal with only one vendor and do not have to deal with system connectivity. But they may lose the advantage of creating systems composed from the “best-of-breed” components. This led to major chaos in the BI market space. Many of the software tools that rode the BI wave (e.g., Savvion, Vitria, Tibco, MicroStrategy, Hyperion) have either been acquired by other companies or have expanded their offerings to take advantage of six key trends that have emerged since the initial wave of surge in business intelligence: • Big Data. • Focus on customer experience as opposed to just operational efficiency. • Mobile and even newer user interfaces—visual, voice, mobile. • Predictive and prescriptive analytics, machine learning, artificial intelligence. • Migration to cloud. • Much greater focus on security and privacy protection. This book covers many of these topics in significant detail by giving examples of how the technologies are evolving and being applied, and the managerial implications. u SECTION 1.4 REVIEW QUESTIONS 1. List three of the terms that have been predecessors of analytics. 2. What was the primary difference between the systems called MIS, DSS, and Executive Information Systems? 3. Did DSS evolve into BI or vice versa? 4. Define BI. 5. List and describe the major components of BI. 6. Define OLTP. 7. Define OLAP. 8. List some of the implementation topics addressed by Gartner’s report. 9. List some other success factors of BI. 1.5 ANALYTICS OVERVIEW The word analytics has largely replaced the previous individual components of computerized decision support technologies that have been available under various labels in the past. Indeed, many practitioners and academics now use the word analytics in place of BI. Although many authors and consultants have defined it slightly differently, one can Chapter 1 • Overview of Business Intelligence, Analytics, Data Science, and Artificial Intelligence view analytics as the process of developing actionable decisions or recommendations for actions based on insights generated from historical data. According to the Institute for Operations Research and Management Science (INFORMS), analytics represents the combination of computer technology, management science techniques, and statistics to solve real problems. Of course, many other organizations have proposed their own interpretations and motivations for analytics. For example, SAS Institute Inc. proposed eight levels of analytics that begin with standardized reports from a computer system. These reports essentially provide a sense of what is happening with an organization. Additional technologies have enabled us to create more customized reports that can be generated on an ad hoc basis. The next extension of reporting takes us to OLAP-type queries that allow a user to dig deeper and determine specific sources of concern or opportunities. Technologies available today can also automatically issue alerts for a decision maker when performance warrants such alerts. At a consumer level, we see such alerts for weather or other issues. But similar alerts can also be generated in specific settings ...
Purchase answer to see full attachment