BI Project Management Tasks Project Project manager Project artifacts ID BI Activity In each row, list the specific task to be completed in order to complete a BI Solution from Start to Finish? 1 2 3 4 5 6 7 8 9 10 11 12 13 BI Activity Purpose Describe the purpose of the BI task Owner/Responsible Stakeholder Who will own and complete the BI activity? Status Date Entered InDate the Progress, activity Closed, or was Pending Entered in Project Plan BI Project Management Tasks 14 15 Project # Sponsor Updated Planned Completion Actual Completion Date the activity is planned to be completed Date the activity was completed services A proven path to success Use Teradata Professional Services to guide your data warehouse implementation. by David R. Schiller F orward-thinking companies realize the importance of an enterprise data warehouse (EDW). They know it is required for obtaining deep operational and strategic insights into their businesses. But organizations differ on how difficult they think it is to implement a data warehouse. Some are reluctant to begin the project because of its scope. Others are convinced they can handle the job internally, unaware that developing a data warehouse is different from working on any other type of technology project in its complexity and level of involvement. Most companies realize they require the services and industry-leading expertise of an accomplished source to guide them through each step of the process. Overcome roadblocks Left alone, many data warehouse implementations run the risk of failure because of obstacles like competing projects, bad hardware architecture and improper design, or a scope too large to achieve timely results. Data quality issues and other broken operational processes can also impede progress. Furthermore, many organizations lack corporate vision and support. Executives who don’t fully appreciate that their data is an important corporate asset can delay funding and hinder governance policies. To overcome these adversities and ensure the successful completion of a data warehouse project, best practices suggest taking advantage of the assistance offered by a knowledgeable and experienced group of service providers. The Teradata Professional Services team, with its group of consultants committed to the success of their customers, can work with your company and help manage your data warehouse project. The team has implemented Photography by iStockphoto PAGE 1 | Teradata Magazine | September 2009 | ©2009 Teradata Corporation | AR-5964 data warehouse projects worldwide in multiple industries. From the knowledge and experience gained through these engagements, the team has developed a proven methodology that is adaptable to any organization. Plan development Before any project is started, its scope must be determined, a plan developed, and the necessary tools and work force established. Your Teradata Professional Services partner will work with you and your team to create a project worksheet detailing each step and task of the implementation process. The worksheet takes into account the expertise your organization has on its enterprise data, business rules and success goals; outside assistance from other third-party vendors; and the overall product and implementation knowledge the Teradata Professional Services team can provide. In the end, you’re given complete flexibility on whether the task ownership and resource staffing are provided by your company, Teradata or other third-party organizations. (See figure, page 2.) When evaluating the scope of the project, you and your team leaders will consult with your Teradata Professional Services partner and the Teradata solution architect about whether the project needs to be broken down into more manageable segments. The desired outcome for each 90- to 120-day iteration will services help determine the tasks, resource requirements, recommended skill sets and assignment duration. Accommodations are provided in the plan for project ramp-up and ramp-down; closeout, internal and external costs; and scheduling considerations such as vacations and holidays. A section enables you to track assumptions until they are agreed upon or otherwise resolved. From this worksheet, statements of work and project plans are created. Not all personnel will start at once, as some tasks aren’t undertaken until later. For example, logical data modeling occurs before physical data modeling, which is needed to complete the design of the extract, transform and load (ETL) processes, and so on. Some activity overlap will occur, but the worksheet can be used to generate the appropriate staffing schedule. Keep in mind that before any external team members are brought on board, you will need adequate lead time to obtain for them the necessary approvals for security accounts, access to the Internet and appropriate facilities, and to establish physical workspaces. Figure Project on-boarding The Teradata Professional Services partner and the project manager will collaborate on preparation details in a phase called project on-boarding. These activities include creating a document of the complete team roster, a project overview and plan, and the current architecture and tools used. The project manager sends the document to the team in advance. Preparing and updating the team with background information and task lists before the project starts is invaluable. It keeps the project on schedule and the team members informed without spending client time at the onset. Team meetings are then held so each function can be explained in detail and information on customer requirements, expectations and protocol can be distributed and discussed. Through this ramp-up process, team members gain an understanding of your company and the project goals and objectives. Throughout the project’s life cycle, circumstances will arise that could expand the scope of the project. When this happens, the Teradata Professional Services partner or program manager will identify the new requirements or additional data sources. This information is used to understand the impact to the current schedule or cost. After these are addressed with your management team, a formal change control process is created to modify the statement of work, the project plan and the resources that are engaged, if necessary. Step by step Regarding each member of the task force as an essential component of the team is conducive to a successful project, whether the person is from your organization, Teradata or another third-party vendor. The table on page 3 shows a breakdown of the typical team members from each of these organizations. An example of a 15-week project, which provides for two weeks of planning, is shown below. The chart shows how each team member is involved at specific times depending on the task and area of expertise. For instance, the Teradata representative sets up and configures the database the week before the project officially begins. This continues on through the first “actual” week. The ETL is designed in week three by a Teradata team member, but your in-house and third-party ETL developers don’t get involved until four weeks later. Enterprise data warehouse phase 1 project detail Weeks Project 1, phase 1 Install hardware, software Database configuration, setup, training customer resources Data discovery/data profiling Project/program management Logical data modeling Physical data modeling ETL design ETL development BI reporting design BI reporting development System/user acceptance testing Go-live Teradata resources Customer resources Third-party resources -1 0 3/24 3/31 3/24 3/31 3/24 3/31 1 4/7 4/7 4/7 2 3 4 4/14 4/21 4/28 4/14 4/21 4/28 4/14 4/21 4/28 5 5/5 5/5 5/5 6 7 8 5/12 5/19 5/26 5/12 5/19 5/26 5/12 5/19 5/26 9 10 6/2 6/2 6/2 6/9 6/9 6/9 11 12 13 6/16 6/23 6/30 6/16 6/23 6/30 6/16 6/23 6/30 A 15-week phase 1 enterprise data warehouse design and development project typically follows these steps, divided among Teradata, customer and thirdparty resources. Weeks -1 and 0 are pre-engagement activities. PAGE 2 | Teradata Magazine | September 2009 | ©2009 Teradata Corporation | AR-5964 With the guidance provided by a flowchart or workstream from the Teradata Solutions Methodology, all parties work together through the different phases of each step in the project. Identify, analyze, design, develop, deploy and improve are the phases, which drill down on each workstream for specific activities, collateral, templates and tools that are necessary to achieve each step’s objective. For example, for the data integration phase of an EDW project, data gathering and cleansing requirements are developed before the data loading process can be created and performed. All data standards must be established in order to build a physical data model. Tasks required to build a high-level data integration architecture are included. And, of course, testing is essential to considering this phase of the data warehouse project ready for production. Teradata Solutions Methodology: Value at every step M any IT professionals plan a data warehouse based on their experiences with other IT projects. But building a data warehouse is much more extensive and intricate, and it needs its own development strategy. A good methodology helps ensure a project’s quality, consistency and timeliness. The patent-pending Teradata Solutions Methodology is a collection of integrated processes, customized tools and quantifiable metrics—from initial strategic planning to technical implementation—that brings value to every piece of the design and implementation. User training and ongoing support are provided to ensure a smooth transition and expert backing. Developed out of more than 23 years of experience implementing data warehouses, the Teradata Solutions Methodology is continually refined with best practices learned from the world’s most successful data warehouse projects. —D.R.S. table Project team resources in house teradata other third party > Project manager > DBA > Developer > Data architect and technical leader > Teradata Professional Services engagement partner > Solution architect > Project manager > Logical and physical data modeler > Data architect and technical leader > BI report designer > ETL designer > Data mappers > ETL developers > BI developers Knowledge transfer After a data warehouse is installed, there are always new sources of data to integrate, additional business questions to ask and changes to regulatory requirements that organizations need to quickly address. Your data warehouse should enable your company to successfully use that data, answer those questions and breathe easier about meeting regulations. To fully utilize the tools and applications available, you must be knowledgeable about your data warehouse environment and confident in your ability to manage the system. For that reason, a driving point of the Teradata Professional Services philosophy is to make its customers more self-sufficient after the implementation is complete but before its team leaves the customer’s site. Part of any Teradata Professional Services engagement includes a variety of mechanisms to transfer knowledge, such as: > Client mentoring > Lunch-and-learn sessions > Informal and formal training curricula > Technical forums Additionally, for two weeks after the project has gone live, a senior-level Teradata consultant remains to ensure continuity of processes, performance and expectations. Teradata also offers a full suite of managed services that can be tailored to your company’s needs, providing another option for ongoing support and access to the entire community of Teradata experts. PAGE 3 | Teradata Magazine | September 2009 | ©2009 Teradata Corporation | AR-5964 Ready for business Choosing the right consultant is key to the success of your data warehousing project. A professional services team with the right skills, knowledge, methodology and best practices can save many hours of unnecessary delay and rework. Teradata consultants average 12 years of experience in data warehousing, and many are highly sought technical and thought leaders in their fields. It’s undeniable that greater business value and insight are gained when your data is transformed into an asset. Working with Teradata Professional Services, your organization can expect a high EDW implementation success rate, and your IT staff will become more self-sufficient, confident and knowledgeable about the data warehouse system and what it can provide your company. T David R. Schiller, CCP, is a certified Teradata Professional with more than 27 years of various IT experience as a practitioner, consultant and executive. He has been with Teradata since 1997 and currently manages Teradata Professional Services marketing programs. online For more information on Teradata Professional Services, visit Teradata.com. insider’s warehouse TECH2TECH From the ground up Business Intelligence Roadmap offers project life cycle methodology for decision-support applications. by Larissa Moss A consultant in the enterprise architecture industry once compared building systems to building airplanes. While a paper or model airplane can be constructed with little forethought, a jet airplane cannot. Similarly, a small stand-alone application can be built without careful planning, but a data warehouse with its many dependent business intelligence (BI) applications cannot. Yet many project teams try to develop their data warehouse and BI applications using no methodology at all. As a result, they quickly become overwhelmed with complexity, unanticipated problems, added tasks, too much rework and wasted time, and a project that seems out of control. 6 stages with 16 steps A methodology I co-developed, the Business Intelligence Roadmap, is designed for data warehouse and BI applications. The 16 development steps of the roadmap are grouped into the following engineering stages of justification, planning, analysis, design, construction and deployment. (See table, page 2.) may be for not implementing a data warehouse or BI solution at your organization. Interview people at all levels in the organization in order to find out their understanding and expectations of a data warehouse and BI project. This will dictate whether an enterprise solution should be architected or whether building departmental or standalone BI applications is more appropriate. Discussing the initiative with others will also help decide the executive sponsor for the project and whether collective sponsorship from all business executives will be achieved for one cohesive enterprise solution. Planning Strategic and tactical plans lay out how the initiative or project will be accomplished. 2 E  nterprise infrastructure evaluation Equally important to supporting an evolving data warehouse are: > Technical infrastructure Justification An assessment, which includes cost justification, is made of a business problem or opportunity. This gives rise to a data warehouse initiative or BI project. Photography by iStockphoto 1 Business case assessment First, define the business drivers and propose a data warehouse or BI solution. Each BI application should clearly define the benefits of either solving a business problem or taking advantage of an opportunity. You may also want to find out what types of BI initiatives your competitors have and what the risks PAGE 1 | Teradata Magazine | June 2009 | ©2009 Teradata Corporation | AR-5923 • Hardware, software, middleware • Database management systems, operating systems • Network components • Developer tools, end-user BI tools > Non-technical infrastructure • Metadata standards, data naming standards, enterprise data model • Data warehouse methodology, guidelines • Testing procedures, change control process, issues management procedures, dispute resolution procedures Identify the technical and non-technical infrastructure components you already have in place and which ones you still need. Evaluate the completeness and effectiveness of those components and identify activities for enhancing or upgrading them. 3 Project planning The difference between data warehouse/BI projects and traditional IT projects is the emphasis on data as opposed to processes. TECH2TECH table insider’s warehouse and data requirements for a solution. Steps for data warehouse/ BI development tracks Engineering stages and data warehouse development steps Justification 1. B  usiness case assessment Planning 2. E  nterprise infrastructure evaluation 3. Project planning Analysis 4. R  equirements definition 5. Data analysis 6. Prototyping 7. M  etadata repository analysis DESIGN 8. Database design 9. ETL design 10. Metadata repository design Construction 11. ETL development 12. Application development 13. Data mining 14. Metadata repository development Deployment 15. Implementation 16. Release evaluation Extract, transform and load (ETL) track Business intelligence (BI) track Metadata repository track √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ Before committing to a project deadline, know the condition of the source files and the effort it will take to sanitize the data. Be sure the project plan considers that datarelated activities can take four to five times the effort of process-related activities. These activities consist of data modeling, profiling, standardization and integration, and data cleansing. Most project deadlines are missed because the project team underestimated the amount of dirty data and its impact on the extract, transform and load (ETL) process. Analysis Detailed analysis of the problem or opportunity is performed to identify business √ √ √ √ √ 4 Requirements definition Stable requirements rarely exist on data warehouse and BI projects because users often demand too much data in an impractical time frame—they want everything, and they want it yesterday. Break the requirements into multiple packages that you can deploy in iterations. Start with requirements that have the highest business value, are the most stable (least likely to change), and are sourced from relatively clean and easily integrated data. Never try to freeze requirements, but allow users to change them when they see BI capabilities unfold. 5 Data analysis An in-depth understanding of the data requirements is produced in this important development step, including standardized data names, definitions, domains, business rules and data quality rules. The two major activities involved are top-down data modeling and bottom-up source data analysis. Topdown data modeling is the creation of a business data model with the participation of the business users. Bottom-up source data analysis refers to the data archaeology performed on the source data to confirm or refute the business rules identified during the business data modeling process. PAGE 2 | Teradata Magazine | June 2009 | ©2009 Teradata Corporation | AR-5923 Only trained data analysts or data administrators should perform this step, not developers. 6 Prototyping Performing system analysis, design, coding and testing allows users to assess and change their requirements with less impact early in the project schedule. The costs of experimenting with different database designs, visualization methods, development tools and programming languages are much lower during prototyping than during development. Some prototypes are nothing more than smoke and mirrors and are used to obtain user buy-in, validate functional requirements or prove a concept. Others with longer life spans produce partially functioning pieces of the application. 7 Metadata repository analysis The four types of metadata and where they are captured are: > Business—in a data modeling tool > Technical—in the relational database management system (RDBMS) and online analytical processing (OLAP) cubes > Process—in ETL, OLAP and other developer tools > Usage—in the RDBMS and manually by the DBA monitoring the data warehouse environment These different types of metadata must be mapped to one another and stored in a repository. Metadata repositories can be purchased or built. In either case, document in a meta model the type of metadata to capture, how to map it and where to store it. Design A product is conceived that will help solve the business problem or enable the business opportunity. 8 Database design A database can be either relational or multi-dimensional, depending on the underlying RDBMS, the purpose of the database, and its access paths and BI tools. A relational design offers maximum flexibility for an operational data store (ODS) or an enterprise data warehouse (EDW). A multi-dimensional design offers maximum performance and is more common for data marts that are tailored toward specific reporting patterns or BI applications. The project’s specific performance considerations should influence the database schema you choose as well as the physical components, such as indexing strategy, data set placement, partitioning and clustering. 9 ETL design ETL functions are designed for different processes: > Initial population is very similar to a system conversion. The main task is to map the selected data elements from the current source files to the target columns in the data warehouse databases. > Historical population is a process that loads historical data from old files. This set of specifications will be slightly different because historical data has usually been archived to off-line storage devices. Since the record layouts of the files and databases change over time, the programs for the historical population have to recognize those changes. > Incremental update refers to the ongoing updates (i.e., inserts) to the data warehouse databases. Two things must be considered for the incremental update: whether to extract all records from the source files or only the changed records (deltas), and whether to pull data from the source files or push out the data from the source systems. In many data warehouse environments, the optimum choice is to have push-out deltas only. 10 Metadata repository design If built in-house, several metadata repository structures must be designed: the database, the migration programs that load and link the Figure Workflow for data warehouse/ BI development tracks A 16-step methodology for designing data warehouse and business intelligence (BI) applications is grouped into the engineering stages of justification, planning, analysis, design, construction and deployment. metadata from the tools where it is captured, the interface programs that communicate with the tools and users, and the online help function. If purchased, the metadata repository must be installed and tested. Construction The conceived product is built, and the design elements are incorporated. 11 ETL development Implementing the ETL process is almost always done through an ETL tool. Some ETL tools are more sophisticated than others. Thus, in some cases, additional code must be written to supplement their functions. Because the ETL process is the most complicated process in the data warehouse environment, it must be thoroughly and PAGE 3 | Teradata Magazine | June 2009 | ©2009 Teradata Corporation | AR-5923 formally tested. This is done with a formal test plan, which contains test cases, test sequence, expected test results, actual test results and a test log. Recommended types of tests are unit, end-to-end integration and regression, performance or stress, quality assurance (pre-production), and user acceptance. 12 Application development The various prototyping results are used to construct a production-worthy frontend BI application. This can be a state-ofthe-practice OLAP cube with reports and canned or parameterized queries. It could also be a state-of-the-art dashboard or scorecard. In either case, the BI application is subjected to the same rigorous testing activities as the ETL process. 13 Data mining A specialized data mining application uses a tool to execute data mining operations against a data pool. This pool is based on an analytical data model, which is usually developed by a statistician. Data mining activities include identifying the business problem domain; collecting, consolidating, cleansing and preparing the data for the analytical data model; and populating the database. The results are often displayed visually in charts (bar, pie and lift), decision trees, scatter plots and histograms. 14 M  etadata repository development Creating a database, writing metadataspecific migration programs to extract the metadata, coding interface programs to deliver metadata to the users, and providing an online help function are some activities to developing a metadata repository. Another development step is writing ETL-specific metadata programs. These programs capture load statistics, reconciliation totals, data quality statistics and error statistics produced during the ETL runs. After each run, these statistics are loaded into the metadata repository. Deployment The finished product is implemented and its effectiveness measured to determine whether the solution meets, exceeds or fails end-user expectations. Additionally, a predefined time frame has been established to determine return on investment (ROI). 15 Implementation After the programs are tested, the data warehouse, BI application and metadata repository are ready for production. The databases and program libraries that will house the ETL and BI programs must be created. Once the data has been loaded into the databases, all security measures should be tested again. The users are then trained, and the support functions—such as a help desk, maintenance of the data warehouse databases, scheduling and running ETL jobs, performance monitoring and tuning, and monitoring usage— are initiated. 16 Release evaluation A data warehouse is not a system but an evolving and expanding environment. It is beneficial, therefore, to review lessons learned at the end of every project. Not only does this enable the team to anticipate and prevent problems in future projects but it also helps to streamline the development process. Any tools, techniques, guidelines and processes that were not useful should be reevaluated and adjusted or, possibly, discarded. Any missed deadlines, cost overruns, disputes and resolutions should be examined, and adjustments to the processes should be made before the next project begins. The release evaluation review is also the perfect forum for repackaging dropped data or functionality and reprioritizing the next projects. Data warehouse/ BI development tracks Although the development steps are presented sequentially and mapped into the engineering stages, these types of projects are rarely executed in this order. Some steps could be combined or bypassed. At a minimum, most projects will divide the development effort into multiple tracks. (See figure, page 3.) The two most common tracks are ETL and BI application. A third track may also be needed: metadata repository. If data mining is a full-time activity at your company, the data mining step can be separated into its own track. The table on page 2 shows which steps apply to each track. While each track can have its own team and set of activities, they are interdependent. All tracks start and end PAGE 4 | Teradata Magazine | June 2009 | ©2009 Teradata Corporation | AR-5923 their activities at the same time, but the activities during the analysis, design and construction stages may diverge in terms of duration and work organization. The back-end ETL track may spend more time in analysis before designing and developing the ETL process, and the frontend BI application track may prototype extensively. The metadata repository track may be organized as a separate but parallel development project, or it may simply involve buying and installing a metadata repository product. Since discoveries made on one track often affect the others, it is important to manage all resources as one project team. Supportive methodology Successful data warehouse and BI initiatives evolve into complicated enterprise-wide decision-support environments. A strong foundation is required to support various BI applications for different types of users. To build such a foundation, the question is not whether a methodology is needed, but what type is the best option. The data warehouse is built in increments to accommodate requirements that are inevitably evolving and morphing. Its methodology, therefore, must support periodic, spiral architectural redesigns and have a strong focus on data and powerful enterprise infrastructure activities. Data warehouse projects are full of challenges, but having to remember hundreds of tasks should not be one of them. That’s why a good methodology specifically designed for data warehouse and BI projects is necessary. Planning of this sort will help strengthen any company’s BI investment. T Larissa Moss, president of Method Focus Inc., has 29 years of IT experience with a focus on data warehousing and data management. She is a worldwide speaker and widely published author. Larissa can be reached at methodfocus@earthlink.net.
Purchase answer to see full attachment