1 | Your Project Name | |
2 | Step 1: Business Case Assessment | |
3 | Determine the business need | |
4 | Identify the business need | |
5 | Determine current financial consequences of the business need | |
6 | Assess the current decision-support system (DSS) solutions | 3 |
7 | Assess current usage of the existing DSS | |
8 | Determine the shortcomings of the existing DSS | |
9 | Perform gap analysis | |
10 | Assess the operational sources and procedures | 3 |
11 | Assess the data quality of operational systems | |
12 | Review file structures and databases | |
13 | Review content (domain) of source data elements | |
14 | Assess the current data movement | |
15 | Review data entry practices | |
16 | Review data extraction practices | |
17 | Review data manipulation practices | |
18 | Review data duplication practices | |
19 | Assess current operational procedures | |
20 | Identify poor data entry practices | |
21 | Identify lack of edit checks | |
22 | Identify defective program code | |
23 | Identify lack of training | |
24 | Assess the competitors ' BI decision-support initiatives | 3 |
25 | Determine the competitors' successes and failures | |
26 | Determine whether competitors gained market advantages | |
27 | Determine the BI application objectives | 6, 10, 24 |
28 | Identify the strategic business goals of the organization | |
29 | Define the overall BI decision-support objectives | |
30 | Define the project-specific BI application objectives | |
31 | Match the overall BI decision-support objectives to the strategic business goals | |
32 | Match the project-specific BI application objectives to the strategic business goals | |
33 | Propose a BI solution | 27 |
34 | Review current DSS solutions | |
35 | Review DSS gap analysis | |
36 | Determine how the BI application will lessen the business pain | |
37 | Create a high-level architecture for the proposed BI solution | |
38 | Consolidate and prioritize unfulfilled requirements from previous BI projects | |
39 | Create a high-level (conceptual) logical data model | |
40 | Perform a cost-benefit analysis | 27 |
41 | Determine costs | |
42 | Determine benefits ( tangible and intangible) | |
43 | Identify short- term benefits to the organization | |
44 | Identify long-term benefits to the organization | |
45 | Calculate the projected return on investment (ROI) | 41, 42 |
46 | Perform a risk assessment | 27 |
47 | Create a risk assessment matrix | |
48 | List the technology risks | |
49 | List the complexity risks | |
50 | List the integration risks | |
51 | List the organization risks | |
52 | List the project team risks | |
53 | List the financial investment risks | |
54 | Assign weights to the risks | 47 |
55 | Rank the risks: low, medium, or high | 54 |
56 | Determine the risks ( ramifications ) of not implementing a BI solution | 54 |
57 | Write the assessment report | 33, 40, 46 |
58 | Describe the business need | |
59 | Describe lost opportunities | |
60 | Describe the proposed BI solution | |
61 | State the cost justification and expected ROI | |
62 | Include risk assessment results | |
63 | Write recommendations (include operational business process improvements) | |
64 | Obtain project approval from business sponsor | 57 |
65 | Step 2: Enterprise Infrastructure Evaluation | |
66 | Section A: Technical Infrastructure Evaluation | |
67 | Assess the existing platform | 64 |
68 | Review hardware | |
69 | Review operating systems | |
70 | Review middleware, especially DBMS gateways | |
71 | Review custom interfaces | |
72 | Review network components and bandwidth | |
73 | Review the DBMS | |
74 | Review tools (CASE, ETL, OLAP, etc.) | |
75 | Review the meta data repository | |
76 | Perform gap analysis | |
77 | Evaluate and select new products | 64 |
78 | Identify the product categories you need to evaluate (hardware, DBMS, tools) | |
79 | List all products being considered for each category | |
80 | Itemize your requirements for the products | |
81 | Weigh each product requirement (scale of 1 to 10) | |
82 | Rank each product against the weighted requirements (scale of 0 to 10) | 81 |
83 | Determine the total score for each product | 82 |
84 | List all vendors of all products | |
85 | Itemize your requirements for the vendors | |
86 | Weigh each vendor requirement (scale of 1 to 10) | |
87 | Rank each vendor against the weighted requirements (scale of 0 to 10) | 86 |
88 | Determine the total score for each vendor | 87 |
89 | Evaluate the product scores and vendor scores | 83, 88 |
90 | Create a short list of products and vendors in each category | 89 |
91 | Have the products demonstrated by the vendors | 90 |
92 | Choose the final product in each product category | 91 |
93 | Obtain business sponsor approval to license the products | 92 |
94 | Write the technical infrastructure assessment report | 67, 77 |
95 | Itemize findings about servers, operating systems, middleware, etc. | |
96 | List the weighted requirements | |
97 | List the product scores | |
98 | List the vendor scores | |
99 | List the product costs | |
100 | List the products on the short list | |
101 | Explain the rationale for selecting or rejecting products | |
102 | Explain the final selection criteria | |
103 | Write the executive summary | |
104 | Expand the current platform | 94 |
105 | Order new products | |
106 | Install new products | 105 |
107 | Test new products | 106 |
108 | Train technical staff on new products | 106 |
109 | Section B: Nontechnical Infrastructure Evaluation | |
110 | Assess the effectiveness of existing nontechnical infrastructure components | 57 |
111 | Review standards for data naming, abbreviations, modeling, etc. | |
112 | Review the use of the development methodology | |
113 | Review estimating guidelines | |
114 | Review change-control procedures | |
115 | Review issues management procedures | |
116 | Review roles and responsibilities | |
117 | Review security processes and guidelines | |
118 | Review meta data capture and delivery processes | |
119 | Review meta data repository functionality | |
120 | Review the process for merging logical data models into the enterprise data model | |
121 | Review data quality measures and the cleansing triage process | |
122 | Review the service-level agreement (SLA) process | |
123 | Review the BI support function | |
124 | Review the dispute resolution process | |
125 | Review the communication process | |
126 | Perform gap analysis | |
127 | Write the nontechnical infrastructure assessment report | 110 |
128 | Itemize findings about inadequate standards, guidelines, procedures, etc. | |
129 | Write recommendations for nontechnical infrastructure changes | |
130 | Prioritize nontechnical infrastructure requirements for the BI project | |
131 | Prioritize nontechnical infrastructure requirements for outside the BI project | |
132 | Write the executive summary | |
133 | Improve the nontechnical infrastructure | 127 |
134 | Create time estimates for creating or modifying new standards, guidelines, procedures | |
135 | Change the guidelines for using the development methodology | |
136 | Modify the roles and responsibilities | |
137 | Create new processes as needed | |
138 | Step 3: Project Planning | |
139 | Determine the project requirements | 94, 127 |
140 | Define data requirements | |
141 | Define functional requirements ( reports , queries, online help function) | |
142 | Define infrastructure requirements (technical and nontechnical) | |
143 | Expand or create the high-level logical data model | 140 |
144 | Validate the requirements with other business people | 140, 141, 142 |
145 | Obtain sponsor approval for the requirements | 144 |
146 | Determine the condition of the source files and databases | 139 |
147 | Review the content of each potential source file and source database (internal and external) | |
148 | Assess source data violations | |
149 | Review technical data conversion rules | |
150 | Review business data domain rules | |
151 | Review business data integrity rules | |
152 | Determine which data elements are critical, important, insignificant | |
153 | Estimate the time needed for cleansing of critical source data | 152 |
154 | Estimate the time needed for cleansing of important source data | 152 |
155 | Review data-cleansing estimates with the business sponsor and prioritize the cleansing effort | 153, 154 |
156 | Determine or revise the cost estimates | 146 |
157 | Review the technical infrastructure assessment report | |
158 | Review the nontechnical infrastructure assessment report | |
159 | Review the project requirements | |
160 | Review the project constraints (time, scope, budget, resources, quality) | |
161 | Review the need for consulting, contracting, training | |
162 | Revise the original cost estimates | 161 |
163 | Revise the risk assessment | 146 |
164 | Review and revise the original risk assessment matrix | |
165 | Determine the likelihood of the risks materializing: low, medium, high | |
166 | Determine the impact of every risk: low, medium, high | |
167 | Define triggers | |
168 | Define a risk mitigation plan | |
169 | Define a contingency plan | |
170 | Identify your assumptions | |
171 | Include assumptions as risks on the contingency plan | |
172 | Review the project constraints as they relate to risk | |
173 | Identify critical success factors | 156, 163 |
174 | Define the success criteria for the BI project | |
175 | Determine critical success factors | 174 |
176 | Review critical success factors with the business sponsor | 175 |
177 | Obtain agreement and cooperation on the critical success factors from the business sponsor | 176 |
178 | Prepare the project charter | 173 |
179 | State the purpose and reason for the BI project | |
180 | State costs and benefits | |
181 | Describe infrastructure and business process improvements | |
182 | Describe the high-level scope (data and functions) | |
183 | List items not in the scope | |
184 | List expectations from the business people (preliminary SLA) | |
185 | Define team structure, roles, and responsibilities | |
186 | List risks, assumptions, and constraints | |
187 | List critical success factors | |
188 | Create a high-level project plan | 173 |
189 | Create a work breakdown structure | |
190 | Determine base estimates for all tasks | |
191 | Identify task dependencies | |
192 | Revise the base estimates for assigned resources | 191 |
193 | Address skill level | |
194 | Address subject matter expertise | |
195 | Address additional administrative activities | |
196 | Address non-work- related activities | |
197 | Identify resource dependencies (resource leveling) | 192 |
198 | Create a critical path method (CPM) or Pert chart | 197 |
199 | Create a Gantt chart | 197 |
200 | Kick off the project | 178, 188 |
201 | Prepare an agenda for the kickoff meeting | |
202 | Call a kickoff meeting | 201 |
203 | Assign roles and responsibilities to core team members | 202 |
204 | Identify extended team members and review their responsibilities | 202 |
205 | Discuss the project charter | 202 |
206 | Walk through the project plan | 202 |
207 | Discuss the concept of self-organizing teams | 202 |
208 | Step 4: Project Requirements Definition | |
209 | Define the requirements for technical infrastructure enhancements | 200 |
210 | Define the requirements for additional hardware | |
211 | Define the requirements for additional middleware | |
212 | Define the requirements for a new DBMS or upgrades to the existing DBMS | |
213 | Define the requirements for the network or upgrades to it | |
214 | Determine the security requirements | |
215 | Define the requirements for development tools (CASE, ETL) | |
216 | Define the requirements for data access and reporting tools (OLAP, report writers) | |
217 | Define the requirements for a new data mining tool | |
218 | Determine whether to license or build a meta data repository | |
219 | Determine how to enhance an existing meta data repository | |
220 | Define the requirements for nontechnical infrastructure enhancements | 200 |
221 | Define the requirements for governance (prioritizing) standards and procedures | |
222 | Define the requirements for the development methodology | |
223 | Define the requirements for estimating guidelines | |
224 | Define the requirements for the scope management process | |
225 | Define the requirements for the issues management process | |
226 | Define the requirements for roles and responsibilities | |
227 | Define the requirements for the security process | |
228 | Define the requirements for the meta data capture and delivery process | |
229 | Define the requirements for logical data modeling | |
230 | Define the requirements for the data cleansing process | |
231 | Define the requirements for the testing procedures | |
232 | Define the requirements for the SLA process | |
233 | Define the requirements for the BI support function | |
234 | Define the requirements for the dispute resolution process | |
235 | Define the requirements for the communication process | |
236 | Define the reporting requirements | 200 |
237 | Collect or create sample report layouts | |
238 | Collect or create sample queries | |
239 | Define business rules for the reports | |
240 | Define aggregation and summarization rules | |
241 | Define reporting dimensions | |
242 | Define query libraries | |
243 | Identify stewards of the libraries | |
244 | Get samples of ad hoc queries (if possible) | |
245 | Define access interfaces | |
246 | Define the requirements for source data | 200 |
247 | Define all source data elements | |
248 | Classify data elements as critical, important, insignificant | |
249 | Define the data domains ( allowable values) | |
250 | Define the significant and obvious business rules for the data | |
251 | Determine the data-cleansing requirements | |
252 | Define the historical data requirements | |
253 | Review the project scope | 209, 220, 236, 246 |
254 | Compare the detailed project requirements to the high-level scope in the project charter | |
255 | Review the project constraints (time, scope, budget, resources, quality) | |
256 | Determine whether the scope is realistic under those constraints | |
257 | Renegotiate the scope, if necessary | 256 |
258 | Create a change-control document | |
259 | Create an issues log | |
260 | Expand the logical data model | 253 |
261 | Add newly discovered entities and relationships | |
262 | Refine the logical data model by resolving the many-to-many relationships | 261 |
263 | Add unique identifiers to each entity | 262 |
264 | Attribute the logical data model with critical data elements | 262 |
265 | Define preliminary service-level agreements | 253 |
266 | Identify or revise the expectations for availability | |
267 | Identify or revise the expectations for security | |
268 | Identify or revise the expectations for response time | |
269 | Identify or revise the expectations for data cleanliness | |
270 | Identify or revise the expectations for ongoing support | |
271 | Write the application requirements document | 260, 265 |
272 | Describe the technical infrastructure requirements | |
273 | Describe the nontechnical infrastructure requirements | |
274 | Describe the reporting requirements | |
275 | Describe the ad hoc and canned query requirements | |
276 | Describe the requirements for source data, including history | |
277 | Describe the data-cleansing requirements | |
278 | Describe the security requirements | |
279 | List the preliminary SLAs | |
280 | Step 5: Data Analysis | |
281 | Analyze the external data sources | 271 |
282 | Identify entities and relationships from each external data source | |
283 | Merge the new entities and relationships from the external data sources into the logical data model | |
284 | Refine the logical data model | 271, 281 |
285 | Fully attribute the logical data model to include all required source data elements | |
286 | Create new entities and relationships where needed to store the new attributes | |
287 | Analyze the layout of all identified source files and source databases | |
288 | Analyze the content of all identified source data elements | |
289 | Create the data-specific business meta data components | |
290 | Analyze the source data quality | 271, 281 |
291 | Apply business data domain rules and business data integrity rules | |
292 | Look for default values | |
293 | Look for missing values | |
294 | Look for cryptic values | |
295 | Look for contradicting values | |
296 | Look for values that violate the business rules | |
297 | Look for missing primary keys | |
298 | Look for duplicate primary keys | |
299 | Determine the severity of the problem | |
300 | Determine the criticality of the problem | |
301 | Expand the enterprise logical data model | 284, 290 |
302 | Merge the project-specific logical data model into the enterprise logical data model | |
303 | Identify data discrepancies and inconsistencies between the logical data models | |
304 | Resolve data discrepancies | 290 |
305 | Discuss the discrepancies with data owners and other business executives | |
306 | Adjust either the project-specific logical data model or the enterprise logical data model | |
307 | Notify other affected project teams | |
308 | Document the discrepancies as meta data and schedule time for resolutions | |
309 | Write the data-cleansing specifications | 304 |
310 | Review the classification of data elements: critical, important, insignificant | |
311 | Write data-cleansing specifications for all critical data elements | |
312 | Write data-cleansing specifications for selected important data elements | |
313 | Step 6: Application Prototyping | |
314 | Analyze the access requirements | 271 |
315 | Review the application requirements document with the subject matter expert and the business representative | |
316 | Analyze the report requirements | |
317 | Analyze the query requirements | |
318 | Analyze the ad hoc requirements | |
319 | Analyze the interface requirements | |
320 | Communicate all your findings to the database administrator | |
321 | Create a skill set matrix for each business person participating in the prototyping activities | |
322 | Indicate computer skill level: beginning, advanced, expert | |
323 | Indicate application knowledge: beginning, advanced, expert | |
324 | Determine the scope of the prototype | 271 |
325 | Determine the objective and the primary use of the prototype | |
326 | Decide which type of prototype to build (show-and-tell, demo, etc.) | |
327 | Select a subset of functions (reports, queries, ETL, interface) | |
328 | Select a subset of sample data from source files and source databases | |
329 | Create a change-control document for the prototype | |
330 | Create an issues log for the prototype | |
331 | Determine the number of prototype iterations | |
332 | Determine the number of prototype participants | |
333 | Determine the time limits for each prototype iteration | 331 |
334 | Estimate the cost and benefit for each prototype iteration | 331 |
335 | Determine the point of diminishing returns for prototyping | 331 |
336 | Select tools for the prototype | 271 |
337 | Review existing in-house tools and find out who uses them | |
338 | Review the availability of new reporting and querying tools | |
339 | Review existing or new graphical tools | |
340 | Review existing or new report distribution tools | |
341 | Review existing DBMS options for the prototype | |
342 | Select the platform on which the prototype will be developed | |
343 | Select one of the installed and tested DBMSs | |
344 | Select one or more existing or new tools | |
345 | Determine training needs for the new tools | |
346 | Schedule training sessions | 345 |
347 | Prepare the prototype charter | 314, 324, 336 |
348 | State the purpose of the prototype | |
349 | State what type of prototype you selected | |
350 | List who will participate (IT and business people) | |
351 | Define what the rules are (scope, time, iterations) | |
352 | Define how you will measure the success of the prototype | |
353 | Design the reports and queries | 347 |
354 | Design the reports | |
355 | Design the queries | |
356 | Design the interfaces | |
357 | Create a physical data model (database design) for the prototype database | |
358 | Identify the data to be used for the prototype | |
359 | Map sample source data or new test data into the prototype database | 358 |
360 | Build the prototype | 347 |
361 | Create the physical prototype database | |
362 | Create sample test data | |
363 | Load the prototype database with sample data | |
364 | Write a selected subset of reports | |
365 | Write a selected subset of queries | |
366 | Write a selected subset of interfaces or other functions | |
367 | Test reports, queries, interfaces, or other functions | 364, 365, 366 |
368 | Document any problems with the tool | 367 |
369 | Document any issues with the reports or queries | 367 |
370 | Document any issues with the interfaces or other functions | 367 |
371 | Document any issues with dirty source data | 367 |
372 | Validate the time and cost estimates for the BI application | 367 |
373 | Demonstrate the prototype | 353, 360 |
374 | Review reports and queries with the business people | |
375 | Review problems and issues with the business sponsor and the business representative | |
376 | Review the project requirements with the subject matter expert and the business representative | |
377 | Document requested changes in the change-control document | |
378 | Analyze the impact of requested changes on other constraints (time, quality, cost, resources) | 377 |
379 | Review impact of requested changes with the business sponsor and the business representative | 378 |
380 | Revise the application requirements document to include approved changes | 379 |
381 | Review lessons learned with the entire project core team and in particular with the ETL step core team | 380 |
382 | Use prototype demonstrations to promote the BI application | 380 |
383 | Perform the next prototype iteration, if applicable | 373 |
384 | Step 7: Meta Data Repository Analysis | |
385 | Analyze the meta data repository requirements | 271 |
386 | Review the technical infrastructure assessment report | |
387 | Perform a cost-benefit analysis for licensing versus building a meta data repository | |
388 | Make the decision to license or build a meta data repository | 387 |
389 | Review the nontechnical infrastructure assessment report | |
390 | Determine the scope of the meta data repository deliverables | |
391 | Prioritize the meta data repository deliverables | 390 |
392 | Update the application requirements document to reflect any changes | 391 |
393 | Analyze the interface requirements for the meta data repository | 271 |
394 | Analyze the meta data sources | |
395 | Analyze word processing files and spreadsheets | |
396 | Analyze DBMS dictionaries | |
397 | Analyze CASE, ETL, OLAP tools | |
398 | Analyze report writers and query tools | |
399 | Analyze the data mining tool | |
400 | Determine what import and export features are available in these tools | |
401 | Determine what import and export features are available in the meta data repository product | |
402 | Analyze the meta data repository access and reporting requirements | 271 |
403 | Review the original meta data repository access and reporting requirements | |
404 | Review the meta data security requirements | |
405 | Identify the access interface media (PDF, HTML) | |
406 | Analyze the feasibility of a context-sensitive help function | |
407 | Determine what reports should be produced from the meta data repository | |
408 | Create the logical meta model | 385, 393, 402 |
409 | Create business meta data entities | |
410 | Create technical meta data entities | |
411 | Determine the relationships between the meta data entities | |
412 | Create attributes for business and technical meta data entities | |
413 | Draw an entity-relationship diagram | |
414 | Create the meta-meta data | |
415 | Describe all meta data entities | 408 |
416 | Name the meta data entities | |
417 | Define all meta data entities | |
418 | Define the relationships between all meta data entities | |
419 | Define the security for meta data entities | |
420 | Define the physical location for meta data entities | |
421 | Define timeliness for meta data | |
422 | Define volume for meta data entities | |
423 | Describe all meta data attributes | 415 |
424 | Name the meta data attributes | |
425 | Define all meta data attributes | |
426 | Define type and length for meta data attributes | |
427 | Define the domain for meta data attributes | |
428 | Define the security for meta data attributes | |
429 | Define ownership for meta data attributes | |
430 | Define the business rules for meta data entities, attributes, and relationships | 423 |
431 | Step 8: Database Design | |
432 | Review the data access requirements | 309, 373 |
433 | Review the data-cleansing specifications | |
434 | Review the prototyping results | |
435 | Review detailed access and analysis requirements | |
436 | Review detailed reporting requirements | |
437 | Review detailed querying requirements | |
438 | Review known ad hoc querying requirements | |
439 | Review data security requirements | |
440 | Determine projected data volumes and growth factors | |
441 | Determine the projected number of concurrent database usages | |
442 | Determine the location of business people | |
443 | Determine the frequency of report and query executions | |
444 | Determine the peak and seasonal reporting periods | |
445 | Determine platform limitations | |
446 | Determine tool limitations (ETL, OLAP, report writers) | |
447 | Determine the aggregation and summarization requirements | 309, 373 |
448 | Review measures (facts) used by the prototype | |
449 | Review the dimensions used by the prototype | |
450 | Review the drill-down and roll-up functions of the prototype | |
451 | Review common reporting patterns among existing reports | |
452 | Determine the most frequently used reporting dimensions | |
453 | Review the logical data model with the data administrator | |
454 | Determine the level of detail (granularity) needed | |
455 | Determine how the detailed data will be accessed (drill-down or ad hoc) | |
456 | Determine how many business relationships (entity relationships) will be needed | |
457 | Design the BI target databases | 435, 447 |
458 | Determine the appropriate database design schemas (multidimensional or entity-relationship) | |
459 | Create the physical data models (database design diagrams) | |
460 | Create the technical meta data for the physical data models | 459 |
461 | Map the physical data models to the logical data model | 459 |
462 | Design the physical database structures | 457 |
463 | Determine how to cluster the tables | |
464 | Determine the placement of datasets | |
465 | Determine how to stripe disks | |
466 | Determine how to partition the tables across multiple disks | |
467 | Determine how much free space to choose | |
468 | Determine how much buffer space to declare | |
469 | Determine how large to set the blocksize | |
470 | Determine the most appropriate indexing strategy | |
471 | Determine whether referential integrity will be enforced by the DBMS | |
472 | Build the BI target databases | 462 |
473 | Create the data definition language (DDL) | |
474 | Define storage groups | |
475 | Define databases | |
476 | Define partitions | |
477 | Define tablespaces | |
478 | Define tables | |
479 | Define columns | |
480 | Define primary keys | |
481 | Define foreign keys | 480 |
482 | Define indices | 480 |
483 | Create the data control language (DCL) | 473 |
484 | Define parameters for the security SYSTABLE | |
485 | Set up group IDs | |
486 | Grant CRUD (create, read, update, delete) authority to group IDs | |
487 | Assign developers, business analysts, and programs to the appropriate group IDs | |
488 | Run the DDL to create the physical database structures | 473 |
489 | Run the DCL to grant authority to the physical database structures | 483 |
490 | Build the indices | 488 |
491 | Develop database maintenance procedures | 483 |
492 | Define database maintenance activities | |
493 | Define database backups (full and incremental backups ) | |
494 | Define disaster recovery procedures | |
495 | Define reorganization procedures for fragmented tables | |
496 | Define the frequency of and procedure for performance monitoring activities | |
497 | Prepare to monitor and tune the database designs | 491 |
498 | Plan to monitor the performance of ETL loads, reports, and queries at runtime | |
499 | Plan to use a performance-monitoring utility to diagnose performance degradation | |
500 | Plan to refine the database design schemas | |
501 | Plan to add additional indices, if necessary | |
502 | Prepare to monitor and tune the query designs | 491 |
503 | Plan to review and streamline all SQL calls in programs | |
504 | Plan to write pass-through queries for OLAP tools, if necessary | |
505 | Plan to utilize parallel query execution | |
506 | Step 9: Extract/Transform/Load Design | |
507 | Create the source-to-target mapping document | 491 |
508 | Review the record layouts for the source files | |
509 | Review the data description blocks for the source databases | |
510 | Review the data-cleansing specifications for source data elements | |
511 | Create a matrix for all target tables and target columns | |
512 | List all applicable source files and source databases for every target table | |
513 | List all relevant source data elements for every target column | |
514 | List data type and length for every target column | |
515 | List data type and length for every source data element | |
516 | Write transformation specifications for populating the columns | 514, 515 |
517 | Combine data content from multiple sources (if needed) | |
518 | Split data content from one data element across multiple columns (if needed) | |
519 | Include aggregation and summarization algorithms | |
520 | Include data-cleansing specifications for each column | |
521 | Include logic for checking referential integrity (if not performed by the DBMS) | |
522 | Include logic for error messages and record rejection counts | |
523 | Include logic for reconciliation totals (record counts, domain counts, amount counts) | |
524 | Test the ETL tool functions | 491 |
525 | Review the transformation specifications in the source-to-target mapping document | 516 |
526 | Determine whether the ETL tool functions can perform the required transformation logic | 525 |
527 | Determine what supplementary custom code must be written | 526 |
528 | Design the ETL process flow | 507, 524 |
529 | Determine the most efficient sequence to extract source data | |
530 | Determine the most efficient sequence to transform, cleanse , and load the data | |
531 | Determine the sort and merge steps in the ETL process | |
532 | Identify all temporary and permanent work files and tables | |
533 | Determine what components of the ETL process can run in parallel | |
534 | Determine what tables can be loaded in parallel | |
535 | Draw the process flow diagram | 533, 534 |
536 | Show the extracts from source files and source databases | |
537 | Indicate temporary and permanent work files and tables | |
538 | Show the sort and merge processes | |
539 | Show the transformation programs | |
540 | Show the error rejection files and error reports | |
541 | Show the load files and load utilities | |
542 | Design the ETL programs | 528 |
543 | Design three sets of ETL programs | |
544 | Design the initial load programs | |
545 | Design the historical load programs | |
546 | Design the incremental load programs | |
547 | Modularize the ETL programs | |
548 | Translate the transformation specifications into programming specifications | 547 |
549 | Set up the ETL staging area | 528 |
550 | Determine whether and how to distribute the ETL process | |
551 | Set up the ETL server | |
552 | Allocate space for temporary and permanent work files and tables | 551 |
553 | Create program libraries | 551 |
554 | Establish program-versioning procedures | 551 |
555 | Step 10: Meta Data Repository Design | |
556 | Design the meta data repository database | 414 |
557 | Review the logical meta model for the meta data repository | |
558 | Design the meta data repository database (entity-relationship or object-oriented) | |
559 | Draw the physical meta model diagram (entity-relationship or object-oriented) | |
560 | Map the physical meta model to the logical meta model | |
561 | Create the DDL for the meta data repository database | |
562 | Create the DCL for the meta data repository database | |
563 | Design backup and recovery procedures | |
564 | Design versioning and archival procedures | |
565 | Install and test the meta data repository product | 414 |
566 | Compile a list of meta data repository products and vendors | |
567 | Compare the meta data repository products to the meta data repository requirements | |
568 | Create a scorecard for each evaluated meta data repository product | |
569 | Create a scorecard for each evaluated meta data repository vendor | |
570 | Narrow the list of meta data repository products and vendors to a short list | 568, 569 |
571 | Arrange for meta data repository product demos | |
572 | Check the vendors' client references | |
573 | License the meta data repository product | 572 |
574 | Install and test the meta data repository product | 573 |
575 | Design the meta data migration process | 556, 565 |
576 | Analyze all sources for extracting business meta data | |
577 | Analyze all sources for extracting technical meta data | |
578 | Design the tool interface process | |
579 | Design the transformations for the extracted meta data | |
580 | Design the load programs for the meta data repository | |
581 | Write the programming specifications for the meta data migration process | |
582 | Write tool interface programming specifications | 578 |
583 | Write transformation programming specifications | 579 |
584 | Write meta data repository load programming specifications | 580 |
585 | Design the meta data application | 556 |
586 | Design the meta data repository report programs | |
587 | Design the media for displaying meta data ad hoc query results | |
588 | Design the access interface process | |
589 | Design the context-sensitive online help function | |
590 | Write the programming specifications for the meta data application | |
591 | Write report programming specifications | 586 |
592 | Write query script specifications | 587 |
593 | Write access interface programming specifications | 588 |
594 | Write online help function programming specifications | 589 |
595 | Step 11: Extract/Transform/Load Development | |
596 | Build and unit test the ETL process | 542, 549 |
597 | Code the ETL programs | |
598 | If using an ETL tool, write instructions for the ETL tool modules | |
599 | Capture the ETL technical meta data for the meta data repository | |
600 | Write code to produce reconciliation totals, quality metrics, and load statistics | |
601 | Unit test each individual program module | 597, 600 |
602 | If using an ETL tool, unit test each ETL tool module | 598, 600 |
603 | Write the scripts to execute the ETL programs and the sort, merge, and load utilities | 601, 602 |
604 | Integration or regression test the ETL process | 596 |
605 | Create a test plan with test cases for the ETL process | |
606 | Create test data for the ETL programs | |
607 | Integration or regression test the entire ETL process | 605, 606 |
608 | Log the actual test results and document any test issues | 605, 606 |
609 | Compare actual test results with expected test results | 605, 606 |
610 | Revise the ETL programs (or the instructions for the ETL tool) | 609 |
611 | Retest the entire ETL process from beginning to end | 610 |
612 | Performance test the ETL process | 604 |
613 | Test individual ETL programs and tool modules that read or write to high-volume tables | |
614 | Test the parallel execution of ETL programs and tool modules against high-volume tables | |
615 | Test the ETL programs and ETL tool modules that perform complicated operations | |
616 | Use full-volume data for performance testing | |
617 | If using a stress test simulation tool, define test components and run a simulation test | |
618 | Quality assurance (QA) test the ETL process | 612 |
619 | Move all ETL programs into the QA environment | |
620 | QA test the entire ETL process from beginning to end | 619 |
621 | Obtain approval from the operations staff to move the ETL process into production | 620 |
622 | Acceptance test the ETL process | 612 |
623 | Acceptance test the entire ETL process from beginning to end | |
624 | Validate all cleansing transformations | |
625 | Validate error-handling routines | |
626 | Validate reconciliation totals | |
627 | Obtain certification for the ETL process from the business representative | 623 |
628 | Step 12: Application Development | |
629 | Determine the final project requirements | 491 |
630 | Review the results of the prototype | |
631 | Review the prototyping programs and scripts | |
632 | Review the change-control document | |
633 | Review the issues log | |
634 | Review existing and mock-up report layouts | |
635 | Review existing spreadsheets | |
636 | Review the latest version of the application requirements document | |
637 | Agree on the final project requirements | |
638 | Update the application requirements document to reflect any changes | 637 |
639 | Design the application programs | 491 |
640 | Design the final reports | |
641 | Design the final queries | |
642 | Design the front-end interface (GUI, Web portal) | |
643 | Design the online help function | |
644 | Write programming specifications | |
645 | Write report programming specifications | 640 |
646 | Write query script specifications | 641 |
647 | Write front-end interface programming specifications | 642 |
648 | Write online help function programming specifications | 643 |
649 | Create a test plan with test cases and a test log | 645, 646 |
650 | Build and unit test the application programs | 639 |
651 | Create sample test data | |
652 | Load the development databases with sample test data | |
653 | Rewrite or enhance prototyping programs and scripts | |
654 | Code the final report programs | |
655 | Code the final query scripts | |
656 | Code the final front-end interface programs | |
657 | Code the online help function programs | |
658 | Unit test each individual program module | 654, 655, 656, 657 |
659 | Test the application programs | 650 |
660 | Integration or regression test all programs and scripts from beginning to end | |
661 | Integration or regression test report programs | |
662 | Integration or regression test query scripts | |
663 | Integration or regression test front-end interface programs | |
664 | Integration or regression test online help function programs | |
665 | Log the actual test results and document any test issues | |
666 | Compare actual test results with expected test results | |
667 | Revise the application programs and scripts | 666 |
668 | Retest the application programs and scripts from beginning to end | 667 |
669 | Performance test complex high-volume programs | 668 |
670 | Use full-volume data for performance testing | 668 |
671 | If using a stress test simulation tool, define test components and run a simulation test | 668 |
672 | Move databases, programs, and scripts into the QA environment | 669 |
673 | QA test the entire application from beginning to end | 672 |
674 | Obtain approval from the operations staff to move the application programs into production | 673 |
675 | Acceptance test the entire application from beginning to end | 669 |
676 | Obtain certification for the application from the business representative | 675 |
677 | Provide data access and analysis training | 650 |
678 | Identify help desk staff to be trained | |
679 | Identify "power users" or other business liaison personnel to be trained | |
680 | Identify business people to be trained | |
681 | Create training materials | |
682 | Create presentation slides and instructor notes | |
683 | Create student workbooks with exercises | 682 |
684 | Create exercise solutions and other pertinent handouts | 683 |
685 | Schedule training sessions | 684 |
686 | Conduct training sessions | 685 |
687 | Measure training effectiveness | 686 |
688 | Step 13: Data Mining | |
689 | State the business problem | 491 |
690 | Define the business problem | |
691 | Obtain commitment for a data mining solution | 690 |
692 | Set realistic expectations for the data mining tool | 691 |
693 | Identify preliminary algorithms relevant to the business problem | 691 |
694 | Collect the data | 689 |
695 | Identify available data sources (operational as well as BI) | |
696 | Extract pertinent data from various internal data sources | |
697 | Acquire pertinent data from external data sources | |
698 | Consolidate and cleanse the data | 689 |
699 | Merge data from various internal data sources | |
700 | Match and merge internal data with external data | |
701 | Review the structure of the merged data | |
702 | Select a sample of data for each analytical data model | |
703 | Select related meta data from the meta data repository | |
704 | Review the data domains and measure the quality and reasonability of data values | 702 |
705 | Validate domain reasonability across active variables | 702 |
706 | Prepare the data | 702 |
707 | Review the frequency distribution of categorical variables | |
708 | Review maximum, minimum, mean, mode, and median for quantitative variables | |
709 | Use statistical distribution parameters to filter noise in the data | |
710 | Eliminate or replace variables with missing values | |
711 | Convert data formats to suit the particular data mining algorithm used | |
712 | Derive new variables from original input data | |
713 | Consolidate customers by assigning a household number to related customers | |
714 | Relate customers with products and services | |
715 | Apply data reduction | |
716 | Apply data mining transformation techniques | 715 |
717 | Apply "discretization" technique | |
718 | Apply "one-of-N" technique | |
719 | Build the analytical data model | 694, 698, 706 |
720 | Create the analytical (informational) data model | |
721 | Select data mining operations with the appropriate algorithms | |
722 | Test accuracy using confusion matrices and input sensitivity analyses | |
723 | Repeat prior steps to train and retrain the model | 722 |
724 | Interpret the data mining results | 719 |
725 | Review the data mining results | |
726 | Look for results that are interesting, valid, and actionable | |
727 | Present the new findings using visualization technology | 726 |
728 | Formulate ways in which the new information can be exploited | 726 |
729 | Perform external validation of the results | 719 |
730 | Compare data mining results to published industry statistics | |
731 | Validate the selection of your variables and time frame against the variables and time frame of the industry statistics | |
732 | Identify the variations between your analysis results and the industry statistics | |
733 | Determine the reasons for the variations | |
734 | Monitor the analytical data model over time | 724, 729 |
735 | Keep validating your analytical data model against industry statistics at regular time intervals | |
736 | When industry statistics change, change your analytical data model and retrain it | 735 |
737 | Research the data mining capabilities of your competitors | |
738 | Monitor your competitors' market share and adjust your model | |
739 | Step 14: Meta Data Repository Development | |
740 | Build the meta data repository database | 575, 585 |
741 | Run the DDL to create the physical meta data repository database structures | |
742 | Run the DCL to grant CRUD authority on the meta data repository database structures | |
743 | If licensing a meta data repository product, set up CRUD authority on the meta data repository product | |
744 | Test all meta data repository product components, especially the meta data repository database | |
745 | Build and unit test the meta data migration process | 740 |
746 | Code the tool interface programs or use the export facility of the various tools | |
747 | Code the meta data transformation programs | |
748 | Code the meta data load programs or use the import facility of the meta data repository product or the DBMS load utility | |
749 | Code the meta data programs that will run during ETL | |
750 | Code the meta data programs to capture load statistics | |
751 | Code the meta data programs to capture reconciliation totals | |
752 | Code the meta data programs to capture data-cleansing metrics | |
753 | Code the meta data programs to capture rejection counts and reasons for rejections | |
754 | Unit test the meta data migration programs | |
755 | Unit test the tool interface programs | 746 |
756 | Unit test the meta data transformation programs | 747 |
757 | Unit the test meta data load programs | 748 |
758 | Unit test the meta data programs that will run during the ETL process | 749 |
759 | Build and unit test the meta data application | 740 |
760 | Code the access interface programs | |
761 | Code the meta data report programs | |
762 | Code the meta data query scripts | |
763 | Code the meta data repository online help function programs | |
764 | Unit test the meta data application programs (or meta data repository product modules) | |
765 | Unit test the access interface programs | 760 |
766 | Unit test the meta data report programs | 761 |
767 | Unit test the meta data query scripts | 762 |
768 | Unit test the meta data repository online help function programs | 763 |
769 | Test the meta data repository programs or product functions | 745, 759 |
770 | Create a test plan with test cases | |
771 | Create test cases for the meta data migration process | |
772 | Create test cases for the meta data repository application programs or product modules | |
773 | Create test cases for the meta data programs that run during the ETL process | |
774 | Create test data for meta data repository testing | |
775 | Create test data for the meta data migration process | |
776 | Create test data for the meta data repository application or product modules | |
777 | Create test data for the meta data programs that run during the ETL process | |
778 | Integration or regression test the meta data repository | 770, 774 |
779 | Integration or regression test the meta data migration process | |
780 | Integration or regression test the meta data repository application or product modules | |
781 | Integration or regression test the meta data programs that run during the ETL process | |
782 | Log the actual test results and document any test issues | 778 |
783 | Compare actual test results with expected test results | 778 |
784 | Revise the meta data repository programs | 783 |
785 | Retest the meta data repository programs from beginning to end | 784 |
786 | Conduct QA testing with operations staff | 785 |
787 | Conduct acceptance testing with the subject matter expert and the business representative | 785 |
788 | Prepare the meta data repository for production | 769 |
789 | Install and test the server platform for the production meta data repository | |
790 | Create DDL and DCL for the production meta data repository database | |
791 | Write operating procedures for the operations staff for running the meta data repository reports | |
792 | Write a reference guide for the help desk staff and the business people | |
793 | Develop performance monitoring and tuning procedures for the meta data repository database | |
794 | Develop meta data repository usage monitoring procedures | |
795 | Provide meta data repository training | 769 |
796 | Identify help desk staff to be trained | |
797 | Identify "power users" to be trained | |
798 | Identify business people to be trained | |
799 | Create meta data repository training materials | |
800 | Create meta data repository presentation slides and instructor notes | |
801 | Create meta data repository student workbooks with exercises | 800 |
802 | Create exercise solutions and other pertinent handouts | 801 |
803 | Schedule meta data repository training sessions | 802 |
804 | Conduct meta data repository training sessions | 803 |
805 | Measure meta data repository training effectiveness | 804 |
806 | Step 15: Implementation | |
807 | Plan the implementation | 618, 622, 659, 719, 788 |
808 | Select an implementation (rollout) strategy | |
809 | Set the implementation date | |
810 | Determine the number of business people for the initial rollout | |
811 | Schedule the necessary resources to participate in implementation activities | |
812 | Schedule the functions to be rolled out | |
813 | Prepare for organizational impact | |
814 | Set up the production environment | 807 |
815 | Set up the production ETL program library | |
816 | Set up the production application program library | |
817 | Set up the production meta data repository program library | |
818 | Create the production BI target databases | |
819 | Create the production meta data repository database | |
820 | Grant appropriate authority on the production BI target databases | 818 |
821 | Grant appropriate authority on the production meta data repository database | 819 |
822 | Grant appropriate authority on all production program libraries | 815, 816, 817 |
823 | Write ETL operating procedures for operations staff | |
824 | Write application reference guides for help desk staff and the business people | |
825 | Implement production security levels for all BI application components | |
826 | Install all the BI application components | 814 |
827 | Move ETL programs into the production ETL program library | |
828 | Move initial load programs | |
829 | Move historical load programs | |
830 | Move incremental load programs | |
831 | Move application programs into the production application program library | |
832 | Move report programs | |
833 | Move query scripts | |
834 | Move front-end interface programs | |
835 | Move online help function programs | |
836 | Move meta data repository programs into the production meta data repository program library | |
837 | Move meta data migration programs | |
838 | Move meta data application programs or product modules | |
839 | Set up the production schedule | 814 |
840 | Set up the ETL process on the job scheduler | |
841 | Add to the job scheduler the meta data programs that run during the ETL process | |
842 | Set up on the job scheduler the regularly scheduled application report programs | |
843 | Set up on the job scheduler the regularly scheduled meta data repository programs | |
844 | Set up the meta data migration process | |
845 | Set up the meta data repository application | |
846 | Load the production databases | 826, 839 |
847 | Run the initial load process | |
848 | Run the historical load process | |
849 | Run the meta data migration process | |
850 | Prepare for ongoing support | 846 |
851 | Establish a schedule for on-call emergency support | |
852 | Schedule database maintenance activities for the production databases | |
853 | Schedule database backups | |
854 | Schedule disaster recovery testing | |
855 | Schedule database reorganizations | |
856 | Schedule database monitoring activities for the production databases | |
857 | Schedule performance monitoring activities | |
858 | Schedule growth monitoring activities | |
859 | Schedule usage monitoring activities | |
860 | Schedule data quality monitoring activities for the BI target databases | |
861 | Schedule activities for reviewing meta data metrics | |
862 | Schedule quality spot checks | |
863 | Develop or review capacity plans for the BI platform | |
864 | Develop capacity plans for processors | |
865 | Develop capacity plans for disk storage | |
866 | Develop capacity plans for network components (including bandwidth) | |
867 | Start production processing (go live) | 850 |
868 | Step 16: Release Evaluation | |
869 | Prepare for the post-implementation review | 850FS+30 days |
870 | Review budget expenditures | |
871 | Review the original project plan and final schedule | |
872 | Review the estimated and actual task completion times | |
873 | Review the issues log (resolved and unresolved issues) | |
874 | Review the change-control procedure and scope changes | |
875 | Review unfulfilled requirements (dropped from scope) | |
876 | Review the effectiveness of the development approach | |
877 | Review the effectiveness of the team structure | |
878 | Review the effectiveness of the organizational placement | |
879 | Review the existing infrastructure (technical and nontechnical) | |
880 | Identify missing infrastructure pieces (technical and nontechnical) | 879 |
881 | Assess the performance of the BI application | |
882 | Review the effectiveness of training | |
883 | Review the implementation (rollout) strategy | |
884 | Review the effectiveness of the release concept | |
885 | Organize the post-implementation review meeting | 850FS+30 days |
886 | Create the preliminary post-implementation review agenda | |
887 | List date, time, and place | |
888 | List invited attendees | |
889 | List topics for discussion | |
890 | List and assign topics for research | |
891 | List questions to be discussed and answered | |
892 | Solicit additional topics and questions from attendees | |
893 | Send out the preliminary agenda to attendees | |
894 | Schedule the meeting at an off-site location | |
895 | Arrange facilitation by a third party | |
896 | Arrange for a third-party scribe to take notes during the meeting | |
897 | Revise and send the final meeting agenda | 893 |
898 | Send out documentation to be discussed during the review | 897 |
899 | Conduct the post-implementation review meeting | 869, 885 |
900 | Introduce the attendees | |
901 | Explain the rules for the facilitated session | |
902 | Discuss each item on the agenda | |
903 | Document discussions, suggestions, resolutions | |
904 | Document action items | |
905 | Assign action items | |
906 | Establish completion or response date for each action item | |
907 | Follow up on the post-implementation review | 899 |
908 | Document unfulfilled requirements to be bundled with the next BI release | |
909 | Write the meeting minutes | |
910 | Publish the meeting minutes | |
911 | Work on assigned action items | |
912 | Monitor the work performed on action items | |
913 | Document the action item results | 911 |
914 | Publish the action item results | 913 |
915 | Implement nontechnical infrastructure improvements | |
916 | Improve the development approach | |
917 | Improve use of the development methodology | |
918 | Improve processes and procedures | |
919 | Improve guidelines | |
920 | Improve standards | |