In order to provide some context and structure, the test steps in this section are divided by project phase. However, it won't always work out as neatly as what's laid out here, so this cannot be a robotic process. For example, there may be occasions when the right time to address a step from the testing section is during the requirements-gathering phase.
The auditor should perform each step at the point in the project where it makes the most sense, based on how the project is run. It is critical for the auditor to understand the methodology of the project and adjust his or her approach accordingly. For example, if the project is using incremental development, where each project phase is executed multiple times, then the auditor may need to audit each phase concurrently or possibly even multiple times. The controls needed for a project generally will be the same regardless of the project methodology, but matching the audit phase to the project and coordinating the timing will be more difficult for some types of projects than for others. Part of the planning process should involve obtaining an understanding of the project methodology used and determining the appropriate timing and method for accomplishing the steps in this program.
The auditors should determine what project management tool is used by the project team and obtain a familiarity with the tool and its terminology. This will allow the auditors to "speak the same language" as the people they are auditing and further enhance credibility.
In addition, some of these steps may be overkill for smaller projects. The auditor should use judgment in determining which of these risks are material enough to address for each specific project. Finally, these steps are written in such a way that they can be used for any sort of IT project, whether it involves acquiring or developing software, procuring new technology, or developing a process. Auditor judgment should be used in determining which steps are most applicable based on the kind of project being audited.
The words software, system, and process are used in the following steps interchangeably and in conjunction with one another. They are basically intended to represent "the thing that is being developed by the project team." The use of one word versus another in a given test step is not intended to convey any specific meaning.
Note | The steps in this section likely should be performed thoroughly at the beginning of the project and then hit again lightly during each phase of the project to ensure that the disciplines are still being followed. Project management may start out strong, but it often wanes as people become busy and are scrambling to meet deadlines. |
This sort of documentation will increase the likelihood that the project is being implemented in a disciplined manner and is following your company's established standards and methodologies. This, in turn, greatly increases the chances that the project will be executed successfully and will produce the business value desired. Also, this documentation can be of benefit to future projects, therefore allowing the company to leverage past efforts. Finally, your company may have specific standards for executing projects based on either internal or regulatory requirements.
Get copies of whatever project documentation exists. Review this documentation and compare it with the standards and requirements of your company. The documents required will vary by company, but look for documents covering areas such as milestones, work breakdown structure (WBS), project approach, statement of work (SOW), requirements, test plans, and design documents. Obtain a copy of your company's project methodology standards and compare them with the methodology being executed on the project being reviewed. When reviewing this documentation, determine whether there seems to be evidence of adequate project and resource planning. Performance of this step is not an exact science. Instead, the auditor is trying to develop a feel for the overall level of documentation and processes established for the project. Some of this documentation will be examined in more detail during later steps. In this step, the auditor should obtain the document(s) that constitute the basic project plan and determine whether customer needs, deliverables, objectives, and scope are clearly defined.
For reasons mentioned in the preceding step, this documentation enhances the quality of current and future projects. However, if it is allowed to become outdated, it quickly becomes useless.
Through interviews with the project team, understand the processes in place for updating these documents when necessary. Look for evidence that updates have been made.
If proper security and change controls are not in place, unauthorized, inaccurate, and /or unnecessary changes may be made to the project documentation.
Ensure that files containing project documentation are locked down and can be modified only by an appropriate subset of project personnel (using techniques described in Chapters 6 and 7). Interview project personnel to understand processes for changing critical project documents. Ensure that there is an approval process required before changes are made to significant project documents and that the approval process cannot be circumvented. The documents that constitute the basic project plan (e.g., customer needs, deliverables, objectives, scope, budget, risks, and communication strategies) should be baselined early in the project such that it cannot be changed without agreement from all key stakeholders.
If these processes are not in place, a system crash or data center disaster could result in the permanent loss of project software and documentation.
Review processes or scripts indicating that project data are backed up. If offsite backup is accomplished via tape storage, review the logs or records indicating that they have been taken offsite. Review written recovery procedures, ensuring that they specify what steps are to be performed for recovery, the order of those steps, and who is to perform each step.
During the course of any project, issues inevitably will arise regarding the project itself or the system, process, or software being developed. Without a robust method of capturing and resolving those issues, there likely will be instances where issues will "slip through the cracks" and not be resolved, resulting in failures in the product or failures in execution of the project.
Review the issues database, issues spreadsheet, or whatever other method has been established for recording and tracking issues. Ensure that the issue-tracking tool records adequate information regarding each issue, including description of the issue, priority level, due date, latest status, and resolution information. Ensure that controls exist over the tool used to track issues, such as backups and security to prevent unauthorized updates. Review processes for escalating issues and for ensuring that issues are tracked to resolution. Review the issues list for evidence that issues are being closed. Interview customers to ensure that the process is working.
During most projects, requests for additional functionality will arise after the project has commenced and the requirements have been established and approved. Without a method for ensuring that these requests are prioritized and dispositioned, these requests may get lost, and/or the scope of the project will be shifting continually, making it impossible to execute the project effectively. A change request process will help prevent scope creep.
Review the change-management process and ensure that it provides allowance for entering, ranking, and approving change requests. Verify that it covers changes to scope, schedule, budget, requirements, design, etc. (i.e., all major elements of the project). Ensure that it records adequate information regarding each change request, including description of the request, priority level, latest status, approval, and resolution information. Select a sample of change requests, and walk them through the process, ensuring that proper approvals were received prior to final resolution.
Project schedules are used to ensure that the project is on track, resources are being used effectively, and all tasks have been accounted for and scheduled.
Review the project schedule, and look for items such as a work breakdown, milestone dates, task dependencies, and the critical path. Look for evidence that the schedule is followed and kept up-to-date. Seek explanations for any significant deltas. Ensure that an escalation procedure exists for any significant schedule or resource overruns, and review evidence that the process has been used. One potential methodology for the project to accomplish this is to have strategic points in the life cycle where it goes through a tollgate process. During this process, the project team goes before a review panel and reports the status of the project, successes and issues, and progress versus the schedule and budget. This helps to identify struggles and runaway failures early.
Without these mechanisms, it is more likely that project budgets will be exceeded and that the appropriate levels of management will not be made aware. Management presumably has a cap on the amount of money it intends to spend on a specific project. If all relevant expenses are not being tracked, management will be unaware if that cap has been exceeded, taking away the opportunity for it to make a decision as to whether to continue.
Obtain a copy of the budget, and compare it with expenses to date. Seek explanations for any significant deltas. Ensure that it includes all costs associated with the project. Ensure that an escalation procedure exists for any significant cost overruns, and review evidence that the process has been used. See the tollgate description from the preceding step for a potential review methodology.
Except for some pure infrastructure projects, most projects are undertaken at the request of the business in order to meet a business need. If the key business stakeholders are not part of the overall leadership and approval structure for the project, the odds of the project getting off track from the business needs increase because information and decisions about the project will be handled by IT people, who may not have the perspective necessary to make all decisions. Conversely, IT personnel also should be part of the structure because they generally bring important knowledge and perspective regarding the elements of success for IT-related projects.
Obtain a copy of the project's leadership structure, and look for evidence that both business and IT leaders and stakeholders are represented.
Projects should not be initiated without approval from the appropriate members of management, who are authorized to allocate resources and funds to new projects.
Review evidence that the project went through the company's standard approval process. If no such process exists, review evidence that an appropriate member of management approved the project prior to startup. Look for evidence that alternative and cost-benefit analyses were performed. Ensure that cost-benefit analyses considered the cost of ongoing maintenance, an element that is often omitted erroneously.
Prior to the startup of an IT project, it is important for qualified technical architects, network personnel, database administrators, and other applicable IT experts to agree that the proposed concept will work in the company's environment. If these sorts of people are brought in early, it is likely that the technical experts can find a way to make the concept work. However, if they are brought in after key elements of the system have been developed or procured, it may be determined that the solution is not technically feasible, leading to costly rework or discontinuation of the project. Likewise, it is important to engage the legal team in order to ensure that regulatory requirements are considered.
Review evidence that appropriate technical and legal personnel were involved in the initial project proposal and that they agreed to the feasibility of the project.
Systems, software, and processes should be built based on the requirements of the end users. If those end-user requirements are not captured and agreed to by the customers, the product likely will not meet their needs, requiring rework and changes. In addition, there are certain standard IT elements that should be included in the requirements definition of any system. Customers may not be aware of these elements and therefore require guidance from the IT team.
Review project documentation for evidence that customer requirements were gathered. Ensure that all key stakeholders, including the project sponsors, were involved in this process. Look for evidence that the key stakeholders agreed to the final list of requirements.
Ensure that the requirements encompass standard IT elements such as
Execution frequency | Service-level agreements (e.g., availability, speed of response to problems) |
Response time | Interface requirements |
Security | Backup/recovery/restart requirements |
Distributed and centralized processing requirements (e.g., the location of the storage and processing in a multitier architecture) | Hardware requirements |
Data retention requirements | Capacity, including needs for future anticipated growth |
Requirements for output distribution | Fault tolerance and redundancy |
Screen definitions |
If this project is intended to replace an existing system, look for evidence that an analysis of the current system was performed in order to determine what is working well and what is not. The results of this analysis should be reflected in the requirements documentation (i.e., the requirements should call for the new system to do the things that work well in the old system and to improve on the things that don't). Error logs and backlog requests from the old system can aid in the effort of determining what is not working well.
There are usually multiple organizations in the IT environment that will be involved in supporting any new system, including network support, operating system support, database support, data center personnel, IT security, and the help desk. If these organizations are not involved in the project early, they may not be prepared to support the system once it is ready, and/or the system may not be in compliance with applicable standards and policies.
Review evidence that other affected IT organizations have been notified of the project and are part of the approval process (as it relates to their readiness to support it).
There are often more system requirements than can be encompassed in the project (or at least in the initial phase of the project). It is important that the most critical requirements be identified and implemented.
Look for evidence that the requirements were prioritized and that the key stakeholders approved the prioritization.
Internal controls are necessary to protect company systems and to ensure their integrity. It is much easier to build controls into new systems up front than to attempt to attach them postimplementation.
This step is referring to the execution of the early-involvement concepts discussed in Chapter 1. The auditor will need to determine what sorts of controls he or she would audit the system for postimplementation and ensure that those controls are being designed into the system. Applicable application controls and infrastructure controls should be considered. The other chapters in Part II provide most of the detail needed in performing those steps. While those chapters discuss the techniques for auditing systems, processes, and software postimplementation, the same information can be used for providing input as to what controls need to be built in during design. In addition, it might be appropriate to assign a financial/operational auditor to the project to ensure that the proper business controls are built into the system logic and workflows.
Purchasing a product from an outside vendor usually is a significant investment and represents a commitment to that vendor's product. If the process for selecting the vendor is inadequate or the contract does not provide the company with adequate protection, it can lead to the purchase of products that do not meet the requirements of the project and a lack of legal recourse.
Review the vendor selection process for elements such as
Ensure that products from multiple vendors are evaluated as to their ability to meet all project requirements and their compatibility with the company's IT environment.
Ensure that a cost analysis has been performed on the products being evaluated. This analysis should include all relevant costs, including product costs, one-time startup costs, licensing fees, and maintenance costs.
Determine whether the vendors' financial stability was investigated as part of the evaluation process.
Determine whether the vendors' experience with providing support for the product for similar companies in the industry was evaluated. This may include obtaining and interviewing references from companies that currently use the product.
Ensure that the vendors' technical support capabilities were considered and evaluated.
Once a vendor is chosen, ensure that the contract clearly identified deliverables, requirements, and responsibilities. The contract should specify how performance will be measured and penalties for nonperformance or delayed performance. It also should provide conditions for terminating the agreement.
Ensure that the contract contains a nondisclosure clause preventing the vendor from disclosing company information.
Ensure that the contract contains a "right to audit" clause allowing you to audit vendor activities that are critical to your company.
Where applicable, ensure that code is put in escrow to protect against the vendor going out of business.
A defined process for tracing requirements to the system design will provide assurance that all requirements are addressed, including such standard IT elements as interfaces, response time, and capacity.
If it exists, review the requirements trace map and verify that all requirements are represented and mapped to a design element. If a trace map doesn't exist, review the process for ensuring that all requirements are encompassed.
This is the document that will be used to actually design the system, software, or process. If key stakeholders have not signed off on it, there is a greater chance that the output of the project will not meet their needs.
Look for the equivalent of a detailed design document and for evidence of customer approval. Note that nontechnical personnel may not be in a position to understand the document, depending on how it is written. If this is the case, ensure that compensating design reviews have been developed that allow the stakeholders to understand the planned design elements.
There is always some fluidity to projects, with the initial set of requirements rarely ending up as the final set of requirements. If key stakeholders are not involved throughout the project, the project runs the risk of getting off-track from customer desires and of making decisions that are not in alignment with customer wishes.
Determine whether there is a direction-setting group containing key customers and whether they are involved in project decisions on a regular basis. Consider interviewing a small sample of customers to obtain their opinion on customer involvement. Look for evidence of periodic project review meetings and periodic communication with key stakeholders.
This quality-control discipline, which involves a review of code and configuration by the developer's peers, can help to increase the odds that the system will be designed with sound logic and a minimum of errors.
Determine whether peer reviews are required by the process, and look for evidence that they are actually occurring.
See step 15.
Validate (either through interviews or design reviews) that the input you provided in step 15 has been encompassed in the design of the system.