Section 8.7. Postmortem Reports

8.7. Postmortem Reports

After the project is done, there is a lot of important information out in the organization that can help a project manager run future projects better. There is a brief period after the software is released to the users when everyone is done with the tasks, but when they still have a lot of strong opinions and remember specific details about how the project went. If this information is documented in a way that can be used to help in the future, it will be a valuable resource for the organization.

A postmortem report is an overall account of the team's experience in building the software, and of the experience of the users and stakeholders in working with the team. The report should contain an honest assessment of how the team members, users, and stakeholders perceived the end product and assessed the decisions made throughout the project. The purpose of the postmortem report is to highlight the team's successes and identify any problems that should be fixed in future releases.

Often, a project manager will gather the project team together with anyone who had anything to do with the software project into a large meeting so they can talk about the project. This is rarely effectiveit just turns into a big, useless meeting (see Chapter 5). In the same way that people are uncomfortable criticizing each others' work or taking that criticism for individual work products, they can be equally uncomfortable doing it for the entire project. This is especially true when it comes to taking criticism about the project from people outside of the team.

A much more effective way to gather this information is to rely on the quality assurance team members, who have expertise gathering and objectively reporting both positive and negative information about the project. In the same way that they assess the health of a single build and write a report that describes it, they can assess the health of the entire project and write a postmortem report that describes the results of that assessment. The project manager can then gather the team, users, and stakeholders together to review the results.

One effective way to gather information for the postmortem report is to use a survey. It is important that this survey covers every phase of software development, so that nobody is left out or singled out. The survey should be constructed so that participants can respond anonymously. Respondents should be asked for both a numeric rating and for their thoughts and comments. Table 8-7 shows some typical questions that might appear on a postmortem survey.

Table 8-7. Sample postmortem survey questions

The "Accounting Rules Automation" feature is complete.
(strongly agree)	5	4	3	2	1	(strongly disagree)
Do you have additional comments about this?
The "Accounting Rules Automation" feature is useful.
(strongly agree)	5	4	3	2	1	(strongly disagree)
Do you have additional comments about this?
The Vision and Scope document said that the software was intended to address the business need that our client team was missing critical assessment numbers. How well did the software project fill this need?
(very well)	5	4	3	2	1	(poorly)
Do you have additional comments about this?

The survey should gather a wide range of information about how well the team performed their function, features of the software that was built, the effectiveness of specific work products and activities, and how well the team met the objectives of the project and of each phase of development. Survey questions typically cover:

• Were the tasks divided well among the team?

• Were the right people assigned to each task?

• Were the reviews effective?

• Was each work product useful in the later phases of the project?

• Did the software meet the needs described in the vision and scope document?

• Did the stakeholders and users have enough input into the software?

• Were there too many changes?

• How complete is each feature?

• How useful is each feature?

• Have the users received the software?

• How is the user experience with the software?

• Are there usability or performance issues?

• Are there problems installing or configuring the software?

• Were the initial deadlines set for the project reasonable?

• How well was the overall project planned?

• Were there risks that could have been foreseen but were not planned for?

• Was the software produced in a timely manner?

• Was the software of sufficient quality?

• Do you have any suggestions for how we can improve for our next project?

Once the survey results are collected, they should be summarized and presented in the postmortem report. The report should focus on the overall results rather than individual responses. Figure 8-1 shows a typical summary section that would appear at the top of a postmortem report. In this example, the questions in the survey were grouped into six categories, and the results were divided into positive, neutral, or negative ratings.

Figure 8-1. Sample summary section from a postmortem report

Each of the remaining sections in the report shows details about the response for each individual question, followed by final sections that offer recommendations and specific actions to take. (The actual postmortem survey should be attached to the report as an appendix.) Each section should summarize the results of one question in the survey and present any comments given (without any identifying factors). It is important to preserve the respondents' anonymity so that they can feel comfortable giving their uncensored opinions. If several respondents had similar or related comments, those comments should be combined into one single statement that summarizes all of them. Not every comment will be constructive; the QA engineer preparing the report should use judgement to decide which comments are relevant. Table 8-8 shows an example of a section of the report that summarizes one question.

Beta

Was the beta test effective in heading off problems before clients found them?

Score: 2.28 out of 5 (12 Negative [1 to 2], 13 Neutral [3], 9 Positive [4 to 5])

All of the comments we got about the beta were negative, and only 26% (9 of 34) of the survey respondents felt that the beta exceeded their expectations. The general perception was that many obvious defects were not caught in the beta. Suggestions for improvement included lengthening the beta, expanding it to more client sites, and ensuring that the software was used as if it were in production.

Individual comments:

• I feel like Versions 2.0 and 2.1 could have been in the beta field longer so that we might have discovered the accounting bugs before many of the clients did.

• We need to have a more in-depth beta test in the future. Had the duration of the beta been longer, we would have caught more problems and headed them off before they became critical situations at the client site.

• I think that a lot of problems that were encountered were found after the beta, during the actual start of the release. Shortly thereafter, things were ironed out.

• Overall, the release has gone well. I just feel that we missed something in the beta test, particularly the performance issues we are experiencing in our Denver and Chicago branches. In the future, we can expand the beta to more sites.

Once the postmortem report has been compiled, the project manager should call a meeting to review it. The attendees should include project team members, stakeholders, users, and any other people who were asked to respond to the survey. The meeting should be a walkthrough (see Chapter 5) of the report. The project manager should write down suggestions or comments that are brought up during the meeting.

The final section of the report is an action list, which should be added after the postmortem meeting. Throughout the review, there will be many recommendations suggested by stakeholders, team members, and other people. The project manager should work with everyone at the meeting to identify specific actions that can be taken to make sure those recommendations are implemented. If possible, he should gain a real commitment to those actions, if the right people are there; if not, he should commit to following up on each of them, in order to make sure that the next project is performed better. By writing down the list of specific actions to take and including the commitments that people make during the meeting, the project manager can ensure that those changes are incorporated into future projects.