10.6 Closing Remarks

Data rule execution is different from the data profiling described in the last two chapters. For columns and structure, you are looking for inaccurate data. Any violations point to wrong data. For data rules, you may be looking at inaccurate data, or you may be looking at correct data that results from the business activity violating one of its business rules. The responsibility of the data is to record the truth. If an exception to a rule is made in the business, the data should reflect this.

Data rule analysis can lead to paralysis through analysis. An overly zealous analyst can dream up countless rules that can be tested. Some of these will prove to be worthwhile, and some will not. A good data profiling analyst can reach a balance between too many rules and not enough rules.

Data rule analysis is an area that is ignored too often. The problem is that data quality assessment is sometimes considered the role of business analysts. To be effective at rule definition, execution, and interpretation, you need to be an expert at SQL or some other rule language. These tend to be very programming oriented. Many business analysts do not have the education or experience to be effective at doing this. They try, it becomes too much of a challenge, and they give up.

The opposite is also true. Rules are, by their very nature, semantically based. This means that the business analyst is the only realistic expert in identifying and expressing them. The technical person who is adept at SQL in all likelihood does not understand the business side. If they are left alone, they will formulate some very obvious and not too important rules.

To make this work, you need a strong working partnership between multiple people who together possess all of the skills needed. Without it, not much rule testing will be done.

Failure to take this step seriously leaves a number of problems undiscovered. By now it should be apparent that each data profiling topic discovers different problems in the data. Leaving out a step will just reduce the effectiveness of the assessment.

The next chapter stays on the topic of data rules. It talks about types of rules that apply to sets of business objects instead of single business objects. These two topics were separated merely to make it easier to describe. Everything said in this chapter about data rules applies equally to the next chapter.