7.3 Casey s Certainty Scale

7.3 Casey's Certainty Scale

Computers can introduce errors and uncertainty in various ways, making it difficult to assess the trustworthiness of digital evidence meaningfully. Although courts are warned to consider the computer systems involved carefully, little guidance is provided.

Business records that are generated by computers present structural questions of reliability that transcend the reliability of the underlying information that is entered into the computer. Computer machinery may make errors because of malfunctioning of hardware, the computer's mechanical apparatus. Computers may also make errors that arise out of defects in the software, the input procedures, the database, and the processing program. In view of the complex nature of the operation of computers, courts have been cautioned to take special care to be certain that the foundation is sufficient to warrant a finding of trustworthiness and that the opposing party has full opportunity to inquire into the process by which information is fed into the computer. (American Oil Co. v. Valenti 1979).

Computer networks complicate reliability considerations because multiple systems and mechanisms are involved. Possibly because of the complexity and multiplicity of computer systems, there is a lack of consistency in the way that the reliability of digital evidence is assessed. To improve our ability to assess the reliability of digital evidence, we need a consistent method of referring to the relative certainty of different types of digital evidence. The scale in Table 7.1 is proposed when attempting to assess the probative value of digital evidence (Casey 2002).

Table 7.1: A proposed scale for categorizing levels of certainty in digital evidence.

CERTAINTY LEVEL	DESCRIPTION/INDICATORS	COMMENSURATE QUALIFICATION	EXAMPLES
C0	Evidence contradicts known facts	Erroneous/incorrect	Examiners found a vulnerability in Internet Explorer (IE) that allowed scripts on a particular Web site to create questionable files, desktop shortcuts, and IE favorites. The suspect did not purposefully create these items on the system
C1	Evidence is highly questionable	Highly uncertain	Missing entries from log files or signs of tampering
C2	Only one source of evidence that is not protected against tampering	Somewhat uncertain	E-mail headers, sulog entries, and syslog with no other supporting evidence
C3	The source(s) of evidence are more difficult to tamper with but there is not enough evidence to support a firm conclusion or there are unexplained inconsistencies in the available evidence	Possible	An intrusion came from Poland suggesting that the intruder might be from that area However, a later connection came from South Korea suggesting that the intruder might be elsewhere or that there is more than one intruder
C4	(a) Evidence is protected against tampering or (b) evidence is not protected against tampering but multiple, independent sources of evidence agree	Probable	Web server defacement probably originated from a given apartment since tcpwrapper logs show FTP connections from the apartment at the time of the defacement and Web server access logs show the page being accessed from the apartment shortly after the defacement
C5	Agreement of evidence from multiple, independent sources that are protected against tampering. However, small uncertainties exist (e.g. temporal error, data loss)	Almost certain	IP address, user account, and ANI information lead to suspect's home. Monitoring Internet traffic indicates that criminal activity is coming from the house
C6	The evidence is tamper proof and unquestionable	Certain	Although this is inconceivable at the moment, such sources of digital evidence may exist in the future

The certainly values (C-values) in Table 7.1 provide a method for a digital evidence examiner to denote the level of certainty he/she has in a given piece of evidence in a given context. This scale is not intended to be used rigidly to categorize types of evidence in general - it is not valid to claim that all NT Event logs have C3 certainly level because in some cases there may be signs of tampering such as deleted log entries, reducing the certainly level of the log to C1. The primary purpose of this Certainty Scale is to help others understand how much weight an examiner has given pieces of digital evidence when making a conclusion based on that evidence. Without these C-values, one might wonder how a digital evidence examiner reached his/her conclusion, particularly if there is disagreement over the certainty assigned to a given piece of evidence. For instance, two digital evidence examiners might make the following conclusions about the same case:

1. Log entries from System 2 indicate that Suspect B was logged in at the time of the crime and is almost certainly the offender.

2. The wtmp log on trusted System 1 (C4) indicates that the offender logged in from System 2. The wtmp log on untrusted System 2 (C2) indicates that two potential suspects were logged in at the time of the crime. However, RADIUS logs (C4) relating to Suspect A's PPP connection show that she disconnected from the Internet long before the crime, indicating that the associated wtmp entry on untrusted System B was not terminated properly, probably due to an abrupt disconnection on her part. Therefore, only Suspect B was logged onto System 2 at the time of the crime. The pacct logs on System 2 (C4) show that Suspect B was using Secure Shell (SSH) at the time of the crime. Although the pacct entry does not indicate which system Suspect B was connecting to using SSH, an examination of his command history (C2) shows that he was connecting to System 1. Based on this evidence, it is probable that Suspect B is the offender.^[1]

It is difficult to assess the validity of the first conclusion because the examiner does not explicate his thought process. Conversely, the though process leading to the second conclusion is clear and easier to access. For instance, another digital evidence examiner might argue that the wtmp log on System 2 is highly questionable (C1) given the erroneous entry associated with Suspect B's logon and the fact that several individuals, including both suspects, had root access to the machine and could have modified the logs. Similarly, it can be argued that anyone with root access to System 2 could have altered the pacct logs, reducing their C-value to C2. Based on these revised certainty values, it is possible (not probable) that Suspect B is the offender but a more reliable source of digital evidence is required to be more certain because any of the (preferably few) people with root access to System 2 could have altered the wtmp, pacct, and command history logs after the crime to implicate Suspect B.

Notably, these certainty values are not simply additive - the circumstances of a case, the questions at issue, and the types of digital evidence involved will determine how much weight each C-value is given and how they are combined. Digital evidence examiners must use their judgment when weighing and combining certainty values.

One major advantage of this Certainty Scale is that it is flexible enough to assess the evidential weight of both the process that generated a piece of digital evidence and its contents, which may be documents or statements. For instance, an e-mail header may be assigned a C-value of C2 in a specific case but the contents may only be assigned a C-value of C1 because there are signs of tampering. In another case, the C-value of an e-mail header may drop to C1 if any inconsistencies or signs of forgery are detected.

Another major advantage of this Certainty Scale is that it is non-technical and therefore easily understood by non-technical people such as those found in most juries. Although it may be necessary at some stage to ask the court to consider the complexities of the systems involved, it is invaluable to give them a general sense of the level of certainty they are dealing with and to help them decide what evidential weight to give the evidence. Only focusing on the complexities, without providing a non-technical overview, can lead to confusion and poor decisions.

Ultimately, it is hoped that this Certainty Scale will point to areas that require additional attention in digital evidence research. Debate over C-values in specific cases may reveal that certain types of evidence are less reliable than was initially assumed. For some types of digital evidence, it may be possible to identify the main sources of error or uncertainty and develop analysis techniques for evaluating or reducing these influences. For other types of digital evidence, it may be possible to identify all potential sources of error or uncertainty and develop a more formal model for calculating the level of certainty for this type of evidence.

^[1]Observe that the use of the word "probable" here corresponds to the C4 level in the certainty scale