Plugging Holes in the Science of Forensics

By HENRY FOUNTAIN
It was time, the panel of experts said, to put more science in forensic science.

A report in February by a committee of the National Academy of Sciences found “serious problems” with much of the work performed by crime laboratories in the United States. Recent incidents of faulty evidence analysis — including the case of an Oregon lawyer who was arrested by the F.B.I. after the 2004 Madrid terrorist bombings based on fingerprint identification that turned out to be wrong — were just high-profile examples of wider deficiencies, the committee said. Crime labs were overworked, there were few certification programs for investigators and technicians, and the entire field suffered from a lack of oversight.

But perhaps the most damning conclusion was that many forensic disciplines — including analysis of fingerprints, bite marks and the striations and indentations left by a pry bar or a gun’s firing mechanism — were not grounded in the kind of rigorous, peer-reviewed research that is the hallmark of classic science. DNA analysis was an exception, the report noted, in that it had been studied extensively. But many other investigative tests, the report said, “have never been exposed to stringent scientific scrutiny.”

While some forensic experts took issue with that conclusion, many welcomed it. And some scientists are working on just the kind of research necessary to improve the field. They are refining software and studying human decision-making to improve an important aspect of much forensic science — the ability to recognize and compare patterns.

The report was “basically saying what many of us have been saying for a long time,” said Lawrence Kobilinsky, chairman of the department of sciences at John Jay College of Criminal Justice in New York. “There are a lot of areas in forensics that need improvement.”
Barry Fisher, a past president of the American Academy of Forensic Sciences and a former director of the crime laboratory at the Los Angeles County Sheriff’s Department, said he and others had been pushing for this kind of independent assessment for years. “There needs to be a demonstration that this stuff is reliable,” he said.

It’s not that there hasn’t been any research in forensic science. But over the years much of it has been done in crime labs themselves. “It hasn’t gotten to the level where they can state findings in a rigorous scientific way,” said Constantine Gatsonis, director of the Center for Statistical Sciences at Brown University and co-chairman of the National Academy of Sciences committee. And rather than being teased out in academic papers and debated at scientific conferences, “a lot of this forensic stuff is being argued in the courtroom,” Mr. Fisher said. “That’s not the place to validate any kind of scientific information.”

Much forensic research has been geared to improving technologies and techniques. These studies can result in the kinds of gee-whiz advances that may show up in the next episode of the “C.S.I.” series — a technique to obtain fingerprints from a grocery bag or other unlikely source, for example, or equipment that enables analyses of the tiniest bits of evidence.

This kind of work is useful, Dr. Kobilinsky said, “but it doesn’t solve the basic problem.”

DNA analysis came out of the biological sciences, and much money and time has been spent developing the field, resulting in a large body of peer-reviewed research. So when a DNA expert testifies in court that there is a certain probability that a sample comes from a suspect, that claim is grounded in science.

As evidence to be analyzed, DNA has certain advantages. “DNA has a particular structure, and can be digitized,” Dr. Gatsonis said. So scientists can agree, for example, on how many loci on a DNA strand to use in their analyses, and computers can do the necessary computations of probability.

“Fingerprints are a lot more complicated,” Dr. Gatsonis said. “There are a lot of different ways you can select features and make comparisons.” A smudged print may have only a few ridge endings or other points for comparison, while a clear print may have many more. And other factors can affect prints, including the material they were found on and the pressure of the fingers in making them.

Sargur N. Srihari, an expert in pattern recognition at the University at Buffalo, part of the New York state university system, is trying to quantify the uncertainty. His group did much of the research that led to postal systems that can recognize handwritten addresses on envelopes, and he works with databases of fingerprints to derive probabilities of random correspondence between two prints.

Most features on a print are usually represented by X and Y coordinates and by an angle that represents the orientation of the particular ridge where the feature is located. A single print can have 40 or more comparable features.

Dr. Srihari uses relatively small databases, including an extreme one that contains fingerprints from dozens of identical twins (so the probability of matches is high), and employs the results to further refine mathematical tools for comparison that would work with larger populations.

“These numbers are not easy to come by at this point,” he said. The goal is not individualization — matching two prints with absolute certainty — but coming up with firm probabilities that would be very useful in legal proceedings.

Other researchers are compiling databases of their own. Nicholas D. K. Petraco, an assistant professor at John Jay College, is studying microscopic tool marks of the kind made by a screwdriver when a burglar jimmies a window. It has been hypothesized that no two screwdrivers leave exactly the same pattern of marks, although that has never been proved. So Dr. Petraco is systematically making marks in jeweler’s wax and other materials, creating images of them under a stereo microscope and quantifying the details, assembling a database that can eventually be mined to determine probabilities that a mark matches a certain tool.

Dr. Petraco, a chemist with a strong background in computer science, looks to industry for ideas about pattern recognition — the tools that a company like Netflix uses, for example, to classify people by the kinds of movies they like. “A lot of computational machinery goes into making those kinds of decisions,” he said.

He figures that if something works for industry, it will work for forensic science. “You don’t want to invent anything new,” he said, because that raises legal issues of admissibility of evidence.

The work takes time, but the good news is that the data stays around forever. So as software improves, the probabilities should get more accurate. “Algorithms and data comparison evolve over time,” Dr. Petraco said.

But it may not be possible to develop useful databases in some disciplines — bite mark analysis, for example. “Using a screwdriver, that’s very straightforward and simple,” said Ira Titunik, a forensic odontologist and adjunct professor at John Jay College. But bites involve numerous teeth, and there are other factors, including condition of the skin, that may make it difficult to quantify them for purposes of determining probabilities.

A few researchers are looking at how errors creep into forensic analysis. The National Institute of Standards and Technology recently established a working group on fingerprints, with statisticians, psychologists and others, “to try to understand the circumstances that lead to human error,” said Mark Stolorow, director of the Office of Law Enforcement Standards at the institute.

In Britain, Itiel Dror, a psychologist who studies decision-making processes, is already looking at human factors. “I like to say the mind is not a camera, objectively and passively recording information,” said Dr. Dror, who has a consulting firm and is affiliated with University College London. “The brain is an active and dynamic device.”

He has conducted studies that show that when working on an identification, fingerprint examiners can be influenced by what else they know about a case. In one experiment, he found that the same examiner can come to different conclusions about the same fingerprint, if the context is changed over time.

The same kinds of contextual biases arise with other decision-makers, said Dr. Dror, who works with the military and with financial and medical professionals. He thinks one reason forensic examiners often do not acknowledge that they make errors is that in these other fields, the mistakes are obvious. “In forensics, they don’t really see it,” he said. “People go to jail.”

Forensics experts say the need for research like Dr. Dror’s and Dr. Srihari’s does not mean that disciplines like fingerprint analysis will turn out to be invalid. “I have no doubt that fingerprint evidence and firearms evidence, once looked into by the appropriate research entities, are going to be shown to be very reliable and good,” said Mr. Fisher, the former American Academy of Forensic Sciences president.

Dr. Kobilinsky said people should not jump to the conclusion that forensic science is bad science. “There’s a lot of experience and knowledge that goes into somebody’s expertise,” he said.

“It’s not junk science. But that doesn’t mean it shouldn’t be improved.”