Secret source code pronounces you guilty as charged [Updated]

Updated, September 18 @ 13:15 EDT (18:15 BST): Mark Perlin, the chief scientist and executive officer of Cybergenetics, says in an e-mail and telephone interview with Ars that "source code is not used to assess forensic software reliability." (More below.)

The results from a Pennsylvania company's TrueAllele DNA testing software have been used in roughly 200 criminal cases, from California to Florida, helping put murderers and rapists in prison.

Criminal defense lawyers, however, want to know whether it's junk science.

Defense attorneys have routinely asked, and have been denied, access to examine the software's 170,000 lines of source code in a bid to challenge the authenticity of its conclusions. The courts generally have agreed with Cybergenetics, the company behind TrueAllele, that an independent examination of the code is unwarranted, that the code is a proprietary trade secret, and disclosing it could destroy the company financially.

A new challenge, pending before the California Supreme Court, concerns some of the company's latest conclusions. The results are evidence in a cold-case murder, yet they differ astronomically from traditional DNA testing. The dispute comes as secret code is creeping into our everyday life in what is known as the Internet of Things. It's in everything, from airplanes to refrigerators, medical devices, and even elevators, light fixtures, and cars.

Volkswagen duped federal pollution regulators for six years, having written proprietary code in its vehicles to allow them to pass emissions tests. All the while, the vehicles would perform better on the road and pollute substantially more than allowed by federal standards.

Secret code now has infiltrated the criminal justice system.

The latest challenge to it concerns a handyman and a convicted sex offender named Martell Chubbs, now accused of a 1977 Long Beach, California murder. Local police were investigating cold cases and arrested Chubbs after DNA taken from the crime scene long ago matched a sample in a national criminal database, the authorities said.

A private company called Sorenson Forensics, testing vaginal swabs from the victim, concluded that the frequency in the profile occurrence in the general population was one in approximately 10,000 for African Americans. The same sample, when examined by Cybergenetics at the company's Pittsburgh lab, concluded that the DNA match between the vaginal sperm sample and Chubbs is "1.62 quintillion times more probable than a coincidental match to an unrelated Black person," according to court records.

The defendant's attorney, Angelyn Gates, wants to examine the TrueAllele's code in a bid to learn how the software reached its conclusion. A trial judge tossed the findings because the company refused to divulge the information. A California appellate court reversed, and Gates appealed to the state Supreme Court. The justices have not decided whether it will rule on the issue.

"They're coming up with these astronomical, insane results," Gates said in a telephone interview.

She said that she has the right to confront the evidence being used against her client. "When you put data into computer and it spits out something, you'd like to know how it did it."

Cybergenetics said its practices are sound and peer reviewed. It told the California appellate court that it provides adequate literature and data to defense experts about how its system works. It discloses its "underlying mathematical model" to enable others to understand its genotype modeling mechanism. And, the company said, it "provides opposing experts the opportunity to review the TrueAllele process, examine results, and ask questions."

UPDATE: Speaking and writing to Ars, Perlin said:

The hallmark of science is empirical testing. Forensic science manufacturers and laboratories must validate their DNA technology. They measure false positive and negative rates, and other reliability metrics. Cybergenetics and others have thoroughly tested the TrueAllele® technology, as published in seven peer-reviewed validation studies.

Judges are the gatekeepers of reliable scientific evidence. TrueAllele has been used in most states, and challenged in six of them. In all six decisions, the courts found TrueAllele to be reliable, and admitted the DNA match results into evidence. TrueAllele is used for both the prosecution and defense.

Source code is not used to assess forensic software reliability. When reviewing a DNA interpretation method, I look at the math and examine empirical results on real data. If source code is available (e.g., a short academic program) I may test the software, but do not read the source code. Computer accuracy is relevant; software text is not.

The larger issue is unvalidated software. The FBI’s Popstats program calculates a combined probability of inclusion (CPI) match statistic. CPI has been used on hundreds of thousands of DNA mixtures, evidence that can implicate or exonerate. After a decade of scientific concerns about CPI’s validity, with recent meltdowns in Washington, DC and Texas, the FBI has yet to publish a rigorous validation study.

UPDATE ENDED.

The company told the court that it keeps its code secret because of the "highly competitive commercial environment" in which it operates.

The appellate court concluded in the Chubbs case that the "vague statements" made from his defense attorney Gates "do not describe in any way how the source code would have any bearing on the reliability of the analysis."

Rebecca Wexler, a visiting scholar at the Berkeley School of Law's Human Rights Center, recently wrote about the topic in Slate:

Coding errors (PDF) have been found to alter DNA likelihood ratios by a factor of 10, causing prosecutors in Australia to replace 24 expert witness statements in criminal cases. When defense experts identified a bug in breathalyzer software, the Minnesota Supreme Court barred the affected test from evidence in all future trials.

Cross-examination can help to protect against error—and even fraud—in forensic science and tech. But for that "legal engine" to work, defendants need to know the bases of state claims.

Chubbs, 54, has pleaded not guilty. He is accused of murdering a 17-year-old girl identified only as "Shelley H."