Lessons from an AI Trainwreck
How to lose $62 million and not spill your drink.
This story is an old story, but I love it anyway. It has everything: arrogant and incompetent leaders, vendor hype, and corruption. An international fugitive is the icing on the cake.
On February 17, 2017, the Houston Chronicle reported that the University of Texas MD Anderson Cancer Center halted work on an AI project for cancer diagnostics. The story revealed that MD Anderson spent four years and $62 million to build a system called the Oncology Expert Advisor (OEA) Powered By IBM Watson. As its champions envisioned, OEA would help community oncologists provide quality care to patients unable to seek treatment directly from MD Anderson physicians.
MD Anderson President Ronald DePinho resigned two weeks later. Reports mentioned the failed OEA project as one reason for his departure.
A dogpile of stories about OEA ensued in Forbes, The Wall Street Journal, MIT Technology Review, Medscape, The Cancer Letter, Health News Review, Health IT and CIO Review, ArsTechnica, and many others. STAT and IEEE Spectrum published excellent post-mortems. Harvard Business School published a business case.
Most reports treated the project as a technical failure, like a rocket that blew up on the launchpad. I beg to differ. There were technical challenges, but that’s not why MD Anderson lit a bonfire with $62 million. The Oncology Expert Advisor was a perfect storm of incompetence, vanity, and mendacity.
Here are some lessons learned for AI projects.
Build your business case first.
MD Anderson desperately needed customers to help pay for OEA. Under a 2012 contract, MD Anderson agreed to pay IBM a minimum of $5.4 million annually if the system went into production.
IBM and MD Anderson completed a pilot project for OEA in October 2013. Then, they spent the next three years hunting for potential external users. They didn’t find any. Nobody wanted to pay for a robot that could diagnose six types of leukemia, and prospects also balked at uploading their medical records to MD Anderson’s Watson cloud platform.
It should have been obvious that prospective users would be scarce. Anyone with the slightest experience in healthcare knows that doctors want control over diagnosis, and hospitals want control over their medical records.
MD Anderson's leadership was so enthralled with their brainchild that they never stopped to ask basic questions. Who wants this? Who will use it? Will they pay for it?
Eschew nepotism.
The project leader for OEA, Dr. Lynda Chin, is a distinguished researcher in genomics. Among other accomplishments, she has held positions at the Harvard Medical School, the Dana Farber Cancer Institute, and the Broad Institute of MIT and Harvard. She was the Founding Chair of the Department of Genomic Medicine at MD Anderson Cancer Center and the Associate Vice Chancellor and Chief Innovation Officer for the University of Texas System.
She is also the spouse of Ronald DePinho, the former President of MD Anderson.
Chin was no stranger to controversy at MD Anderson. In 2012, she and DePinho caused an uproar when the Cancer Prevention and Research Institute of Texas (CPRIT) awarded MD Anderson a $20 million grant after bypassing standard review. Nobel laureate Alfred Gilman, CPRIT’s chief scientific officer, subsequently quit. So did CPRIT’s peer reviewers, who included premier cancer scientists across the U.S.
In May 2013, The Cancer Letter reported that Chin spent almost $2 million on a 25,000-square-foot office suite. DePinho had recently announced austerity measures, including suspending merit raises, slowing recruitment, and suspending capital projects.
In an internal survey, one faculty member wrote: If we don’t align with their agenda, we will be fired, in the words of Dr. Chin. That might explain why pragmatists were unwilling to say, “This OEA project is nuts.”
Don’t fuck around with budgets and contracts.
An audit report published in November 2016 by the University of Texas Audit Office identified numerous exceptions to University procurement policies in the OEA project. Project leadership did not use proper contracting and procurement procedures, failed to follow IT Governance processes for project approval, did not effectively monitor vendor contract delivery, and overspent pledged donor funds.
MD Anderson executed seven agreements and eight amendments with IBM and PwC. The auditors noted that the contract value for many of these agreements was just below the Board approval threshold, suggesting deliberate “structuring” to avoid scrutiny. The cancer center awarded contracts to IBM and PwC under non-competitive procurement. Moreover, per the audit report, MD Anderson appears to have paid IBM and PwC for work they did not do.
In response to the audit report, project leader Lynda Chin claimed she was not required to follow IT Governance policies because the effort was a “research” project. Every research institution has rules for contracts and budgets, and violations are a fireable offense. Harvard Medical School, the Dana Farber Cancer Institute, and the Broad Institute all have rules for budgets and contracts. It seems disingenuous for Chin to argue that her project could ignore institutional rules.
Do not make false claims.
In October 2013, MD Anderson and IBM announced OEA with a press release and a promotional video. The press release said that MD Anderson “is using” IBM Watson. The video footage showed white-coated doctors using a computer while a narrator described OEA as a working system.
Subsequent Scientific American, Business Insider, and Technology Review stories claimed that OEA was already helping oncologists. According to the stories, OEA could ingest the patient’s medical records and pair that information with knowledge from medical journals, textbooks, and treatment guidelines.
IBM Watson Health CTO Rob High wrote, "Doctors at the MD Anderson Cancer Center in Houston are using Watson to drive a software tool called the Oncology Expert Advisor, which serves as both a live reference manual and a virtual expert advisor for practicing clinicians.”
All of these claims were false. OEA was not in production and never went into production. When IBM abandoned the project in September 2016, the company declared OEA “not ready for human investigational or clinical use, and its use in the treatment of patients is prohibited.” The false claims about OEA misled the public and potential partners and damaged the credibility of MD Anderson and IBM.
MD Anderson’s OEA video showed a vast data center while the narrator spoke about synthesizing vast amounts of data. In reality, the OEA team worked with tiny datasets. In a 2014 paper, the OEA team said they used 400 patient cases for training and 200 for testing. In another paper published in 2018, they said they worked with 848 cases: 585 for model development, 175 for training, and 88 for validation testing.
You could run that shit on a laptop.
Hire people who know what they are doing.
Quoted in STAT, Lynda Chin said that she and her team struggled to get Watson to deal with the idiosyncrasies of medical records: the acronyms, human errors, shorthand phrases, and different writing styles. “Teaching a machine to read a health record is much harder than anyone thought,” she said.
Health records are messy, who knew? A competent project team would know this and plan accordingly.
Do not build white elephants.
MD Anderson integrated OEA with ClinicStation, a homegrown medical records system. That’s nice. Except that MD Anderson replaced the ClinicStation system with Epic Systems in March 2016.
Citing the cost of maintaining ClinicStation, MD Anderson began searching for a new EHR system in late 2012 and announced its selection of Epic in early 2013. In November 2013, the Board of Regents approved a contract with Epic Systems and appropriated $60 million to cover software licensing, implementation, maintenance, and support services.
OEA was a white elephant from the moment the Board approved the contract with Epic Systems. It could not go into production unless MD Anderson integrated it with Epic EHR. That would mean a complete rebuild of the system’s back end.
In a February 2014 Board of Regents meeting, MD Anderson President DePinho claimed that OEA could ingest data from the million patients treated at the center since 1944. None of the Regents asked about the impact of switching from ClinicStation to Epic EHR, a strategic move the Board had approved three months earlier.
They don’t hire Regents for smarts.
Epic implementations do not go unnoticed; they are massive projects that affect everyone in the hospital. The OEA team knew their main data source was going away. Epic schema would have been available internally long before the go-live date. Why not build an interface?
I think the OEA team knew the project was dead long before IBM pulled the plug.
Do not trust dodgy donors.
In February 2012, Mr. Low Taek Jho thought he had cancer. He sought treatment at MD Anderson Cancer Center.
Doctors at the Center diagnosed an infection and ruled out cancer. Jho, a Malaysian-born businessman, was so pleased that he donated $1 million.
The following November, Jho flew his grandfather to MD Anderson for leukemia treatment. That inspired him to pledge $50 million for the Oncology Expert Advisor project.
What a heartwarming story.
In October 2016, Interpol published a red notice for Jho. Singapore wanted him to question him about his role in the 1MDB scandal, a conspiracy of corruption, bribery, and money laundering that looted the Malaysian sovereign wealth fund.
In civil forfeiture proceedings, the United States Department of Justice alleged that Jho diverted funds to his accounts. The DOJ said he used the money to buy a private jet, a superyacht, hotels, and artwork by Picasso and Monet.
And a robot that diagnoses cancer, or so it appears.
Jho recently settled the civil asset forfeiture claims with the U.S. government. He denies the criminal allegations and remains an international fugitive.
IBM pulled the plug on Oncology Expert Advisor in September 2016, around the same time the allegations against Jho surfaced. I’m sure that was pure coincidence.
MD Anderson has never disclosed how much of Jho’s pledge it collected. In fiscal year 2017, the cancer center deprecated Oncology Expert Advisor and wrote off the entire investment.
Postlude: did Watson work?
MD Anderson's medical oncology staff told auditors that OEA's internal pilot testing achieved a prediction accuracy of “near 90 percent.” This seems roughly consistent with this paper's finding of 82.6% prediction accuracy. For OEA, prediction accuracy means that the system recommended a treatment consistent with human doctors' recommendations.
In a separate paper published in The Oncologist, researchers wrote that OEA recommended approved therapy options linked to supporting evidence (99.9% recall; 88% precision) and screens for eligible clinical trials on ClinicalTrials.gov (97.9% recall; 96.9% precision).
IBM used these results to claim that Watson worked. But did it?
MD Anderson treats 175,000 unique patients every year. For that second paper, the researchers narrowed the population to focus on a class of patients who sought treatment over a two-year period. Let’s say the total population was 300,000 patients. The researchers found 848 cases from that pool they could use for model development, testing, and validation.
You can get remarkable results with a model when you cherry-pick your cases.
Suppose you’re the Chief Medical Officer at Giant Metro Teaching Hospital. Someone calls and offers a service that can diagnose 0.2% of your cancer patients. It will cost millions, and you must upload your medical records.
You tell them to get lost.
What an incredible story, Thomas. I had no idea so much crazy stuff was happening behind the curtains at MD Anderson. Fraud always follows the same pattern. In my investigation of Hippocratic AI, which I’m wrapping up (https://open.substack.com/pub/sergeiai/p/hypocritical-ai-the-fast-track-to) - part 2 should be out on Monday - I found that the culprits are the usual suspects: marketing hype, corruption, incompetent leadership, toxic corporate culture, and the tight-knit circle feeding at the VC money trough.