More Trainwreck Thoughts
Why did MD Anderson launch OEA, and why did it fail?
Previously, I wrote about MD Anderson’s failed Oncology Expert Advisor project (OEA). Here are a few additional thoughts on two key questions.
Why did MD Anderson launch OEA?
“Imagine a world without cancer.” MD Anderson marketed OEA as a humanitarian project: doctors in white coats healing toddlers from the developing world. This positioning was “healthwashing” – a deliberate intent to cultivate support by appealing to the public interest in science and health.
OEA was a commercial venture. MD Anderson’s contract with IBM included this provision:
If the pilot program becomes fully operational and a Production Statement of Work is executed, the institution will pay IBM a fee of approximately one-third of the value derived from using the Watson Service within the institution, with a minimum charge of $450,000 per month.
We don’t know how the contract defined “value,” but IBM wasn’t expecting payment in butterflies and rainbows. There are two possibilities: MD Anderson planned to offer OEA diagnostics as a subscription service, or OEA remote diagnostics would draw more patients to MD Anderson for treatment.
I doubt they planned a subscription service. The original scope of OEA narrowly targeted a single type of cancer: lower-risk myelodysplastic syndrome (MDS) leukemia. MDS affects older people, and Medicare covers supportive care only. It seems unlikely that “partner” hospitals would pay a subscription fee to target unreimbursable treatments.
There are about 15,000 new MDS cases annually in the United States, of which roughly 10,000 are low-risk. However, MD Anderson treats only 300 cases annually, likely due to the lack of insurance coverage. It seems odd that MD Anderson focused a modeling project on a disease it rarely treats.
MD Anderson hosts more clinical trials than any other cancer center. The pharmaceutical company sponsoring the trial usually pays for the treatment. Matching patients with clinical trials was an OEA core capability. If it worked, OEA would expand MD Anderson’s clinical trial services market.
To add some zest to this cocktail, MD Anderson President DePinho and his wife, Lynda Chin, co-founded two pharmaceutical companies: Aveo Oncology and Metamark Genetics. MD Anderson faculty complained about the potential conflict of interest:
DePinho’s performance on May 18th Stockwatch and the $$ millions that this couple gained personally due to this activity, Dr. Chin’s use of her time and institutional funds to support travel to Metamark and Aveo to oversee their businesses, the waivers filed to allow the Aveo drugs to be tested here,... it goes on and on.
Ironically, MD Anderson chose to target OEA at a disease that affects older adults. The winsome children from the developing world in the promotional video were props.
Why did OEA fail?
Project leader Lynda Chin spoke with STAT just before she left the University of Texas. She cited three “roadblocks” that she and her team encountered: messy medical records, difficult deployment in clinical practice, and insufficient data to draw insights that could “significantly advance the standard of care.”
Chin told the University of Texas audit team a different story. She said the team could not pilot OEA with external partners because of “a lack of engagement or interest.” Additionally, she cited issues with compliance and information security reviews.
That’s five reasons. Let’s examine them.
Messy medical records. Everyone in healthcare knows this is a problem, especially with legacy systems. Eighteen months after starting work and after spending $2.5 million in consulting fees, the OEA team asked the Board of Regents for another $14 million for a scope expansion:
The initial focus of the pilot program was to identify insights related to a particular group of leukemia patients. This proposed change order/project change request expands the pilot program to include five additional types of leukemia.
By this point, the OEA team had spent enough time with the data to understand the level of effort. If the challenge were unsurmountable, there would be no reason to expand the scope. Data wrangling may have slowed progress, but it did not kill the project.
Difficult deployment. According to STAT, Chin said her team also wrestled with deploying the system in clinical practice:
Watson, even if guided by doctors, is as close as medicine has ever gotten to allowing a machine to help decide the treatments delivered to human beings. That carries with it thorny questions, such as how to test the safety of a digital treatment adviser, how to ensure its compliance with regulations, and how to incorporate it into the daily work of doctors and nurses.
Those are legitimate questions. But since OEA never made it past prototyping, Chin’s team only wrestled with them conceptually, like Sartre wrestling with the problem of resistance while sipping apricot cocktails at the Bec de Gaz. They are also the questions one asks before investing in a tool for clinical practice.
Insufficient data. The showstopper, according to Chin, was insufficient data:
Finally, the project ran into a bigger obstacle: Even if you can get Watson to understand patient variables and make competent treatment recommendations, how do you get it access to enough patient data, from enough different sources, to derive insights that could significantly advance the standard of care?
Watson did not have a connected network of institutions feeding data about specific cohorts of patients.
Chin is right – you are much more likely to build a successful diagnostic model if you have data for all cancer patients, not just those from one medical center. But the OEA team knew they did not have this. If “insufficient data” was the showstopper, MD Anderson shouldn’t have started the project.
Lack of interest from partners. It’s a serious problem when nobody wants your product. Smart people analyze customer needs before they invest in a large project. That means building PowerPoint slides and speaking with prospective users before rolling in the supercomputer.
OEA failed because nobody wanted it, which raises the question of how the team spent four years and $62 million on a vanity project.
Compliance and security reviews. Chin told the auditors that “the time needed for compliance and information security reviews of the cloud-based data repository” prevented the OEA team from running pilot projects with partners. Note that this explanation conflicts with the claim that partners weren’t interested. You won't get to a security review if a potential partner isn’t interested in your stuff.
But let’s suppose that one or more partners were sufficiently interested in OEA to commence a security review. The Board of Regents approved the external phase of the project in February 2014. IBM pulled the plug in September 2016, so the OEA team had 30 months to complete security reviews.
It does not take that long to do a security review. It’s more likely that OEA failed security reviews because prospective partners had to upload their HIPAA-protected health records to MD Anderson’s Watson instance in IBM Cloud, and there was no fucking way they were going to do that.
The OEA project failed due to poor leadership. Let’s enumerate the management issues:
Improper leadership roles. Lynda Chin is a highly experienced and credentialed medical researcher. However, her resume shows no experience or training in artificial intelligence or IT project management. Her appropriate role in OEA was executive sponsorship, with no hands-on responsibilities.
Incompetent project management. The project badly needed a hands-on manager steeped in University of Texas IT policies with the power to say “no.”
Failure to assess user interest. Given the initial concept of a diagnostic engine, the project team should have first invested in a demand assessment. The team spent more than enough time and money on this task.
Failure to assess project risks. Risk assessments are essential for a project of this size, but I do not believe the OEA team performed one. Any reasonable assessment would have identified the potential challenge of working with medical records.
Successful projects choose the best available technology to address a well-defined problem. MD Anderson seems to have put the cart before the horse: They started with IBM Watson and tried to figure out what to do with it.
The dodgy donor provides comic relief in this story but is not the root cause of failure. The allegations about Low Taek Jho surfaced after the project collapsed. However, the donor’s pledge of $50 million may have given the OEA team false assurance that they did not have to worry about budget approvals.