Notes on Gartner's DSML MQ (3)
Vendor Assessments
This post is Part Three of a three-part series. Part One introduced the MQ and covered Gartner’s Critical Capabilities for Data Science and Machine Learning Platforms. Part Two covered Gartner’s methodology and scoring approach.
Part Three covers Gartner’s assessments of individual vendors. It’s a 14-minute read, which is a bit long. I’ve added section headers for each vendor. You can read the whole thing or skip to your favorite vendor.
Big Cloud
In 2021, Gartner positioned Amazon Web Services, Google, and Microsoft as Visionaries and Alibaba Cloud as a Niche Player. In the latest report, Gartner gave all four much higher scores on Ability to Execute. This increase is partly due to improved product and customer adoption and reflects Gartner’s scoring change.
AWS, Microsoft, and Google rank first, third, and fifth in total raw product score. Gartner’s IT clients will be thrilled to see the cloud service providers (CSPs) scored as Leaders. When IT leaders choose a CSP, they load up on credits. Then, they try to persuade everyone to use the CSP’s tools so they don’t look stupid at the end of the year for buying all that capacity.
Amazon Web Services
The SageMaker suite checks off more feature boxes in Gartner’s RFI than any other vendor. One small problem: data scientists still cringe when asked to use SageMaker. Even in SageMaker “reference” customers, AWS struggles to get data scientists to use the product. SageMaker feels like every module has a separate development team, and they all hate one another.
Gartner says AWS focuses more development efforts on operational machine learning at the expense of analytics for decision support and insight. No shit, Sherlock — this is obvious to anyone who spends two minutes evaluating SageMaker. According to Gartner, customers must supplement SageMaker with other services for decision support.
Think about that. AWS gets the highest product score of all vendors evaluated. However, customers choose “additional services to provide core functionality.” Ergo, Gartner’s product scoring model excludes or underweights capabilities that real-world customers consider “core functionality.”
Gartner dings AWS in generative AI because nobody wants to use Amazon Titan. Right, Titan sucks. Who cares? Foundation models are a dime a dozen. It’s like dunking on Verizon because they don’t produce all of the movies you get on FIOS.
Customers don’t choose AWS for its ability to build models; they choose the company because of its infrastructure skills. Amazon Bedrock offers customers models-as-a-service from popular model developers like Anthropic and Meta. It’s a cafeteria where you decide what to eat.
Microsoft
In the real world, Azure Machine Learning was better than SageMaker in 2021. Since then, it has gone sideways. The product and engineering team suffers from high attrition because Satya’s laser-like focus on GenAI sucks the air out of the room.
Gartner notes that Microsoft continually rebrands its services and breaks APIs, creating messes for data science teams.
Google
Gartner rated Google a Visionary in 2021. That was remarkable because Google launched Vertex in May 2021, several months after Gartner released the 2021 MQ. That position demonstrates the power of branding and analyst relations.
Google Vertex AI is considerably better today than three years ago (thanks partly to my former boss, Nenshad Bardolliwalla). Google was the last of the three leading CSPs to offer a DSML product; they learned by studying what AWS and Microsoft did wrong. Of the three top CSP DSML platforms, Vertex has the most coherent user interface by far.
Unfortunately, Vertex isn’t an option for your data science team unless your organization embraces GCP. Vertex isn’t the only platform-centric product in this MQ, but Google’s limited share in the CSP market makes the issue more visible.
Google drew the fifth-highest product score. AWS outscored Google in only three categories: Deployment, Generative AI, and Model Management. If Google is weak in “core data science use cases,” the product scores should reflect that.
Gartner says that Generative AI is a strength for Google and a weakness for AWS. However, the category scores say the opposite—just another one of those unexplained mysteries, like what happened to Jimmy Hoffa.
Alibaba
I don’t know shit about Alibaba.
Oracle
Oracle ranks fifth in CSP market share. Gartner awarded the company an Honorable Mention, which you can redeem for a cup of coffee at Starbucks if you give them five dollars.
Oracle bought a leading data science vendor in 2018 and killed it. The company announced a move to Nashville LOL and says it will focus on its health informatics business. I don’t think anyone in this category worries about what Oracle will do.
IBM
Hardly anyone uses IBM Cloud; I will cover them below under Big Legacy.
Big Data
This category includes vendors whose core business is a data platform. Gartner included Databricks and Cloudera in the MQ, with Snowflake waiting in the wings.
Databricks
Gartner rated Databricks as a Leader in 2021 and again in the latest MQ. Databricks is a quality company with the best machine-learning credentials in this category.
Many people aren’t aware that the Spark project's original goal was to build a better platform for machine learning in Hadoop. Apache Spark 1.0, released in May 2014, included Spark SQL in Alpha release; Spark SQL adoption accelerated rapidly and soon buried the ML bits.
In 2018, Databricks launched MLflow, the most widely used data science workflow software. The acquisition of Mosaic ML strengthens Databricks’ machine learning street cred and gives it an immediate entry into the generative AI space.
Gartner complains that “some users” are unhappy with Databricks performance tuning and cluster management. That’s ridiculous. I found one comment in Peer Insights from November 2018.
Databricks Machine Learning works with data in Databricks Lakehouse. If your data isn’t there, you put it there. All the platform-centric vendors say they can deliver a hybrid architecture but show you Rube Goldberg shit.
Cloudera
Cloudera moved from Niche Player to Visionary. Cloudera Machine Learning is a decent product. However, hardly anyone uses it unless they also use Cloudera Data Platform, which narrows the potential market.
Gartner is unhappy that Cloudera targets expert data scientists and lacks a low-code user interface. Tough. Low-code is for losers.
Cloudera has dabbled with machine learning and AI for years. Today it offers data science tooling that is “good enough” for some users, but nothing to write home about. Prospective customers who are serious about data science and AI will choose Databricks or Snowflake over Cloudera every time.
Snowflake
Gartner gave Snowflake an Honorable Mention, which is good for a cup of coffee at…eh, I already used that joke. Snowflake is bulking up with data scientists, and they just acquired TruEra. Nevertheless, Databricks has a ten-year head start in machine learning. Snowflake has much catching up to do, and they know it.
Others
SAP and Teradata used to compete in this MQ but are (rightly) MIA in 2024. SAP tried to buy its way into this market in 2013 when it acquired KXEN. SAP isn’t interested in selling tools; they embed machine learning in solutions.
Teradata is in the same position as SAS: not dead, just flat. At thirty-four bucks, that stock price is full of hot air.
Big Legacy
The legacy players are IBM, Mathworks, and SAS. Gartner rated all three as Leaders in 2021. In the latest MQ, SAS remains a leader, but just barely. IBM dropped out of the Leaders quadrant and is now a Challenger. Mathworks got hosed and is now a Niche Player.
IBM
IBM scored low marks for its product, and rightly so. IBM watsonx is a quodlibet of old toss plus a few new bits stacked on Red Hat OpenShift.
IBM fell out of the Leaders quadrant because it didn’t score well on Completeness of Vision (the horizontal axis). That’s an Analyst Relations FAIL—IBM didn’t do a good job cultivating analysts or preparing for the MQ presentation.
I was going to insert a joke about heads rolling in Armonk, but it’s no joke:
IBM rarely shows up in qualified DSML opportunities because it sells mostly to its existing “Blue” customers. I only lost a deal to IBM once; the Client Exec bundled software into an ELA and told the customer it was “free.”
Mathworks
Poor Mathworks. They work hard, building intense user loyalty and doing good work. Gartner views that as a negative, slamming the company for targeting “technical LOB users” in engineering. Those “technical LOB users” will give up the product when you tear it from their cold, dead hands.
MATLAB and Simulink are serious products that require some skill and expertise from the user. This makes Gartner sad. Remarkably, MathWorks can build a billion-dollar business without catering to children.
Gartner is also unhappy that Mathworks doesn’t position itself on the cutting edge of Generative AI. It’s the first time in history that Gartner cared about cutting-edge anything.
SAS
Gartner scored the SAS product offering in the middle of the pack; viability and other factors dragged the company over the line into the Leaders quadrant, but just barely.
Gartner thinks SAS is too expensive. No shit. Gartner dings SAS on cost in every MQ and every category for twenty years. SAS just ignores them. Cost does not affect a vendor’s MQ position.
Independent Software Vendors (ISVs)
This category includes Altair, Alteryx, Dataiku, DataRobot, Domino, H2O, and KNIME.
Altair
Altair, a Niche Player in 2021, acquired RapidMiner, a Visionary, and other assets. Gartner thinks 2+2=5, so Altair is now a Leader. In real-world M&A, 2+2=3. Altair’s R&D team will now spend most of their developer effort paying back technical debt.
How long will it take to clean that up? Altair acquired Datawatch in 2018, and DataWatch acquired Panopticon in 2013. Panopticon remains a standalone product. Each of the acquired products has its merits, but it seems unlikely that Altair will ever integrate all those bits. Not while we’re young.
Gartner notes that Altair has yet to integrate all of its acquired assets into a cohesive product. That’s a nice way to say that Altair currently offers a bag of parts.
Alteryx
Alteryx plummeted from Challenger to Niche Player. Under Gartner’s new rules, products optimized for insight and decision support don’t score well, as Alteryx discovered.
Alteryx is a data prep and blend tool for business insight and decision support. It always had code-centric functionality, but it sucked. Expert data scientists will use it again when first-class travelers fly Spirit Airlines.
Gartner worries about the impact of private ownership on the company. It shouldn’t. Private ownership will help Alteryx. They won’t have to spin bullshit stories for Wall Street about how they are so much more than prep and blend. The company can take consolation in having four times the revenue of the other players in this category.
DataRobot
DataRobot moved from Visionary to Leader. The company sells the leading software for automated machine learning but has always struggled to define its market. Expert data scientists tend to be lukewarm about AutoML; on the other hand, business analysts don’t need cutting-edge AutoML and are perfectly happy with tools embedded in Power BI, Qlik, or Salesforce.
Does DataRobot want to help customers with decision support and insights, or does the company want to help with operational machine learning? The company seeks to pivot away from the former towards the latter, so it moved up in this MQ.
DataRobot has always had a code-based API and a graphical user interface. In the last three years, the company has invested in a data science workbench to complement its core AutoML user experience. Still, there is work to do: the workbench itself doesn’t compare well with other data science platforms on the market.
Although AutoML is in DataRobot’s DNA, its current strengths are in governance, deployment, and monitoring. Gartner rated DataRobot’s model monitoring best in class, which is well-deserved.
Dataiku
Gartner rated Dataiku as a Leader in 2021, and the company repeats in this year’s MQ. Dataiku offers a nice collaborative workflow and robust data connectors. The company only recently added a model export capability; it’s not very good, so what you build in Dataiku may have to stay in Dataiku. That’s sad because Gartner says Dataiku model serving isn’t great.
In my recent Substack on Dataiku, I said this:
…the DSS back end is less than robust. (I’m being diplomatic.)...The workloads that Dataiku can’t push down to K8S or a scalable data platform run on a server. That’s fine if you have a small data science team or you don’t need to do serious machine learning. DSS will struggle if you want to do computationally intensive stuff.
Gartner awarded 3.4 points out of 5 on performance and scalability. That’s not terrible. It’s not the worst score, just below average. The good news is that Dataiku is more scalable than Posit. Salut tout le monde! Sortez le champagne!
Gartner complains about Dataiku’s “complicated pricing model.” It’s not that complicated. You pay a fee for the platform and a fee per user. Persona-based pricing makes perfect sense. Simple and predictable. If you want to see complex pricing, try to budget for SageMaker.
Domino Data Lab
Domino moved from Niche Player to Visionary. Domino benefited from Gartner’s increased focus on operational machine learning. The company drew a best-in-class score for Model Management and above-average scores in every product category except Generative AI.
Gartner complains that Domino customers are concentrated in a few industries. That’s true for most of the vendors in this MQ because those four industries (Banking, Insurance, Pharma, and Government) spend the most on DSML Platforms.
Gartner is also unhappy that Domino focuses on serving large expert data science teams. Domino supports large teams without falling over, which is a good thing.
H2O.ai
H2O was a Visionary in 2021 and remains a Visionary. H2O built its reputation with a scalable machine library but has broadened its product suite considerably in the past five years. The company competes with DataRobot for AutoML but with a stronger appeal for expert data scientists.
Gartner dunks on H2O for targeting a few key industries. See comments above, under Domino. DSML vendors target those industries for the same reason Willie Sutton robbed banks.
KNIME
Gartner scored KNIME as a Leader for six straight years, then a Visionary in 2021, and now a Niche Player. O, how the mighty hath fallen. KNIME distributes the KNIME Analytics Platform under an open-source license, which Gartner hates. The company has traction in pharma, loyal users, and partners who build custom solutions.
KNIME's low-code user interface targets a broad user base. Gartner thinks this is a problem because it thinks vendors in this category should support every user persona.
Open-Source Enablers
Anaconda and Posit fall into this category; Gartner thinks they stink.
Gartner does not like open-source software. You can’t blame them. Organizations that embrace open-source software don’t need Gartner to tell them which products are good. They just try the product.
Somebody should tell Goldman Sachs that open-source software is terrible.
In the Critical Capabilities report, Gartner says that business leaders should not allow data scientists to build models for production with “unmanaged” open-source tools. The vendors in this category provide organizations with tools to manage open-source software, but Gartner trashes them anyway.
Anaconda
Anaconda drew the lowest total product score, and Posit drew the second lowest.
Gartner slams Anaconda for its reliance on open-source components. Uh yeah. That’s Anaconda’s DNA. They help customers use open-source software.
Gartner is also unhappy that Anaconda targets expert data scientists, the people who drive the most value.
Posit
Posit participated in the MQ, but that didn’t help them much. Gartner hates Posit, too, because it focuses on insight delivery over ML engineering. Yes, that is what most R users do, which is why Posit is in business.
Gartner just doesn’t think insight delivery is important. Tell that to the Chief Scientist at a pharmaceutical company.
In Gartner’s world, people are either data scientists who write code and build deployable models or business analysts who need a graphical user interface. The typical R user is an analyst or researcher who wants to extract insight from data but prefers to write code. The concept makes a Gartner analyst’s head explode, but there are armies of such people in pharma and elsewhere. That is Posit’s target market.
Gartner also dings Posit for failing to give lip service to GenAI. Considering its customer and user base, Posit’s conservative approach to GenAI makes sense.
Thanks for reading!