This post is Part One of a three-part series. I will publish the remaining parts over the next two weeks. Part One is a seven-minute read.
Gartner’s 2024 Magic Quadrant for Data Science and Machine Learning Platforms is now available. Congratulations to the Leaders: Altair, Amazon Web Services, Databricks, Dataiku, DataRobot, Google Cloud Platform, Microsoft Azure, and SAS.
For everyone else, there’s always next year. You might get some good draft picks, LOL.
Several vendors, including Amazon, Databricks, Dataiku, Domino, Google, Microsoft, and SAS, offer licensed copies of the Magic Quadrant report. DataRobot also offers the Critical Capabilities for Data Science and Machine Learning Platforms, Machine Learning Engineering report.
Vendors that made it into the Leaders Quadrant praise the wisdom and intellect of Gartner analysts. Others smile, kowtow to the analysts, and whisper that Gartner rigs the whole thing.
Screaming at the umpire has no value. It smells of sour grapes. Gartner analysts do not accept suitcases full of cash, but Gartner's institutional bias affects its analysis. I will discuss this later in the series.
Many Gartner clients treat Magic Quadrants like a horse race. Open the report, check the upper right-hand box, choose Leaders for the shortlists, done. It’s the Ricky Bobby approach to software selection: If you ain’t a Leader, you’re a Loser.
Check some of the previous MQs for this category, and you will see why that’s a dumb move. Gartner rated TIBCO a Leader in 2021. If you bought TIBCO, you are now struggling to figure out what to do with it.
Correction: the person they hired to replace you is now struggling to figure out what to do with it.
Gartner says this report is a “new” Magic Quadrant. That’s pedantry. Gartner published a Magic Quadrant for Data Science and Machine Learning Platforms in 2021 (and previous years) with the same mix of vendors. Until 2017, Gartner called it the “Advanced Analytics” MQ. This year’s MQ is “new” because Gartner changed its evaluation approach.
It’s a rules change, not a new game.
In previous DSML MQs, Gartner mixed tools for insight and decision support with tools for operational machine learning. That was a hot mess. Gartner’s revised scoring model strengthens the focus on operational machine learning; there is much less emphasis on data viz and more focus on deployment, governance, and scalability.
Gartner continues to promote the “citizen data scientist” concept. In most organizations, expert data scientists lead operational machine-learning projects. Business leaders do not entrust important projects to rubes with tools; they assign people who know what they are doing. Most expert data scientists prefer a code-based user interface because it offers greater control than a graphical interface.
I can dunk on “citizen data science” at length, but I will save that for another day.
Seven Gartner analysts contributed to this report. I’ve met some of them in person and attended briefings with most. The team includes one analyst, Peter Krensky, who has deep experience in this category and six who do not. If your organization has an established data science and machine learning program, you have dozens of people who know more about the discipline than anyone on this team.
Surprisingly, this team is an upgrade. Gartner hasn’t previously assigned its “A” team to cover data science and machine learning. A few years ago, a DataRobot presenter showed two Gartner analysts a word cloud; they were so excited they nearly wet themselves. It reminded me of George H.W. Bush fawning over a supermarket scanner:
Then he grabbed a quart of milk, a light bulb, and a bag of candy and ran them through an electronic scanner. A look of wonder flickered across his face again as he saw the item and price registered on the cash register screen.
Collectively, the current team holds four master’s degrees and two doctorates. That’s a lot of credentials. Too bad they never learned how to write. Grammarly found 341 issues in this report and rated the text 29/100 on the Flesch Readability score. That’s a rigorous way to say the writing sucks.
Muddled writing reveals muddled thinking. Clean up your act, Gartner.
Gartner begins the report with one of its classic “Strategic Planning Assumptions”: the analysts predict that your organization will retrain half your data analysts as data scientists, and your data scientists will retrain as “AI engineers.”
When you plan investments in computing platforms, assumptions like this help if you clearly define “data analyst” and “data scientist.” Gartner doesn’t define “data analyst anywhere. Here’s how they define “data scientist:”
The data scientist role is critical for organizations looking to extract insight from information assets for “big data” initiatives and requires a broad combination of skills that may be fulfilled better as a team. For example, collaboration and teamwork are required for working with business stakeholders to understand business issues. Analytical and decision modeling skills are required for discovering relationships within data and detecting patterns. Data management skills are required to build the relevant dataset used for the analysis.
That’s a hot mess.
Large organizations have ten data analysts for every data scientist. Most don’t work in IT; they work in Marketing, Logistics, Credit, Risk Management, Operations, etc. Business domain expertise is their strength. Many already have “analytical and decision modeling skills,” which they don’t use 24/7 because simpler analysis is enough.
When the CEO wants to know how many scooter pies we sold in Duxbury last week, you don’t need TensorFlow.
Will data scientists turn into “AI engineers?” Sure. And citizen data scientists will eat the world.
Gartner defines the Data Science and Machine Learning Platform category with four paragraphs of word salad. I included a verbatim quote in the original version of this article, but that made Gartner sad, so go and read it yourself.
Let’s jump to the list of capabilities. First, the core requirements:
Import data from databases, data warehouses, and file stores located on-premises and in the cloud
Build and evaluate models using a library of core data science and machine learning techniques, methods, algorithms, and processes
Deploy, host, and serve models in the platform for usage in services and applications
Stop right there. Gartner believes that you need a single platform for model development and model deployment. That is nonsense; machine learning pipelines are code packages. Leading development platforms export model pipelines in containers, and the top deployment platforms can import containers.
There are some items missing from Gartner’s list of required capabilities:
Support diverse computing platforms
Provide governance and control over all deployed models
Preserve, curate, and secure assets across the development and deployment lifecycle
Monitor the status of development projects and deployed models
Monitor and control infrastructure costs
Real-world organizations do not adapt their computing platform architecture to the needs of a vendor’s DSML offering. They embrace a computing and data storage architecture and then consider conforming tools.
I will choose my cloud service provider based on their data science tooling, said no CIO ever.
Governance is essential for enterprise AI at scale. You need formal controls built into the platform when you have a hundred data scientists. Controls are also necessary if you want to “democratize” AI.
If you don’t think governance is mandatory, you don’t work for a bank, insurance company, pharmaceutical firm, healthcare provider, or any company with a presence in the European Union.
Next, Gartner enumerates what it calls “standard” capabilities, which elaborate on the “required” capabilities.
Ability to build models from structured and unstructured data sources, including text, images, video, audio, and geospatial
Low-code interface for model development suitable for nonexpert data science roles, including business users and domain experts
Notebook-based code interface for data scientists to perform data access, preparation, model development, and publication tasks
Postdeployment model life cycle management to retrain, retire, or adapt models based on detecting and analyzing data, feature, and model drift
Support for MLOps-based processes and tools for model deployment at scale in different operational environments
A code-based interface for expert data scientists is essential. You also need a low-code interface for model consumption, but a low-code interface for model development is fluff.
Anyone who knows what they are doing with machine learning knows how to write code. People who don’t know what they are doing should stay out of the way.
Finally, we get to “optional” capabilities:
Platform-generated recommendations for the best way to prepare, integrate, and model data, as well as automated creation of machine learning models based on manually selected target prediction
Advanced interfaces that facilitate more complex modeling for simulation, optimization, and deep learning-based use cases
Custom SDKs that provide more control and flexibility for code-based model development and integration with services and applications
Functionality for working with GenAI models, such as large language models, through tracking, selection, and monitoring of prompts, models, and outputs
Techniques and tools that increase the transparency and interpretability of models
The first four are optional, but the last one is not. Every model has a business stakeholder; most stakeholders aren’t data scientists. Opaque models are dead on arrival.
Gartner doesn’t say how they construct the list of Critical Capabilities; they simply hand it down as gospel.
Their list is not terrible. It includes many of the essential capabilities but omits a few. It contains some fluff that you may not need. It’s also very high level. Gartner writes the RFI for vendors at a higher level of abstraction than most real-world RFIs and RFPs.
The most significant weakness is that Gartner’s hypothetical requirements aren’t the same as yours. For the sake of argument, let's suppose that Gartner’s Critical Capabilities reflect a consensus of leading-edge organizations. (They don’t, but let’s pretend.) Furthermore, suppose the Critical Capabilities are unbiased and do not tilt the playing field toward a vendor or a class of vendors.
Suppose those two things are true. Why do you want to build that? Nobody ever secured a competitive advantage by doing what everyone else does.
Asking Gartner what you need in a data science and machine learning platform is like asking your doctor what kind of car you need.
Part Two covers Gartner's methodology for scoring vendors in the Magic Quadrant. Yes, the methodology is dull. However, it affects how vendors land in each quadrant, so we need to know how Gartner makes the sausage.