The Joy of Batch Inference
Meryem Arik, CEO and co-founder of Doubleword, pitched me recently on the company’s new service, Doubleword Batch Inference.
Batch Inference? My eyes glazed over. Cool people want to talk about AI agents or the shit they can do with Moltbot. Doesn’t everyone do batch inference?
In five minutes, I found eight companies using batch inference for production applications:
Cashfree Payments, a fintech company based in India, uses an LLM to crawl and index merchant websites to extract information. Running the process in batch reduces costs and still delivers a dramatic reduction in the time needed to onboard a merchant.
DoorDash uses a batch process in PySpark to profile millions of consumers, merchants, and items. The process itself is model agnostic; the company periodically tests and evaluates the results from different LLMs.
Instacart developed a system using an LLM to suggest complementary ideas. Testing showed that real-time inference was too slow and expensive, so the company switched to a batch architecture.
Netflix treats a user’s history as a “sentence” and uses LLMs to predict the next movie. To do this at scale, they run offline batch jobs to generate embeddings.
Pinterest uses LLMs to evaluate the quality of search results with offline batch inference. They run quality checks offline to label massive datasets, which they use for further analysis.
Spotify runs batch inference to transcribe and analyze podcast episodes to extract metadata, chapters, and summaries.
Uber runs a nightly process to summarize insights from the previous day’s customer support interactions.
Zalando, a European e-commerce company, performs postmortem analysis for every database incident. The company batch processes these reports through an LLM for pattern detection, analysis, and summarization.
Here are some things you can do with large language models that don’t require real-time inference:
Summarize bulk documents
Assess call center quality
Scrape competitor websites and mine the docs for insight
Review financial transactions for fraud.
Localize documentation
Tag content at scale for SEO
Analyze patent applications
Redact PII from documents
We’re just beginning to leverage the potential of large language models. As we expand the scale and scope of applications, batch inference makes economic sense.
OK, you want to do batch inference. Great. Big brand model vendors like Anthropic, Google, and OpenAI all have batch APIs. Are they the best choice for your application?
Yes, if you are stuck on proprietary models. They are also a good choice if your biggest headache is spending all the money in your budget. Batch inference pricing for the big brands runs from $1.25 to $2.00 per million input tokens. That may sound cheap, but costs mount quickly with enterprise-scale data, especially for a repeatable process.
Suppose you want to process the Justice Department’s recent Epstein file dump. It’s about 1.5 billion tokens, which will cost you about $3,000 just for the input tokens if you use Google Gemini. That’s a lot to pay for information about Bill Gates’s STD.
By the way, the “bigs” turn off automatic key-value caching for batch inference. That means you may have to pass that file more than once, depending on the task you want to perform and how you organized the input files.
Forget about a real SLA, too. Batch inference is a side gig for the big model vendors. Instead of a real SLA, their standard is “finish or fail” in 24 hours. That’s a polite way to say they will run the job for 24 hours, but they don’t guarantee that the job will finish. They only bill you for the part that runs.
It’s like a restaurant that guarantees service in one hour. The waiter delivers your appetizer, but when the hour’s up, he says, “Sorry, we don’t have time to cook your main course, here’s a bill for the appetizer.” Then, he kicks you out.
You’re serious about minimizing inference costs. Okay, for starters, work with one of the open weight families: DeepSeek, Gemma, Llama, Mistral, or Qwen. Yes, I know, there are other open model families. But those five currently have the broadest availability on independent hosting platforms. Open weight models invite vendor competition, which drives down costs.
Vendors that sell proprietary models build training costs into their inference pricing. With open weight models, you don’t pay for training costs. You also don’t pay for Super Bowl ads, overhead, and Sam Altman’s lifestyle.
Amazon Bedrock and Google Vertex support Gemma, Llama, and Mistral, as well as their proprietary models. You will pay through the nose, but you can go to their big shows, chat with their sales reps, and get some free merchandise.
The real price competition is among the specialist service providers:
Predibase (acquired by Rubrik)
With the specialists, your batch inference costs are a fraction of what you pay the “bigs.” Pricing varies considerably depending on which model you use; to compare vendors, you need to specify a model. The table below shows Doubleword leading in the price war for batch inference with the flagship Qwen model.
At a nickel per million, the Epstein dump costs just $75 for input tokens. That’s a reasonable price to pay for information about Bill Gates’s STD.
DeepInfra promotes low prices, but lacks any sort of SLA. Predibase isn’t competitive; they focus on fine-tuning and AI agents, which is fine if you’re into that shit. If you want low-cost batch inference, you can count them out.
Doubleword uniquely offers an SLA, while the others only offer “Finish or Fail.” Dine at Doubleword, and they promise to bring your appetizer, an entree, and a nice dessert. The others will chug away for 24 hours and maybe get the job done. Or not.
How does Doubleword drive the costs out of inference?
Focus: the Doubleword team has a laser-like focus on inference; it’s all they do. While other AI startups mess around with the current thing, the Doubleword team knows that inference is the “last mile” to AI value. Focus is the most important attribute for any startup, and the most easily forgotten.
Design: When you build an inference stack, you optimize for latency, or you optimize for cost. It’s one or the other; you cannot optimize both at once. Doubleword optimizes its stack to minimize cost. This approach affects the company’s choices of cloud platform, hardware, models, runtimes, and orchestration.
Kaizen, or continuous improvement: technology advances through tinkering. Continual small improvements on the margin over time lead to great leaps forward. The Doubleword team constantly searches for new ways to make inference better and cheaper. Learn more about the company’s engineering advances here, or the blog here.
There are some limitations to using Doubleword Batch Inference. Doubleword deploys a limited number of models, choosing those that are on the cutting edge of model performance for their size. Currently, the service supports the Qwen family. That’s a temporary problem if you’re stuck on other models; Doubleword plans to add new models in a few weeks.
It’s comparable to your shopping experience at Costco versus a conventional supermarket. Safeway has 40 brands of chocolate sandwich cookies; Costco carries Kirkland cookies.
You pay for variety. Your kids don’t care; they just want cookies.
LLMs are a commodity. With most applications, you won’t see a material difference in the results you get from different open-weight models. The burden of proof is on those who demand the more expensive model.
Right now, Doubleword Batch Inference runs only in Doubleword’s cloud, in European regions. The service currently has a 200MB file limit; Doubleword plans to double that soon. Customers with larger files split them and send them sequentially, so no big deal. Doubleword expects to introduce connectors to S3, Azure Blob, and BigQuery this quarter; Databricks and Snowflake connectors may follow.
Doubleword launched Batch Inference in general availability four weeks ago. Since then, the company has processed more than 50,000 batches with only one missed SLA, an outstanding record for a new service. The company reports rapid growth in batch and token counts.
Learn more about Doubleword here. The company has nine sample workbooks you can run, and offers credits to help you get started.
Note to Readers: I do not accept compensation in any form from companies that I write about.
Interesting.
Thanks for a comprehensive and balanced write up!
Only thing is lots of people would have second thoughts about sending docs to any external LLM for PII redaction. But maybe fine for a local one or one you trust.