Wednesday, July 22 2026

The path toward  end-to-end Data Science Agents you can trust in production

12:24

By Yann Debray, CPO of Probabl &
Gaël Varoquaux, CSO of Probabl & Research Director at Inria

TL;DR

Data scientists are already adopting AI coding assistants and agents at certain stages of their workflows, especially for coding, but we’re not yet at a point where we can delegate entire data science workflows or machine learning experiments to agents and blindly trust the outcomes. The missing puzzle pieces in the capabilities of Data Science Agents are proper methodology and statistical thinking. In this post, we discuss the types of AI solutions that data scientists are already using in their workflows, and the capabilities that would be required to adopt a Data Science Agent that can autonomously iterate through end-to-end workflows and produce outcomes you can confidently trust.

The path towards agentic data science

We started our last post by stating that the agentic moment of data science is on the horizon [1]. We, data scientists, are already coding faster – and let’s be honest, often better – thanks to AI. We can easily prompt our coding assistants to generate code snippets or even entire machine learning pipelines.

But unlike software engineering where AI now writes the majority of code shipped in production [2], we’re not yet at a stage where we can confidently adopt AI to autonomously execute loops of iterations through our workflows and experiments, and trust the models that come out on the other end.

As we explained in our last post, the reason for this is that coding agents excel at – precisely – coding but not at data science methodology nor statistical thinking, which are core to producing machine learning models that can be trusted in production [1].

The challenge is that proper methodology cannot be verified as code. Code has a compiler that tells you when you're wrong, data science doesn't. The code may work, but if the methodology is wrong you may not be able to trust the resulting model.

If we want to be able to adopt AI agents in our data science workflows without compromising on trust, we need to harness agents to follow proper methodology and statistical thinking. In this post, we delve into a bit more detail about why we haven’t seen the agentic transformation in data science yet and propose a path forward.

Coding vs. data science: Why the use cases diverge

Agents are good at coding for two reinforcing reasons. First, code execution gives a verifiable feedback signal (e.g., tests pass or fail, the program runs or throws). Second, code has been open-sourced at a scale with no data science equivalent. ML pipelines and their underlying datasets are largely proprietary, so models have far less public signal to learn from. Data science workflows often lack both a clean pass/fail signal and a comparable open corpus. Even where a proxy exists, like the training loss curve, it doesn't capture the judgment calls that matter. Investigating whether you're using the right features, catching data leakage, and knowing if your model actually solves your business problem.

This brings us to methodology. Agents need to know and follow proper methodology. Let’s take the example of data leakage, which is one of the most common mistakes in machine learning. In simple words, leakage occurs when information from the test data accidentally slips into the training data [3]. This produces a model that looks great in testing because it's essentially memorizing the answers, but performs poorly once it encounters real, unseen data. Avoiding this means carefully separating training and test data from the start, and testing only on data the model has never seen. Sounds simple in theory, but in practice preparing and transforming raw data to avoid leaks takes real, careful work.

Statistical thinking is another skill we need to teach agents. When a model suddenly fails in production, the solution may not necessarily be to find and fix a bug in the code, but rather to ask questions like "why did this result change from last week?" or “can we trust the result after we have fixed the bug?”. Many factors can influence why predictions are right on one day and wrong on another. Statistical skills are crucial to apply scrutiny to models and ultimately to trust them in the wild.

Agentic Data Science vs. Data Science Agents

Before we continue any further, we should clarify what we mean by “agent” in the context of data science.

The term “agent” implies autonomy [4]: a system that can independently scope a task, execute it, and validate the result with minimal human-in-the-loop.

As we know from the software engineering context, coding agents are already automating large portions of engineering workflows with limited human intervention. Anthropic's analysis [5] of roughly 400,000 Claude Code sessions found that in a typical session, people make most of the planning decisions (what to do) while Claude makes most of the execution decisions (how to do it). In practice, this means an engineer can tell an agent like Claude Code to add a feature, and the agent will carry out that decision: implementing the change, running tests, and preparing a ready-to-review pull request.

By that bar, a Data Science Agent would be one system that owns an end-to-end data science workflow [6] the way Cursor or Claude Code owns a coding task. As Eric Ma puts it, the data scientist handles the inquiry while the agent handles the implementation [7]. To the best of our knowledge, such a solution doesn't exist yet.

The agentic data science workflow by Probabl, the Tabular AI company by the creators of scikit-learn

Figure 1: The agentic data science workflow

However, what does exist is Agentic Data Science: agents applied to different stages of the data science workflow, but not yet the full process – currently strong at engineering, since methodology and statistical thinking still lag behind [1]. This is how agents are being used so far at different stages of the data science workflow [8 - 10]:

Data ingestion: Pulling raw data from source systems into a usable location for the pipeline. To date, there is no agent specifically for this stage of the workflow.
Data cleaning: Handling missing values, duplicates, outliers, and inconsistent formats before the data is trustworthy enough to be used to train a model. Some notebook agents, such as Julius, handle cleaning as a side effect of exploration.
Data exploration, data preprocessing, and feature engineering: Understanding the data's structure and relationships, then transforming raw fields into inputs a model can actually learn from. For example, ML-engineering agents like AutoKaggle handle feature engineering within their train/tune loop. However, such agents mostly focus on Kaggle-style benchmarks where the feature space is far narrower than in real-world production environments.
Model/estimator training: Selecting an algorithm, fitting it to data, and tuning hyperparameters against a defined metric. ML-engineering agents cover this stage. For example, MLE-STAR, Google’s own frontier ML-engineering agent, has a dedicated leakage-detection module built into it [11].
Model evaluation: Judging whether the model actually solves a business problem, based on real-world validity, fairness, and leakage checks. Outside of dedicated evaluation tools like Probabl's skore, this work mostly falls to general ML-engineering agents like AIDE, rather than tools built specifically for evaluation.
Deployment: Packaging and shipping a validated model into a production environment. This stage depends on AutoML/lifecycle platforms, such as DataRobot, rather than on specific agents.
Monitoring models in production: Tracking drift, performance decay, and data quality once the model is in production, and flagging when it needs attention or retraining. This stage is well covered by specific purpose-built tools, such as WhyLabs, rather than by specific agents.

In a nutshell, agentic solutions cover only some stages in the data science workflow: model/estimator training and to a lesser extent feature engineering. Data ingestion still relies mostly on human judgement, while deployment and monitoring depend on platforms and purpose-built tools rather than agents. Data cleaning and model evaluation sit in between: agentic solutions perform these steps indirectly as a side effect of other tasks rather than as their dedicated focus.

We don't think that's a coincidence. Training has a clean, automatable signal to optimize against, the loss curve, which is exactly the kind of feedback agentic solutions excel. Feature engineering benefits from a weaker version of that same signal, downstream model performance, which explains its partial coverage. Monitoring also has a clean signal, drift metrics, and that signal is already well served by dedicated platforms; deployment is governed by similarly purpose-built infrastructure.

The remaining stages are different in kind. Data ingestion, data cleaning, and model evaluation are all driven by judgment calls – for instance, deciding which source is trustworthy, how to impute missing values, and which metric matters most – that don't reduce to a single number to chase. That's precisely why agentic solutions might still be so limited there.

The conditions for end-to-end Data Science Agents

If there’s no verifiable feedback signal to enforce validity checks, it begs the question of what can we do to get agents to follow proper methodology and practice statistical thinking?

We think better evaluation signals can be used to play the same instrumental role as the compiler does in software engineering.

Today's agentic tooling already follows a clear pattern: wherever success can be measured automatically, such as minimizing a loss curve during training or monitoring drift in production, agents have made rapid progress. Where success still depends on human judgment, such as deciding whether data is trustworthy, features are meaningful, or a model actually solves the right problem, automation remains limited.

That suggests the bottleneck is not the models themselves, but the absence of reliable ways for agents to verify their own work. This is the inspiration behind skore, our own open source library. It catches methodological errors before they impact models and surfaces the metrics that matter through automated reports, turning subjective judgment into measurable checks.

On Hugo Bowne-Anderson’s podcast, Vanishing Gradients, he points to why the gap remains. Agentic data science hasn't taken off the way agentic coding has because it risks producing "vibe scientists" as people who can't verify whether what the model produced is actually real or significant [12].

Eric Ma [13] frames this shift similarly: the data scientist's core job is moving toward defining and evaluating what actually matters, since AI tools can build but can't decide what's worth measuring.

The next generation of data science agents is therefore unlikely to emerge from more capable foundation models alone, but from richer proxy signals that let agents evaluate the workflow stages that today still require human expertise.

Thomas Wiecki makes a similar point on Hugo Bowne-Anderson's podcast [7]. He argued that the missing piece is a causal evaluation layer that can actually measure whether an agent's decision improved a real-world outcome, not just whether the code ran.

At Probabl, we’re building open source skills and the first statistical harness for data scientists who want to adopt agents in their workflows and trust the outcomes. The work starts here: https://github.com/probabl-ai/skills