By Guillaume Lemaitre, scikit-learn core maintainer & VP Product Strategy at Probabl
Data science has the power to transform how enterprises understand their world and make high-impact decisions. When executed well, it delivers profound business value through optimized operations and strategic advantages that compound over time.
However, the potential for data science remains constrained by a myriad of organizational challenges and friction points that exist throughout data science workflows, from misaligned processes to fragmented tooling. It’s clear that enterprises deserve better tooling for data science–tooling designed for data scientists, by data scientists.
In this post, I break down the evidence about these friction points and present our vision at Probabl for building the next generation of data science tooling, anchored in our deep understanding of the scientific practice of data science.
Industry insights provide a clear signal that we have to change the status quo of enterprise data science and AI.
According to RAND, more than 80% of AI projects fail, which is twice the rate of traditional IT projects [1]. Further research finds that 87% of projects never even reach production [2], and Gartner predicts that through 2026 60% of AI initiatives will be abandoned because the data held by enterprises is simply not yet AI-ready [3].
It’s also clear that these failures are rarely the result of poor data science or AI in itself; they tend to be organizational. For example, RAND finds that challenges often come from the top when business leaders do not clearly articulate specific problems that need to be solved and lean into technology-first thinking, prioritizing trendy AI solutions when simpler analytical approaches might actually suffice [1].
Many AI projects also fail because enterprises lack the necessary data to train models, and even when data exists, the unglamorous work of data wrangling consumes a disproportionate amount of time [1, 6]. This is compounded by a process mismatch, where frameworks designed for software engineering are applied to data science, despite it being an R&D process where the end product is often unclear at the outset [4-5].
If we want to build solutions that solve these challenges, we must understand exactly where frictions live in enterprise data science workflows.
To do so, let’s think of the data science workflow as it’s visualized in Figure 1. We can distinguish the types of work by their nature rather than by job titles. Upstream work centers on making raw data available and usable, while downstream work focuses on putting models into production. Between these lies the data science work–the scientific core of the process–which demands methodological rigor and careful experimental design.
Figure 1: A typical data science workflow (arrows represent potential failure points)
While the data science ecosystem provides many powerful tools for these different types of work, the connective tissue to make them work as a coherent whole is still missing. What’s more, the current tooling landscape wasn't designed with the data-scientists’ work as its focus, but rather production, which is closer to engineering. As a result, frictions and potential failure points exist throughout the data science workflow, which we must solve for.
These frictions include the following:
Friction at the boundaries: Currently, the transitions between upstream work (data collection, data engineering), data science work, and downstream work (MLOps) are not always optimal. In the worst case scenarios, handoffs between stages result in lost context, rewritten code, and undocumented methodological decisions. Moving from prepared data into experimentation, or from validated model to production, requires knowledge that current tools neither capture nor transfer.
Friction between stakeholders: Domain experts struggle to articulate data requirements clearly. Those leading data science teams find it difficult to translate model performance into business impact. Business leaders set objectives without understanding what AI can realistically achieve. Technical work proceeds without clarity on what success looks like.
Friction within the data science work itself: Experiments need review, results require interpretation, and methodological choices must be justified. Yet without structured processes for this validation cycle, quality control risks becoming ad hoc and inconsistent.
Friction in data science tooling: Most critically, existing tools tend to misunderstand what data science is. Data science is not software engineering. It transforms data–the essential ingredient–into impactful results through three dimensions: coding, business understanding, and scientific methodology. This scientific dimension changes everything. A methodological mistake can invalidate an entire project, leading teams to perfectly solve the wrong business problem, regardless of code quality or data accuracy. Existing tools do not address the core challenge of data science work–ensuring scientific excellence, maintaining methodological context across iterations, and translating statistical findings into business narratives.
These frictions create an opportunity for a new generation of data science tooling.
At Probabl, we’re on a mission to remove these frictions and unleash the full potential of data science teams. We imagine a world where data science moves at the speed of insights, where experiments build naturally on previous work, and where the path from a question to an answer is measured in days, not months.
This requires a fundamental shift in building tools for data scientists, by data scientists. Towards this end, you can expect us to double down on the following principles.
We build specifically for the data scientist.
This choice is rooted in our identity as stewards of scikit-learn. We have shaped how millions of practitioners practice machine learning. We understand that the data scientist holds a unique position as a hybrid professional who bridges quantitative excellence with business context. While AI can automate parts of code generation, it cannot replicate the contextual understanding and data-driven reasoning required for high-stakes decision making in enterprises. We believe the work carried out by data scientists is where better tooling is most needed to achieve faster outcomes and impact.
We build ecosystem-first architecture, not one-size-fits-all platforms.
We understand that data scientists have strong preferences for their tools. They are more likely to assemble a curated set of libraries within their existing environment than to adopt a rigid, all-in-one platform. Similarly, we understand that enterprises operate on diverse infrastructures. Attempting to latch a one-size-fits-all data science platform onto these heterogeneous infrastructures creates integration challenges, vendor lock-in, and operational frictions. We offer a modular, slot-in approach that respects autonomy and works with an existing technology stack rather than replacing it. This ecosystem delivers value where data scientists actually work while integrating seamlessly with diverse enterprise infrastructures.
Data science is fundamentally a process of empirical discovery.
What distinguishes data scientists from software engineers is not just technical skill–it is that data science is fundamentally empirical discovery. Data science exists to uncover novel insights from data–to reveal patterns, relationships, and knowledge that weren't previously known or understood. Because data science is discovery, it naturally demands suitable methodological approaches. When you're uncovering insights that will drive business decisions, methodological mistakes can lead to false discoveries, misguided strategies, and wasted resources. Our solutions ensure that empirical discovery is supported throughout the entire process, from data validation to stakeholder communication.
If data science is the discipline of unlocking value from data, then we should practice what we preach: we will augment data scientists by leveraging AI.
The goal is to reduce time-to-value by deploying AI where it genuinely accelerates the workflow, such as through natural language interfaces and automated diagnostics. We build intelligent systems that learn from the essence of data science practice to provide contextually relevant guidance. As advancements emerge like the SOTA tabular foundation models TabICL [7] and TabPFN [8], we integrate these innovations while maintaining the principle that AI should always empower the data scientist rather than replace him or her.
To truly serve data scientists, we must address friction across the entire workflow.
Upstream, we bring databases closer to predictive modeling to reduce reliance on constant intervention from data engineers. Downstream, we view MLOps through the practitioner's lens to ensure seamless handoffs while preserving scientific context. Finally, we provide tools to help translate technical results into business language, recognizing that communication is where value is ultimately realized or lost. By providing tools that help data scientists explain model behavior and translate technical results into business narratives, we seek to ensure that scientific excellence leads to real-world impact.
At Probabl, we’re committed to putting our principles into practice. Just as we standardized the practice of machine learning by creating scikit-learn, we intend to do the same for the entire enterprise data science practice.
References:
[1] Ryseff, J., Bruzelius, E., & Scobell, W. (2024). The Root Causes of Failure for Artificial Intelligence Projects and How They Can Succeed: Avoiding the Anti-Patterns of AI. RAND Corporation. https://www.rand.org/pubs/research_reports/RRA2680-1.html
[2] Lorica, B. (2019, July 19). Why do 87% of data science projects never make it into production? VentureBeat. https://venturebeat.com/ai/why-do-87-of-data-science-projects-never-make-it-into-production/
[3] Gartner. (2025, February 26). Lack of AI-Ready Data Puts AI Projects at Risk [Press release]. https://www.gartner.com/en/newsroom/press-releases/2025-02-26-lack-of-ai-ready-data-puts-ai-projects-at-risk
[4] Freischlag, C. (2022). Machine learning projects and Scrum: A major mismatch. Towards Data Science. https://medium.com/data-science/machine-learning-projects-and-scrum-a-major-mismatch-c155ad8e2eee
[5] Ribeiro, D. (2025). Why Agile Doesn't Work for Data Science and How DSLP Fills the Gap. LinkedIn. https://www.linkedin.com/pulse/why-agile-doesnt-work-data-science-how-dslp-fills-gap-diogo-ribeiro-gnirf/
[6] Oleli, D. (2018, July 13). Bridging The Data Scientist Talent Gap Starts With Defining The Current Role. Forbes. https://www.forbes.com/sites/forbestechcouncil/2018/07/13/bridging-the-data-scientist-talent-gap-starts-with-defining-the-current-role/
[7] Jingang Qu, David Holzmüller, Gaël Varoquaux, Marine Le Morvan. (2026). TabICLv2: A better, faster, scalable, and open tabular foundation model. https://arxiv.org/abs/2602.11139 Code: https://github.com/soda-inria/tabicl Installation: https://pypi.org/project/tabicl/
[8] Noah Hollmann, Samuel Müller, Katharina Eggensperger, Frank Hutter. (2023). TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second. https://arxiv.org/abs/2207.01848. Code: https://github.com/PriorLabs/TabPFN. Installation: https://pypi.org/project/tabpfn/.