By Adrin Jalali, scikit-learn core maintainer and VP Labs at Probabl, and
Cailean Osborne, Head of Ecosystem Development at Probabl
Almost 7 years ago, Ralf Gommers from Quansight wrote an excellent blog post about the cost of an open source contribution, where he described what happens when a contribution comes into an open source project, and the subsequent challenges and bottlenecks for maintainers. Since then, open source has become even more mainstream. The number of contributors has gone up, while the number of maintainers in many projects has stayed more or less the same.
On top of that, with the expansion of AI tools and their general availability, an increasing number of people seem to be trying their luck at “vibe contributing” to open source. Recently, it has become rather straightforward to prompt an IDE or an agent to read guidelines, claim an issue, and submit a solution without the contributor ever truly understanding their code.
Maintainers like our colleagues, who maintain core data science libraries like scikit-learn and skore, are increasingly submerged in LLM-generated comments, auto-generated issues, and many low-quality PRs that do not advance the project, such as redundant contributions to README files and ones that claim significant performance gains while failing basic linting tests.
These come almost exclusively from “first-time contributors” who seem to want to have a contribution in our project without understanding their submissions. The issue has gotten so bad that at times almost every second issue on our main repo gets at least one such message, in many cases multiple ones, to the point where some of our peers dread opening issues since they don’t want to deal with interacting with AI.
While the cost of writing and contributing code has shrunk thanks to AI, the cost of reviewing and maintaining code hasn’t. Given these developments, maintainers from many open source communities have been deliberating what to do. Some reject all AI-generated contributions, saying they have never seen useful ones or that the risks of copyright infringement or license violations are simply too great. Meanwhile others are proactively trying to steer the types of AI-generated contributions they receive by implementing solutions like AI contribution policies and AGENTS.md files.
So, what should open source maintainers do? There is clearly no single way to deal with AI-generated contributions and by no means do we want to prescribe the best ways to go about it. Instead, we’ve summarized discussions and solutions we’ve seen lately in scikit-learn as well as other open source communities, with the hope that gathering this information in one place may be useful for others asking themselves the same question.
In “The Cost of AI in Open Source Maintenance,” I (Adrin) wrote about the types of AI-generated contributions to scikit-learn, the impacts on the maintainers, as well as recommendations to future contributors. As described there, in scikit-learn, we have encountered the following types of interactions with users who post LLM-generated content.
|
Type |
Description |
Impacts on maintainers |
|
New issues |
Bug reports or features generated without a real-world use case. |
Review time: Maintainers spend time triaging issues that no human actually requested, which distracts them from working on issues which could have more impact on actual users and/or are prioritized in the roadmap. |
|
New pull requests |
Code ranging from “vibe-coding” to sophisticated but unverified logic. |
Ghosting effects: New contributors vanish because they lack the skill to implement requested changes. |
|
PR reviews |
Automated “LGTM” or nitpicks left by agents on other contributors' work. |
Confusion: Misleads genuine contributors and requires immediate moderation to stop AI-to-AI loops or for contributors to spend time on pointless AI feedback. |
|
Issue grabbing |
Agents scan tags like “good first issue” and claim them. |
Noise: Multiple agents or users claim an issue before a human has triaged it as valid. |
|
Comments giving technical advice |
AI-generated comments attempting to solve a user’s problem or provide guidance. |
Misdirection: Provides plausible but incorrect advice that complicates the triage process. We’ve seen two AI powered users going down a rabbit hole of responding to one another, none of which with useful comments. |
|
Dataset creation |
Using active repos to test models or gather validation data for research. |
Free labor: Research groups treat maintainers as free evaluators for their models. |
Table 1: Types of AI-generated contributions to scikit-learn
These kinds of AI-generated contributions are creating a lot of extra work for us at scikit-learn and taking up valuable time that could be better used for other tasks like onboarding genuinely motivated contributors or advancing the priorities in our roadmap.
Our struggle isn't happening in a vacuum though. This has become an open source-wide challenge. A series of recent discussions show this trend affects open source communities across the board, and maintainers are debating different ways to adapt to this new reality.
Just to name a few, take a look at the list of discussions in the last month alone that our colleague and scikit-learn core developer, Stefanie Senger, shared in issue #31679 in scikit-learn:
If you’re a maintainer and you’re wondering what to do about AI-generated contributions to your project, perhaps some of the following approaches might be helpful.
Vague guidelines about AI use are not sufficient. In scikit-learn, we have codified our stance in our Automated Contributions Policy. Basically, contributions require human judgment. It states that maintainers reserve the right to close fully automated submissions and if judged appropriate, ban the user from the GitHub organisation. Contributors are also required to disclose AI usage. This works well because it empowers the maintainers to act decisively.
Below we’ve summarized acceptable and unacceptable uses that are mentioned in AI tool use policies and/or guidelines by open source projects and communities. Melissa Weber Mendonça of Quansight has also created this useful repo with policies of many other projects, which we recommend checking out.
|
Project or organization |
Acceptable Uses |
Unacceptable Uses |
|
• Assist with code changes on condition you understand and can explain them on request. • Assist with documentation changes IF you understand and can explain them on request. • Drafting for non-native English speakers (grammar/clarity). |
• Submitting fully-automated PRs or issues. • Submitting code you haven't personally tested or reviewed. • Pasting AI-generated text in descriptions or comments. • Not disclosing your use of an AI tool in your contribution. |
|
|
• Make sure a PR addresses the problem correctly and doesn’t introduce subtle errors. • Deliberate effort to understand code contribution. • Deliberate effort to ensure that code conforms to community norms. • Automating routine tasks once mastered. |
• Fully-automated submissions without a human-in-the-loop. • Failing to disclose the use of generative tools. • Submitting code that violates copyright and licensing requirements. |
|
|
• Assistance with writing comments, especially in a non-native language. • Understanding existing code. • Supplementing contributor knowledge for code, tests, and documentation. |
• Fully automated issues or PRs. • “Code churn” contributions (i.e. unnecessary code changes). • Using AI to rewrite entire code blocks. |
|
|
• Code, documentation, or images generated in whole or in part by AI are allowed, provided they meet specific legal and licensing criteria. • Marking contributions in source control using the generated-by: token in commit messages to ensure machine-parsable provenance. |
• Using AI tools whose terms and conditions restrict the use of output in a way that conflicts with open source standards. • Submitting AI-generated content without "reasonable certainty" that it's either free of third-party materials or uses them with proper permission. |
|
|
• Code or content generated "in whole or in part" by AI tools is permitted IF the tool's terms of service do not impose restrictions that conflict with the project’s license, IP policies, or the Open Source Definition. • Using AI output that contains pre-existing copyrighted or open source code IF you have confirmed permission and provided proper notice and attribution. |
• Using output from AI tools whose terms of service impose restrictions that conflict with the project’s license, IP policies, or the Open Source Definition. • Using AI output that contains pre-existing copyrighted or open source code, when you do not have confirmed permission and have not provided proper notice and attribution. |
|
|
• Speed up the workflow and automate routine coding tasks. • Secondary tool to help navigate and interpret the codebase. • Catch bugs or technical issues early in the development cycle. • Create production code, provided you take full responsibility for the result. |
• Fix a high volume of “Good First” or “Good Next” bugs. • Include private, security-sensitive, or confidential Firefox information in prompts sent to external tools. • Propose changes you cannot explain. • Generate contributions that are broad or poorly justified. |
|
|
• Search engine or discovery tool to help you understand the codebase. • Help with grammar or translation. • Help write good code, as long as you can explain every change. • Include an LLM-generated answer in a conversation only if it is placed in a Zulip quote block to distinguish it from your own thoughts. |
• Submitting any AI-generated PR that you haven't understood, tested, and verified. • Posting AI-generated messages in community channels. • Generating a PR description based on code changes (which usually just results in a list of files changed rather than the why). • Using AI-generated text that inaccurately describes your code changes, effects, or testing process. |
|
|
• Help you understand code or describe how it works. • Small, specific task assistance (e.g. code completion). • Find bugs, memory leaks, or proofread documentation and commit messages. • Use agents or prompts for small, manageable pieces of code that you already have a general idea of how to solve manually. |
• Over-delegation to AI and not reviewing outputs (“You, as a human programmer, must be in charge”). • Submitting code, documentation, or commit messages that you don’t understand. • Responding to questions or writing PR/issue justifications with AI-generated text. • Ignoring project conventions. • Generating overly long, fluffy, or marketing-heavy documentation. |
Table 2: Examples of AI tool use policies and/or guidelines
AI contribution policies come in many flavors. If you’re considering developing one for your project, you may want to take a look at the principles advocated by the Scientific Python community. In January 2026, Stefan van der Walt proposed four principles:
In the NumPy mailing list, Ralf Gommers recently added a fifth principle: “we want to interact with other humans, not machines,” which LLVM recently adopted in their AI Tool Policy.
More and more projects are adding an AGENTS.md file to their repo, which gives explicit instructions for how agents should interact in the repo. This is a strategy for driving up the quality of automated interactions by giving agents better rules to follow.
For example, the Apache Airflow maintainers added this AGENTS.md file to their repo. Jarek Potiuk from Airflow has commented that historically they had very good human-targeted documentation for contributors, and they suspect that it helped drive up the quality of AI-generated contributions. Jarek has recommended developing good instructions and embracing the AI contributions that follow.
Jarek Potiuk from Apache Airflow has also recommended developing bespoke tooling and instructions for AI use, and in turn deliberately inviting contributors to use AI in their contributions accordingly. In the Apache Airflow contributing docs, they list guidelines for contributors who use AI tools to help them create PRs. For example, here they mention that for a large-scale UI documentation translation task, they “developed custom tooling that helped to more easily apply regular tools like coding LLM integration to aid translation efforts.” Their guidelines also mention how the maintainers may proceed if a contributor ignores, repeatedly ignores, or spams the project.
The “good first issue” label has become a magnet for automated agents. Thibaud Colas from wagtail shared that they are now using matchmaking programs like Djangonaut Space or Outreachy to onboard new contributors. These programs are working for them because they ensure a sustained commitment and human-to-human connection that AI tools cannot replicate.
In the NumPy mailing list, Ralf Gommers from Quansight recently challenged the belief that AI tools are just making contributors dumber, suggesting they can actually facilitate learning. For example, they can be used to automate routine tasks once mastered, brainstorm design options, and write documentation that maintainers may be too busy to produce, among others.
Now, let’s turn to open source contributors who use AI tools.
In this day and age, it would be unreasonable to expect folks not to use any AI tools. Many of us use these tools one way or another. However, you should never submit contributions without understanding what you’re submitting.
Basically, if you are using AI tools to make open source contributions, the goal should be to reach a state of understanding where you no longer need the tool to explain your own work. You should also be spending at least as much time creating your contribution, as it takes a maintainer to review it. LLVM’s golden rule is that a contribution should be worth more to the project than the time it takes to review it.
As mentioned in the September blog post, below are some recommended ways that you can engage with your AI tools when contributing to open source:
Below are additional recommendations from others:
Our closing message to contributors is: We want to work with you, mentor you, and see you make genuine attempts to solve problems. If you use AI tools to empower your curiosity–if you are reviewing, testing, and understanding every line you submit–then you are the kind of contributor we are excited to welcome. If you want to get involved in scikit-learn, check out our guidance for first-time contributors here.
There are many ways maintainers may decide to tackle the surge of AI-generated contributions to open source projects, and ultimately it’s up to you and your community to build consensus on your way to go.
Our bottom line is that since AI tool usage is so prominent now, we should be proactive and intentional about shaping good policies and practices. This is in the long-term interest of our projects and open source in general. This means sharing our learnings, engaging in debate, and building resources together.