Training Example: Apache Spark – Review the Data, Give Your Score & Compare to the Real AI Evaluation

Adaptive Query Execution
Spark SQL adapts the execution plan at runtime, such as automatically setting the number of reducers and join algorithms.

Support for ANSI SQL
Use the same SQL you’re already comfortable with.

Structured and unstructured data
Spark SQL works on structured tables and unstructured data such as JSON or images.

TPC-DS 1TB No-Stats With vs. Without Adaptive Query Execution
Accelerates TPC-DS queries up to 8x

[H2] Apache Spark™ community
[H3] Have questions?
[H4] StackOverflow
For usage questions and help (e.g. how to use this Spark API), it is recommended you use the
StackOverflow tag apache-spark
as it is an active forum for Spark users’ questions and answers.
Some quick tips when using StackOverflow:
Prior to asking submitting questions, please:
Search StackOverflow’s
apache-spark tag to see if
your question has already been answered
Search the ASF archive for
user@spark.apache.org

Please follow the StackOverflow code of conduct
Always use the apache-spark tag when asking questions
Please also use a secondary tag to specify components so subject matter experts can more easily find them.
Examples include: pyspark, spark-dataframe, spark-streaming, spark-r, spark-mllib,
spark-ml, spark-graphx, spark-graphframes, spark-tensorframes, etc.
Please do not cross-post between StackOverflow and the mailing lists
No jobs, sales, or solicitation is permitted on StackOverflow
[H4] Mailing lists
For broad, opinion based, ask for external resources, debug issues, bugs, contributing to the
project, and scenarios, it is recommended you use the user@spark.apache.org mailing list.
user@spark.apache.org is for usage questions, help, and announcements.
(subscribe)
(unsubscribe)
(archives)

dev@spark.apache.org is for people who want to contribute code to Spark.
(subscribe)
(unsubscribe)
(archives)
Some quick tips when using email:
Prior to asking submitting questions, please:
Search StackOverflow at apache-spark
to see if your question has already been answered
Search the ASF archive for
user@spark.apache.org

Tagging the subject line of your email will help you get a faster response, e.g.
[Spark SQL]: Does Spark SQL support LEFT SEMI JOIN?
Tags may help identify a topic by:
Component: Spark Core, Spark SQL, ML, MLlib, GraphFrames, GraphX, TensorFrames, etc
Level: Beginner, Intermediate, Advanced
Scenario: Debug, How-to

For error logs or long code examples, please use GitHub gist
and include only a few lines of the pertinent code / log within the email.
No jobs, sales, or solicitation is permitted on the Apache Spark mailing lists.
[H5] Reporting security issues
Please see the Security page for information on how to report sensitive security
vulnerabilities, and for information on known security issues.
[H4] Chat rooms
Chat rooms are great for quick questions or discussions on specialized topics. The following chat rooms are not officially part of Apache Spark; they are provided for reference only.

Spark with Scala is for questions and discussions related to using Spark with the Scala programming language.
[H3] Events and meetups
[H4] Conferences
The Spark Summit website archives at spark-summit.org, from years 2013 to 2017, are no longer available, so the links to them have been removed. Instead, we have
most of the video recordings.

Spark Summit (2013-2017) video playlist
[H4] Meetups
Spark Meetups are grass-roots events organized and hosted by individuals in the community around the world. Check out meetup.com/topics/apache-spark to find a Spark meetup in your part of the world. Below is a partial list of Spark meetups.

Bay Area Spark Meetup.
This group has been running since January 2012 in the San Francisco area.
The meetup page also contains an archive of past meetups.

Barcelona Spark Meetup

Bogotá Spark Meetup

Boston Spark Meetup

Boulder/Denver Spark Meetup

Hangzhou Spark Meetup

Israel Spark Users

London Spark Meetup

Maryland Spark Meetup

México City Spark Meetup

NYC Spark Meetup

Seattle Spark Meetup

Shanghai Spark Meetup

Shenzhen Spark Meetup

Tokyo Spark Meetup

Vancouver Spark Meetup

Washington DC Area Spark Meetup
If you'd like your meetup or conference added, please email user@spark.apache.org.
[H3] Issue tracker
The project tracks bugs and new features on JIRA. If you'd like, you can also subscribe to issues@spark.apache.org to receive emails about new issues, and commits@spark.apache.org to get emails about commits.
[H3] Powered by
Our site has a list of projects and organizations powered by Spark.
Add yours by emailing `dev@spark.apache.org`.

[H5] Latest News

Spark 4.1.2 released
(May 21, 2026)
Preview release of Spark 4.2.0
(May 01, 2026)
Preview release of Spark 4.2.0
(Apr 09, 2026)
Preview release of Spark 4.2.0
(Mar 12, 2026)

Archive

Download Spark

Built-in Libraries:

SQL and DataFrames
Spark Streaming
MLlib (machine learning)
GraphX (graph)
Third-Party Projects

This guide documents the best way to make various types of contribution to Apache Spark,
including what is required before submitting a code change.
Contributing to Spark doesn’t just mean writing code. Helping new users on the mailing list,
testing releases, and improving documentation are also welcome. In fact, proposing significant
code changes usually requires first gaining experience and credibility within the community by
helping in other ways. This is also a guide to becoming an effective contributor.
So, this guide organizes contributions in order that they should probably be considered by new
contributors who intend to get involved long-term. Build some track record of helping others,
rather than just open pull requests.
[H2] Contributing by helping other users
A great way to contribute to Spark is to help answer user questions on the user@spark.apache.org
mailing list or on StackOverflow. There are always many new Spark users; taking a few minutes to
help answer a question is a very valuable community service.
Contributors should subscribe to this list and follow it in order to keep up to date on what’s
happening in Spark. Answering questions is an excellent and visible way to help the community,
which also demonstrates your expertise.
See the Mailing Lists guide for guidelines
about how to effectively participate in discussions on the mailing list, as well as forums
like StackOverflow.
[H2] Contributing by testing releases
Spark’s release process is community-oriented, and members of the community can vote on new
releases on the dev@spark.apache.org mailing list. Spark users are invited to subscribe to
this list to receive announcements, and test their workloads on newer release and provide
feedback on any performance or correctness issues found in the newer release.
[H2] Contributing by reviewing changes
Changes to Spark source code are proposed, reviewed and committed via
GitHub pull requests (described later).
Anyone can view and comment on active changes here.
Reviewing others’ changes is a good way to learn how the change process works and gain exposure
to activity in various parts of the code. You can help by reviewing the changes and asking
questions or pointing out issues – as simple as typos or small issues of style.
See also https://spark-prs.appspot.com/ for a
convenient way to view and filter open PRs.
[H2] Contributing documentation changes
To propose a change to release documentation (that is, docs that appear under
https://spark.apache.org/docs/),
edit the Markdown source files in Spark’s
docs/ directory,
whose README file shows how to build the documentation locally to test your changes.
The process to propose a doc change is otherwise the same as the process for proposing code
changes below.
To propose a change to the rest of the documentation (that is, docs that do not appear under
https://spark.apache.org/docs/), similarly, edit the Markdown in the
spark-website repository and open a pull request.
[H2] Contributing user libraries to Spark
Just as Java and Scala applications can access a huge selection of libraries and utilities,
none of which are part of Java or Scala themselves, Spark aims to support a rich ecosystem of
libraries. Many new useful utilities or features belong outside of Spark rather than in the core.
For example: language support probably has to be a part of core Spark, but, useful machine
learning algorithms can happily exist outside of MLlib.
To that end, large and independent new functionality is often rejected for inclusion in Spark
itself, but, can and should be hosted as a separate project and repository, and included in
the spark-packages.org collection.
[H2] Contributing bug reports
Ideally, bug reports are accompanied by a proposed code change to fix the bug. This isn’t
always possible, as those who discover a bug may not have the experience to fix it. A bug
may be reported by creating a JIRA but without creating a pull request (see below).
Bug reports are only useful however if they include enough information to understand, isolate
and ideally reproduce the bug. Simply encountering an error does not mean a bug should be
reported; as below, search JIRA and search and inquire on the Spark user / dev mailing lists
first. Unreproducible bugs, or simple error reports, may be closed.
It’s very helpful if the bug report has a description about how the bug was introduced, by
which commit, so that reviewers can easily understand the bug. It also helps committers to
decide how far the bug fix should be backported, when the pull request is merged. The pull
request to fix the bug should narrow down the problem to the root cause.
Performance regression is also one kind of bug. The pull request to fix a performance regression
must provide a benchmark to prove the problem is indeed fixed.
Note that, data correctness/data loss bugs are very serious. Make sure the corresponding bug
report JIRA ticket is labeled as correctness or data-loss. If the bug report doesn’t get
enough attention, please send an email to dev@spark.apache.org, to draw more attentions.
It is possible to propose new features as well. These are generally not helpful unless
accompanied by detail, such as a design document and/or code change. Large new contributions
should consider spark-packages.org first (see above),
or be discussed on the mailing
list first. Feature requests may be rejected, or closed after a long period of inactivity.
[H2] Contributing to JIRA maintenance
Given the sheer volume of issues raised in the Apache Spark JIRA, inevitably some issues are
duplicates, or become obsolete and eventually fixed otherwise, or can’t be reproduced, or could
benefit from more detail, and so on. It’s useful to help identify these issues and resolve them,
either by advancing the discussion or even resolving the JIRA. Most contributors are able to
directly resolve JIRAs. Use judgment in determining whether you are quite confident the issue
should be resolved, although changes can be easily undone. If in doubt, just leave a comment
on the JIRA.
When resolving JIRAs, observe a few useful conventions:
Resolve as Fixed if there’s a change you can point to that resolved the issue
Set Fix Version(s), if and only if the resolution is Fixed
Set Assignee to the person who most contributed to the resolution, which is usually the person
who opened the PR that resolved the issue.
In case several people contributed, prefer to assign to the more ‘junior’, non-committer contributor

For issues that can’t be reproduced against master as reported, resolve as Cannot Reproduce
Fixed is reasonable too, if it’s clear what other previous pull request resolved it. Link to it.

If the issue is the same as or a subset of another issue, resolve as Duplicate
Make sure to link to the JIRA it duplicates
Prefer to resolve the issue that has less activity or discussion as the duplicate

If the issue seems clearly obsolete and applies to issues or components that have changed
radically since it was opened, resolve as Not a Problem
If the issue doesn’t make sense – not actionable, for example, a non-Spark issue, resolve
as Invalid
If it’s a coherent issue, but there is a clear indication that there is not support or interest
in acting on it, then resolve as Won’t Fix
Umbrellas are frequently marked Done if they are just container issues that don’t correspond
to an actionable change of their own
[H2] Preparing to contribute code changes
[H3] Choosing what to contribute
Spark is an exceptionally busy project, with a new JIRA or pull request every few hours on average.
Review can take hours or days of committer time. Everyone benefits if contributors focus on
changes that are useful, clear, easy to evaluate, and already pass basic checks.
Sometimes, a contributor will already have a particular new change or bug in mind. If seeking
ideas, consult the list of starter tasks in JIRA, or ask the user@spark.apache.org mailing list.
Before proceeding, contributors should evaluate if the proposed change is likely to be relevant,
new and actionable:
Is it clear that code must change? Proposing a JIRA and pull request is appropriate only when a
clear problem or change has been identified. If simply having trouble using Spark, use the mailing
lists first, rather than consider filing a JIRA or proposing a change. When in doubt, email
user@spark.apache.org first about the possible change
Search the user@spark.apache.org and dev@spark.apache.org mailing list
archives for
related discussions.
Often, the problem has been discussed before, with a resolution that doesn’t require a code
change, or recording what kinds of changes will not be accepted as a resolution.
Search JIRA for existing issues:
https://issues.apache.org/jira/browse/SPARK
Type spark [search terms] at the top right search box. If a logically similar issue already
exists, then contribute to the discussion on the existing JIRA and pull request first, instead of
creating a new one.
Is the scope of the change matched to the contributor’s level of experience? Anyone is qualified
to suggest a typo fix, but refactoring core scheduling logic requires much more understanding of
Spark. Some changes require building up experience first (see above).
It’s worth reemphasizing that changes to the core of Spark, or to highly complex and important modules
like SQL and Catalyst, are more difficult to make correctly. They will be subjected to more scrutiny,
and held to a higher standard of review than changes to less critical code.
[H3] MLlib-specific contribution guidelines
While a rich set of algorithms is an important goal for MLLib, scaling the project requires
that maintainability, consistency, and code quality come first. New algorithms should:
Be widely known
Be used and accepted (academic citations and concrete use cases can help justify this)
Be highly scalable
Be well documented
Have APIs consistent with other algorithms in MLLib that accomplish the same thing
Come with a reasonable expectation of developer support.
Have @Since annotation on public classes, methods, and variables.
[H3] Error message guidelines
Exceptions thrown in Spark should be associated with standardized and actionable
error messages.
Error messages should answer the following questions:
What was the problem?
Why did the problem happen?
How can the problem be solved?
When writing error messages, you should:
Use active voice
Avoid time-based statements, such as promises of future support
Use the present tense to describe the error and provide suggestions
Provide concrete examples if the resolution is unclear
Avoid sounding accusatory, judgmental, or insulting
Be direct
Do not use programming jargon in user-facing errors
See the error message guidelines for more details.
[H3] Behavior changes
Behavior changes are user-visible functional changes in a new release via public APIs. The term ‘user’ here refers
not only to those who write queries and/or develop Spark plugins, but also to those who deploy and/or manage Spark
clusters. New features and bug fixes, such as correcting query results or schemas and failing unsupported queries
that previously returned incorrect results, are considered behavior changes. However, performance improvements,
code refactoring, and changes to unreleased APIs/features are not.
Everyone makes mistakes, including Spark developers. We will continue to fix defects in Spark as they arise.
However, it is important to communicate these behavior changes so that Spark users can be prepared for version
upgrades. If a PR introduces behavior changes, it should be explicitly mentioned in the PR description. If the
behavior change may require additional user actions, this should be highlighted in the migration guide
(docs/sql-migration-guide.md for the SQL component and similar files for other components). Where possible,
provide options to restore the previous behavior and mention these options in the error message. Some examples include:
Bug fixes that change query results. Users may need to backfill to correct existing data and must be informed about
these correctness fixes.
Bug fixes that change the query schema. Users may need to update the schema of tables in their data pipelines and must
be informed about these changes.
Removing or renaming Spark configurations.
Renaming error classes or conditions.
Any non-additive changes to the public Python/SQL/Scala/Java/R APIs (including developer APIs), such as renaming
functions, removing parameters, adding parameters, renaming parameters, or changing parameter default values. These
changes should generally be avoided, or if necessary, done in a binary-compatible manner by deprecating the old function
and introducing a new one instead.
Any non-additive changes to the way Spark should be deployed and managed: renaming argument names in deployment scripts,
updates to the REST API, changes to the method of loading configuration files, etc.
This list is not meant to be comprehensive. Anyone reviewing a PR can ask the PR author to add to the migration guide
if they believe the change is risky and may disrupt users during an upgrade.
[H3] Code review criteria
Before considering how to contribute code, it’s useful to understand how code is reviewed,
and why changes may be rejected. See the
detailed guide for code reviewers
from Google’s Engineering Practices documentation.
Simply put, changes that have many or large
positives, and few negative effects or risks, are much more likely to be merged, and merged quickly.
Risky and less valuable changes are very unlikely to be merged, and may be rejected outright
rather than receive iterations of review.
[H4] Positives
Fixes the root cause of a bug in existing functionality
Adds functionality or fixes a problem needed by a large number of users
Simple, targeted
Maintains or improves consistency across Python, Java, Scala
Easily tested; has tests
Reduces complexity and lines of code
Change has already been discussed and is known to committers
[H4] Negatives, risks
Band-aids a symptom of a bug only
Introduces complex new functionality, especially an API that needs to be supported
Adds complexity that only helps a niche use case
Adds user-space functionality that does not need to be maintained in Spark, but could be hosted
externally and indexed by spark-packages.org
Changes a public API or semantics (rarely allowed)
Adds large dependencies
Changes versions of existing dependencies
Adds a large amount of code
Makes lots of modifications in one “big bang” change
[H2] Contributing code changes
Please review the preceding section before proposing a code change. This section documents how to do so.
When you contribute code, you affirm that the contribution is your original work and that you
license the work to the project under the project’s open source license. Whether or not you state
this expl

[H2] Current committers

Name
Organization

Sameer Agarwal
Deductive AI

Michael Armbrust
Databricks

Dilip Biswal
Adobe

Ryan Blue
Databricks

Joseph Bradley
Databricks

Matthew Cheah
Palantir

Felix Cheung
NVIDIA

Mosharaf Chowdhury
University of Michigan, Ann Arbor

Bryan Cutler
IBM

Jason Dai
Intel

Tathagata Das
Databricks

Ankur Dave
Databricks

Aaron Davidson
Databricks

Thomas Dudziak
Meta

Erik Erlandson
Red Hat

Robert Evans
NVIDIA

Wenchen Fan
Databricks

Huaxin Gao
Apple

Max Gekk
Databricks

Jiaan Geng
NetEase

Joseph Gonzalez
UC Berkeley

Thomas Graves
NVIDIA

Martin Grund
Databricks

Stephen Haberman
LinkedIn

Mark Hamstra
ClearStory Data

Seth Hendrickson
Stripe

Herman van Hovell
Databricks

Liang-Chi Hsieh
Databricks

Yin Huai
Databricks

Shane Huang
Intel

Dongjoon Hyun
Apple

Kazuaki Ishizaki
IBM

Xingbo Jiang
Databricks

Yikun Jiang
Huawei

Holden Karau
Fight Health Insurance

Shane Knapp
UC Berkeley

Cody Koeninger

Andy Konwinski
Databricks

Hyukjin Kwon
Databricks

Ryan LeCompte
Quantifind

Haejoon Lee
Databricks

Haoyuan Li
Alluxio

Xiao Li
Databricks

Yinan Li
Google

Yuanjian Li
Databricks

Davies Liu
Juicedata

Cheng Lian
Databricks

Yanbo Liang
Facebook

Jungtaek Lim
Databricks

Sean McNamara
Oracle

Xiangrui Meng
Databricks

Xinrong Meng
Databricks

Mridul Muralidharan
LinkedIn

Anton Okolnychyi
Databricks

Andrew Or
Facebook

Kay Ousterhout
LightStep

Sean Owen
Databricks

Bingkun Pan
Baidu

Cheng Pan
NetEase

Tejas Patil
Meta

Nick Pentreath
Automattic

Attila Zsolt Piros
Cloudera

Anirudh Ramanathan
Signadot

Imran Rashid
Cloudera

Charles Reiss
University of Virginia

Josh Rosen
Databricks

Sandy Ryza
Databricks

Kousuke Saruta
AWS

Saisai Shao
Datastrato

Prashant Sharma
IBM

Anish Shrigondekar
Databricks

Gabor Somogyi
Apple

Ram Sriharsha
Pinecone

Chao Sun
OpenAI

Maciej Szymkiewicz

Daniel Tenedorio
Databricks

Jose Torres
Databricks

Peter Toth
Apple

DB Tsai
Databricks

Takuya Ueshin

Marcelo Vanzin
Cloudera

Shivaram Venkataraman
University of Wisconsin, Madison

Allison Wang
Databricks

Gengliang Wang
Databricks

Yuming Wang
eBay

Zhenhua Wang
Alibaba

Patrick Wendell
Databricks

Yi Wu
Databricks

Andrew Xia
Alibaba

Reynold Xin
Databricks

Weichen Xu
Databricks

Takeshi Yamamuro
NTT

Jie Yang
Baidu

Kent Yao
Microsoft

Burak Yavuz
Databricks

Xiduo You
NetEase

Matei Zaharia
Databricks, Stanford

Ruifeng Zheng
Databricks

Shixiong Zhu
Databricks

[H3] Becoming a committer
To get started contributing to Spark, learn
how to contribute –
anyone can submit patches, documentation and examples to the project.
The PMC regularly adds new committers from the active contributors, based on their contributions
to Spark. The qualifications for new committers include:
Sustained contributions to Spark: Committers should have a history of major contributions to
Spark. An ideal committer will have contributed broadly throughout the project, and have
contributed at least one major component where they have taken an “ownership” role. An ownership
role means that existing contributors feel that they should run patches for this component by
this person.
Quality of contributions: Committers more than any other community member should submit simple,
well-tested, and well-designed patches. In addition, they should show sufficient expertise to be
able to review patches, including making sure they fit within Spark’s engineering practices
(testability, documentation, API stability, code style, etc). The committership is collectively
responsible for the software quality and maintainability of Spark. Note that contributions to
critical parts of Spark, like its core and SQL modules, will be held to a higher standard when
assessing quality. Contributors to these areas will face more review of their changes.
Community involvement: Committers should have a constructive and friendly attitude in all
community interactions. They should also be active on the dev and user list and help mentor
newer contributors and users. In design discussions, committers should maintain a professional
and diplomatic approach, even in the face of disagreement.
The Apache Way: Committers should follow and
understand The Apache Way such as
Lazy Consensus.
Apache projects are managed independently.
A community that obviously favors one specific vendor in some exclusive way will often
discourage new contributors from competing vendors, and this would be an issue for the
long-term health of the project.
The type and level of contributions considered may vary by project area – for example, we
greatly encourage contributors who want to work on mainly the documentation, or mainly on
platform support for specific OSes, storage systems, etc.
The PMC also adds new PMC members. PMC members are expected to carry out PMC
responsibilities as described in Apache Guidance, including
helping vote on releases, enforce Apache project trademarks, take responsibility for legal and license issues,
and ensure the project follows Apache project mechanics. The PMC periodically adds committers to the PMC
who have shown they understand and can help with these activities.
[H3] Review process
All contributions should be reviewed before merging as described in
Contributing to Spark.
In particular, if you are working on an area of the codebase you are unfamiliar with, look at the
Git history for that code to see who reviewed patches before. You can do this using
git log --format=full <filename>, by examining the “Commit” field to see who committed each patch.
[H3] When to commit/merge a pull request
PRs shall not be merged during active, on-topic discussion unless they address issues such as critical security fixes of a public vulnerability. Under extenuating circumstances, PRs may be merged during active, off-topic discussion and the discussion directed to a more appropriate venue. Time should be given prior to merging for those involved with the conversation to explain if they believe they are on-topic.
Lazy consensus requires giving time for discussion to settle while understanding that people may not be working on Spark as their full-time job and may take holidays. It is believed that by doing this, we can limit how often people feel the need to exercise their veto.
All -1s with justification merit discussion. A -1 from a non-committer can be overridden only with input from multiple committers, and suitable time must be offered for any committer to raise concerns. A -1 from a committer who cannot be reached requires a consensus vote of the PMC under ASF voting rules to determine the next steps within the ASF guidelines for code vetoes.
These policies serve to reiterate the core principle that code must not be merged with a pending veto or before a consensus has been reached (lazy or otherwise).
It is the PMC’s hope that vetoes continue to be infrequent, and when they occur, that all parties will take the time to build consensus prior to additional feature work.
Being a committer means exercising your judgement while working in a community of people with diverse views. There is nothing wrong in getting a second (or third or fourth) opinion when you are uncertain. Thank you for your dedication to the Spark project; it is appreciated by the developers and users of Spark.
It is hoped that these guidelines do not slow down development; rather, by removing some of the uncertainty, the goal is to make it easier for us to reach consensus. If you have ideas on how to improve these guidelines or other Spark project operating procedures, you should reach out on the dev@ list to start the discussion.
[H3] How to merge a pull request
Changes pushed to the master branch on Apache cannot be removed; that is, we can’t force-push to
it. So please don’t add any test commits or anything like that, only real patches.
[H4] Setting up remotes
To use the merge_spark_pr.py script described below, you
will need to add a git remote called apache at https://github.com/apache/spark,
as well as one called apache-github at git://github.com/apache/spark.
The apache (the default value of PUSH_REMOTE_NAME environment variable) is the remote used for pushing the squashed commits
and apache-github (default value of PR_REMOTE_NAME) is the remote used for pulling the changes.
By using two separate remotes for these two actions the result of the merge_spark_pr.py can be tested without pushing it
into the official Spark repo just by specifying your fork in the PUSH_REMOTE_NAME variable.
After cloning your fork of Spark you already have a remote origin pointing there. So if correct, your git remote -v
contains at least these lines:
apache git@github.com:apache/spark.git (fetch)
apache git@github.com:apache/spark.git (push)
apache-github git@github.com:apache/spark.git (fetch)
apache-github git@github.com:apache/spark.git (push)
origin git@github.com:[your username]/spark.git (fetch)
origin git@github.com:[your username]/spark.git (push)
For the apache repo, you will need to set up command-line authentication to GitHub. This may
include setting up an SSH key and/or personal access token. See:
https://docs.github.com/en/authentication/connecting-to-github-with-ssh
https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens
To check whether the necessary write access are already granted please visit GitBox.
Ask dev@spark.apache.org if you have trouble with these steps, or want help doing your first merge.
[H4] Merge script
All merges should be done using the
dev/merge_spark_pr.py,
which squashes the pull request’s changes into one commit.
The script is fairly self explanatory and walks you through steps and options interactively.
If you want to amend a commit before merging – which should be used for trivial touch-ups –
then simply let the script wait at the point where it asks you if you want to push to Apache.
Then, in a separate window, modify the code and push a commit. Run git rebase -i HEAD~2 and
“squash” your new commit. Edit the commit message just after to remove your commit message.
You can verify the result is one change with git log. Then resume the script in the other window.
Also, please remember to set Assignee on JIRAs where applicable when they are resolved. The script
can do this automatically in most cases.
Once a PR is merged please leave a comment on the PR stating which branch(es) it has been merged with.
[H3] Policy on backporting bug fixes
From pwendell:
The trade off when backporting is you get to deliver the fix to people running older versions
(great!), but you risk introducing new or even worse bugs in maintenance releases (bad!).
The decision point is when you have a bug fix and it’s not clear whether it is worth backporting.
I think the following facets are important to consider:
Backports are an extremely valuable service to the community and should be considered for
any bug fix.
Introducing a new bug in a maintenance release must be avoided at all costs. It over time would
erode confidence in our release process.
Distributions or advanced users can always backport risky patches on their own, if they see fit.
For me, the consequence of these is that we should backport in the following situations:
Both the bug and the fix are well understood and isolated. Code being modified is well tested.
The bug being addressed is high priority to the community.
The backported fix does not vary widely from the master branch fix.
We tend to avoid backports in the converse situations:
The bug or fix are not well understood. For instance, it relates to interactions between complex
components or third party libraries (e.g. Hadoop libraries). The code is not well tested outside
of the immediate bug being fixed.
The bug is not clearly a high priority for the community.
The backported fix is widely different from the master branch fix.

[H5] Latest News

Spark 4.1.2 released
(May 21, 2026)
Preview release of Spark 4.2.0
(May 01, 2026)
Preview release of Spark 4.2.0
(Apr 09, 2026)
Preview release of Spark 4.2.0
(Mar 12, 2026)

Archive

Download Spark

Built-in Libraries:

SQL and DataFrames
Spark Streaming
MLlib (machine learning)
GraphX (graph)
Third-Party Projects