diff --git a/site/versioning-policy.html b/site/versioning-policy.html index 97727c04f1..2239caf8cf 100644 --- a/site/versioning-policy.html +++ b/site/versioning-policy.html @@ -164,19 +164,33 @@

Spark versions

Each Spark release will be versioned: [MAJOR].[FEATURE].[MAINTENANCE]

+ +

Each feature release will have a merge window where new patches can be merged, a QA window when only fixes can be merged, then a final period where voting occurs on release candidates. These windows will be announced immediately after the previous feature release to give people plenty -of time, and over time, we might make the whole release process more regular (similar to Ubuntu). -

  • MAINTENANCE: Maintenance releases will occur more frequently and depend on specific patches -introduced (e.g. bug fixes) and their urgency. In general these releases are designed to patch bugs. -However, higher level libraries may introduce small features, such as a new algorithm, provided -they are entirely additive and isolated from existing code paths. Spark core may not introduce -any features.
    • +of time.

    +

    Alpha components

    @@ -274,9 +288,10 @@

    Alternatives to breaking an API

    Release cadence

    -

    The branch is cut every January and July, so feature (“minor”) releases occur about every 6 months in general. -Hence, Spark 2.3.0 would generally be released about 6 months after 2.2.0. Maintenance releases happen as needed -in between feature releases. Major releases do not happen according to a fixed schedule.

    +

    Starting with Apache Spark 4.3.0 (the first quarterly feature release after 4.2.0), feature releases occur quarterly (every 3 months), containing new features, +improvements, and bug fixes. Major releases occur annually (every 12 months), typically in a stable window once per year (subject to release discussion on the dev list), allowing breaking +changes and dependency upgrades. Maintenance releases happen as needed in between for critical +bug fixes and security patches.

    Spark 4.2 release window

    @@ -303,16 +318,78 @@

    Spark 4.2 release window

    +

    Illustrative transition: 2026 and 2027

    + +

    The calendar below is an example to show how the community expects to bootstrap the faster cadence; +exact dates stay subject to the usual release-discussion and voting process.

    + + +

    Maintenance releases and EOL

    -

    Feature release branches will, generally, be maintained with bug fix releases for a period of 18 months. -For example, branch 2.3.x is no longer considered maintained as of September 2019, 18 months after the release -of 2.3.0 in February 2018. No more 2.3.x releases should be expected after that point, even for bug fixes.

    +

    We plan to ship a release every 3 months. Every fourth release bumps the major version (x.0.0). +Within each major line, the first release is the major release, the releases in between are feature +releases, and the last release in the line is the LTS release.

    + +

    The following table summarizes the maintenance window for each release type:

    + + + + + + + + + + + + + + + + + + + + + + + + + + +
    Release TypeMaintenance Window
    Major (x.0.0)6 months
    Feature (x.[1–2].0)6 months
    LTS (last x.y.0 in the major line)18 months
    Maintenance (x.y.z, z ≥ 1)N/A (patches only)
    + +

    Release branches other than LTS will, generally, be maintained with bug fix releases for a period of +6 months (see the Major and Feature rows in the table above).

    + +

    Under this cadence, the last release in each major line is designated as the LTS (Long-Term +Support) release for that major line and is maintained for 18 months (for example Apache Spark +5.3.0 and 6.3.0). The x.0.0, x.1.0, and x.2.0 trains are maintained for 6 months +each as usual. LTS releases provide a stable target for ecosystem projects and downstream vendors to +standardize around.

    + +

    As an exception while the project transitions into this cadence, Spark 4.x still ships 4.3.0 as the +first quarterly release on the new train, but the 4.x LTS will be Apache Spark 4.5.0 (the last +planned 4.x release) rather than 4.3.0. 4.5.x therefore receives the 18-month LTS +window in the same role as the last release on newer major lines.

    -

    The last minor release within a major release will typically be maintained for longer as an “LTS” release. -For example, 3.5.0 was released on September 13th 2023 and would normally be maintained for 31 months until April 12th 2026.

    +

    Critical security patches will be backported to all actively maintained branches. Critical bug fixes +(e.g., correctness issues) that may introduce behavior changes will be evaluated by the community +on a case-by-case basis.

    -

    As an exception from the normal versioning policy, version 3.5.x has an “extended” LTS period to allow for migrations to be completed. This extended LTS period will end November 2027. During the 3.5.x extended LTS period, we will only include security fixes. This extend LTS only applies to the primary Apache Spark project/repo and does not apply to sub projects with separate repos/releases (namely: Spark Connect for Swift/Rust/Go and Spark Kubernetes operator). Additionally, as Java 8 support may be removed from other projects (including Hadoop), should a dependency have a security fix that is not backported to a Java 8 compatible version we may decide to mark that vulnerability as a won’t fix or release the new version without Java 8 support.

    +

    As an exception from the normal versioning policy, version 3.5.x has an “extended” LTS period to allow for migrations to be completed. This extended LTS period will end November 2027. During the 3.5.x extended LTS period, we will only include security fixes. This extended LTS only applies to the primary Apache Spark project/repo and does not apply to sub projects with separate repos/releases (namely: Spark Connect for Swift/Rust/Go and Spark Kubernetes operator). Additionally, as Java 8 support may be removed from other projects (including Hadoop), should a dependency have a security fix that is not backported to a Java 8 compatible version we may decide to mark that vulnerability as a won’t fix or release the new version without Java 8 support.

    diff --git a/versioning-policy.md b/versioning-policy.md index cce0805b08..cdf7a616f9 100644 --- a/versioning-policy.md +++ b/versioning-policy.md @@ -17,19 +17,28 @@ These small differences account for Spark's nature as a multi-module project. Each Spark release will be versioned: `[MAJOR].[FEATURE].[MAINTENANCE]` -- **MAJOR**: All releases with the same major version number will have API compatibility. -Major version numbers will remain stable over long periods of time. For instance, 1.X.Y may last -1 year or more. -- **FEATURE**: Feature releases will typically contain new features, improvements, and bug fixes. +- **MAJOR**: Major releases occur annually (every 12 months) as **x.0.0** releases. These releases may +include **breaking changes**, third-party dependency upgrades, API removals and deprecations, and other changes that are not +compatible with the previous major line. Within a major line, all **x.y.z** releases share API compatibility +as described in the **FEATURE** and **MAINTENANCE** bullets below. +- **FEATURE**: Feature releases (**x.y.0** with **y ≥ 1**) occur quarterly (every 3 months) and contain new features, performance +improvements, API additions, and bug fixes. To ensure safe and predictable upgrades for downstream +projects, feature releases have the following compatibility: + - No third-party dependency upgrades (e.g. Parquet, Arrow, ORC, Hadoop, Netty) by default. Upgrades + required to address **security** issues may be allowed; any other exception is decided case by case by + the release managers and the community. + - No behavior or semantic changes (e.g. SQL semantics, execution behavior, configuration defaults). + Optimizer changes that only improve performance while preserving query results are allowed; optimizer + changes that alter results are not. Exceptions might occur case by case (e.g., security issues). + - Public APIs may be added but not changed or removed. + Each feature release will have a merge window where new patches can be merged, a QA window when only fixes can be merged, then a final period where voting occurs on release candidates. These windows will be announced immediately after the previous feature release to give people plenty -of time, and over time, we might make the whole release process more regular (similar to Ubuntu). -- **MAINTENANCE**: Maintenance releases will occur more frequently and depend on specific patches -introduced (e.g. bug fixes) and their urgency. In general these releases are designed to patch bugs. -However, higher level libraries may introduce small features, such as a new algorithm, provided -they are entirely additive and isolated from existing code paths. Spark core may not introduce -any features. +of time. +- **MAINTENANCE**: Maintenance releases (**x.y.z** with **z ≥ 1**) will occur on an ad hoc basis depending on specific patches +introduced (e.g. critical bug fixes and security patches) and their urgency. They contain bug fixes and +security patches only; they do not introduce new features.

    Alpha components

    @@ -96,9 +105,10 @@ In cases where there is a "Bad API", but where the cost of removal is also high,

    Release cadence

    -The branch is cut every January and July, so feature ("minor") releases occur about every 6 months in general. -Hence, Spark 2.3.0 would generally be released about 6 months after 2.2.0. Maintenance releases happen as needed -in between feature releases. Major releases do not happen according to a fixed schedule. +Starting with Apache Spark 4.3.0 (the first quarterly feature release after 4.2.0), feature releases occur quarterly (every 3 months), containing new features, +improvements, and bug fixes. Major releases occur annually (every 12 months), typically in a stable window once per year (subject to release discussion on the dev list), allowing breaking +changes and dependency upgrades. Maintenance releases happen as needed in between for critical +bug fixes and security patches.

    Spark 4.2 release window

    @@ -108,13 +118,53 @@ in between feature releases. Major releases do not happen according to a fixed s | Mid May 2026 | QA period. Focus on bug fixes, tests, stability and docs. Generally, no new features merged.| | Late May 2026 | Release candidates (RC), voting, etc. until final release passes| +

    Illustrative transition: 2026 and 2027

    + +The calendar below is an **example** to show how the community expects to bootstrap the faster cadence; +exact dates stay subject to the usual release-discussion and voting process. + +- **2026**: Ship **Apache Spark 4.2.0** on the timeline above (the last *large* feature drop before the + transition). **4.2.x** is not affected by this policy (development began before the SPIP); existing + commitments, including the **18**-month maintenance window for **4.2.x**, are unchanged. After **4.2.0** + is generally available, plan **Apache Spark 4.3.0** as the first quarterly feature release on the new + train (for example, roughly **three months** after the **4.2.0** GA date for the start of the **4.3** merge/RC + cycle—this is not a fixed rule, only an illustration of quarterly feature releases). +- **2027**: Ship **Apache Spark 5.0.0** as the next **annual** major. Follow with quarterly **5.1.0**, + **5.2.0**, and **5.3.0** feature releases; **5.3.0** is the **5.x** LTS as the last **5.x** feature release + (for example targeting calendar quarters **2027 Q1** through **Q4** if the **5.0.0** major lands early + in the year). +

    Maintenance releases and EOL

    -Feature release branches will, generally, be maintained with bug fix releases for a period of 18 months. -For example, branch 2.3.x is no longer considered maintained as of September 2019, 18 months after the release -of 2.3.0 in February 2018. No more 2.3.x releases should be expected after that point, even for bug fixes. +We plan to ship a release every 3 months. Every fourth release bumps the major version (**x.0.0**). +Within each major line, the first release is the major release, the releases in between are feature +releases, and the last release in the line is the LTS release. -The last minor release within a major release will typically be maintained for longer as an "LTS" release. -For example, 3.5.0 was released on September 13th 2023 and would normally be maintained for 31 months until April 12th 2026. +The following table summarizes the maintenance window for each release type: -As an exception from the normal versioning policy, version 3.5.x has an "extended" LTS period to allow for migrations to be completed. This extended LTS period will end *November 2027*. During the 3.5.x extended LTS period, we will only include security fixes. This extend LTS only applies to the primary Apache Spark project/repo and does not apply to sub projects with separate repos/releases (namely: Spark Connect for Swift/Rust/Go and Spark Kubernetes operator). Additionally, as Java 8 support may be removed from other projects (including Hadoop), should a dependency have a security fix that is not backported to a Java 8 compatible version we may decide to mark that vulnerability as a won't fix or release the new version without Java 8 support. +| Release Type | Maintenance Window | +| ----- | ----- | +| Major (x.0.0) | 6 months | +| Feature (x.[1–2].0) | 6 months | +| LTS (last x.y.0 in the major line) | 18 months | +| Maintenance (x.y.z, z ≥ 1) | N/A (patches only) | + +Release branches **other than LTS** will, generally, be maintained with bug fix releases for a period of +6 months (see the **Major** and **Feature** rows in the table above). + +Under this cadence, the **last** release in each major line is designated as the LTS (Long-Term +Support) release for that major line and is maintained for **18 months** (for example Apache Spark +5.3.0 and 6.3.0). The **x.0.0**, **x.1.0**, and **x.2.0** trains are maintained for **6 months** +each as usual. LTS releases provide a stable target for ecosystem projects and downstream vendors to +standardize around. + +As an exception while the project transitions into this cadence, **Spark 4.x** still ships **4.3.0** as the +first quarterly release on the new train, but the **4.x** LTS will be **Apache Spark 4.5.0** (the last +planned **4.x** release) rather than **4.3.0**. **4.5.x** therefore receives the **18**-month LTS +window in the same role as the last release on newer major lines. + +Critical security patches will be backported to all actively maintained branches. Critical bug fixes +(e.g., correctness issues) that may introduce behavior changes will be evaluated by the community +on a case-by-case basis. + +As an exception from the normal versioning policy, version 3.5.x has an "extended" LTS period to allow for migrations to be completed. This extended LTS period will end *November 2027*. During the 3.5.x extended LTS period, we will only include security fixes. This extended LTS only applies to the primary Apache Spark project/repo and does not apply to sub projects with separate repos/releases (namely: Spark Connect for Swift/Rust/Go and Spark Kubernetes operator). Additionally, as Java 8 support may be removed from other projects (including Hadoop), should a dependency have a security fix that is not backported to a Java 8 compatible version we may decide to mark that vulnerability as a won't fix or release the new version without Java 8 support.