[Feature] Add Spark 4 platform images - #314
Merged
royischoss merged 5 commits intoSep 23, 2026
Merged
royischoss merged 5 commits into
royischoss merged 5 commits into
Conversation
Adds separately maintained Spark 4.2.0 CPU and CUDA application images for ML-13080 without changing the existing Spark 3 recipes or configured defaults. Both variants use pinned upstream images, Spark 4.2.0, Scala 2.13, Hadoop 3.5.0, Temurin 25.0.4+7, and Python 3.11, with local validation for image metadata and CPU/CUDA JAR parity. --- - Added `docker/spark/Dockerfile.spark4` for the CPU image using a digest-pinned upstream Spark base. - Added `docker/spark/Dockerfile.spark4.cuda` for the CUDA image using digest-pinned Spark and CUDA bases without downloading Spark or Java again. - Added `docker/spark/scripts/ce-customize-spark4.sh` for Python 3.11 and a writable `/home/spark`. - Added `docker/spark/scripts/validate-spark4.sh` for Spark, Scala, Java, Hadoop, Python, architecture, locale, ownership, entrypoint, CUDA, cuDNN, and NVIDIA metadata validation. - Added `docker/spark/scripts/jar-parity.sh` to compare every CPU and CUDA Spark JAR by SHA-256. - Added independent Spark 4 build and validation targets to `docker/spark/Makefile`. - Documented Spark 4 build, validation, connector limitations, selection, manual publication, and evidence capture in `docker/spark/README.md`. --- - [ ] I have tested the changes in this PR - [ ] I confirmed whether my changes require a change in documentation and if so, I created another PR in MLRun for the relevant documentation. - [ ] I confirmed whether my changes require a changes in QA tests, for example: credentials changes, resources naming change and if so, I updated the relevant Jira ticket for QA. - [ ] I increased the Chart version in `charts/mlrun-ce/Chart.yaml`. - [ ] I confirmed that the installation works both on a local Docker Desktop environment and on a real cluster when using the required [prerequisites](https://docs.mlrun.org/en/stable/install-mlrun-ce/kubernetes-install.html#prerequisites). - [ ] If installation issues were found, I updated the relevant Jira ticket with the issue and steps to reproduce, or updated the prerequisites documentation if the issue is related to missing or outdated prerequisites. - [x] If needed, update https://github.com/mlrun/ce/blob/development/charts/mlrun-ce/README.md with the relevant installation instructions and version Matrix. - [x] If needed, update the following values files for multi namespace support: - [x] [Admin values](https://github.com/mlrun/ce/blob/development/charts/mlrun-ce/admin_installation_values.yaml) - [x] [User values Node Port](https://github.com/mlrun/ce/blob/development/charts/mlrun-ce/non_admin_installation_values.yaml) - [x] [User values ClusterIP](https://github.com/mlrun/ce/blob/development/charts/mlrun-ce/non_admin_cluster_ip_installation_values.yaml) --- - Built both `linux/amd64` images with `make build-spark4-all`. - Ran `make validate-spark4-all`; both images passed Spark, Scala, Java, Hadoop, Python, architecture, entrypoint, user, home-directory, ownership, locale, and image-metadata checks. - Validated CUDA 12.8.1, cuDNN 9.8.0.87-1, `nvcc`, cuDNN libraries, labels, and NVIDIA runtime environment metadata. - Ran SparkPi successfully in both images. - Ran `make jar-parity-spark4`; all 276 JARs matched by SHA-256. - Confirmed the protected Spark 3 files have no diff from `development` and the existing Spark 3 Make targets are unchanged. --- - Ticket link: https://ecliptos.atlassian.net/browse/ML-13080 - External links: https://hub.docker.com/_/spark and https://hub.docker.com/r/nvidia/cuda - Design docs links (Optional): `docker/spark/README.md` --- - [ ] Yes (explain below) - [x] No No configured image defaults, chart values, resource names, ports, Secrets, ConfigMaps, or installation behavior are changed. --- The Spark 4 images intentionally exclude S3A, ABFS, GCS, and BigQuery connectors, so connector support requires a later immutable image revision. JFrog publication uses immutable tags and remains a manual checkpoint; attach the resulting repository digests, pull-by-digest output, image inspection output, version output, JAR inventories, and source commit to ML-13080. - Confirm both immutable JFrog images were published successfully and attach the required evidence to ML-13080. - Installation on Docker Desktop and a real Kubernetes cluster still requires human confirmation.
royischoss
reviewed
Sep 22, 2026
royischoss
left a comment
royischoss
left a comment
Collaborator
There was a problem hiding this comment.
Hey looks good few comments
Collaborator
Author
Thanks for the detailed review, @royischoss. I addressed all four comments:
We also added the cloud connector family shipped by the Spark 3 image so Spark 4 can serve as a like-for-like platform image. Both CPU and CUDA images now install the eight artifacts listed in The rebuilt images pass local validation: all eight connector artifacts are present, the S3A/ABFS/GCS classes resolve, no duplicate connector versions are present, CPU/CUDA environment parity passes, and all 284 JARs match by SHA-256. Authenticated provider testing and BigQuery compatibility validation against Spark 4.2 remain deferred to the activation work. |
royischoss
reviewed
Sep 23, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
📝 Description
Adds Spark 4.2.0 CPU and CUDA application images for ML-13080 without changing the existing Spark 3 recipes or configured defaults.
Both variants use centrally pinned upstream images, Spark 4.2.0, Scala 2.13, Hadoop 3.5.0, Temurin 25.0.4+7, and Python 3.11, with validation for runtime metadata and CPU/CUDA parity.
🛠️ Changes Made
docker/spark/Dockerfile.spark4for thelinux/amd64CPU image.docker/spark/Dockerfile.spark4.cudafor thelinux/amd64CUDA image without downloading Spark or Java again.docker/spark/Makefile.docker/spark/scripts/ce-customize-spark4.shfor Python 3.11, manifest-driven connector installation, and a writable/home/spark.docker/spark/scripts/jars-4.2.0.txtto both images.docker/spark/scripts/validate-spark4.shfor Spark, Scala, Java, Hadoop, Python, connector inventory, architecture, locale, ownership, entrypoint, CUDA, cuDNN, and NVIDIA metadata validation.docker/spark/scripts/jar-parity.shto compare every CPU and CUDA Spark JAR by SHA-256.docker/spark/scripts/env-parity.shto enforce shared runtime metadata while allowing documented CUDA-specific and build-provenance differences.docker/spark/Makefile.docker/spark/README.md.✅ Checklist
charts/mlrun-ce/Chart.yaml.🧪 Testing
linux/amd64images withmake build-spark4-all.make validate-spark4-all; both images passed Spark, Scala, Java, Hadoop, Python, architecture, entrypoint, user, home-directory, ownership, locale, and image-metadata checks.nvcc, cuDNN libraries, labels, and NVIDIA runtime environment metadata.JAVA_VERSION, the expected CUDAPATHaddition, allowed CUDA-only variables, and intentional Spark build-provenance omissions.development.🔗 References
docker/spark/README.md🚨 Breaking Changes?
No configured image defaults, chart values, resource names, ports, Secrets, ConfigMaps, or installation behavior are changed.
🔍️ Additional Notes
ML-13080 originally excludes storage-provider libraries and validation; the images nevertheless package the same connector family as the published Spark 3 image so Spark 4 can serve as a like-for-like replacement. Authenticated provider testing and BigQuery compatibility validation against Spark 4.2 remain deferred to activation work.
GCS consumers must use
org.apache.hadoop.fs.gs.GoogleHadoopFileSystem, and AWS consumers using SDK v1 credential-provider classes may need to migrate their configuration for Hadoop AWS 3.5.0.JFrog publication uses manually managed tags; attach the resulting repository digests, pull-by-digest output, image inspection output, version output, JAR inventories, pinned base references, and source commit to ML-13080.
Warnings