Amazon EMR introduces Long Term Support for Apache Spark

Amazon EMR has introduced Long Term Support (LTS) releases, beginning with emr-spark-8.1.0 and Apache Spark 4.1. These designated versions will receive 36 months of support, providing fixes for critical and high-severity security, bug, and data-corruption issues to help users manage production workloads on their own schedules.

The update includes full support for Apache Iceberg v3, offering new geospatial, high-precision timestamp, and schema-evolution capabilities. Additionally, Spark SQL queries can now reference catalogs by name, including cross-account and Amazon S3 Tables catalogs, with automatic detection for Apache Iceberg, Delta Lake, and Apache Hudi formats.

Furthermore, Amazon EMR on EKS now supports interactive Apache Spark sessions via Spark Connect. This allows data engineers and scientists to develop and debug applications interactively from managed notebooks in Amazon SageMaker Unified Studio or local IDEs like Jupyter and Visual Studio Code. The Spark Connect architecture decouples the application client from the Spark driver, enabling persistent Spark contexts that span across cells and scripts.