What is Managed Service for Apache Spark?
Managed Service for Apache Spark is the current product name for Google Cloud’s Apache Spark offering, unifying the former Dataproc on Compute Engine (cluster deployment) and Google Cloud Serverless for Apache Spark (serverless deployment) products under one brand. For serverless workloads, you submit your Spark code and the platform automatically provisions the required resources, runs the job, and releases the resources afterward.
Unlike traditional cluster operation, serverless mode removes the need to provision, configure, and continuously manage clusters. Billing applies only for the duration of actual job execution.
Core Features
- Two operating modes: cluster mode (managed Dataproc clusters on Compute Engine) and serverless mode (no cluster management required)
- No cluster management in serverless mode: run Spark batch workloads without provisioning your own clusters
- Automatic scaling: resource adjustment during job execution based on demand
- Open-source compatibility: support for common open-source data tools for batch processing, querying, streaming, and machine learning
- Integration with the Google Cloud data ecosystem, including BigQuery and Cloud Storage
Typical Use Cases
Ad-Hoc Data Analysis
Data scientists and analysts use serverless mode for exploratory analysis with Spark without provisioning or managing clusters.
Scheduled ETL Pipelines
Regularly executed Spark ETL jobs benefit from serverless mode since no clusters need to be maintained between executions. Orchestration tools can be connected for workflow control.
Workloads Requiring Persistent Clusters
For workloads with consistently high utilization or specific cluster configurations, cluster mode on Compute Engine remains available within the same service.
Benefits
- No infrastructure management or cluster tuning in serverless operation
- Faster iteration cycles for data engineers
- Cost-effective: billing based on actual usage
- Flexibility to choose between cluster and serverless mode within the same service
Integration with innFactory
As a certified Google Cloud Partner, innFactory supports you in adopting Managed Service for Apache Spark: migrating existing Spark jobs, designing data pipeline architecture, and optimizing costs between cluster and serverless operation.
Typical Use Cases
Frequently Asked Questions
What is Managed Service for Apache Spark?
Managed Service for Apache Spark is the current name for the Google Cloud service formerly known as Dataproc Serverless for Apache Spark (serverless deployment) and Dataproc on Compute Engine (cluster deployment). It runs Apache Spark jobs without you provisioning your own cluster, with the platform handling the infrastructure automatically.
What is the difference between serverless mode and cluster mode?
In cluster mode, you provision and configure your own Dataproc clusters on Compute Engine. In serverless mode, you only submit Spark code, and the platform automatically handles provisioning, scaling, and releasing resources.
How is serverless mode billed?
Serverless mode is billed on a usage basis per Dataproc Compute Unit (DCU) hour. Costs are only incurred during actual workload execution time, not for idle time.
What workloads is this service suited for?
The service suits batch processing, querying, streaming, and machine learning workloads using open-source data tools built on Apache Spark, especially when no persistent cluster management is desired.
Note: All product information on this page has been compiled with care, but is provided without guarantee and may be outdated or incomplete. Cloud services evolve rapidly — features, pricing, SLAs, and availability change frequently. Authoritative and up-to-date information can only be found on the official product page of Google Cloud (official documentation). This page does not represent an offer by Google Cloud.
