Managed Service for Apache Spark (formerly Dataproc) enables fast provisioning of Apache Spark and Hadoop workloads for big data applications.
What is Managed Service for Apache Spark?
Managed Service for Apache Spark is the umbrella brand, effective since April 2026, for Google’s managed Spark offerings: the cluster-based option (formerly Dataproc on Compute Engine) and the serverless option (formerly Google Cloud Serverless for Apache Spark). The service supports Apache Spark, Hadoop, Presto, and other open-source tools. API endpoints, client libraries, CLI commands, and IAM role names continue to carry the Dataproc name, so existing automation continues to work unchanged.
Core Features
- Fast provisioning: Clusters ready quickly with pre-configured images
- Serverless option: Spark jobs without cluster management with automatic scaling
- Native GCP integration: Direct connection to BigQuery, Cloud Storage, and the Gemini Enterprise Agent Platform (formerly Vertex AI)
- Autoscaling: Automatic adjustment of cluster size based on workload, including scale-down to zero worker nodes
- Spot VM support: Cost savings through preemptible instances for suitable workloads
Typical Use Cases
ETL and Data Processing
Migrate existing Hadoop or Spark ETL pipelines to the cloud with minimal code changes. The service supports common Spark APIs and libraries.
Data Lake Analytics
Analyze large amounts of data in Cloud Storage with Spark SQL or Presto. Direct integration with BigQuery enables hybrid analytics across data lake and data warehouse.
Machine Learning with Spark MLlib
Train ML models on large datasets with Spark MLlib. Integration with the Gemini Enterprise Agent Platform for model deployment and monitoring.
Benefits
- Open-source compatibility: Run unmodified Spark, Hadoop, and Presto workloads
- Cost efficiency: Usage-based billing and Spot VMs for temporary workloads
- Fast migration: Migrate existing on-premises workloads without major refactoring
- Flexible options: Choose between cluster-based and serverless depending on requirements
Integration with innFactory
As a certified Google Cloud partner, innFactory supports you with Managed Service for Apache Spark: migration from on-premises Hadoop clusters, optimization of existing Spark jobs, architecture of data lake analytics solutions, and cost optimization through proper cluster configuration.
Available Tiers & Options
Managed Service for Apache Spark on clusters
- Full control over cluster configuration
- Spot VMs for cost optimization
- Autoscaling support, including scale-down to zero worker nodes
- Cluster management required
Managed Service for Apache Spark serverless
- No cluster management
- Automatic resource scaling
- Fast startup without cluster provisioning
- Fewer configuration options
Typical Use Cases
Technical Specifications
Frequently Asked Questions
What is Managed Service for Apache Spark (formerly Dataproc)?
Managed Service for Apache Spark is Google's unified brand for managed Spark workloads, covering both cluster-based deployment (formerly Dataproc on Compute Engine) and serverless deployment (formerly Google Cloud Serverless for Apache Spark). The API, client libraries, CLI, and IAM names remain Dataproc.
What's the difference between Managed Service for Apache Spark and Dataflow?
Managed Service for Apache Spark is optimized for existing Spark/Hadoop workloads, while Dataflow is a fully serverless service for Apache Beam pipelines. For migrations from on-premises Hadoop clusters, Managed Service for Apache Spark is usually the better choice.
Can I run existing Spark jobs without changes?
Yes, the service is compatible with Apache Spark, Hadoop, Hive, Pig, and Presto. Existing jobs can typically be migrated without major code changes.
How is Managed Service for Apache Spark billed?
Billing covers the Compute Engine resources used plus a small management surcharge per vCPU-hour for cluster operation. Spot VMs can significantly reduce costs for suitable workloads; Google Cloud's current pricing page is authoritative for exact figures.
Is the service GDPR-compliant?
Yes, Managed Service for Apache Spark is available in EU regions. Data can be encrypted with Customer-Managed Encryption Keys (CMEK), which supports GDPR requirements.
Note: All product information on this page has been compiled with care, but is provided without guarantee and may be outdated or incomplete. Cloud services evolve rapidly — features, pricing, SLAs, and availability change frequently. Authoritative and up-to-date information can only be found on the official product page of Google Cloud (official documentation). This page does not represent an offer by Google Cloud.
