Cluster Toolkit is Google’s open-source tool for deploying HPC, AI, and ML clusters on Google Cloud reproducibly and in line with best practices. Instead of assembling infrastructure manually, teams define their cluster architecture in a YAML blueprint and let it automatically generate Terraform and Packer configurations.
What is Cluster Toolkit?
Per the official documentation, Cluster Toolkit simplifies the deployment of high performance computing (HPC), AI, and machine learning (ML) workloads on Google Cloud. The open-source tool provides customizable blueprints that help provision infrastructure aligned with Google Cloud best practices; it can also be tailored and extended to address a broad range of deployment needs.
Components
Per the documentation, Cluster Toolkit consists of four building blocks:
- Cluster blueprint: A YAML file that defines the cluster’s architecture by specifying which modules to use and how to configure them.
- Modules: Reusable, configurable building blocks that define specific resources like schedulers, storage, or compute nodes.
- The gcluster tool: A command-line utility that compiles the blueprint and modules into a deployment folder.
- Deployment folder: A generated directory containing the Terraform and Packer configurations needed to deploy the cluster. It can be deployed directly or customized further before deployment.
How it works
Users first set up their environment via Cloud Shell or a local Linux or macOS terminal, clone the Cluster Toolkit repository, and use example blueprints to create their own blueprint file. Running gcluster create generates a deployment folder with the necessary Terraform and Packer configurations; the subsequent gcluster commands then deploy the cluster on Google Cloud via Terraform or Packer. After deployment, jobs can be submitted to the HPC cluster, and the Google Cloud resources in use can be analyzed via Cloud Monitoring.
For running clusters, Cluster Toolkit documentation states support for updating specific configurations, such as resizing a Slurm partition or updating a GKE node pool. For fundamental architectural changes, such as switching VPCs or the scheduler, a full cluster redeployment is required instead.
Telemetry
By default, Google collects non-personally-identifiable usage statistics about Cluster Toolkit, such as commands used, exit codes, execution latency, names of standard blueprints and modules, machine types, and the toolkit version in use. This telemetry can be turned off via gcluster telemetry off and back on via gcluster telemetry on.
Core Features
- Turnkey cluster deployment: Efficiently create and deploy HPC, AI, and ML clusters following Google Cloud best practices.
- Configurable open-source framework: Customize and extend via your own modules and blueprints.
- Partner integrations: Seamless interoperability with Intel DAOS, DDN EXAScaler, and Slurm.
- Monitoring integration: Performance visibility through integration with Cloud Monitoring.
- Reproducible infrastructure: Terraform- and Packer-based deployment folders instead of manual configuration.
Typical Use Cases
HPC clusters for scientific computing
A research team uses an example blueprint to quickly deploy a Slurm-based HPC cluster on Google Cloud via gcluster create and gcluster deploy.
Standardized AI/ML training environments
A platform team defines company-wide blueprints so different teams can create reproducible AI and ML clusters following the same best practices.
Integration with specialized storage
For I/O-intensive HPC workloads, Cluster Toolkit is combined with partner solutions such as Intel DAOS or DDN EXAScaler.
Iterative adjustment of existing clusters
Running Slurm partitions or GKE node pools are adjusted via the update mechanisms Cluster Toolkit supports, without rebuilding the entire cluster.
Benefits
- Best-practice infrastructure without manual setup: Blueprints encapsulate proven Google Cloud architecture patterns for HPC/AI/ML.
- Reusability: Modules can be reused across different blueprints and projects.
- Openness: As an open-source project, customizable and extensible for specific requirements.
- Ecosystem integration: Direct integration with established HPC partners and Cloud Monitoring.
- Transparent deployment: Terraform and Packer configurations are inspectable and version-controlled.
Integration with innFactory
As a certified Google Cloud partner, innFactory helps you adopt Cluster Toolkit: designing blueprints suited to your HPC, AI, or ML workloads, integrating with your existing Terraform infrastructure, and connecting it to monitoring and cost-control processes.
Contact us for a consultation on Cluster Toolkit.
Typical Use Cases
Technical Specifications
Frequently Asked Questions
What is Cluster Toolkit?
Per Google Cloud documentation, Cluster Toolkit simplifies deploying high performance computing (HPC), AI, and machine learning (ML) workloads on Google Cloud. It's an open-source tool with customizable blueprints that help provision infrastructure aligned with Google Cloud best practices; the toolkit can also be tailored and extended for a broad range of deployment needs.
What components make up Cluster Toolkit?
The documentation names four components: the cluster blueprint (a YAML file defining the cluster's architecture and modules), modules (reusable, configurable building blocks for things like schedulers, storage, or compute nodes), the gcluster command-line tool that compiles the blueprint and modules into a deployment folder, and the generated deployment folder itself, containing the Terraform and Packer configurations needed for deployment.
Which partner solutions does Cluster Toolkit integrate with?
Per the documentation, Cluster Toolkit integrates seamlessly with various partners, including Intel DAOS, DDN EXAScaler, and Slurm. It also integrates with Cloud Monitoring to provide performance visibility into deployed clusters.
What does Cluster Toolkit cost?
Cluster Toolkit itself is a free, open-source tool. Costs are incurred for the Google Cloud resources it provisions, such as Compute Engine or Google Kubernetes Engine, per the applicable Google Cloud price list.
Note: All product information on this page has been compiled with care, but is provided without guarantee and may be outdated or incomplete. Cloud services evolve rapidly — features, pricing, SLAs, and availability change frequently. Authoritative and up-to-date information can only be found on the official product page of Google Cloud (official documentation). This page does not represent an offer by Google Cloud.
