What is Amazon FSx for Lustre?
Amazon FSx for Lustre is a fully managed, high-performance parallel file system based on the open-source Lustre file system. It is designed for workloads that require extremely fast access to large data sets: high-performance computing, machine learning training, video processing, and financial modeling.
FSx for Lustre integrates seamlessly with Amazon S3 as a data repository. Objects in S3 are transparently presented as files in the Lustre file system without needing to copy them manually. After processing, results can be written back to S3 via data repository tasks.
Core Features
- Sub-Millisecond Latency: Parallel file system with consistently low access times (on the SSD storage class)
- S3 Integration: Data repository linking to S3, automatic import of object listings, and write-back via data repository tasks
- Multiple Storage Classes: SSD for consistent low latency, Intelligent-Tiering for elastic cost-efficient scaling, HDD for capacity-oriented workloads
- Scratch and Persistent: Temporary file systems for short-term processing or persistent, replicated file systems for long-running workloads
- POSIX-Compatible: Standard POSIX file system interface for existing Linux applications with no changes required
- Integration with AWS Services: Connects with Amazon SageMaker AI, AWS Batch, AWS ParallelCluster, as well as Amazon EC2, ECS, and EKS
Typical Use Cases
Machine Learning Training: ML training jobs require fast access to large datasets. FSx for Lustre delivers the performance needed to fully utilize GPU clusters instead of waiting on I/O, and accelerates training jobs with SageMaker AI by eliminating the initial S3 download step.
High-Performance Computing: Simulations, genome sequencing, and scientific computations benefit from the parallel file system’s high throughput across many parallel accesses, for example in combination with AWS ParallelCluster.
Video Rendering: Film and media production uses FSx for Lustre for rendering pipelines that need simultaneous access to large media assets.
Benefits
- Fully managed without Lustre administration overhead
- Seamless S3 integration as a data repository
- Consistent performance even under high parallelism
- Flexible storage classes and deployment options for temporary and permanent workloads
Integration with innFactory
As an AWS Reseller, innFactory supports you with Amazon FSx for Lustre: HPC architecture design, S3 integration for ML pipelines, performance optimization, and hybrid storage configuration.
Typical Use Cases
Frequently Asked Questions
What is Amazon FSx for Lustre?
Amazon FSx for Lustre is a fully managed parallel file system based on the open-source Lustre file system. It delivers sub-millisecond latencies and throughput rates reaching multiple TBps for compute-intensive workloads such as HPC and machine learning.
How does the S3 integration work?
FSx for Lustre can be linked directly to an S3 bucket as a data repository. S3 objects are transparently presented as files in the Lustre file system, and changes can be written back to S3 via data repository tasks. The file system serves as a high-performance access layer for S3 data.
What storage classes does FSx for Lustre offer?
FSx for Lustre offers the SSD, Intelligent-Tiering, and HDD storage classes. SSD delivers consistent sub-millisecond latency at throughput levels of 125, 250, 500, or 1,000 MB/s per TiB of provisioned capacity. Intelligent-Tiering scales throughput independently of storage capacity in increments of 4,000 MB/s.
Scratch vs. Persistent - what is the difference?
Scratch file systems are suited for temporary, short-term processing without data replication. Persistent file systems replicate data and automatically replace failed file servers, making them suitable for longer-running, throughput-focused workloads.
Note: All product information on this page has been compiled with care, but is provided without guarantee and may be outdated or incomplete. Cloud services evolve rapidly — features, pricing, SLAs, and availability change frequently. Authoritative and up-to-date information can only be found on the official product page of AWS (official documentation). This page does not represent an offer by AWS.