What is Amazon Kinesis Data Streams?
Amazon Kinesis Data Streams is a serverless real-time data streaming service that enables continuous capture, processing, and analysis of large data volumes. The service processes data with millisecond latencies and serves as a central building block for event-driven architectures.
Kinesis Data Streams is based on the concept of shards, where each shard processes up to 1 MB/s of input data and 2 MB/s of output data. With On-Demand mode, manual capacity planning is eliminated as the service automatically scales based on data volume.
The service is designed for scenarios where data from many producers needs to be received in parallel and processed independently by multiple consumers. Typical sources include web applications, IoT devices, mobile devices, and logging systems.
Core Features
- On-Demand Mode: Automatic scaling without manual shard management based on actual throughput
- Enhanced Fan-Out: Dedicated throughput of 2 MB/s per consumer for parallel processing
- Data Retention: Configurable retention period from 24 hours to 365 days
- Server-Side Encryption: Automatic encryption of data at rest with AWS KMS
- Kinesis Client Library: SDK for easy development of consumer applications with automatic load balancing
Typical Use Cases
Real-Time Log Aggregation: Organizations stream application logs from hundreds of microservices via Kinesis Data Streams to central analytics systems. Low latency enables timely error detection and alerting.
Event-Driven Architectures: Kinesis Data Streams serves as a central event bus, distributing business events such as orders, payments, or status changes in real time to downstream services.
IoT Data Streaming: IoT devices send sensor data via Kinesis Data Streams to processing pipelines that detect anomalies, compute aggregations, and store data for long-term analysis.
Benefits
- Millisecond latencies for time-critical processing scenarios
- Flexible scaling from a few KB/s to GB/s throughput
- Multiple consumers can independently process the same data stream
- Seamless integration with Lambda, Flink, EMR, and other AWS services
Integration with innFactory
As an AWS Reseller, innFactory supports you with Amazon Kinesis Data Streams: from streaming infrastructure architecture and producer/consumer application development to capacity planning and cost optimization.
Typical Use Cases
Frequently Asked Questions
What is Amazon Kinesis Data Streams?
Amazon Kinesis Data Streams is a serverless streaming service for capturing, processing, and storing data streams in real time. The service continuously processes gigabytes of data per second from hundreds of thousands of sources.
What is the difference between On-Demand and Provisioned Mode?
In On-Demand mode, Kinesis automatically scales based on data volume. In Provisioned Mode, you manually configure the number of shards. On-Demand is suited for variable workloads, Provisioned for predictable loads.
How long is data stored in Kinesis?
By default, data is stored for 24 hours. The retention period can be extended up to 365 days, which is useful for replay scenarios and data reprocessing.