What is Amazon Kinesis?
Amazon Kinesis is a family of fully managed services for processing streaming data in real time. Kinesis enables ingestion, processing, and analysis of continuous data streams with low latency at scale.
The Kinesis family includes several services: Kinesis Data Streams (core streaming platform), Kinesis Data Firehose (managed data delivery), and Amazon Managed Service for Apache Flink (formerly Kinesis Data Analytics; SQL and Apache Flink for stream processing). Kinesis Video Streams is a separate service for video ingestion, covered on its own product page.
Kinesis is available in numerous AWS regions, including EU regions for applications with requirements around processing data within the EU. The service scales automatically from a few MB to very large data volumes per second, depending on the chosen mode and region.
How Does Kinesis Data Streams Work?
Kinesis Data Streams is the core streaming platform. Producers send data (records) to streams, and consumers read data in parallel. Each record consists of a partition key, sequence number, and a data blob (currently up to 10 MiB per record, with very large records absorbed via burst capacity).
Stream Architecture
Shards: Parallelization units in Kinesis Data Streams. In Provisioned mode, each shard supports up to 1 MB/s or 1,000 records/s of write throughput and up to 2 MB/s or 2,000 records/s of read throughput. Streams scale by increasing the shard count.
Partition Key: Determines shard assignment via hashing. Records with the same partition key land on the same shard, guaranteeing order. High-cardinality keys ensure even distribution.
Retention: The default is 24 hours and can be increased up to a maximum of 365 days. Data remains replayable within the retention period.
Provisioned vs. On-Demand Mode
Provisioned: Manual shard configuration with full control over capacity. Shard splitting/merging for adjustments. Typically more cost-effective at consistent load.
On-Demand: Automatic sharding based on workload, billed per GB processed. New On-Demand streams start with a baseline throughput and scale automatically; some regions support significantly higher maximum throughput than others. Ideal for variable or hard-to-predict workloads.
Kinesis Data Firehose
Firehose is fully managed: no shards, no consumer management. Automatic batching, compression (e.g., Gzip), and format conversion (e.g., JSON to Parquet). Direct delivery to S3, Redshift, OpenSearch, Splunk, or custom HTTP endpoints.
Firehose buffers data briefly (near-real-time) before delivery. Transformations are possible via Lambda. The service is well suited for simple ETL pipelines without custom consumer logic.
Typical Use Cases
Real-Time Clickstream Analysis
E-commerce websites stream clickstream data (clicks, views, purchases) to Kinesis. Lambda or Amazon Managed Service for Apache Flink process events in real time for user segmentation, product recommendations, and conversion tracking.
Log Aggregation and Monitoring
Applications stream logs to Kinesis Firehose. Firehose batches, compresses, and delivers to S3 for cost-effective long-term storage or to OpenSearch for real-time search and alerting. CloudWatch Logs subscription filters can automatically send logs to Kinesis.
IoT Telemetry Processing
IoT devices send telemetry data to Kinesis Data Streams, which scales automatically with the workload in On-Demand mode. Lambda or Amazon Managed Service for Apache Flink handle anomaly detection, aggregation, and event routing.
Real-Time ETL for Data Lakes
Kinesis Data Streams serves as the ingestion layer, Lambda handles transformations, and Firehose delivers data into an S3 data lake, often with format conversion for more efficient queries with Athena or Glue.
Event Sourcing
Kinesis Data Streams can serve as an event store for event-sourcing architectures, since state changes remain replayable within the retention period and consumers can build read models from them.
Amazon Kinesis vs. Alternatives
Kinesis vs. Google Cloud Pub/Sub: Pub/Sub is a simpler messaging system with at-least-once delivery and optional ordering keys. Kinesis offers guaranteed ordering per partition key, longer retention, replay capability, and Enhanced Fan-Out.
Kinesis vs. Azure Event Hubs: Event Hubs is conceptually similar to Kinesis Data Streams (partitions instead of shards, a Capture feature similar to Firehose) and additionally offers Kafka protocol compatibility. Kinesis stands out with its On-Demand mode, long retention, and Enhanced Fan-Out.
Kinesis vs. self-managed Apache Kafka: Kafka offers more control and Kafka-specific features but requires significantly more operational effort (cluster and broker management). Kinesis suits teams looking for a managed solution without Kafka expertise; Amazon MSK is the AWS alternative for teams that need Kafka itself.
As multi-cloud experts, we provide vendor-neutral advice on the optimal solution for your requirements.
Integration with innFactory
As an AWS Reseller, innFactory supports you with:
Architecture Design: Designing scalable streaming architectures with Kinesis, event-driven design, Lambda integration, and analytics pipelines, plus selecting the right Kinesis service for your use case.
Migration: Migrating existing streaming solutions (e.g., Kafka, RabbitMQ, on-premises log aggregation) to Kinesis, including hybrid setups for gradual migration.
Performance Optimization: Partition key optimization for even shard utilization, use of Enhanced Fan-Out for latency-sensitive consumers, and shard scaling strategies.
Security & Compliance: Kinesis implementations with encryption in transit (TLS) and at rest (KMS), VPC endpoints for private connectivity, and IAM policies following least privilege.
Cost Optimization: Analysis of your Kinesis usage and the trade-off between Provisioned and On-Demand mode, as well as retention optimization.
Contact us for a non-binding consultation on Amazon Kinesis and real-time streaming architectures on AWS.
Available Tiers & Options
Kinesis Data Streams (Provisioned)
- Predictable costs
- Control over shard count
- Ideal for consistent workloads
- Manual shard management
- Over-provisioning for variable loads
Kinesis Data Streams (On-Demand)
- Automatic scaling
- No shard management
- Pay-per-GB
- Ideal for variable workloads
- Higher costs at consistently high load
Kinesis Data Firehose
- Fully managed
- Automatic batching
- Direct integration with S3, Redshift, OpenSearch
- No infrastructure management
- Higher latency than Data Streams
- Less flexibility in processing
Typical Use Cases
Frequently Asked Questions
What is Amazon Kinesis?
Amazon Kinesis is a family of services for real-time data streaming. Kinesis Data Streams enables ingestion and processing of streaming data in real time. Kinesis Data Firehose automatically delivers data to storage targets such as S3, Redshift, or OpenSearch. Amazon Managed Service for Apache Flink (formerly Kinesis Data Analytics) runs SQL queries and Apache Flink jobs on streams. Kinesis Video Streams is a separate service for video ingestion.
How much does Amazon Kinesis cost?
Kinesis Data Streams in Provisioned mode is billed per shard-hour plus PUT payload units; in On-Demand mode it is billed per GB written and read. Kinesis Data Firehose is billed per GB ingested. Extended retention and Enhanced Fan-Out incur additional charges. Current prices are available on the official AWS pricing page.
When should I use Data Streams vs. Firehose?
Data Streams is suited for low-latency real-time processing, custom consumer logic (Lambda, EC2, EKS), multiple parallel consumers, and replay capability within the retention period. Firehose is suited for near-real-time delivery directly to storage targets such as S3 or Redshift without custom processing logic. Firehose is simpler to operate, Data Streams is more flexible.
How does Kinesis sharding work?
Shards are the parallelization units in Kinesis Data Streams. In Provisioned mode, each shard supports up to 1,000 records or 1 MB/s of write throughput and up to 2,000 records or 2 MB/s of read throughput. Records are distributed to shards based on the partition key; the same partition key always maps to the same shard, guaranteeing order. In Provisioned mode, shard management is manual via split/merge; in On-Demand mode it is automatic.
What is Enhanced Fan-Out?
Enhanced Fan-Out gives each registered consumer dedicated read throughput per shard via HTTP/2 push instead of shared polling throughput. This reduces latency and increases throughput when multiple consumers read the same stream. Additional charges apply per consumer-shard-hour and per GB retrieved.
How do I integrate Kinesis with Lambda?
Lambda can use Kinesis Data Streams as an event source and automatically polls the stream to invoke functions with batches of records. Batch size and window are configurable. The parallelization factor enables multiple concurrent Lambda invocations per shard. This is well suited for event-driven processing without managing servers.
Note: All product information on this page has been compiled with care, but is provided without guarantee and may be outdated or incomplete. Cloud services evolve rapidly — features, pricing, SLAs, and availability change frequently. Authoritative and up-to-date information can only be found on the official product page of AWS (official documentation). This page does not represent an offer by AWS.