What is STACKIT Intake?
STACKIT Intake is a fully managed data ingestion service that accepts high-volume data streams via the Apache Kafka protocol and writes them directly to Apache Iceberg tables. It forms the ingestion layer of the STACKIT Data Lakehouse, writing to tables managed via the Iceberg REST catalog of STACKIT Dremio. Intake is designed as a temporary buffer, not a permanent data store.
Core Features
- Kafka protocol compatibility: Use existing Kafka client libraries and data producers such as Debezium without modification
- Automatic schema management: Data types are inferred from JSON payloads, with automatic schema evolution of the target Iceberg table
- Buffering with delivery guarantee: Messages are buffered for up to 24 hours until reliably written to Dremio
- Dead letter queue: Automatic handling of malformed or unprocessable messages
- Scalable throughput: Dynamic capacity adjustment between 1 GB/h and 20 GB/h
- Management via portal, CLI, and Terraform
Typical Use Cases
Data lakehouse ingestion: Intake serves as the ingestion layer: data flows via the Kafka protocol directly into Iceberg tables that can be analyzed in STACKIT Dremio.
Change data capture (CDC): Changes from operational databases are captured using tools like Debezium and brought into the data lakehouse in real time.
IoT and telemetry: Device data, sensor readings, and application logs are captured at high volume in real time and made available for downstream analytics.
Benefits
- GDPR-compliant, operating exclusively in German data centers
- Kafka protocol compatibility avoids lock-in for data producers
- Automatic schema management reduces manual maintenance effort
- Seamless integration with STACKIT Dremio for analysis directly on incoming data
Integration with innFactory
As an official STACKIT partner, innFactory supports you in designing real-time data pipelines: from connecting existing Kafka producers to schema design and integration with STACKIT Dremio for downstream analytics.
Typical Use Cases
Technical Specifications
Frequently Asked Questions
What is STACKIT Intake?
A managed data ingestion service that accepts data streams via the Apache Kafka protocol and writes them directly to Apache Iceberg tables in the Dremio Iceberg REST catalog. It serves as a bridge between data producers and the STACKIT Data Lakehouse.
Is Intake a permanent data store?
No. Intake is designed as a temporary buffer, holding messages for up to 24 hours until they are reliably written to the target Iceberg table. They are then removed from the buffer.
What data formats are supported?
Currently, STACKIT Intake processes JSON data only. Data types are automatically inferred from JSON payloads, and the schema of the target Iceberg table is automatically managed and evolved as new fields appear.
What is the maximum throughput?
Throughput can be dynamically adjusted between 1 GB/h and 20 GB/h to match actual data volume.
Do I need to replace my existing Kafka clients?
No. Since STACKIT Intake supports the Apache Kafka protocol, existing Kafka client libraries and common data producers such as Debezium can be reused without modification.
Note: All product information on this page has been compiled with care, but is provided without guarantee and may be outdated or incomplete. Cloud services evolve rapidly — features, pricing, SLAs, and availability change frequently. Authoritative and up-to-date information can only be found on the official product page of STACKIT (official documentation). This page does not represent an offer by STACKIT.
