Consumers

Consumer Overview

Consumers are client applications that read messages from StreamHouse topics. A consumer subscribes to one or more topics and reads messages starting from a specified offset. Consumers track their position (offset) so they can resume where they left off after restarts.

Consumer Groups

A consumer group is a set of consumers that cooperatively consume from a topic. Each partition is assigned to exactly one consumer in the group, allowing multiple consumers to process a topic in parallel. If a consumer joins or leaves the group, partitions are rebalanced automatically. Consumer group offsets are stored in the metadata store.

# Start multiple consumers in a group
# Terminal 1
streamctl consume --topic user-events --group analytics --auto-commit

# Terminal 2 (gets different partitions)
streamctl consume --topic user-events --group analytics --auto-commit

Offset Management

Consumers can manage offsets in two ways: automatic commit (offsets are committed periodically) or manual commit (the application explicitly commits offsets after processing). Manual commit provides exactly-once processing semantics when combined with idempotent downstream writes.

• Auto-commit: Offsets committed every 5 seconds (configurable). Simple but may result in at-least-once delivery.
• Manual commit: Application calls commit after processing. Enables exactly-once when combined with transactional writes.
• Seek: Consumers can seek to a specific offset, the beginning, or the end of a partition at any time.

Fetch Behavior

When a consumer requests messages, the StreamHouse agent determines which S3 segments contain the requested offset range. Segments are fetched from S3 (or the local cache), decompressed, and the relevant records are streamed back to the consumer. For tail reads (consuming near the head of the partition), the agent often serves directly from the in-memory write buffer, avoiding S3 entirely.