Yaks – Yet Another Kafka on S3

Name: Yaks – Yet Another Kafka on S3
Availability: InStock
Author: markbergz

by markbergz·Mar 5, 2026·1 point·0 comments

Visit Project View on HN

AI Analysis

●●●BangerWizardryBig Brain

S3-backed Kafka eliminates broker state management; inspired by Warpstream but open-source.

Strengths

•Kafka wire protocol compatibility means zero client code changes—genuine plug-and-play for existing infrastructure.
•Stateless agent design with groupcache distributed read cache solves real operational pain (no broker rebalancing, horizontal scaling without coordination).
•Event sourcing + S3 latency tradeoff is transparent and well-justified for throughput-oriented workloads (logs, CDC, event sourcing).

Weaknesses

•Early stage (explicitly WIP, not production-tested)—unproven at scale against Warpstream/AutoMQ in real deployments.
•Higher end-to-end latency is unsuitable for low-latency requirements, limiting addressable market.

Post Description

I'm a long time reader of HN and some articles have been really inspiring for me. "Kafka is dead, long live Kafka" by the Warpstream team is one of those: https://www.warpstream.com/blog/kafka-is-dead-long-live-kafk...

At the time, I was working on a team pushing terabytes of logs through Kafka. A stateless broker backed by S3 would have saved us many headaches from hand-holding the Kafka cluster. It really struck me as a simple and elegant solution for our use case. Ever since then, I've been slowly tinkering and rewriting various implementations to better understand how to build a similar S3-backed broker.

yaks aims to be simple and only require a S3 bucket and Postgres database. Metadata is written to the db using an event sourcing model and reads to S3 are cached using the popular groupcache library.

Thanks for checking this out.