HN Showcase
Back to browse
GitHub Repository

The inference engine the open-source world built for itself.

151 starsPython

ZSE – Open-source LLM inference engine with 3.9s cold starts

by zyoralabs·Feb 26, 2026·58 points·9 comments
Visit ProjectView on HN

AI Analysis

●●●BangerWizardrySolve My Problem

3.9s cold starts vs 45s+ for quantized models—real infra pain solved tangibly.

Strengths
  • •Native .zse pre-quantized format with memory-mapping cuts cold starts 11.6× vs bitsandbytes on real A100 hardware
  • •Custom CUDA kernels (zAttention, zKV cache quantization) deliver 63-70% memory reduction vs FP16 without sacrificing throughput
Weaknesses
  • •Benchmarks only on A100-80GB hardware; consumer GPU validation and long-tail latency (HDD, older SSDs) under-tested
  • •No comparison against vLLM's latest quantization modes or other emerging inference engines beyond cold-start metrics
Target Audience

ML engineers deploying LLMs on limited VRAM or serverless platforms; cost-conscious inference teams

Similar To

vLLM · bitsandbytes · llama.cpp

Post Description

I've been building ZSE (Z Server Engine) for the past few weeks — an open-source LLM inference engine focused on two things nobody has fully solved together: memory efficiency and fast cold starts.

The problem I was trying to solve: Running a 32B model normally requires ~64 GB VRAM. Most developers don't have that. And even when quantization helps with memory, cold starts with bitsandbytes NF4 take 2+ minutes on first load and 45–120 seconds on warm restarts — which kills serverless and autoscaling use cases.

What ZSE does differently:

Fits 32B in 19.3 GB VRAM (70% reduction vs FP16) — runs on a single A100-40GB

Fits 7B in 5.2 GB VRAM (63% reduction) — runs on consumer GPUs

Native .zse pre-quantized format with memory-mapped weights: 3.9s cold start for 7B, 21.4s for 32B — vs 45s and 120s with bitsandbytes, ~30s for vLLM

All benchmarks verified on Modal A100-80GB (Feb 2026)

It ships with:

OpenAI-compatible API server (drop-in replacement)

Interactive CLI (zse serve, zse chat, zse convert, zse hardware)

Web dashboard with real-time GPU monitoring

Continuous batching (3.45× throughput)

GGUF support via llama.cpp

CPU fallback — works without a GPU

Rate limiting, audit logging, API key auth

Install:

----- pip install zllm-zse zse serve Qwen/Qwen2.5-7B-Instruct For fast cold starts (one-time conversion):

----- zse convert Qwen/Qwen2.5-Coder-7B-Instruct -o qwen-7b.zse zse serve qwen-7b.zse # 3.9s every time

The cold start improvement comes from the .zse format storing pre-quantized weights as memory-mapped safetensors — no quantization step at load time, no weight conversion, just mmap + GPU transfer. On NVMe SSDs this gets under 4 seconds for 7B. On spinning HDDs it'll be slower.

All code is real — no mock implementations. Built at Zyora Labs. Apache 2.0.

Happy to answer questions about the quantization approach, the .zse format design, or the memory efficiency techniques.

Similar Projects