GitHub Repository

Tiny, stateless Go router that dispatches OpenAI-compatible requests to single-model vLLM and sglang backends with zero external dependencies

2 starsNix

LLMhop – A tiny, stateless router for LLMs with a NixOS module

Name: LLMhop – A tiny, stateless router for LLMs with a NixOS module
Availability: InStock
Author: mlenz

by mlenz·Jun 5, 2026·2 points·0 comments

Visit Project View on HN

AI Analysis

●●SolidNiche GemSolve My Problem

Stateless Go proxy routes LLM requests by model name to vLLM backends.

Strengths

•Pure Go with zero external dependencies, safe behind any load balancer
•NixOS module automates llama.cpp, vLLM, and sglang deployment via Quadlet
•Secret references via env vars and files, no plaintext in config

Weaknesses

•Only routes by model field, no request transformation or rate limiting
•Two stars on GitHub means limited community testing and adoption

Post Description

LLMhop is a tiny stateless proxy for LLM inference servers. It tackles an issue I faced when trying to serve more than one local LLM at once which is not natively supported by vLLM. The LLMhop binary inspects the model field of the request and routes it to the correct backend service with optional handling of authentication. In addition, it contains a NixOS module to run llama.cpp, vLLM, and sglang via Quadlet/Podman and auto-register with the proxy.