HN Showcase
Back to browse
Orchestera – Managed Apache Spark on Kubernetes in Your Own AWS Account

Orchestera – Managed Apache Spark on Kubernetes in Your Own AWS Account

by iamspoilt·Feb 19, 2026·3 points·1 comment
Visit ProjectView on HN

AI Analysis

●●SolidSolve My ProblemDark Horse

Spark without Databricks markup, but Kubernetes management is still ops work.

Strengths
  • •Genuine cost arbitrage: zero compute markup on EC2, billed directly to AWS account.
  • •Removes boilerplate — native SDK handles cluster lifecycle, autoscaling, notebook provisioning.
  • •Fault-tolerant design with multi-AZ optimization and pipeline monitoring built-in.
Weaknesses
  • •Kubernetes is still Kubernetes; 'fully managed' still requires AWS account ops and Spark expertise.
  • •Databricks offers collaborative notebooks, SQL, Delta Lake governance — not just cheaper compute. This solves price, not the feature gap.
Target Audience

Data engineers and ML practitioners using Apache Spark; teams wanting to avoid Databricks/EMR markup.

Similar To

Databricks · AWS EMR · Apache Spark on Kubernetes (self-managed)

Post Description

I built Orchestera as a PaaS that allows you to orchestrate Apache Spark clusters in your own AWS account, with no additional markup on compute via EC2 instances.

I built this because I was tired of the compute markup that products like AWS EMR and Databricks charge for the convenience of using Apache Spark via their platforms. One can argue that Databricks is a superior product with a lot of additional value in their offering but I don't see that with AWS EMR Apache Spark at all (given my personal experience working with it).

My motivation to build this was to be able to create your own Apache Spark cluster without needing any understanding of the underlying data infrastructure engineering and quickly get to the point of writing Spark pipelines, whether as Python applications or Jupyter notebook.

It took me almost an year to build it with a day job and of course I used AI for frontend design and video narrations, the infrastructue engineering that goes behind it comes with quite a bit of experience in the industrly. The backend that orchestrates the cluster is written with the following:

- Django and DRF for API - Temporal for async workers - Pulumi that is run via Temporal workers to orchestrate the cluster - Karpenter for node auto-scaling based on Spark executor workloads and requests - Librechat for Spark History server and MCP based debugging for Spark pipeline run analysis

There are currently no caps on the CPU limits so you can try this out today in your own personal AWS accounts for free.

Also looking for feedback on HN.

Similar Projects