Back to browse
GitHub Repository

A CLI tool that turns your Python scripts into a parallel data processing engine

3 starsRust

Ril: a parallel data streaming tool for Python

by dpesic·Jun 16, 2026·2 points·0 comments

AI Analysis

●●SolidSolve My ProblemBig Brain

4x speedup over pandas with constant memory via streaming workers.

Strengths
  • Matches multiprocessing.Pool performance without manual Pool configuration
  • Pipeline syntax (load | clean | save) mirrors Unix stream processing
  • Peak memory stays flat regardless of dataset size
Weaknesses
  • Parallel Python processing is crowded (Dask, Ray, joblib)
  • Polars or DuckDB faster for pure DataFrame transforms
Category
Target Audience

Data scientists and Python developers processing large datasets

Similar To

Dask · Ray · multiprocessing.Pool

Similar Projects

Developer Tools●●Solid

Fleet – Python supervisor for running coding agents in parallel

Centralized beads queue eliminates per-project init for 50+ parallel Claude sessions.

Ship ItNiche Gem
sermakarevich
3525d ago