DemandMap – Memory Mapping S3 into Polars on macOS Without FUSE

Basically this exists to solve one of my biggest pain-points with dataframe libraries and big data. It's basically impossible to lazily download data on S3 with polars unless you're processing it strictly sequentially, and even then, it won't work with arrow and numpy data.

This allows you to "download" a multi-gigabyte file on S3 into a Polars DataFrame in <100ms.

# Demo

alloc = demandmap.S3Alloc( "./cache.bin", # number of blocks capacity=512, # one megabyte block (per request chunk size) block_size=1048576 )

buf1 = alloc.get(S3_PATH) # Big file assert buf1.nbytes > 400000000 # But this takes ~100ms df = pl.DataFrame([buf1])

It's one of those problems that manages to be both simple and difficult. I'd wager <5% of devs know what a memory map is (higher on HN) and I'd wager 1% of devs who know what mmap is know you can catch page faults in user-space, and yet another 1% of those devs know that it's possible to do userfaultfd in macOS with truly obscure mach_send_msg calls.

I'd really like to build a cross platform user faulting library covering Linux and Windows too, because nearly everyone who's touched a dataframe has had this exact problem.