Running a vision model on every screenshot on-device

hi author here, Screenmind is privacy first Microsoft recall alternative . It runs on gemma 4 which is one of the fewer models supporting vision audio and reasoning all 3, so your data never leaves you machine.

With screenmind you can keep a track of your timeline , how much time you spent on what..search any screenshot with any text on it.. and the coolest thing, you can chat with your screen history, like what did alex texted me on discord or did i received any mail from Microsoft, if it was on your screen , you can prompt it in the cha. and also you can make automations on top of it, like send me my whole day report on slack(it has integrations )..you can also write automation either though plain English for not so coders or use the python for devs who want to deep dive, and you can save voice memos(with a screenshot) with just a hotkey, and get you meeting transcribed and summarised(auto detects meeting)

the hardest part which i faced was keep running screenmind as a background service it would not have been not hard if chat feature didn't existed, as running local model requires compute ..and keep analyzing screenshots continuously will keep all the resouces hogged up for that i came up with a perceptual has cache .. the three tier cache system reduces inference upto 40% for an average user(which is me)..and to reduce the inference time more i came up with three modes..fast balanced and accurate..where the tradeoff is between time and accuracy

for now i use it daily on my 4gb gtx 1650 with fast mode, works pretty fine also it would be much faster on high end machine , it also has a mcp server so you can just ask claude desktop/cursor about the bug you saw in morning..

supports windows/mac/Linux

being upfront about rough edges , it is not extensively tested on mac and installation has some friction , for which i m working on one click installer thing

(reposting- i put up an earlier version a few months back, comments got flagged cuz of new account so couldn't reply to any )

repo:github.com/ayushh0110/ScreenMind

curious about anyone have idea for how to approach multi monitor support