In the last year, this project idea has repeatedly come to mind: a wearable AI personal assistant that remembers everything I see and hear.

A number of problems have led me to this idea. A few examples include:

This project genuinely doesn't leave my mind - I've thought about it almost every single day for the last few months. I'm not exactly sure why it piques my interest so much, but I know I want to work on it. I'm planning on spending most of my free time in the next few months working on this and hope to have a reasonable proof of concept by the end of the summer.

Smart glasses can easily capture everything we see and hear in our waking life, so collecting and storing the data is trivial. Actually ingesting this data in a useful way for accurate memory is the hard part.

I've been tinkering with both long context and various RAG techniques to solve this. Using just RAG with frame embeddings has poor results when it comes to understanding motion or relationships within scenes. On the other hand, Gemini's long context window solves these problems and is great with video memory, as seen in Google's Project Astra. But even a context window this big isn't large enough to store context about my entire day (12+ hours of footage), let alone a week.

I'm currently experimenting with a combination of RAG and graph memory. Ideally, RAG lets us send only relevant clips to the LLM, while (multimodal?) graph memory stores complex relationships.

Please do reach out if you have any ideas on how I can make this a reality or if you're interested in helping me build this out!

I'm no expert in agentic AI (I've mostly stuck to web dev), but I'm attempting to learn as much as I can to make this happen.