I ran Needle 2, a 45M parameter model for tool calling on an ESP32-S3.
I have conveniently trimmed out the 78 seconds prefill stage so you can now only see the ultra fast 1.27 tps tool call generation.
It’s not about the utility, it’s about the friends we made along the way ✨
I recently purchased this piece of hardware and was overloaded with ideas on things to build with it. Before I jumped to my larger plan, I wanted to quickly try Needle 2 model on an ESP32.
Since this hardware has a microphone, speaker & a touchscreen (+ a few sensors), my plan is to hook up my Hermes Agent, manage work Claude agents, other home automation stuff etc. through voice and maybe through the touch screen but mostly voice. Can’t think of practical interaction patterns for the touch screen right now other than the obvious visual read-only view.
I am planning to setup a Pipecat based backend for the voice assistant and this box will sit on my desk. I have a few ideas on modes of operation:
- A wake word detection kind of setup and talk to my Hermes Agent and whatever.
- Always on rolling buffer kind of setup like I did in hotmic, a tool call model like Needle 2 inferences if any action is required and asks for permission, and after this, it can be offloaded to my Hermes Agent or whatever other kind of deterministic tasks/scripts/actions that I want to configure.
In all of this, this box should act as a thin collector layer to stream audio in and out, inferencing can happen somewhere else (unlike this experiment that I tried) on one of my old GPU machines.
I’ll write a longer post once I finalize how I want to make my desk agent and build it.