Listening to my Mac

I wanted my Mac to transcribe the sound coming out of it, not the microphone, live, for free, and without sending audio to a server. The result is a small menu bar app called System Audio Agent. This post covers the two problems that took the most time.

The repo was not mine first. Anand Velpuri started it in November 2025 and wrote the early versions. I picked it up in July and reworked most of it.

Capturing audio without a virtual driver

The usual way to record what a Mac plays is to install a virtual audio device and route sound through it. Then you forget to switch it back and your AirPods go silent. I did not want anyone installing a driver for a small app.

Apple's screen recording API can capture audio too, but it insists on video as well. So the app requests a 2 by 2 pixel frame once a second and ignores it. The catch is that macOS now treats the app as a screen recorder, so on first launch it asks for screen recording permission for an app that only wants sound. Slightly awkward, but acceptable.

Keeping the transcript intact

For speech to text I used Apple's built-in speech recognizer, set to run on device. There is no API key and nothing leaves the machine.

The recognizer returns live guesses while someone speaks. After a pause, though, it often discards its guess and starts fresh, without signalling that the previous sentence was finished. So every time the speaker paused, the transcript lost text.

The fix is simple:

Keep two pieces of text: everything confirmed so far, and the current guess.
When a new guess does not continue the previous one, save the previous one before replacing it.
Number every recognition session, so a late result from an old session cannot overwrite the new one.

Takeaways

The screen recording API is a driver-free way to capture system audio, at the cost of a screen recording permission prompt.
Live speech recognizers can silently reset. Track confirmed text separately from the current guess.
It is not a lot of code, but it took much longer than it looks. It is English only for now, with the language hardcoded to en-US.

Capturing audio without a virtual driver

Keeping the transcript intact

For speech to text I used Apple's built-in speech recognizer, set to run on device. There is no API key and nothing leaves the machine.

The fix is simple:

Keep two pieces of text: everything confirmed so far, and the current guess.

When a new guess does not continue the previous one, save the previous one before replacing it.

Number every recognition session, so a late result from an old session cannot overwrite the new one.

Takeaways

The screen recording API is a driver-free way to capture system audio, at the cost of a screen recording permission prompt.

Live speech recognizers can silently reset. Track confirmed text separately from the current guess.

It is not a lot of code, but it took much longer than it looks. It is English only for now, with the language hardcoded to en-US.