Aligning pen strokes with locally transcribed audio on an e-ink tablet

Sean · 2 August 2026 · notes from an unfinished build

I take meeting notes on a reMarkable Paper Pro Move. The problem with paper, and with paper-like things, is that three weeks later you find a line that says resource allocation and it means nothing. You wrote it because someone said something. The something is gone.

So I spent a while making the ink and the audio agree. The recording and the transcription run on my Mac; the tablet is where I write. When the meeting ends a document comes back to the tablet with the transcript on the left and my handwriting in the margin, each line joined by an arrow to the part of the transcript that was being spoken while I wrote it.

These are notes on the parts that were harder than I expected. It has run end to end once, on my own setup. There is no installer and nothing to download.

Two devices, two clocks

The naive version is to timestamp strokes with wall-clock time on the tablet, timestamp the transcript with wall-clock time on the Mac, and subtract. This does not work well enough. The two clocks disagree, they disagree by an amount that drifts, and NTP corrections can move one of them mid-meeting, which is a wonderful way to produce notes that are confidently attached to the wrong sentence.

What I do instead: wall time is metadata only, kept for display and audit and never used for placement. Pairing establishes clock samples between the two machines, and from then on every line event and every transcript segment is recorded as elapsed milliseconds from one session epoch. There is exactly one epoch per meeting and both sides measure against it. Placement is then subtraction on a single monotonic axis rather than a negotiation between two calendars.

The consequence worth stating plainly: a session is the unit of correctness. If you cannot say which session an event belongs to, you cannot place it, and the honest response is to not place it.

A stroke is not a word

I anchor on stroke start — the moment the pen goes down, not when the line is finished. You start writing while the sentence is still being said, and if you anchor on completion you land on whatever came next.

Even then, the timing is only good to about a second. Not because the tablet's input is slow, but because the gap between hearing something and starting to write it is itself variable and roughly that size. So I do not pretend to word-level precision. A note anchors to a paragraph and a phrase inside it, not to an exact word. Claiming the exact word would be a better demo and a worse tool.

When it cannot tell where a line belongs, it leaves the line in the margin with no arrow at all. This is the rule I am most attached to: every acknowledged line ends up either anchored or loose, and never silently dropped. A note in the wrong place is worse than a note with no arrow, because a wrong arrow is a claim and a missing arrow is just an absence you can see.

What e-ink does to a live interface

I wanted an audio level meter on the tablet so you can see the Mac is really hearing the room. On an LCD this is free. Here it is a budget.

The panel cannot absorb rapid updates, so level updates are capped at about 10 Hz and are explicitly lossy — they are exempt from the monotonic sequencing everything else obeys, because a dropped level frame costs nothing and a queued one costs you a stale meter. Most of the level data never reaches the screen and should not. Designing for e-ink is mostly deciding what to throw away.

A related annoyance: there are no screenshots. Calling QML's grabToImage on this hardware segfaults the stock app, so every image or video of this thing is a camera pointed at a screen, including mine. If you have solved that on this platform I would like to hear it.

Two audio tracks, never mixed

The Mac captures the microphone and the system audio as two separate tracks and never mixes them. Mixing early is tempting and destroys the thing you need later: track identity has to survive all the way to diarization, because "someone in the room said this" and "someone on the call said this" are different facts, and once summed you cannot recover which is which.

What broke

The failure I care about most: two people talking over each other collapsed into one wrong sentence. Not garbled — plausible, and wrong. A rehearsal with two input voices also produced three speaker labels. Diarization runs as a post-pass over the finished recording, and overlapping speech is where it earns or loses its keep. This is why I will not make an accuracy claim, and why the returned document shows the transcript rather than a summary. A summary of a sentence that was never said is a very efficient way to be confidently wrong.

Where it is

One end-to-end run, on one setup: a Paper Pro Move and an Apple-silicon Mac with 16 GB. The transcription runs on the machine — no audio is uploaded anywhere, which was the point rather than a feature. It is not finished, there is no public installer, and it does not run on the rM1 or rM2, which are a different CPU architecture entirely.

Recording other people is a thing you should have agreement about. That is not a feature I can build for you.


If you want the download link the day it opens — I expect that to be later this month — you can leave an address on the project page. There is nothing to download yet. I am happy to answer questions about any of the above.