A few months ago, I was calling a travel agency almost every day over a dispute. After too many repetitive calls, I had a simple thought: “Why couldn’t I give a voice agent the relevant documents and let it handle this conversation for me?”
That question got me exploring voice agents.
I started looking at the best products in the space. I started experimenting with open-source projects like Pipecat, LiveKit, and OpenAI’s Whisper, playing around and trying to understand where the latency in a voice-to-voice system actually came from.
My agent needed to retrieve documents and call APIs. That meant values from the user’s speech eventually had to become structured tool arguments: order IDs, room codes, phone numbers, dates, amounts, and similar values.
I was using Deepgram for speech-to-text. An ASR transcript could look perfectly understandable to a person:
My order ID is seven eight three two nine and my room code is one oh five.
But the API behind the agent did not want spoken English. It wanted something closer to:
order_id = 78329
room_code = 105
Deepgram’s number formatting helped, but it was not reliable enough for the structured values I needed. If seven eight three two nine becomes the wrong order ID, or one oh five becomes the wrong representation, every stage after it can work perfectly and the tool call will still be wrong.
That led me down the rabbit hole of inverse text normalization, or ITN: converting spoken-form text from an ASR system into the written forms that downstream systems actually expect.
The hard part wasn’t converting words to numbers
Some transformations are straightforward:
one hundred twenty three → 123
twenty dollars → $20
There are well-defined rules behind these transformations. Given the spoken form, a deterministic parser can usually calculate the written value.
But then I started running into examples like:
meet me at two thirty
→ 02:30
the room number is two thirty
→ 230
The same spoken form can have more than one structurally valid interpretation.
I was actually dealing with two separate problems:
- What written forms are valid for a piece of spoken text?
- Which valid form makes sense in this sentence?
The first problem is mostly deterministic. We already know how numbers, dates, times, currencies, phone numbers, and other structured values are formed.
The second problem is contextual. A parser looking only at two thirty cannot always know whether the speaker meant a time, room code, or something else.
A deterministic baseline: text-processing-rs
I first tried text-processing-rs, a deterministic inverse text normalizer.
It uses grammar rules to convert spoken forms into written values. When multiple interpretations are possible, it prefers longer matches and follows a fixed parser order:
Money > Measurements > Dates > Times > Decimals > Ordinal numbers (first, second, 23rd) > Regular numbers (one, twenty three, 105)
It was extremely fast, but the parser order does not change with the surrounding sentence.
One of the failures I eventually retained in my benchmark was:
the room code is one oh five
text-processing-rs:
the room code is 01:05
expected:
the room code is 105
01:05 is a perfectly reasonable interpretation of one oh five in isolation. The problem is the words room code should change the decision.
I could keep adding custom rules for cases like this:
"room code" → prefer an identifier
"meeting starts at" → prefer a time
but that would quickly turn into a growing collection of hand-written contextual exceptions.
So I started looking for an ITN system that could actually use the surrounding sentence when making the decision.
A contextual baseline: Thutmose
The next system I found was NVIDIA’s Thutmose Tagger, a contextual neural ITN model. Its released English model passes the sentence through bert-base-uncased once, then uses two classification heads to make predictions for each token:
Replacement tag:
What should this word become?
e.g. <SELF>, <DELETE>, _19, 8, 4_
Semiotic class:
What type of value is this?
e.g. DATE, TIME, MONEY, CARDINAL
Thutmose uses tags such as <SELF> for unchanged tokens, <DELETE> for removed tokens, and _19, 8, 4_ for replacements; underscores mark how the output should attach when reconstructing the final text.
For example, Thutmose represents the sentence at the token level like this:
INPUT WORD REPLACEMENT TAG SEMIOTIC CLASS
the <SELF> PLAIN
year <SELF> PLAIN
nineteen _19 DATE
eighty 8 DATE
four 4_ DATE
The replacement tags are then combined to produce 1984.
But when I tested it on the structured values I cared about for voice-agent tool calls, I still found surprisingly simple failures:
input:
the room code is one oh five
Thutmose:
the room code is 1 oh 5
expected:
the room code is 105
Two things about the architecture started bothering me:
First, the neural model has to learn both:
What type of value is this?
+
How should each input word be transformed?
Second, training a token-level tagger requires token-level targets. A dataset might originally contain:
spoken: the year nineteen eighty four
written: the year 1984
but Thutmose needs something closer to:
the → <SELF>
year → <SELF>
nineteen → _19
eighty → 8
four → 4_
Its preprocessing pipeline therefore has to derive these alignments before training. Thutmose uses GIZA++ as part of this process, so alignment errors can become incorrect training labels.
My first attempt: BIO tagging
I already had deterministic parsers from text-processing-rs that could normalize a value once they knew its span and type.
So I split the problem:
Neural model:
Find the span + predict its type
Deterministic parser:
Convert that span into the written value
I used BIO tagging: B marks the beginning of a span, I marks tokens inside it, and O marks everything else.
For example,
the O
room O
code O
is O
one B-DIGIT_SEQUENCE
oh I-DIGIT_SEQUENCE
five I-DIGIT_SEQUENCE
I used a DeBERTa-v3-small token classifier with 10 normalization classes based on the deterministic types I wanted to support. With a B and I label for each class, plus O, that gave me 21 possible labels.
This was cleaner than predicting normalized text directly: DeBERTa found the span and type, while deterministic code handled the transformation.
But validation exposed the weak point: span boundaries. One wrong token could send the parser the wrong span.
Then I realized the deterministic parsers could already enumerate:
which spans are parseable
+
which types each span could belong to
+
what written values they could produce
I was using a neural model to rediscover structure I could enumerate exactly.
That led to the architectural change that became the core of Premove:
Instead of asking the model to discover the candidates, why not generate every structurally valid candidate first and ask the model only to choose between them?
The final architecture
The idea was simple:
Let deterministic code generate the structurally valid outputs for every contiguous span. Let the neural model choose between them.
I no longer asked the model to discover spans, predict classes, or construct normalized text.
Generating candidates from every span
If a sentence has n tokens, there are n(n + 1) / 2 contiguous spans, so there are O(n²) spans to consider.
the room code is one oh five
has 7 tokens, which gives 28 possible spans.
Conceptually, Premove tests each span against the deterministic parsers:
the
the room
...
one
one oh
one oh five
oh five
Most produce nothing. But one oh five can produce multiple structurally valid candidates:
105 → identifier / number-like interpretation
01:05 → time interpretation
In the actual implementation, I do not make a separate Python→Rust call for every span. Premove collects the span texts, deduplicates repeated ones, and sends them to Rust in one batch. But logically, every contiguous span is still checked for deterministic output.
Scoring candidates in context
Once the deterministic parsers generate candidates, Premove has to decide which one fits the sentence.
The full sentence passes through DeBERTa-v3-large once, producing a contextual vector for every token.
Each candidate is then scored using three features:
1. Contextual source span
For the source span, we take the first token vector, the last token vector, and the mean of all token vectors in the span.
one oh five
│ │ │
▼ ▼ ▼
h1 h2 h3
The mean summarizes the whole span, while the first and last vectors preserve information about its boundaries and order that an average alone can lose.
2. Candidate type
Each candidate also stores which parser types produced it.
Premove supports 13 types, so I represent them as a 13-value vector where each position represents one parser type:
[1, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0]
where
CARDINAL 1
DIGIT_SEQUENCE 1
TIME 0
DATE 0
...
1 = this type produced the candidate
0 = it did not
3. Candidate replacement
Finally, the scorer needs to know the actual replacement being proposed.
one oh five → 105
one oh five → 01:05
Since their source representation is identical, the scorer also represents the replacement itself. I also encode the proposed replacement using DeBERTa’s token embeddings, without another Transformer pass.
The three features above are concatenated and passed through a small feed-forward network that outputs one score per candidate.
The gold graph
There might be more than one candidate path that produces the correct target.
The full candidate graph contains both correct and incorrect possibilities:
one oh five → 105
one oh five → 01:05
one → 1
oh → 0
five → 5
...
The Gold Graph keeps only the paths that reconstruct the target 105 exactly.
Training with a structured loss
During training, each example contains the spoken input and its expected normalized output:
input: the room code is one oh five
expected: the room code is 105
Premove generates the candidate graph from the input, then uses the expected output to build the Gold Graph containing only paths that reconstruct that target exactly.
The model assigns a score to every candidate. A path through the sentence gets the sum of the scores of the candidates it uses.
During training, we compare all paths through the candidate graph with only the paths in the Gold Graph:
The loss is:
L = log Z_all − log Z_gold
Z is the sum of the exponentiated path scores in the corresponding graph, computed with dynamic programming rather than enumerating every path.
So if the model gives a high score to one oh five → 01:05, the loss increases.
If it gives more score to paths that produce 105, the loss decreases.
This also gives us hard negatives for free: every structurally valid Rust candidate outside the Gold Graph automatically competes with the correct ones.
Decoding with dynamic programming
Candidate scores cannot be considered independently because their spans can overlap.
┌──── one oh five → 105 (3.0) ────┐
START ──────┤ ├──── END
└─ one oh → 10 (2.4) ─ five → 5 (1.0) ─┘
The bottom path has no overlapping candidates and has the higher total score.
Premove represents candidates as edges over the source sentence, with unchanged characters as KEEP edges scored at 0. Dynamic programming moves left to right, stores the best score at each position, and returns the highest-scoring path.
Training the final model
The final DeBERTa-v3-large model was trained on 418,000 examples: a broad Google ITN base followed by conversational and structured-value adaptation. The later stages included replay from earlier data to reduce forgetting, and the entire model was fine-tuned rather than only the scoring head.
Results
I evaluated the final model against NVIDIA Thutmose and text-processing-rs on a frozen 1,500-row synthetic benchmark, including a dedicated 400-row voice-agent subset.
| Backend | Voice-agent accuracy | Overall accuracy | Mean latency |
|---|---|---|---|
| Premove ITN | 99.50% (398/400) | 89.70% | 56.49 ms |
| NVIDIA Thutmose | 67.00% (268/400) | 59.39% | 15.98 ms |
| text-processing-rs | 68.25% (273/400) | 55.79% | 0.14 ms |
Premove was much more accurate on the voice-agent subset while keeping warm inference under 60 ms.
The benchmark was held out from training and checkpoint selection, but it is synthetic rather than production traffic.
Limitations
Premove is relatively large: DeBERTa-v3-large gives the scorer roughly 435 million parameters, with a first download of around 1.6 GB. It is also English-only today.
The current benchmark is synthetic, so the results should not be read as universal ITN performance. Premove is strongest on the structured, context-dependent cases I built it for.
The main lesson I took from building it was not that neural models should replace rules. It was the opposite:
Use deterministic code to define what outputs are structurally valid. Use a neural model where context is needed to resolve ambiguity. Then use an exact algorithm to make the final choices compatible.