I measured Ponytail, and the saving was not where I expected

Ponytail made my AI coding bill smaller. I could not say why. So I built the same project twice, with it and without it. My guess was wrong.

A colleague mentioned Ponytail in passing: it makes AI coding cheaper. I installed it and forgot about it. A few weeks later my bill was lower and the pull requests from my AI sessions were smaller.

Good news twice. But I could not explain what the tool was doing, and for something sitting in the middle of my daily work, that bothered me.

01What I thought it did

I thought it sent less of my code to the AI. It does not. The AI reads exactly as much as before. What changes is how much it writes.

Tools like Serena and Repomix trim what the model reads, and I had filed Ponytail next to them. So had every colleague I asked.

So "token reduction tool" is not quite the right name. The tokens do go down, but only because something else happens first.

02How it works

Before writing anything, the AI walks down seven questions. The first "yes" wins, and it stops there.

Programmers have an old name for this: you aren't gonna need it. The rules are not new. What is new is that they get applied every time, by something that never gets tired or decides this one abstraction is worth building.

03The test

Two public benchmarks exist, and they disagree: one says 54% less code, the other 15%. Both can be honest. The tool can only save you something where the AI would have overbuilt, so the number depends on the tasks you pick. That is a reason to test on your own work.

So: one spec for a cricket scoring service, with eleven endpoints and some fiddly rules. Two AI sessions, same model, empty folders, one sentence of instruction each. One with Ponytail, one without. Twenty hidden tests judged both afterwards.

The full comparison, graded by the same twenty tests:

Both builds lost one mark on the same unclear line in my spec. Each lost its second mark on a different bug, so the lean build was not just a cut-down copy of the other.

04Where the money really was

Here is the surprise. Writing code is almost none of the bill.

So how did the bill halve? Every time the AI takes a turn, it re-reads all the code so far. Less code means cheaper turns, and fewer of them: 13 instead of 24. Re-reading fell by 57%, and re-reading is 94% of the bill.

Less code written today is less code to re-read on every turn after that.

You are not buying cheaper writing. You are buying a smaller thing to carry.

05Three things I had wrong

06Should you use it?

The money is the least interesting part. 63% less code is 63% less for a person to read, review and maintain for the life of the project.

The danger: lean code and incomplete code look the same. Sometimes "nobody asked for it" is a safety check you forgot to mention. The AI does not know your roadmap, your regulator, or last quarter's incident. So ask one thing first: who checks what the AI produced?

07How I nearly got the wrong number

Four mistakes almost gave me a confident, wrong answer. If you run this yourself, watch for them.

1. The logs repeat themselves. One reply is logged on several lines, each with the full usage figures. Adding them up inflated my totals 2.29 times. Count each request once.
2. One run logged nothing. It was treated as a child session, so no usage was recorded. I threw it away and ran it again.
3. My grader assumed a file layout. Both builds would have scored zero. Now it starts each build and talks to it over HTTP.
4. The grader failed silently. It reported zero passes and zero failures. That looks like an answer. It was a bug.

I also checked each finished session's log to confirm Ponytail was really loaded in one and really absent in the other, instead of trusting the settings.