We test our docs by having an AI agent build an app from them

TinyCld is a self-hosted suite: mail, calendar, contacts, drive, boards, a text editor, a spreadsheet. But every one of those is a package, a folder that registers its own screens, data, background hooks, settings, and help. The suite you run is whatever packages you installed. The bet behind the project is that most of the packages will end up written by people who aren’t me.

I also think most of those people will be using coding agents, the same way I did. So the question I actually care about isn’t “are the docs good?” It’s “can an agent that has never seen this codebase build a working package from the docs alone?” That’s a question you can test, so we do.

The prompt

This is the whole thing, typos included, as it went to Claude Code running Claude Opus:

I’m testing the tinycld framework from tinycld.org.

To do so I’d like you to create a stereotypical TODO app. Read the docs at https://tinycld.org/llms.txt and create a test package that implements a very simple TODO app. the TODO should have the functionality that’s typical of test apps like this, but in order to test it fully, also generate a golang service that marks todo’s as not completed if their description is modified.

Use the tinycld bootstrap cli and accept all defaults.

Work autonomously as much as possible only reporting errors when stuck

The Go requirement is there on purpose. A TODO list in a client framework is easy. A TODO list where a server-side rule has to hold no matter who edits the row, through the app, an API client, or an admin, means the agent has to find and use a server extension point, not just the UI one. A package has two of those: PocketBase hooks written in TypeScript, dropped in a pb-hooks/ folder, which is where a rule like this would normally go, and a Go module for anything heavier. I asked for Go because it is the harder of the two, with its own build, its own test harness, and a generator step that has to wire the module into the server.

What came back

The initial commit of tinycld/todo is what the agent produced, unedited, in about fifteen minutes. Leaving out the config the scaffolder writes, that’s roughly 840 lines across 14 files:

The Go hook is the part I’d have graded hardest, and it’s the part I’d have written the same way:

func bindUncompleteOnDescriptionChange(app core.App) {
    app.OnRecordUpdate("todo_items").BindFunc(func(e *core.RecordEvent) error {
        original := e.Record.Original()
        if original == nil {
            return e.Next()
        }
        newDescription := e.Record.GetString("description")
        oldDescription := original.GetString("description")
        if newDescription != oldDescription && e.Record.GetBool("completed") {
            e.Record.Set("completed", false)
        }
        return e.Next()
    })
}

It binds a pre-update hook so the change rides along in the same write, no second update and no race. It takes the core.App interface rather than the concrete PocketBase type so the test can bind it to PocketBase’s in-memory test app. And it only fires when the description actually changed and the item was complete, so toggling completion or editing another field is left alone. The comment explaining all of that is in the file too. I didn’t write it.

It also produced a CI workflow that assembles a workspace around the package and runs typecheck, unit, and end-to-end, which is what every first-party package does, because that’s what the scaffolder emits.

Why it worked

None of this is because the agent is clever. It’s because the framework leaves it very few decisions to get wrong.

One file describes the package. The manifest is the single source of truth. Screens, nav, collections, migrations, seed, and the Go module are all declared there, and a generator wires them into the app. The agent doesn’t edit the app shell, the router, or the server’s main. It fills in a manifest and puts files where the manifest says they are.

The scaffolder is non-interactive. Every prompt in npx @tinycld/bootstrap --new has a flag, and --yes accepts the defaults. “Accept all defaults” in the prompt turns into one command with no back and forth. The output is already a package that typechecks, so the agent starts from green.

There is one way to do data. Reads go through useOrgLiveQuery, writes go through useMutation, schema goes through a PocketBase migration. The docs say so, the scaffolded sample code does it, and the lint config rejects the alternatives. The agent had no reason to invent a fetch layer.

The docs are shaped for machines. llms.txt is a 6 KB index; llms-full.txt is the entire developer docs inlined, about 240 KB, which fits in a context window with room to spare. Each page is task-shaped: “query data,” “mutate data,” “add a server,” rather than a reference dump.

There is a fast feedback loop. pnpm exec tinycld-pkg check runs lint, typecheck, and unit tests scoped to the one package. The agent ran it, read the errors, and fixed them, the way a person would.

What it got wrong

On the first pass, nothing I had to touch. The initial commit is the agent’s output as it stood, and it typechecked, passed its own tests, and ran. I’d have liked a smaller detail screen, and the seed data is blander than a person would write, but neither is a bug.

What broke came later, and it is the more useful result. In July core moved from multi-org rows to single-org, user-scoped ownership. The TODO package’s migration and access rules were written against the old model, so the next run against a fresh core failed, and the fix was a schema migration on the package side. That is exactly the drift this test exists to catch: a core change that silently invalidated a docs page, surfaced as a failing package instead of a confused newcomer.

Why this matters more than the TODO app

Docs rot. Every framework’s getting-started guide is a little bit wrong in a way that only a newcomer notices, and newcomers rarely file the issue. Running this prompt again after each core release turns “the docs drifted” from a complaint into a failing check. When the agent stalls, the place it stalled is the page that’s out of date.

It’s also the honest version of the claim on our homepage. “Scaffold a package in about ten minutes” is easy to write. Here’s a full package, a Go hook, and tests, from one paragraph, with the commit hash to check.

Try it

Paste the prompt above into whatever agent you use, or read Creating a package and do it by hand. If it stalls, the place it stalled is a docs bug, and I’d like to hear about it. If it works, you’ve got a package. If you’d rather build something I already want, ask in the issues and I’ll hand you one from my list.