The docs pipeline

The pipeline lives in lib/pipeline/ and is driven by the tasks in trigger/. This page follows one run from start to finish. For the product view, read How Docup works.

1. Read the repository#

readRepo resolves the branch to a commit, downloads the tarball for that exact commit, and streams it through gunzip and tar-stream. For each entry it:

1. strips the leading owner-repo-sha/ folder,
2. runs skipReason(path, size) and the owner's ignore matcher,
3. stops keeping files at 1,500 files or 12 MB,
4. buffers the rest, then runs contentSkipReason on the bytes.

The result is a RepoSnapshot: { sha, files: Map<path, text>, skipped }. Ignore patterns are compiled by createIgnoreMatcher: a trailing-slash name is a folder at any depth, a pattern with * becomes a regular expression (** matches across folders, * within one, and a glob without a slash matches the file name anywhere), and anything else is a path or path prefix.

2. Tools#

The model never touches GitHub. createRepoTools(snapshot) returns three read-only tools over the in-memory snapshot: list_files, grep and read_file. Each limits its output (300 paths, 40 matches, 400 lines) so a long answer cannot flood the context.

toolStepLimit(n) allows n - 1 tool steps and forces the last step to be a plain answer by setting toolChoice: "none" and clearing the list of active tools. Without this extra precaution, some models (such as Gemini) occasionally ignore the none choice and attempt further tool calls, running out of steps and returning nothing.

3. Plan#

planDocs calls the planner model with the repository name, your instructions, up to 500 file paths and the first 6,000 characters of the README. It has 15 steps, a six-minute timeout, and a structured output schema: pages, each with path, title, summary and sourcePaths.

toPlannedPages cleans the result. Paths are lowercased, non-alphanumeric runs become hyphens, at most two path segments are kept, duplicates are dropped, files the model invented are removed from sourcePaths, and the list is capped at 15. A plan with no pages fails the run.

4. Write#

writePageWithRetries writes each page with generateText, the writer model and the same tools, 12 steps and a four-minute timeout. The system prompt requires plain GitHub-flavored Markdown with no frontmatter, no HTML and no top-level heading, fenced code with a language, callouts only in the three allowed types, links only to pages in the outline, and no file paths or line numbers in prose.

The user prompt carries the page title, path and summary, your instructions, the starting files, and the list of all pages in the docs.

Quality gates:

• A wrapping Markdown fence in the reply is stripped.
• A reply is checked by looksLikePage: it must be at least 80 characters long and contain a markdown heading (# or ## ), or be at least 400 characters long.
• If the reply does not look like a page, a fallback call runs without tools. It receives all the tool results from the first call (up to 120,000 characters) and is instructed to write the complete page from what was already read. If the fallback reply still does not look like a page, it throws an error.
• On failure, the page is tried again up to four attempts. The pause between retries increases with each attempt, waiting 1,500 ms * attempt (or 20,000 ms * attempt if the error was due to exceeding a quota or rate limit).
• From the third attempt, the stronger planner model is used because the fast model already failed twice. A retry also tells the model to read only the files it needs and to finish with the full page.

During a full repository sync, pages are written four at a time with mapWithLimit, which keeps results in order. During updates after a push, pages are rewritten three at a time.

5. Save#

Pages are inserted as pending, then updated one by one to written or failed, with markdown and ai_markdown set to the same text. A run where every page failed is a failed run. Otherwise the project becomes draft_ready and last_synced_sha is stored.

Updating after a push#

update-docs reuses the same reader, tools and writer. The additions:

1. Diff. compare calls GitHub's compare endpoint between last_synced_sha and the head. It returns null when the old commit is gone or the diff lists 300 files or more.
2. readDiff. From the file list alone, it works out renamed and removed files, the pages whose sources changed (affected), pages whose sources are all gone (orphaned), and changed files no page covers (uncovered) that pass the path rules.
3. Early exit. If nothing is affected or uncovered, the run succeeds without any model call.
4. Daily limit. A Redis counter allows 15 AI updates per workspace per UTC day.
5. decideDocsChanges. For uncovered files, the writer model sees each file's first 60 lines (40,000 characters in total), the page list and commit messages, and returns structured output: files to attach to existing pages and a yes/no for new pages.
6. planNewPages. Only on "yes", the planner model plans at most min(3, 30 - existing) pages. Existing paths are never reused.
7. Rewrite. Each affected page is written with the current text in the prompt and an instruction to keep structure and wording and change only what the code makes wrong. If the owner edited the page, the prompt adds that their wording must survive. An identical result is a no-op.
8. Apply. An unedited page is saved with a compare-and-set on its current text. If the text moved in the meantime, the result becomes a suggestion. On a live site, an unedited page also updates published_markdown.
9. Finish. If anything went live, the site cache is cleared and index-site is triggered. last_synced_sha moves forward only on success.

Failure behavior#

• Model calls have timeouts, and a stuck call is cut off so the page is retried.
• Tasks do not retry as a whole: retry: { maxAttempts: 1 }.
• A failed page never erases existing text.
• Errors stored on a run are shortened and free of sensitive details; the full error stays in the task log.