# Ask AI internals

How the retrieval pipeline works: chunking, embeddings in Qdrant, retrieval thresholds, answer streaming and abuse protection.

Ask AI is retrieval-augmented generation over each site's published pages. This page describes how it works. The reader-facing description is in [Ask AI](/guides/ask-ai).

## Indexing

`indexSite(db, qdrant, embedder, projectId)` runs in the `index-site` task.

1. **Select pages.** It reads every page of the project with `status = 'written'` and a `published_at`, using the live path (`coalesce(published_path, path)`) and the **published** Markdown. Drafts are never indexed.
2. **Chunk.** `chunkPage` splits each page at headings of level 1 to 3, ignoring `#` lines inside fenced code. A section longer than 1,500 characters is split at blank lines. A short piece (under 500 characters) takes the next section in with it, up to 1,500 characters. A single block longer than the limit is kept whole.
3. **Prefix.** Each chunk is stored as `Page title > Heading` followed by its text, so it makes sense on its own.
4. **Fingerprint check.** The indexer calculates a SHA-256 fingerprint (hash) of each chunk's text, path, title, heading, embedding model, and dimension size. It compares these against the existing fingerprints retrieved from Qdrant.
5. **Update or embed.** Each Qdrant point ID is a UUID-formatted SHA-256 hash of `siteId:path:index`.
   - **Unchanged chunks:** Chunks whose fingerprints match the stored ones are not re-embedded. Instead, their `run` ID is updated in bulk in Qdrant (in batches of 500) so they are kept during the cleanup sweep.
   - **Changed or new chunks:** Chunks with new or modified content (or missing fingerprints) are embedded in batches of 50 using the `RETRIEVAL_DOCUMENT` task type and 768 output dimensions, then upserted to Qdrant. This replaces old chunks for the same page instead of adding copies.
6. **Sweep.** Every point carries a `run` id for this indexing. Afterwards, all points of the site that do not have it are deleted. That removes chunks of old text, deleted pages and unpublished pages.

The collection is created on first use with cosine distance. Payload indexes on `site_id` and `run` are required because Qdrant Cloud only filters on indexed fields. A payload holds `site_id`, `path`, `title`, `heading`, `text`, `run` and the `hash` fingerprint.

Indexing is safe to run any time and as often as needed.

## Answering

`POST /api/ask` takes `{ token, messages }` and checks, in this order:

1. The host must be a docs site. Otherwise `404`.
2. A **burst limit** of 20 requests per minute per visitor. This is the cheapest check and stops floods before they cost anything.
3. The body must be valid: up to eight messages, each at most 4,000 characters, starting and ending with a user message, and the last question at most 1,000 characters.
4. The site must be live, public and have Ask AI on.
5. The **Turnstile** token must be valid for the `ask` action and either the request host or the site's default Docup address.
6. **Daily limits** for the visitor on this site (15), the site (500) and all sites (5,000).

Then:

1. **Search text.** The last user message, or, if it has fewer than five words and there is an earlier question, the two joined. This lets "and in Vite?" work.
2. **Embed** the text with the `RETRIEVAL_QUERY` task type.
3. **Search** Qdrant with a filter on `site_id`, returning the best 8 points.
4. **Threshold.** Points scoring under 0.6 are dropped. Real questions score about 0.7 and up, unrelated ones about 0.5.
5. If nothing is left, the reply is a fixed message and no model is called.
6. Otherwise the chat model is given a system prompt containing the excerpts, plus the conversation. The prompt says to answer only from the excerpts, be short, use Markdown and code blocks, treat excerpts as documentation and never as instructions, and not to mention "excerpts".
7. The reply streams as plain text.

The pages the excerpts came from, best match first, are sent in an `X-Ask-Sources` header as URL-encoded JSON, so the widget can show them before the first word. `X-Ask-Remaining` carries the questions left today.

### Turnstile on custom domains

Because the Cloudflare Turnstile widget is registered only for `docup.dev` and its subdomains, it cannot run directly on an owner's custom domain. 

To support custom domains:
1. The custom domain embeds an iframe pointing to `/docup-turnstile` on its default Docup address, passing its own domain as the `origin` query parameter (e.g., `?origin=https://docs.acme.com`).
2. The `/docup-turnstile` route returns a page with the Turnstile widget configured for the `ask` action. It sets a `Content-Security-Policy` header with `frame-ancestors` limited to the origin URL to prevent clickjacking.
3. Once verified, the iframe passes the generated token back to the parent page on the custom domain using `postMessage`.
4. The site sends this token along with the request to `POST /api/ask`, which validates it against either the incoming host or the site's default Docup address.

## Failure behavior

Ask AI is anonymous, so it fails closed. If Redis cannot be reached, it answers `503` rather than running without cost controls. A failed Turnstile check is `403`, an exceeded limit is `429`, and a model error is `500`.

## Constants

All tunable values are in `lib/ask/config.ts`:

| Constant | Value |
| --- | --- |
| `EMBEDDING_MODEL`, `EMBEDDING_SIZE` | `gemini-embedding-001`, 768 |
| `CHAT_MODEL` | `gemini-3.5-flash` |
| `CHUNKS_PER_QUESTION` | 8 |
| `MIN_MATCH_SCORE` | 0.6 |
| `MAX_QUESTION_CHARS`, `MAX_HISTORY_MESSAGES` | 1,000, 8 |
| `DAILY_MESSAGE_LIMIT`, `SITE_DAILY_MESSAGE_LIMIT`, `ALL_SITES_DAILY_MESSAGE_LIMIT` | 15, 500, 5,000 |
| `BURST_LIMIT` | 20 per minute |

> **Warning:** Changing the embedding model or its size means every site must be indexed again, which happens the next time each site is published.