# Optimize for search and AI answers

Everything Docup does for search engines and AI assistants, and the settings that make your docs easy to find and quote.

People find documentation through search engines, through AI assistants that answer questions, and through links others share. Docup builds for all three by default. This guide lists what is automatic and what you can tune.

## What every site gets automatically

| Feature | What it does |
| --- | --- |
| **Page titles and descriptions** | Each page has a title and meta description. They default to the page title with the site name, and the page summary. |
| **Canonical links** | Each page declares its own address as canonical, and points to its Markdown copy. |
| **Structured data** | JSON-LD describes the page: a breadcrumb trail everywhere, a `WebSite` entry on the home page and a `TechArticle` with headline, description and modified date on other pages. |
| **Open Graph and Twitter cards** | Pages carry `og:` and `twitter:` tags with a 1200 by 630 image. |
| **Sitemap** | `/sitemap.xml` lists every indexable page with its last modified time. |
| **robots.txt** | Allows crawlers, points to the sitemap and optionally turns away AI crawlers. |
| **llms.txt** | An index of your docs for AI tools, plus `llms-full.txt` with every page in one file. |
| **Markdown pages** | Every page is also served as clean Markdown at `<page>.md`. |
| **Fast, server-rendered pages** | Pages arrive as complete HTML, and finished pages are cached. See [Performance](/docs-site/performance). |
| **Generated social images** | Every page gets its own card with its title and summary. |

## Tune titles and descriptions

### For the whole site

Open [Search and sharing](/settings/search) and write a **Site title** and **Description**. They are used for the home page and as the fallback for the site. Keep the title under 70 characters, and describe what your product does and who the docs are for in 120 to 160 characters. The settings page previews the Google listing.

### For one page

Each page has its own panel next to the editor:

- **Search title**: replaces "Page title \| Site name" completely. Include the keyword people search for, and the product name. Aim for 50 to 60 characters; the limit is 70.
- **Search description**: the snippet shown under the title. State what the reader will learn or be able to do, in 120 to 160 characters. The limit is 300. An empty field defaults to the page summary.
- **Hide from search**: adds `noindex` and removes the page from the sitemap and `llms.txt`.

Write titles that match the question: "Connect GitHub to Docup" beats "GitHub". Use one idea per page so each page can win one query.

## Help AI assistants cite you

Assistants such as ChatGPT, Claude and Perplexity look for pages that answer a question directly. To be easy to quote:

- **Lead with the answer.** The first paragraph of a page should say what the page covers in plain words.
- **Use question-shaped headings** where it fits: "How are updates triggered?".
- **Keep sections self-contained.** Ask AI and many crawlers read pages in sections, split at headings.
- **Use tables and lists** for facts such as limits, options and defaults.
- **Keep `llms.txt` on** so tools can read the whole site in one request.
- **Allow AI crawlers** in [Search and sharing](/settings/search) if you want to be quoted. You can see which bots visit in [Analytics](/guides/analytics).

## Social sharing

By default each page generates its own card with the page title and summary, which makes shared links look specific. If you prefer one image for the whole site, switch the **Social image** option to **My own image** and upload a 1200 by 630 PNG, JPG or WebP up to 2 MB.

## Control crawlers

| Goal | Setting |
| --- | --- |
| Keep the site out of search entirely | Turn **Search engines** off. All crawlers are blocked and pages get `noindex`. |
| Appear in search but not in AI training or answers | Keep **Search engines** on and turn **AI crawlers** off. |
| Disable files generated for AI assistants | Turn **llms.txt** off to disable both `/llms.txt` and `/llms-full.txt`. |
| Hide one page | Use **Hide from search** on that page. |
| Add a custom rule | Use **robots.txt** extra lines, up to 1,000 characters. |

Known AI crawlers are asked to stay away with `Disallow: /`. Well-behaved crawlers follow it; `robots.txt` is a request, not enforcement. Docup requests the following bots to stay away when AI crawlers are turned off: `GPTBot`, `OAI-SearchBot`, `ChatGPT-User`, `ClaudeBot`, `Claude-User`, `Claude-SearchBot`, `PerplexityBot`, `Google-Extended`, `CCBot`, `Applebot-Extended`, and `Bytespider`.

## Check your work

1. Open `/sitemap.xml`, `/robots.txt` and `/llms.txt` on your site.
2. View the source of a page and look for the title, description, canonical link and JSON-LD.
3. Paste a page link into a social preview tool or a chat app and look at the card.
4. Submit the sitemap in Google Search Console.
5. Return to **Analytics** after a few days and watch crawler visits and 404s.

Fix every address listed under **Not found (404)**; each is a link someone followed that led nowhere.