# Search and sharing

Control search results, social images, crawlers, robots.txt and the machine-readable files Docup generates for AI tools.

**Settings → Search & sharing** decides how your docs appear in search engines, in link previews and to AI tools. Everything here takes effect on the live site when you save.

## Search results

**Site title** and **Description** are used for your home page and as the defaults for the site.

- The title defaults to your project name and can be up to 70 characters.
- The description defaults to the summary of the Overview page and can be up to 300 characters. 120 to 160 characters works best.
- A preview shows how the listing looks on Google.

Every other page uses its own title and summary. Change them per page in the editor; see [The page editor](/guides/editor). A page title appears as "Page title | Site title" unless you set a search title for that page.

## Structured data and canonical links

Every page includes structured data (JSON-LD) that tells search engines what the page is: a breadcrumb trail on every page, a `WebSite` entry on the home page and a `TechArticle` entry on other pages. Each page also declares its canonical address and points to its Markdown copy. You do not need to configure any of it.

## Social image

Choose what is shown when a link is shared in Slack, X, LinkedIn or a message.

- **Generated**: a card per page showing the page title and summary on a card themed to your site. This is the default.
- **Custom**: one image used for every page. PNG, JPG or WebP, 1200 by 630 pixels, up to 2 MB.

Save after uploading to switch to your image.

## Crawlers and AI

| Setting | When on | When off |
| --- | --- | --- |
| **Search engines** | Google, Bing and others may list your docs. | Every crawler is blocked and every page gets `noindex`. |
| **AI crawlers** | AI bots may read your docs. | `GPTBot`, `OAI-SearchBot`, `ChatGPT-User`, `ClaudeBot`, `Claude-User`, `Claude-SearchBot`, `PerplexityBot`, `Google-Extended`, `CCBot`, `Applebot-Extended` and `Bytespider` are asked to stay away. |
| **llms.txt** | Publishes `llms.txt`, `llms-full.txt` and a `.md` version of every page. | Those files are not served. |

AI crawlers cannot be allowed while search engines are off, because everything is already blocked.

### Extra robots.txt lines

Add your own lines at the end of the generated `robots.txt`, up to 1,000 characters:

```text
User-agent: SomeBot
Disallow: /internal/
```

## Generated files

These files are built from your published pages and update when you publish. The settings page links to each of them.

| File | What it is |
| --- | --- |
| `/sitemap.xml` | Every indexable page with its last modified time. Pages hidden from search are left out. |
| `/robots.txt` | Crawl rules, a link to the sitemap and your extra lines. |
| `/llms.txt` | A title, a summary and a link to every page grouped by section, following the llms.txt convention. |
| `/llms-full.txt` | All pages in one Markdown file, for tools that read a site in a single request. |
| `/<page>.md` | One page as Markdown. For example `/guides/editor.md`, and `/index.md` for the overview. |

In the Markdown files, callouts become plain block quotes such as `> **Note:** ...` so any tool can read them.

> **Note:** An unpublished site, a private site, or a site with search engines turned off, answers `robots.txt` with `Disallow: /` so crawlers stay out.

Watch how crawlers use these files in [Analytics](/guides/analytics).