Search and sharing
Control search results, social images, crawlers, robots.txt and the machine-readable files Docup generates for AI tools.
Settings → Search & sharing decides how your docs appear in search engines, in link previews and to AI tools. Everything here takes effect on the live site when you save.
Search results#
Site title and Description are used for your home page and as the defaults for the site.
- The title defaults to your project name and can be up to 70 characters.
- The description defaults to the summary of the Overview page and can be up to 300 characters. 120 to 160 characters works best.
- A preview shows how the listing looks on Google.
Every other page uses its own title and summary. Change them per page in the editor; see The page editor. A page title appears as "Page title | Site title" unless you set a search title for that page.
Structured data and canonical links#
Every page includes structured data (JSON-LD) that tells search engines what the page is: a breadcrumb trail on every page, a WebSite entry on the home page and a TechArticle entry on other pages. Each page also declares its canonical address and points to its Markdown copy. You do not need to configure any of it.
Social image#
Choose what is shown when a link is shared in Slack, X, LinkedIn or a message.
- Generated: a card per page showing the page title and summary on a card themed to your site. This is the default.
- Custom: one image used for every page. PNG, JPG or WebP, 1200 by 630 pixels, up to 2 MB.
Save after uploading to switch to your image.
Crawlers and AI#
| Setting | When on | When off |
|---|---|---|
| Search engines | Google, Bing and others may list your docs. | Every crawler is blocked and every page gets noindex. |
| AI crawlers | AI bots may read your docs. | GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Google-Extended, CCBot, Applebot-Extended and Bytespider are asked to stay away. |
| llms.txt | Publishes llms.txt, llms-full.txt and a .md version of every page. |
Those files are not served. |
AI crawlers cannot be allowed while search engines are off, because everything is already blocked.
Extra robots.txt lines#
Add your own lines at the end of the generated robots.txt, up to 1,000 characters:
User-agent: SomeBot
Disallow: /internal/Generated files#
These files are built from your published pages and update when you publish. The settings page links to each of them.
| File | What it is |
|---|---|
/sitemap.xml |
Every indexable page with its last modified time. Pages hidden from search are left out. |
/robots.txt |
Crawl rules, a link to the sitemap and your extra lines. |
/llms.txt |
A title, a summary and a link to every page grouped by section, following the llms.txt convention. |
/llms-full.txt |
All pages in one Markdown file, for tools that read a site in a single request. |
/<page>.md |
One page as Markdown. For example /guides/editor.md, and /index.md for the overview. |
In the Markdown files, callouts become plain block quotes such as > **Note:** ... so any tool can read them.
An unpublished site, a private site, or a site with search engines turned off, answers robots.txt with Disallow: / so crawlers stay out.
Watch how crawlers use these files in Analytics.