Optimize for search and AI answers

People find documentation through search engines, through AI assistants that answer questions, and through links others share. Docup builds for all three by default. This guide lists what is automatic and what you can tune.

What every site gets automatically#

Feature	What it does
Page titles and descriptions	Each page has a title and meta description. They default to the page title with the site name, and the page summary.
Canonical links	Each page declares its own address as canonical, and points to its Markdown copy.
Structured data	JSON-LD describes the page: a breadcrumb trail everywhere, a WebSite entry on the home page and a TechArticle with headline, description and modified date on other pages.
Open Graph and Twitter cards	Pages carry og: and twitter: tags with a 1200 by 630 image.
Sitemap	/sitemap.xml lists every indexable page with its last modified time.
robots.txt	Allows crawlers, points to the sitemap and optionally turns away AI crawlers.
llms.txt	An index of your docs for AI tools, plus llms-full.txt with every page in one file.
Markdown pages	Every page is also served as clean Markdown at <page>.md.
Fast, server-rendered pages	Pages arrive as complete HTML, and finished pages are cached. See Performance.
Generated social images	Every page gets its own card with its title and summary.

Tune titles and descriptions#

For the whole site#

Open Search and sharing and write a Site title and Description. They are used for the home page and as the fallback for the site. Keep the title under 70 characters, and describe what your product does and who the docs are for in 120 to 160 characters. The settings page previews the Google listing.

For one page#

Each page has its own panel next to the editor:

• Search title: replaces "Page title | Site name" completely. Include the keyword people search for, and the product name. Aim for 50 to 60 characters; the limit is 70.
• Search description: the snippet shown under the title. State what the reader will learn or be able to do, in 120 to 160 characters. The limit is 300. An empty field defaults to the page summary.
• Hide from search: adds noindex and removes the page from the sitemap and llms.txt.

Write titles that match the question: "Connect GitHub to Docup" beats "GitHub". Use one idea per page so each page can win one query.

Help AI assistants cite you#

Assistants such as ChatGPT, Claude and Perplexity look for pages that answer a question directly. To be easy to quote:

• Lead with the answer. The first paragraph of a page should say what the page covers in plain words.
• Use question-shaped headings where it fits: "How are updates triggered?".
• Keep sections self-contained. Ask AI and many crawlers read pages in sections, split at headings.
• Use tables and lists for facts such as limits, options and defaults.
• Keep llms.txt on so tools can read the whole site in one request.
• Allow AI crawlers in Search and sharing if you want to be quoted. You can see which bots visit in Analytics.

Social sharing#

By default each page generates its own card with the page title and summary, which makes shared links look specific. If you prefer one image for the whole site, switch the Social image option to My own image and upload a 1200 by 630 PNG, JPG or WebP up to 2 MB.

Control crawlers#

Goal	Setting
Keep the site out of search entirely	Turn Search engines off. All crawlers are blocked and pages get noindex.
Appear in search but not in AI training or answers	Keep Search engines on and turn AI crawlers off.
Disable files generated for AI assistants	Turn llms.txt off to disable both /llms.txt and /llms-full.txt.
Hide one page	Use Hide from search on that page.
Add a custom rule	Use robots.txt extra lines, up to 1,000 characters.

Known AI crawlers are asked to stay away with Disallow: /. Well-behaved crawlers follow it; robots.txt is a request, not enforcement. Docup requests the following bots to stay away when AI crawlers are turned off: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Google-Extended, CCBot, Applebot-Extended, and Bytespider.

Check your work#

1. Open /sitemap.xml, /robots.txt and /llms.txt on your site.
2. View the source of a page and look for the title, description, canonical link and JSON-LD.
3. Paste a page link into a social preview tool or a chat app and look at the card.
4. Submit the sitemap in Google Search Console.
5. Return to Analytics after a few days and watch crawler visits and 404s.

Fix every address listed under Not found (404); each is a link someone followed that led nowhere.