Robots.txt Generator

How Robots.txt Generator works

Create a valid robots.txt with allow and disallow rules, crawl delay and a sitemap reference. Presets for WordPress and staging. Everything happens inside this page: the tool reads your input, runs the calculation or conversion locally, and hands the result back to you. Nothing is transmitted, because there is no backend to transmit it to.

That design choice is deliberate. Most free tool sites upload your files to a server, process them there, and quietly keep a copy. Toolcin cannot do that — the site is a set of static files. If your connection dropped the moment the page finished loading, every tool would still work.

Presets and what they mean

Allow everything is the normal production setting. Block everything is for a staging site — it stops crawlers touching the domain at all, which is what you want before launch. WordPress defaults add the usual admin, cart and search paths.

Custom mode lets you list paths to block, one per line, plus the user-agent they apply to.

The rule that breaks sites

robots.txt controls crawling, not indexing, and a Disallow prevents a crawler from even fetching the page — which means it cannot see a noindex tag. Blocking a page to keep it out of the index therefore does the opposite: the URL can still appear with no description.

To keep a page out of search results, do not block it. Serve a noindex header or meta tag and let the crawler read it.

Things worth adding

The Sitemap line should be absolute. Do not block /assets/ — search engines need CSS and JavaScript to render a page, and blocking those is the most common self-inflicted rendering failure.

A long list of disallowed paths that no longer exist is harmless but misleading. Keep the file short enough that someone can read it and tell what it does.

Good to know

• Works offline once loaded, including on flights and in locked-down networks.
• No file size limit beyond your device's own memory.
• Free to use, with no account and no usage counter.
• Results are produced locally, so they are not logged or shared with anyone.

Questions