Add the requested inputs
Use the labelled fields in the workspace above. If example values are provided, try them first or replace them with your own.
Marketing & growth · Free online tool
Robots.txt builder
Draft a robots.txt group with allow, disallow, sitemap and optional crawl-delay directives. Turn a short list of crawler paths into a reviewable robots.txt starting point before publishing it at the site root.
3 guest runs per rolling 24 hours10 with a free accountPrivate inputsMobile-friendly
Robots.txt builder workspace
Runs locally in your browser
robots.txt controls crawling hints; it does not remove an already indexed page or protect private content.
robots.txt draft
Use it with confidence
How to use the robots.txt builder.
Calculate and review
Use the main action when needed. Each entered path is validated and written as a directive under one user-agent group, followed by optional delay and sitemap lines.
Try a worked example
Allow a public asset path, disallow an internal search path and add the full sitemap URL for a chosen user-agent.
Check the assumptions
Crawler support and rule precedence vary. Robots.txt is public, does not secure private content and does not reliably remove indexed URLs.
Helpful answers
Robots.txt builder FAQs.
Is the robots.txt builder free?
Yes. Guests receive 3 runs per rolling 24 hours, verified free accounts receive 10, and Premium subscribers receive unlimited normal use.
Does UtilHub upload or save my inputs?
No. This tool runs in your browser and UtilHub does not receive or store the values you enter. Your browser or device may still retain normal local history or form data.
How does this robots.txt builder work?
Each entered path is validated and written as a directive under one user-agent group, followed by optional delay and sitemap lines.
What should I check before using the result?
Crawler support and rule precedence vary. Robots.txt is public, does not secure private content and does not reliably remove indexed URLs.