What is a robots.txt file?

A robots.txt is a text file that communicates to the search engine spiders (also known as robots) which pages on your website can be crawled and which cannot. Read more information about robot.txt here. 

A default robots.txt is automatically included with every Cratejoy theme and blocks customer information and checkout pages from being crawled. It also tells the search engine spiders where to find your domain’s sitemap. 

Can I edit the robots.txt file on my Cratejoy store?

Absolutely! There is a default robots.txt file for all stores, but it is actually hidden until you override it. To do this, you'll have to create a new file using the code editor, name it robots.txt, and save it under the html files. 

Steps to creating your own robots.txt:

  1. Go to my.cratejoy.com
  2. Click on the Design tab
  3. Click on the Code button next to your published theme
  4. Click on Add new in the outermost directory 
  5. Name the new file robots.txt
  6. Click Save
  7. You can either write directly onto the new file or write it the robots.txt ahead of time in plain text and paste it in when you are ready.
  8. Click Save in the upper right hand corner

What would an example robots.txt look like?

User-agent: *
Disallow: /checkout/
Disallow: /*preview_theme_id*

This would block all search engines ("*" is everyone) from indexing your checkout page and unpublished themes. Note: it's a good idea to check your current robots.txt (yourdomain.com/robots.txt) and copy that as a starting point.

I host my site on a different platform - how does this impact me?

If you host your site on another platform and use Cratejoy for checkout purposes only, you'll want to prevent the unused pages on the Cratejoy domain from being indexed (aka block the pages from showing up in search results). Blocking pages from being indexed will prevent your current and future customers from finding unused pages on the Cratejoy domain when using a search engine such as Google, Yahoo, Bing, etc. to find you online. This will prevent user confusion and adheres to SEO (search engine optimization) best practices.

Useful tools to help you write and test your robots.txt!