Robots Control

Set global robots defaults, add extra crawler rules, and apply post-level indexing overrides only where needed.

Overview

Robots Control manages both the default robots meta output and your additional robots.txt rules.

Admin path: Airygen SEO -> Settings -> Robots Control

This module works best when you keep one safe sitewide default and only create exceptions for special pages.

Settings

The page includes Settings and Robots.txt.

Settings

Use this section for your normal sitewide robots behavior.

  • Enable default robots meta turns default sitewide robots output on or off.
  • Indexing lets you choose Use default, Index, or Noindex.
  • Following lets you choose Use default, Follow, or Nofollow.
  • Max image preview lets you choose Use default, Large (best quality), Standard, or None.
  • Max video preview lets you choose Use default, Unlimited (no cap), 30 seconds, 60 seconds, or Disable video preview.
  • Extra directives includes toggles such as noarchive, nosnippet, noimageindex, and notranslate.
  • Custom directives lets you add comma-separated directives that are not already covered by the built-in controls.
  • Preview robots meta value shows the exact robots meta string that will be output.

Robots.txt

Use this section when you need crawler rules beyond the meta tag.

  • Additional rules lets you add extra lines to the generated robots.txt.
  • Preview URL gives you a direct link to review the live robots.txt output.

Editor Panel

The editor panel lets you override the defaults on individual posts.

  • Preview shows the current robots result for the post.
  • Use defaults keeps the post on the sitewide rules.
  • Use custom data unlocks post-level directives.
  • Indexing directive lets you choose Inherit default, Index, or Noindex.
  • Link following directive lets you choose Inherit default, Follow, or Nofollow.
  • Post-level controls also include media preview limits, extra directives, and a Custom directives field.

How to Use

  1. Open Airygen SEO -> Settings -> Robots Control.
  2. In Settings, turn on Enable default robots meta if you want Airygen SEO to output a sitewide robots meta tag.
  3. Choose your default Indexing and Following behavior, then set Max image preview and Max video preview only if your site needs those limits.
  4. Turn on any Extra directives that match your policy, such as noarchive or nosnippet, and add anything else to Custom directives if needed.
  5. Check Preview robots meta value to make sure the final robots string says exactly what you intended.
  6. In Robots.txt, add any Additional rules you need for crawler instructions, then open the Preview URL to confirm the live file.
  7. Save the module, and use the editor panel only on pages that need a different indexing rule from the global default.

Verify these items after saving:

  • The sitewide robots meta string matches your intended default.
  • robots.txt contains only the additional rules you really want public.
  • Most pages still use the global default instead of unnecessary overrides.
  • Special pages with post-level overrides use the exact directive you chose.

SEO Benefits

Clear robots rules help you control which pages are indexed, how previews appear in search, and how crawlers should interpret special areas of the site. Good defaults also reduce accidental indexing mistakes.

User Cases

Sites with low-value utility pages

Before: Search engines may index pages that are not useful as landing pages, such as thin utility or filtered content.

After: The site owner keeps a clean default policy and uses page-level overrides only where low-value pages need Noindex.

Sites with special crawler instructions

Before: The team needs extra crawler rules, but they are scattered across manual edits and hard to review.

After: Robots Control keeps both robots meta defaults and robots.txt additions in one place, which is easier for non-developers to manage.

FAQ

Should most pages use the default robots settings?

Yes. In most cases, a single clean default is safer than many post-level exceptions.

What is the difference between robots meta and robots.txt?

Robots meta controls page-level indexing and preview behavior. robots.txt provides crawler instructions at the file level.

When should I use Custom directives?

Use them only when the built-in controls do not cover the directive you need. Most users can leave this field empty.