About XavierSeekBot

XavierSeekBot is the official crawler for the XavierSeek search engine, part of the Xavier Media network.

Purpose

XavierSeekBot crawls selected websites to discover and index content for our search tools. The bot focuses on:

• Fetching metadata, page titles, and descriptions
• Collecting images (for previews and visual search features)
• Supporting XavierSeek’s public search index and utilities

Technical Details

User-Agent string:

XavierSeekBot/1.0 (+https://www.xavierseek.com/bot.html)

XavierSeekBot respects robots.txt directives and Crawl-delay where provided.

Robots.txt Examples

You can control XavierSeekBot using standard robots.txt rules:

Allow everything:

User-agent: XavierSeekBot
Disallow:
  

Disallow everything:

User-agent: XavierSeekBot
Disallow: /
  

Allow only a specific folder:

User-agent: XavierSeekBot
Disallow: /
Allow: /public/
  

Set crawl-delay (10 seconds):

User-agent: XavierSeekBot
Crawl-delay: 10
  

Contact

If you have any questions, concerns, or would like to block our bot, please contact us:

• Website: contact us @ XMG.biz