Detect Search Crawlers via JavaScript

bots, javascript, web-crawler

Solution

This is the regex the ruby UA `agent_orange` library uses to test if a `userAgent` looks to be a bot. You can narrow it down for specific bots by referencing the bot userAgent list here:

/bot|crawler|spider|crawling/i

For example you have some object, `util.browser`, you can store what type of device a user is on:

util.browser = {
   bot: /bot|googlebot|crawler|spider|robot|crawling/i.test(navigator.userAgent),
   mobile: ...,
   desktop: ...
}

Problem

I am wondering how would I go abouts in detecting search crawlers? The reason I ask is because I want to suppress certain JavaScript calls if the user agent is a bot. I have found an example of how to to detect a certain browser, but am unable to find examples of how to detect a search crawler: `/MSIE (\d+\.\d+);/.test(navigator.userAgent); //test for MSIE x.x` Example of search crawlers I want to block: ``` Google Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) Googlebot/2.1 (+http://www.googlebot.com/bot.html) Googlebot/2.1 (+http://www.google.com/bot.html) Baidu Baiduspider+(+http://www.baidu.com/search/spider_jp.html) Baiduspider+(+http://www.baidu.com/search/spider.htm) BaiDuSpider ```

Original source

Related problems