CIM
Competitor Intelligence Monitor

CIM Bot

Information about the Competitor Intelligence Monitor web crawler.

User-Agent

CompetitorIntelMonitorBot/1.0 (+https://competitorintelligencemonitor.com/bot)

What does this bot do?

CIM Bot reads publicly available web pages on behalf of Competitor Intelligence Monitor users who have configured your site as a competitor source, and on behalf of users who check their own site. It reads:

  • pricing pages, changelogs, job boards and news about a competitor;
  • any public page a user chooses to watch for changes to its main text;
  • the public technical tags of pages, such as titles, canonical links and structured data;
  • your robots.txt file, and for a site check your llms.txt file, your sitemap and up to 25 of your public pages;
  • the page a suggested competitor came from, to check that the quote taken from it is really there.

App-store reviews are read from the app store's public feed rather than from your site. To measure a brand's presence outside its own website, CIM also reads public pages of Wikipedia, Wikidata, the web archive and news sources. Ad creative is not collected.

Official records and app listings

For a competitor's profile, CIM Bot also asks three public services for official facts, each through its documented public interface. None of them is searched or crawled beyond the addresses below.

  • SEC EDGAR (www.sec.gov and data.sec.gov): the list of company names and numbers, and the recent filings of a company a user has confirmed, about once a day. As the SEC asks of automated tools, every request declares who is asking: its User-Agent starts with Competitor Intelligence Monitor and a contact address the SEC can write to, followed by the User-Agent above. Without a contact address set, the SEC is not asked at all.
  • USAspending (api.usaspending.gov): U.S. federal contract awards. A company is looked up by its exact name, and its awards are read only after a user confirms it is the right company, every 3 to 7 days.
  • Apple's App Store lookup (itunes.apple.com/lookup): the version, price, rating and release notes of an app, asked by the app's number for apps linked from a competitor's own site or chosen by a user, at most 15 requests a minute. The App Store is never searched.

Compliance

  • We respect robots.txt — if you disallow our bot, we will not crawl your site.
  • We crawl at a polite rate. Scheduled checks wait at least 30 seconds between requests to the same site. A site check asks a site for at most one page a second, and runs between 10:00 PM and 6:00 AM Central time.
  • We only access publicly available pages — no login-required content.
  • For each page it checks, CIM stores a saved copy of what it read (the extracted text and, for a page shown in a browser, a screenshot) so that changes can be compared. A page that has not changed is not saved again.
  • We use official APIs and licensed data providers wherever available.

How to block our bot

Add the following to your robots.txt:

User-agent: CompetitorIntelMonitorBot
Disallow: /

Contact

Questions or concerns about our crawler?

CIM Bot — Competitor Intelligence Monitor