OplioBot, our web crawler
How to stop OplioBot, how to recognise it, and what it reads on your website and why.
In short
- It reads a few public pages, at most 1 a second.
- It obeys your robots.txt.
- To stop it, add two lines to your robots.txt, or email us.
How to stop it
Add these lines to your robots.txt. OplioBot never uses a copy of your robots.txt older than 24 hours, so it stops within 24 hours:
User-agent: OplioBot
Disallow: /Or email us your domain. We will stop visiting it, delete what we kept from it, and no one using Oplio will be able to contact it through us:
How to recognise it
Every request it makes carries this user agent:
OplioBot/1.0 (+https://oplio.app/bot)Why it visits
Oplio helps freelancers and small agencies find small businesses that may need their services. When one of them searches for, say, dentists near Denver, OplioBot visits those businesses’ websites to read the contact details they publish there. It also visits a business’s website when one of them asks to research it, whether that website came from a search, or they typed it in or listed it in a file.
What it reads
- Your homepage, and at most 5 other pages, such as Contact, About or Team.
- On a site that we know several businesses share, such as a mall’s or a directory’s, only the homepage. When a business’s website is a page on another site, such as its listing in a directory or on a booking site, only that page, and on it only an address that carries the business’s name, or the only free email address there.
- Only public pages: it never signs in, fills in a form, pays or solves a CAPTCHA.
- It never keeps a team member’s own phone number or social profile from a team page, or the address of whoever made your website.
Details
- Your robots.txt, and, only when needed: the start of your sitemap where your platform keeps it (such as /sitemap.xml, following one redirect on your site) when your robots.txt names none; your /favicon.ico when your homepage declares no icon; and, when your homepage opens over plain HTTP, your HTTPS address once, to see whether it works.
- If your homepage loads a Google Tag Manager container, that container’s file, from Google’s servers, to see which tags your site uses. It is read, never run.
- If your site’s security certificate has expired but is still your site’s and signed by an authority browsers trust, OplioBot reads your pages once more without verifying it, so the website check can say so. If its chain is incomplete, OplioBot completes it as browsers do, fetching the issuer’s certificate from the address your certificate gives, on your certificate authority’s server and at most once a day, and reads your pages only if the completed chain checks out. If the certificate is for another site, or no one vouches for it, OplioBot reads nothing your site serves.
- If your homepage builds itself in the browser and shows nothing without it, OplioBot may read your pages once more, within the hour, in a headless browser. It runs their scripts and asks only for what they need from your own site and a few public code libraries (scripts, styles and the data they load), each after robots.txt and at the same pace. It never asks for images, fonts or frames, or for anything on other sites, and it never posts or submits anything.
- It keeps facts, not your pages: the business email addresses, phone numbers and social profile links your pages publish; the names and roles of the people your pages name as your team (owners, managers and staff); the tools your site runs on (such as WordPress, an analytics tag or a booking widget), from your pages, the headers your homepage answers with and the names of the cookies it sets (never their values), each with the short snippet that shows it; and what a website check needs (such as whether your homepage works on phones or names a phone number). Each comes with a link to the page it was on. Never your pages’ text, beyond those snippets of at most 80 characters, which never hold a person’s details.
- When your team or About page names its people in sentences rather than as a list, an AI service we use reads that page’s text, without its email addresses and phone numbers, to find their names and roles. We keep only those, each with a link to the page, never the text.
- If a page it reads says you do not want unsolicited messages, OplioBot keeps none of your contact details: only that you ask not to be contacted, and the page that says so.
- When someone using Oplio asks for a speed check of your homepage, Google PageSpeed Insights loads it once, as Google, not as OplioBot. Never on a site that asked us to stop.
How it behaves
- It reads your robots.txt first, and follows its rules for OplioBot, or the rules for all crawlers (
User-agent: *) if none name it. - If your robots.txt sets a Crawl-delay, it waits that long between pages, up to 30 seconds. If it asks for longer, OplioBot reads nothing on your site.
- It asks your site for at most 1 page a second, and never more than 2 at a time.
- It reads a site again about once a month, every 30 days, and sooner when someone using Oplio refreshes a search that includes the business or researches it, though not for the same request within 24 hours. When it can, it first asks whether anything changed.