What it is
Unfenced fetches web pages on behalf of AI assistants, and converts them to text those assistants can read. Each request is made because a person asked their assistant a question that needed your page - it is not a bulk crawl, we do not build a search index, and we do not store copies of your pages for redistribution.
How to identify it
Requests that declare themselves carry this User-Agent:
Unfenced/1.0 (AI agent fetcher; +https://unfenced.ai/bot)
We do not publish a fixed IP range: traffic originates from the machine of whoever is running Unfenced, which for self-hosted installations is their own computer. The User-Agent is the identifier to match on.
How to block it
A robots.txt rule addressed to us is honoured:
User-agent: Unfenced Disallow: /
Allow works as you would expect, longest match wins, and
* and $ are supported - so
Disallow: / with Allow: /public/ means what you intend.
We read the file once an hour per site rather than once per request.
Where it applies. Unfenced does two different things, and
robots.txt is addressed to one of them. When a request announces itself
as Unfenced/1.0 - an automated agent, which is what the Robots Exclusion
Protocol is written for - your rules are obeyed and the request is not sent. When
Unfenced is instead standing in for a person who asked their assistant for one
specific page, it behaves as their browser does: your rules are read and recorded,
and the fetch proceeds. We would rather state that boundary than let you assume a
stricter one.
Blocking by User-Agent at your edge, CDN or WAF stops the declaring case:
if (req.headers["user-agent"] contains "Unfenced") -> 403
It does not stop the other one, and we would rather say so. A fetch made for a person who asked their assistant for one specific page presents that browser's own User-Agent - a current Chrome string - because it is a real browser on a real machine, doing what their browser would do. Nothing in that request carries the word "Unfenced", so a User-Agent rule will not see it, and you should not build a policy on the assumption that it will.
For that traffic the controls that work are the ones you already use for browsers you do not want: a sign-in wall, a challenge page, or a rate limit on the address it arrives from. A challenge works because we do not solve CAPTCHAs - we wait a few seconds for a challenge script to clear itself, as a browser would, and give up when it does not. And the fastest control of all is to write to us: see below. We can stop the traffic at the account responsible, which no edge rule can do.
Answering 429, or 503 with a
Retry-After, needs no configuration and slows us down. A refusal
puts that whole host into an automatic back-off, so we wait rather than retrying
harder, and repeated refusals lengthen the wait - 30 seconds, then two minutes, then
ten, then an hour, up to six. One exception, stated because you may be watching for
it: a single refusal from a host that has just been serving us is recorded and not
acted on, since plenty of sites serve most paths and refuse a few, and that is a
website rather than a wall. The second in a row starts the back-off. If you send
Retry-After we wait at least that long from the first one, whichever of
the two forms you send it in, up to a ceiling of six hours. Sending a plain
503 with no header is read as your server having a bad afternoon rather
than as a refusal, and buys no back-off at all: a single outage should not cost a
caller your whole site for six hours, so if you mean "slow down" rather than "we are
broken", name the wait.
What it does not do
- It does not attempt to defeat a CAPTCHA.
- It does not use credentials it was not given, or sign in to anything on its own.
- It does not follow links to crawl your site broadly - it fetches the page it was asked for.
- It does not impersonate another company's crawler, or claim to be a search engine.
-
It slows down for hosts that push back, rather than retrying harder - and it waits as
long as a
Retry-Afterasks for.
If it is causing you a problem
Please tell us rather than only blocking us - a pattern that hurts one site usually hurts others, and we would rather fix it. Write to abuse@unfenced.ai with the date, a sample path, and roughly how much traffic you saw, and we will trace it to the account responsible.
For a copyright notice, a law-enforcement request, or anything else a lawyer should see: legal@unfenced.ai. The full procedure, and what happens after a report, is on Abuse reporting and takedown.