We Built Your Website a Second Website — Just for the Robots

A little behind-the-scenes from the workbench — the part where we admit we built your site a twin.

↓ Skip to the long version / results

By now you’ve met the bot zoo (Part 1), watched the bots run up the bill (Part 2), and learned why a “page not found” can be the busiest page you own (Part 3). So here’s the natural next question — the one a client actually asked us:

“If the robots are such a problem, why don’t you just send them somewhere else?”

Funny thing. That’s more or less exactly what we did. We built your website a second website — an identical copy, on its own machine — and we point the robots at it. It sounds a little mad out loud. Here’s why it’s the calm, sensible answer.

First rule: don’t hurt the SEO

Before any clever architecture, we had a hard constraint: real estate SEO is our bread and butter. The whole reason your listings show up in search is that crawlers like Googlebot can reach every page, read it cleanly, and index it. So you can’t just block bots — block the wrong one and you’ve quietly vanished from search results, which for a real estate site is roughly the worst thing that can happen.

So the goal wasn’t “keep bots out.” It was subtler: let the good ones in, send the bad ones away, and make sure neither group ever slows down a human looking at a kitchen photo. That constraint shaped everything that follows.

Same food, different seating (and why it isn’t cloaking)

Here’s the move. When a request arrives, our system classifies it in microseconds — person, or machine? People stay on your main server, tuned for the full experience: listings, search, maps, the works. Machines get routed to a separate, dedicated bot server.

The crucial detail: the bot server serves the same content — same pages, same listings, same structured markup a search engine wants to read. Nothing is hidden, nothing is changed. The only difference is which physical machine answers the door.

Now, the moment you describe “we treat crawlers differently,” a sharp reader thinks: isn’t that cloaking? Good instinct. Let’s be precise, because this matters.

Cloaking means showing search engines different content than you show people — feeding the crawler a keyword-stuffed page while humans see something else. Search engines penalize it, because it’s a lie told to the index.

What we do is not that. The content is identical. Picture a diner with a smoking and a non-smoking section: same kitchen, same menu, same food — you just get seated at a different table. Nobody’s being deceived about what’s for dinner; they’re seated where it makes sense.

That’s the whole accuracy case: identical content, different server is a seating arrangement, not a different menu. Search engines have no problem with where a page is served from; they care whether the page is honest. Ours is.

Something always has to give

Site speed is itself a ranking factor, so the bot copy can’t be a stale, half-broken mirror — it has to stay fast and current too. Fine. We keep it in sync.

But here’s the physics. When thousands of machines hit your site at once — and they do; one crawler can request thousands of listing URLs in rapid succession — something on the system slows down. You can add servers, add memory, add cleverness (we did all three), and the bots still come. At some point during a real spike, capacity gets stretched and a piece of the machine has to wait.

As a certain starship engineer put it: “I cannot change the laws of physics.” Neither can we. So instead we made a choice about who waits. We arranged things so that when a spike stretches capacity, it’s the bot server that slows down — never the one your buyers are on. A crawler waiting an extra beat for a listing page is a non-event; the page still gets indexed. A buyer waiting an extra beat is a buyer who clicks away.

That’s the quiet genius of the split. Pain is going to land somewhere during a surge. We pointed it at the visitors who don’t mind.

The happy accident: the spam mostly vanished

We built the bot server to protect performance. We got something else for free.

Think about what bots actually do. The good ones read pages and leave. The bad ones hammer the login page, submit junk through your contact forms, probe for vulnerabilities, and scrape your listings at machine speed — none of which a real buyer does. For years the painful part of fighting that noise was that it was mixed in with real traffic: tune your defenses too strict and you block a real agent, too loose and the spammers run wild. With humans and bots sharing one front door, every rule was a compromise.

Separating them dissolved that compromise. Once obviously-bad bots are turned away — and turned away fast, in essentially 0 milliseconds, before any of your site’s heavy machinery even wakes up — the noise that used to wash over your forms and login page mostly just… stops arriving on the human side. There’s no longer a crowd of bad actors standing in the same line as your buyers.

The result owners notice: fewer junk leads to wade through, fewer “did someone try to break in?” alerts, and the freedom to pay attention to the handful of real inquiries that come in. When the noise drops, the signal gets louder. That’s the whole series in one sentence.

We’ll be honest about the numbers: the exact before-and-after on spam and login attempts is something we want to pull from real data rather than eyeball, so we’re not going to put a hard number on it yet. The mechanism, though, is solid and running in production today.

The long version (for the curious)

If you like the gears, here’s how the split works under the hood — at a high level, since the real edge decisions are Part 5’s job.

Everything starts at our edge. Before a request reaches a server, it passes through our network edge, where the obviously-bad actors get stopped cold. (How that bouncer makes its calls is where we’re headed next.)

Then the sort. Requests that make it through get classified by our main server in microseconds — cheap lookups against the visitor’s stated identity, before any application code runs. Three outcomes:

• Looks like a person → stays on the main web server with all 174 of the client sites we host, for the full experience.
• Recognized crawler (whitelisted) → handed to the bot server, which serves the same content, optimized for clean indexing. We maintain 116 whitelisted crawler patterns across seven categories — Google’s family, the AI search bots, Bing/Microsoft, social link-preview bots, uptime monitors, the other search engines, and platform/embed tools — so the legitimate machines we want are served, not blocked.
• Unrecognized bot → the bot server turns it away with a clean refusal in 0 milliseconds — no PHP, no database, no cache lookup, nothing. Just “no.” A note on language: we say “recognized,” not “verified” — we match the patterns a crawler announces; we’re not doing a full identity check. An important honesty distinction.

A subtle but vital design rule: the main server keeps its own copy of the bot-handling logic. If the bot server ever goes down, the main server doesn’t forget how to handle bots — Google keeps crawling, buyers keep getting the full experience, bad bots keep getting turned away. The split is a performance optimization, not a single point of failure.

The rate limits. On top of the sort, there are speed limits — a machine behaving like a swarm gets throttled. Login pages and the old XML-RPC endpoint (a classic brute-force target) are kept on a short leash, while listing and search pages get generous bursts, because a single human map-search legitimately fires dozens of background requests per click. Get that limit wrong and you throttle a real buyer panning around a map.

A war story, because it earns the point. When we first tried to turn away a particularly aggressive SEO-audit crawler, it didn’t work — the bot kept getting served full pages. The cause was a three-bug pileup, and it’s the best illustration of why “just block the bots” is harder than it sounds:

• The bot server was reading the wrong field for the visitor’s real address, so every bot looked like it came from the same internal machine.
• So a rule meant to trust our own internal traffic was trusting every bot — the “turn it away” step never fired, and unrecognized bots were quietly served full pages.
• And the bot server’s recognized-crawler list had drifted out of sync — only 64 patterns instead of the full 116 — so once blocking worked, some legit AI and search crawlers got caught until we re-synced.

Fix all three and the proof was satisfying: the aggressive auditor got a clean “no,” and Googlebot sailed straight through. That’s the difference between “we block bots” and “we block the right bots without touching the ones that pay your bills.”

One last framing note, because we promised to make it carefully: none of this is cloaking. We never show a crawler a different page than a person would see — same page, different seat. Google’s own guidance defines the problem as showing different content to users versus search engines, not where a page is served from. (As always: that’s current guidance, and search policies do evolve.)

Sources

• Google Search Central — Spam policies: Cloaking (the official definition: showing different content to users and search engines). Current guidance; policies change.
• Google Search Central — Managing crawl budget for large sites (why high-volume crawling is a real operational concern for big sites). Current guidance; policies change.

If you’re hosting your own real estate site and the bots are winning, this is the kind of thing we live in so you don’t have to. We’re a small shop that actually builds this — happy to look at what’s hitting your site and talk through what would help. 👉 Come say hello, no pitch, just a conversation.