How Thousands Rogue Bots Accidentally Forced Me to Rewrite My Frontend Search Engine

Hi, everybody. 👋

This stems from my curiosity over my statistics on Cloudflare.

As I've observed from the dashboard — specifically on the Analytics ➡️ HTTP Traffic — the listed countries wildly varied each day. And also, very odd number of requests. Thus, I pulled the data from Cloudflare's GraphQL API — available for Free Tier, capped at 10k per 24h — to analyse it with a user interface (UI).

I've been observing outside Cloudflare dashboard since March 2026. Built the UI from scratch.

But in these few days, there's a more peculiar pattern. Well, not "more". "Peculiar" is indeed already with its own uniqueness — so perhaps, "different peculiarity".

Well. Turned out, those were mostly rogue bots. I'd noticed the bombardments were more active around 1-2 weeks ago. On my domain and subdomains. Those php, .env, credentials, and so forth scanning eggplants.

You know, akin to those that just plonked their hindquarters to [path] which has been specifically stated in robots.txt that YOU SHALL NOT PLONK YOUR THING ON [PATH] — and they plonked themselves conveniently on [path] nonetheless.

I have my custom security rules on Cloudflare. Thus, most of them were either blocked or given managed challenge. Cloudflare logs everything though, either blocked or allowed requests. I was inclined to dissect them, hence the API bit above.

---

The Search Feature

And therefore, I had massive data from Cloudflare, flung that to HTML.

Chrome said — Wow mate, that will take at least 700 MB of RAM to render. Could you chunk that? — And I said, with my code — Er. No. Just render it, please. — And Chrome rendered it, with occasional freezes and blanks, very occasionally.

I'd constructed a frontend search / filtering feature for the datasets — search (inclusion) and exclusion logic. It worked fine weeks ago because the data was considerably small back then.

But with this massive data, there was an oddity.

There was count discrepancy. I thought — Hm. — Specifically happened when I had more than 1K elements to rummage through.

I first thought it was from the loop method being implemented.

I'd prefer for...of rather than forEach.

🤔 No, that wasn't it.

After some more debugging...

💥

The main issue was at the dynamic regex creation in massive loops!

It took me about an hour to identify the daftness centroid.

By goblins everywhere.

But without me being bombarded by bots, I wouldn't know the caveats of regex in loops! Especially when searching through large datasets in the frontend, the DOM trees with thousands of elements.

My summary note:

Using dynamic regular expressions inside iterative loops introduces significant performance bottlenecks and functional bugs.

I had to re-engineer my entire JavaScript filtering architecture, battle forced browser reflows, optimise thread frame rates, and rewrite regex logic — all because thousands of automated scripts decided to throw a party on my endpoints. Proper poetic justice.

Rather comical, really. Because in this LLM era, most humans don't visit sites. Unless the sites are LLM interfaces. Or Google Search. Or Reddit. Just not like the old days. Very similar to industrial revolution, the shift!

That ought to be a blues song. Or reggae. "No human, no cry." Or, "hand production methods to machines." 🎵

Very similar, that. Let's do carry on.

One example of 24-hour statistics:

I list the rogue IP addresses based on my custom filter:

Summary in groups:

From that example above, I have — at least — 16k DOM elements.

Side note, they were orchestrated from different cloud services. Rotating user agent string, path request, and IP address (obviously — country). And that was the origin of that peculiar country-request counts on my HTTP Traffic dashboard. The rotation they'd done.

Anyway, I'd experienced massive DDoS in my line of duty back then. And that made the physical servers hung. A week without proper sleep to reconfigure things is indeed quite an experience.

Back to my current bot specimen here. A few of them even masquerading as legitimate bots — presumably comandeered by a tactical gremlin. One specimen:

⬆️ Googlebot coming from HostPapa.

Not a chance, mate.

Not. A. Chance.

That's like saying — I'm a toaster. — Well no, you are not. For now.

Moving on. Let's proceed to the frontend part, the regex-employing search.

---

Frontend Bit

Here's the list of drawbacks from my experience.

The regex part:

• When constructing a regular expression with the global flag (/g), JavaScript turns the object into a stateful instance. Executing .test() repeatedly updates the internal lastIndex pointer.

  • Issue:

    Testing subsequent strings without resetting lastIndex = 0 causes .test() to start searching mid-string, yielding false negatives on elements that actually contain the target term.

  • Symptom:

    Mismatches between inclusion and exclusion counts on identical search strings.

  • Resolution:

    Manually set it to 0 using regex.lastIndex = 0;.

And getting the string part:

• Pairing heavy loop computations with .innerText reads forces the browser's rendering engine to calculate layout geometry for every element.

  • Issue:

    .innerText is computationally expensive because it considers CSS styles and element visibility.

  • Symptom:

    Severe thread stuttering, dropped frames, and potential browser freeze on larger datasets.

  • Resolution:

    .textContent reads raw strings directly from memory in C++ bindings, running absolutely FASTER without triggering forced reflows.

So because of that, I needed to rewrite the methods.

Roughly as shown below.

From:

const searchString = searchInput.value.trim(); const escapedSearchString = searchString !== "" ? searchString .replace(/\./g, "\\.") .replace(/\(/g, "\\(") .replace(/\)/g, "\\)") : null; const searchRegex = escapedSearchString ? new RegExp(escapedSearchString, "gi") // ← Regex bit. : null; // —————————————————————————————————— // Inside a loop. const text = elm.innerText // ← innerText expensive method. .replace(/\s{2,}/g, " ") .trim() const matchesSearch = searchRegex?.test(text); // ← Substring search / match. if (matchesSearch) { // ... DOM manipulation. }

To:

const searchString = searchInput.value.trim().toLowerCase(); // —————————————————————————————————— // Inside a loop. const text = elm.textContent // ← innerText is swapped to this. .replace(/\s{2,}/g, " ") .trim() .toLowerCase(); const matchesSearch = searchString ? text.includes(searchString) // ← Regex and test() are swapped to this. : true; if (matchesSearch) { // ... DOM manipulation. }

The advantages:

• String.prototype.includes is written in native C++. It bypasses regex object creation, state management, and memory overhead completely.
• Zero regex state bugs. No lastIndex traps to track down or escape sequences to worry about.
• Predictable as intended. It checks raw character matches symmetrically for both positive searches and exclusions.

Also, I avoid instantiating new RegExp inside a loop. If possible, define it once outside the loop.

For substring search method — well — because this is not a complex pattern matching like filtering email pattern and so on, so employing includes preceded by toLowerCase is recommended — as I've shown above.

And for processing, I employ requestAnimationFrame with batches within a setTimeout block. Well. That. If not, when I typed something, the UI would freeze for some seconds. The search trigger is oninput. Mm.

I imagine that freeze is like when we ask a bloke for a direction, he would just stand there for some seconds. — Hi, mate. Do you know where Tesco is? — Bloke just stands there for 10 seconds. His brain hits 100% usage, then answers — Don't know, mate.

That bit should be in a film.

---

Virtualisation or Windowing Technique

Well, my method above by dumping thousands of elements to HTML without proper chunking isn't recommended. That is simply because it isn't for public consumption — or real-world users — only for my own research and analysis.

Say, we have 100k elements.

Instead of putting 100k elements in the HTML tree, we should render what fits on the screen — for instance, 20 visible rows + a few off-screen buffer rows.

As the user scrolls, we dynamically swap the text content of those 20-30 DOM elements. So the DOM size will always be 20-30 elements, very fast and efficient.

And for the search feature, we do that from the raw JS array of 100k plain objects in memory. It will be absolutely more efficient than reading string from the DOM elements.

And surely, we can also combine that with a Web Worker, so the search runs on a background CPU core.

---

Right, then. That'd be all. Thanks for visiting. 👋