How Thousands Rogue Bots Accidentally Forced Me to Rewrite My Front-End Search Engine

Hi, everybody. 👋

This stems from my curiosity over my statistics on Cloudflare.

As I've observed from the dashboard — specifically on the Analytics ➡️ HTTP Traffic — the listed countries wildly varied each day. And also, very odd number of requests. Thus, I pulled the data from Cloudflare's GraphQL API — available for Free Tier, capped at 10k per 24h — to analyse it with a user interface (UI).

I've been observing outside Cloudflare dashboard since March 2026. Built the UI from scratch.

But in these few days, there's a more peculiar pattern. Well, not "more". "Peculiar" is indeed already with its own uniqueness — so perhaps, "different peculiarity".

Well. Turned out, those were mostly rogue bots. I'd noticed the bombardments were more active since about 1-2 weeks ago. On my domain and subdomains. Those php, .env, credentials, and so forth scanning eggplants.

You know, akin to those that just plonked their hindquarters to [path] which has been specifically stated in robots.txt that YOU SHALL NOT PLONK YOUR THING ON [PATH] — and they plonked themselves conveniently on [path] nonetheless.

I have my custom security rules on Cloudflare. Thus, most of them were either blocked or given managed challenge. Cloudflare logs everything though, either blocked or allowed requests. I was inclined to dissect them, hence the API bit above.

---

The Search Feature

And therefore, I had a massive amount of data from Cloudflare and flung that into the DOM.

Chrome said — Wow mate, that will take at least 700 MB of RAM to render. Could you chunk that? — And I said, with my code — Er. No. Just render it, please. — And Chrome rendered it, with occasional freezes and blanks, very occasionally.

I'd constructed a front-end search / filtering feature for the datasets — search (inclusion) and exclusion logic. It worked fine weeks ago because the data was considerably small back then, top ~300 kB. Quite different than ~3 MB.

And with this massive amount of data, I spotted an oddity in the search results.

There was count discrepancy. I thought — Hm. — Specifically happened when I had more than 1k elements to rummage through.

Well to be fair, it perhaps had already had the count-discrepancy issue with small data, but I simply didn't notice it back then because the output was "viewable". I focused on the string itself rather than the count.

Right. I first thought it was from the loop method being implemented.

I'd prefer for...of rather than forEach.

🤔 No, that wasn't it.

After some more debugging...

💥

The main issue was at the dynamic regex creation in massive loops!

It took me about an hour to identify the daftness centroid.

By goblins everywhere.

But without me being bombarded by bots, I wouldn't know the caveats of regex in loops! Especially when searching through large datasets in the front end, the DOM trees with thousands of elements.

My summary note:

Using dynamic regular expressions inside iterative loops introduces significant performance bottlenecks and functional bugs.

I had to re-engineer my entire JavaScript filtering architecture, battle forced browser reflows, optimise thread frame rates, and rewrite regex logic — all because thousands of automated scripts decided to throw a party on my endpoints. Proper poetic justice.

Rather comical, really. Because in this LLM era, most humans don't visit sites. Unless the sites are LLM interfaces. Or Google Search. Or YouTube. Or Reddit. Just not like the old days. Very similar to industrial revolution — the shift!

That ought to be a blues song. Or reggae. "No human, no cry." Or, "hand production methods to machines." 🎵

Very similar, that. Let's do carry on.

One example of 24-hour statistics:

From those numbers, I've built proper honeypots, haven't I. Such magnetism. Or attraction. Attraction is the power, act, or quality of drawing rogue bots toward my sites. Just like in real life.

A: You are very attractive, B!

B: Oh, thanks. (Cockroaches from nowhere jump to head.) AAAARGGH!

A: The proof! Very attractive.

B: Quite so. (Stands still.) Oh by golly, these beings emit foul stench! With a bit of lime and lemon though, mm mm.

I listed the rogue IP addresses based on my custom filter:

Indeed, not "thousands" from the listed IP addresses per day, specifically in this example. But those 842 IP addresses did 13,984 requests. Imagine eight hundred and forty-two strangers rattle our doorknob thirteen thousand nine hundred and eighty-four times from 00:00:00 to 23:59:59. One day is 24 hours = 24 × 60 = 1,440 minutes. So rattles per minute = 13,984 : 1,440 = 9.71111. Nearly ten rattles per minute. Thinking if we have any exposed pem.key files inside.

Summary in groups:

From that example above, using the maximum 10k API limit, I had — at least — well, plenty. About ~200k elements and roughly ~7% with event listeners. Tremendous JUNGLE, that. Absolutely mad setup. I'd counted it. Say, a table, it's not one element, the children? thead, tbody, tr, th, td, and so on? And then the div which consisted of another divs. Inside each div, there were code, mark, and span tags. Plus the buttons! And I made those particular mark and spans as function triggers, like proper buttons! Mad stuff.

Side note, generic method to count all the elements within an HTML document:

document.querySelectorAll("*").length;

Or to be more specific, counting all elements inside a particular parent element:

parentElement.querySelectorAll("*").length;

Right. That.

But Chrome, being composed as always — Aye, do carry on, old chap. Just another 200,000 nodes for the pile.

The initial construction above had no issue since it was constructed from a plain object and I didn't use regex g flag for .test(). And ~200k-element rendering was still within Chrome's timeout limit — not triggering the "wait or exit page" dialog. It was under six seconds to complete.

The issue appeared when I ran my front-end search engine — which travelled through the DOM with g flag for .test() method.

It was comical — I'd built the search engine by referencing the constructed DOM because back then, the data was considerably small. And with large data, I didn't change it to referencing the in-memory plain object, instead, well... still that. My functional fixedness was and is rather thick in this. Because refactoring? Brain said — Oh sure, but I'm quite sleepy now. — Oh, you bloody menace.

Side note about the automated eggplants, they were orchestrated from different cloud services. Rotating user agent string, path request, and IP address (obviously — country). And that was the origin of that peculiar country-request count on my HTTP Traffic dashboard. The rotation they'd done.

This sort of peculiarity:

⬆️ Ah.

On Google Analytics, we cannot dissect those eggplants. Because Google Analytics is client-side based.

Anyway, I'd experienced massive DDoS in my line of duty back then. And that made the physical servers hung. A week without proper sleep to reconfigure things is indeed quite an experience.

Back to my current bot specimen here. A few of them even masquerading as legitimate bots — presumably comandeered by a tactical gremlin. One specimen:

⬆️ Googlebot coming from HostPapa.

Not a chance, mate.

🤔 Hang on. Indeed, it's like Bingbot comes from GOOGLE-CLOUD-PLATFORM.

Not. A. Chance.

---

Rogue Bot Bit

This is the reconnaissance and initial access phase — "first wave" bots. Like burglars randomly rattle doorknobs, then mark them if they're unlocked. And there's the "second wave" — not necessarily the same burglars. The first-wave parties can sell the data of "unlocked doors" to the second-wave parties. The second-wave parties then will install backdoors and moving the furniture.

Most first-wave scanning bots don't actually search for domain names. They scan raw IP addresses. Scanning by IP range is the ultimate high-volume, low-effort move in cybersecurity.

Here is how these automated eggplants pull off IP range scanning in a nutshell:

1. Set up speed-demon tools.

   Using lightweight, high-speed scanners like Masscan or ZMap on a basic cloud server, a bot can blast out millions of packets a second.

2. The numbers game.

   Instead of hunting specific websites, they treat the IPv4 internet as a simple list of numbers — pinging every address sequentially from 0.0.0.0 to 255.255.255.255.

3. Spray a blind payload.

   The bot sends rapid-fire HTTP requests straight to open ports (usually port 80 or 443) on every IP, looking for specific vulnerable paths like /wp-login.php, /.env, or vendor setup scripts.

4. Log and move on.

   If an IP responds with a success code, the bot logs that IP address to a target list for "second wave" exploitation and instantly moves to the next number — scanning through the entire internet in under an hour for pennies.

Well. A glimpse of that. Business, mate. Rather shady, that. Proper shady.

Moving on. Let's proceed to the front-end part, the regex-employing search.

---

Front-End Bit

Here's the list of drawbacks from my experience. Plonked a massive number of elements into the HTML and tried to rummage through those.

The regex part:

• When constructing a regular expression with the g (global) flag, JavaScript turns the object into a stateful instance. Executing .test() repeatedly updates the internal lastIndex pointer.

  • Issue:

    Testing subsequent strings without resetting lastIndex = 0 causes .test() to start searching mid-string, yielding false negatives on elements that actually contain the target term.

  • Symptom:

    Mismatches between inclusion and exclusion counts on identical search strings.

  • Resolution:

    We can manually set it to 0 using regex.lastIndex = 0.

    But that's a bit daft, innit? Adds complexity to the code.

    So use .includes() rather than regex engine for simple search. We'll see how it works shortly.

And getting the string part:

• Pairing heavy loop computations with .innerText reads forces the browser's rendering engine to calculate layout geometry for every element.

  • Issue:

    .innerText is computationally expensive because it considers CSS styles and element visibility.

  • Symptom:

    Severe thread stuttering, dropped frames, and potential browser freeze on larger datasets.

  • Resolution:

    Well, don't use .innerText, then.

    Instead, .textContent.

    .textContent reads raw strings directly from memory in C++ bindings, running absolutely FASTER without triggering forced reflows.

So because of that, I needed to rewrite the methods.

Roughly as shown below.

From:

const searchString = searchInput.value.trim(); const escapedSearchString = searchString !== "" ? searchString .replace(/\./g, "\\.") .replace(/\(/g, "\\(") .replace(/\)/g, "\\)") : null; const searchRegex = escapedSearchString ? new RegExp(escapedSearchString, "gi") // ← Regex bit. : null; // —————————————————————————————————— // Inside a loop. const text = elm.innerText // ← innerText expensive method. .replace(/\s{2,}/g, " ") .trim() const matchesSearch = searchRegex?.test(text); // ← Substring search / match. if (matchesSearch) { // ... DOM manipulation. }

To:

const searchString = searchInput.value.trim().toLowerCase(); // —————————————————————————————————— // Inside a loop. const text = elm.textContent // ← innerText is swapped to this. .replace(/\s{2,}/g, " ") .trim() .toLowerCase(); const matchesSearch = searchString ? text.includes(searchString) // ← Regex and test() are swapped to this. : true; if (matchesSearch) { // ... DOM manipulation. }

The advantages:

• String.prototype.includes is written in native C++. It bypasses regex object creation, state management, and memory overhead completely.
• Zero regex state bugs. No lastIndex traps to track down or escape sequences to worry about.
• Predictable as intended. It checks raw character matches symmetrically for both positive searches and exclusions.

Also, I avoid doing .test() with g or y flag inside a loop.

y flag is the sticky flag (info on MDN).

Careful with g and y FLAGS! If possible, do the .test() outside the loop. But of course, if unavoidable, there's that manual reset regex.lastIndex = 0 we can put. Example:

const regex = /abc/g; const str = "abc abc"; console.log(regex.test(str)); // true (lastIndex is now 3) console.log(regex.test(str)); // true (lastIndex is now 7) console.log(regex.test(str)); // false (lastIndex reached end, // resets to 0 automatically, // BUT FAILING FIRST) // BUT if we want to test from the start again mid-way: regex.test(str); // true (lastIndex is now 3) regex.lastIndex = 0; // ← MANUAL RESET! console.log(regex.test(str)); // true (starts from 0 again!)

⬆️ Or simply omit the g (or y) flag from the regex. i, m, s, and u are completely safe. They don't touch lastIndex at all.

Therefore, for substring search method — well — because this is not a complex pattern matching like filtering email pattern and so on, so employing .includes() preceded by toLowerCase is very recommended — as I've shown above. We'll see the summary below.

For managing RAM usage, I have a button (function) to clear the DOM lavish jungle, remove all dynamic listeners (click, input) from it, clearTimeout, and cancelAnimationFrame so Chrome's Garbage Collector (GC) knows it has permission to throw them away. But it won't right away collect the garbage, somewhere around 1-3 minutes after. My top RAM usage was about 2.1 GB, just for 24h-span data. If I took data for more than 24 hours, say, 48 hours, well. Hilarious, for just that one Chrome tab. And when I executed the cleaner function, the RAM usage slowly dropped to ~600 MB, ~400 MB, and to its idle state, about ~290 MB, without refreshing the tab. Very slowly, but started around 1-3 minutes after the button click.

When I cleared the DOM — with that number of elements and more things to detach — Chrome finally lost its stiff upper lip and shouted at me:

Oh yes, I'd tried plenty of workarounds — queueMicrotask(), requestAnimationFrame(), to Web Worker. Well, three were plenty. I mean, the code for each one was bloody DIFFERENT! Right. Anyway, Chrome's shouting just moved to that different method, not the original click event anymore. Admitting defeat is easier anyhow.

And for overall processing, I do employ requestAnimationFrame with batches within a setTimeout block — hence clearTimeout and cancelAnimationFrame for clearance. Well. That. If not, when I typed something, the UI would freeze for some seconds. The search trigger is input event.

I imagine that freeze is like when we ask a bloke for a direction, he just stands there for some seconds. — Hi, mate. Do you know where Tesco is? — Bloke just stands there for 10 seconds. Brain hits 100% usage. There's a sound of laptop fans howling from nowhere. Then finally answers — Don't know, mate.

That bit should be in a film.

Or that "freeze" bit from Hollywood. Literally, the scoundrel freezes in mid-run. That should be a physics-defying spell. With 100% brain usage. Deducing which refrigerator series is the most affordable and good. Without a knowledge bank. There's a sound of laptop fans howling from nowhere. Too.

Because — Please remain stationary. — or — Kindly desist, sir! — is not quite fitting for a tech jargon. For instance:

The UI remains stationary.

Or:

The UI is simply kindly desisting.

git add . && git commit -m "Fixed bug where UI kindly desisted for 10 seconds during 400k element render"

---

Regex and Native String Search Summary

For simple string searches, where no complex regex patterns like \d+ or \s{2,} are needed:

• text.toLowerCase().includes(target.toLowerCase())

  Fastest. 🥇

• /target/i.test(text)

  Middle. 🥈

  No lastIndex issue here — if used in .test() — since we don't use g flag.

• /target/.test(text.toLowerCase())

  Slowest. 🥉

  Since .toLowerCase() allocates a brand-new string in memory. JavaScript scans the string twice. First, it iterates through every single character to build the lowercased copy. Then, it iterates through that copy a second time to execute .test().

So if we don't need wildcards or regex features:

lowercase + .includes()

— wins every day of the week.

---

Virtualisation or Windowing Technique

Well, my method above by dumping thousands of elements into the HTML without proper chunking isn't recommended. That is simply because it isn't for public consumption — or real-world users — only for my own research and analysis.

Say, we have 100k elements.

Instead of putting 100k elements in the HTML tree, we should render what fits on the screen — for instance, 20 visible rows + a few off-screen buffer rows.

As the user scrolls, we dynamically swap the text content of those 20-30 DOM elements. So the DOM size will always be 20-30 elements, very fast and efficient.

And for the search feature, we do that from the raw JS array of 100k plain objects in memory. It will be absolutely more efficient than reading string from the DOM elements.

And surely, we can also combine that with a Web Worker, so the search runs on a background CPU core.

---

About Consuming Cloudflare GraphQL API During Local Development

When building locally, Hot Module Replacement (HMR) and auto-reloading can fire off dozens of live API requests every time we save a file — easily triggering a soft ban from Cloudflare.

We can use mock / sample data during active UI development, and only hit the live CF GraphQL endpoint when we explicitly need to test real responses. Save our rate limits for when they actually count.

We can include the samples in our UI as options, like so:

⬆️ Radio inputs.

I have two versions as radio inputs, S (small) and L (large). By default, the in-application, small data is the picked option. If then I want to analyse the live data, I'll pick the Live option. Er, I have plenty of samples — it's simply because of reasons. But too many of those will utterly clutter the UI.

So my setup:

• Local development: I have those three radio inputs.

• Live application: only Live option — because the large sample data would burden the first load, using import.

  

Plus, I cache the live API response using node-cache for both environments.

To get a mock / sample data — well — we need to get them live first. Copy the response from browser's network tab ➡️ make a .js file for that ➡️ import it to our application. ✨

Or, if you're familiar with Postman / Bruno / Insomnia / so on, you know the drill. I am confidently redundant.

We don't want to get shouted at by CF.

CF: Hold on a second... high request frequency? Repeating pattern? Unnatural speed? WE'VE FOUND A ROGUE BOT, BOYS! IT'S THIS WATERMELON WITH PORT 3214 FROM A VPN! 📢

I am trying to ANALYSE the bots!

CF: Sounds like something a rogue bot would say, innit? Into the rate-limit bin you go, plonker!

Perfect irony, that.

---

Thanks for visiting. 👋