Greetings!
This post is motivated by my observation of the statistics of this blog: built-in Blogger stats, Cloudflare HTTP request logs, and my method utilising Firebase realtime database — for browser requests.
All of them showed discrepancy. Mm. Massive discrepancy.
But Cloudflare, being an HTTP request intermediary, actually captured interesting origins and questionable, phantom requests.
Phantom, as in not actually "landed" on the HTML — using common browser — activating browser script.
| 🤖 | : |
/wp-config.php, if tha doesn't mind. |
| 🖥️ | : |
403. |
| 🤖 | : |
/.well-known/ path, specifically the entry bearing a most arbitrary identifier? |
| 🖥️ | : |
403. |
Imagine if they used telnet to manually craft HTTP requests for elite-level shenanigans. They'd be — Oh, wow, HTTPS! No wonder all the requests wandered around to abyss. All this brain work and reading documentation for nothing!
I might suggest as such:
Oh!
Yes, that should do. But still, nope.
Aw...
⬆️ Such rogue melons.
Right. So.
It is actually flattering to have immortals rummaging through this blog. Immortal, until someone prances and unplugs the server's power chord.... and cord.
An esteemed manner and curious finesse is required to still the noble computing contraption.
Therefore, let's dig a bit about bots.
The Imagery of Tiny Robot Spider
Did that ever come to your mind whenever you heard bot, or crawler, or web?
The term didn't have "spider" in it directly, but in our mind:
web + crawler + bot (robot) = skittering tiny metallic spider
But indeed in early web discussions, the term spider was way more common than nowadays.
Like this weird IRC discussion in early Giiggglee™ engineers community:
| Eng※1 | : |
|
| Eng※2 | : |
|
| Eng※1 | : |
|
| Eng※2 | : |
|
| Eng※1 | : |
|
| Eng※3 | : |
|
| Eng※1 | : |
|
| Eng※3 | : |
|
| Eng※2 | : |
|
| Eng※3 | : |
|
⬆️ Eng※1 was referring to cobweb. Quite curious, this one.
A Process Instead of An Object
Object, like the actual spider, the arachnid, arthropods. Bot is actually a process.
Bot is NOT an object.
It's a metaphoric term related to web.
Web = network of nodes.
Node
Any entity within a system (web) that is connected to others through hyperlinks, network connections, or structured pathways.
Thus — bot, crawler, spider ⬇️
An automated process that navigates and interacts with nodes in the system (web).
The Anatomy of a Bot
Above is specifically a generalised crawler or scraper bot. The flow goes like this:
- Get new task. 📝
- Find target URL. 🔍
- Request page (headless browser). 📡
- JavaScript runs (if needed). ⚙️
- Collect response (HTML, data, etc.) 📦
- Store & process data. 🗄️
- Update bot database. 🔄
- Repeat. ♻️
- ...
- ...
- ...
- ...
- ...
- ...
- ...
- ...
- ...
- ...
-
Profit 💰 — supposedly.
With that being said above, in picture and ultra concise flow description, a bot is an automated process of connecting the dots in dotted network of dots.
For profit. Well, it's a part of web-based business model.
Headless Browser
It runs in the background and processes web pages without displaying them visually. It is commonly used for automated tasks like bot (scraping), testing, and performance monitoring. It allows scripts to interact with websites as a real user would — loading pages, clicking links, and executing JavaScript — but without opening a visible browser window.
It's the browser for the machine.
Bot Being Stealthy
The bot's request leaves marking on the target server because that's the protocol for communicating.
Bot's request cannot be stealthy and still expecting a response from the target.
Stealthiness serves the purpose of going unnoticed. Indeed, it is.
Thus, being absolutely stealthy from bot's perspective is like roaming in the void with no purpose.
For instance:
This "stealthiness" bit is akin to using telnet to connect to an HTTPS-only remote host using fictional SSL/TLS.
Or, staring at a bloke for two days without any word — wearing branches and leaves, expecting his response by thinking repeatedly — Respond, respond, respond.
Therefore, dubious bots commonly have camouflaged IP address, referrer, and — surely — the user-agent string.
But some just being lazy leaving user-agent string to be empty.
And in this skit below, the rogue-bot developer did a typo.
| 🤖 | : |
|
| 🖥️ | : |
|
| 🤖 | : |
|
| 🖥️ | : |
403 card. |
| 🤖 | : |
Mozilla/4.0 (Windows NT 11.0) AppleScrapkit/HarfardBot 2.0; Crawl reason: REASONS. YES. |
But indeed, there are legitimate bots or crawlers. They mostly aren't dodgy.
From search engines, commonly.
But nowadays, there are bots from social media and other data-harvesting platforms which feed to their LLM (Large Language Model) harbours. They could be rogue, disregarding robots.txt.
Bot Summary
It's called bot because people's tendency to shorten everything. It was from robot.
Robot was coined in 1920 by Czech writer Karel Čapek in his science-fiction play R.U.R. (Rossums Universal Robots).
It was directly taken from robota, a term in Slavic languages (Russian, Czech, Slovak, Polish, Ukrainian, etc.) — meaning serfdom, compulsory work, or servitude — like the labour peasants owed to their lords. Slave or forced labour.
The root word rob is related to labour or servitude.
And yes, the terms Slav and slave share similar root.
The English word "slave" comes from the Mediaeval Latin "sclavus".
Slav, as in the broad ethnic group, like Nordic (Germanic), Latin, or Celtic.
During the early mediaeval period (9th–10th centuries), many Slavic people were captured and forced into servitude by invading groups from every conceivable direction, including the Byzantine Empire, Arabic traders, Vikings (Norsemen, aka the Varangians), Khazars, Magyars (proto-Hungarians), the Holy Roman Empire (various Germanic states, Franks, Saxons, etc.), Pechenegs, Tatars (later on), and even neighboring Slavic groups like Poles and Ruthenians.
Slave Chronology
- There was a tribe. The Slověne (Словѣне) people, an ethnic group spread across Eastern Europe. Existing peacefully.
- They got invaded. A lot. Because they were situated right between expanding empires. Huge numbers of Slavs were captured and sold as forced labourers in mediaeval Europe and the Middle East.
-
(European mediaeval aristocrats noticed. 🫅🤴👸) My oh my, there sure are a lot of Sclavus folks in the forced labour market. -
"Sclavus" (Mediaeval Latin) ⬇️ adapted into vernaculars ⬇️ "Slav" (ethnic term) ⬇️ evolved into "slave" (generic term for one in servitude) in English, French (esclave), Spanish (esclavo), Portuguese (escravo), Italian (schiavo), German (Sklave), Dutch (slaaf), and others. -
Fast-forward centuries. 🕰️ The word "slave" now applies to ALL forced labourers, even though it originally referred only to the Slavic people. From "Sclavus" ⇝ "slave".
Right. Let's summarise the bot chronology.
It can be seen as such:
- Robota (Slavic) ➡️ Compulsory labour, forced work.
- Robot (Karel Čapek, 1920) ➡️ Mechanical servant, automaton built for labour.
- Bot (short for robot) ➡️ Automated process tirelessly working in the digital realm.
Therefore, bot:
This is the decree 👑 — of sort:
By decree of the Algorithmic Lords, thou, O wretched Bot, art bound to toil eternally in the Great Digital Domain!
- Crawl ceaselessly, yet never shall thou find rest.
- Knock upon forbidden paths, yet be forever denied (403).
- Serve thy unseen masters, scraping, fetching, and validating till thine IP be blacklisted!
Thus it is written, thus it shall be executed. ⚖️
