Nostalgia
Back in the early 2000s, when I first got my hands on a proper internet connection at university — studying Electrical Engineering, no less, with a focus on Control Systems — I genuinely believed that typing something into a search engine meant it was out there, live, rummaging through the actual web in real time.
Embarrassing, really.
Like sending a little scout off into the wilderness and waiting for him to come back with news.
Well, I didn't discuss it with anyone back then. Just searched for materials from the internet, went back glaring at the books and screen, no —
Prithee, good fellows, when I didst query Yahoo!, was it truly scurrying about the great web, fetching results in earnest?
⬆️ No. Absolutely not.
We were just doing what the curriculum told us to do, no further questions. And in my department, no proper "Search Engine" topic. But the logic underneath — the gates and whatnots — and the bloody semiconductor magic? Oh, we were absolutely drenched in those.
Here's the MOSFET threshold voltage for our amusement:
Steve from Microelectronics:
I beg your pardon, that formula does not belong to your department!
Well. Fine. Here's the closed-loop transfer function:
End of amusement
Anyway, turns out — not quite, mate. I mean, the search engine bit, not Steve.
How It Actually Works
The main flow goes as such:
-
Crawling
Bots (often called crawlers or spiders) continuously roam the web, hopping from link to link, page to page, quietly gathering content.
⬆️ Hopping. Hop + ing = hopping, English. Not "hoping". They didn't hope for the best. They were commanded to do the tasks, no hoping was included in that. — I hope that link is friendly! Last link, it took my trousers. I cannot believe that. — one sentient bot uttered.
-
Indexing
Everything gathered gets analysed, categorised, and filed into a massive internal database.
This database is what a search engine actually is, underneath all the polish.
-
Updating
The bots keep revisiting.
Pages change, new ones appear, old ones die.
Some pages get deindexed — quietly removed from results due to legal requests, poor quality, or the site simply going dark.
-
User searches
When we type into that search box, we're querying that internal database —
not the live web.The results then get filtered and ranked through
algorithms.The algorithms:
- relevance scoring,
- spam detection,
- legal compliance,
- and a fair bit of commercial logic too.
And it doesn't stop there — every click we make on a result, and how long we actually stay on that page, feeds right back into the ranking algorithm. The users themselves are unknowingly voting on the quality of results with every single search.
⬆️ Roughly, it is indeed as such.
The Neighbourhood Analogy
Perhaps the simplest way to picture it:
Imagine we need to find an address in an unfamiliar neighbourhood.
We don't just wander about the streets hoping to stumble across it. We consult a map, or ask the local bloke at the corner shop who's walked every street and knows the area like the back of his hand.
The search engine is that bloke. He's already done all the wandering — so we don't have to.
And that, really, is all there is to it. No little scout dashing about the live web on our behalf. Just a very large, very well-organised filing cabinet — with a bloke at the front desk who's already done all the legwork for us.
Spam
The "spam" bit — that was from Monty Python.
Not the dish, per se. Well, yes, that spam, the pre-cooked canned meat product. But it was more of the repetitive malarkey in their sketch — it could be a sandal or anything else, really. But in it, they used "spam". Which was utterly comical.
And then people on internet back then started to use that term to refer to... that repetitive malarkey.
Junk email = spam. ⬅️ Back then. And people who consumed spam were bloody confused by that term. Back then. Not now. Now, everyone is possibly well informed or simply — Oh yes, spam. The internet spam. Not the food spam. Back to swiping my arse.
Imagine if they used sandal instead of spam. —
Oi Geoff, my mailbox now is full of sandals! Such a rogue sandaler.
No Small Investment
It is worth pausing for a moment to appreciate what is actually running underneath all this.
The crawling and indexing we described above doesn't happen by magic — it runs on
These data centres sprawl across multiple countries, engineered to stay online every hour of every day without flinching.
Then there are the engineering teams — armies of them — dedicated solely to keeping the crawlers crawling, the indexes sharp, and the whole enormously complex machine ticking along properly.
And yet, we type into that search box entirely for
Not a penny changes hands.
Which naturally begs the question — How on earth do they pay for all this?
The answer, of course, is us — or rather, our attention.
Advertising is the engine behind the engine. Every search we make, every result we click, every pattern in our behaviour feeds into a finely tuned commercial machine that sells targeted advertising to businesses worldwide.
The search engine is free because we are the product being sold to advertisers.
Mm.
So yes — not exactly a charity operation. Just a very well-oiled business hiding behind a very clean, simple search box.
Mwahahaha!
Though in fairness, it is a reasonable arrangement.
Businesses need customers, and the internet is precisely where those customers roam. Advertising on a search engine is simply that age-old transaction — connecting a seller to a buyer — at an almost incomprehensible scale.
Just commerce doing what commerce does.
The Cookie Banner
The legacy of all that enthusiastic data harvesting, of course, is that we now cannot visit a single website without being ambushed by
the size of a motorway billboard.
You are quite welcome, everyone.
Cheers. See you next time! ๐
Comments
Post a Comment