Please Do NOT Feed The Robots - Thank You

Search businesses send-out crawlers and robots to learn from the Web, to find and feast upon fresh, meaningful, helpful, insightful, lean and nutritious content. Google's mission is 'to organise the world's information and make it universally accessible and useful'. A core and critical component of that mission is to identify real food and ignore artificial junk food.

How Fat Is The Web?

As of 2023^[1], there are thought to be more than 50 billion web pages on the Internet. Of these, at least 4.77 billion are believed to be indexed. Less than 10% of the Web is routinely monitored, crawled, tracked and remembered by search engines.

90% of the Web is fat.

Please do not feed the robots. Thank you.

Googlebot (and others) demonstrate an ability to exercise restraint when faced with the all-you-can-eat buffet that is the Web, eating only what they need to learn from. Googlebot recognises junk food when it sees it and chooses the healthy option.

Natural Language Processed Food

AI models capable of writing content have, to date, acquired their intelligence by being trained using the vast amount of textual data that's presented on the Web. We're now seeing AI generated content appear right across the web - from the interpretation of song lyrics through to the writing of dissertations. Not all of this processed content is good and healthy. There will come a time (in fact, it may already have come) when AI models will be unable to trust the Web because it has become so full of fatty, chewy, processed food.

It's already clear that Google understands what bad web content looks like and tastes like. By continually selecting only the choicest cuts of the Web, Google works hard to present, what it considers to be, the best Search results available. This model is not about to fail anytime soon. As AIs write a greater portion of the Web's content, so Google cannot afford to super-size itself and must avoid the junk, becoming ever-more choosy about what it trusts and consumes.

Is This Steven Writing?

Search engines understand language in ways you or I do not or cannot. A Natural Language AI processes text by chewing the syntax of the content it consumes; breaking-down sentences, identifying parts of speech and creating branches of dependencies. Entities such as dates, people and prices are labeled and classified. The sentiment behind the content is assessed to calculate what the overall opinion, feeling, or attitude of the text may be. In short, it's likely that Googlebot will be able calculate whether or not it's me that's written a page of text.

You can think of the analysis of such a pattern of communication as the measurement of the personality of your website.

Google, therefore, has to be able to spot and avoid processed food. For, if it weren't able to make healthy choices for itself, it would end-up spiraling on an unnatural and unhealthy binge. Natural language will have its own tone of voice, its own texture and its own taste. Artificial, processed language will have a consistency that resembles beige, puréed baby food.

Garbage In - Garbage Out

AIs train themselves on the Web because it's, currently, a reliable source of natural language. The larger the portion of the processed Web (that gets written by AIs) becomes, the smaller the portion of the natural Web (that can be reliably learned from) will become. But don't worry, Google's probably cutting-out more than 90% of the fat already. The future's already been written - AI will increasingly choose quality artisan content, from which it can discern meaning, over highly processed content which just tastes artificial and just the same as everything else.

Should You Use AI Content On Your Website

Why ever not? Most people love a sausage now and then and there's nothing wrong with eating a sausage as part of a healthy, balanced diet. AIs output content based upon broad match criteria to generate broadly acceptable content. Search wants to relish the finest cuts of meat and will not spend its time eating processed sausage because it knows it's bad for its health. Google's business model is built upon returning the best results it can cook-up. And, so, the search engine needs your website to have its own voice, to be opinionated, to be certain of its message, to be clear about its alignment with your Customers' own voices and values.

Tasting Notes

What prompted this blog post was a routine inspection of the digital estate, this morning; proving that it's not only AIs which do their learning from the Web. This notion of quality came to mind after I performed a routine inspection of a new website of a former Customer of ours. We'd built a one-page, super-scroll website for a startup business and were content with the #1 Google rank it achieved and maintained for the vehicle finance broker. Having successfully started-up, said Customer decided to go-it-alone and build their own Wordpress website. 👏🏻

Now, one month after the startup launched their own website, I got curious as to how the my first website perfoms against our custom design and build. A quick colon search site:www.ourfinancebroker.co.uk on Google, this morning, revealed that Google's still not yet indexing any of the pages from the new website but is ranking to two former blog pages from our startup site as well as seven PDFs.

Every SERP link returned by Google 404s when clicked. I fully expected Google to have updated its index with the new website content after one month - maybe Google's biding its time. Maybe it's not only what Google eats that is important but also how it's cooked. 👨🏻‍🍳

• Is This Steven Writing?^[2]