Why Conservatives Can’t Just “Build Their Own Google”
Why Conservatives Can’t Just “Build Their Own Google”
By Daniel Greenfield
A few months ago, Google began aggressively censoring my articles. My Front Page Magazine articles went from appearing on the first page to the seventh page. As did my old personal site. Even specific searches for particular articles often no longer turn up results on the illegal monopoly search engine.
This isn’t entirely unusual and has happened to a number of other people writing about Islamic terrorism.
I wasn’t particularly surprised because I had spent quite a few years warning conservatives that Big Tech was coming for them only to be greeted with incomprehension. (Fast forward to today and everyone is engaged on it, but generally in the wrong ways, going down the Section 230 rabbit hole.)
Conservatives now champion DuckDuckGo, which is a cute privacy oriented way to get results from Microsoft Bing. Since Bing is already privacy oriented, I’m not sure why that’s much of a help. But I was recently introduced to Gigablast which, on casual inspection, actually seems to function like an actual search engine. It may not be pretty, but it fills me with some 90s nostalgia. It’s a lot smaller, but it bills itself as the “the only non-Big Tech search engine in the U.S. that still crawls the web.”
That’s an important distinction, as we’ll see.
And when I use Gigablast, I actually seem to get the kind of results I would have gotten from Google in 2012, instead of the usual grab bag of redundant corporate media spam, assorted spam from Google’s garbage index which sucks in every single spam page from the third world, but keeps out conservative content, and Google’s attempts to reword my query to match what its algorithms say most smartphone users are searching for.
But Gigablast’s situation also breaks down the problem of just telling conservatives to build their own search engine or other service.
Google and Microsoft are the only search engines that spend hundreds of millions of dollars annually to maintain a real-time map of the English-language internet.
“It costs more money than we can afford,” said Gabriel Weinberg, chief executive of DuckDuckGo. In a written statement for the House antitrust subcommittee last year, the company said that “an aspiring search engine start-up today (and in the foreseeable future) cannot avoid the need” to turn to Microsoft or Google for its search results.
And that’s not a search engine. It’s just remixing leftovers from Google and Microsoft into Thursday’s meatloaf.
But it’s not just about the money. Sites will block or interfere with search engines crawling the web that aren’t Google or Microsoft. And here’s where we get to the fundamental problem of Big Tech monopolies.
You can’t just make your own Google today.
When FindX started to develop an alternative to Google in 2015, the Danish company set out to create its own index and offered a build-your-own algorithm to provide individualized results.
FindX quickly ran into problems. Large website operators, such as Yelp and LinkedIn, did not allow the fledgling search engine to crawl their sites.
That’s a point that Gigablast makes on its blog when it comes to Cloudflare, which is widely used.
It has come of notice to me recently while spidering the web that Cloudflare, a Content Distribution Network (CDN), is now offering as they put it: “DDos Security” by using several tactics that prevent free web search engines from spidering public web sites.
…and Google…
The big tech companies are buying up all the best content on the internet and prohibiting smaller search engine companies from indexing it. For example, Google owns youtube.com and is increasingly prohibiting potentially-competitive search engine crawlers from indexing its content.
…and the United States Senate
Then I followed up, twice, actually, asking if Google was throttled the same way, or if I could be on the same whitelist that Google is on, and was met with deafening silence. I need to be able to spider somewhat fast to get any sort of coverage on these big websites. To surmise, if you want to start a search engine to compete against Google, the U.S. government will make sure you do not get far.
And as Matt Wells of Gigablast notes,
“Google prevents their search results from being combined with search results from other search engines to create a meta search engine. So, in other words Google is saying something like “we are free to use your stuff how we see fit, but you can’t use our stuff how you see fit.”
So when you’re searching something like DuckDuckGo, it’s drawing on Bing’s index.
Google gets treated differently by webmasters (an old un-pc term) because it dominates over 80% of search traffic. When Google gulags you, you’re gone. But you can’t just duplicate another Google. Not unless your search engine provides enough traffic and even then, a whole bunch of major sites will likely block you, either on principle or because they’re already in bed with Google.
This is another example of why the dot com monopolies can’t just be fixed by building alternatives. Not when the infrastructure is already rigged in favor of dot coms like Google. The only answer is to break up the big monopolies, particularly Google.