On Getting Found, Or, Search Engines: Is There A Difference?

I have a story today that comes from my predilection to “self-syndicate”, meaning that I post my stories far and wide, in the same way a newspaper columnist is syndicated nationally—or beyond.

After I post, I know others will also post my stories to their sites, a topic that was itself the subject of a recent conversation.

To keep track of it all, I use the Google...but I recently wondered if that’s actually the most effective tool for the job—or not—so as an experiment I recently challenged several search engines to go out and seek the same search term.

We find out today...and the results are, indeed, interesting.

So here's the rules of the game: on the afternoon and evening of November 29th, I posted my story "On Stimulating The Future, Or, "It's The Ytterbium, Stupid!"" on 27 sites. The next morning I conducted the searches you'll see referenced in this discussion using as a search term the exact words of the title, in quotes, just as it appears above. During the course of writing this story, we'll revisit the same sites to see if the results have changed.

Let The Contest Begin!



So the first search was conducted on Google, which found 849 results.

The reason that happens is because the tags associated with (or the proper nouns that appear in) a story often trigger websites to place that material on pages with other stories with matching tags or names, as you can see from this example at RootsWire. (The story appears twice because it was updated after it was posted.)

This creates lots of iterations of the same title on the same site under different categories, a situation other search providers seek to reduce; this being the one of the points behind all those recent ads for Microsoft's Bing search engine.

A quick note about "search consistency": seeking for the same search term at Google on multiple occasions will yield different results each time, even if the two searches are conducted immediately after one another. For example, my search this morning found 852 results--and then, just a few minutes later, 653. (By the way, if you click on these links now, some other number of results will appear, which is its own comment on consistency.)

We next visit Bing, where 16 results were initially found. Interestingly, some of the links were the ones I placed, but 6 of the 16 were multiple iterations of the same story on three sites.

As with Google, visiting Bing today might yield 57 links--or 2530, or 152, or 26--and despite Bing's advertising claims that they make searching simpler by eliminating Internet "clutter", a huge number of the links I'm seeing here are links to the weather in virtually every city in Maine; all of these linked back to "Weather Underground" weather reporting pages...and all of those pages were from the same basic address: insert name here.wunderground.com.

Next was Yahoo!, reporting 887 results (and then, after clicking through a few pages, 1580). There was an interesting variation to the pattern of what they found, however: more results from the first 50 were links to the original 27 postings than appeared to be the case with either Google or Bing.

The search today found 860 results...four times in a row...which is by far the most consistent results reporting so far--even if the results from the other day were completely different.

Lycos found 67 iterations of the posting...or 50......and then 49...with roughly a dozen of the first 30 listings being "duplicative" entries, which is fairly consistent reporting. Returning to the site today, the search engine found 69 listings--and it was also able to do that four times in a row....which makes it at least the "consistency equal" of Yahoo!

Dogpile (a product of the fine folks at Infospace) aggregates results from Google, Yahoo!, Bing, and Ask.com into one set of results...and for some reason the first result on the second page was for a futures trading opportunity ("I'm shocked to discover there's gambling here...!").

That said, Dogpile "sniffed out" 40 results, with many of those being "duplicate" instances of the same story from the same site. Conducting the same search today yields 6 additional results--all of which appear to be duplicates of the previous 40.

On four further attempts to search, the original 40 results were found.

WebCrawler, another Infospace property, located 38 results; again, the results are highly duplicative. It is not possible to enter the entire search term at this site, instead, the term...

"On Stimulating The Future, Or, "It's The Ytterbium,

...was used.

Four additional searches were conducted today, with 38 results found each time.

(Because the page-naming conventions of both Dogpile and WebCrawler insert an ! into the page names upon which results are presented, they can't be linked here, and you'll just have to visit the pages on your own.)

Remember Altavista?

Altavista found 904 iterations of the story, then 17,800 on today's search. There is an option to either search "Worldwide" or "USA", the Worldwide search, conducted immediately after today's USA search, found, oddly enough, 2460 results--and for at least the first several pages, which was as far as I looked, the results were the same as for the USA search.

Four additional searches, conducted today, located the same 17,800 results.

One strange idiosyncrasy of the site is that it won't actually display those 17,800 results: instead, it only displays the first several pages of results (in this case, 7 pages), and then just stops, with no additional pages made available beyond that point. There is an "advanced settings" page available, but it does not offer any solution for this problem.

Ask.com displays 240 results on the first search--and they were the only site to report the listing on the Times of India site right there on the front page (which, if you return to the site, is no longer the case)--but on the down side, 1/3 of the results on that first page were "sponsored results".

After the first page, ½ of all results are "sponsored", and the results are highly duplicative. By page 10 of the results, as few as 2 of the 13 results on the page are not sponsored.

Today's searches located 452 links, then 449, then 452, and then (take a guess...) 449.

Ever heard of Duck Duck Go? Neither had I before this story. They feature an unusual format that displays some results, and then, when you click "more results", displays those below the first results on the same page; a pattern that continues until all results are displayed.

The Duck located 35 results on the first attempt, with no duplicates. The "wunderground" domain was represented--but only once.

Apparently recognizing that their searches are not going to give every result, the site encourages you to also search at YouTube, flickr, twitter, amazon, and Google.



Turning off the "safe search" feature yields 43 results, including BhamLinks.com (a news aggregator from Birmingham, Alabama), Pshcye's Links, ("Esoteric Subjects on the Web"), and the "Li-Ion" page from the Journalism that matters site (Li-Ion, by the way, is the abbreviation used to describe lithium ion batteries.)

Conducting additional searches on the site today yields the same 43 results.

Finally, Cuil. I had never heard of this site before...and apparently, they've never heard of me, either, with zero results reported for my query. Searching the "127 billion web pages" they purport to scan today provided no results again during four additional checks--which makes this site the most consistent of the search engines I examined.

I conducted a test search for pizza (with no quotes). 809,000,000 results were found...but only two were displayed on the "All Results" tab: one for Pizza Hut, one for the Wikipedia entry for pizza (which featured the story of how pizza was introduced into Pakistan, of all things). Even more odd: on the same page you can look up "Pizza franchises" and other pizza related results "categories", and there's a "Timeline for Pizza" with entries like: "2004 Melbourne, Australia" and "1993 Pizza was".

All of this appears to be at odds with the intent of the site's operators:

"Popularity is useful, but has dominated search results so heavily that it gets harder and harder to find the page you want, especially if your search is a complex one. Cuil respects popular pages and recognizes that for many simple searches, popularity is an easy answer to your question. But for a deeper search, establishing relevancy is more than a numbers game. Cuil prefers to find all the pages with your keyword or phrase and then analyze the rest of the content on those pages..."

And The Winner Is...

Those are the results: so, what about conclusions?

The first conclusion we can reach about all of this is that the number of results that any search engine locates on any particular visit are highly variable--and so much so that the number of results presented appears to be virtually random (with the notable exception of Cuil, which seems to be consistently unable to find anything).

With that said, if I was quickly looking for this particular story, it appears that some of the odd search engines might be the best choices, including Lycos, Duck Duck Go, and Ask.com.

On the other hand, if the idea was to determine how far a story has been distributed, Google seems to be the winner.

There is another reason to use search engines, that being to find information about a topic that you currently don't know enough about; this test is not well suited to answer the question of which search engine is best for that purpose...and it's a test that we'll save for another day.

So that's today's story: we visit quite a few search engines, we learn that the results you get are almost always entirely unpredictable, and, in what might be the most important lesson of the day, we're learning that deifying Tiger Woods can backfire on you, big time.

On Better Ways To Spend Billions, Or, Microsoft Wants Yahoo! Why?

For the past couple of years Microsoft has pursued Yahoo!; and over the past week the news has been full of discussion regarding Microsoft’s (now rejected) $44 billion takeover offer. Rumors suggest Yahoo! might seek as much as $56 billion.

The company that put Bill Gates on the map reports that the intent of merging the two is to create an online community and search service that could rival Google, and to develop new services that will turn that online community into a new form of online cash register down the road.

As a fake consultant, it’s my real fake job to offer advice on these sorts of deals; and since Microsoft’s CEO, Steve Ballmer, does not appear to have been well served by his “real” consultants, today’s discussion is, literally, advice from a fake consultant.

To understand the logic of the deal, we need to understand exactly what Yahoo! does for a living—and the most important thing they do is “host” an online community of millions (Yahoo! Groups) in the US. They also own a major e-mail service, the HotJobs website, and are trying to establish an open-source competitor to the Windows Mobile operating system for your phone or Blackberry-like device.

The reason internet communities are important is because they are one of the most effective ways to gather potential online customers in one place…which allows the owner to gather advertisers to the same place. It also becomes possible to establish online “shopping malls” (think of eBay and their Power Sellers); and just as in the terrestrial world, it’s possible to charge “virtual rent” for the online space. (You can also make great money processing the online payments, which is why eBay paid so handsomely for PayPal.)

Another way to gather customers is to provide an ubiquitous search service that everyone wants to use…in other words, Google. If billions of “eyes” are on your site daily doing searches, you really have something to sell to advertisers, and Google is the world leader in that market.

If you’re really smart (and Google is), you keep records of where your visitors go and use that data to develop a process to deliver advertising to visitors that will be tailored to what interests each of those visitors. Then you buy the largest online advertising agency (AdSense). Then you convince the owners of millions upon millions of websites to let you place your advertising on their sites, and you use your process to decide which ads go where.

Yahoo! has a search service, but Google’s is immensely more popular. Microsoft owns the MSN internet service, which also operates a search service and some online community features, but the two combined do not begin to approach the popularity of Google.

Who saw “Roger & Me”?

One of the most memorable characters from the film was Genesee County, Michigan’s Sheriff’s Deputy Fred Ross, who was charged with the task of performing evictions in a county that was losing thousands of GM jobs at a time—over 30,000 lost in the County’s largest city, Flint, by the time of the film’s 1989 release.

He posed a question (and an answer) that is as relevant in this instance as it was in the film. To paraphrase: “What’s the point of a woman marrying someone else who’s already broke? She can be broke by herself—and do it for less money.”

Now we get to the advice part.

If Microsoft doesn’t marry Yahoo!, what are they to do to finally grow the internet business they have never really seemed to be able to get off the ground?

A business that can challenge Google…especially in China.

Here’s a radical thought: instead of spending $56 billion on someone else’s shareholder equity, why not invest a similar amount internally and make MSN into the world’s coolest place to be?

Since China is the market of the future, let’s start our investment program right there.

Why not create a new Chinese television show, not unlike a melding of “The Apprentice” and “American Idol”, offering a big prize…say, $50 million…for the best internet business idea? See if Bill Gates will be a judge alongside a famous Chinese business personality…or the iconic character who awards the cash to the winner.

“China’s Next Mega-Millionaire…sponsored by MSN”.

In parallel with the show you create the MSN community that runs the voting, the contestant video audition upload service (online viewers vote for these, too…keeping eyeballs on the site 24/7), hosts all the chat and gossip and “news” and daily voting results that you can pump out, and sells the merchandise generated…and you get to deliver advertising to billions of eyeballs week after week…on TV, through personal appearance tours, and on the Web.

Of course, the recent announcement that all mobile phone carriers except Nokia now offer Windows Mobile-enabled devices means you can port all this content to even more eyeballs by phone.

All of this could easily be done for under $1 billion…including marketing, and the odds and incidentals that might be needed to get it going. We also assume a few well-placed “licensing fees” will assist the process, and expect the cost on that to be manageable as well.

Naturally, the second part of the prize is MSN operating the business in conjunction with the new “Mega-Millionaire”…and the ongoing relationship with the Chinese public is further reinforced.

And then you do it all over again.
And again. And again.

The payoff from being China’s coolest ongoing “product release” party?
You’re gonna need extra staff to count all the money.

But I’m not done yet.

Want to quickly make MSN “the” world’s coolest site to visit?

Free, legal music downloads.
That’s right…any song you could ever want, from anywhere in the world musicians can be found, free.

Offer each of the major recording library owners some absurd amount of money on an annual basis ($500 million each, guaranteed, upfront, nonexclusive?), in exchange for the distribution rights to the libraries. Offer to distribute new music for new artists, as iTunes does today…and pay those unsigned artists based on how many free downloads MSN customers request. This makes MSNZune the most desirable site from the artists’ and customers’ perspective—and knocks iTunes down to size in a big hurry.

It’s a win for the library owners as well—free and legal makes pirating pointless…and it creates opportunities for distribution of desired media in limited packaging for premium prices.

The total annual cost: this could be done for under $5 billion annually…and it should be possible to recover that cost in the form of advertising revenue as you grow the site. Even if you take 3 years of no revenue at all, the cost is below $15 billion—and you can sell Zune players and tons of artist-related products at the same time.

By the way, if you want to monetize music…has anyone at Microsoft ever considered making an offer for LiveNation? There’s a synergistic brand, not yet fully developed, that could be leveraged off all the other ideas we just discussed. The immaturity of the brand means it could be purchased with far more “bang for the buck” than Yahoo!, with a real growth potential for the future.

What kind of growth potential?
Can you say LiveWorld?

Hey, don’t you sell XBOX?

Think MSNXBOX. In addition to the XBOX Live service, you add “meet the developers” chats and interaction (watch the G4 channel and then either buy them out or do it better)…MSNTESTCORPS for beta testers and others (more viral marketing potential and a “testbed” for new game concept discovery); and the MSNToolshed, for those who want to build out their own versions of the games (an apprenticeship program? an international “unknown talent” hiring hall? an ongoing focus group? all of the above? you bet).

MSNFREEGAMES does just what it sounds like: it provides free (ad supported) games to XBOX, PC, and cell phone users alike…and contests like crazy. $50 million in giveaways annually, distributed worldwide? That’s peanuts compared to $56 billion…and it grows property you already own that’s already demonstrated it can be of interest to the gaming public…and again, you don’t have to enrich someone else’s stockholders to do it.

And of course, MSNZune, the Zune device itself, and the XBOX complex all gain presence at future MSNLiveNation events…and giant MSN “tour tents”, swag giveaways…and “meet the musician”, “meet the game character”, and product release events become more and more melded together from then on.

Even more so with the release of “Rock Band”, which seems to be the vehicle by which all of this might be realized…especially as new bands are added to the game…creating the demand for “add-on” versions that let you be the band you love the best…and naturally you’re willing to pay MSNLiveNation fat money to get tickets to the show when that same band comes to town.

This means MSN suddenly becomes the means by which you can sell Zunes, XBOX, games, and concert tickets…and a place that gathers the participants in all of those communities for a world of hungry advertisers.

Here’s another idea: develop an online music creation service that gives XBOX owners and PC owners (and you too, Apple and Linux!) free tools that allow them to make music and video, along with online musician and fan communities and artist collaboration tools that let artists meet, link up, and build an audience at the site…which means MSNArtistBuilder could quickly become another of the most important gathering places on the ‘Net….with the output of artists who work over the service ported to the MSNZune service, as we described above…not to mention the potential “Rock Band” tie-ins.

How to promote it?

How about the first “Planetary Battle of The Bands”, with videos and voting from the site, by XBOX Live or computer or cell phone, with a $10 million prize for the winners to guarantee some serious worldwide media attention? And then do it again every six months.

Ka-ching!

We still haven’t really discussed what might be done with MSNSearch, but here’s an idea no one else has today: why not spend some R&D money to develop a “media hunter” engine that allows users to hum a tune which the service uses to locate the song? Then expand the idea into a tool for searching video

Universal media search that’s better than anyone else’s: an advantage that should be worth quite a bit, and if it costs $1 billion; or even $5 billion…compared to $56 billion, so what?

Spend the extra money to make this useable in every language you can find...in other words, as far as the competition is concerned, hit ‘em where they ain’t.

Now even if you go nutty with the money, there’s no way all of this could possibly cost over $40 billion (and don’t be afraid, Mr. Ballmer, to send me a lil’ taste of that delicious Redmond Cheddar if all this strikes a responsive chord); and we haven’t even come to the part where Microsoft might be able to truly realize their fondest dream…the part where they stop selling so much software—and start renting more of it.

The copy of Windows that came loaded on your computer was purchased by the machine’s builder…but why should Microsoft only make one sale? Why not “rent” maintenance, upgrade, and new product services by the month? This is already underway, and by moving the process to an MSN branded location we lay the groundwork for the inevitable next step: Office Online becomes MSNOffice and that rental process begins.

A version of Office can be developed that allows finished work to be stored and retrieved by the user from their own machines, and the Office program functions are rented for…say…$9.95 a month, which means there’s a huge potential future revenue stream—and it’s one Microsoft has always wanted to access.

The advertising supported version, available for free, offers even more potential for “eyeball gathering” (along with reducing the OpenOffice and piracy threats), and MSNOfficeFree users’ “sessions” would last much longer than the microseconds most users spend looking at pop-ups as they wait for other sites to load today.

So that’s my advice, Mr. Ballmer: instead of investing $56 billion to marry someone who’s already broke, invest the same money in Microsoft and build something truly revolutionary.

And, finally, consider this: if I’m wrong, I’ll go away, quietly, peacefully…you’ll never hear from me again.

But Steve…if I’m right…you have the chance to enrich millions of registered Microsoft shareholders.

Need I say more?