[Note: this post is adapted from a long comment I made on one of Albert Wenger’s recent posts]
Albert, I think you’re absolutely right that the search game is just getting going. If you define search as “finding relevant web pages,” then Google has search locked up and solved. But if you define search as “providing the information you need to make the best decisions all the time” then there’s a vast and fertile green field ahead. At the end of the day, the goal of search isn’t to deliver links – it’s to deliver wisdom.
So how are we going to do that?
Let’s start with a thought experiment: imagine that you yourself are hired to be a search engine. Now imagine that your wife is a search user and so naturally any time she needs something you want to give her the absolute best and most useful answers you can.
Let’s assume that you have unlimited time, resources, and patience to search. But let’s add a constraint: the only way your wife can communicate is by barking 2-10 words at you and waiting silently.
How you would go about answering your wife’s queries?
For example, let’s say she asks “pizza broadway.” You can probably guess what she wants, but at least for me Googling that phrase returns junk. You, on the other hand, would take what you know about your wife’s tastes, then you’d compile a list of restaurants on Broadway, then you’d figure out which ones serve pizza, then you’d look at Yelp to see how well reviewed each pizza place is, and then perhaps you’d call your friends who eaten at those restaurants to get their opinion (which you would discount based on how much you trusted their taste in pizza.) Ultimately you’d hand your wife a simple summary telling her that the best pizza on Broadway was at “Pizza X,” located at 3333 Broadway (click here for directions or here to have it delivered.) You might even provide a summary of why you chose that answer if it was a close call.
Next consider a slightly more complicated (but much more important) query: “best college for my child.” Even with your human judgment and everything you know about your wife and children, that’s a mighty hard question to answer. But it’s not impossible, and it probably has an objectively correct answer (or at least there’s a small set of equivalently good options.)
So how would you get to the answer? Personally, I would start by fighting back against the initial constraint of this thought experiment, i.e. that you can’t communicate with the searcher. I’m sure you’d be able to do a much better job of answering the college query if you could have a conversation with your wife that lets her volunteer additional relevant information (e.g. whether your kid prefers small or big schools.) That additional information would let you figure out which variables to optimize and thus come up with a much better answer.
I’m using this thought experiment to try to make three points:
1) Search is a much larger and much more economically and socially valuable problem than “find the best link.”
2) A great answer to a hard question requires sophisticated aggregation of information from a variety of sources.
3) Instead of starting with an interface (e.g. “text in a box”) and figuring out how to
stretch that approach, one should start by figuring out what information one
needs from the user and then design the interface to most effectively gather
that information.
Facebook’s graph search is extremely exciting because it’s the first approach I’ve seen by a large player that recognizes each of those three principles:
1) Graph search lets you solve an entirely new class of problems. As a real life example – last week I spent about 20 minutes compiling a list of people to invite to my
birthday party. I could have done that in an instant by querying “close friends
who live in New York.”
2) The whole system is based on Facebook’s massive structured dataset. Cost-of-data-acquisition was what killed off the 1980’s “question answering” AI systems, but Facebook has basically solved that problem by conscripting an army of human volunteers.
3) It’s still a textbox, but the addition of the “concept chaining” system (i.e. “in New York”, “from 2003”) lets users massively but easily increase the expressive power of
their queries.
Obviously it’s going to be a long, long time before a computer can tell your children where to go to college. But it’s going to happen.
The search systems of the future aren’t going to be location-based OR mobile OR social. They’re going to be all of the above. They’re going to combine cutting-edge machine learning and NLP with crowd-sourcing and old-fashioned manual data entry to create comprehensive, varied knowledge bases. They’re going to use embarrassingly parallel systems to process data that’s organized with ontologies from linked data and stored in not-only-SQL data-stores. And they’re going to use best-practices from Human Computer Information Retrieval (HCIR) to expose interfaces that let searchers effortlessly deploy statistical reasoning to get the exact answers they need to live better lives.
Imagine a world where your mobile phone can instantly find every band similar to your favorite bands you’ve never heard, or can provide you all the core evidence for and against investing in an entrepreneur, or can save you hundreds of millions of dollars by correctly estimating how many factories you need to build to meet demand for your new widget.
Those answers are coming. And they’re coming soon.
[Edit: if you like this post, help more people see it! Please share, or upvote this post in the Hacker News “new” queue ]