InkyAnswers: a New Natural Language Search Engine (beta)
I got a ping from natural language search creator Lance Bradley introducing InkyAnswers.com. The site's not live yet, but he sent me a login to check it out.

In the interest of asking intelligent questions I went back and read over Danny Sullivan's gently chiding Hello Natural Language Search, My Old Over-Hyped Search Friend.

Questions for Lance Bradley:
Q) Could you discuss the origins of your interest in natural language search?

My initial idea was a method of machine translation. As I was doing preliminary research into the subject, it hit me that the same idea would also improve search, and that the search implementation would also be significantly easier. Although Inky has evolved dramatically from those initial concepts, I think that approaching the problem from an MT perspective ultimately proved to be invaluable. From the start it was not about finding matching words, it was about divining the concept that the words were getting at, then finding other words that do the same-- which was the essence of my initial translation idea.

Q) What are the chief difficulties of natural language search?
These vary among the different methods of implementation and the differing resources available to the engineer. Some of my biggest headaches have come from simply trying to get the data in a timely manner. I do not have the infrastructure or resources to maintain a web crawl, so for every query I have to download html from between 40 and 60 web sites. Depending upon their throughput, this step alone can take up to 10 seconds. From the start we determined that we wanted Inky to respond in under 15 seconds. So now I have 5 seconds to parse the html into plain text, split that into sentences, and do millions of database lookups, all on 2 moderately equipped servers.

As for more general difficulties, they are similar to the problems with plain old search. Knowing the time at which some piece of information was written, and rather it's still relevant today, can be difficult. So queries about things that change regularly can return out of date information. ("freshman point guard for Kansas Jayhawks" returns Russell Robinson, a freshman two years ago when the retrieved article was written, but now the junior starter, instead incoming freshman Sherron Collins)

I've also gotten frustrated with the dynamic nature of the web at times. Due to new data appearing and sites going unresponsive, results do unexpectedly change. We've tried to solve this by caching sites that we come across, but then you have to ask at what point does that data legitimately expire and newer data become more relevant. Overall, currency is a wicked problem.

Q) What is your vision for your project?

I think Inky has the potential to drastically alter the mobile search environment in the coming months and years. The current user experience for mobile search is nowhere near as satisfying as searching via a web browser. With slow browser loads, clumsy navigation menus, difficult to follow links, and text you can only pray will be readable on the tiny screen, it can be very frustrating.

Inky allows the user to interact with the search engine through text messaging, which most people are already very comfortable with. We are in the unique position to provide this experience because our results are very concise, and therefore shorter than the 160 character limit on SMS messaging. We do not require AIM or any other software to be configured on the phone.

Ultimately, my vision for Inky is take the "Ah-hah!" experience currently provided on the web by big search and neatly package it in your pocket using an existing, universally supported interface.

Q) How do you currently fund your project?

We're funded by an Angel who is also a principal in the project.

Q) John Flowers, in this interview, describes his IM-facing search product as "natural language response." How is his project similar or different from what you're doing?

I believe that by "natural language response" John means that his system provided collections of sentences as opposed to links, although I'm not sure. Although we do return plain sentences for a small percentage of queries, most of our results are resolved down to concise answers, something kozoru didn't do. For example, the query "author of The Baroque Cycle" concisely returns "Neal Stephenson" followed by relevant sentences and links. You even get a picture and Wikipedia excerpt most of the time. Like kozoru, Inky doesn't force the user to enter a question.

Our systems also have little in common functionally. To the best of my understanding, kozoru's parsing system was based upon word pairing. Inky uses information about each word from the web itself to conceptually parse the query and determine the relevant results. This procedure gives us a massive amount of semantic and lexical information that we use to find relevant passages and answers.

Q) In Hello Natural Language Search, My Old Over-Hyped Search Friend, Danny Sullivan wonders "how searchers are somehow magically going to go beyond "keywordese" to natural language searching." What is your response to this?

I think Mr. Sullivan was spot on in his article, although he did fail to mention some key points. Namely, the success of Yahoo Answers and the importance of NLS when/if reliable voice recognition technology is developed.

Although I do maintain that Inky is a Natural Language Search Engine, we do not require the query to be entered in the form of a question. Our NLP algorithms tend to work quite well with phrases and even keywords. Users are free to enter the query however they please. You can ask a question, "How deep is the Ocean?" and "Why do cats purr?" tend to make more sense than their keyword counter parts; phrases also make the most sense sometimes, examples include: "last Beatles album" and "fastest production car in the world"; or you can stick with keywords: "car Back to the Future" or "North Carolina beaches". All of those examples, except for "Why do cats purr?" return concise, 1 or 2 word answers, even with a little picture of the answer. We use natural language processing to build a dramatically better SERP, not to impose constraints on the users.

Overall, I'm glad that someone finally called out all the egomaniacs who are very talented at creating a tremendous amount of hype, but roll out products that differ very little from plain old search. (Do not construe that comment to be directed at Powerset however; Barry Pell does seem to get it, and I'm anxious to see what comes of all that cash.)

Q) What advantages do you think natural language search has over, say, query refinement by sites like Ask?

Actually, I think natural language search could benefit from query refinement, but the methods of refinement would have to evolve to meet the needs of questions and answers. It could be used to disambiguate nouns with multiple meanings, for example: "What is the top speed of a jaguar?" could refer to the animal or the car. In a perfect world, query refinement would be done automatically and results for each displayed.

We actually use some very basic query refinement in Inky, however it's all done inline so the user doesn't notice it. NL and QR aren't mutually exclusive concepts.

Q) Why make your natural language search a mobile phone txt-based search format? Txt is not really natural language?

I should clarify that our mobile version isn't txt-based in the sense that you can query it using short hand ("what r u doing"), and I don't see that feature being added. We simply allow users to interact with Inky via SMS on their phone, which most likely has predictive text.

Q) What keeps you enthusiastic about this project?

Search in general is a very exciting field that's relatively young compared to other types of software. Search engines have a lot of room for improvement, and it's safe to say that they will be radically different ten years from now. To be in a position to guide and contribute to something as powerful and useful as search at such an important time in its development is very satisfying.

Q) What advice can you offer other solo developer/entrepreneurs out there?

Well, I'm still in the phase where I'm soaking up advice, so I don't think I'm really qualified to advise others. But if I had to, I would tell them to find inspiration. I've personally drawn quite a bit of it from traditional Geekdom. I can't tell you how many times I've thought I was completely stuck until I heard an encouraging line from an episode of Firefly or a NerdTV interview.

I should also clarify that I'm not solo on the entrepreneurial front. I'm supported by some very experienced and accomplished business leaders who have been instrumental in guiding myself and Inky. So I would also advise other first time entrepreneurs to seek out those who have succeeded.
Garrett French - MarketSmart Interactive | permalink | +marktd |