Tuesday, February 13, 2007

The Time To Build NLP Applications

The Time To Build NLP Applications:[...]I am proposing that a more sophisticated search engine would be explicit about ambiguity (rather than let the user and documents figure this out for themselves) and would take information from many sources to resolve ambiguity, recognize ambiguity and synthesize results.

Sure, those are interesting research goals. But the paths to those goals are not even close to being mapped out. Recognizing ambiguity is an especially difficult problem, because we do not have robust methods for recognizing and quantifying what we don't know. Stupid mistakes, even bugs, get blended with subtle distinctions in a hodgepodge of candidate interpretations, most of which will appear baffling to users.

This is why I believe that tested NLP techniques will creep into search from the inside, building out from islands of confidence. I like a evolutionary developmental biology analogy: complex, more versatile species did not arise by wholesale redesign of old species, but by co-opting robust, highly conserved modules into new roles. Reading recommendation: The Plausibility of Life by Marc W. Kirschner and John C. Gerhart.

No comments: