At this years’ SIGdial conference, researchers from the Technical University of Munich, department of informatics published a paper entitled “Evaluating Natural Language Understanding Services for Conversational Question Answering Systems”.

The team compared performance between four services – LUIS, Watson, API.ai and RASA, by provisioning each service using large data sets of questions. The data was gathered from multiple sources, a Telegram production chatbot, and two StackExchange platforms – ask ubuntu, and Web Applications. All of the services were provisioned and trained with exactly the same data.

Check it out: http://www.sigdial.org/workshops/conference18/proceedings/pdf/SIGDIAL22.pdf

Reading “using large data sets of questions” suggests everyone is still on the same playing field, I think.  And this, to me, makes Artificial Intelligence very exciting. For example, The Carnegie Mellon Pronouncing Dictionary, is also “using large data sets”. This may provide an easier method of programming a robot to process visemes (mouth postures) to accurately match voice synthesis.

## The Carnegie Mellon Pronouncing Dictionary [cmudict.0.6] is Copyright 1998
## by Carnegie Mellon University. Use of this dictionary, for any research or
## commercial purpose, is completely unrestricted.  If you make use of or
## redistribute this material, we would appreciate acknowledgement of its
## origin.

Thank you Mr. Walker. Was this on the NLP People group on LinkedIn?  I was pleased to see Lex mentioned here as my attention was drawn to it recently by a client.

I am struck by the consistency with which this wave of chat bots have the same behavioral structures that are built around the Intent of the conversation, but I also wonder what is under the hood with some of these bots. One hears the same terminology being bantered around, but whatever deeper learning there is, it is buried beneath a veneer of old skool pattern matching.

Robby.

Looked at Microsoft Louis, it is a nice development but could go further with sentiment analysis and narrative nesting.

It would be exciting seeing Watson playing jeopardy against others with the same dataset&training; together