From http://see.stanford.edu/materials/ainlpcs224n/transcripts/NaturalLanguageProcessing-Lecture03.html :

So does everyone know what the Turing test is? A less-known fact is there’s actually been an attempt for a bunch of years now to actually implement the Turing test, which has been referred to as the Loebner Prize. And the Loebner Prize was sponsored by Hugh Loebner, who’s this guy who really has nothing to do with AI.

He made his money actually doing shows on Broadway, but somehow he had heard of the Turing test and he decided that he’d set up the Loebner Prize to have people compete to do the Turing test. And if you can succeed in doing it, it’s got a million-dollar prize, but I guess the problem is that no one’s really close to succeeding and the yearly prize for having the best - the most human-like computer of the ones that enter is only $2,000, and $2,000 isn’t enough money to produce a lot of serious work. So it tends to be a bit of a hobbyist’s competition of doing the Loebner Prize.

But nevertheless, the Loebner Prize has taught us some things that Alan Turing did not think of, and one of the things that it’s taught us is that, you know, the Turing test, as he set it up, he kind of forgot the HCI angle. The kind of HCI - human communication angle of what happens when people communicate.

And so the way it’s set up in the Loebner Prize, there are a bunch of judges and they communication by typing, so it’s, you know, kind of like doing it with instant messaging.

I wish it was like instant messaging!

Now, yeah, so the interesting thing that this has shown, really, is sort of the kind of social side of communication side of things, rather than actually the artificial intelligence.

Totally agree!

So one of the people who’s done quite a bit of - done some of these hobbyist entries is one of my fellow compatriots, Jason Hutchens, who did this for a number of years.

[...]

So what it’s doing is it’s being an ngram language model. It’s a pair of ngram language models which have a lot of data in it, some of which may well have been derived from [inaudible]. And so what it’s doing is for each time, it’s taking the user’s sentence, it’s finding one content word in the user’s sentence, and then it’s running a markov ngram model in both directions, so there are two ngram models.

[...]

On the other hand, I mean, I think for a lot of linguists it’s been a real eye-opener, or that they’re still kind of in denial as to how well these ngram language models work. Because the fact of the matter is, you know, there’s no explicit representation of grammar whatsoever, but simply if you’re learning up these ngram probabilities, you’re kind of inefficiently and badly, in some sense, because you’re spewing around billions of parameters, but you’re nevertheless, you’re learning all the grammar as you go along.

That, you know, you learn that after a form of the verb “to be,” you can get present participle forms of verbs - “I am singing -” and you can get passive participles - “I was hit -” that you don’t get other forms. Like you don’t get “I was make,” all right? You just pick up all of that stuff as ngrams. And so yeah, the thing to notice here is, well, actually, this is pretty grammatical stuff. And you should notice that too when you do the assignment, since one of the things in the assignment is to generate text.

And what you should find is by the time you have a decent trigram model that although the sentences will normally make not very much sense - and you know, quite a few of these don’t make very much sense, either. Some of them are okay, but some of them don’t make very much sense. They’ll nevertheless come out sort of grammatical.

I like a multi-agent approach; in my system, I include MegaHAL and ALICE and my own logic agents and humor agents and grammar agents, etc. Each agent responds with a score that increases if the agent program has a method to handle the input pattern. The agent response with the highest score is selected by a controller to return to the user. The user can use feedback to modify scores, so that if the user prefers another agent’s response, he can inhibit the agent the controller selected and reinforce an agent whose response he prefers.

Anyway, it’s interesting to see an academic mention the loebner prize, although he gets the prize details wrong, and completely glosses over the silliness of the fake typing requirement :)

The problem with statistical nlp models, for me, is that when they make a mistake, I want to correct them. I want to tell the program what mistake it made, and how to correct it, and why. But the way the academics have designed their programs, you have to modify a training text (possibly with annotations) and retrain. So I want to take their programs and wrap them with an online-learning agent. Maybe memorize corrections at runtime, then when the program has downtime, put the statistical NLP module to sleep, add the corrections to the training file and retrain…