Great blog post on the LP Contest:

http://sciencesamosa.wordpress.com/2012/05/31/yakety-yak-dont-talk-back-the-2012-loebner-prize-at-bletchley-park/

Interesting article, Rich, but please please PLEASE tell me you didn’t really say DOCTOR Spock!

Quoting from the linked blog:

—-
In many ways, however, Turing’s thought experiment was a much more stringent form of the “Duck Test” used by Loebner. Turing’s original test did not stipulate a time limit, less still a number of judges that needed to be fooled. The test could continue in open-ended, freewheeling discussion indefinitely, presumably getting tougher for the machine all the while.
—-

However, going back to Turing’s “Computing Machinery and Intelligence” ( http://cogprints.org/499/1/turing.html ):

—-
I believe that in about fifty years’ time it will be possible, to programme computers, with a storage capacity of about 109 [sic, should this be 10^9?], to make them play the imitation game so well that an average interrogator will not have more than 70 per cent chance of making the right identification after five minutes of questioning.
—-

So Turing’s test was less stringent than Loebner’s, since Turing proposed a five-minute session, and a 70% threshold for correct identification.

Also, Loebner’s insistence on a character-by-character protocol violates the spirit of Turing’s test. Quoting Turing’s paper again:

—-
In order that tones of voice may not help the interrogator the answers should be written, or better still, typewritten. The ideal arrangement is to have a teleprinter communicating between the two rooms. Alternatively the question and answers can be repeated by an intermediary.
—-

Note that “teleprinter” does not necessarily mean character-by-character protocol. Quoting wikipedia ( http://en.wikipedia.org/wiki/Teleprinter ):

—-
In 1925 Creed acquired the patents for Donald Murray’s Murray code, a rationalised Baudot code, and it was used for their new Model 3 Tape Teleprinter of 1927. This machine printed received messages directly on to gummed paper tape at a rate of 65 words per minute
—-

Thus, the typing speed of the operator and the typing speed that the receiver saw were essentially disconnected, since the printing occurred at a fixed rate.

Also, the same wikipedia article on Teleprinters says:

—-
Exchange systems such as Telex and TWX. These created a real-time circuit between two machines, so that anything typed on one machine appeared at the other end immediately. US and UK systems had actual telephone dials; German systems did “dialing” via the keyboard. Typed “chat” was possible, but because billing was by connect time, it was common to prepare messages on paper tape and transmit them without pauses for typing.
—-

So once again, in typed chats of Turing’s time using teleprinters, the typing characteristics of the sender were essentially eliminated by the teleprinter. The modern equivalent is the message-by-message protocol, which has become so popular that no one (except Loebner) uses character-by-character anymore.

[Message-by-message protocol is better because reading is faster than writing, so when you’re on the receiving end of a character-by-character message it feels silly to have to wait for the sender to find the right keys. Message-by-message also allows for editing before you hit “enter”, so you have more control over your communication. It’s interesting how in the online ai-class and Thrun’s robotic car class at udacity.com, most of the mechanical writing was edited out, so that the viewers didn’t have to sit through the slow process of handwriting. The main point was that the content was the important part, not the process by which the letters that conveyed the content were written onto the screen.]

Turing’s point was clearly to eliminate any way of identifying intelligence by physical characteristics; Loebner’s character-by-character protocol is plainly an attempt to re-introduce physical characteristics.

Quoting Turing’s paper again:

—-
The new problem has the advantage of drawing a fairly sharp line between the physical and the intellectual capacities of a man. No engineer or chemist claims to be able to produce a material which is indistinguishable from the human skin. It is possible that at some time this might be done, but even supposing this invention available we should feel there was little point in trying to make a “thinking machine” more human by dressing it up in such artificial flesh. The form in which we have set the problem reflects this fact in the condition which prevents the interrogator from seeing or touching the other competitors, or hearing -their [sic, shouldn’t the “-” be deleted?] voices.
—-

Loebner’s character-by-character protocol misses Turing’s main point, which is that factors such as typing speed or accuracy are irrelevant to Artificial Intelligence.

Robert, I couldn’t agree more. All other chatbot competitions use a “message-by-message” format, which is less technically complicated, and is much easier for both judges and confederates to handle, since there’s no tedious waiting for a message to “emerge” from a string of incoming characters. The one downside to using a message-by-message protocol is in the delay time “gap” between chatbots and humans. Most chatbots can deliver a response almost immediately, no matter how long the response text is, whereas Humans would still have the delay involved with them formulating their reply, and then typing it out before hitting the “send” button. Of course, this is easily overcome with a little programming “common sense”, by checking the size of the bot’s response, and calculating the appropriate delay required for a Human to deliver the same, and then adding a small, random “fudge factor” before sending the response out after the proper wait time has passed. In fact, the software can use a similar (or even identical) time delay function for the human responses, to “normalize” the time delay, and possibly even “match” the delays between Human and chatbot, to eliminate (or come close to eliminating) even this distinction between the two “contestants”.

Just a naive comment. This did happen to me while chatting with real persons.
They where good at chatting in English while me being poor at finding the right words
fast enough and me slow at typicng having to correct my spelling again and again
them lost interest in me being too slow to respond to their “flirty” remarks.

They asked me if I where drinking coffee or something stronger and not paying attention
They felt neglected if my response where too slow. A kind of opposite to what a chat bot
would do. Chat Bot doing almost Instant response with a lot of texts that is spelled correctly.

Edx’s CS188.1x “Artificial Intelligence” started today:
https://www.edx.org/courses/BerkeleyX/CS188.1x/2012_Fall/about

Lecture 1, Segment 3 “What is AI?” reminded me of this thread, when Dan Klein says:

The problem was, the Turing test, in order to really do well, you don’t just really concentrate on programming intelligence, you concentrate on things like, don’t spell too well, humans don’t do that. And so you build in some type of typo Turing machines and then you think, wait a minute, if I get asked about the square root thirty-five, I better not have an answer. And so you go through basically trying to mimic things that probably you didn’t really value in the human in the first place.

But maybe the human has autistic savant calculation abilities, or maybe he has a calculator right there and is very fast with it, or maybe even he has a screen scraper that quickly detects math questions and answers it for him; so you don’t necessarily have to dumb down the computer’s responses. But Loebner’s protocol requires it; I wish he’d abandon it. (To me it’s obvious that he has some sort of emotional attachment to his protocol because he wrote it, and he uses it to “weed out the riff-raff”; but this is all social intelligence and a distraction from the real goal of artificial intelligence, I think.)