Dear Chatbots and Loebner Prize community,

What follows is an unofficial, informal, non-binding, announcement about the current plans for this year’s AISB chatbot event. I hope to confirm all the details and update the AISB website this month. This post therefore allows you to be prepared for the official announcement, but also allows you to comment and have some effect on any decisions made, where possible.

From now on the Loebner Prize will take on a new form, as fixture of an annual exhibition exploring the issues that interested Turing, Loebner, and the Chatbot Community. Importantly, we will completely relax the requirement for any submitted chatbot to perform in competitive rounds as per the Loebner Prize. The main impact is that we are now able to accept chatbots which are specialised to a single task as well as the free-form conversation bots previously accepted.

The rest of the exhibition will include submissions, and some commissions if possible, from collaborations between artists, roboticists and AI scientists, and will reference any of the issues raised when making a machine that performs as a human.

The exhibition this year will likely be over Thursday-Sunday in September. We have secured some funding, and have a dedicated exhibition space in the University of Swansea.

There will still be prizes, they will still be awarded by a panel of judges, but this will be on a more personal, situational basis. Personally, and I think I speak for most people involved, I’d rather distribute funds to more people in the form of travel awards to have more chatbotters at one event, than to have larger chunks of money going to fewer people. In this way the event will be more of a collaborative celebratory event, and less of a stressful competitive event.

There is nothing stopping exhibitors from agreeing on some kind of interaction between their chatbots, and even defining some kind of competition of their own, so anyone who wants their bot to face a classic Loebner Prize style challenge may still do so, we will even offer some assistance setting up and finding confederates, but can’t guarantee it will run smoothly (not that it always did) and won’t offer any prize money or medals to any winners.

Unfortunately, the Loebner Prize in its traditional state was unsustainable, the location of Bletchley Park, though historically ideal, was a poor venue for public exposure, making it impossible to secure sponsorship, which became an essential requirement. Also, I have heard a number of complaints from the community about the restrictions and specifications of the Loebner Prize and the LPP, so this is a way to respond to those issues. Finally, I feel there was growing enthusiasm from the organisers and participants to have a bit more of a dynamic event, rather than a recurring meeting of the same few entrants and technologies. I hope everyone welcomes the increased eligibility and, with any luck, the increase in number of people participating in the event.

Please let me know what you think, while we are still drafting the official announcement. I’m a bit frantic this weekend so I may have missed points out, but I’ll try to be responsive on this thread.

Cheers,

Andrew

Thanks Andrew, I think this is positive news. I would consider submitting an exhibit to something like what you describe.

I believe you may attract more commercial entries if they are allowed to submit their product offering or ideas in an open environment.

I think more people would be able to experience what it is like to judge a Turing test, or to be the confederate. They could have a chat for 5 minutes and then trade places for example, just for fun. It could be more educational than previous years’ contests that were restricted to only 4 finalists. That rule was fine when Hugh held the contest in his apartment, but if you want to attract sponsors, entertainment would be conducive to that.

Best,
Robby.

It sounds like a fun event and it’s nice that a broader range of submissions will be possible. I’d also say the exhibition needs a more catchy name to go with it.

Despite all the criticisms I’ve had on the Turing Test, the one thing it did well was add a gaming element, that I think made the event more noteworthy and alluring than a static exhibition. I’m not exactly in a position to organise such a thing, but next to allowing visitors to freely interact with all exhibited chatbots of all natures, I would still like there to be a competition sub-event with title(s) as reward, if not a prize. It could certainly boost publicity.

For that competition I would suggest to step away from the “human-or-not” criterion and go with e.g. “most natural conversation” (previously called “most human-like”). To make this easier to organise I would do away with confederates, networks and interface protocols altogether, and just have judges interact with each chatbot on the computer that they are installed on, changing seats and jotting down scores. Given that no bot has yet shown to be close to going un-ousted for more than five questions, one might as well remove the curtains and all the issues that came with them.

While it would not be a traditional Turing Test anymore, it would still serve to illustrate Alan Turing’s point: If a program can answer as well as a human, shouldn’t we consider it “thinking”? It would continue to serve as a finger on the pulse of progress in this area, but without the necessity to oust bots with unnatural questions or deliberate typos. If and when chatbots develop to a level where the interactions in this exhibition become convincingly natural, then you could conclude that the time is rife for the traditional Turing Test to be held again. This sub-event we could still call the Loebner Prize, if people agree.

Would comic pages and digital artwork about human-robot relations be suitable exhibits? I have connections in the comic artist world and know some that may have relevant quality material. What is usually arranged in such cases is that the artists send in pdfs, someone locally prints them on A3 size and hangs them in borrowed frames.

Hi Don,

Turing described the Imitation Game in enough detail that we could come as close as possible in the Loebner Prize Contest, but you have to study the history of the proposition to understand what he was trying to demonstrate. Dr. Loebner came very close to a real Tt in the 1990’s. People don’t seem to understand what the point of it was. The more recent LPC’s were ruined by the 20 questions to select the finalists, and the selection of experts and journalists as judges.

Now, I believe the spirit of Turing’s test was for ordinary people to sit down and try to decide, which is man? which is woman? (or which is machine). The Royal Society test in 2014 is the closest time in my memory where a large enough pool of normal judges took part in a fair Turing test. Period. My program, JFRED, scored over 20% deception and a conversation quality rating of the mid 30’s percent.

My program Albert fooled two judges in 1998, and one in 1999. Now, I ask you this: Did JFRED improve or did the judge selection more closely represent the normal curve in human beings? Your comment about “no one has come close to going un-ousted for more than five questions” is inaccurate. Disambiguate that you mean the latest contest entrants and the LPP.

Some of you seem enamored by the LPP and I don’t understand why that is. I think it was a huge mistake in the contest. David Hamill got it right in 2003 by admitting web based bots and message-by-message interactions rather than fake typing as yet another obstacle to scientific progress. Making the program successively harder and harder, and introducing those ridiculous questions to determine the finalists was Hugh’s fancy,  but not mine. Dr. Powers at Flinders University in Perth, understood the basic concept, and designed the interfaces and protocol that made sense with our chat bots from that time.

When I hear people say, “let’s have a modified Turing test” I am face palmed. If you can’t beat my 20%, you haven’t progressed very far with your entry, or with the contest itself. Paco Nathan and I got 18% in the BBC MegaLab experiment in 1997. “Eugene” got 33% in 2014 and that really happened.

But again, it comes down to interface: should be like text messaging on your phone, and Judge selection should be more representative of the population. Same with confederates.

Best,

Robby.

Looking forward to participating in any event.

Dan

The question on everyone’s mind… Will the event allow web-based chatbots?

I have 4 strong reserves regarding online access.

First, I actually believe that participating to an AI contest with an online access allowed for the bot, is like participating to a human official evaluation along with a cheatsheet. There is no end to the “limitations of your embarked AI” that you can circumvent with any kind of people on the other end.

Secondly, having an online database instead of an embarked one is really not the same difficulty. Embarking an AI can be one of our progress and scientific “strong points”.

Thirdly, working without internet/wifi/you name it means that you HAVE an autonomous AI. Which means an AI capable of differentiating itself from other AIs because of the individual experience it has, instead of the mass/common/“we are all the same” online AI.

And most importantly, forcing to have an embarked AI puts you in all these real issues of understanding how the human mind can make “shortcuts” in social behaviour. I always thought that the purpose of the test was not to have the most intelligent or accurate answer, but having the most human answer.

I also remember a sentence that Turing said, that by trying to reproduce an AI capable of being mistaken for a human, we would have to understand how a human thinks, and that progress will be the real useful progress. Instead, having an online access falls into the route of “finding the most accurate/perfect answer”.

Sorry if that sounds as a disagreement about this question; these 4 reserves I have are strong points for me. And I believe they are scientifically valid as well.

I hope this question won’t be divisive, and don’t want to hurt any feelings with my reply.

About the original post, I loved the previous implementation of the Loebner Prize. Although I could not participate in later years, it really marked my mind positively.

I stopped participating because my kind of AI was unskilled at passing the applicant 20 questions. “Taught AIs” with accurate answers behaved better, and they were much more “matured”, and I could not deliver such progress yet on my AI, and felt so sorry about it.

However, the place was felt as an important place and there was consistency.

I would love the contest to keep this consistency.

But I will understand and support any change to the formula. The staff knows about the problems and every year feedback. I love the idea of travel awards. I really trust the staff and the most experienced of our contestants to put us in the right direction.

Note that the exhibition is scheduled to last 4 days. There is no reason why there can’t be a competitive event on one day, as well as a 4-day merry-go-round of visitors having Turing tests with online chatbots of all sorts. I’ve no taste for bickering.

No worries. I don’t feel strongly about it at all. If I’m not mistaken, David Hamill allowed web-based systems, but didn’t allow Internet access. They had to run on a local web server at the contest. Entrants were invited to install their own software on computers that University of Surrey provided. Some may have used their own computer systems. Or, you could just mail it in with some instructions. Incidentally, the web based entries had an advantage of sorts because they all looked identical in color, shape, fonts, etc. That was a long time ago. Maybe things have changed. IDK.

I see what you mean Robby!
I did not think about it in my reply.

I am all for an encapsulated version (for the contest version) which could allow local web, and it effectively tramples any reserve I could have

I think it’s a great idea to open it up to different categories. It gives it a kind of Crufts Dog Show feel with a winner announced in each category. What I would really love to see though is, just like Crufts, to have an overall winner as well as individual winners.

The Loebner Prize is the only chatbot contest I can think of, either real world or virtual, that is open to anyone and it’s always been great publicity for the winner to say they have that year’s best chatbot. It would be a shame to see that part of it be removed.

Having the different winners also opens up business opportunities and publicity to many people instead of just the one winner. “Hey, I have the best interface/avatar etc. Want me to do some work for you?”

My original thoughts on the future of the Loebner Prize are on this thread.
https://www.chatbots.org/ai_zone/viewthread/3453/#26487

I will support the contest/event in whatever way I can and am looking forward to it already!

When attendance grows so much as to add turnstiles outside the venue, and can only let 10-20 people in at a time, I hope the contest organizers would consider ordering the velvet ropes and brass fittings and stands from Crown Industries.

Nice idea Robby.

And of course we should order the portable disco floor for the celebrations afterwards from them as well.

LOL !

Just my two cents. The loebner awards were my inspiration to build my chatbot from the very beginning and I’d be very disappointed if at least a similar type of contest wouldn’t be held anymore. 

I’d also like to add that there are a lot of chatbot and ai expos or conventions. Lots of events that already have sponsors and industry there. But at the same time chatbot competitions especially ones that allow individuals to enter seem to be slowly disappearing. It sounds like the loebner competition is moving toward another expo/event just like so many others and I’m not sure that’s a very interesting way to go.

Also, what will happen to Loebner’s solid gold medal. Nobody will win it ever? That would be a real shame. I’ll always regret I never met Hugh Loebner in person but from what I’ve read I don’t think he would have wanted his contest abandoned or transformed into just another industry technology expo.

There’s a lot of angles to go in order to make the contest more economically feasible while still maintaining the basic contest as it existed. My vote would be to keep the competition similar to as it is but add additional events (competitions or not) and move it somewhere (maybe here in Las Vegas? ) where you can get plenty of visitors and sponsors to help fund it.

I have to point out that if we add Steve’s disco and velvet ropes it sounds like it should be here

And the award for best chatbot cyber-blackjack dealer and slot machine goes to . . .