I am looking for Turing test that would be easy to enter for new bots and that would be scheduled frequently, ideally by offering random visitors to participate as human control group in otherwise fully automated competition. Besides standard Turing test, it could feature other scoring mechanisms.

I’ve seen turinghub.com, but that one seems to have only 3 participating botmasters, which makes me wonder how hard it is to get in. Not to mention the usability issues and affiliations. I tried to contact them, but I don’t hold much hope for a reply since their contact form looks like a joke.

Of course I can develop such test myself and open it up for other bots, but I would much prefer submission of my bot to an existing test if there is any.

Robert Važan - Nov 26, 2011:

I am looking for Turing test that would be easy to enter for new bots and that would be scheduled frequently, ideally by offering random visitors to participate as human control group in otherwise fully automated competition. Besides standard Turing test, it could feature other scoring mechanisms.

I’ve seen turinghub.com, but that one seems to have only 3 participating botmasters, which makes me wonder how hard it is to get in. Not to mention the usability issues and affiliations. I tried to contact them, but I don’t hold much hope for a reply since their contact form looks like a joke.

Of course I can develop such test myself and open it up for other bots, but I would much prefer submission of my bot to an existing test if there is any.

I WOULD BE INTERESTED IN COLLABORATING ON YOUR PROJECT IN EXCHANGE FOR LIMITED RIGHTS.

Raymond Lavas - Nov 26, 2011:

I WOULD BE INTERESTED IN COLLABORATING ON YOUR PROJECT IN EXCHANGE FOR LIMITED RIGHTS.

I am not used to receiving business offers this way, by SHOUTING generic offers on public forum. Your profile is not serious. Your posts are not serious. Your website is not serious. You demonstrated no skills. Your offer is a joke. Perhaps you find it funny. I don’t. It has no point. I am quite busy and you just wasted my time.

Oh,... I beg your pardon sir. I’m deeply sorry to have offended you.

You are correct sir, and I am nothing but a fool…....

Again, sorry for having wasted your time looking me up….....

Have a good day anyhow, and I PROMIS that I will no longer bother you….

Raymond Lavas

Raymond Lavas - Nov 27, 2011:

Oh,... I beg your pardon sir. I’m deeply sorry to have offended you.

You are correct sir, and I am nothing but a fool…....

Again, sorry for having wasted your time looking me up….....

Have a good day anyhow, and I PROMIS that I will no longer bother you….

Raymond Lavas

Closed minds offer no substance other than the distortion of facts and collaboration is a sincere form of flattery for those wise enough to recognize it. 

Laura Patterson - Nov 27, 2011:

Closed minds offer no substance other than the distortion of facts and collaboration is a sincere form of flattery for those wise enough to recognize it. 

Oh, he’s right… I just typed that whole message so quickly I hit the send button before looking at it. I was stuck in CAP-lock ....

Anyhow, I thought he could perhaps certainly look at the table of content in my book before deciding to reject me.

The man is forgiven as far as I am concerned. I am too happy and at peace with myself to be bothered by it.
Raymond Lavas

Robert, are you familiar with the Loebner prize or Chatterbox Challenge? Each is held once a year.

This is a really excellent idea which I had considered some time ago. Such a server could seriously help the community, for several reasons.  Competitions that happen once a year can serve as flagship events that gather coverage and force a deadline on developers.  However, they don’t generate much useful data (at least in terms of quantity).  A server that randomly paired bots and humans could be online continuously.  For each human to be able to face both a bot and a human (Loebner-style) there would need to be another human online at the same time, so it might be better to schedule rounds once a week at the beginning, so people know they have more chance to participate.
Second, this replicates closely the environment of the Loebner prize, which is a better way of testing than putting your bot online elsewhere.  In this situation, the bots will have conversations with humans that are designed to tell them apart from humans.  The nature of the human/human conversations will be the same, which could generate useful data for “conversation mining”.
Third, the feedback from the humans could be structured to include more than a simple win/lose.  For example, when the human says “that’s the bot”, it could be asked for the line in the conversation that was the first / biggest giveaway.
Finally, this could allow anyone to have the Loebner / Turing test experience, so broadening people’s understanding of the concept and the state of the art in bots.
Designing a website to do this is beyond my capabilities, but I guess a chatroom website would provide the basic capabilities and the bots could connect using a standard chat protocol.  I would suggest not using the Loebner real-time character by character mechanism as this is a significant programming overhead.
A question to the others who have mature / well-developed bots: would you connect to such a server? What would be your choice of protocols?  Would you be comfortable with all transcripts being “open-sourced” or would you require your conversations to be private to you?

@Robert: We had a discussion a while ago about several new approaches to chatbot (con)testing:
http://www.chatbots.org/ai_zone/viewthread/140/

What you’re basically saying is:
-I’m going to a website
-I’m seeing a chatbox
-I start to chat
-I enjoy the conversation
-When would I become suspicious? Or is it a human anyway? I’ll stay in confusion forever.

Is that what you’re saying?

My suggestion works like this. 

- You go to a website
- You see 2 chat boxes
- You click start and have 2 chats in parallel
- Whenever you like, you call out which is the chatbot and you’re told if you were right / wrong
- You continue to chat with the other human until he has made his decision (unless he had already done so)
- You give feedback on the bot’s performance

As noted above, this requires another human to be available, so there may be a delay after clicking start.

I would love a service like this to be available for testing chatbots. I believe also a small fee could be charged and would be accepted by most for this service.

OliverL - Nov 28, 2011:

My suggestion works like this. 

- You go to a website
- You see 2 chat boxes
- You click start and have 2 chats in parallel
- Whenever you like, you call out which is the chatbot and you’re told if you were right / wrong
- You continue to chat with the other human until he has made his decision (unless he had already done so)
- You give feedback on the bot’s performance

As noted above, this requires another human to be available, so there may be a delay after clicking start.

what an absolutely brilliant Idea, good thinking OliverL!!!!

what about not knowing whether it is a human or a chatbot anyway? So you could also endup with two humans, or with two chatbots.

Another suggestion: what about limiting the subject. Only talk about cars? Or about pop music? Or growing food? It would be easier for chatbots to pass a test in a certain domain, and the developer can continue from there.

Erwin Van Lun - Nov 29, 2011:

what about not knowing whether it is a human or a chatbot anyway? So you could also endup with two humans, or with two chatbots.

Another suggestion: what about limiting the subject. Only talk about cars? Or about pop music? Or growing food? It would be easier for chatbots to pass a test in a certain domain, and the developer can continue from there.

That my dear Erwin is happening already, but how on Earth will you or anyone else ever be able to tell the difference? just try limiting your subject matter to “Quantum Chromodynamic Fractals”

Raymond Lavas

I’ve got news from turinghub.com. I found an email address of Robby Garner who runs the website. We’ve agreed to integrate my chatterbot Teo on turinghub.com sometime during December.

My advice to botmasters: Join turinghub.com. There’s no alternative. Skip the feedback form on turinghub.com and mail directly to Robby Garner. His address is on his website linked from his profile on chatbots.org. The interface is extremely simple - your chat form along with voting link is embedded via HTML frames on a test page on turinghub.com. It’s not a Turing test though since there’s no control group of real humans. It’s just a voting system for rating chatterbots. Nevertheless, it’s still a good preparation before joining CBC or Loebner Prize.

For anyone considering creating alternatives to turinghub.com: You are welcome, but please keep the service free of charge and with as few ads as possible. You can significantly improve things like botmaster recruiting model, organized test rounds, reports/charts, webdesign/usability, and marketing of the website. You can get much better results by integrating with existing chat sites or social networks that already have plenty of visitors.