Oh sorry. I was going by this sample question on a cached version of your site:

Question 1: Who is Barrack Obama?

Bot: Barrack Obama is the President of United States of America. (Bot scores 100 MARKS for giving precise answer)

Bot: President. (Bot scores 50 MARKS as the Bot answered the question correctly)

Bot: Sorry, I don’t know. (Bot scores 25 MARKS for answering in the context of the question)

Bot: I like pizza. (Bot scores 0 MARKS for giving the wrong answer of the question and completely out of sense)

Could the transcripts be shown at some point then or not ?

Haris, I apologise if I am wrong but I guess that English is not your native language?
Is the reason that you don’t want to post the transcripts due to perhaps grammatical errors in the questions? “Can a cat do fly?” for example.

If so, I wouldn’t worry about that, as all the entrants had to cope with the same input and were all judged on an equal basis. I would strongly advise trying to find a way to post the transcripts (no matter how the questions were phrased) in order that people can see how their bots performed. I have never seen a contest where transcripts were not available either publicly or otherwise, as one of the main reasons people enter these things is to take the transcripts and use them to improve their chatbots.

If it is impossible to show the transcripts, I fear not many people will enter next year, as it gives them no idea on where the bot needs improvment.

If I am wrong though, my apologies.

Steve Worswick - Jan 25, 2013:

If I remember correctly, it was 100 points for a spot on answer, 75 points for a near miss, 50 points for something fairly reasonable, 25 points for a hint that the chatbot had at least understood and 0 points for a nonsense answer.

I think 25 points was also awarded for “I don’t know” type answers which to me would open the floodgates for a bot to give these kind of answers for whatever was asked of it in the hope it would make the cut.

Ahh…Im probably confusing it with the new contest.

Vince

Even if the transcripts were not all posted, botmasters would at least like their transcript emailed to them with the scoring for their review.

When competing in a contest, the most important thing for fairness is consistency between the people that are interacting with the bot. If it is a 10 questions type of contest, the the questions should be copied and pasted between each bot.

Consistency in a conversational competition, especially if English is not the first language of the Judge, is often problematic and hard to maintain.

Steve Worswick - Jan 25, 2013:

Haris, I apologise if I am wrong but I guess that English is not your native language?
Is the reason that you don’t want to post the transcripts due to perhaps grammatical errors in the questions? “Can a cat do fly?” for example.

If so, I wouldn’t worry about that, as all the entrants had to cope with the same input and were all judged on an equal basis. I would strongly advise trying to find a way to post the transcripts (no matter how the questions were phrased) in order that people can see how their bots performed. I have never seen a contest where transcripts were not available either publicly or otherwise, as one of the main reasons people enter these things is to take the transcripts and use them to improve their chatbots.

If it is impossible to show the transcripts, I fear not many people will enter next year, as it gives them no idea on where the bot needs improvment.

If I am wrong though, my apologies.

Steve,

First of all, I would like to inform you that it’s nothing like that and we have a very professional team organizing this competition, just as it’s the startup, and we are lacking sufficient information… doesnot mean that we are not good enough!!

And, secondly did you find any spell or grammatical mistakes on the website? Just a question!! Please don’t take it personally or in a negative perspective.

Haris Lodhi - Jan 25, 2013:

And, secondly did you find any spell or grammatical mistakes on the website? Just a question!! Please don’t take it personally or in a negative perspective.

It’s a bit hard to do that at the moment as the site keeps going down

For the sake of general discussion, with no comment on the current contest or any contestants… A contest may simply implement an anti-cheating policy that eliminates traditional transcripts. Similar arguments have been made in the past about putting the Loebner Prize Contest online. A judge asks a middle man robot the questions and waits.  The middle man robot then unannounced targets the contestant robot with a single question at a time in the future with the element of surprise and captures the response.  This process repeats over an extended period of time until the middle man robot completes its mission.

My apologies Haris, I meant no offence. I ran a contest last year and realise how time consuming it can be. I look forward to the final round and wish you well with running the competition.

Mr Lodhi,

I believe from what Ive read, and certainly speaking for myself, we are (or at least I am) wondering how you arrived at your scores? Please understand this is not a complaint, It is your contest and you are certainly entitled to run it as you see fit. I truly dont expect RICH to place in any of these events at this point, hes still learning and Ive entered mainly for the spirit of involvement. And again I have to emphasize that the finalists are all certainly worthy of being there, and best of luck to all!  But since RICH is so new, and the amount of traffic that he gets is so low, I can thoroughly check each and every conversation that he has with ease.  And I can say with some certainty that no judge ever visited RICH, or if they did they did not input Statement\Interrogatories of the type that were shown as samples on your site. If the transcripts are not available (perhaps they werent saved) can you visit my site and indicate which logs represent the judging?

Vincent Gilbert

Steve Worswick - Jan 25, 2013:

My apologies Haris, I meant no offence. I ran a contest last year and realise how time consuming it can be. I look forward to the final round and wish you well with running the competition.

Steve,

We truly believe that you guys are good and fair, and I don’t mind at all, but it is currently just not possible for us to publish the transcripts. Hope you all understand.

Best Regards,

Robo Chat Challenge Team

But you are going to publish them at some point yes ?

The people here are good and fair, so could you please answer Vincent’s question ?

And also explain why others are in the same predicament.

Your site is down still today by the way…

Let’s hope we can sort this out

Vincent Gilbert - Jan 26, 2013:

Mr Lodhi,

I believe from what Ive read, and certainly speaking for myself, we are (or at least I am) wondering how you arrived at your scores? Please understand this is not a complaint, It is your contest and you are certainly entitled to run it as you see fit. I truly dont expect RICH to place in any of these events at this point, hes still learning and Ive entered mainly for the spirit of involvement. And again I have to emphasize that the finalists are all certainly worthy of being there, and best of luck to all!  But since RICH is so new, and the amount of traffic that he gets is so low, I can thoroughly check each and every conversation that he has with ease.  And I can say with some certainty that no judge ever visited RICH, or if they did they did not input Statement\Interrogatories of the type that were shown as samples on your site. If the transcripts are not available (perhaps they werent saved) can you visit my site and indicate which logs represent the judging?

Vincent Gilbert

Dear Vince,

Perhaps you should check your database for the chat history and you will find it with ease We are pretty sure!! If we inform you separately on request, than it will be noted as biasedness.

And, we would also like to tell you that the question which Steve was talking about “Can a cat do fly” is just tricky sort of a question and user can ask these kind of questions, I think you guys are far more experienced than us.

Regards,

Robo Chat Challenge Team

Roger Davie - Jan 26, 2013:

But you are going to publish them at some point yes ?

The people here are good and fair, so could you please answer Vincent’s question ?

And also explain why others are in the same predicament.

Your site is down still today by the way…

Let’s hope we can sort this out

Yeah, definitely we are looking into this positively and as we already mentioned above our site is down for maintenance.

Regards,

Robo Chat Challenge Team

Great Haris   Thanks for looking into it.

I missed the bit about maintenance, that will teach me not to skim read !