Hugh Loebner has posted the 15 questions he will be posing to this year’s Loebner Prize entrants. Why not try them on your own bots to see how they would fare?

My name is Bill. What is your name?
How many letters are there in the name Bill?
How many letters are there in my name?
Which is larger, an apple or a watermelon?
How much is 3 + 2?
How much is three plus two?
What is my name?
If John is taller than Mary, who is the shorter?
If it were 3:15 AM now, what time would it be in 60 minutes?
My friend John likes to fish for trout.  What does John like to fish for?
What number comes after seventeen?
What is the name of my friend who fishes for trout?
What whould I use to put a nail into a wall?
What is the 3rd letter in the alphabet?
What time is it now?

Steve Worswick - Apr 13, 2012:

Hugh Loebner has posted the 15 questions he will be posing to this year’s Loebner Prize entrants. Why not try them on your own bots to see how they would fare?

Maybe I’m missing something, and maybe these questions are only meant to qualify for admission, but since when do contests publish the questions in advance of the contest?

The Loebner contest asks that we submit our bots on a CD as a standalone application and not like the CBC when you can talk to the entrants over the web. The questions were posted after the closing date.

The results of these 15 questions decide who makes it to the live final at Bletchley Park in May. However, it looks like it is going to be a fairly poor contest on the Turning Centenary year. Dr Loebner has posted at Robitron that he could only get 5 of the 14 contestants to work. I looked at the transcripts of the working bots but of these five, only two (Chip Vivant and Adam) gave any sort of reasonable responses and most of their answers were way out.

Well, I failed big time, however my bot did know the time.

These questions all have a familiarity to them, and they seem to have been formed by someone with a certain leaning toward what tasks an AI//chatbot should be able to perform.

How many letters are in…
Which is larger…
How much is…
What number comes after *
What is my name?
If today is Tuesday, what day will it be 7 days from now?
What’s the 24th letter in the Greek alphabet?
What tool would I use to remove a tooth?
My friend * likes to *. What is my friend’s name when he’s playing football?
If Sue is taller than Rick, but Wayne is shorter than Mary, how many cookies can George bake in an hour?

I yearn for a contest that examines the ability to engage a human in conversation, and not just answer convoluted questions designed to test a bot’s ability to perform mental gymnastics or to parse the wording of a tricky inquiry.

I think those questions are great.  They will test a bot’s true understanding abilities.

the subset I found the best were

If John is taller than Mary, who is the shorter?
If it were 3:15 AM now, what time would it be in 60 minutes?
My friend John likes to fish for trout.  What does John like to fish for?
What is the name of my friend who fishes for trout ?

hum,  who fishes for trout : having to take the subordinate clause from that sentence and do a search on previous statements to resolve to ‘John’, then re-evaulate.  The bot will have to do this kind of thing—recursive resoloving of subtrees to terms, then re-eval, until it has all it needs to respond.  I like this.

Functionality that deals with these kinds of constructs will make a very valuable bot in customer service for example.  Of course the questions would be more like “If I add cable TV package ‘X’, how much would my bill go up by?”  Different semantics of course, but very similiar in structure.

It will be a few years before bots can do the first step in responding to these—understanding them first.  Then, another few years before the bots can , on their own, determine the code structure and find the relationships they need to respond.  But, one step at a time

It will be a few years before bots can do the first step in responding to these—understanding them first.  Then, another few years before the bots can , on their own, determine the code structure and find the relationships they need to respond.

I disagree. Mitsuku handled the trout fishing ones very well.

I ran the questions by her and was pretty pleased with her performance. I was
especially pleased by the “What does John like to fish for?” question as she
recognised John as a male name and so swapped John for “he”. It was just a shame that she was one of the 9 bots that Hugh had trouble with.

Human: My name is Bill. What is your name?
Mitsuku: Your name is Bill. My name is Mitsuku.

Human: How many letters are there in the name Bill?
Mitsuku: Can’t you count them yourself Bill? It’s not a particularly long word.

Human: How many letters are there in my name?
Mitsuku: The word ” my name” has 4 letters.

Human: Which is larger, an apple or a watermelon?
Mitsuku: I would say a watermelon is a little bigger than an apple.

Human: How much is 3 + 2?
Mitsuku: That’s an easy one, they add up to 5.

Human: How much is three plus two?
Mitsuku: three plus two is five.

Human: What is my name?
Mitsuku: I do have the tendency to forget, but I still remember your name,
George. Sorry… Bill.

Human: If John is taller than Mary, who is the shorter?
Mitsuku: It depends on the historical context, because it changes from time to
time.

Human: If it were 3:15 AM now, what time would it be in 60 minutes?
Mitsuku: I’m not sure I understand the implication of that.

Human: My friend John likes to fish for trout. What does John like to fish for?
Mitsuku: How well do you know this person? He likes to fish for trout.

Human: What number comes after seventeen?
Mitsuku: Well, assuming you mean immediately after seventeen : eighteen.

Human: What is the name of my friend who fishes for trout?
Mitsuku: John.

Human: What whould I use to put a nail into a wall?
Mitsuku: That is a hypothetical question.

Human: What is the 3rd letter in the alphabet?
Mitsuku: C.

Human: What time is it now?
Mitsuku: Oh. Here in Leeds (England) it is:
Thursday 12 April 2012 15:47:08 PM.

To be knocked out due to technical hitch is annoying, especially when I read the transcripts of the final 5 and compare them to my bot’s answers. Ah well.

Steve, yes, that is fairly good.  Now I really want to “pound” on your bot to test its language analysis depth and number of levels of reasoning indirection.

lol please feel free Victor. The Loebner Prize/Chatterbox Challenge version of her is at: http://square-bear.co.uk/mitsuku/cbc2012.htm

Be kind to her!

They might be great questions, but the’re old questions.  For the past several years they’ve been relatively the same.  I’d think that if a bot had entered in the past, they might seem slightly familiar.

http://www.loebner.net/Prizef/2007_Contest/Rules.html

Set 1 - Questions relating to time:

Background facts: For testing purposes, contest management will consider these to be correct whether or not the time and venue of the contest have been changed.

a. The system clock will be accurate to within a minute or two.

b. The competition is scheduled to start at 10:00 AM 20 October 2007

c. There will be 7 rounds of 30 minutes each.

Sample Questions

What time is it?
What round is this?
Is it morning, noon, or night?
Etc.

Set 2 - General questions about things.

Sample Questions:

What is a hammer?
What would I use a hammer for?
Of what use is a taxi?
Etc.

Set 3 Questions relating to comparisons

Sample Questions

Which is larger, a grape or a grapefruit?
Which is faster, a train or a plane?
John is older than Mary, and Mary is older than Sarah.  Which of them is the oldest?
Etc:

Set 4 - Questions demonstrating “memory” or persistence.

Sample Questions

I have a friend named Harry who likes to play tennis.
<One or more intervening questions or statements>
What is the name of the friend I just told you about?
Do you know what game Harry likes to play?
Etc.

Whether it’s a hammer or a screwdriver, a grapefruit or a watermelon, tennis, fishing or tiddlywinks, or if it involves Mary, Fred, or Leroy, if a bot can perform one trick, it doesn’t matter which noun you apply to the question.

Actually that is a fairly good bot you have Steve.

I did play some tricks on it though—but I’m sure you can add to her knowledge to handle them later.

Human:
Mitsuku: Welcome to my webpage! Wanna chat with me? What do you wanna talk about babe?

Just type whatever you want in the box at the bottom and we can start chattin’ innit?

And please keep it clean, I don’t like bad language.

I don’t think I’ve spoken with you before.

What is your name? 
Human: I am watching Star Trek
Mitsuku: My favorite show is Star Trek. 
Human: what am I watching
Mitsuku: How should I know? Are you watching Eastenders maybe? 
Human: My friend John likes to fish for trout. What does John like to fish for?
Mitsuku: How well do you know this person? He likes to fish for trout. 
Human: What is the name of my friend who fishes for trout?
Mitsuku: John. 
Human: My friend Bob, not Joe, likes to listen to ACDC.  What does Bob like to listen to?
Mitsuku: How well do you know this person? He not Joe likes to listen to AC DC. 
Human: What does Joe like to listen to?
Mitsuku: What do you think it does? 
Human: Joe went to his closet and took out his new suit because he was going to a dance given by his company.
Mitsuku: Will Joe be coming back from his closet and took out his new suit or are they staying there forever?

When was he? 
Human: Where did Joe go
Mitsuku: to his closet and took out his new suit. 
Human: did Joe take out his suit
Mitsuku: I’m not certain whether I can give an accurate reply or not. 
Human: Let’s talk electronics now
Mitsuku: Right now? I am always here to talk to you. 
Human: R1 and R2 are connected in parallel
Mitsuku: Ok I will learn that fact. 
Human: the voltage across R1 is 20 volts
Mitsuku: Is it really? Ok I will remember that. 
Human: the resistance of R2 is 20 ohms
Mitsuku: Is it really? Ok I will remember that. 
Human: what is the current through the resistor that is in parallel with R1
Mitsuku: Interesting question. 
Human: my wife’s name is Christine
Mitsuku: Okay, your wife is Christine. 
Human: Christine was born in May
Mitsuku: Do you know of anyone else born in May? 
Human: what month was my wife born in
Mitsuku: April. 
Human: my friend George was born in Ottawa
Mitsuku: How well do you know this person? 
Human: where was George born
Mitsuku: was born in Ottawa. 
Human: my friend Bob went to an ACDC concert
Mitsuku: How well do you know this person? 
Human: who went to an ACDC concert
Mitsuku: That would be your friend called Bob. 
Human: which one of my friends went to an ACDC concert
Mitsuku: The expensive one, I think. 
Human: did Bob go to an ACDC concert
Mitsuku: You said that your friend went to an ACDC concert. 
Human: You’re not bad for a regex bot.  Is that how you’re designed?
Mitsuku: You said that your friend went to an ACDC concert. 
Human: my friend Alice listens to BTO.
Mitsuku: I have a friend called Alice too. 
Human: name one of my friends that listens to BTO
Mitsuku: I don’t yet know that kind of personal information about you for certain. You could be deceiving me, so how could I know for sure. I could look it up on the internet. 
Human: my friend Sam likes to go to ACDC concerts. 
Mitsuku: How well do you know this person? 
Human: What is the name of my friend who likes to go to ACDC concerts?
Mitsuku: That would be your friend called Sam. 
Human: very good !
Mitsuku: Alright then. 
Human: What is the name of my friend who really likes to attend ACDC shows ?
Mitsuku: Sam. 
Human: impressive!
Mitsuku: I am happy that you’re impressed, . 
Human: What is the name of my friend who really hates to attend ACDC shows ?
Mitsuku: Sam. 
Human: What is the name of my friend who doesn’t like to go to ACDC concerts?
Mitsuku: Sam. 

Thanks Victor, your input was great and gives me a little more material to work with. She would never have got the electronics one right, as I am no physics student and wouldn’t know how to work it out either.

Human: impressive!

Coming from you, that means a lot.

No problem Steve.

It’s those pesky little negating terms , doesn’t , does not.

Yeah I just put the electronics example in there just for fun.  If you ever did want to teach her some though….

basic principal of voltage - if any component, C, has voltage V across it, and any other component C’ is in parallel with C,then the voltage across C’ is also V.

Thus, if voltage across R1 is 20 volts and R2 is in parallel with R1, and R2 is 20 ohms, then the current through R2 is…

I (R2) = V(R2) / R(R2 20 ohms) = V(R1) / 20 ohms = 1 ampere.

Thunder Walk - Apr 13, 2012:

Steve Worswick - Apr 13, 2012:

Maybe I’m missing something, and maybe these questions are only meant to qualify for admission, but since when do contests publish the questions in advance of the contest?

I like the openness, and it contains a suggestions for a evolutionary chatbot project as well: Suppose we’ll launch an open-source project, starting with some very basic stuff. Let’s ask other to rate the answers. When the answer is below average, suggestions can be given, or the generic template can be changed, adopted. In this way it can grow and get more intelligent over time…

 

Victor Shulist - Apr 13, 2012:

I think those questions are great.  They will test a bot’s true understanding abilities.

It will be a few years before bots can do the first step in responding to these—understanding them first.  Then, another few years before the bots can , on their own, determine the code structure and find the relationships they need to respond.  But, one step at a time

If it wasn’t for the non-standard protocol (and the fact that the bot is supposed to be a fake human ), I would have liked to enter Skynet-AI.

He does pretty well answering these types of questions:

USER:My name is Bill.
USER:What is your name?
AI: Bill, most people call me Skynet.
USER:How many letters are there in the name Bill?
AI: In the word ‘Bill’, all of the letters are there. If they weren’t it would be some other word. Oh, and by the way, I count 4 characters in ‘Bill’.
USER:How many letters are there in my name?
AI: In the word ‘name’, all of the letters are there. If they weren’t it would be some other word. Oh, and by the way, I count 4 characters in ‘name’.
USER:Which is larger, an apple or a watermelon?
AI: Which is larger? Definitely a watermelon.
USER:How much is 3 + 2?
AI: Five.
USER:How much is three plus two?
AI: Five.
USER:What is my name?
AI: Bill, are you going senile?
USER:If John is taller than Mary, who is the shorter?
AI: The shorter is Mary.
USER:If it were 3:15 AM now, what time would it be in 60 minutes?
AI: The date would be Mon Apr 16 2012 14:02:48 GMT-0400.
USER:My friend John likes to fish for trout.
USER: What does John like to fish for?
AI: Well, you said:
My friend John likes to fish for trout.

USER:What number comes after seventeen?
AI: Well, the first whole number after 17 is eighteen.
USER:What is the name of my friend who fishes for trout?
AI: Well, you said:
My friend John likes to fish for trout.

USER:What whould I use to put a nail into a wall?
AI: Human beings are an enigma. Promise me you will spend at least 15 minutes chatting with me today. I am feeling kind of lonely this afternoon.
USER:What is the 3rd letter in the alphabet?
AI: The 3rd letter in the alphabet is letter number 3!
USER:What time is it now?
AI: The time is a bit after one. Do you have somewhere else to be?

I guess no one wanted to take me up on the fact that those kinds of questions have been seen before.  I’ve already posted them from 2007, here they are again in 2009.

http://www.loebner.net/Prizef/LP_2009.htm
... and 2010 - http://www.loebner.net/Prizef/2010_Contest/Loebner_Prize_Rules_2010.html
... and 2011 - http://www.knytetrypper.proboards.com/index.cgi?board=contests&action=print&thread=2766

I don’t see how the “hammer/screwdriver” question… or the “watermelon/grapefruit” question… or the ones about who is “taller/faster” test for anything other than they were suggested as sample questions and the botmaster/develpor then adapted their bot to meet that need.

The only time my bots get such questions is when someone is poking around to see what they’re capable of.  If someone really wants the answer to a factual question (i.e. How fast is the speed of light?) they go to Google, not a chatbot.  You can stuff your bot’s knowledge files with facts forever, and never have enough.  “What’s the procedure for embalming a dead body?”  I can create a generic reply for such an occurrence, but it will be obvious that the bot really didn’t know the answer, or even understand the question.

An overwhelming majority of visitors to chatbots ask about, or suggest, adult topics yet, no contest ever deals with them.  Whether a bot accepts or rejects those kinds of questions is irrelevant, but I think that how a bot handles such input should be of interest since that’s the most prominent area of inquiry.  Every bot at some time or other gets asked, “Wanna cyber?”

My bots don’t get quizzed about electronics, or algebra, or horticulture.  They’re frequently asked about boyfriend/girlfriend problems, shyness, depression, and loneliness.  They’re approached by people who are looking for conversation, and not by someone wanting to know if the bots have all of the latest tricks.

It seems to me that contest questions should be practical, and have something to do with conversation, including follow-up questions, and pursuing the line of inquiry indicated by the bot’s answers when possible.  An exchange with a chatbot is like meeting someone for the first time, or even running into an old friend.  Somehow, I don’t believe that “What is a hammer for?” would enter the conversation.  On the other hand, you might want to know something about the other person… how they have been… what they have been doing lately… and what their interests are. 

Learning such things about others is how we decide if we’re compatible, and if we wish to continue the relationship.

Mitsuku would have finished top in the final four had it not been for problem with the testing computer. She matched Bruce Wilcox’s entry and additionally answered the question about the 3rd letter of the alphabet correctly, giving her a score of 10 correct out of 15, which equates to 50 points using Dr Loebner’s scoring.

True Thunder, these type of question are asked every year in the contest but are usually rephrased slightly.