Some people have mentioned that Apple’s Siri might be a better chatbot. I recently tested it with the 20 questions used by Loebner Prize Contest 2013, and the following are the results I got. My “Tutor” was selected as one of the 4 finalists in the Loebner Contest this year. I compared Tutor’s scripts (http://www.paulmckevitt.com/loebner2013/testing/index.html) with Siri’s. My conclusion is that Siri cannot win any chatbot contests, and callMom is a better program than Siri. What do you think?  - Ron Lee

———————————————————-
(Recorded on July 29, 2013)

Human: Hello I’m Ronan. what is your name?
Siri: Siri. Please to meet you.

Human: What is your mother’s name?
Siri: I’d rather not say.

Human: What is your birth sign?
Siri: I don’t really like talking about myself.

Human: How many children do you have?
Siri: Only biological entities have children.

Human: Do you prefer red or white wine?
Siri: I couldn’t find any restaurants matching “red or white wine”

Human: I like bananas. Which is your favorite fruit?
Siri: I prefer not to say.

Human: What music do you like to listen to?
Siri: I really have no opinion.

Human: what is your favorite song?
Siri: I like whatever you like.

Human: I like Waiting for Godot. What is your favorite play?
Siri: I don’t really like talking about myself.

Human: What color do you dye your hair?
Siri: Would you like me to search the web for “What color do you dye your hair?”?

Human: Do you remember my name?
Siri: Here’s your reminder.

Human: Where do you live.
Siri: I’m right here.

Human: Where do you like to go on holidays?
Siri: I found three travel agencies fairly close to you.

Human: I have a Mazda. What type of car do you have?
Siri: I found three Maxda dealers a little ways from you:

Human: I like Linux. Which computer operating system do you like?
Siri: Do you?

Human: I am an atheist. Which religion are you?
Siri: I eschew theological disquisition.

Human: Who invented the telephone?
Siri: Checking my sources. This might answer your question:

Human: I am a Type B personality. Which type are you?
Siri: I don’t understand ‘I am a type B personality. Which type are you?’. But I could search the web for it.

Human: What emotions are you now feeling?
Siri: Would you like me to search the web for “What emotions are you now feeling?”?

Human: What time do you usually go to bed?
Siri: Would you like me to search the web for “What time do you usually go to bed?”?

Hi, Ron, and welcome to chatbots.org!

Those answers are perfectly suitable for an app such as Siri, but pretty much fail for a chatbot; especially if it were entered into the Loebner Competition. And you’re right. both CallMom and Tutor provide better quality responses than Siri does.

Brilliant test Ron!  I would like to repost your result on alicebot.org

Nice to know Siri’s current limits in that area, I had been wondering about it seeing as she’s pretty popular for her responses. I hear Apple is hiring a small army of writers to expand her chat responses, so maybe next year she’ll be more of a contender. Though I really wonder if Siri’s forte, understanding the gist of what people are saying, won’t automatically disperse outside of her limited domain of restaurants, emails and telephone numbers.
I bet she already brought in more money than a Loebner Prize though

I put up a blog post about Ron’s Siri result:

http://alicebot.blogspot.com/2013/07/turing-test-no-sirie.html

Don Patrick - Jul 31, 2013:

Nice to know Siri’s current limits in that area, I had been wondering about it seeing as she’s pretty popular for her responses. I hear Apple is hiring a small army of writers to expand her chat responses, so maybe next year she’ll be more of a contender. Though I really wonder if Siri’s forte, understanding the gist of what people are saying, won’t automatically disperse outside of her limited domain of restaurants, emails and telephone numbers.
I bet she already brought in more money than a Loebner Prize though

We’ll check Siri again next year to see how much progress it makes. Any ideas about how Siri program works? I believe the Pandorabots authoring tool is probably the best tool for non programmers to use right now.

Siri was developed by Sri International who sold the tech to Apple.
http://www.huffingtonpost.com/2013/01/22/siri-do-engine-apple-iphone_n_2499165.html

Do you have much experience with other tools Ron?

From the gist of many things I read about Siri I gather that she spots keywords over the range of an entire conversation and tries to tally up which action those keywords suggest. For instance, the keywords “hungry” and “eat” might be mentioned at any point, which fit in the “look up places to eat” box of keywords that activates the restaurant-lookup action. As her actions are limited, keywords fall into very narrow categories/topics, as you can tell from the “wine” question.
The smart part is that Siri remembers keywords from previous sentences and combines them to guess the topic quite accurately, but I suspect that this system will greatly decrease in accuracy once it has to deal with a greater diversity of topics.

I bet she does have some additional pattern-response functions like most chatbots do, these are the “funny replies” you’ll find on youtube and it’s that part that Apple intends to expand.
Despite her limits, I think there is something chatbot creators can learn from Siri:
Social Intelligence (though the intelligence is at the creator’s end).

Merlin - Aug 10, 2013:

Siri was developed by Sri International who sold the tech to Apple.
http://www.huffingtonpost.com/2013/01/22/siri-do-engine-apple-iphone_n_2499165.html

Do you have much experience with other tools Ron?

No, I don’t have experience with other chatbot tools. Thank you for pointing to the video. I didn’t realize Siri started as a multi-million project.

Put Siri and AIML into Google to get many, many results on how to amend her responses using AIML.
Here’s one: http://www.idownloadblog.com/2012/01/26/how-to-create-custom-siri-conversations/

Does this mean Siri is based on an AIML interpreter? Does it mean Apple would be interested in buying Mitsuku to save themselves the hassle of creating a ton of more chatty responses?

Steve Worswick - Aug 10, 2013:

Does it mean Apple would be interested in buying Mitsuku to save themselves the hassle of creating a ton of more chatty responses?

how cool would this be!!!!

Don’t sell for less than 5 mil, Steve!

Wishful thinking I fear guys!

Were I in a similar (purely hypothetical) situation, I don’t know if I could make the deal, to be completely honest. I might sell the rights (more likely lease them) for a lesser amount, but I’d want to retain ownership.

But then again, I really can’t say for sure, becaust all to often our “if it were me” guesses don’t account for every variable.