The knowledge of chatbot systems are still far from human adults. Test comparing with adults misleads the direction that most of the system try to cheat the testers by some simple methods.

If the comparison group is composed by 7-8 years old children, and the chatter may ask more application questions. The results may reflect the real diversity of the chatbots.

This was suggested at the 2011 contest in Exeter too. Although the panel of judges suggested the human confederates be 2-3 years old. I agree that it is a good idea and would be an interesting twist on the current Junior Prize where the judges are children. To have the confederates being children too would be a very interesting experiment.

I’m not sure at what age children learn to type these days, but I imagine that would be a necessity.

The idea was that as long as the child could understand and respond to a question, the parent could read the question and type the child’s reply into the text box. The earliest this was generally agreed on was a child aged between 2 and 3. Once a chatbot had convinced a judge it was a 2-3 year old, repeat the test with a 4 year old, 5 year old and so on until we found a level that chatbots were on a par with.

Sometime before the first LP, when the committee was developing the rules, I suggested using children as human confederates (This would have been in the late 1980’s.) 

The idea was quickly shot down, and they used restricted topics instead.

You were right. I believe using children as human confederates could provide better discrimination than restricted topics. Because without general knowledge understanding, knowledge in restricted topics can not be really understood.

When no chatbot can really understand the conversation, the only way is cheating the human chatter.

We should not forget why we want computers chat with human. We expect computers to help us to solve real problems, not only play text game with us.

Hugh Loebner - Jun 15, 2013:

Sometime before the first LP, when the committee was developing the rules, I suggested using children as human confederates (This would have been in the late 1980’s.) 

The idea was quickly shot down, and they used restricted topics instead.

I would think it no less than logical that the confederates in the Junior Prize should be of similar age to the judges. Isn’t the whole point of the Junior Prize to conduct the test at a child level? Children miss knowledge in similar areas as AI, but children can think as well as adult humans. Therefore a comparison to a child confederate would be as good an argument for “thinking” as a comparison to an adult confederate.