Thought i would share this article:
  http://www.ecnmag.com/news/2014/11/alternative-turing-test-artificial-intelligence

the artificial agent passes if it develops a creative artifact from a subset of artistic genres deemed to require human-level intelligence and the artifact meets certain creative constraints given by a human evaluator. Further, the human evaluator must determine that the object is a valid representative of the creative subset and that it meets the criteria.

I think we’ve discussed the Lovelace test at some point before. This version has only dropped the “must be inexplicable” clause. Long argument short: creativity is not the same as intelligence, and once again it will be very hard to judge to what degree the AI is doing something original.

Being something of an illustrator, I can tell you that method and rules of thumb make up a surprisingly large part of creative decisions. The fact that some artists aren’t fully aware of the patterns they stick to only makes them seem more intelligent to people who are equally ignorant. Similarly, people attribute my creative skills to some magical talent, but as many artists will tell you, there is no such thing as talent, only practice.

From the paper:

the test provides no threshold at which one can declare an artificial agent to be intelligent

I’ll add to that: If every time someone holds up a hoop telling us to jump, we jump, nobody would get any useful work done.

Good point about the hoop Don.
There is generally so much emphasis on e.g. the Turing test - when passing a bot off as a human is at the same time extremely difficult and of dubious use, when there are so many other potential uses for chatbots.