Facebook AI Research (FAIR) published a set of 20 tasks that they believed any AI should be able to solve.

This month, I took some time to create an AI that completes all the tasks with 100% accuracy. This score might be the highest anyone has achieved on these tasks.

Facebook AI Research is organized towards the goal of automatic text understanding and reasoning. The 20 bAbI QA tasks are a step on the path and method to benchmark AI algorithms.

Antoine Bordes Artificial Tasks for Artificial Intelligence, ICLR keynote, 2015.

Seems there’s going to be a fair number of A.I.‘s in the unemployment lines if this task list is to be challenged.

It is a fascinating example as it shows how knowing event words like “in”, “picked up”, “went to” is all that is needed to answer the questions correctly, without any understanding of the nouns involved.

Getting such generic ‘event’ words straight* would be very worthwhile but I hope FAIR also asks questions requiring particular knowledge of the nouns; otherwise AI is being defined down as content-independent communication.

(*) I mean knowing whether their source/target nouns are things, people, or places.

The challenge presented a number of interesting insights. To the best of my knowledge, no deep learning/neural net solution has been been able to solve all the tasks (even though much of AI research is using these techniques).

Simple tasks are a good milestone enabling bench marking on a restricted data set. Computational intelligence from the current conversation is one of the many things AI needs to solve.

When you add in the requirement to understand nouns, you dramatically increase the amount of data that you must store in memory.  There are some other tasks, like the Loebner competition that do require a greatly increased grounding in worldly data

If AI enthusiasts solved all the tests a grammar school student could, we would be well on our way to general AI. Beyond the tasks themselves, FAIR also presented how they achieved Natural Language Generation (NLG) to create questions for the tasks. NLG techniques are important for chatbots if they are to sound human.

The Allen Institute for Artificial Intelligence also has a number of AI tasks to solve, like the math challenge I solved earlier.

I would think you need a lot of specialised logic systems to get all of the questions right. It sounds along the lines of the common sense used for Winograd Schemas, where indeed focus does not lie on nouns but on connections between verbs. Focusing on nouns would mostly test knowledge, while focusing on event relations tests a generally applicable intelligence. The task list does contain a few basic inferences where the necessary noun knowledge is provided, and I think that’s only fair.
My system can answer about half of these, but would need adjustments and specific SHRDLU-like systems for the rest. I’m not inclined to jump through other people’s hoops just because they’re holding them up, but it looks like a solid list of basic abilities and a job well done to tackle them all. In retrospect the 20 tasks seem to make a good checklist for logic necessary for the Winograd Schema Challenge, if that ever comes around again.

This discussion is making me more aware of such “specialized logic systems”. You do need a few but perhaps not that many: some before/after system to track the order of events; some spatial system to track position information (in, north, next to); some assumption of persistence of ownership (“has”). Such basic ‘rules of context’ may not be that diverse. How many are needed? It must depends on complexity.

I guess now I need to think about how to add ‘rules of context’ to my keyword engine.

To give you some ideas, I worked on common sense axioms, about 20 to 30 of them. Axioms for size, amount, time, location, transfers, general physics, time, lifespan, feelings, karma. You’d need increasingly more rules to cover ever more detailed systems like hydromechanics, but the basics go a long way.
Other approaches like neural networks instead just apply the frequency and amount of co-occurring words. For instance, I can imagine how one would answer the following merely by matching words from the question to the statement that contains most of the same words in the same sequence, and answering the difference, “Mary”.
“Mary gave the cake to Fred.”
“Fred gave the cake to Bill”
“Who gave the cake to Fred?”
You could even mimic inferences by matching words across two statements. I’m not a fan of this but it looks like it would work for a lot of the Facebook tasks without requiring any logic systems at all, just word matching.

I liked your Wordpress article. Very nicely written.

I am inclined to believe that very large servers will be needed to hold vast databases of assorted information. Even given this premise, the program will still need a method of sorting everything out from nuance, slang, expression, puns, double entendre and so on.

Short story - All collected knowledge is worthless in the hands of a 2 year old. It needs to learn how to learn and what to learn.

That’s very deep, Art; and very true.

I prefer programming the basics and learning the rest. Even nature gave us a visual cortex subsystem to process visual information. Might as well skip a million years of trial-and-error learning and program a spatial reasoning system right away. The same goes for language: The phrase “He was beside himself” is in conflict with a spatial axiom if taken literally, which could allow a computer to learn that this must therefore be some sort of figurative expression.
Or even more simple, you could just teach it: https://www.youtube.com/watch?v=hRG7fZuQSk0

Don,

Your statement, “I prefer programming the basics and learning the rest.” Is that to be taken as once you provide the programming, then the computer is to learn the rest of what is required? Your next-to-last sentence provides that the computer could learn about figurative expressions.

How have you been able to program your A.I. (bot) so that it is able to learn and if so, to what extent?

I didn’t mean to imply ambiguity based on your first statement but wanted to make sure how you intended it to be.

Thanks!

Pardon my late night grammar . The A.I. learns knowledge by reading, I just program how it learns and how it should employ that knowledge. For instance:
I programmed pronoun resolution: This allows the program to learn the gender of people and names in general, so it can refer to them appropriately.
I programmed syntax rules: These allow the program to make a solid guess at whether an unrecognised word is a verb, adjective, name, etc. It then permanently adds the word to its vocabulary. The more words it learns, the better it can categorise further words.
I programmed semantic parsing: This allows the program to extract e.g. subject-verb-object facts from plain English. It then stores those in its database. These facts are used in common sense axioms and inferences, which improve its ability to handle ambiguity. It still gets about 20% wrong.

The way I would (but don’t) handle expressions is to have the common sense axioms trigger if e.g. “When pigs fly” contradicts a known fact “pigs can not fly”, then have the program ask clarification. It could then store the original phrase in a file alongside the user’s “I mean…” explanation and use that to substitute every future occurrence of “When pigs fly” with “It will never happen”.

I suppose figurative speech isn’t basic enough to be included on Facebook’s list.

Thanks Don.

Here’s what might serve as a useful if not interesting listing of idioms in American English. There are many other somewhat amusing sayings and quotes on this site. How and if one wishes to employ them is anyone’s guess!

http://www.smart-words.org/quotes-sayings/idioms-meaning.html

That is a useful list. I had to rote learn idioms in school myself, so it seems only fair to furnish A.I. with the same material.

Merlin, got any code or pseudo code or a demo of the AI that can solve all those tasks?

Since the Facebook tasks were all file based, I don’t have anything on-line that you can look at.

I coded the AI in Python and only did the English tasks (not Hindu). I took an approach more similar to what I would use in a chatbot than a traditional machine learning approach. Since Skynet-AI can automatically pull data from the current conversation, I am interested in techniques where the answer to a question can be gleaned from the current conversational log. The Facebook tasks lend themselves to this type of exercise. Most answers are in the conversation (although a few, like the yes/no questions require the answer to come from a separate list)

There are 4 files per task that I tested against.
1000 question train/test sets.
10000 question train/1000 question test.