Read the article here

While this is interesting news (and its a brit startup so a bonus as I’m biased), I had a play with this last night for a while, and frankly it’s poor.

Asking it a bunch of the test questions we use on Caesar, it failed to answer ~80% of them at all, and of the 20% it did provide an answer for, the reliabilty of that knowledge was questionable at times.

It seems to function fine with simple Who, What, Where queries, but “push the boundaries” a little more, especially into the semantic realm, and it performs badly.

For example, one of Caesar’s test questions is “Can a fish run?”, to which Caesar correctly answers no, fish don’t have legs, legs are required for running.  Evi didn’t have an answer.  Many other of Caesar’s test queries also resulted in similar devoid answers from Evi.

Moving past the performance aspect, I find it rather interesting that within the space of a few weeks, 2 of the internet behemoths have purchased small, independent startups/companies, for quite large sums of money, in the domain of NLP and KBS.

The race for an intelligent web is certainly in play, but I think the technology of these kind of systems to enable the afore mentioned, needs to be a lot more progressed than it currently is.

Thoughts anyone?

Before I reply… I don’t recall if there’s a way we can talk with Caesar.  Is it an online bot, or something we’d need to download?

In any event, can you give me some idea about how Caesar arrived at the understanding that fish can’t run because they have no legs?

Caesar isn’t available publicly at the moment as it is a commercial endevour, however, we do hope to have our closed beta of a Deep QA service online within the next month or so…it’s very much a “when its ready” sort of deadline.

In answer to your question, Caesar is able to come to that answer by way of learned properties of entities, and the functional requirements of verbs.

The action of walking or running, has a functional requirement that legs are a property of the entity within the statement.  A fish does not have this property, so thus the functional requirement of the verb is not fulfilled, hence, it can not walk or run.

Thats a very simplified overview, but is essentially how it all boils down.

Dan Hughes - Apr 18, 2013:

The action of walking or running, has a functional requirement that legs are a property of the entity within the statement.

I take it you’ve never had the pleasure of seeing an older, out of balance washing machine “walk” across a non-level floor.

Also, a river “runs”, too, and while a washing machine has “feet”, neither it nor a river have legs.

haha very true. 

However, easily solved, include the addition that “animate” is a functional requirement of the verb and you can safely omit rivers and washing machines from walking and running in the traditional sense of a query.

Inanimate objects comply with a differing sense of the walk/run verbs and instead would qualify that, which would denote an entirely different meaning, while still allowing your washer to walk, and rivers to run

I realise we are going way off topic but Dan, do you have a list of these test questions. Mitsuku can also work out whether something can run or walk judging by the number of legs it has. I am curious to know what other test questions Caesar has.

Dan,

This isnt meant to be critical, but in point of fact fish do run http://en.wikipedia.org/wiki/Salmon_run Salmon runs are probably the best known usage of the word in this context, but I believe that the term is applied to any body of fish that is moving upstream to spawn.
As Dave pointed out, the problem is the complexity of word usage in the english language.

Vince

Vince, true, but a salmon run would actually be a composite noun, which references a specific time and event, and not an action verb in that context. So the truth that to run, you need legs, still applies.

Still, nothing like a passing example to get a bunch of developers pulling things apart, thats how progress happens

Steve, there are quite a few test questions & statements we use, some are very simple like Can a fish run?, others are pretty complex.  For example, “John and Jane are going to the cinema with Bill and Ben at 8.30pm to watch a movie.  They all had a good time.”, allows us to apply a complex set of actions, conjunctions, prepositions and co-reference to which then we can query with a question set such as “Who did John see a movie with?”, “What time did Bill go with Ben to see a movie?”.

At present there are around 100 such test questions, which we run on Caesar after each major module update to ensure that all forms of sentence construction are understood, and that the various relationships and such that can exist between entities, elements, objects and actions all still happen as expected.

A lot of our time at the moment is fine tuning, as most of the work of converting a sentence to a more structured form is complete, we want to be sure that there is nothing we have missed, or overlooked before moving onto the next mammoth task of a more granular understanding of verb functionality.

Vincent Gilbert - Apr 21, 2013:

This isnt meant to be critical, but in point of fact fish do run http://en.wikipedia.org/wiki/Salmon_run Salmon runs are probably the best known usage of the word in this context, but I believe that the term is applied to any body of fish that is moving upstream to spawn.

Don’t forget about the Grunion!

Many people enjoy catching grunion at events called “Grunion Runs.”

Vincent Gilbert - Apr 21, 2013:

As Dave pointed out, the problem is the complexity of word usage in the english language.

I didn’t realize that’s what I was pointing out, Vince. Thanks!

I think it would be interesting if you’d put up a no-frills beta test page.  It might prove beneficial.  Who could workout the kinks better than a bunch of botmasters?

I’ve been considering that also recently, a more “techy” beta which shows some of the “guts” of the process so that more technical minded folks such as you guys here can pick at it a little more and maybe provide some feedback to us.

We’re currently a key phase in development right now, and its slow, purely because the complexity of the task involved requires a lot of “sit down brainstorming” on the correct way to tackle it.

That said, once we have this piece of the puzzle in a place, a lot of functionality and cognitive ability just plugs together.  I can’t wait!