My chatbot works with something I have dubbed ‘semantic patterns’ for interpreting natural language input. Basically these are ‘template phrases’, with the ‘wildcards’ containing the meaning.

You can think of it like a primitive form of POS tagging or something. Anyway, this is part of how my bot interprets language, and after some regex trickery it knows what to store.

So, A pattern could be ‘A [class] is a [parentclass]’, and this makes my bot understand that ‘A dog is an animal’ is information about the class ‘dog’ being a subclass of ‘animal’

Anyway, basic stuff, not very exciting. Now, I tend to store these patterns together with an example phrase, mainly for my own memory, but I just realized this makes for a nice unit test occasion: for all the patterns, run the example phrase through yokobot, and see if she still interprets the phrase as intended, even as her code grows more complex.

So I built a rudimentary version of this, and the outcome looks something like the below screenshot - and it’s nice because it allows me to nicely spot ambiguities in language interpretation, regressions because of new code / patterns introduced, ‘clashing’ patterns, etc:

http://i.imgur.com/BoQi8bL.png

... Would love your thoughts on this. Do AIML bots or other bots you have/use do something similar? How can I improve on this?

I had a similar idea some months ago but haven’t implemented it (I really should). In my case I finetune my syntax rules every now and then to cover newly discovered exceptions, but because of their complexity this often causes one of the other 100 syntax rules to stop working correctly. Adding an ambiguous word to the program’s vocabulary can cause the same. So I keep a file around with various basic phrases that I run by my program, while I check if it is processing every single word and clause correctly. As this is inhumanly painstaking I figured it would do better to automate it: Have the program read a standard text file line by line, compare it to the standard “correct” interpretation also stored on file, and have it warn me if something blew up again. Basically a self-diagnostic.

If you frequently tinker on your patterns, something like your standard test is a great time-saver. And yet somehow I haven’t made time for it.

An easy implementation would be to check if the output Y to input X complies with the standard, but as I also change my output template often that doesn’t work for me.
Perhaps another possibility is to write several variations of a type of input on one line and see if they’re all interpreted the same as the first most basic sentence: “I have a cat. I am a cat’s owner. A cat is mine. A cat belongs to me.” etc. Then you could compare just the inputs and don’t need to keep the “correct” interpratation on file alongside it. Thanks for giving me an excuse to brainstorm

Looking at your list by the way, I would recommend doing some root forming on at least the basic verbs “is/are/was/be/been”, “have” and “do”, and changing your “the” patterns to a list of “the/this/that/my” etc. Covers more ground.

your list of patterns is impressive. Harry also uses language templates with wild cards.

my approach to regression testing includes both language pattern recognition and “mental” behavior. for example, given certain facts, what inferences will Harry make. are these inferences the same as expected?

Sounds like great minds think alike! Great to see I’m not the only one thinking in this direction.

@Don (or is it Patrick? ), that is/was EXACTLY my situation, especially what you say with adding ‘broader’ patterns later on overwriting earlier more specific patterns.

One other way I’ve sought to leverage this is the following which may be interesting for you as well to look into: a patterns ‘specificity’ is correlated with the number of ‘wildcards’ in there, and so you can give them priority based on that, that’s how I do it. The LESS wildcards, the more specific the pattern.

So, say, the pattern

[instance] likes [classes]  (which, roughly put, should map to an internal interpretation of INSTANCE_HAS_PREFERENCE_TOWARDS_CLASS with preference level ‘like’)

will take precedence over [instance] [performsactionon] [classes] (where you can see that the middle part will be any verb, e.g. Tom catches butterflies)

So the ‘like’ in the first pattern, being hardcoded in it, is more specific, and so that is the pattern my bot should use first.

I just finished the code for running unit tests manually, but as you can see, now comes the boring part of acting on them and fixing the bugs, of which there are unfortunately a-plenty

@Toborman Thanks! This is just a subset actually, more specifically the one about ‘statements’ about ‘onthology’ (i.e. classes having properties and parent classes). There’s lots more patterns dealing with actions, events, feelings, common expressions and whatnot. I’d be glad to share if you’re interested.

This is just a subset actually, more specifically the one about ‘statements’ about ‘onthology’ (i.e. classes having properties and parent classes). There’s lots more patterns dealing with actions, events, feelings, common expressions and whatnot. I’d be glad to share if you’re interested.

Yes, I’m very interested.

I think your pattern prioritising method is very clever. I’ll keep the concept in mind, but although we do have similar problems, I don’t actually use pattern matching. For comparison, this is one of 40 syntax rules I wrote just to rule out that e.g. the word “programs” might mean the verb:

Translated: “have” can only be followed by verbs ending with -ed/-en/-ing, unless *insert annoying exceptions*. What I did automate is that when a word is syntactically impossible according to the rules, a red button lights up and the word and sentence are added to an error log for examination. I guess with pattern matching that would be the equivalent of not finding any match.

(It’s “Don” only as in “Don Alehandro de la Vega” )

I’m having another idea. I could code some c++ Windows software that copy-pastes sentences one by one to a chatbot’s input field and presses enter, so chatbot creators could feed textfiles to their chatbots without manual typing. The thing is, I have no use for it myself so I’d put a price tag on it to cover my hours.

Don Patrick - Jun 11, 2015:

I’m having another idea. I could code some c++ Windows software that copy-pastes sentences one by one to a chatbot’s input field and presses enter, so chatbot creators could feed textfiles to their chatbots without manual typing. The thing is, I have no use for it myself so I’d put a price tag on it to cover my hours.

It might be possible to whip something up like that very fast with AutoIt or AutoHotKey:

https://www.autoitscript.com/site/autoit/
http://www.autohotkey.com/

On Linux I use something similar called AutoKey:

https://code.google.com/p/autokey/

Thanks for the tip. Autohotkey looks rather interesting for saving programming time and creating user interface windows, which is a real pain in c++. I already have the clipboard, keypress and mouse click codes lying around though; I added task automation functions to my AI last month.
Strange how I’m a programmer yet rarely realise that my own work can be automated too.

Don Patrick - Jun 12, 2015:

Strange how I’m a programmer yet rarely realise that my own work can be automated too.

Totally agree Don, I’m not using this stuff nearly enough myself. These tools are great for building test installations but I mostly do that by hand as well. Although, currently testing mainly meaning ‘trainig’, the human component in that is rather important

My bot actually works a lot like the idea Patrick suggests - I ‘feed’ her knowledge with a system of natural language files with simple phrases, which looks something like this:

http://i.imgur.com/xYzCIqY.png

(how does one include an image inline here?)

This keeps me on my toes! No inputting knowledge except through her learning mechanism!

Wouter Smet - Jun 12, 2015:

My bot actually works a lot like the idea Patrick suggests - I ‘feed’ her knowledge with a system of natural language files with simple phrases, which looks something like this:

This way you are actually feeding predicate logic into the system. The problem with this is that your system will only be able to react correctly to a question, if the needed predicate logic has been submitted by you. Unless you have some way for the system to build it’s own predicate logic by inferences from other data, which I doubt because then you wouldn’t have to input the data in the form you show here.

Here is some training data from our system:

- There was no other like it in any of the stores.
- he sometimes looked at it on the sly on account of the old leather strap that he used in place of a chain.
- she turned white for just a moment.
- He looked thin and very serious.
- He needed a new overcoat and he was without gloves.
- He simply stared at her fixedly with that peculiar expression on his face.
- That was as much as she could do or say.
- You would not dare to interrupt her description of their advantages.
- you could never afterward entertain the same feeling toward your parents.
- He wrote plays and smoked cigarettes in it all day long.
- But every room-hunter was made to visit his room to admire the lambrequins.
- In it was an iron cot, a washstand and a chair.
- Its four bare walls seemed to close in upon you like the sides of a coffin.
- Through the glass of the skylight you saw a square of infinity.
- She carried a typewriter made to be lugged around by a much larger lady.
- Why didn’t you keep up with us?
- I asked the lady to have a look at your lambrequins.

We take those sentences from several novels and short stories, to prevent bias in formulating test sentences. When we started testing and training the system, there was an abundance of the words test and testing in the sentences

When our system is fully functional, it will be able to actually read full stories and documents, and build knowledge from that, as it does build it’s own predicate logic for the learned information.

Wait, I’m a bit confused I think - what do you mean exactly by ‘predicate logic’ in this context?

I suspect the answer about my bot to your question is, yes, Yoko is only able to answer questions correctly if the knowledge is already in some form in her database, but I could be misinterpreting.

I think I’m confused because some bot learning/training is about ‘language’, whereas other learning/training is about - knowledge - and while my example is about feeding knowledge, yours is about training in language? (If not, not sure what can be made by a computer program of an input like ‘that was as much as she could do or say.’)

I occasionally use Wikipedia and academic blog articles for testing (not training) my NLP. It misunderstands about 30% of that, which is why for basic knowledge, I use a file with simple phrases just like Wouter. The very reason being that I do use inference and want to minimise the chance of misunderstandings in the base knowledge. In my opinion, Wouter’s system is well adaptable.

Woah, your bot can understand about 70% of the phrases in a wikipedia article? Understand not just, dunno, POS tagging, but actually extracting some meaning from them?

That’s hugely impressive if you ask me, Give Yoko a ‘because’ in a phrase and she crawls into a corner sobbing (so to speak )).

This is turning into a show-and-tell session.

I’m not sure if 30% misunderstanding equals 70% understanding, but feel free to take that with a grain of salt
Some pieces of text that I tested came from Simple Wikipedia instead, and I had a great deal of trouble with the page format (bullet lists, tables, non-sentences, headers), so I just copy-pasted the main paragraphs for reading and it got the gist of that right.
As for understanding, you know what, I’ll show you a schematic. It extract facts at semantic (meaning) level, after POS and grammar levels. How, is deliberately not apparent from this example.
Not covered are facts about indirect objects, facts about location, cause-and-result relations, or unspoken intent. For now it mainly ignores the word “because” and just stores the bare facts after it. The inference engine takes it into consideration either way.
I did sacrifice 3 years of my life to program this, so I should have something to show for it (and yet still get beat by a majority of pattern matching at Turing Tests). I’m just saying, there is plenty of potential in your approach.

Wouter Smet - Jun 12, 2015:

Wait, I’m a bit confused I think - what do you mean exactly by ‘predicate logic’ in this context?

What I mean is that the sentences you enter are all (as it seems) in the form of predicate logic. It seems to me that your bot is aimed at answering questions about how things relate to each other. However, the simple question that pops up for me is how your bot will handle a sentence like this one:

The lights where on in the house but it seemed that there was nobody home.

Unless you have all the possible relations between entities and other entities and actions encoded in predicate notation (like you showed), not to forget having some POS-tagging system in place that can break down this sentence to do something with it, your system can do very little with that. And I’m not even talking about how the system should formulate a response (if any) on such a sentence.

I don’t want to shoot down your efforts, but in my view the approach seems pretty basic (at least currently). However, Don says he sees potential and I know that his system, although he states it uses a simillar approach, is much more comprehensive then he makes it to appear here