Hi folks,

I just started working/pondering one aspect, and would love to hear the thoughts/suggestions/critique of any fellow coders here. It’s about ‘lexical’ or ‘syntactic’ parsing with pattern phrases. (I am trying to avoid the use of corpusses for ‘POS tagging’ as long as possible) This is a VERY long post, so it’s likely that nobody bothers to read it, but it still serves to organize my thoughts

Allright, so, this is in the context of my still-in-infancy chatbot Yoko, who works by matching use input to ‘pattern phrases’ who in turn are linked to ‘meaning + parameters’ which is then processed by looking up or storing in her knowledge database etc etc.

So as a concrete real-world example: the user inputs ‘are cats animals?’, this matches (the generated regex-version of) the ‘are [somethings] [somethings]’ pattern, which is linked in turn to the meaning QUESTION_HAS_CLASS_PARENTCLASS which is then processed from there.

Currently these pattern phrases are defined in a bunch of .json files as follows:

(the “1” and “2” here indicate which [something] maps to which parameter for the further processing, the rest should be pretty obvious)

Anyway, this is quickly running out of hand, as I’m starting to reach the 100s range of these, and still getting nowhere close to capturing even a fraction of possible phrase structures with them. This caputres ‘are cats animals’, but for ‘is a cat an animal’ I need another block like this, etc…

I’ve considered many improvements for this verbosity, and already have a bunch of ad-hoc ones in place. To stick with the cats/animals thing a ‘sub pattern’ that matches both ‘a something is’ and ‘somethings are’ for example. But the singular/plural distinction affects other parts of the same phrase in so many ways, that I prefer sticking to patterns for entire phrases for now.

Or another one would be to write the parameter meaning right inline: “a [something|class] is a [something|parentclass]”, but I don’t like how this hurts readability.

(already, and in the future, there is of course some separate logic for processing more complex phrases like ‘cats are animals that can meow’. But now I’m focusing on ‘atomic’ ones)

Ok, here’s how I think I’ll have my new pattern definitions look:

All the wildcard and multiple choice stuff should look familiar to people with AIML/chatscript/regex experience, but those are just there to make this as realistic as possible - they are taken straight from Yoko, but but that’s not the new and exciting part.

The new part is that there is now ONE block where previously there would have been THREE, because they are all associated with the same meaning. Moreover, there is no [something] anymore, with a separate line to map each [something] to a meaningful parameter, but rather the parameters THEMSELVES contain both the ‘lexical form’ information and the semantic (parameter) meaning, while the whole is still readable.

I love that the pattern phrase itself contains all this syntactic+parameter meaning, the definition is so much more compact, while it all remains very readable, to the point where I consider not bothering with example phrases anymore (though they are great for unit tests!).

Sadly this improvement will also come at a cost of slightly increased complexity in a way: instead of almost everything in the pattern phrase being a [something] or [somethings], there will now be many more cases:

BEFORE:
a [something] is a [something] -> CLASS_HAS_PARENT_CLASS (‘a cat is an animal’)
a [something] can [something] -> CLASS_HAS_ACTION (‘a cat can meow’)

AFTER:
a [class] is a [parentclass]
a [class] can [performaction]

... The extra complexity is that I will introduce a separate mapping, that translates all those [class] type parameters in TWO others: a lexical and a synthactic one. The added benefit is that the lexical one will be more precise than before, which will allow me in the future to incorporate irregular plurals and verb conjugations, etc etc…

Here is the additional mapping that accompanies the new-style patterns,  again in JSON because I love it:

As a legend of the above: ‘semanticparameter’ is what will be taken as input to the semantic processing functions (that are directly tied to Yoko’s knowledge), ‘patternparameter’ is what will occur in patternphrases, and ‘lexicalform’ is what will be translated into regexes for the actual pattern matching.

... so, this gives me the power of my more descriptive+compact pattern phrases, AND the flexibility of more regexes (and grammar-specific exceptions for what a regex can’t capture)  for each of those lexical forms (currently ‘plural noun’ and ‘3rd person present verb’ are both simply the same [somethings] regex, which more or less matches ‘whatever ends with an s’).

YOU MADE IT THROUGH THE POST! What do you think? Am I overlooking something? Is the introduction of the separate extra mapping not worth the more compact pattern phrases?

Wouter Smet - Sep 9, 2013:

Currently these pattern phrases are defined in a bunch of .json files as follows:

{
"pattern" : "are [somethings] [somethings]",
"examplephrase" : "are cats animals?",
"meaning" : "HAS_CLASS_PARENTCLASS",
"params" : {"class" : "1", "parentclass" : "2"}
} 

Wouter, is there a way to upload JSON like this at the Yokobot.com upload page?

Allright, so, this is in the context of my still-in-infancy chatbot Yoko, who works by matching use input to ‘pattern phrases’ who in turn are linked to ‘meaning + parameters’ which is then processed by looking up or storing in her knowledge database etc etc.

Skynet-AI takes a very similar approach (which reminds me that I have not posted about “JAIL” in some time).

You need to start thinking about long term care and feeding of your bot. Specifically:
- Will it only be you maintaining Yoko, or will others/novices be adding to the knowledge base.
- Will you be editing the raw JSON files or using a user interface. Many find a spreadsheet like UI easier to do bulk edits of simple input/response pairs. Some bots have custom UIs for knowledge management.
http://bragisoft.com/
http://www.personalityforge.com/book.php
- It looks like you are ultimately compiling the Json files. If so, consider using Regex syntax directly for your input side. For example: (a/each/every) vs (a|each|every), or (cat/cats) vs (cats?). It might be easier to troubleshoot later.

Think through your strategies for:
Spelling or grammar corrections.
Symbolic Reduction: Reducing complex grammatic forms to simpler ones. (This is the basis of AIML bots)
Divide and Conquer: Flow control, splitting an input into two or more subparts, and combining the responses to each.
Synonyms: Mapping different ways of saying the same input.
Variations: Different ways of responding to the same concept.
Headword vs Bag of words: Detecting keywords anywhere in the input.
Conditionals: Certain forms of branching, possibly based on context. (“Yes.” will mean different things depending on what the bot asked.)

As you move from hundreds of pattern/response pairs to thousands, debugging will become increasingly important. Watch the size of the names of your object classes “patternparameter” vs something like “IN”. Although it doesn’t make much difference when your bot is small, when it grows large, the difference in naming convention may start to impact file/memory sizes. I classify this as the signal (pattern) to noise (overhead) ratio.

“are [classes] [parentclasses]”, are you looking up synonyms for the items in brackets or just using it as a placeholder?

“grammar-specific exceptions for what a regex can’t capture” - can you give a couple examples?

∞Pla•Net - Sep 11, 2013:

Wouter, is there a way to upload JSON like this at the Yokobot.com upload page?

{
"records": [
{ "input":"HELLO" , "output":"Hi there friend!" },
{ "input":"WHAT IS YOUR NAME" , "output":"My name is Yoko." },
{ "input":"TEST" , "output":"I passed the test." },
{ "input":"BYE" , "output":"Goodbye my friend." }
]
} 

 

At the moment not, no. It should be noted that the second example is actually against Yoko’s design philosophy of parse input -> interpret -> Knowledge base -> generate output. The ‘name’ question, for all its popularity, actually required some specific exception code.

The other examples are indeed pretty close to how Yoko works, because they are of the type ‘common formulations’, which bypass all the difficult knowledge-base stuff and are mapped pretty straight from input to output.

(if you’re curious here is the concrete file with those things: http://www.yokobot.com/lib/language/lang_parse_sentiments.json)

A way for users to input those used to be there, but I scrapped that for the moment. As for how I am planning to make Yoko reasonably smart all on my own, see the next answer I’m about to give in this same thread

ONE MORE THING: besides the ‘common formulations’ data, there IS a mechanism in Yoko for ‘plugging’ specific conversations in her, which I caled ‘plugins’. A plugin is basically a php file that can grab user input before Yoko does, and use its own mechanisms for generating an output. Yoko relies on plugins to:

- answer algebra questions (give it a shot, she does pretty well I would say)
- answer ‘42’ to things containing ‘meaning of life’ (this was more a dummy to test/demonstrate how plugins work)
- answer questions and commands about her virtual 2D world you can enable (this is still in its most early phases, but a plugin keeps this code nice and separate)
- scrape a definition out of wikipedia when she does not know the answer to a ‘what is’ or ‘who is’ question. This was eventually moved to her core code, and is severely limited because, again, it’s against my ‘use structured knowledge base to find answer’ principles.

Here is some info on her plugin mechanism, it’s really simple to create one:
http://www.yokobot.com/index.php?p=about&s=codeinternals#plugins

If you know PHP and would love to use Yoko as your chat engine to play around with, I’ll gladly give you a way to apply/upload your own plugins to her! The Q+A of your ‘records’ example would be pretty trivial to implement for example.

@Merlin,

Thanks for taking the time for your thought-out response. You raise interesting questions, and I’m glad to discover that I have spent time thinking (and sometimes addressing) pretty much all of them by now over the past weeks/months. All of them are addressed in Yoko’s about section in one way or another, but it will be fun to talk about Yoko in this ‘exam’ form, challenge accepted

Skynet-AI takes a very similar approach (which reminds me that I have not posted about “JAIL” in some time).

JAIL is Skynet-AI’s ‘engine’? Please do share, I’m always curious to learn from others! I write almost every singular though I have in Yoko’s ‘about’ page, more for my own benefit then anybody else’s though.

- Will it only be you maintaining Yoko, or will others/novices be adding to the knowledge base.

For as long as possible, just me. And perhaps it will always be like that. While I am having some fun manually entering her input data (in particular her ‘basic knowledge’ file here: http://www.yokobot.com/lib/knowledge/yokoknowledge_basic.json), I am happily looking at other existing available knowledge bases and converting those to Yoko format. Not sure I need an army of volunteers when armies of volunteers before me have already done so much, and much of the result of their efforts is publicly available. To be more concrete:

So far I’ve played around with the free available AIML basic set, openCyc files, WordNet, FrameNet, DBPedia, ChatScripts ‘raw’ source files, the ‘mindpixel’ project assertions…

Besides that alread structured (in some cases more than others) data, I am happily scraping from various ‘top 100’ lists from wikipedia and others, lists of countries, capitals, currencies, languages, top 1000 bands of all time, top 1000 most-used verbs/adjectives/adverbs… top 1000 movies and actors, biology taxonomies, domain-specific onthologies, hopefully at some point when her parser is more robust also the first phrase of simple wikipedia articles…

As you can see from the json file, Yoko’s format is basically ‘simple natural language phrases’ anyway, so it’s a breeze to write a small script that converts those sources into Yoko’s input format, and Yoko’s ‘brain’ takes care of the structured storing.

Currently, my main focus is making that brain robust, so while I’ve played around with the above (http://www.yokobot.com/lib/knowledge/yokoknowledge_hollywood.json ), I haven’t bothered to actually load those files into Yoko yet. I reset her knowledge base very often these days. 

- Will you be editing the raw JSON files or using a user interface. Many find a spreadsheet like UI easier to do bulk edits of simple input/response pairs. Some bots have custom UIs for knowledge management.

manual editing with Sublime Text 2 baby! (but see the above response, the json files that simulate conversations to ‘feed’ knowledge in her database will increasingly be generated by some big open data source + some scripting)

In the beginning I did store ‘common sentiment expressions’ and their responses in a db table with an interface, but I abandoned that because I see those more as ‘language’ than as ‘knowledge’, and I want to preserve the DB as much as possible for the latter. It was a tough choice though, and maybe I’ll come back on it

http://bragisoft.com/
http://www.personalityforge.com/book.php

Interesting stuff! I will definately look into these for inspiration. I currently already use
https://developers.google.com/chart/interactive/docs/gallery/orgchart to visualize the ‘onthology’ parts of Yoko’s knowledge, but it’s not ideal and like so many aspects of chatbotteering, I still got lots to learn on this front.

- It looks like you are ultimately compiling the Json files. If so, consider using Regex syntax directly for your input side. For example: (a/each/every) vs (a|each|every), or (cat/cats) vs (cats?). It might be easier to troubleshoot later.

I considered and rejected this approach. Regexes are in fact already possible in my patterns, but I want to keep them as ‘clean’ as possible so avoid them with (optionalword|optionalwordb)? one of the few exceptions.

In the strategy I outlined in my OP, ultimately the parameters will be replaced by things like [plural_noun], which in turn are a regex, but in the long run this will have to be more than just a regex to deal with English irregularities more properly.

Spelling or grammar corrections.

Not a big priority right now, I see this as more of a ‘luxury’ problem. I DO have a list to correct common user mistakes (‘your a’ -> ‘you’re a’, ‘r u’ -> ‘are you’ etc), but that’s as complex as I want to go for now, not spending precious CPU cycles on going through a corpus or stuff like that.

Symbolic Reduction: Reducing complex grammatic forms to simpler ones. (This is the basis of AIML bots)

Yup, that’s a big one, and I am currently taking some first steps towards this, both the obvious ‘could you tell me if’, ‘I wonder if’... stuff and the more complex (but interesting) stuff with anafora resolution.

Yoko can already do this: ‘ducks are animals that can quack’ will be correctly interpreted as ‘ducks are animals + ducks can quack’ and processed accordingly, AND every following mention of ‘they’ and ‘them’ will be correctly replaced with ‘ducks’.

So I’m quite glad with how this is going to be honest! As with every aspect, lots more to do of course. But I’m convinced to tackle this, and actually rather convinced that Yoko may eventually be more powerful in this regard than many current chatbots.

Divide and Conquer: Flow control, splitting an input into two or more subparts, and combining the responses to each.

Yup, see above, besides other splittings such as points and commas within the same input.

Synonyms: Mapping different ways of saying the same input.

Yes, there is a synonym replacement going on in the parsing (though I sometimes wish this could also be json files I think, because the language-knowledge thingie I mentioned above - it’s just so MUCH though). The most obvious currently visible example in Yoko is ‘human > person’.

Other ‘ways of saying the same’ besides words are also in place, to assure that ‘do you know what X is’ maps to ‘what is X’ etc etc…

Variations: Different ways of responding to the same concept.

Check. In fact, there is no single output that has only one formulation at the moment, and Yoko chooses not just randomly, but randomly while avoiding already used ones.

Headword vs Bag of words: Detecting keywords anywhere in the input.

I do this for ‘common formulations’ (like ‘how are you’) and separate ‘domains’ like answering math questions or the time or date, but not in the ‘main’ engine.

Conditionals: Certain forms of branching, possibly based on context. (“Yes.” will mean different things depending on what the bot asked.)

Yoko is only JUST now getting to the point where I can start doing this, so there’s not much code for that yet (I gave it a first stab yesterday in fact!), but yes it’s crucial. I think here is another domain that Yoko should eventually outperform many other bots, because I am disciplined about CONSCIOUSLY asking questions to the user, and storing the appropriate ‘I just asked X, so a yes will mean that X is true’ info.

At first I asked counter-questions as part of her answer templates because it’s so tempting and natural. ‘I’m fine, how are you’, right? But no more of that. When Yoko asks a question, the code should always know exactly what was asked, so responses of the user can be handled appropriately. Almost all transcripts with existing bots I have seen seem to not do this at all by the way.

As you move from hundreds of pattern/response pairs to thousands, debugging will become increasingly important. Watch the size of the names of your object classes “patternparameter” vs something like “IN”. Although it doesn’t make much difference when your bot is small, when it grows large, the difference in naming convention may start to impact file/memory sizes. I classify this as the signal (pattern) to noise (overhead) ratio.

Interesting point. When performance becomes more of a concern, I’ll definitely add layers of caching and compactifying, but I think it will always be a priority to keep the ‘base’ input as human-readable as possible. It gets translated to messy stuff (including lotsa regexes) early and often, but I want to spare my eyes from that, except when debugging of course.

“are [classes] [parentclasses]”, are you looking up synonyms for the items in brackets or just using it as a placeholder?

They are placeholders, indicating to capture ALL phrases with that structure (both represent ‘plural noun’), or more concrete, something like this regex:

... The second part of information captured is that, even though both are plural nouns and thus morphologically/lexically the same (so why not say ‘are [pluralnoun] [pluralnoun]’ right?) ,the function processing it, will know that the first parameter is the ‘class’ and the second ‘parent class’, and query the knowledge base accordingly.

“grammar-specific exceptions for what a regex can’t capture” - can you give a couple examples?

For example irregular noun plurals, consider this page:
http://en.wiktionary.org/wiki/Appendix:English_irregular_nouns

The first ‘rules’ part in that article lends itself nicely to regular expressions, whereas the rest is more sensibly implemented using a dictionary for replacements, at least in Yoko’s architecture. Of course one can use regexes for a find/replace operation in a string, but that’s the distinction I meant.

JAIL (JavaScript Artificial Intelligence Language)

JAIL is a language I developed to make designing and coding Chatbots/AI easier. In the next few days I will provide an update.

At one point in time, I tried to see if the industry was interested in creating an AI language. At the time I could not get much support, so I went off and created my own framework. Like you, I like the JSON model for many things and it is portable between most languages.

Virtually everything that Skynet-AI does is done via JAIL/Javascript.

Merlin - Sep 11, 2013:

Think through your strategies for:

Spelling or grammar corrections.

Symbolic Reduction: Reducing complex grammatic forms to simpler ones. (This is the basis of AIML bots)

Divide and Conquer: Flow control, splitting an input into two or more subparts, and combining the responses to each.

Synonyms: Mapping different ways of saying the same input.

Variations: Different ways of responding to the same concept.

Headword vs Bag of words: Detecting keywords anywhere in the input.

Conditionals: Certain forms of branching, possibly based on context. (“Yes.” will mean different things depending on what the bot asked.)

Good list, Merlin.

Couldn’t agree more! Actually, some kind of ‘acid test’ of input questions/conversations that checks against each of these might have a lot of value compared to the typical ‘free flow conversation’ Turing test kinda stuff.

To continue the list with some other worries/problems/issues that I have ran into conceptualizing a chatbot:

- Syllogisms and other logic: fruits can be eaten and a banana is a fruit - can a banana be eaten?

- Causality and events: you want to go outside but it’s raining, what would you do next? (find an umbrella) When asked what you did yesterday, or have planned for tomorrow, what type of answer do you give?

- passive versus active: the cake was eaten by Tom; what did Tom do? (though this can be taken to be the category ‘different ways to say the same thing’

- preferences: do you have a consistent answer when asked what you like, dislike? If somebody tells you about a fact or thing that is not to be liked (‘my cat died’), do you react appropriately? Will the same preference be given tomorrow?

- ‘pop culture’ and other rapidly changing knowledge: does the bot have a way to update its knowledge based on current language and other fads? When asked about ‘the election’ or ‘the situation in Syria’, or even ‘asl?’ or Twitter, does your bot know what this is about?

- ‘meta questions’ about language: can your bot realize a question is about the word and not the object? ‘how many characters are in xyz’, ‘what is the second word in ‘I love you’’ etc?

- smart handling of non understood input: does the bot know the difference between total gibberish, something not understandable because it looks like french, or a phrase like ‘what is your favorite adsfs;akljdrqwerlxcv’ where only one part is gibberish?

- short term memory: does the bot remember what the user told about themselves? When the user says their age early in the conversation, can the bot reply correctly when asked about it later? Or even across chat sessions somehow?

- idioms and figures of speech: are these recognized as such?

- humor / math / poetry / ... : can the bot ‘generate’ a poem, joke, .... about something specific when asked for it?

- spatial reasoning: can the bot understand that something is ‘close’ to something else, that something being in a city means its also in the country containing that city?

- pronoun resolution: does the bot have a strategy with ‘it’ and ‘they’ and things like that?

- spontaneous talk: does the bot take initiative to maintain the conversation when it seems to be dying out? Is there a mechanism for when NO input is provided?

- conflicting information: if one source/user says a fact, and another says something directly contradicting it, does the bot discard one of both? Does it have a strategy to choose which one? If the user says he’s 15 years old, and then that he’s 18 years old, does this raise any flags?

... Man, this list can go on for a while

Priorities, priorities
Given that I don’t know much about patterns, if “meanings” like CLASS_HAS_OBJECT form the core of Yoko’s workings, then it makes perfect sense to group all your patterns by meaning as you intend to. It should save you a lot of code-chasing later. If that means you also have to write a separate code for plurals that can be applied to all patterns, that’s all the better.
I found that the more things I could combine, the more flexible and powerful my program became, and the easier it was to debug.

Similarly, if possible, I would advise making global arrays like ‘var A = “a/each/every”’ that you can insert into your patterns “[A] [class] is [A] [parentclass]” (I don’t know if this is correct code, but you get the idea). That way, if later you realise that e.g. the word “one” should be in there too, you only have to edit one array instead of 1000 patterns.

You have very, very well understood how I am reasoning/implementing language understanding for Yoko.

Indeed I am separating out like that, and indeed one of the advantages I view this as having is that quirks of language are / will be easy to implement in the long run. Almost every grammatical feature (either on the ‘phrase structure’ and the ‘word’ level, but let’s focus on the atter) in English can be pretty accurately captured using a combination of:

- word shape, most easily expressed in regular expressions: (plurals: [noun]s)
- ‘regular’ exceptions to the rule, most easily expressed in regular expressions: words ending with ‘sys’ have the plural replace that with ‘ses’ or explicit lists depending on the case (list of nouns that stay the same in plural form)
- a list of individual exceptions. (ox -> oxen…)

... and that’s it. I just used plurals of nouns here for illustration, but verb conjugations, proper nouns, etc… will all fit into this list. Note there are some ‘edges cases’ relying almost entirely of one of the above 3.

For example proper nouns will be best captured using a big fat list of proper nouns, most easily categorized by boy names, girl names, company names, band names, geographical names. This is EXACTLY the type of lists and patterns I am using to build Yoko, and in the context of something earlier mentioned: this is also why I don’t rely that much on ‘learning language from visitor conversations’, except for inspiration in which cases to cover. The stuff described above is quite easy to scrape together from different sources online.

And that’s very, very encouraging for a lone chatbot wolf like myself