I have a question.  Are the people here that make bots always able to figure out why their bots make mistakes?  Can you always pinpoint the reason a sentence is parsed or understood incorrectly?

Oh my GAWD, no! Trust me; as a botmaster, I’ve lost not only time and patience, but also hair, sleep, weight, and sometimes even the will to live, looking for a cause to why Morti was constantly ending his sentences with the word, “it”! He just started doing it one day, for no apparent reason. I tried and tried to locate the source of the problem, and even resorted to wiping his database, and his script, and starting over, all to no avail. then one day he stopped, and he hasn’t done it since. I still have no clue why. I’m gonna need therapy over that for a LOOOOOOONG time!

Ah, ghosts in the machine. I remember hearing about the “it” issue, lol. I didn’t realize it just “disappeared”. How strange. (And a little spooky! )

I mostly have problems with unintended consequences of algorithms. Occasionally I’ll get some odd parses and have to trace back what rule created the monstrosity. The problem is generally incompleteness or too much generality on my own part, but it can be frustrating to sort out.

My earliest version of a grammar training interface printed the line “Good! One moment while ALEX parses your sentences…” after the user input text. When I decided to get rid of it, I simply could not find where it was written. That sentence mocked me for about two months until I stumbled upon the line of code! Now I make sure all of my output contains a reference to the file that produced it.

I’ve pretty much re-vamped the debugging script for Morti, for exactly the same reason. When I’m running the debugger, it now spits out almost as many lines of info as there are lines of code! But reading through the debug file gives me an exact map of the script’s execution, including timestamps for each section of code. It comes in handy for tracking down all sorts of troubles. I wish that I had done that before the “it” fiasco. It would have saved me a lot of grief, I think.

Are the people here that make bots always able to figure out why their bots make mistakes?  Can you always pinpoint the reason a sentence is parsed or understood incorrectly?

That’s a good question. Personally, I’ve made it part of my believe system that all bugs can be found. It allows me to put my teeth into something and hang on for dear life until I get where I wanted to!
Sometimes, I’m just lucky and see the error right in front of me without having to look for it. Other bugs need to be hunted down for months before they finally reveal themselves. Some algorithms are just difficult ‘as a whole’. I rewrote the flow recognition algorithm about 20 times. In all, it took me somewhat 6-7 months (for 200 neurons or so) before it was working.
But in the end, the threading bugs are, without a doubt, the most difficult. You have no idea how frustrating it is to have a correct result 99% of the time with a huge fail for exactly the same input 1% (in various places, for no apparent reason). These types of bugs can’t be ‘seen’ in the code. The only way I have been able to solve these, is by making a complete mental picture of the process (like imagining a clock-work mechanism) and going over it again and again (trying to imitate parallel processing in my mind) until you find the problem. The fact that a bug can be in different layers of the application (neural code, interpreter, designer,...) doesn’t make things any easier either.
At the moment, I still have 1 left (please , let it be just 1   ), which mysteriously doubles the result on occasion.  Ah well, hunting season never ends for debuggers, does it!

You have no idea how frustrating it is to have a correct result 99% of the time with a huge fail for exactly the same input 1%

Yeah I do, I’ve done lots of debugging in C# with 4 other languages being used concurrently.  SQL, ASP, AJAX, JavaScript.  Layers of languages that don’t always agree with each other.  Things that defy logic.  A capital letter deep within an XML file for SQL that should be lowercase.  Using an == sign where it will give the right result most of the time where I needed =.

What I was most interested in was parsing “errors” and how you correct them, and whether it seems absurd at all to keep adding rules.  When do you stop adding rules and change the upper level logic?

Toby Graves - Feb 11, 2011:

What I was most interested in was parsing “errors” and how you correct them, and whether it seems absurd at all to keep adding rules.  When do you stop adding rules and change the upper level logic?

I have an excellent participial phrase extractor* that works amazingingly well, except for certain cases when it completely implodes. I figured out when and why the algorithm fails—that part wasn’t especially difficult—and decided that the fail cases exemplified a class of sentences that required going back to the “upper levels” so to speak, and changing the entire extractor (including how it’s called).

Since I’m still going through specific modules and error correcting, it’s not so difficult to find the source of parsing errors. Occasionally the source will be something unrelated and surprising, but that’s only happened a few times. I’m sure the task will be much more difficult once the various modules are working more in unison.

*It’s a bit more generalized than participial phrases, but that’s a whole other banana.

What I was most interested in was parsing “errors” and how you correct them, and whether it seems absurd at all to keep adding rules.  When do you stop adding rules and change the upper level logic?

Flow recognition algorithm = upper level logic
Flow definitions = rules
Threading bugs = parsing errors caused in upper level logic.

Toby Graves - Feb 11, 2011:

<blockquote>
What I was most interested in was parsing “errors” and how you correct them, and whether it seems absurd at all to keep adding rules.  When do you stop adding rules and change the upper level logic?

I find following types of parsing errors.

[ul][li]New Inputs - stuff the bot never encountered before and should handle. (the use of “add” instead of plus in a math problem- “what is two add two”)[/li]
[li]Marginal Inputs - Things caught by a general rule, but if they were more common I would create a more specific rule for.(Topics the bot is not interested in)[/li]
[li]Hard Input - Things I haven’t figured out how to code but need to. (Word Math problems)[/li]
[li]Shadowed Input - Things that are picked up by a higher priority rule that would be better handled by a lower priority one. (Change priority or crisper rule)[/li]
[li]Bugs - Bad rules, bad output, coding errors[/li]
[li]Bad language - spelling and grammar errors[/li][/ul]

As the AI expanded I added a Debug mode also that allows me to see the exact path from input to output. For Skynet-Ai, data is added at an average of about 10k/month. Each version includes refactoring and introduces new bugs as the code path changes.

I usually find the problem, but being an animist at heart, I think bots can just have a mind of their own.

I have actually created a set of “Troubleshooting guides” for my system.

So, if you see this error, of if the bot is behaving this or that way, see this document.  It starts basically with a flow chart of “symptoms”.

So far so good.  Most of the time, the “problems” weren’t problems with the bots core logic, but bad word data.

Once, initially, the database had it that “too” and “to”—I had “to” as adverb and “too” as preposition !  Of course it is the other way around.  Thus, it had problems generating some parse tree options I have since made the bot tolerant of that.

You can even use “2” in place of “too”,“two” or “to”, it doesn’t care (although an option I will have in it later, it can correct you, if you like).

Patti Roberts - Feb 12, 2011:

I usually find the problem, but being an animist at heart, I think bots can just have a mind of their own.

hehe.. that’s the idea

That’s one of my “pet peeves” that I’m having to learn to live with, in that it actually bothers me when people type “to much info”, or “I went too the store”. I know that it’s a minor thing, but it grates on the back-side of my soul like I’m sliding down a banister that’s wrapped in barbed-wire. And don’t even get me started with “than/then”...

Anyway, I should seriously consider creating a “troubleshooting guide” for my chatbot script. One could really come in handy, right about now.

Dave—thanks for the chuckle.  That was great.  Your posts are always good.  I used to really get a kick out of what Chuck used to come out with.  Hope he returns to chatbots soon.

Anyway, yes, agreed.  two/to/too/2 : what we have to live with as chat bot developers!

This actually has me a bit “worried” (too strong a word, perhaps, but it’s more than just “concerned”), with respect to this year’s CBC. It plainly states in the rules that “netspeak” shorthand will likely be used, and since I steadfastly refuse to use such a horrible grammatical construct, I don’t know how well Morti will fare. But this is getting away from the topic, so I’ll shut up now.

no, I think this is on topic (well, your thread Toby, so if it’s not on topic, nuke it Dave) 
Grace knows by context which you mean (if she sees ‘to’ (as adverb—which it is not, but she checks anyway) modifying adjective like ‘late’ , then it is obvious you meant ‘too’).  Of course she doesn’t check that it is the literal string ‘to’ (instead if it is a synonym of ‘to’.  Everything is done by synonym, there is absolutely no hard coding of any text.  But Dave, for you, I will have it scream at them!

That’s one of my “pet peeves” that I’m having to learn to live with, in that it actually bothers me when people type “to much info”, or “I went too the store”.

It took me a while before I found the errors.
From a developer’s point of view, I can completely understand you. Though I must admit, as a person, I am of the other type. No doubt, this has all to do with my early childhood years: As a toddler, I used to watch my ‘big sisters’ study and write in their books. Naturally, I wanted to imitate them, so I wrote books full of gibberish (my first inveted language ). Later on, I did a similar thing with English. I had picked it up from here and there, but couldn’t write it, so I began writing it phonetically.  This was a big problem later on, when I had to unlearn everything at school. And it never got straitened out properly, I’m afraid. I once managed to cramp in 56 spelling errors or so, in a dictation of 5 lines. Spell checkers have been a godsend for me   But, they don’t pick up things like ‘then’, ‘than’, ‘of’, ‘off’,... and it’s all the same to me.