If your bot has only a few conversations per day, then looking through the conversation logs is pretty easy.  However, if you have lots of conversations per day (or whatever period you choose to monitor), how do you go about it? 

We have found that looking through each conversation is tedious at best, and downright annoying at worst (cybersexing or gibberish).  To this end, we implemented a sort of high level visual filter to enable quick filtering of conversations- allowing us to see each conversation at a somewhat high level including user name, lines of conversation, and stop-word subtracted input word list. A more detailed view is shown for each individual conversation giving word-stop subtracted frequency and ranking and some other meta info.

conversation list

detailed conversation (the left hand conversation has high repetition = uninteresting, while the one on the right has low repetition = interesting)

We have found this very useful for tweaking an otherwise auto updating bot.

What are others doing?

Hey, Carl.

Right now, Program O just uses a simple DB query, with the only search parameter being the user_id. A system such as you describe would be awesome, but it sounds like a bit of time/effort to implement, at first. When I get to the point where I can investigate something of this nature more fully, I’d like to have a chat with you about it.

I look at them in the following order:

1 - The Turing Test logs
2 - People talking to the bot from a competition website
3 - Known users who contribute greatly
4 - Logs where people have tried to teach the bot (I get an email of each learned fact and who taught her it)
5 - Logs between 50-100 interactions
6 - Recent logs with more than 25 interactions

I rarely get past stage 4 unless I have a lot of time on my hands.

RICH has a management console that lets you manipulate logs which is pretty handy. Its fairly basic, but it lets you search by keyword or keyphrase, either question\statement or response. It also allows you to import exchanges as various things. Going through the logs has been interesting and there are a couple of things Ive noticed.

1. the weird tendency for people to attempt to communicate with you, through the logs, while ignoring the perfectly sound “Contact us” email app. ?

2. An flaw in Alan Turings test becomes glaringly obvious. Without a benchmark having been established at some point in the past, a successful Turing test does not necessarily represent an advancement in the machines intelligence.

It sould just as easily mean that people are becoming dumber…

All of my logs are downloaded into .TSV files.
Each “volley” contains an IP address, date, time, browser type, User Input, and Bot Response.

I take single volleys that should have been responded to differently and copy them to a separate file. I later go back and make the modifications. When I have a time that I want to work on them, I go through them and change the bot. I usually sort the log by Conversation, User Input or Bot Response, depending on the focus. If I am working on a topic, I may pull all the related inputs of a certain type (and also go through past logs), and try to create general rules to handel the situation. I can collect these either by keyword search, full regular expressions or programatically.

Merlin - Apr 20, 2013:

All of my logs are downloaded into .TSV files.
Each “volley” contains an IP address, date, time, browser type, User Input, and Bot Response.

I take single volleys that should have been responded to differently and copy them to a separate file. I later go back and make the modifications. When I have a time that I want to work on them, I go through them and change the bot. I usually sort the log by Conversation, User Input or Bot Response, depending on the focus. If I am working on a topic, I may pull all the related inputs of a certain type (and also go through past logs), and try to create general rules to handel the situation. I can collect these either by keyword search, full regular expressions or programatically.

We do the same, though it is through a somewhat combursome process of downloading either a parsed (by some keword(s)) conversation log (input, response, topic, it, name, index, date, userID) or the “unknown input” log (logs all input where no specific response was generated) from mySQL -> excel as csv.

The conversation log (~700,000 volleys in the active log which covers only the past 12 months) is very useful for seeing frequency/variations of certain keyword or patterned input (pretty easy to do in excel). 

Unknown inputs are useful for dealing forward with spelling and oddly patterned input (often non-native english grammer patterns). 

What would be ideal at this point is something more dynamic though, as the mySQL -> excel part is pretty tedious, and should be able to be automated, given the appropriate time and motivation of course.