Hi
I’m working on a (semester wide..like 2-3 months) project along with my two project partners(just a reminder of time scale n human resource available).

In this project we are going to design a forum(possibly ready-made and lightweight like punBB which can be easily modified with needs) with AI bots(Intelligent Agents).
Those bots must be capable of

1)Detecting type of query (that is CONTEXT AWARENESS... whether given post is just a greeter..like HI/Hello msgs or Specific like “What are the configurations/models available in dell Laptops for gaming?”) 
2)Flagging of SPAM messages/posts. (This post is marked as spam)
3)In case of ambiguity take help from Human (Forum Admin in our case) & learn from this experience so that next time admin doesn’t have to reply manually.

Hope I summed it right…

Now problem we are facing
There are so many choices and we need to select the right combination to reach our target in given estimate.

We are thinking of integrating AIML based chatterbots with php based forums.
But we did’nt find any logical explanation (logical..don’t wanna dig into the code right now) AIML Interpreter.(Please point me if there are any good/easy information available about AIML interpreter.)
Will AIML suffice our need? (I mean NLP..sorry im not quite familiar with AIML methodology)
Is/are there any automatic tool for conversion of NL information to AIML?

Or

Using Natural Language Toolkit (NLT based on python http://www.nltk.org/ ) will be better option?

We are confused with many possible options available (don’t know which individual combination will work)
Any help will be greatly appreciated!

P.S. We are quite new in AI domain (As you have already noticed )
P.P.S. Forgive me if I posted this in wrong section or my post is too confusing.I’ll try my best to explain in subsequent posts.

Hi, Daisy, and welcome to chatbots.org.

Your post is in the right place, and it’s not all that confusing, either, so no worries there.

Now please bear in mind that, for what I’m about to say, I’m not an expert with any of this. Don’t get me wrong, I have a decent bit of background knowledge, but there are folks here who are far more educated and experienced than I am, and may have other, possibly better, ideas and suggestions.

First off, it’s a good idea to acquaint us with your skills and abilities on the technical side. I see you’ve stated that you’re not all that familiar with AI in general, but how proficient is your team with PHP and other web-based languages, and is anyone on the team at least a little familiar with forum software? Integration and modification of forum software can vary greatly in difficulty, depending on one’s familiarity with the chosen package, as well as the actual package itself. You mentioned PunBB (a script that I’m not familiar with, making it more difficult for me, just as an example), but there are other choices, as well, such as PHPBB, or SMF (packages that I am familiar with, again, just as an example). the package that you choose will greatly determine the outcome for your project, so it would be good to carefully weigh all options before making a decision.

Now as for choosing between chatbot software to integrate with your forum package, you have even more choices, but before you can choose which solution would serve best, you need to determine whether to use an AIML interpreter, a solution that uses NLP, or perhaps a hybrid of sorts, that combines the best of both options (I can say with some confidence that the third option here would be a new animal, since nobody I know of has created something of this nature). The current overall opinion here is that AIML is a good “First Step” for beginners, but that for truly “intelligent” conversation, you need to use a different sort of solution that involves NLP, and I (sort of) agree, up to a point. I feel that AIML has a great deal of untapped potential, and can do a great deal more than has been demonstrated so far, but my skills in this area aren’t up to the challenge of demonstrating the idea to best effect. However, that being said, the nature of your project isn’t exactly “intelligent” conversation, and more accurately falls into the category of an Expert System, so AIML may, indeed, be able to suit your needs.

Another possibility to consider, and one that may potentially be closer to that “third option” of combining NLP and pattern matching (though not AIML), is to use ChatScript. ChatScript was created by Bruce Wilcox, an active member here at chatbots.org, and is another example of untapped potential. It has been used as the engine for prize winning chatbots (Suzette won the Loebner Competition in 2010, and Rosette won just last year). Of course, there’s a bit of a learning curve here, but the basics of the language that ChatScript uses are simple to learn, so the only real challenge would be to integrate it with your chosen forum software.

No matter which path you choose here, there are folks that can help you, with advice and technical expertise, so don’t be afraid to ask questions. That is, after all, one of the reasons we all hang out here.

There is no ready made project that has the integration of NLP and a forum. But, If you are just going to do a proof of concept, then there may be a number of interesting opportunities.

Spam detection is a large topic on its own. It was covered in the Stanford AI course:
http://www.wonderwhy-er.com/ai-class/
look for section 8 Spam Detection
It talks about using a Bayes rules to detect spam.

One question I have is, do you need a full BB system that will live on after the class or is this project just for the course. If it is just for the course, then you could do some things that are more on the bleeding edge.

AIML is good at specific pattern matching but is not so good when you are doing “fuzzy matches”.
AIML Interpreters:
http://www.alicebot.org/downloads/programs.html

The background of your team will shape your project. If you give us more on the skills the team has we could point you in a better direction.

Merlin - Feb 24, 2012:

One question I have is, do you need a full BB system that will live on after the class or is this project just for the course. If it is just for the course, then you could do some things that are more on the bleeding edge.

Great point, Merlin.

Merlin - Feb 24, 2012:

AIML is good at specific pattern matching but is not so good when you are doing “fuzzy matches”.
AIML Interpreters:
http://www.alicebot.org/downloads/programs.html

This was more or less the point I was trying to make, though I realize now that I wasn’t all that clear about it. My bad for trying to post while enjoying the wonderful~ness that is a migraine.

Also, should you choose to use AIML for this project, I strongly urge you to consider Program O. Version 2.0 has recently been released, and it has some features that would be eminently suitable for forum integration. Of course, I’m just a bit biased here, since I’m a lead developer on the Program O project, but…

Merlin - Feb 24, 2012:

The background of your team will shape your project. If you give us more on the skills the team has we could point you in a better direction.

Thought I already said that, Merlin.

Another point to consider (and one that many folks don’t, most of the time) is that, while it’s great to have strong skills in whatever scripting languages are going to be used for the project, you also will need to have someone on-board that has at least a moderate degree of skill at creative writing. Speaking from personal experience, not many “coders” that I’ve worked with are all that good in the creative writing department, and creative writing skills are required when building responses and (more importantly) patterns to match. As an exercise to illustrate this, write down a set of 5 questions (of any nature), and give a copy to a creative writer, and one to a programmer, giving the same instructions: Write out as many different ways as you can to ask the same question. Then compare the results. I think you’ll be amazed at what you’ll see.

Interesting question ....

          “What are the configurations/models available in dell Laptops for gaming?”

meaning…...

(A) “What configurations are recommended for gaming on Dell laptops?”

(B) “What models of Dell laptops are recommended for gaming”

?

So removal of the word ‘are’, and realize the meaning (when ‘recommended’ is inserted).

I know it was just an example question Daisy (welcome by the way ), but your bot will have to realize the translation of that question so that it makes sense first.

Good example question though, I’m going to work on how my own approach will translate that to one of the more precise meaning.

Question - is (A) or (B) the true meaning of what you were wanting to convey? or something else.  I’m interested from a research point of view.

The main reasoning is— ‘models’ usually are ‘available’, but ‘configurations’ usually aren’t ‘available’, they are more ‘recommended’.... (that is, one more speaks of models being available or not, but you won’t say “is a configuration available?” . .you’d more likely say “is this or that configuration recommended or not…  . .for a specific purpose . .in this case, gaming).

That is a really good example question, thanks.

Also, the bot will have to know to try another possibility in order to determine the closest possible meaning of…

“configurations available in Dell laptops”

The bot will have to ‘guess’ and realize that when ‘in’ is replaced by ‘of’, it makes sense…

“configurations available OF Dell laptops”

We say ‘the configuration OF something’ (not configuration ‘in’ something).

These are the things the bot must know in order to have true semantic understanding.

Those are good points, Victor, and certainly something to consider, but given the stated goals of the project, I’m not sure how much priority should be placed on “true semantic understanding”.

You’re right Dave.  Perhaps though, if Daisy wanted to continue the work, or, later to take it to the next level

Firstly, Thanks all for replying to my queries.I’ll try to respond chronologically.

My team’s strengths

As I said there are 3 members in all in our group.I’ll summarize their skills both from technical and non-technical point of view.
Firstly ME,
I have descent skills in php (well not the best to hack into Facebook’s).Also I’m in charge of researching/surfing/reading new stuff for our project and conveying it to my team.Also Im trying to “Dive into python” and taking lessons from http://code.google.com/edu/languages/google-python-class/

Then my SECOND partner, he is a very good “OUT OF BOX” thinker ..also a strong coder n has nice background with databases n mathematics (probability , Combinatorics stuffs).

My Third partner is decently experienced in Java and design stuff like uml and same goes with maths skills n Also the person to cool off when things go out of hand and when we go nuts.

Well I think its fair combination for our projects needs.

Why I chose punBB instead of phpbb3?
I don’t need all complex/addon features which phpbb provides.Just post/reply topics and few basic forum features will do.We are going to strip down punBB as well(hope they don’t use any binary or compressed code) to our needs.To be honest, the SIZE of the files cause me to choose punBB over phpbb (both BBs are new to us so we started with the ligher/smaller one).

do you need a full BB system that will live on after the class or is this project just for the course?
Answer depends on how much we can perceive and fathom in our goals.
If we are going with AIML and aim for the END PRODUCT usable by customer then we must find a way to convert Given INFORMATION IN NL to AIML structure.
(NL Infomation——->AIML files) with some automated gui based module.
(We cant ask end user to handcode each AIML file).

If we can’t then we must take some test cases (black box) and provide more sophisticated functionality to our project (bleeding edge?) from Research point of view.
Here we may consider using Hybrid methods/chatscripts/NLTK in natural language processing.As well as “true semantic understanding” as DAVE and VICTOR were talking about.

Also ,does anyone know about “Chomskyan linguistic superstructure” ?
I read about it here http://www.ai-forum.org/topic.asp?forum_id=1&topic_id=16699
in Arthur’s post. Sounds interesting.

Again, How AIML interpreter works? (or its just stupid question ).Logical explanation would be great.
Configuring those various bots gave me hard time(most of the functions were deprecated and badly documented) but finally installed program-O bot successfully on XAMPP!

Hope I answered all your queries.Sorry if I’m missing any.

The way an AIML interpreter works is quite simple in theory, but somewhat complex in execution. First, you have a repository of categories, each with only one input pattern, and one template (output). The interpreter takes a user’s input, and searches the repository for the best match to the input, based on several factors, such as whether the input exactly matches a particular input pattern, or if the pattern has wildcards (very important to have, or you would be adding hundreds and hundreds of thousands of categories to catch every possibility), etc. Once the closest pattern is found, then the template is processed. This template can contain simple text, or perhaps certain “tags” that can represent different variables for either the bot or the user (or even the time of day), or it can contain special tags to “redirect” the output to other categories, either instead of, or in addition to, the currently processed template. There are even condition tags, to change the output based on the value of a given variable (again, such as the time of day), and much, much more. This is why I say that AIML has a lot of untapped potential, as all but a very few AIML botmasters use these various features to their full potential.

I hope this helps to answer your questions. I’ve got an errand to run, so I’ll get into what I know of some of the different interpreters at a near, but later, point.

The simplest way to do your project might be to use a forum and the “bag of words” model to return a canned response to post on the forum. Especially since one of your team understands probability.

http://en.wikipedia.org/wiki/Bag_of_words_model

One step beyond this would be to use topic models to characterize blog posts:
http://en.wikipedia.org/wiki/Topic_model
http://www.cs.umass.edu/~mccallum/courses/gm2011/20-lda.pdf
http://www.cs.umass.edu/~wallach/talks/topic_modeling.pdf

JAVA Topic Modeling Tools - MAchine Learning for LanguagE Toolkit (MALLET)
http://mallet.cs.umass.edu/topics.php

The limitation of this is that it is not really used as a conversational engine like you could get with chatbot software. But it would allow you to characterize posts which you might then use to drive responses.

If you are looking for a more conversational approach in the forum then use chatbot software.
AIML is easier but less flexible in recognizing patterns. It does have large sets of canned patterns and responses.
ChatScript is more flexible but more “programmatic”.

If you want more “bleeding edge” ideas and know JavaScript, let me know.

I think you will find it much more difficult to translate Arthur’s concepts about “Chomskyan linguistic superstructure” into viable project code in the deadlines you have for your project.

Here is a little history on forumbots.  Program E, created by Paul Rydell, is an ALICE AIML chatbot which was modded for phpBB version 2.0. When phpBB version 3.0 was released, it was very different from phpBB 2.0. 

So unfortunatley the phpBB version 2.0 forumbot mod was incompatible with the phpBB new version 3.0.  The Program E forumbot mod was enormously popular. It is considered the best forumbot mod ever made because it used the forum, admin panel, private message system and could even send email from the forum, according to a funny YouTube video demo.

Thanks all again for replying so quickly.
We were thinking of using AIML chatterbots with automatic AIML generator script.
The Script (mostly based on Natural language Toolkit http://www.nltk.org) using python language which will generate FAQ and GLOSSARY format from given INFORMATION.
I read one paper about the same..here’s its link http://je-lks.maieutiche.economia.unitn.it/index.php/Je-LKS_EN/article/viewFile/414/404
It talks about how we can automatically generate AIML KB(Knowledge Base) if provided format is in FAQ and GLOSSARY(Something like a movie script).

As far “Topic Model” and “Bucket of Words” classification, we are working on it as well.
I’m just not sure which one to choose and how far we must be focused on NLP.
I’ll get back soon with chosen method.

Daisy,

That’s awsome, Daisy!  Thanks for that really cool link.

Glossary?  Ha, ha… Wow!  How about a giant glossary, better known as… A dictionary?
I just taught my chatter robot Vira ( http://8-I.us ) over a hundred thousand dictiionary definitions by uploading her robot brain database with an open source dictionary written in XML.

So, an AIML/RiveScript/Mitsuku A.I./ProgramO engine integration of php and perl cgi-bin with a hundred and fifty thousand records may losely be an example of the good point Dave Morton made on this thread.  And I would like to give a shoutout to Victor again, because Perl is a powerful language for A.I. , and after reading about Victor’s success I started coding in Perl again, after spending a few years on PHP.

Lastly, I built a light weight, basic functionality forumbot for punBB on my localhost. I may upload it to the web at some point.  A note about python… It is great, and I fully support it, but it tendd to be less quick than other options.

8PLA • NET - Mar 1, 2012:

I just taught my chatter robot Vira ( http://8-I.us ) over a hundred thousand dictiionary definitions by uploading her robot brain database with an open source dictionary written in XML.

  You ► What does “lottery” mean?
Vira ► I will search for it. 
  You ► what does the word “destroy” mean?
Vira ► I may try to find out. 

Am I phrasing the question incorrectly?

  You ► What does lottery mean?
what does the word destroy mean?

Vira ► May there be only one? You tell me. Lottery ... A scheme for the distribution of prizes by lot or chance esp a gaming scheme in which one or more tickets bearing particular numbers draw prizes and the rest of the tickets are blanks ....

  You ► what does the word destroy mean?
Vira ► Interesting question. 

( Not sure why she ignored “destroy”, so I’ll have to investigate. )

Thank you so much Steve, I really appreciate your chatter robot expertise!  I hope you are in first place this year. You deserve to be the winner.  That reminds me to add a regular expression for alphanumeric only to strip off quotes etc.  Thanks for finding that for me.  I also want to add more animation.  These contests are so much fun, but they are a lot of work.  Thanks again for the tip! That is a clear example of good sportsmanship on your part.