AI Zone Admin Forum Add your forum

NEWS: Chatbots.org survey on 3000 US and UK consumers shows it is time for chatbot integration in customer service!read more..

RSS AI Zone  Forum Home  >  Development  >  ChatScript  >  Thread
Tokenizing Issue
  
 
Posted: Jul 25, 2013

I have written the following rule:

#! Is Mnajdra Temple really the oldest structure in existence?
u: (<< be [Mnajdra mnajdra] [“oldest” “most ancient”] >>) aaa

Now in theory, this should work. However, :prepare demonstrates that there is a tokenizing issue preventing it from verifying properly. Here is a screenshot: http://i.imgur.com/ZyMwYUl.png?1

In summation:

is : mainverb
mnajdra_temple_the : mainsubject
structure : mainobject
existence : object2

How can I get it to tokenize in a manner where mnajdra is isolated? I’ve tried creating a concept like so:

concept: ~mnajdra (Mnajdra mnajdra)

But it doesn’t fix anything. Any suggestions? I apologize if there is an obvious fix that I’m missing.
 

 
Posted: Jul 25, 2013
   [ # 1 ]

Currently you have, by default, proper-name merging turned on using $token = ...

If you don’t want it to treat a series of words which in fact represent a single entity, then you should turn it off.
Otherwise, accept it and use it - add the full form to your concept: 
concept: ~mnajdra (“Mnajdra Temple” Mnajdra mnajdra)
u: (<< be ~mnajdra [“oldest” “most ancient”] >>) aaa

which would accept “Is mnajdra the oldest temple?” as well as “is mnajdra temple the oldest”
 

 
Posted: Jul 25, 2013
   [ # 2 ]

Also, your screenshot shows you doing :prepare ~mnajdra   as an input, which is meaningless.
The user is not going to type that in. :prepare is used to show how the system is handling user input.
No user will enter a concept name as input.
 

 
  login or register to react
‹‹ escaping underscore      flow diagram for processing ChatScript ››