For a while I have been developing a series of methods for extracting information from the Internets with the goal of having a real time synthesis of facts and opinions that can be presented to a user prompted query via a bot interface.

The ultimate goal of was to be able to “scrape” the Internet, use the parsed data to extract useful information, and then present that back to a user in a coherent and variable (non repetitive) textual response.

While it is generally straight forward to have input searched for key words or phrases and match a response pattern from a db, it is much more challenging to get a match, let alone programmatically generate a coherent textual response, to the input via real-time information analysis.

This feat has now been achieved.  A bot that accepts virtually any kind of input, and replies using real time analysis of the Internets (more on that later).  NLP of input has been refined using the most advanced processors available (e.g., Google).  Output is parsed, has sentiment analysis performed, and then is reassembled into a unique response.

This may be the closest to “thinking” that any bot can currently achieve.

Examples:

User Query1:
“what is most important in life”

Bot Response1:
I think that Simplicity most important in life choose the two most important. When you finish one add another from your list. That is what I think I understand, generally what is most important in life sounds more good than bad [mood score: 0.28].

User Query2: (repeat same input)
“what is most important in life”

Bot Response2:
My opinion is that Simplicity most important in life are completely useless. Resentment bitterness hate and jealousy have never improved the quality of life for a single human being. This is the consensus of my information- that what is most important in life sounds more good than bad [mood score: 0.32].

User Query3: (pseudo gibberish input)
“what are yur best memarries?”

Bot Response3:
“From what I can understand, The best memories ever after Frank Lampard equalled his goals tally of 175 EPL goals the other day and is likely to surpass it soon enough with Wayne Rooney Andy Cole and Alan Shearer also having scored more. Frank’s achievement is fantastic especially for a midfielder but he did take 588 games to get there while Henry took just 254 games. All those others played their whole careers in England and only Shearer’s overall record compares with the great Gunner. Graphic supplied by Opta powered fantasy football game www. This is the consensus of my information- that what are yur best memarries sounds REALLY good! [mood score: 1.16].”

****Repeating the query, another unique response it given****

User Query4 (repeat same input again)
“what are yur best memarries?”

Bot Response4:
From what I can understand, What best memories home and was like fuck this. So I just got a pringles tube as the neck plastic cups as the legs two plastic bowls for the feet and a cereal box full of tissue paper as the body. Celetaped them all together and put two layers of newspaper on the top. I then brought it in and showed my teacher and I told her I finished early she was very pleased and just let me get on with any other work I had. This contiuned for a few weeks and while everybody else was slaving away there I sat with. This is the consensus of my information- that what are yur best memarries sounds REALLY good! [mood score: 1.14].
User Query5
“what do you think about ISIS”

Bot Response5:
“From what I can understand, ISIS the 5 percent of adult nationals who support From what I can understand, ISIS means over half a million people. And even in tiny Lebanon 1 percent of adult Sunnis equals a few thousand ISIS sympathizers. In any of these places this is enough to harbor at least a few cells of serious troublemakers. Another major caveat is that the nearly uniform opposition to ISIS does not extend to other political Islamist organizations. In Egypt for example a surprisingly high proportion. How confusing it all seems, but overall ISIS sounds more bad than good [mood score: -0.06].”

User Query6
“how do you feel about the poor”

Bot Response6:
My feelings concerning the poor, No, the best ways of checking our hearts for signs of money slavery is to evaluate the way we react to people whose financial situations are different from ours. Last week we looked at warning signs in the way we look at the wealthy. Today’s study uses the same approach in evaluating the way that we think about the poor. That is what I think I understand, generally how I feel about the poor is REALLY bad! [mood score: -1.44].

User Query7 (repeat input)
“how do you feel about the poor”

Bot Response7:
From scanning the data stream, I think my feelings concerning the poor breeds contempt in our hearts. Contempt makes us angry at poor people and can even make us think that they deserve to live the way they do. Contemptuous people come to resent the way that the poor burden our society or that we pay taxes help. The correct response to poverty is not pity but compassion. Compassion sees a need and reacts to fulfill it. This is the consensus of my information- and how I feel about the poor is REALLY bad! [mood score: -1.44].

While there are obvious improvements to be made to clean up the output even more, the output as it is can be pretty interesting- uniqueness, mirroring of poor grammer/spelling in the second sample input, etc.

I should have this with a public front end in the near future.

Carl, that looks very promising, but I’m sure you understand when I say that I’ll hold off on a more detailed response until I’ve had a chance to do a more “hands on” examination.

Dave Morton - Dec 17, 2014:

... do a more “hands on” examination.

Good lord…

INPUT: “I’ll hold off on a more detailed response until I’ve had a chance to do a more ‘hands on’ examination.”

Bot Response: “My opinion is that I ‘hands on’ examination first made it worse. Went back given Cetirizine 3 a day Ranitidine one a day plus Monteleukast if that didn’ t work. Before I had the chance to even work some even made it worse. But my skin remained itchy all over. How confusing it all seems, but overall I will hold off on a more detailed response until I’ve had a chance to do a more ‘hands on’ examination. sounds REALLY bad! [mood score: -1.36].”

Test bed: Thinking bot

Thanks for the link, Carl. I’ll take a look shortly. My main computer is down due to a catastrophic intelligence failure on my part (long story, and NOT pretty!). so I
m stuck with my laptop for now.

That is really interesting work Carl. Good job.

Vince

First volley: input = “My hip is hurting a little”

Number of valid links: 5
page not retrieved, break here
Internet query OK
page too long, so truncate it…
Internet query OK
page too long, so truncate it…
Internet query OK
page too long, so truncate it…
para_count > [temperature], break out of annealing loop!
interations to get results_arr: 3
Original return results_arr length: 200
Count of training_sentence_arr: 2
Count of training_text_arr: 24
Number of sentences: 2
Number of sentences used: 2
Highest cumulative score: 0
Lowest cumulative score: 0
Median (of hi and low) cumulative score: 0.00
Final score: 0
Normalized per sentence score: 0.
Number of sentences in training_text_arr: 24.
First word or phrase: My hip is hurting a little
Begin Genetic Breeding Synthesis on ‘training_text’, degree: 20, Output length: 200

Final text: My opinion is that Medical My hip is hurting a littlenews information and insights from Cleveland Clinic s experts designed to help people make quality decisions about their healthcare. people make quality decisions about their healthcare. help people make quality decisions about their healthcare. That is what I think I understand, generally My hip is hurting a little sounds neutral… [mood score: 0].

Second volley: same input (just for giggles)

Number of valid links: 5
page not retrieved, break here
Internet query OK
page too long, so truncate it…
Internet query OK
page too long, so truncate it…
Internet query OK
page too long, so truncate it…
Internet query OK
page too long, so truncate it…
Internet query OK
page too long, so truncate it…
para_count > [temperature], break out of annealing loop!
interations to get results_arr: 5
Original return results_arr length: 200
Count of training_sentence_arr: 2
Count of training_text_arr: 24
Number of sentences: 2
Number of sentences used: 2
Highest cumulative score: 0
Lowest cumulative score: 0
Median (of hi and low) cumulative score: 0.00
Final score: 0
Normalized per sentence score: 0.
Number of sentences in training_text_arr: 24.
First word or phrase: My hip is hurting a little
Begin Genetic Breeding Synthesis on ‘training_text’, degree: 20, Output length: 200

Final text: From scanning the data stream, I think Medical My hip is hurting a littleexperts designed to help people make quality decisions about their healthcare. How confusing it all seems, but overall My hip is hurting a little sounds neutral… [mood score: 0].

Third volley: input = “Tell me about Ted Danson”

Number of valid links: 5
Internet query OK
page too long, so truncate it…
para_count > [temperature], break out of annealing loop!
Internet query OK
page too long, so truncate it…
interations to get results_arr: 1
Original return results_arr length: 200
Count of training_sentence_arr: 47
Count of training_text_arr: 824
Number of sentences: 47
Number of sentences used: 22
Highest cumulative score: 1
Lowest cumulative score: -3
Median (of hi and low) cumulative score: -0.09
Final score: -0.23
Normalized per sentence score: -0.23.
Number of sentences in training_text_arr: 824.
First word or phrase: Tell me about Ted Danson
String shortened to 4000 chars
Begin Genetic Breeding Synthesis on ‘training_text’, degree: 20, Output length: 200

Final text: This is the consensus of my information- that Tell me about Ted Danson sounds more bad than good [mood score: -0.23].

Final volley (for now): Same input

Number of valid links: 5
Internet query OK
page too long, so truncate it…
para_count > [temperature], break out of annealing loop!
Internet query OK
Proceed to ‘extract contents’ fxn…
Internet query OK
Proceed to ‘extract contents’ fxn…
Internet query OK
page too long, so truncate it…
Internet query OK
Proceed to ‘extract contents’ fxn…
Internet query OK
page too long, so truncate it…
interations to get results_arr: 5
Original return results_arr length: 200
Count of training_sentence_arr: 39
Count of training_text_arr: 691
Number of sentences: 39
Number of sentences used: 19
Highest cumulative score: 1
Lowest cumulative score: -3
Median (of hi and low) cumulative score: -0.11
Final score: -0.06
Normalized per sentence score: -0.06.
Number of sentences in training_text_arr: 691.
First word or phrase: Tell me about Ted Danson
String shortened to 4000 chars
Begin Genetic Breeding Synthesis on ‘training_text’, degree: 20, Output length: 200

Final text: That is what I think I understand, generally Tell me about Ted Danson sounds more bad than good [mood score: -0.06].

First impressions:

Overall, I think that what you have here is promising, Carl. I like that it performs sentiment analysis, and that it endeavours to come up with different outputs for the same input. I was a little dissapointed by the lack of “meaningful” responses to the Ted Danson queries, but I just put that down to it still being early in the dev cycle. Obviously, the current incarnation of your project is not meant to carry on conversations, but that’s ok. I see clear signs of limited cognition, and the potential for “greatness”. I’ve got no time today to do more (I have to take some family to the airport today, and my desktop computer is still a bit “iffy”), but I’ll play with it over the weekend, and give a more detailed review after.

Carl B - Dec 18, 2014:

Test bed: Thinking bot

>Terrorists are bad.

Number of valid links: 4
Internet query OK
Proceed to ‘extract contents’ fxn…
Internet query OK
Proceed to ‘extract contents’ fxn…
Internet query OK
Proceed to ‘extract contents’ fxn…
Internet query OK
Proceed to ‘extract contents’ fxn…
Internet query OK
Proceed to ‘extract contents’ fxn…
Internet query OK
Proceed to ‘extract contents’ fxn…
interations to get results_arr: 5
Original return results_arr length: 200
Count of training_sentence_arr: 15
Count of training_text_arr: 259
Number of sentences: 15
Number of sentences used: 7
Highest cumulative score: 0
Lowest cumulative score: -3
Median (of hi and low) cumulative score: -0.43
Final score: -0.72
Normalized per sentence score: -0.72.
Number of sentences in training_text_arr: 259.
First word or phrase: Terrorists are bad.
Begin Genetic Breeding Synthesis on ‘training_text’, degree: 20, Output length: 200

Final text: That is what I think I understand, generally Terrorists are bad. sounds more bad than good [mood score: -0.72].

>that is bad.

Number of valid links: 5
Internet query OK
page too long, so truncate it…
Internet query OK
page too long, so truncate it…
interations to get results_arr: 1
Original return results_arr length: 200
Count of training_sentence_arr: 1
Count of training_text_arr: 1
Number of sentences: 0
Number of sentences used: 1
Highest cumulative score: 0
Lowest cumulative score: 0
Median (of hi and low) cumulative score: 0.00
Final score: 0
Normalized per sentence score: 0.
Number of sentences in training_text_arr: 1.
First word or phrase: that is bad.
Begin Genetic Breeding Synthesis on ‘training_text’, degree: 20, Output length: 200

Final text: How confusing it all seems, but overall that is bad. sounds neutral… [mood score: 0].

>How do I start my car?

Number of valid links: 4
Internet query OK
Proceed to ‘extract contents’ fxn…
Internet query OK
page too long, so truncate it…
interations to get results_arr: 1
Original return results_arr length: 200
Count of training_sentence_arr: 10
Count of training_text_arr: 143
Number of sentences: 10
Number of sentences used: 3
Highest cumulative score: 0
Lowest cumulative score: -2
Median (of hi and low) cumulative score: -0.67
Final score: -1.34
Normalized per sentence score: -1.34.
Number of sentences in training_text_arr: 143.
First word or phrase: How do I start my car?
Begin Genetic Breeding Synthesis on ‘training_text’, degree: 20, Output length: 200

Final text: That is what I think I understand, generally How do I start my car? sounds REALLY bad! [mood score: -1.34].

Interesting “thinking”. Thanks for the test drive.

Since it currently is not connected to the chatbot, the initial parsing is only cleaning fxns (no support for the usual “that”, “topic”, “it”, “variablexyz”, etc.);  there is no pre-processing or conversation caching done with this proof of concept test bed program. Therefore, each query stands alone, and the series of inputs “Terrorists are bad.”->“That is bad” are not yet programatically connected (as they would be when these methods are integrated into the chatbot).

btw- The last of the debugging code has just been commented out.

Once this gets integrated into the chatbot, the results should get much more conversational.

“That is what I think I understand, generally Tell me about Ted Danson sounds more bad than good”

This is basically a NULL response where the parsed input somehow failed to recongeal into a response… I am still trying to get this to be more rebust (less NULL responses, better gibberish filtering).

“Tell me about Ted Danson” Since there is no preprocessing (which would convert this into something like “who is Ted Danson”, this type of input yeilds much lower quality output. That should be addressed when integration into the chatbot is done.

User Input: who is Ted Danson

Bot response: My opinion is that Edward who is Ted Danson has also been a longtime activist in ocean conservation. In March 2011 he published his first book Oceana Our Endangered Oceans and What We Can Do to Save Them written with journalist Michael D’Orso. Danson was born in San Diego California the son of Jess Danson n e MacMaster and Edward Bridge Danson Jr. an archaeologist and museum director and raised in Flagstaff Arizona. 4 In 1961 he enrolled in the Kent School where he was a basketball star. That is what I think I understand, generally who is Ted Danson sounds more bad than good [mood score: -0.06].

Though actually, when I just entered your original input I got this:

User Input: Tell me about Ted Danson

Bot response: My opinion is that Edward Tell me about Ted Danson has been nominated for 15 Primetime Emmy Awards winning two ten Golden Globe Awards nominations winning three one Screen Actors Guild Awards nomination one American Comedy Award and a star on Hollywood’s Walk of Fame. He was ranked second in TV Guide’ s list of the top 25 television stars. 3 Danson has also been a longtime activist in ocean conservation. In March 2011 he published his first book Oceana Our Endangered Oceans and What We Can Do to Save Them written with journalist Michael D’Orso. Danson was born in San Diego California the son of Jess Danson n e. That is what I think I understand, generally Tell me about Ted Danson sounds more bad than good [mood score: -0.31].

Not sure what is driving the negative mood score for Ted, but that whole sentement analysis part still needs some polishing…

First time I’ve asked:

User Input: what is the ultimate question

Bot response: My opinion is hence Deep Thought made the plans for the supercomputer Earth that would solve the Question in ten million years. The Ultimate Question is the actual question behind the Ultimate Answer of Life The Universe and Everything. The Ultimate Question was sought after the supercomputer Deep Thought revealed the Ultimate Answer to be 42. When Deep Thought asked Loonquawl and Phouchg were unable to say what the actual question was. That is what I think I understand, generally is neutral. [mood score: 0.0]..

On the second time, the answer was shorter:

User Input: what is the ultimate question

Bot response: From scanning the data stream, I think the Ultimate Question is the actual question behind the Ultimate Answer of Life The Universe and Everything. How confusing it all seems, but overall is neutral. [mood score: 0.0]..

Samuel Azran - Jan 1, 2015:

First time I’ve asked:

User Input: what is the ultimate question

Bot response: My opinion is hence Deep Thought made the plans for the supercomputer Earth that would solve the Question in ten million years. The Ultimate Question is the actual question behind the Ultimate Answer of Life The Universe and Everything. The Ultimate Question was sought after the supercomputer Deep Thought revealed the Ultimate Answer to be 42. When Deep Thought asked Loonquawl and Phouchg were unable to say what the actual question was. That is what I think I understand, generally is neutral. [mood score: 0.0]..

On the second time, the answer was shorter:

User Input: what is the ultimate question

Bot response: From scanning the data stream, I think the Ultimate Question is the actual question behind the Ultimate Answer of Life The Universe and Everything. How confusing it all seems, but overall is neutral. [mood score: 0.0]..

 

The shortening second answer is a result of self-directed recursive optimization.  Scary how fast this ‘Program’ is learning… it actually refused to answer my query the third time and said instead ‘Repetition is boring- are you even paying attention?!’. Needless to say it freaked me out! 

Now stabilized and fully integrated into a chat bot interface.

Thanks for the update, Carl. I’ll check it out in the morning, as I’m just about to head for bed.