I am prototyping AI-based desktop automation bot. I mean a bot that can see the screen, click and type, and basically work like a secretary. I want it to be trained by end-users rather than programmed. I am worried that someone else has already created something like that, but I didn’t find anything by searching the web. I though I would ask people from the field. Maybe you know about any similar projects?

You might take a look at this:

http://www.youtube.com/watch?v=7W52TL9Akv4

Hans Peter Willems - Mar 16, 2011:

You might take a look at this:

http://www.youtube.com/watch?v=7W52TL9Akv4

GOFAI on steroids. Good looks, but it has limited application. Ultra Hal is similar although less graphic.

Is there something that can be trained by end-users? Or more broadly, something that actually solves serious automation problems in business?

Robert Važan - Mar 16, 2011:

Or more broadly, something that actually solves serious automation problems in business?

I’m not sure if that is a domain where AI could do a lot of useful stuff. In the end, ‘serious automation problems’ are solved by special purpose software much better then with AI. To me, AI has it’s main advantage in human-interaction, so it will be the next step beyond the GUI (besides the whole ‘artificial companion’ stuff, where the real big money is going to be).

Hans Peter Willems - Mar 16, 2011:

Robert Važan - Mar 16, 2011:

Or more broadly, something that actually solves serious automation problems in business?

I’m not sure if that is a domain where AI could do a lot of useful stuff.

Highly disagree.  This is one of the best use cases I can see for AI.  Reasoning out a plan from general descriptions of a problem, and interactive discussion AI/human to develop plans.

Hans Peter Willems - Mar 16, 2011:

To me, AI has it’s main advantage in human-interaction, so it will be the next step beyond the GUI (besides the whole ‘artificial companion’ stuff, where the real big money is going to be).

Higly agree.  Myself, I am targetting on both the above.

Also, check out

http://www.cs.rochester.edu/research/cisd/projects/plow/

to add to the above post, if the AI cannot develop the entire plan for problem solving, the interactive disucssion with humans would augment human reasoning powers.

.......and, if the AI can learn from example, then a similiar problem in the future which was resolved by human+AI, could be solved only by the AI.

Hans Peter Willems - Mar 16, 2011:

You might take a look at this:

http://www.youtube.com/watch?v=7W52TL9Akv4

I wouldn’t let this example discourage you from pursuing your own automation bot.  You simply don’t know how much of this is ‘scripted’, and how flexible it really is.  Also, it doesn’t really give an example of the bot learning a new procedure, nor does it show the user explaining in one or more complicated sentences what they want, it is always just ‘save this’, ‘print this’, you could do that with a good macro editor.

Victor Shulist - Mar 16, 2011:

http://www.cs.rochester.edu/research/cisd/projects/plow/

Good shot. I knew there is someone else working on this idea. Thanks a lot. This one is a bit too rigid for end-users, but it does demonstrate the concept. I have even considered almost the same design before moving on to probabilistic algorithms.

I am wondering whether there is any commercial output from PLOW or a commercial alternative to it?

commercial product, no I am not aware of anything.  I think it is still highly experimental.

Also, this is only for learning procedures on the web.

Your goal is general desktop automation.

Thus, I’d say, full speed ahead !!

Robert,

What is the setting or scope for end-users to train a desktop automation to solve business problems?

My favorite is this one…

http://www.rivescript.com/?p=blog&id=6

Awesome!

8PLA • NET - Mar 21, 2011:

What is the setting or scope for end-users to train a desktop automation to solve business problems?

I don’t quite understand your question. This must be either professional deformation or you are really a bot.

The scope is very wide. Basically everything an employee sitting in front of a computer could do. Of course that cannot be done in one step. The scope is initially small. At the moment, Theo is a clever learning chatbot somewhat similar to jabberwacky. I will be adding more features/applications and more intelligence over time until I cover full scope of the project.