@CR - excellent site.

Perhaps this is more of a chicken and egg kind of problem. I tend to agree that for a fully self learning system (similar to how we learn), you need multiple input channels so that the data from one channel can be compared to another (or the rendering output in one format, matches the concept expressed in another format). But, I think it is very difficult to make a system like that from the start. I would propose a boots-trapping technique.
Here is what I am trying to do: first get the text part working well enough, next finish the visual algorithms and also get them working well enough. Only then start designing the details for the algorithm that integrates both into a self learning system, cause only then are all the data structures known and available for testing.
In my experience, I generally start something with a clear picture in mind, only to end up with something completely different. This was the case for text, it will probably be the case for the visual algorithms I have on paper, and I can only dream of the final stage at the moment.

@Jan: that’s exactly what is happening right now. Different disciplines are working on their own. NLP about from visual search for example.

Leslie Pack Kaelbling, MIT professor, said during her talk during the last AAAI (2010 in Atlanta) that disciplines should start to work together.

I personally believe when disciplines are starting to work together, this industry will really kick some a**. (if you can say that in English like this… ;-s)

Actually, the vision is to translate the real world into a virtual world in our brain. There are may guys focus on it. 

I believe what we need to do is building a representation model which can support both natural language and the result of vision.

Along the vein of developing artificial senses, I thought this link was interesting: http://www.boingboing.net/2010/09/13/electronic-artificia.html About artificial skin being developed at Stanford and Berkeley.

cool example CR! tx

It’s strange, the more you start thinking about anything, the more you see it everywhere. I keep finding all these cool articles discussing advances in robotics. I found another article which describes the development of artificial limbs that can be linked with the brain so you can “feel” temperature and pressure. It hasn’t been realized yet, but the idea is pretty amazing.

so much is happening in this field. Until last year, when I didn’t start to work for Chatbots.org, I did a lot of research and wrote about it on my blog. Lot’s on cool stuff:

http://www.erwinvanlun.com/ww/C165/

Once, I will go back to my research. It’s jsut soooo cool.