After machines become intelligent, since they will be able to process information a million times faster than biological neurons, that raises a question I’ve been wondering about for years: what will be the approximate magnitude of their IQs, relative to human IQ tests?

I once naively assumed the answer would be just the average human IQ (= 100) times one million, but I should have known better: IQ is not a linear measurement, since it is based on the normal distribution (“bell curve”) from statistics:

While one standard deviation is 15 points, and two SDs are 30 points, and so on, this does not imply that mental ability is linearly related to IQ, such that IQ 50 means half the cognitive ability of IQ 100. In particular, IQ points are not percentage points.
http://en.wikipedia.org/wiki/Intelligence_quotient

The crux of the situation is that one must relate processing speed to test performance. This is quite tricky. Once that is achieved using standard deviations, one can then merely divide by 15 to involve standard deviations to derive an IQ measurement. I finally found an article online called “Setting Time Limits on Tests” (Wim J. van der Linden, 2011, Applied Psychological Measurement, 35(3) 183–199) that indirectly addressed this question. After I saved the PDF file, I could no longer find the article online, so I uploaded the article to here, if you want a copy:

http://www.divshare.com/download/20336325-412

Most tests, especially IQ tests, have time limits, and the above study was able to statistically relate the risk (= pi sub theta in the article) of a test taker not finishing a test before the time limit, dependent on that test taker’s speed parameter (= tau, in the article). Of course, with very brief time spans in which to finish the test, the risk of not finishing the test on time is almost 1, and with enough time, the risk of not finishing the test on time drops to almost 0. Inbetween those two extremes is obviously a curve that drops from 1 to 0, which can be seen in the article’s Figure 2, plotted for five different values of tau.

To twist the article’s results into a form that answers my question, it’s easier to regard the article’s probabilistic curve as a measure of the test score outcome as a percentage, subtracted from 1. With that mental shift, the formulas in the article should allow nearly direct calculation of test performance based on processing speed. In particular, a human’s test performance for a given speed as measured in standard deviations (= tau) would be compared to a machine’s theoretical performance (= tau * 1,000,000 / 15), since the machine is 1,000,000 times as fast as the human, so the machine’s risk of non-completion would reach 0 one million times faster than the human. Of course the answer depends somewhat on which part of the curve is used, but an order of magnitude estimation should be very nearly the same in any case.

I was really hoping to give you all the answer, but the article turned out to be maddeningly frustrating since it did not give a clear-cut formula that spanned the early derivation to the final results, so I was not able to make these calculations in a reasonable period of time. I even went to the trouble to find an online calculator for the lognormal function and then to plot the results using typical parameters they showed, but my curves did not very closely resemble their results. I won’t say they have irreproducible results, but they do have a lot of missing parts that the reader has to fill in at what is likely an impractically long time expenditure. I’ll admit it: I failed. :-(

Therefore if anybody has time, inclination, and sufficient mathematical prowess, I encourage them to answer this question for all of us regarding the order of magnitude of intelligent machine IQ based on my earlier comments. The article demonstrates a lot of heavy knowledge of statistics that I don’t have, like the trick of using cumulants to overcome the problems of using moments…

“In some cases theoretical treatments of problems in terms of cumulants are simpler than those using moments.”
http://en.wikipedia.org/wiki/Cumulant

...and the tricks of simplification that allowed the problem to reduce to using the cumulative distribution function (cdf) from the lognormal distribution, so I was impressed, although the critical parts I am presumably not understanding seem to come from the details of the lognormal curve they are using, such as the exact parameters.

http://en.wikipedia.org/wiki/Log-normal_distribution

_____________________________________________

Mr. Atkins:
_____________________________________________

Machines may have an AIQ:
_____________________________________________

Artificial Intelligence Quotient.
_____________________________________________

php > echo md5(”$myName Friday, November 02, 2012”);
_____________________________________________

19cf35acc6a93a6cafd77b1fae0cab96
_____________________________________________

OK, I put in a little more time on this, and here’s a status report.
(And this was supposed to be only a quickie project, just based on passing curiosity…!)

I realized right after posting that the online calculator I had carelessly pulled up was calculating the pdf, not the cdf, so now with the correct lognormal cdf calculator…

http://www.solvemymath.com/online_math_calculator/statistics/continuous_distributions/lognormal/cdf_lognormal.php

...my plots attempting to copy Figure 2 in the article look a lot more like Figure 2. What follows are two plots I made using (free) Microsoft Works Spreadsheet.

  I’m assuming Formula 1 in the article means the desired graph is…

1 - lognormal(mean, variance, x)

...since they say “FsubTtot (. | tau, alpha, beta) is the cumulative distribution function for the conditional distribution of Ttot given tau.”

...and using the following two different values of variance I made the following plots…

1 - lognormal(0, .24, x)
http://www.divshare.com/download/20461477-872

1 - lognormal(0, .48, x)
http://www.divshare.com/download/20461481-f03

Figure 2 in the article:
http://www.divshare.com/download/20461572-390

My plots look correct, since per Wikipedia the lognormal cdf looks like…
http://en.wikipedia.org/w/index.php?title=File:Lognormal_distribution_CDF.svg&page=1
...so 1 minus this curves will look like the article’s Figure 2 plots and my plots.

But so far much of this is still maddeningly unclear to me. According to Wikipedia and the calculator’s constraints, the standard deviation must be greater than 0, yet the figure seems to plot negative values. I understand on a normal curve one standard deviation away from the mean, similar to the dotted lines shown, but that means I don’t know what the x axis represents other than an ambiguous tau that I can’t fit into the plotted equations. The article doesn’t give values of alpha or beta, either, which appear in their original formula, so there’s no way I can match up the article’s graph with any parameters with certainty. And where does the total test time (Ttot) come into play, in the plots? Ugh.

Since my curiosity won’t leave me alone until I answer my original simple question, I’ll probably keep putting in time on this every so often, but it looks like it’s going to take a lot more time. Once I can clearly match my curves to Figure 2, I should be able to make the calculations easily that determine test performance measured in standard deviations, which then easily translate to IQ measures.

What good is any of this? Well, I suspect that after machines become intelligent, companies will start marketing them using the IQ scale that is well-known to us today, in sales pitches such as “...with an IQ of 22,000 in the geometrical theorem proving domain, this little hummer can quickly solve your worst packing optimization problems in seconds. And with as little as $5,000 down, we can have this electronic marvel delivered to your office the same day! Call now!” After all, we often still use measurements based on old technology, like car engines measured in horsepower, nuclear weapons measured in tons of dynamite, and lasers measured in lumens, which is based on candelas, which is candlepower. Therefore this little study is a peek into the future to give an estimate of order of magnitudes in IQ that we’ll likely be hearing in conjunction with intelligent machines. Besides, I haven’t heard of anyone asking this question before, much less answering it.

I suspect the next hurdle, though, even if I can get my curves looking like those in the article, is to find a calculator that can handle exponents of around one million when calculating the cdf, without overflow or underflow. :-(

Thanks for writing this interesting article.

I’m still making progress.

Now that I’m reading the article in more detail, I’m starting to pinpoint mistakes that van der Linden made that make the article confusing. For example, there is poor wording, maybe because his native language is Dutch, such as: “the distribution of Ttot converges to normality with [when?] the length of the test is known”. He also never mentioned the units of tau anywhere in the article, which is one of the things my math teachers stressed even in elementary school: always put the units. That omission might even be OK, except he states that tau is speed, yet tau clearly goes negative on Figure 2, which doesn’t make any sense for a speed value in this context.

I’m finally understanding his graph in Figure 2, though. The most important insight is that tau isn’t speed per se, despite his description to the contrary: tau evidently is the difference in speed from the average speed, where that average speed of 0 is shown in the middle of the graph of Figure 2. Therefore the dotted vertical lines do show 1 standard deviation above and one standard deviation below the average speed, as he partly states, which implies 1 standard deviation of the speed variable tau is .24, as he states. Part of my misunderstanding came from my not having seen or ever thought about what a pdf curve would look like when overlaid with a cdf curve and with standard deviation markings. Here’s an example from Google images of what such a triply superimposed graph looks like:

http://ars.els-cdn.com/content/image/1-s2.0-S0038092X11001757-gr1.jpg

That means that there is essentially an invisible pdf graph overlaid on the article’s cdf graph that reaches its peak at tau=0, which will correspond to a risk of exactly .5 on the vertical axis (which always happens on the cdf of any symmetrical pdf), and the standard deviation dotted lines apply to both the pdf and cdf graphs at once.

I plotted another example of the lognormal cdf, this time using a smaller standard deviation (.10) instead of his stated .24 to try to make my curve fall sooner, and now my curve drops primarily over a tau range of .4, which is roughly what his curves do, also, so I’m getting closer…

http://www.divshare.com/download/20574995-7fd

...but that raises a new complication in that now our standard deviations are off by about a factor of 2. I’m also getting a long tail on the top of my curve where it’s up against the top of the chart like an asymptote, which doesn’t look like his curves (although it might, depending on where he’s starting at speed “0” using some unspecified offset of his).

As for the unknown values of alpha and beta, it sounds like he’s using two justifications to ignore those: (1) Alpha and beta go away per a later article, per his statement “In the sequel, it is assumed that the items in the test have been calibrated previously, with parameters alpha(i) and beta(i) estimated with enough precision to treat estimation error as negligible.”, but he doesn’t say if they go away in this article. (2) It sounds like he’s trying to combine alpha and beta with tau in this article in order to ignore the affects of alpha and beta on tau, and he also mentions tau as possibly a “nuisance parameter”, but it’s not clear to me from his description what he concludes as far as relevance of alpha and beta to this article’s final formulas.

Thanks, 8PLA. As for the erf(x) function on which this cdf function depends, fortunately I found that it doesn’t involve putting the x into the exponent, only on the upper limit of the definite integral, which means I won’t run into the huge exponents I’d feared…

http://en.wikipedia.org/wiki/Error_function

...unlike the Normal distribution’s pdf, which has an x squared in the exponent…

http://en.wikipedia.org/wiki/Normal_cumulative_distribution_function

Whew. I didn’t relish trying to find a calculator to handle e^((10^6)^2) = e^(10^12).

As a result of this study I’ve also been pressured to look up how to represent the pdf and cdf functions of statistical curves like normal and lognormal. I’ve found there are different ways to do that in various computer languages, but I don’t yet know if there is a standard mathematical convention for that. At any rate, all examples show the variable x first, with the mean and standard deviation following the x, as your example showed, such as:

In Mathworks:
P = normcdf(X,mu,sigma)
[P,PLO,PUP] = normcdf(X,mu,sigma,pcov,alpha)
http://www.mathworks.com/help/stats/normcdf.html

In Excel:
normdist(X,mean,stdev,cumulative)
http://kfoster.ccny.cuny.edu/classes/fall2011/ecob2000/lecturenotes4.html

In SPSS:
Cdf.Normal(X,mean,stdev)

In Matlab:
normcdf(X,mean,stdev)

That’s one value of these miniature side excursion topics of mine: they force me to learn a lot of things than can be useful, even if only notation.

Here’s a new status report.

Since it’s clear I understand his main graph (Figure 2) now, it’s also clear there are major problems with his published graph. As I noted before, his standard deviation doesn’t match the slopes of the figures, but is off by about a factor of 2. Also, if you notice where his curves hit the top asymptote at 1 on the left, they shoot past 1, which is mathematically impossible. I’m starting to think now that he just drew in those curves with a pen and a French Curve as an approximation, rather than using a software tool, which is ironic since he mentions free statistical tools like R will produce such curves readily. Conclusion: Irreproducible results in his plots.

After much thought, I finally concluded the information I need doesn’t exist in his article. I need a baseline speed at the mean of the chart that I can multiply to produce faster speeds on which I can measure their risk value (= 1 - performance), but I can’t do that with a mean of 0, which he has. This is analogous to trying to figure out information about the height of people in general from a Normal curve that has been normalized so that the average height is 0! In reading the article in more depth I can understand now that he was forced to produce a mean of zero since it is through that normalization that he was able to disregard the alpha variable, which was otherwise a nuisance, but in doing so, he also threw away speed values, which I needed. That wouldn’t have been so bad except he kept describing the horizontal axis as speed instead of difference in speeds, which was the only reason his article was useful to me: I was trying to relate speed to performance. So that misleading description wasted a lot of my time.

For a while I had the idea that even in the absence of numerical values that I could estimate from his graphs some numerical values quite closely, but given my observation above that his graphs are very inaccurate, I can’t even do that. As a result, I’ve become very disenchanted with the article. His knowledge of statistics is very impressive (I wouldn’t have known about “Liapuonov’s theorem on the limit of nonidentical variables”, for example), but his graphs and presentation and clarity are so poor that it renders his study almost useless to anyone who needs to actually use that as a foundation for another study. This situation wouldn’t be so bad except this topic is difficult for finding any applicable research. I looked extensively before and again last night, and this was the only (somewhat) applicable article I could find. I finally concluded yesterday that I was at a standstill on this project. There are probably statistical tricks to recover the lost data, but those are beyond my knowledge. Already I’m skirting the edge of my statistical knowledge throughout this entire project. So I gave up.

About an hour after I gave up, the thought hit me, “Why not create my own graph, from scratch?” I could create my own non-zero mean and then work from reproducible charts and formulas. What made me think this might be possible is that (out of the entire article) he did give one statement that almost gave some numerical speed information: “The middle curve is for 6,000 seconds, which corresponds to 1 hr 40 min for this test of 78 items.” Although this isn’t exactly speed information, it gives a general idea of the range.  This means 78 test items per 6,000 seconds = .013 test items per second required to finish the test on time.

One problem with this is that is that this implies test completion, which statistically never happens completely (100%): instead, the likelihood of completion nears 100% only asymptotically. Still, it gives a ballpark estimate, since at the third standard deviation (99.7%), nearly everyone will have completed a test by that time. Then I realized that van der Linden was working with two significant digits, which I was, also, which meant the “68-95-99.7 Rule”...

http://en.wikipedia.org/wiki/68-95-99.7_rule

...becomes a “68-95-100 Rule” when rounded to two significant digits. That seemed to me to be an acceptable mathematical justification for using his baseline speed figure at the third standard deviation. Everything starts to fall into place then, because an average speed would occur at exactly the first standard deviation, which would be exactly 1/3 of that base speed (=> .0043), and since the mean is essentially the average speed, I then would have a mean to use.

So I launched into creating my own curve based on this single piece of numerical information. I was off to a rough start because I made two strategic errors along the way: (1) I was still using his standard deviation of .24, but since he derived that value in a very different way due to his simultaneous manipulation of tau and alpha, and since his definition of speed was very different than mine, that produced problems in my graphs. (2) I should have started by plotting the pdf first, since the mean in such a pdf is clearly visible underneath the peak, whereas the mean is harder to see in cdf curves.

So I’m currently correcting those problems. I’m rushing off to my regular job now, so this post will contain some spelling errors, but posting this now will cut my following post in half. If my new approach is successful, I’ll likely have some numerical values to share in my next post. Either that or I’ll have to give up entirely for a long time.

I thought to be a “nice guy”, and edit the post, only to correct any spelling errors, but I found none, so left it be.

While reading your post initially, I was beginning to think that, with so many problems that you’re finding with the article/study, perhaps it would be good for you to do your own study, and write an article about it upon completion. then I got to “I gave up”, and i yelled, “NOOOOO!!!”.

Then you came to the same conclusion that I had, earlier during my reading, and my heart rejoiced! I’m sincerely hoping that you complete your study with no unsurmountable problems, and that it concludes to your satisfaction.