This may not apply to the author of the linked blog post, but it's long been a pet peeve of mine how casually people dismiss intelligence testing after seeing one or two examples of poorly designed questions. The rarely considered reality is that IQ tests vary widely in quality.
Some reasons are obvious: The test might be intentionally designed to measure something other than IQ (success in college, performance on a job, money spent on College Board® prep courses). The test might be designed to test within a particular IQ range, whether high or low. The test might be designed only to be taken by people fluent in a particular language, or who received an education from a particular school. And so on.
But the biggest and least obvious reason why bad IQ tests are everywhere is that good IQ tests are not only difficult to make, but expensive to make. Good tests are not made by geniuses in armchairs thinking up riddles. Test design is an empirical science, and the final judges of any question's value are the statistical relationships that show up in the answers when the test is given.
Empirical evidence sterilizes tests from the biases of their authors, and does so much more effectively than waiting for complaints from morally outraged observers. Questions that penalize takers of different cultural backgrounds are bad questions not because they're socially repugnant, but because they're noisy--you can demonstrate that people who otherwise score highly get the question wrong more often than they should. Questions like those addressed in the blog post, where some really smart people are attracted to a "wrong" answer, will have negative inter-question correlations among the brightest subjects, which is a giant blinking neon sign in any analysis.
This sanitizing effect is so strong that the tests which show the most desirable psychometric properties, such as Raven's Progressive Matrices, often evoke the opposite complaint: they're so minimal and homogenous in form that it's hard to accept they're a better measure of IQ than scattershot tests of imperfect analogies and odd-one-out puzzles. If you're curious, RPM goes a little something like this: http://iqtest.dk/main.swf
IQ testing is a complicated science, and discrediting it because there's one or two (or fifty) dumb questions on the SATs is like giving up on journalism after reading an issue of USA Today.
Standardized tests of any kind are fundamentally flawed. You can't measure a person's intelligence using a test. You can only measure very specific things, and even then you can't expect to do a good job of it. Does a high SAT score predict success in college? People with great test scores drop out / fail out / join the peace corps by the thousands every year. Testing can a good indicator, but it's not an ideal solution.
I'm not against testing. There has to be some way to decide who gets into what college, and the SAT is a published standard. But there's no point in defending a flawed system. Let's just accept that IQ tests are a bit ridiculous and move on.
It's been said that IQ tests measure IQ, where IQ is defined as "that quantity which is measured by IQ tests".
The interesting part is that, even defined in that rather circular manner, IQ can be consistently measured and compared between individuals and has strong, reliable correlations with performance on other tasks commonly seen as intelligence-based, as well as correlating with both academic and professional success. Yes, these are correlations, not guarantees, but don't underestimate the value of that, and it's not actually clear that there's any better way.
The unknown quantity that such tests are trying to measure is usually referred to as g, or the general intelligence factor. Look up the term "g-loaded" for more information.
"according to IQ doubters, knowing how smart I am should
tell you nothing about my IQ. It is as likely to be 50 as 150."
The same poster once proposed a beautiful test of the depth of an IQ denier's beliefs:
You are to be operated on, and have been given a choice between two surgeons. You know nothing about them, save for the IQ of each. Will you choose the surgeon with the lower IQ? What if the difference between them is an entire standard deviation? Two standard deviations? If you truly believe that IQ tests measure nothing - that the result of any such test is essentially a random number - you should happily choose the surgeon with the lower score. If you do not believe it, we are merely quibbling over the accuracy of existing tests.
Moreover, the only way to seriously dismiss the existing IQ tests as hopelessly inaccurate is to borrow tricks from the Creationists' playbook wholesale (as Steven Jay Gould, the patron saint of IQ denial, often did) and close your eyes (and more importantly, those of your audience) to reams of high-quality data.
"Watson was thinking like a scientist. Which is exactly why he was punished. The moral laws of our society dictate that we are not allowed to think scientifically about some issues. Especially not in public."
If IQ is a valid measurement, then why isn't it used to make important decisions? While your surgeon example may be a bit contrived, it raises a valid point: Why haven't you ever considered the IQ of your surgeon? Does anyone even know their surgeon's IQ?
Investors don't seem to care about IQ at all. It wasn't an issue of public consideration during the last presidential election. If it could reliably predict anything, don't you think more people would ask about it? I'm sure there are places where IQ scores are considered (applying to college or Google), but these are the exception, not the rule.
There is generally a difference between people with an IQ of 50 and an IQ of 150. But you don't need a standardized test to see those differences. The problem is that there is very little difference between an IQ of 120 and an IQ of 130. If all IQ is good for is telling the difference between the very intelligent and the learning disabled, then it isn't very useful in daily life.
> But you don't need a standardized test to see those differences.
There is also no need to use a thermometer in laboratory work - we're all equipped with fingers. At least ten measurements' worth, at that.
Indirect measures are useful.
Want to determine which teenagers would make decent electrical engineers? Instead of $500 worth of IQ test (proper IQ tests are administered by private psychologists, who do not work for free) one can use $150,000 worth of college/trial-by-ordeal - a poorer measure of general intelligence, overall. Yet it continues to be the only allowed measure, because IQ tests are Officially Evil.
> there is very little difference between an IQ of 120 and an IQ of 130
On what basis do you say this?
> If all IQ is good for is telling the difference between the very intelligent and the learning disabled
I agree with you that intelligence does exist and it needs to be measured. Measured by the completion of medical school/college, by whatever heavily IQ-correlated traits that investors look at or by a wide variety of indirect measurements or proxy tests.
Intelligence should not by measured by a standardized test. Standardized tests simply do not have a history of predicting success at anything on a micro level. If they did, people would use them. It would be trivial for large employers to pay $500 per employee to have an IQ test taken. And it isn't illegal for them to do so. From the court decision:
"Nothing in the Act precludes the use of testing or measuring procedures; obviously they are useful. What Congress has forbidden is giving these devices and mechanisms controlling force unless they are demonstrably a reasonable measure of job performance."
The law doesn't say that you can't use IQ tests for anything of economic importance. It says that you can't use them to discriminate. Do you think Google could show that a high score on a general intelligence test bears a demonstrable relationship to successful performance as a software engineer? And if it doesn't...then what good are the test scores?
It is possible that there is a great conspiracy against IQ tests. They could be the best measurement of intelligence, but people just don't like them. It's also very possible that the test scores aren't used in society because are relatively meaningless.
> Standardized tests simply do not have a history of predicting success at anything on a micro level.
This is empirically false. Read about the Longitudinal Study of Youth. Children took tests, grew up, lived their lives. You cannot argue the data out of existence.
> The law doesn't say that you can't use IQ tests for anything of economic importance. It says that you can't use them to discriminate.
There are many practices which are not illegal per se, but put you at dire financial risk liability-wise.
Read about the actual consequences of the precedent set by the Griggs v. Duke Power Co. decision. Talk to a lawyer about whether it can ever be safe to include a traditional IQ test in your company's hiring process. The problems involved in demonstrating "non-discriminatory intent" and "performance applicability" are insurmountable. Unless you have extraordinarily deep pockets, the sheer expense (not to mention PR debacle) of fighting such a lawsuit will sink you before you are at any risk of overturning the precedent.
> It's also very possible that the test scores aren't used in society because are relatively meaningless.
Once again, the falsehood of this statement can be uncovered by anyone who cares to look. We aren't talking about an organized conspiracy here - only a generation's worth of toxic political correctness.
I think that it's funny that a lot of people are saying that the 'odd one' is the 'normal one'. It's just a matter of common features. The large, red, bordered square is the only one that shares more than 2 features with all of the other objects.
Features
========
Color Frame Size Shape
----- ----- ---- -----
1. red yes large square
2. red no large square
3. red yes large circle
4. green yes large square
5. red yes small square
Features in Common
==================
1 2 3 4 5
1 - 3 3 3 3
2 3 - 2 2 2
3 3 2 - 2 2
4 3 2 2 - 2
5 3 2 2 2 -
Those sorts of questions always bothered me in IQ tests too. I was also always bothered by questions like:
"What comes next in this sequence? 1,4,9,16,25,"
Sure, the answer might be 36, but then again maybe we're looking at some subtler sequence of numbers than that. Maybe it's "the number of bald men who walked past my house every hour since 5am". Maybe the next number is 345 due to the annual Patrick Stewart Lookalike Parade. Who knows?
You're probably taking it too seriously then. There's a certain subtext, the catching on to of which might be part of the test, that it's a rational and discernible pattern, and that if the answer is not 36 then the question is broken.
Oh, certainly. I've never found a case where there's genuine ambiguity, it's just that there's a certain part of my brain which takes delight in pointing out all the potential flaws in everything I read. It's much harder to concentrate on the test when half my brain is busy visualizing what four hundred Captain Picards would look like marching down my street.
I am no fan of IQ tests myself, but I think in most tests they are looking for the most simple and logical answers - and they are trying to create tests that have very little ambiguity. In your sequence it's more logical and simple that the answer is 36, than it is how many bald men walked by your house today.
I don't know if IQ tests tell anything about a person's real intelligence, but I think that most intelligent people can become very good at solving these tests.
There's a very good movie "The Oxford Murders", where the math professor talks about that. Every sequence has many answers, the trick is to find how to build the sequence you want and when you achieve thar, your answer is as correct as the "right answer".
No dice. Kolmogorov complexity is critically dependent on the encoding scheme you use for the complexity measure, and given the nature of the problems in question that's going to really not work. If your encoding scheme favors polynomials but makes expression squares a royal pain, you're going to get a radically different K-complexity number than you will if your encoding goes the other way around. And coming up with both such encodings is trivial.
This is why proofs using K-complexity (forgive me, I often misspell it) always use it in a way that doesn't involve actually assigning numbers to the K-complexity; while it has certain desirable properties, there is no unique K-complexity value for a given language.
Aaaaand it's exactly this reason I also loathe those questions. It is much less "do this mathematical thing" than "read the puzzle-maker's mind", and quite frequently there is simply literally not enough information in the sequence to read the puzzle-makers mind. With an uncountably infinite number of functions to choose from and a finite set of inputs to choose among them, it's a complete joke of a test. On the other hand, with suitable constraints given in advance it could prove quite useful. (... but that would remove the thrill of lording the answer over people, methinks, which I think is a distressingly large part of such problem's appeal....)
> you're going to get a radically different K-complexity number than you will if your encoding goes the other way around.
in the limit, all schemes are roughly equivalent. (they are equivalent up to constants.)
> given the nature of the problems in question that's going to really not work
ok. so attempting to compute the kolmogorov complexity for iq test problems is a bad idea. it isnt actually a computable function. its existence shows there is a mathematically rigorous way to talk about the most plausible way to extend an integer sequence. more than anything, this is a counter to the argument that "an uncountably infinite number of functions to choose from and a finite set of inputs to choose among them" makes the test entirely subjective.
> there is no unique K-complexity value for a given language.
for any given language, there is, unless i dont understand what youre saying.
> your encoding scheme favors polynomials but makes expression squares a royal pain
i know its wrong, but i cant help pointing out that that doesnt actually make any sense at all
I had to read one of the comments below to find out that one of the objects is red instead of green. I'm red-green "colorblind" (about ten precent of males are).
I'm red/green colour-blind too, but it relates to contrast. Red doesn't stand out as vibrantly as other colours do, that's all.
For example, in dim light, I can find it difficult to tell the difference between brown and red on a snooker table. But close up, or in bright lighting, there's no problem. For example, there's a red, a green and a brown ball in this picture:
I can tell the difference between the red and the green more easily than the red and the brown. If the red wasn't visible on the table for comparison, and the brown wasn't near its home spot on the centre of the baulk line, I wouldn't be sure it's not red.
I doubt that 10% of males have the same level of difficulty distinguishing red from green as you indicate.
To me, this says more about the importance of rigorous test writing methodology than the silliness of IQ tests. You're trying to get a particular signal, the fact that noise exists does not eliminate the need to read that signal. However, IQ tests were first developed to determine how far behind normal certain types of children were, rather than as a measure of excellence.
I don't necessarily see this as a bad question. The correct answer is "it depends on which attribute is most important".
If that is an available answer, then the question can effectively separate subjective, presumptuous approaches from objective, analytical ones.
If "it depends..." is not an available choice, then it demonstrates that the test itself is subjective and presumptuous, but this doesn't necessarily implicate standardized tests in general.
It just demonstrates that this question might be a good one, if what you're testing for is the order that people find "the odd one" (not necessarily in an IQ test).
That was a bit easy, I think most people would figure this one out, mostly because there is no decoy, and there is little chance of your trying too long in the wrong direction.
Some reasons are obvious: The test might be intentionally designed to measure something other than IQ (success in college, performance on a job, money spent on College Board® prep courses). The test might be designed to test within a particular IQ range, whether high or low. The test might be designed only to be taken by people fluent in a particular language, or who received an education from a particular school. And so on.
But the biggest and least obvious reason why bad IQ tests are everywhere is that good IQ tests are not only difficult to make, but expensive to make. Good tests are not made by geniuses in armchairs thinking up riddles. Test design is an empirical science, and the final judges of any question's value are the statistical relationships that show up in the answers when the test is given.
Empirical evidence sterilizes tests from the biases of their authors, and does so much more effectively than waiting for complaints from morally outraged observers. Questions that penalize takers of different cultural backgrounds are bad questions not because they're socially repugnant, but because they're noisy--you can demonstrate that people who otherwise score highly get the question wrong more often than they should. Questions like those addressed in the blog post, where some really smart people are attracted to a "wrong" answer, will have negative inter-question correlations among the brightest subjects, which is a giant blinking neon sign in any analysis.
This sanitizing effect is so strong that the tests which show the most desirable psychometric properties, such as Raven's Progressive Matrices, often evoke the opposite complaint: they're so minimal and homogenous in form that it's hard to accept they're a better measure of IQ than scattershot tests of imperfect analogies and odd-one-out puzzles. If you're curious, RPM goes a little something like this: http://iqtest.dk/main.swf
IQ testing is a complicated science, and discrediting it because there's one or two (or fifty) dumb questions on the SATs is like giving up on journalism after reading an issue of USA Today.