I'm curious to see more about the distribution of questions and answers people had, and how the HN population may differ from the NYT's. There will certainly be self selection bias here, but if you're willing to share how you did with others, please enter it here:
https://docs.google.com/forms/d/17e5BIL0lH8OHsGj89Zdtdl8GeCV...
People familiar with unit testing and test driven development will feel at home with this kind of puzzle. That doesn't mean that they will be less biased in social/political decisions, it just means that this test will fail to prove a point.
Seriously, though, it seems a bit of leap from the existence of confirmation bias to explaining away the public outpourings of US Politicians about their financial crises and foreign policy disasters - in the absence of better data as to just why the given statements were made ascribing this to confirmation bias seems itself open to accusations of confirmation bias! :)
I mean, it also illustrates how training and systematic reasoning can improve these things. Whether explicitly or implicitly, I picked up certain skills and procedures for problem solving (from programming and math contests) that I now use by default. Trying a bunch of examples, coming up with a hypothesis, trying to disprove it, testing edge cases…
This doesn't mean I always use these—at the very least, I have to explicitly jump into "problem solving" mode—but it means they can be useful.
It's still a meaningful difference, and could very well apply to lots of things beyond this kind of puzzle.
That's exactly what I was thinking. I (sometimes) follow TDD, and I applied it to this problem. I made sure to include negatives, 0, positives, and include primes here or there to help avoid issues with multiplication/exponentiation. After a few of these, I felt pretty confident that the rule was simple.
I tried floating point numbers. Also, at 28 decimal places, the test breaks; it's not arbitrary precision. So, technically, the answer isn't simply "any ascending sequence of numbers".
I actually avoided going that far to avoid getting bad data. I was trying to answer the question, "What does the experimenter THINK his rule is?" rather than what will the computer do. Since the computer can't be infinite, it will inevitably fail with overflow, underflow, and such.
I was relieved, in fact, when it worked with negatives and floats in a "safe" range.
I also tested with 1,1,2 and 1,2,2 to make sure that the required increase applied to ALL of the values, not just a specific pair.
I too tried to test if it was only one pair that was significant. However I grew impatient and didn't try to come up with more tests when I thought I had a sufficient answer to explain my most vexing observation (negative, positive, positive out of combinations involving negative numbers).
The observation to brainstorm for ways of proving that a statement is in fact wrong, and exhausting them, is such an eloquent way of wording the hunt for a negative.
The slightly-shorter 0.60000000000000000, 0.60000000000000001, 0.60000000000000002 will break it too, for what it's worth. If you punch those in to a Javascript console, you can see that the FP representation of all three is 0.6.
Yes I did, though not to 28 places as tptacek had done. With 2 to 4 place floats, including negatives, the test worked as expected.
Like others here have said it wasn't a particularly hard "rule" to figure out. Easy to immediately rule out geometric relationship as in 2^y, which didn't leave a whole lot of possibilities to test. For the commenters here, I'd attribute ease of finding the solution to familiarity with the kinds of problems that programming presents.
Which leads to the idea there's value in learning even the rudiments of programming. Logically, it should encourage better problem-solving skills in general. We might think there this has important implications for our educational systems. But I know, that's probably not realistic at all.
I know the question isn't directed at me, but I thought you might want to know. I couldn't try negatives or floating point on the iPhone. The keypad didn't have the option.
Based on the context of the question and the UI of the testing interface, fractions seem unlikely to be an intended part of the question. I likewise wouldn't bother testing unicode U+216x roman numerals.
It refused to accept both fractions and imaginary numbers. I did test negatives and zeros since that was really the only remaining set I could think of.
Most importantly I used about 6 tests (3 right 3 wrong) to come up with the answer and then did another 17 looking for the trick. After all, it couldn't just be that simple right?
The presentation of the problem -- and I know that trusting the problem state is unwise sometimes in cognitive-bias tests, since many such tests are actually designed to be "we said we were asking X but actually meant Y" -- indicated a simple rule, rather than one which would behave differently on different classes of numbers.
So after the tests listed above I felt confident enough to guess.
To talk about a certain fraction of real numbers you have to have a distribution over them. In general we take the uniform distribution if no distribution is explicitly given. That doesn't work for real numbers (it doesn't even work for natural numbers). (See https://math.stackexchange.com/questions/14777/why-isnt-ther...)
If there's no implicit default distribution, we have to pick on. I can pick one where they cover an arbitrary high percentage of real numbers..
Down the rabbit hole of pedantry: we don't need a distribution, just a measure, if we want to talk about how many reals it accepts. The Lebesgue measure is implied on the Reals if none is given, and the computable reals have measure zero.
We can't reasonably talk about a percent coverage, since the Lebesgue measure of the reals is infinite, but as a non-technical description, 'zero percent' is morally equivalent to saying it only covers a measure-zero set.
Exactly. I'm just as 'No' averse as the next person but the 'No' I'm averse to is the one where you make wrong assumptions and it comes back to haunt you afterwards.
I was just writing this out! I believe practice in unit testing is what let me get this question right. I almost submitted n^1, n^2, n^3 before I realized I didn't get any wrong answers and something didn't feel right. I credit this 'instinct' to having written thousands of tests, and trying to make them fail to make sure my tests weren't lying to me.
It's funny. Even though I got the right answer, looking back, I still see many holes in my tests. I only tested positive and negative integers. I didn't even think to try fractions, decimals, hex values, text, etc. and I was already expecting a confirmation bias test.
It's probably not "fair" to say I got it in zero... but I did. :)
Now that the NYT has done it, this puzzle has probably attained enough popularity now that you really ought to change it up a bit now if you're going to run it yourself. Granted, the space of hypotheses as simple as "increasing/decreasing" is pretty small, but your ability to fool people with the first sample run is almost unbounded, so that helps.
Similar story here. I got it in zero because this problem shows up at the early part of HPMOR.
I suspect that the basic idea behind it is about right (people who insist on failures before committing to a theory will probably do "better"). But it seems to me that this test will be best at selecting people who've seen it before and can pretend they didn't (or even remember to ask negative questions when someone asks you to guess three numbers to get the job).
Yeah, I'd like to see other sorts of questions that you can only get right by looking for disconfirming evidence, but that don't have anything to do with choosing a sequence of numbers.
There's a selection bias - those of us who got it right are more likely to fill that out (:
As a result we can't really rely on overall accuracy, but we can break it out by yes/no to account for the selection bias to get a profile for how a HN correct and incorrect differ.
My answer was: "The sequence is of an increasing real variable, where each subsequent value is greater than the preceding value. It's monotonically varying."
When I clicked "I think I know it", nothing happened. I don't want to click their "I don't want to play; just tell me the answer". But it seems like the right answer. I can't answer your form question if it is the right anwer, since I haven't clicked on their link and don't know for a fact whether it is or not.
Although I used the wrong term, it's strictly increasing.
Your last question is "have you seen a test about confirmation bias before?" But the text of this test says that it's about "why no-one likes to be wrong", which means it's pretty obviously about confirmation bias (and therefore that the test-taker should be wary about just confirming their first intuition).
Intersting that the split of correct/wrong answers from the HN crowd is 78%/22% - the exact opposite of the general population : 22% / 78%! The HN community does think different :)
In the sense that they guessed the rule correctly. The correct answer was the dominant hypothesis in my mind before I tried any sequences, and others here report the same.
It just means that this test was described at least several times in different "computer" media over last few years and almost everyone has read about it.
Same thing with all "logic" puzzles.
3 9 27 yes (is it exponential series?)
4 16 64 yes (is it only odd numbers?)
5 7 9 yes (is it any numbers of the same parity?)
6 7 8 yes (is it any set of increasing numbers?)
6 7 6 no (just to confirm that it's x<y<z, and not something like x<=y<=z)
It would be relevant to include a histogram of #yes_answers - #no_answers in the summary, to test whether people are biased towards positive rather than negative tests. I think the raw data suggests that it does although I totally failed to create a histogram in google spreadsheets within 5 minutes.
The result summary is visible here: https://docs.google.com/forms/d/17e5BIL0lH8OHsGj89Zdtdl8GeCV...
The raw answers are visible here: https://docs.google.com/spreadsheets/d/1ZxR2_eOUtNLXwgKfLO1J...