My initial thought was "Yes, you're absolutely right. That must shift the results", and I think that selecting for any particular subset of the overall population will shift it, but given that the researcher is already using a convenience sample (i.e., easily available college students) is it really going to change it more than the convenience sample already does?
Like, I can see there being a difference between a truly random sample from the overall population vs. students who are required to participate, but how much of a difference are you going to see between local students who volunteer vs. local college students doing this 'as part of their homework'.
Yeah, the biggest problem seems to be taking a bunch of extremely hormonal 18-20 year olds in an awkward and idiosyncratic social situation (university in general, and often forced dorm life for undergrads), and then calling that a universal study of human nature. Over and over and over again.
It’s not the only thing to shift the results. I took one of those surveys asking the classic question of whether I’d like money now, or a decent bit more money in a few months.
There I was, earning nothing, a little shy on cash, but I had a job offer in a few months for a decent little software job. I said I preferred the money now.
Uhhhh doesn't that violate informed consent? "Participate in a study or don't graduate" doesn't feel as consensual as research ethics normally require...
When I took intro psych in 1986, we were required to do 3 studies, but some of the studies did offer cash. Those usually were the first to get filled. I think I only got money for one of the three.
I feel like undergrads who are participating in a study only because they’re told to will shift the results a bit.