Wow. There's a lot of negativity and criticism on here, don't be disillusioned, you got something up and out in 2 weeks that says more about you than most other people spending time "commenting" on here :-)
Best of luck, and get on that usability thing these people are talking about...
Thanks for the encouragement. I've been meaning to play with mechanical turk for awhile and this seemed like a good idea. I'm going to hire a professional and get a snappy design going as quick as possible.
I went ahead and bought a review. I just spent a few hundred dollars launching a mini-site, what is an extra $7? Rounding error -- if they catch a typo I'll consider it money well spent.
I'll let you know about the quality of results. One request: my results should not be available on a publicly accessible URL. It doesn't bother me, personally, but it will bother some people.
This is a holiday promotion aimed at my customers. They're overwhelmingly non-technical folks, and not too sophisticated at the Internet. (I am well-aware that most people here would not purchase my software. I'm totally OK with that: you are not the audience.)
I asked the Turkers three questions (and will tell you my rationales):
1) Is it IMMEDIATELY clear what the site offers when you open it? [Rationale: This site is a sledgehammer, not a scalpel. I want them to understand what it offers before they hit their back button. If thought is required, I failed at design.]
2) Try to print out Christmas bingo cards. Tell me how long it takes you from me giving you that instruction to something coming out from your printer. [Rationale: My target is that fourty-something mother with kids, with no special computer skills, can accomplish this in 90 seconds. There is actually a bit of an overlap between Turkers and my market.]
3) If you thought your friends would find Christmas bingo to be a fun activity, would you recommend this site for it to them? Why or why not? [Rationale: The Turking population has a significant overlap with my core audience. Why not ask a quick thumbs up/down?]
What it taught me:
1) The site achieves its singular purpose fairly well.
2) It is attractive enough to move several testers into mentioning it to their friends. (As in, not hypothetical.)
The feedback lets me know "OK, the site is basically OK and needs to be built-out more than rethought from scratch". Knowing that, and the peace of mind from knowing that, are worth $7 to me easily. I hope this service continues getting developed, because I can see a lot of uses for it in my business.
(Yeah, I could roll my own with the Turk API... but I value my time at $100 an hour. This means I never want to be playing the sit-at-computer-refreshing-Amazon-so-I-can-authorize-fourty-cent-payment game.)
We'll see what happens. I'm watching everyones results like a hawk right now too. You may see a few responses at night (someone who just did a request already has 2) but I find its during the day when they come the quickest.
If you're interested in catching typos you should specify that to the reviewers. The reviewers are good at doing what they are told.
Do you have any suggestions on the publicly accessible URL? I don't want to force people to create usernames/passwords. I think its a hassle for a quick-use service. Or at least its a pet peeve of mine.
How about specifying a password with the request and requiring that password in the URL to access the feedback later on. This way you can subscribe to the results in any feed reader but still have some protection in case people try to guess your URL to see the feedback.
<3 I can't disagree with what you're saying. My design skills are weak but I wanted to validate the concept first. Today I've seen enough interest to validate moving forward with the project. I'm going to invest in a site facelift to bring the presentation to Web 2.0 standards. Thank you for taking the time to look and post your thoughts here. -- Raphael
It's not usability testing at all: in a usability test we're interested in how people use the product, not how they say they use the product (although that can be interesting too).
Lets assume you've never posted a job to Mechanical Turk before. Here is what you need to do:
1. Create an Amazon Web Services account
2. Figure out how much you want to spend (and don't forget to calculate Amazon's share)
3. Prepay the exact amount or more
4. Create a Human Intelligence Task template
5. Publish the Human Intelligence Task
6. Login to Amazon's portal regularly and navigate several pages to find your results. Have fun clicking refresh.
7. Review the results and promptly pay your workers (or they may ignore future requests)
How much is your time worth? Feedback Army spends the majority of the fee towards paying the mechanical turk workers. The Feedback Army software saves you from many of these steps and conveniently transforms the results into an RSS feed.
I have never used Mechanical Turk before and I see that you are adding some value.
If you improve the look of your own site you might have something here.
Another idea might be to offer larger scale tests with graphical reporting on the results, basically using the low cost/test advantage that Turks offer.
Thanks. This is a pretty new project for me (< 2 weeks old) following the release early, release often mentality. I've seen enough interest to validate the idea and so I'm reaching out to a few designers I know to get a proper face lift. I'll also dump a few dollars into making some design firms aware of the service and see where that takes me.
I tell you what else you will provide, after you've gotten going - a relationship with the Turks. They'll know what you pay for (and that you will pay), and you should know how to get good results out of them.
I tried feedbackarmy - from a purely ROI perspective I think the usability methodology is flawed.
It seems that the folks who give the feedback are actually trying to be a bit too helpful - to the extent that they think about questions as a challenge and try their best to answer them.
For example - If I pose a question that "What does this site do?" in a usbility experiment, I am expecting to figure out if the audience immidiately grasp the utility of a site.
However in this case it seems that the audience takes that as a challenge modifying the question to mean "Can you figure out what this side does?" and try out very hard to find the utility of a site and in the process totally defeating the purpose of a usability review.
It might be an artifact of the way I posed the question, but then I was just following the sites examples. I think using the site in the way it is setup (messaging, mturk) the concept does not give a good ROI w.r.t usability.
However, one thing that I did like was that the turkers gave some suggestions to improve the site - some of which was really valuable. So maybe if you change your positioning to something like an active feedback/testing tool it might work better.
1)That page is why. If someone has the capacity to produce (or have produced) a website, they have the capacity to create a form with two elements and garner feedback. It is incredibly lackluster, easy to replicate and certainly not worth $7 for 10 responses.
2) Based on the comments here people aren't sold on Usability Feedback on websites when the website you've created doesn't suit the needs people are coming to the site for in the first place.
3)The process lends itself quite easily for scammers and cons. There's almost no security measures in place, the design is inherently broken and doesn't provide any peace of mind from the process that you have to perform to get "feedback".
I'm not sold. This can be done about 50 ways better than what it is now; I really do not see myself paying any amount of money for the presentation you've put in front of me. It's an effort, I wont say good effort, I wont say bad effort. It's an effort made, and at this point I'm going to say no to this.
It's $7, lighten-up. I'm about to spend $15/person/hour for deeper usability tests and now I can ask less in-depth questions but as often as I want for much less? Pure gold.
Best of luck, and get on that usability thing these people are talking about...