Mauled by a PANDA

So I almost got mauled by a PANDA today.

If you don't know what I'm talking about, on mturk, a PANDA is link that will automatically accept a HIT. It's short for "Preview AND Accept," which is the kind of link you get if you click the "automatically accept next HIT" checkbox from an active HIT. (I'm butchering this explanation, but it's common enough that if you google it, you'll find tons of detailed posts about how to use PANDA links.)

So anyway, you can accept up to 25 HITs (until they expire), so it's kinda a common thing to use a PANDA link for popular batches. Basically, you're just grabbing as many of them as you can before they all disappear. If you worked them one by one (waiting until you completed one to accept another), you might not be able to get as many.

But since I've been more of a casual turker, I didn't really try anything with PANDA links until recently. And in the last few days, I've bee playing with opening a PANDA link in a separate tab or window, then auto-refreshing that tab while I work from my "Hits Assigned To You" queue.

And that's what I was doing his morning when I started turking at my house. Then my Internet connection went down. I knew I had a few HITs that would expire if I didn't get to them within an hour or so, so I decided to head to my office to work there for a while. Cleaned up the few HITs I had accepted from home, opened a new PANDA window, and kept working away. A few hours later, I closed my PANDA window so I could finish up what I had accepted and then go to lunch.

But the HITs kept coming.

It didn't make sense... until I realized that my home Internet must have come back up... and I never closed the PANDA tab there... Oops! Felt like the classic I Love Lucy scene in the chocolate factory!

Much overdue update

So it's been a while :)

I asked a question in my last post: could I take a break from turking to enjoy something else (a vacation, etc) and still want to come back to it?

Apparently, the answer is yes. Over the summer I stepped away from turking for a few days to a week on several occasions, and I still came back. And I'm doing very well.

Here's my latest: it's been almost 6 months now since I created my mturk account. As of this writing, I have submitted 14,003 HITs, 13,997 approved, 6 rejected. I am still working primarily for Crowdsurf, and that's been an interesting ride as they have started making some moves to leave mturk (or at least offer an alternative that will pay a better rate than mturk in order to encourage us to leave).

Mturk more than doubled the rates it charges requesters over the summer. Several large requesters have already left, and the increase will likely curtail or cut off participation from some smaller academic requesters whose surveys have been previously been an mturk staple. I'm not sure how much of this is seasonal and how much of it is a reaction to the price hike, but it seems like at any given time, there are fewer HITs available overall, and those that are available are mostly very low paying HITs.

But on the plus side, mturk doesn't have any sort of noncompete or user agreement that says you can't work for a requester outside of mturk, so I'm guessing requesters will continue to post penny jobs to mturk and take their more experienced workers elsewhere for higher paying jobs.

And speaking of higher pay... at the time of my last post, one month in, I was thrilled to make over $100 in a week. Now... I usually have one day each week where my total is over $100. So yeah... I still think it's worthwhile to continue to turk :)

One Month

So, here I am, one month later... still turking!

Quick stats update: I have now submitted 2844 HITs. 2737 are approved, 104 pending, 3 rejected. 

I hit a couple of milestones this week. One evening, I completed 360 HITs... a new one-day record for me. They were "numbers" HITs, penny hits that went fast. I need to keep my eye out for more of them, because the next big goal I'm shooting for is 5000 total approved HITs. 

This was my best week yet, financially speaking. I made about $105. That was partly because I got sick and spent a little extra time on mturk, and partly because of bonuses from Crowdsurf, the requester I'm doing the most work for.

On Friday, I was "promoted" on Crowdsurf. This gave me access to longer, more involved, but higher paying HITs. It's still taking me far too long to work on each task, but I can see how it will get easier as I master the interface (and maybe start working on a desktop system instead of this ultrabook) and become more comfortable with my understanding of their guidelines. Right now, I lose a lot of time double-checking stuff and sometimes even looking it up the Guidelines document. If I can get to the point where my "first pass" at editing is solid enough (and I don't feel like I need to run it through again from the beginning), that will improve my time significantly. But baby steps for now :)

I feel a little bit uneasy about focusing so much on one requester, but honestly, I haven't found any others that appeal to me as much or that have work available pretty much any time I'm looking for it. The bonus structure also pulls you in -- reach certain goals in daily or weekly jobs and the bonus pay works out to be a lot better than the "base" rate. 

My biggest turking test is yet to come: Can I step away from it for a week (I'll be taking a vacation to visit family next week), and then come back to it with the same enthusiasm and commitment? Guess I'll have to wait and see :) But I have one more "solid" week of turking before that happens.

Three Weeks and still going!

Had to post a quick update since I've now reached the three week mark. So far, I've submitted 1865 HITs, 1847 approved, 15 still pending, 3 rejects (no new rejects this week). And I'm now averaging over $10 per day. It's hard for me to figure out what that equates to hourly since I only turk a few minutes here and a few minutes there, but it's definitely not a 'living wage.' Probably less than $4/hr. It's not a lot, but it does add up. Three weeks in, and I have $165 in my account now. So I think I'll keep going :)

Ups and downs

I had intended to post on Tuesday to give a two-week update, but real life had other plans and kept me pretty much occupied :)

There's a lot of hit-or-miss involved when you're a new mturker. It seems like some really good (both in terms of being interesting and high paying) stuff is reserved for those who have difficult to find quals (because the qualification tests are not being offered) or who have Master status on mturk, and the secret to achieving this Master status seems to be something of an enigma even to those who possess it. Then there are times when you just happen to log in at the right time and find a great batch that fits your qualifications. 

I'm still doing well. Tuesday of this week was actually my best yet in terms of total number of HITs -- I happened to be online when some easy numbers HITs became available, and along with some of the regular work I had already done that day, I finished the evening with 170 HITs submitted (and all approved quickly)! It was also my best day yet in terms of daily earnings: $16.41! But that wasn't all from the HITs I worked that day. That was also the day I had a couple of delayed bonuses credited to me. 

A few updates: For over a week now, I've made over $5 each day. I now have over $90 in my account. I passed the 1000 approved HITs mark, and currently have a total of 1273 approved. 

I'm still a newbie for sure, but with that kind of a buffer, I'm feeling more confident to try out different kids of HITs that I might have been avoiding before (for fear of rejections) or even from new requestors who don't have reviews on Turkopticon yet. At this point, if I come across an interesting batch, I feel comfortable testing the water because a couple of rejections aren't going to really change my percentage of approved HITs. 

I've started to focus on building my numbers with one particular transcription requester, Crowdsurf, in the hopes of working my way up to better paying work with them. They're a company I feel good about doing work for -- they provide captions for the hearing impaired. But they're also a very frustrating company. 

Here's how it works: they post a transcription HIT. They pay based on the length of the clip you transcribe. In order to continue working for them and to move up, you have to maintain a certain Quality Control score. Unfortunately, the quality of the audio we're transcribing isn't always great. There's a lot of video of college classes with professors who do not speak English well, or speak it with such a heavy accent it is sometimes impossible to know what they're saying (especially in the context of a clip that is less than 25 seconds). 

After you submit a transcription, it is "graded" by their Quality Control folks. If you take on a difficult HIT, you're more likely to get docked in QC for saying that something is inaudible or unknown, or for mis-hearing something because you don't have enough context. So if you're trying to keep your score up, you avoid submitting any of these. However, if you're a worker who is trying to keep your score up, you're probably a great worker and would do the best possible job on the HIT you're avoiding. So there's a pretty deep flaw there in their business model :) Sometimes I see HITs with a popup saying to just do the best possible job, you won't be rejected and your score won't go down. But I've done them and still had my score affected. So worker beware!

Unfortunately, even on relatively clear clips, there seems to be disagreement among the QC folks about certain guidelines... so you can submit one HIT and get docked for doing it one way and have the QCer tell you to do it another way... then submit another one done the other way and have another QCer tell you to do it the first way! 

And yet, despite the frustration, I have managed to keep my score up in the range that I want it to be, and even though it still takes me a lot longer to do some HITs than what they're really worth, I do enjoy knowing that what I'm doing might actually benefit someone. In my free time. And make a little spending money. I mean, come on, that beats Candy Crush any day!

10 Days Later

Thursday was Day 10, the last day of my mturk probation period. So Thursday evening, I was determined to max out my allowed HITs before calling it a night. I knew I had returned a few HITs during the day, so I figured I'd max out in the low 90s again. Apparently, the HIT limit is increased on Day 10, rather than after Day 10. By the end of the evening, I had 101 HITs submitted! Felt good to see a three digit number on my daily stats page for the first time. I have now submitted 709 HITs in total. I have 3 rejections (that could have been avoided if I had installed TO earlier). 19 are still pending. And the total I have earned so far in my first 10 days: $39.98. In my spare time. At my discretion. 

Now obviously I hope that as I continue with mturk and put up good numbers, I'll have access to higher value HITs and make a little more than I am right now. But already, after just 10 days, I can see that my daily earnings are generally going up, and some new opportunities did open up for me at the 500 HIT approval level. I'm cautiously optimistic that this will continue... and that I will continue to be interested enough to keep turking :)

My first week

One week ago today I began my turking experiment. It's been interesting.

Saturday, I reached my probation period daily limit on HITs in almost no time, with 93 submitted. Sunday, I could hardly find an HIT I wanted to do. I managed to get 32 in, and it took me longer to do that (mostly for searching and checking back at different times during the day for new HITs) than it did to max out the day before.

Yesterday I didn't have much time for turking. Then last night I stumbled on a new set of HITs from a requester with a great TurkOpticon score, and happily worked through about 25 of them before that well ran dry. Between that and a few extra data entry HITs, I still managed to turn in 77.

Remember that first rejection I received on my second day of turking? I heard back from the requester and they overturned it! Yes! But lest I celebrate too much... I also got two rejections from HITs I turned in last week, from a requester with a bad reputation on TO. If I had been using TO from the start of my turking experience, I would have avoided that. But knowing that they have a bad reputation makes it a little easier to take the rejections.

So... my numbers for my first week as of this writing: 412 HITs submitted, 385 accepted, 2 rejected, 25 pending. Total earnings for my first week: $24.66. That's a horrible number if you're trying to earn a living. But look at it this way: pretty much any other "work from home" promotion out there involves you spending money at the beginning. You have to buy supplies or stock or books or a program... and there's no guarantee you'll earn enough to cover those costs, let alone actually make money. Here's a site where I spent no money at all to get started, and at the end of my first week, I actually earned something. Not much. But something. It required work, and there is definitely a learning curve, but this is something I can see myself continuing to do.

Most of the tasks are kinda boring. But unlike what I do in my day job, these are tasks that you can complete and have approved quickly. And even if it's a low value task, it's nice to have that sense of completion. I'm also finding that when I'm feeling stuck on a creative project, spending a few minutes on a data entry or other repetitive task kinda helps me get un-stuck.

So where do I go from here? Right now, I want to reach that 1000 HIT approved level. It seems like that is the first truly significant milestone in terms of what requesters with "good" HITs require from workers.

I found the ceiling

Today is a rainy, miserably day... so I had some time on my hands for turking. 

But after a couple of hours, I found myself cut off:

There are no HITs available for you at this time. Please try again tomorrow.

When I got the message, I had submitted 93 HITs. Seemed like an odd number. So, I poked around and here's what I found: During your first 10 days on mturk (and today is Day 5 for me), you can attempt no more than 100 HITs per day. Unfortunately, sometimes the HIT you think you're accepting isn't the one you wind up getting -- or you get one that is particularly tricky and you're just not confident enough to send in an answer. In that case, you "return" the HIT. That doesn't count against you technically (it does not have any impact on your approval number), but they do keep track of that. So apparently I submitted 93 HITs today and returned 7 to hit my daily probationary limit.

Since I am just kinda feeling my way through this process, I didn't know how that worked. Until today. So now that I know that, for the next 5 days I can focus on getting as close as possible to that 100 mark, and fill it up either with short, easy HITs (on days when I'm just filling in dead space between projects) or spend a little more time on more involved (but higher paying) HITs on days when I have more time on my hands. 

So, as of this writing, today I submitted 93, had 92 approved, for $3.93. A couple more HITs from previous days were also approved, so now I have 266 total approved HITs and a total for this (partial) week of $13.11. Keep in mind, that's for a "casual" turker who was trying to figure things out on her own and not really trying to do it seriously for a living. The way I see it, I didn't "lose" any time to turking since I did it all during short gaps in my day (waiting for a big job to finish printing, a video to render, uploading large files, on hold with someone on the phone, etc) or, for today, to fill time I would have otherwise spent playing games. 

And since I couldn't spend the rest of this rainy afternoon turking, did a little searching and read a few blogs with suggestions for beginning turkers. One really nice tool that I wish I had installed from day one is Turkopticon, https://turkopticon.ucsd.edu/. It gives you quick access to reviews of requesters to help you avoid turking for bad employers. I'll definitely be making use of it in the future.

Day 4

Today was the day I had the least time for Turking so far. I spent some extra time this evening trying to find new things to try, and still wound up completing 32 hits by the end of the night, 17 of those still pending, and a whopping $.67 earned.

The most interesting thing I did today was take a "qualification test" for another requester doing transcription. This time, it looks like small scanned passages from books. The "test" transcription was in English, and after passing that I was offered a handful of additional passages that appeared to be Dutch. I definitely prefer the scanned book passages to the receipts or even audio/video transcripts, so we'll see where that goes.

Some of my previous pending HITs have cleared, bringing my total for this first (partial) week, including a few 'bonuses,' to $8.42. 

Day 3

I spent the first part of today focusing on boring and relatively easy HITs from a requester who approves HITs quickly. I was determined to break that 100 HITs approved threshhold.

And I did! Worked my way through about 30 of the data entry type tasks in between phone calls and other projects. I was also able to pick up some $.06 data entry hits, a nice change from the one-and-two penny jobs I've been using to build up my HIT count. I also tried a few short audio transcriptions. I might have to explore that a little more, depending on the subject they can be fun.

Then I rewarded myself with 16 more interesting (and somewhat more risky) HITs that I actually enjoyed.

At the end of my third day, I had completed 54 HITs. 38 were approved, 16 were still pending. And I made $2.22 on the ones that had paid already. 

More of my work from the last two days has also been approved, bringing my "paid" totals to $1.44 for day one, and $2.07 for day two. It's not a lot, but still, that's more than $5 for stuff I did with otherwise "wasted" moments during the last three days. 

Day 2

My adventures in Turking continue :)

Spent some time today trying to figure out how to get some more qualifications so I can work on "better" HITs. Turns out, a lot of it is based on your HIT acceptance rate (keep it above 95%) and your total number of completed and approved HITs. There are thresholds where new opportunities open up: 10, 50, 100, 200, 1000 -- any requester (company or individual creating the HIT and thereby also dictating the terms) can filter out Turkers based on combinations of these statistics about a Turker's work. Then there are specialized qualifications that you can get based on taking a test or demonstrating that you can handle their lower level HITs.

So I wanted to do more HITs today. I wanted get closer to reaching that 100 approved HITs mark. About 2/3 of my HITs from yesterday wound up being approved. I still have several that are "pending," and I've learned that they can stay pending for up to 30 days. A pending HIT doesn't count towards your successful HIT count.

So today, in addition to exploring a few new options, I went back to the requesters who were quick to approve my HITs... and in my spare bits and pieces of time, I completed 59 HITs total today. Some are still pending. But I'm closer to that first big milestone: 100 approved and completed HITs.

Unfortunately, I also reached a different, less pleasant milestone today: my first rejection. It was a HIT worth less than a dime (most of them are worth less than a dime)... but it still stung! How could a college educated, literate, tech savvy, middle-aged woman fail at a simple task? The truth is, I'll probably never know what I did wrong. And it WILL haunt me on some level as I work on additional HITs from that requester. But life goes on...

So after two days of Turking, I've made less than $1.50 and have a slightly bruised ego. Yeah, off to a GREAT start!

Day 1

This morning I got the email - "Congratulations, your Mechanical Turk account has been approved"!

I was eager to get started. So as soon as I had a few spare moments, I logged in and began to look at what was available. The site provides a list of "HITs" - Human Intelligence Tasks - along with a brief description and the promised reward. The banner boasted of over 300,000 HITs available at that very moment, so I hoped I'd find a few interesting things to try.

The first thing I learned is that on mturk, there are HITs that are available... and then there are HITs that are available to you. As a brand-new Turker, once I chose to see only the HITs that I was qualified to work on, that list of 300K shrunk to around 500. 

Of those 500, you figure out pretty quickly that some of the HITs are scams or questionable at best. Some of it is probably meant to boost search engine ranking by asking turkers to search for certain companies and keywords. Some of it is probably meant to inflate the target website's ad revenue or make it look like they have more genuine visitors than they really do. Some are outright phishing schemes that want you to sign up on some other website and give out your info.

Turking is a dirty job. There are groups of HITs reserved only for turkers who have an "adult content qualification." 

Then there's the legit stuff. Or at least, I hope it is. There were some interesting surveys and academic research projects to try. They advertised a reward of anything from a nickel to nearly a dollar, and a potentially higher reward if you met certain requirements. These were kinda fun! But you can only take a survey once. 

I tried a few of the quick, boring penny tasks. These were mostly tedious data entry tasks, entering single lines from often blurry shopping receipts from foreign countries. There were also tasks for copying business card information, and tasks that I presume were checking up on the work of other turkers (looking at a scanned item and choosing the correct transcription, with the option of entering your own if you felt no one got it right).

There were a couple of "creative" tasks available: writing summaries or headlines from larger blocks of text, answering questions, writing a paragraph in a story, etc. They generally paid a bit more per task, but there were far fewer of these available.

I also tried my hand at audio transcription, short snippets of what I imagine was folks giving out their personal info in response to a telemarketing call. And I tried a few of the video based tasks, but most of them wouldn't work properly on my computer.

I don't know exactly how much time I spent Turking on this first day. I was just dabbling and learning the ropes, sneaking in a HIT here and there when I had a spare moment. I can tell you I completed 42 tasks... and as of the end of the day... earned a whopping $0.63. Now, not all of my HITs have been processed yet, and I'm assuming most of the ones still "pending" will eventually be approved... but even so, at best I probably made a dollar, if that.

On the flip side... I did this all in the spare bits and pieces of time that I might have otherwise spent playing games. And possibly spent money on those games. And even after my 42 tasks... I'm still intrigued. And I'll be back tomorrow.

Turking

As part of its web services, Amazon has a site that kinda looks like it's stuck in the 90s and promises to connect an army of Internet workers to employers who are willing to pay them a little pocket change in exchange for completing quick, simple tasks that require human intervention.

The site is mturk.com, Amazon's Mechanical Turk.

I've heard a lot of folks sings its praises, and I've heard plenty of complaints. But I decided that it was time to check it out for myself. So tonight... I signed up. I was ready to explore the weird world of Internet Turkers. But it seems like my first task would be getting accepted into the system in the first place so I could try it out. I completed the simple sign up form... and found out I'd have to wait around 48 hours for my request to be approved.