Topic: Very preliminary concept (Get paid to do science) (Read 2706 times)
Also, in my mind... this would just be bringing back the days of "human computers", to do the jobs that noone is currently doing because the people in charge of the projects have neither the time nor resources to really have someone examine their data to the fullest extent. The result is a millions of monkeys on typewriters approach, half assing everything. I think we can do better with a crowdsourced approach, even with fraud and noise to filter out.
I have been thinking more and more about this concept and have been convinced that the existence of something like this would be a huge benefit to humanity. However, I have another year or so of work to complete my current project before I can start working on it, and after that... it depends on what happens. If someone else creates something along these lines I will contribute as much as possible. This is really a $15 billion a year industry just sitting there waiting to be taken from academia.
and get paid $5-$10 for a job well done. Others could donate to this site, somehow invest in it and filter out bad submissions. The payout could be a function of the number of submissions and there would be extra rewards for more productive contributors...
You should lower the payout. At $5 to $10 a job you are paying graduate student wages.
Of course if you marketed it as a way for graduate students to procrastinate then they could double their paychecks.
This is a pretty awesome idea, people will get to learn more science, get paid, and also contribute further to science at the same time. It's a win:win for both sides almost.
I have been also toying with the idea what happens after the GPU-mining comes totally unprofitable. I mean not to hijack this thread but so called classification etc. "mundane" classification tasks can be in my opinion be rather easily solved with neural networks. So my suggestion is to make rather an distributed computer cluster where "mining rigs" can join. Environment could have ready make neural networks etc. plus ability to upload&train your own. Then any researcher could simply buy cluster time. WIN-WIN solution. Plus pretty affordable.
So I was at the zoo the other day and talking to one of the guys running the research aspect of it. Apparently they are working on something similar to this idea. They are trying to devise a way to have zoo-goers report on animal behaviour they observe and filter out the nonsense. The current strategy is to make it part of an ipad app that provides facts about the animals. It sounded like there was a threshold of "learn more" buttons that need to be clicked before the data would be taken seriously.
Anyway I think it is promising that people are beginning to think along these lines.
Anyway I think it is promising that people are beginning to think along these lines.
I am glad there is interest in this idea. Here is an example of a simple task that must be repeated hundreds of times. I used some of the terminology PawShaker provided here:
Fullsize here: http://i49.tinypic.com/fwuiz8.png
A quick draft writeup. I think better when I put ideas to keyboard.
https://docs.google.com/document/d/1nPI1tsrVt9QMD6HgesA77PvdQ-cD7m_0wLYuBv00NJk/edit
https://docs.google.com/document/d/1nPI1tsrVt9QMD6HgesA77PvdQ-cD7m_0wLYuBv00NJk/edit
Fullsize here: http://i49.tinypic.com/fwuiz8.png
Wouldn't it also sometimes be an option for the process not to be deterministic? For example, "Use any software to count the cells on this set of images". The worker would have to say what he did, give the numerical results, and also supply the processed images*. This way the researcher can easily see if the work has sufficient quality, without having to do all the manual tweaking of edge-cases. The idea here is that checking if something is done correctly takes a lot less time than doing it yourself.
*Normally software such as this can show the outlines of the cells it detected, for example.
*Normally software such as this can show the outlines of the cells it detected, for example.
[...]
That looks promising in general, but the power of one SC to affect the final product seems excessive. (Only two must approve) Perhaps a less linear process with a more distributed system would work better. (Multiple SCs submit revisions or "no revision", they all vote on the preferred option)
What system would you suggest for dealing with users with large amounts of rejected work?
That looks promising in general, but the power of one SC to affect the final product seems excessive. (Only two must approve) Perhaps a less linear process with a more distributed system would work better. (Multiple SCs submit revisions or "no revision", they all vote on the preferred option)
What system would you suggest for dealing with users with large amounts of rejected work?
In theory instructions should be so precise that process should be deterministic. Anyone following instructions should (theoretically) arrive at materialy same answer. This is supposed to be a scientific observation, hence reproducible.
In practice human factor is not that easy to account for. Hence need for some arbitration and quality control. SC are supposed to be more experienced and cases when their verdict would be questioned will be so rare that they will not bias the study.
I must admit that I wrote spec with corporate context in mind. I could use some lateral thinking and explore a self-organising network of humans.
Well SC has power to deny reward for inadequate work. In practice such workers drop out very quickly. Of course, repeated offenders will be suspended. Depending on the supply of recruits, suspended users will be retrained or dismissed.
A quick draft writeup. I think better when I put ideas to keyboard.
https://docs.google.com/document/d/1nPI1tsrVt9QMD6HgesA77PvdQ-cD7m_0wLYuBv00NJk/edit
https://docs.google.com/document/d/1nPI1tsrVt9QMD6HgesA77PvdQ-cD7m_0wLYuBv00NJk/edit
That looks promising in general, but the power of one SC to affect the final product seems excessive. (Only two must approve) Perhaps a less linear process with a more distributed system would work better. (Multiple SCs submit revisions or "no revision", they all vote on the preferred option)
What system would you suggest for dealing with users with large amounts of rejected work?
Wikipedia is human language, which can contain other things beside truth. If you farm out eg data processing in a picemeal fashion, you'll be able to see if some part of it differs significantly from others. So when you figure out who screwed up you can mark them down or if it's a malicious error throw them out.
You'd need a 'web of trust' type identifying system. Management doesn't have to know who workers are, as long as someone trusted in the chain does. This would also double as a referral system too.
I also see this as being handy for collaboration. Don't have anyone at your job interested in the work you're interested in? Pay a fee and get matched with someone with your interests. For example, if you're great at analysis but crap at writing it up, you might be able to find someone familiar with your area to get the written parts ready for publication, either for a fee or for coauthor credits or both.
You'd need a 'web of trust' type identifying system. Management doesn't have to know who workers are, as long as someone trusted in the chain does. This would also double as a referral system too.
I also see this as being handy for collaboration. Don't have anyone at your job interested in the work you're interested in? Pay a fee and get matched with someone with your interests. For example, if you're great at analysis but crap at writing it up, you might be able to find someone familiar with your area to get the written parts ready for publication, either for a fee or for coauthor credits or both.
A quick draft writeup. I think better when I put ideas to keyboard.
https://docs.google.com/document/d/1nPI1tsrVt9QMD6HgesA77PvdQ-cD7m_0wLYuBv00NJk/edit
https://docs.google.com/document/d/1nPI1tsrVt9QMD6HgesA77PvdQ-cD7m_0wLYuBv00NJk/edit
[...]
This is interesting, and likely would be tried. What would stop the "third-world" individuals from signing up themselves? Also, how do you explain the number of people that contribute to wikipedia?
This is interesting, and likely would be tried. What would stop the "third-world" individuals from signing up themselves? Also, how do you explain the number of people that contribute to wikipedia?
Culture differences in most cases (language) and economical differences in remaining ones. After all you need to own a computer with reasonable internet connection.
Wikipedia gets really huge reviewing power and still you can find BS in it. It all depends on how sensitive your study is to human factor artifacts.
There are always "human factor artifacts". I'm not sure what the best way to deal with them would be for this system. Have an odd number of people perform the same task then assess how different the odd man out is vs having the same number of trained researchers do it (techs, interns, grad students, whatever)? Incentivize the other people contributing to score the work of their peers accurately. I agree this is an issue, but it is not unsolvable. I don't think wikipedia bureaucracy is nearly as insidious as the bureaucracy I have observed.
sub
(sounds cool, I have experience in science as well as the kind of image manipulation you're talking about)
(sounds cool, I have experience in science as well as the kind of image manipulation you're talking about)
IMO, R doesn't make pretty graphs, but it can depict exactly the info you want, then you make it pretty in illustrator.
Thanks for the kudos, btcbtc113!
I don't want to derail this thread so I'll pm you, but I'm interested in the illustrator prettifying.
Also, I do think R can produce charts sufficiently pretty to make them easy on the eye. Check this out:
http://organofcorti.blogspot.com.au/2012/04/41-slushs-pool.html
[...]
This is interesting, and likely would be tried. What would stop the "third-world" individuals from signing up themselves? Also, how do you explain the number of people that contribute to wikipedia?
This is interesting, and likely would be tried. What would stop the "third-world" individuals from signing up themselves? Also, how do you explain the number of people that contribute to wikipedia?
Culture differences in most cases (language) and economical differences in remaining ones. After all you need to own a computer with reasonable internet connection.
Wikipedia gets really huge reviewing power and still you can find BS in it. It all depends on how sensitive your study is to human factor artifacts.
In my experience, the most important thing participating in the scientific process teaches you is to acknowledge the degree of uncertainty to be dealt with and, also, subjectivity in interpreting your results. Never "believe" what you read. Wikipedia is great because this is made obvious, the appeal to authority veil has been pushed aside.
Sounds quite interesting and a step up from mechanical turk. Eg lots of people might want pretty charts for their work and not know how, only the chart they want.
You might want to have the option to be paid in btc if that's your preference though.
You might want to have the option to be paid in btc if that's your preference though.
OoC, I want you to know you were crucial in getting me to finally give in and start messing with R. You and stochastic here along with some people IRL. But you were constantly helpful and suggesting it until I realized how much I was limiting myself by relying on excel and my stubbornness to make what was in my head happen... and if I was going to incorporate something into my "workflow" it may as well be R which anyone can use instead of matlab, or worse, sigma plot and brethren. IMO, R doesn't make pretty graphs, but it can depict exactly the info you want, then you make it pretty in illustrator.
Sounds quite interesting and a step up from mechanical turk. Eg lots of people might want pretty charts for their work and not know how, only the chart they want.
You might want to have the option to be paid in btc if that's your preference though.
You might want to have the option to be paid in btc if that's your preference though.
[...]
This is interesting, and likely would be tried. What would stop the "third-world" individuals from signing up themselves? Also, how do you explain the number of people that contribute to wikipedia?
This is interesting, and likely would be tried. What would stop the "third-world" individuals from signing up themselves? Also, how do you explain the number of people that contribute to wikipedia?
Culture differences in most cases (language) and economical differences in remaining ones. After all you need to own a computer with reasonable internet connection.
Wikipedia gets really huge reviewing power and still you can find BS in it. It all depends on how sensitive your study is to human factor artifacts.
I have seen a system for transcribing recording into words. The biggest hassle is constant need to train new people. You will be surprised how hard is to train a person who is willing to be paid low end rates. The retention of people who learn quick is not high. The end of the story was that pay rates went up to retain people.
There is half-joke about an ad for a person with PhD with appropriate level salary to cut leaves into small pieces. When one applicant realized what task he is going to perform, he asked recruiter why they need a person with PhD to do such mandate and repetitive task. The answer was simple: no one else understands how important it is to do this task precisely as documented in the instruction.
However, if you do things that are not culture specific (like language specific in the case I have seen) you may have access to cheep foreign labor. There are places in the world where people will work for one hour get paid one dollar and kiss you on your legs for the privilege. What is low end pay rate in US, may be very good for high IQ person in third world country.
I already think about all these people who would like to subscribe and then subcontract the job keeping $9 per hour in their pocket. Not that it did not cross my mind.
There is half-joke about an ad for a person with PhD with appropriate level salary to cut leaves into small pieces. When one applicant realized what task he is going to perform, he asked recruiter why they need a person with PhD to do such mandate and repetitive task. The answer was simple: no one else understands how important it is to do this task precisely as documented in the instruction.
However, if you do things that are not culture specific (like language specific in the case I have seen) you may have access to cheep foreign labor. There are places in the world where people will work for one hour get paid one dollar and kiss you on your legs for the privilege. What is low end pay rate in US, may be very good for high IQ person in third world country.
I already think about all these people who would like to subscribe and then subcontract the job keeping $9 per hour in their pocket. Not that it did not cross my mind.
This is interesting, and likely would be tried. What would stop the "third-world" individuals from signing up themselves? Also, how do you explain the number of people that contribute to wikipedia?
Jump to: