[open-science] IPCC report

Thu Oct 3 07:24:54 UTC 2013

Hi Marc,

On Wed, Oct 2, 2013 at 5:09 PM, Couture Marc <marc.couture at teluq.ca> wrote:

>  Daniel suggests the following for the crowdcrafting application:
>
>
>
> >
>
> > * Each citizen is given a reference. and asked "can you read this"
>
> >
>
> > […]
>
> >
>
> > * they click questions such as "is this paper on a public site?" "is it
> the publisher
>
> > site?" "can you access the full-text?" "if not, how much does it cost?"
> "please save
>
> > the URL"
>
> >
>

Actually that was suggested by Peter :-) And I like his proposal.

>
>  I don’t know if this forum is the right place to discuss these details;
> maybe one should put up an EtherPad or Google Doc  to that effet. Anyway,
> I’ll go ahead.
>
Well,  I think that as the first e-mail included this list, it is fair to
continue the discussion, unless someone says we should move to another
place :-)

>
>
> I don’t have any concrete experience with crowdcrafting, only a knowledge
> derived from reading about some projects, mainly Galaxy Zoo, and I had a
> look at Ross Mounce’s app for OA journals copyright policies (a subject I
> know very well).
>
>
>
> All I can say is that different people will come up with different answers
> to these questions; some will find an OA version, others won’t. And I know,
> because I do it on a regular basis for a journal, that OA versions are
> sometimes quite hard to find.
>

IMHO this is also interesting :-) The perception of users regarding OA
versions could be something really good to analyze too. It would be cool to
show which references generated more disagreement regarding the chosen
licenses. For example, for our ForestWatchers.net project, we analyze the
disagreement for specific
tasks<http://forestwatchers.net/pybossa/app/besttile/>(check the map,
and select the Heat Map layer), and it reveals very useful
information: the most cloudy areas, are the most difficult ones.

>
>
> My point is that I don’t know what would be the quality of a survey in
> which each reference is checked by only one person. In Galaxy Zoo, for
> instance, they combine answers from many participants and empirically
> determined thresholds (number of participants and % of agreement) for an
> answer to be considered reliable (in their case, as reliable as what would
> be obtained by a trained astrophysicist).
>

You have raised a very good point: using only one user is bad, really bad.
For this reason, and by default, CrowdCrafting and any PyBossa server sends
an application task to at least 30 different persons, so you can do the
statistical analysis correctly.

PyBossa allows you to change that value, to have a bigger sample, or a
smaller one, but by default, if you do not touch anything you will get 30
samples per task. In other words, each task is going to be reviewed by 30
different people, so as you can see we are doing the same as Galaxy Zoo,
and other citizen science projects.

CrowdCrafting and PyBossa also supports "golden tasks" (in PyBossa terms I
called them calibration tasks) which means that you can weight the answer
of the users based on the answer given to the golden ones.

Thus, the application is covered by the server unless the authors decide to
change that value to something like one (i.e. for testing purposes). In
summary, CrowdCrafting follows the same pattern as GalaxyZoo.

I hope this is much clearer now :-)

Cheers,

Daniel

PS: PyBossa also supports task priorities, so you can change the priority
of apps as you wish.
PS2: PyBossa is the open-source software that powers CrowdCrafting.org and
ForestWatchers.net

>
>
> Marc Couture
>
>
>
>
>
>
>
> *De :* open-science-bounces at lists.okfn.org [mailto:
> open-science-bounces at lists.okfn.org] *De la part de* Daniel Lombraña
> González
> *Envoyé :* 2 octobre 2013 04:15
> *À :* Peter Murray-Rust
> *Cc :* open-science; okfn-discuss
> *Objet :* Re: [open-science] IPCC report
>
>
>
> Hi Peter,
>
>
>
> On Wed, Oct 2, 2013 at 9:57 AM, Peter Murray-Rust <pm286 at cam.ac.uk> wrote:
>
> Great,
>
> I'll probably try to hack bits of the report today.
>
>
>
> Nice!
>
>
>
>
> The way I see it is something like:
>
> * Crowdcrafting is given 9000 references:
>
> * Each citizen is given a reference. and asked "can you read this"
>
> * they are expected to paste the text into Google or some other search
> engine (maybe Microsoft Academic Search)
>
>
>
> In the application I mentioned before, I basically avoid the copy&paste
> action by giving a button that will directly do the search in Google
> Scholar in a new tab. You can adapt it easily to any service :-)
>
>    * they click questions such as "is this paper on a public site?" "is
> it the publisher site?" "can you access the full-text?" "if not, how much
> does it cost?" "please save the URL"
>
>
>
> It makes a lot of sense :-) Once the user has all that info, they will
> click a button with the Save URL or Send report, and a new task will be
> loaded for them.
>
>
>
>
> and repeat.  It's up to our app to keep track of the results.
>
>
>
> Yep! You will be able to get all the answers via JavaScript, or if you
> prefer, download them, do the statistics first, and then generate the
> output. This is up to you to decide how do you want to achieve it.
>
>
>
>
> There's slightly more cut and paste than normal, but many citizens should
> have high motivation. My guess it will take about 0.5-3 mins. We may need
> notes on how to navigate some journals. Their interfaces are awful.
>
>
>
> That's something you can address in the tutorial. Every CrowdCrafting.org
> application can have one, so all you have to do is to create one for yours
> :-)
>
>
>
>
> We also need a wiki/mail - e.g. how do we find the cost for Journal X...
>
>
>
> That's something CrowdCrafting does not have, but there should be free
> Wikis or Etherpads to coordinate yourself.
>
>
>
>
> I think it could be exciting, rapid and very worthwhile.
>
>
>
> Indeed! If promoted well, you will get a lot of people!
>
> Cheers,
>
> Daniel
>
> _______________________________________________
> open-science mailing list
> open-science at lists.okfn.org
> http://lists.okfn.org/mailman/listinfo/open-science
> Unsubscribe: http://lists.okfn.org/mailman/options/open-science
>
>

-- 
http://daniellombrana.es
http://citizencyberscience.net
http://www.shuttleworthfoundation.org/fellows/daniel-lombrana/
··························································································································································
Please do NOT use proprietary file formats to share files
like DOC or XLS, instead use PDF, HTML, RTF, TXT, CSV or
any other format that does not impose on the user the employment
of any specific software to work with the information inside the files.
··························································································································································
Por favor, NO utilice formatos de archivo propietarios para el
intercambio de documentos, como DOC y XLS, sino PDF, HTML, RTF, TXT, CSV
o cualquier otro que no obligue a utilizar un programa de un
fabricante concreto para tratar la información contenida en él.
··························································································································································
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/open-science/attachments/20131003/de2176c8/attachment-0001.html>