[School-of-data] BIG data... What is BIG?

Wed Jul 9 15:24:13 UTC 2014

I guess a good definition of big data is that it’s a term used to sell tech to folks who may not otherwise need it. In that case, the limit is whatever the tool you’re selling can process.

It’s also a great way for columnists to discuss the decline of western civilisation symbolised by Mark Zuckerberg. In that case, it’s everything that gets collected by Americans. 

Best, 

- Friedrich 

p.s. I honestly don’t think there’s a good definition. “Not reasonably processable on one machine” could be a guideline, but that can be a good dozen terabytes these days?

On 09 Jul 2014, at 16:53, Simon Cropper <simoncropper at fossworkflowguides.com> wrote:

> Hi,
> 
> I have been exploring various projects that claim to handle BIG data but to be honest most do not qualify what BIG actually means.
> 
> I remember the days when programs specified the maximum number of records, maximum number of fields and maximum number of tables in a database that could be manipulated at any one time. Why don't these types of specs get provided for languages and libraries anymore?
> 
> What are people's impression of what BIG actually means when used to describe large datasets?
> 
> To me BIG is millions of records and multiple linked tables.
> 
> -- 
> Cheers Simon
> 
>   Simon Cropper - Open Content Creator
> 
>   Free and Open Source Software Workflow Guides
>   ------------------------------------------------------------
>   Introduction               http://www.fossworkflowguides.com
>   GIS Packages           http://www.fossworkflowguides.com/gis
>   bash / Python    http://www.fossworkflowguides.com/scripting
> 
> _______________________________________________
> school-of-data mailing list
> school-of-data at lists.okfn.org
> https://lists.okfn.org/mailman/listinfo/school-of-data
> Unsubscribe: https://lists.okfn.org/mailman/options/school-of-data

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 842 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://lists.okfn.org/pipermail/school-of-data/attachments/20140709/9603d45f/attachment-0002.sig>