[School-of-data] BIG data... What is BIG?

Wed Jul 9 16:00:09 UTC 2014

Hi there,

One aspect of "big data" in engineering is that the database doesn't fit in
a single machine and therefore is distributed among several computers
(large search engines index for instance). Those sort of data requires
specific algorithms development that can handle concurrent tasks on
multiple clusters of machines.
I don't think something that doesn"t fit in your RAM can be called big
data.

But I agree with Friedrich : it is mostly a keyword for the industry to get
excited about sth new.
Unlike "Big Oil" it does still have a sexy feeling for the audience when
said by journalists or policy makers.

2014-07-09 17:24 GMT+02:00 Friedrich Lindenberg <friedrich at pudo.org>:

> I guess a good definition of big data is that it’s a term used to sell
> tech to folks who may not otherwise need it. In that case, the limit is
> whatever the tool you’re selling can process.
>
> It’s also a great way for columnists to discuss the decline of western
> civilisation symbolised by Mark Zuckerberg. In that case, it’s everything
> that gets collected by Americans.
>
> Best,
>
> - Friedrich
>
> p.s. I honestly don’t think there’s a good definition. “Not reasonably
> processable on one machine” could be a guideline, but that can be a good
> dozen terabytes these days?
>
> On 09 Jul 2014, at 16:53, Simon Cropper <
> simoncropper at fossworkflowguides.com> wrote:
>
> > Hi,
> >
> > I have been exploring various projects that claim to handle BIG data but
> to be honest most do not qualify what BIG actually means.
> >
> > I remember the days when programs specified the maximum number of
> records, maximum number of fields and maximum number of tables in a
> database that could be manipulated at any one time. Why don't these types
> of specs get provided for languages and libraries anymore?
> >
> > What are people's impression of what BIG actually means when used to
> describe large datasets?
> >
> > To me BIG is millions of records and multiple linked tables.
> >
> > --
> > Cheers Simon
> >
> >   Simon Cropper - Open Content Creator
> >
> >   Free and Open Source Software Workflow Guides
> >   ------------------------------------------------------------
> >   Introduction               http://www.fossworkflowguides.com
> >   GIS Packages           http://www.fossworkflowguides.com/gis
> >   bash / Python    http://www.fossworkflowguides.com/scripting
> >
> > _______________________________________________
> > school-of-data mailing list
> > school-of-data at lists.okfn.org
> > https://lists.okfn.org/mailman/listinfo/school-of-data
> > Unsubscribe: https://lists.okfn.org/mailman/options/school-of-data
>
>
> _______________________________________________
> school-of-data mailing list
> school-of-data at lists.okfn.org
> https://lists.okfn.org/mailman/listinfo/school-of-data
> Unsubscribe: https://lists.okfn.org/mailman/options/school-of-data
>

-- 
Clément Renaud

@clemsos
www.clementrenaud.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/school-of-data/attachments/20140709/eb23be80/attachment-0002.html>