[openbiblio-dev] How to speed up BibServer?

[Mark MacGillivray]

This is most likely a slowdown of the translators rather than bibserver
itself, although even then I am unsure why it would be going so slow. If
you have a lot of translations to run, it would probably be best to do them
directly first using the bibtex translator and creating bibsjon output.
then upload the bibjson directly for faster response.


On Thu, May 2, 2013 at 3:42 PM, Henryk.Krajinski <
Henryk.Krajinski at open.ac.uk> wrote:

> Hello,
>
> I've uploaded  120,000 BibTex formatted files  (one file consists one
> record) into BibServer over HTTP (./upload).
> Just after migration the BibServer showed 500 items in the collection.
> About 18 hours later it showed 25,000 items in the collection.
> Then 42 hours since the upload it showed 57,000 items in the collection.
>
> The migration process is running but it's taking up very little CPU time
> which means the process isn't CPU bound, there doesn't appear to be a
> resource issue.
> Based on the log activity we estimated that it is taking 3 seconds to
> process one record.
>
> Then it looks like to process 120,000 files we need at least 100 hours.
> We are going to process millions of records then we need to speed the
> process up.
>
> I am wondering if anybody is familiar with this kind of problem and would
> have any idea of how to speed up the process?
>
> Regards
> Henryk
>
> --
> The Open University is incorporated by Royal Charter (RC 000391), an
> exempt charity in England & Wales and a charity registered in Scotland (SC
> 038302).
>
>
> _______________________________________________
> openbiblio-dev mailing list
> openbiblio-dev at lists.okfn.org
> http://lists.okfn.org/mailman/listinfo/openbiblio-dev
> Unsubscribe: http://lists.okfn.org/mailman/options/openbiblio-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/openbiblio-dev/attachments/20130502/c4093490/attachment.html>