[openbiblio-dev] How to speed up BibServer?

[Henryk.Krajinski]

Hello,

I've uploaded  120,000 BibTex formatted files  (one file consists one record) into BibServer over HTTP (./upload).
Just after migration the BibServer showed 500 items in the collection.
About 18 hours later it showed 25,000 items in the collection.
Then 42 hours since the upload it showed 57,000 items in the collection.

The migration process is running but it's taking up very little CPU time which means the process isn't CPU bound, there doesn't appear to be a resource issue.
Based on the log activity we estimated that it is taking 3 seconds to process one record.

Then it looks like to process 120,000 files we need at least 100 hours.
We are going to process millions of records then we need to speed the process up.

I am wondering if anybody is familiar with this kind of problem and would have any idea of how to speed up the process?

Regards
Henryk

-- 
The Open University is incorporated by Royal Charter (RC 000391), an exempt charity in England & Wales and a charity registered in Scotland (SC 038302).