[openbiblio-dev] updates from JISC open bib project

Thu May 12 20:41:20 UTC 2011

I was swatting a few bugs with the export routine earlier this week and
have completed the conversion.

The nquads 'export' of the new version with entity URIs is being
uploaded to archive.org... very slowly unfortunately (*A*DSL here).

The solr indexer is ready to go - although I think the production schema
is out of sync with the current version, I'll switch the schema later on
tonight but I may need guidance as to how to correctly (ie by which
user, etc) reboot the solr container later.

How to index:

(from within the openbiblio environment:)

paster indexnquads development.ini *.nquads

Few command flags to note:

--batch=, -b=   -- set the number of 'docs' to batch up before sending
an update to solr. --batch=1000 is a good option.

--json   -- instead of trying to update solr, serialise the solr docs to
disc. solrupdate0.json, solrupdate1.json, etc. These files can be used
to rapidly update solr by the following:

paster indexjson development.ini solrupdate*

Ben

On Thu, 2011-05-12 at 20:26 +0200, William Waites wrote:
> The medline stuff is being uploaded as we speak, has finished being
> generated. Announcement to follow along with some vital statistics. It
> will just be available for download, not for querying as it is rather
> large.
> 
> For the reincarnated bibliographica, I'll just trash the existing
> store and load it in once Ben tells me where the data that is.
> 
> Cheers,
> -w
> 
> * [2011-05-12 19:20:42 +0100] Mark MacGillivray <mark at cottagelabs.com> écrit:
> 
> ] This week we are working through the data we recently received from
> ] PubMed and preparing a fresh instance of our bibliographica, which
> ] will also have a SOLR index available. Additionally, we are working on
> ] some visualisations which should show just how useful this sort of
> ] open bibliographic data can be.
> ] 
> ] Ben, Will - is all going well with the plan to have some
> ] visualisations, a new instance, and a SOLR index up by Sunday? Please
> ] reply with some details to this on the dev list, so we can keep people
> ] up to date with the coming changes.
> ] 
> ] Thanks,
> ] 
> ] 
> ] Mark
> ] 
> ] _______________________________________________
> ] openbiblio-dev mailing list
> ] openbiblio-dev at lists.okfn.org
> ] http://lists.okfn.org/mailman/listinfo/openbiblio-dev
> 