[iRail] UK railway data

[Yeri Tiete]

Cool. You're my hero.

De Lijn is about 1Gb of data as well if I'm not mistaken, and importing it takes about 2 hours. 
If we can wget those 2Gb once per week and import it at night, that'd be great. (*)

How much data does it hold by the way? I didn't really check it in detail.
All of Great Britain? Just UK? What about London underground? 

That 5Mb archive holding updates, what does it actually contain? 

(*) This brings me to another thing though: to prevent inconsistent/duplicate data, we drop both De Lijn & STIB schema and recreate/reimport everything. 
Dropping just the data takes significantly longer. 
However, this brings a problem that, if anyone would request the data at that period of time, the database would be empty. Do you have a solution/idea for this? 
This is something Maarten Cautreels worked on. 

Thanks again!

--
Kind regards,
Yeri Tiete
FlatTurtle

Avenue du port, 86c - 18, 1000 Brussels

m: +32 (0) 474 61 0139
t:   +32 (0) 2 669 1001
f:   +32 (0) 2 669 1002


On 21 Aug 2012, at 18:41, Quentin Kaiser wrote:

> Hey ! 
> 
> I think I'll write some blog post about it because there is a lot of things to describe and explain but here is some updates : 
> 
> - train stations with TIPLOC (timing point location, specific id in their system) and geolocation is up here : http://data.quentinkaiser.be/NetworkRail/Stations.xml.  I have to update geolocation because my coordinates transformation script wasn't doing so well (I had to transform from northing/easting to latitude/longitude).
> - I'm working on semi-static data (schedules). It's really BIG data (1.9Gb of json) that we have to parse. Updates are provided daily via an archive on amazon S3 containing around 5Mb of json. So if you want to provide up-to-date informations via data.irail.be I can write a wget/parse/store daily cronjob to get it done. 
> - As TheDatatank is mainly using MySQL I wrote a script to read data from the 1.9Gb and insert it into tables, it's currently running and I don't know exactly when it's going to finish :D
> - they don't provide some kind of database scheme so I spent some time to see what was going on into the raw json with mongo.
> 
> As soon as I get a working local database with schedules I'll push everything in here : https://github.com/QKaiser/NetworkRailResource
> 
> Quentin
> 
> 
> 
> 
> 2012/8/19 Yeri Tiete <yeri at flatturtle.com>
> Hi Quentin,
> 
> What's the status on this? Had any time to get it to work? 
> 
> 
> --
> Kind regards,
> Yeri Tiete
> FlatTurtle
> 
> Avenue du port, 86c - 18, 1000 Brussels
> 
> m: +32 (0) 474 61 0139
> t:   +32 (0) 2 669 1001
> f:   +32 (0) 2 669 1002
> 
> 
> On 09 Aug 2012, at 15:52, Yeri Tiete wrote:
> 
>> Cool. Many thanks Quentin :)
>> 
>> --
>> Kind regards,
>> Yeri Tiete
>> FlatTurtle.com
>> Sent from my Nexus 7
>> 
>> Quentin Kaiser <kaiserquentin at gmail.com> wrote:
>> 
>> Hi all,
>> 
>> I think I can do it ! Besides, it's a really interesting API (message-oriented with brokers and subscribers), take a look at this document [1] even if you don't want to develop something out of it. There is plenty of examples for different languages.
>> I'll take a closer look on this tonight. I keep you posted ;-)
>> 
>> Cheers !
>> 
>> Quentin
>> 
>> [1] : http://www.networkrail.co.uk/WorkArea/DownloadAsset.aspx?id=30064782140
>> 
>> 2012/8/9 Yeri Tiete <yeri at flatturtle.com>
>> Dear All,
>> 
>> We're looking for volunteers to implement Network Rail data into our DataTank interface.
>> 
>> It requires a signup, but after that it's free to use. 
>> 
>> Who's up for it? *puppy eyes* 
>> 
>> --
>> Kind regards,
>> Yeri Tiete
>> FlatTurtle
>> 
>> Avenue du port, 86c - 18, 1000 Brussels
>> 
>> m: +32 (0) 474 61 0139
>> t:   +32 (0) 2 669 1001
>> f:   +32 (0) 2 669 1002
>> 
>> 
>> 
>> _______________________________________________
>> iRail mailing list
>> iRail at list.irail.be
>> http://lists.rootspirit.com/mailman/listinfo/irail
>> 
>> 
>> 
>> 
>> -- 
>> Quentin Kaiser
>> 
>> 
>> _______________________________________________
>> iRail mailing list
>> iRail at list.irail.be
>> http://lists.rootspirit.com/mailman/listinfo/irail
> 
> 
> _______________________________________________
> iRail mailing list
> iRail at list.irail.be
> http://lists.rootspirit.com/mailman/listinfo/irail
> 
> 
> 
> 
> -- 
> Quentin Kaiser
> 
> 
> _______________________________________________
> iRail mailing list
> iRail at list.irail.be
> http://lists.rootspirit.com/mailman/listinfo/irail

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/irail/attachments/20120821/3139faf5/attachment-0003.html>