[@OKau] OpenAus

Sun Jun 28 07:55:48 UTC 2015

As it turns out, I can run most of the queries off a single table with all the data in it now I'm on a VPS. Nearly 700k rows and counting.
I realised that if I recode the tenders pages now it will simplify everything for users and I won't have to explain those issues in the documentation because they no longer exist. 
Not quite finished so it's still buggy but search by financial/calendar or any other two dates is implemented in the tenders sub-project. 
I'm now doing a script to create graphs that works with contract terms of any length (to avoid issues where the end date of a contract is beyond the presently hard-coded maximum year & mucking up the visual display) by generating the min and max dates of all contracts for that query on the fly. 
I'll have to move the queries from searching by publish dates to searching by contract start dates as there are many cases where the publish date is after the start date of the contract and this breaks the graphs if the graph is generated using the publish dates of the tender contracts.

Rosie Williams BA (Sociology)________________________________________
 NoFibs.com.au - Open Data Reporter | InfoAus.net - Founder and Developer 

From: budgetaus at hotmail.com
To: okfn-au at lists.okfn.org
Date: Fri, 26 Jun 2015 14:20:15 +1030
Subject: Re: [@OKau] OpenAus

I'm in two minds about allowing users to select date ranges for results. I implemented it with the grants data but not the tenders data. There's a couple of reasons for that and I did have two options for the grants data originally, both by start date and end date. I took that back to just selecting data by start dates because I thought it was too complex for the typical user and would just confuse people. I was assuming people applying for grants would be looking for specific grants by the date they start and whether they'd prefer to search by calendar or financial year but I'll clarify this during consultation. Thanks for bringing it up.
The tenders data has it's own problems with the 500,000 rows of data queries simply would not run so I had to break this data up into 6 different tables, and write 6 queries creating 6 subtotals. The first is actually all amended tenders which is why it looks like it overlaps in time with the other subtotals. Like I said, I haven't done any of the user documentation yet. 
Now that I've moved to a VPS perhaps I can combine all the tenders data into one table and have the one total. I'm new to dealing with such large amounts of data but it would cut down the code to a sixth of it's current size and make a lot more sense from a user perspective. Having said all that,  this is not a finished product/service (more of a starting point) and will only be going ahead and being improved with feedback from users if it gains enough financial support for me to survive on. I have to be able to pay for the VPS for example (along with everything else).
It's a lot of work so I'm just taking it one stage at a time and apportioning tasks to before and after funding - deciding what to leave till after I find out if it is going to actually run at all. 
Thanks for taking an interest.
Rosie Williams BA (Sociology)________________________________________
 NoFibs.com.au - Open Data Reporter | InfoAus.net - Founder and Developer 

From: craig.thomler at gmail.com
Date: Fri, 26 Jun 2015 11:53:32 +1000
To: okfn-au at lists.okfn.org
Subject: Re: [@OKau] OpenAus

Hi Rosie, It look good, however I can't see any reference to time frames to 'place' the data by period. This makes it hard to judge whether the data is aggregated over a long period and represents the net total from successive governments. Some more context in this regard, as well as the capability to bound searches by dates (at minimum financial year) would assist in making sense of the data and help people use it effectively. Cheers,
Craig _________________________________________________
Craig Thomler

http://egovau.blogspot.com
http://twitter.com/CraigThomler
http://au.linkedin.com/in/craigthomler
http://www.slideshare.net/CraigThomler 

Mobile: 0411 780 194 (International: +61 411 780 194)Phone: 02 6161 4508 (International: +61 2 6161 4508)Skype: craig.thomler

On 26 June 2015 at 09:43, Rosie Williams <budgetaus at hotmail.com> wrote:

Hi all, 
I've just migrated the project I'm about to launch to my very first VPS. I've been given quite a bit of free memory for the first week so I thought I'd give the site a public airing and see how it performs. I'm trying to guage how much memory I'll need going forward. 
The project already feels large and it has only just begun so has the potential to get quite massive if it all goes ahead and I end up adding grants data from multiple agencies/jurisdictions. However at the moment I'm mainly interested in what I'll be needing in the coming months. 
I'm not quite up to the user documentation. Domain name is currently http://ausgov.org and will be changing after testing to http://openaus.net.au 
Datasets used are Commonwealth Tenders, DSS Grants, DCCSDS Grants and the ACNC register. Data is not absolute freshest available but is quite recent. I'm also using some data from Wikipedia which I'll add in shortly.
Source is open now too.

Rosie Williams BA (Sociology)________________________________________
 NoFibs.com.au - Open Data Reporter | InfoAus.net - Founder and Developer 

_______________________________________________

okfn-au mailing list

okfn-au at lists.okfn.org

https://lists.okfn.org/mailman/listinfo/okfn-au

Unsubscribe: https://lists.okfn.org/mailman/options/okfn-au

_______________________________________________
okfn-au mailing list
okfn-au at lists.okfn.org
https://lists.okfn.org/mailman/listinfo/okfn-au
Unsubscribe: https://lists.okfn.org/mailman/options/okfn-au 		 	   		  

_______________________________________________
okfn-au mailing list
okfn-au at lists.okfn.org
https://lists.okfn.org/mailman/listinfo/okfn-au
Unsubscribe: https://lists.okfn.org/mailman/options/okfn-au 		 	   		  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/okfn-au/attachments/20150628/b6e02b26/attachment-0004.html>