[OpenSpending] Portuguese database of contracts

Thu Jan 23 16:31:00 UTC 2014

Hi Rufus

- Portuguese database:

>From the db point of view, the data is the same.

The db structure is different since one contract in mine has two
many-to-many with entities
 (a contract between several entities is possible), while in the
OpenSpending is a Many-To-One, as pointed out in this blog
entry<http://community.openspending.org/2013/12/processing-data-for-open-spending/>in
openspending by Daniel and Olaf.

Maybe there is another difference, which I believe is less relevant: the
categories in the public procurements in EU create a tree (CPVS).
In my db this is created using a nested set
model<https://en.wikipedia.org/wiki/Nested_set_model>(very expensive
to add, very easy to read because it is static), which I
don't know how is implemented in OpenSpending.

I talked with Olaf and I believe the issue of the dataset in OpenSpending
will be solved soon or is already solved :)

- automate analysis:

I think I don't know enough about database design and database technology
to help you at this point.

- OpenSpending

I already took a look at the code, and It seems quite good,
congratulations. I will definitely contribute when and where I can.
>From what I understood, OpenSpending uses Pylons. I tried SQLAlchemy and
Jinga2 before, but Pylons I've never used before; I think it is no problem
thought for small to medium issues. :P

Cheers,
Jorge

On Thu, Jan 23, 2014 at 12:29 PM, Rufus Pollock <rufus.pollock at okfn.org>wrote:

> Hi Jorge,
>
> You've come to the right place and this is very exciting.
>
> Some immediate things:
>
> - Here is some info on existing work<https://github.com/openspending/datatoload/issues/6>by an OpenSpending community member  scraping contracts from
> base.gov.pt  with results live at
> https://openspending.org/pt_ajustes-diretos - would you be interested in
> connecting here?
>
> - We're also interested in collecting contract data more broadly across
> countries - see the list of "Contract" issues in the data to load tracker<https://github.com/openspending/datatoload/issues?labels=Type%3A+Contracts>.
> We've also been thinking about how to automate analysis - e.g. pushing
> datasets automatically into Google BigQuery or Amazon Redshift (more info
> here <https://github.com/openspending/ideas/issues/23>)
>
> - The OpenSpending platform is built in python :-) Given your python
> expertise you may be interested in this! Code is at
> https://github.com/openspending/openspending and hacking instructions are
> here <http://community.openspending.org/help/development/>
>
> Regards,
>
> Rufus
>
>
> On 23 January 2014 10:08, Jorge Cardoso Leitão <jorgecarleitao at gmail.com>wrote:
>
>> This is a cross-post from the mailing list of open-government, as it was
>> suggested
>>  there that this update would make more sense here.
>> Sorry for that.
>>
>> --------------------
>>
>> Hi.
>>
>> I'm a portuguese doing my PhD in Dresden (on physics) and on my free time
>> I do some analysis to the portuguese public procurements. I recently found
>> the open knowledge foundation.
>>
>> In concrete, I have a website and API using Django ORM (github<https://github.com/jorgecarleitao/public-contracts>)
>> that daily crawls* the official portuguese database<https://index.okfn.org/country/Portugal/spending> of
>> public procurements and I provide open sql access to a remote mysql
>> database I designed and populated (see readme<http://public-contracts.readthedocs.org/en/latest/tools/database.html>).
>> I'm now asking questions to the data like "When do portuguese
>> municipalities contract most?<http://contratos.publicos.pt/quando-e-que-os-municipios-mais-contratam>
>> "
>>
>> It seems that you are already doing this kind of stuff for some time,
>> and, in particular, I'm doing what the international transparency in
>> Slovakia is doing<http://tender.sme.sk/en/reports?cut=contract_date:2014,01>
>> .
>>
>> What I've done so far is of a very limited scope: only now I'm finding
>> that there is an actual community very interested in this kind of stuff,
>> which I'm thrilled to find!
>>
>> So, with this e-mail I'm trying to introduce myself and this little
>> project to this community, in the hope of receiving some feedback. I'm a
>> little overwhelmed by the amount of information the foundation has on the
>> website, and some guidance would be great. E.g., does this database and/or
>> project fits in the foundation in any way?
>>
>> In the meanwhile, how can I help you?
>>
>> I like this subject a lot and I agree with the vision of the foundation.
>> I'm not an expert in database or web development, thus I'm more into posing
>> scientific questions. Nevertheless, I'm also capable of designing and
>> querying a database, git, etc.
>>
>> Cheers,
>> Jorge
>>
>>
>> *The crawling was made posterior to confirmation with the official
>> database provider that I'm allowed to do so ;-)
>>
>> _______________________________________________
>> openspending mailing list
>> openspending at lists.okfn.org
>> https://lists.okfn.org/mailman/listinfo/openspending
>> Unsubscribe: https://lists.okfn.org/mailman/options/openspending
>>
>>
>
>
> --
>
>
> * Rufus Pollock Founder and Executive Director | skype: rufuspollock |
> @rufuspollock <https://twitter.com/rufuspollock> The Open Knowledge
> Foundation <http://okfn.org/> Empowering through Open Knowledge
> http://okfn.org/ <http://okfn.org/> | @okfn <http://twitter.com/OKFN> | OKF
> on Facebook <https://www.facebook.com/OKFNetwork> |  Blog
> <http://blog.okfn.org/>  |  Newsletter <http://okfn.org/about/newsletter> *
>
> _______________________________________________
> openspending mailing list
> openspending at lists.okfn.org
> https://lists.okfn.org/mailman/listinfo/openspending
> Unsubscribe: https://lists.okfn.org/mailman/options/openspending
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/openspending/attachments/20140123/c58d17ef/attachment-0002.html>