[openspending-dev] How to update data from specific source?

Sun Nov 22 09:54:45 UTC 2015

Thanks for the answer, Friedrich.
Now the data seems stuck: all sources are "Still analysing" for hours... =(

[2015-11-21 10:10] Friedrich Lindenberg:

> Hey, 
> 
> I've seen this error before, as well, but I'm not sure what the origin is. In any case, the solution you've chosen (creating a new dataset) seems like the most sure-fire way to make sure that the thing gets reset fully. Re-configuring the model might also work. 
> 
> Cheers, 
> 
> - Friedrich 
> 
> On Fri, Nov 20, 2015 at 6:47 PM, <andmrm at yahoo.com.br> wrote:
>    Even dropping and reloading the data didn't work, for I got errors like this
>    for each line:
>            (IntegrityError) insert or update on table "orcamento_federal__entry"
>            violates foreign key constraint "fk_orcamento_federal_orgao" DETAIL: Key
>            (orgao_id)=(3) is not present in table "orcamento_federal__orgao". 'INSERT
>            INTO orcamento_federal__entry (id, orgao_id, funcao_id, uo_id, gnd_id,
>            time_id, subfuncao_id, pago, amount, rppago) VALUES (%(id)s, %(orgao_id)s,
>            %(funcao_id)s, %(uo_id)s, %(gnd_id)s, %(time_id)s, %(subfuncao_id)s,
>            %%(pago)s, %(amount)s, %(rppago)s)' {'subfuncao_id': 3, 'orgao_id': 3,
>            %'time_id': 1, 'uo_id': 3, 'amount': 5310158.0, 'funcao_id': 3, 'pago':
>            %95113.48, 'gnd_id': 1, 'id': '3f4c569fa77019f2c1b9e7847f0894f9e25b86f4',
>            %'rppago': 2717187.94}
>    Even using the same data and the same model.json that was working before.
>    So I created another dataset, added the data, deleted the entire dataset (to
>    test), added the data again and it seems to work.
>    So... the only way to update data is to delete the entire dataset? Am I
>    missing something?
>    [2015-11-19 15:52] :
>    >
>    > Hello! In a dataset, is there a way to update data from a specific source?
>    >
>    > We have a dataset with many sources (one source for each year), and we want to
>    > update the current year data weekly.
>    >
>    > After a source is marked as "Successfully loaded" there is no button in the
>    > website interface to unload it, and the "/sources/{id}/delete" endpoint seems
>    > not to work neither.
>    > The "Delete loaded data" button unloads all the sources, what is bad in this
>    > case, because then we have to reload ALL the sources (years) again and we just
>    > need to update the current year, not all of them.
>    > So I tried to just add a new source for the current year (duplicating it), the
>    > interface returned errors about duplicated PK (what is expected), but I hoped
>    > it to replace old lines with new lines where PKs collided, updating the data.
>    > After some months doing that, it seems I was wrong and instead of updating the
>    > data OpenSpending is keeping old lines when PKs collide, inserting only new
>    > lines (with new PKs).
>    >
>    > So, is our only option to drop and reload ALL the sources?
>    >
>    > Is there a way to delete data based on a field value?
>    > e.g.: delete lines where year==2015
>    >
>    >
>    > Thanks for the attention
>    _______________________________________________
>    openspending-dev mailing list
>    openspending-dev at lists.okfn.org
>    https://lists.okfn.org/mailman/listinfo/openspending-dev
>    Unsubscribe: https://lists.okfn.org/mailman/options/openspending-dev
> 
> 