[openspending-dev] How to update data from specific source?

[Daniel Fowler]

Hi,

Thanks Friedrich.  

I’ve resolved the “Still analysing” problem.  Can you try re-adding the dataset?

Also, I want to let you know that we’re working on a new version of OpenSpending.  You can find out more about how to get involved here:

http://community.openspending.org/get-involved/


-- 
Daniel Fowler
daniel.fowler at okfn.org
+251 93 368 4124 | skype: dan.d.fowler  |  @danfowler
Open Knowledge International - A world where knowledge creates power for the many, not the few
http://okfn.org/  |  @okfn 


> On Nov 22, 2015, at 12:54, <andmrm at yahoo.com.br> <andmrm at yahoo.com.br> wrote:
> 
> 
> Thanks for the answer, Friedrich.
> Now the data seems stuck: all sources are "Still analysing" for hours... =(
> 
> 
> 
> [2015-11-21 10:10] Friedrich Lindenberg:
> 
>> Hey, 
>> 
>> I've seen this error before, as well, but I'm not sure what the origin is. In any case, the solution you've chosen (creating a new dataset) seems like the most sure-fire way to make sure that the thing gets reset fully. Re-configuring the model might also work. 
>> 
>> Cheers, 
>> 
>> - Friedrich 
>> 
>> On Fri, Nov 20, 2015 at 6:47 PM, <andmrm at yahoo.com.br> wrote:
>>   Even dropping and reloading the data didn't work, for I got errors like this
>>   for each line:
>>           (IntegrityError) insert or update on table "orcamento_federal__entry"
>>           violates foreign key constraint "fk_orcamento_federal_orgao" DETAIL: Key
>>           (orgao_id)=(3) is not present in table "orcamento_federal__orgao". 'INSERT
>>           INTO orcamento_federal__entry (id, orgao_id, funcao_id, uo_id, gnd_id,
>>           time_id, subfuncao_id, pago, amount, rppago) VALUES (%(id)s, %(orgao_id)s,
>>           %(funcao_id)s, %(uo_id)s, %(gnd_id)s, %(time_id)s, %(subfuncao_id)s,
>>           %%(pago)s, %(amount)s, %(rppago)s)' {'subfuncao_id': 3, 'orgao_id': 3,
>>           %'time_id': 1, 'uo_id': 3, 'amount': 5310158.0, 'funcao_id': 3, 'pago':
>>           %95113.48, 'gnd_id': 1, 'id': '3f4c569fa77019f2c1b9e7847f0894f9e25b86f4',
>>           %'rppago': 2717187.94}
>>   Even using the same data and the same model.json that was working before.
>>   So I created another dataset, added the data, deleted the entire dataset (to
>>   test), added the data again and it seems to work.
>>   So... the only way to update data is to delete the entire dataset? Am I
>>   missing something?
>>   [2015-11-19 15:52] :
>>> 
>>> Hello! In a dataset, is there a way to update data from a specific source?
>>> 
>>> We have a dataset with many sources (one source for each year), and we want to
>>> update the current year data weekly.
>>> 
>>> After a source is marked as "Successfully loaded" there is no button in the
>>> website interface to unload it, and the "/sources/{id}/delete" endpoint seems
>>> not to work neither.
>>> The "Delete loaded data" button unloads all the sources, what is bad in this
>>> case, because then we have to reload ALL the sources (years) again and we just
>>> need to update the current year, not all of them.
>>> So I tried to just add a new source for the current year (duplicating it), the
>>> interface returned errors about duplicated PK (what is expected), but I hoped
>>> it to replace old lines with new lines where PKs collided, updating the data.
>>> After some months doing that, it seems I was wrong and instead of updating the
>>> data OpenSpending is keeping old lines when PKs collide, inserting only new
>>> lines (with new PKs).
>>> 
>>> So, is our only option to drop and reload ALL the sources?
>>> 
>>> Is there a way to delete data based on a field value?
>>> e.g.: delete lines where year==2015
>>> 
>>> 
>>> Thanks for the attention
>>   _______________________________________________
>>   openspending-dev mailing list
>>   openspending-dev at lists.okfn.org
>>   https://lists.okfn.org/mailman/listinfo/openspending-dev
>>   Unsubscribe: https://lists.okfn.org/mailman/options/openspending-dev
>> 
>> 
> _______________________________________________
> openspending-dev mailing list
> openspending-dev at lists.okfn.org
> https://lists.okfn.org/mailman/listinfo/openspending-dev
> Unsubscribe: https://lists.okfn.org/mailman/options/openspending-dev