[ckan-dev] DataStorer and preview troubleshooting

Mon Jul 30 17:42:18 UTC 2012

Thanks, guys... So I have a possibly related question that pertains to
ElasticSearch.

When I do an API search call against the *working* CSV resource, the
returned JSON references what I presume is ElasticSearch stuff, namely
5 shards and this:

"_index": "ckan-data.cityofsantacruz.com"

in addition to the actual data returned.

Today while troubleshooting, I noticed that I'm getting several errors
in my ElasticSearch logs similar to this:

[2012-07-28 22:35:57,899][DEBUG][action.index             ]
[Gatecrasher] [ckan-data.cityofsantacruz.com][3],
node[t67o_ONdTLOlKEVkKqd-LQ], [P], s[STARTED]: Failed to execute
[index {[ckan-data.cityofsantacruz.com][2d4beb1b-80b5-4df4-a7ec-305bd3313ce4][smbshr.pl],
source[host=%22%20%2DFOOBAR%7Cecho%20%22%20Sharename%22%0Aecho%0Aecho%20%22%20%20SomeShare%20%20Disk%20%22%60id%60%20%23%22]}]

Now my question is -- where does "ckan-data.cityofsantacruz.com" from?
 I grepped my entire home directory (where I installed ElasticSearch,
CKAN, extensions, etc.) as well as /etc/apache2, /etc/supervisor,
everywhere else I could think of and I couldn't find *any* references
to that hostname -- or even the string "ckan-data" -- other than in
the logs.  Is that the "dynamically generated node name" that the ES
config file documentation is referring to?

I'm also wondering if host=FOOBAR in there is indicative of another problem :-)

Again, ElasticSearch, Solr, Nginx -- everything seemed to work just
fine on Friday when I demo'ed the Recline graphing functionality on
this server for some colleagues.  So I'm not sure what is now
happening.

Thanks again for any suggestions!
Jim

On Mon, Jul 30, 2012 at 10:18 AM, Rufus Pollock <rufus.pollock at okfn.org> wrote:
> Great debugging Jim. I've pinged David Raznick who knows the most
> about celery stuff and may have insight into this rather obscure
> behaviour.
>
> Rufus
>
> On 30 July 2012 18:04, Jim Craner <jim at codeforamerica.org> wrote:
>> I did some more testing based on past troubleshooting advice from this list...
>>
>> So I can upload data via the API:
>>
>> prompt$ curl -XPUT http://localhost/api/data/<resource_id>/1 -d
>> '{"test":"data"}' -H 'Authorization: <API key of my admin user>'
>>
>> successfully replaces the CSV that was in that resource with the
>> "test, data" table and preview works, API calls against it work, etc.
>>
>> So it seems that it's just when files are uploaded via the web
>> interface that no DataStore is being created.  My celery log is still
>> showing the "AttributeError: 'NoneType' object has no attribute
>> 'read'" error referenced in the pastebin I made
>> (http://pastebin.com/jtXUbvJx) but I can't tell if this is the
>> archiver or datastorer process or something else altogether.
>>
>> Any suggestions greatly appreciated!
>> Thanks,
>> Jim
>>
>>
>> On Sun, Jul 29, 2012 at 9:19 PM, Jim Craner <jim at codeforamerica.org> wrote:
>>> <premature enter-hitting>
>>>
>>> Thanks for any advice you can suggest!
>>> -Jim
>>>
>>> On Sun, Jul 29, 2012 at 9:19 PM, Jim Craner <jim at codeforamerica.org> wrote:
>>>> Hi,
>>>>
>>>> I successfully had CKAN 1.7.1 running, including DataStorer storing
>>>> CSV resources and the ability to preview them with Recline.
>>>>
>>>> I logged in today and resources that previously displayed previews
>>>> properly in Recline now show the Dreaded Pink Fail Bar of Doom.
>>>>
>>>> I then tried to upload some new test CSV resources, including ones I'd
>>>> successfully uploaded and previewed before, and got the same thing.  I
>>>> checked the celeryd.log and it was yielding the following:
>>>>
>>>> [2012-07-29 21:08:44,507: INFO/MainProcess] Got task from broker:
>>>> datastorer.upload[814f9861-a34c-4b4c-993f-f7c8358cd9b4]
>>>> [2012-07-29 21:10:04,677: ERROR/MainProcess] Task
>>>> datastorer.upload[814f9861-a34c-4b4c-993f-f7c8358cd9b4] raised
>>>> exception: AttributeError("'NoneType' object has no attribute
>>>> 'read'",)
>>>>
>>>> I pasted the entire traceback from the celeryd.log here:
>>>> http://pastebin.com/jtXUbvJx
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> -----
>>>> Jim Craner, 2012 Fellow
>>>> Code for America
>>>> jim at codeforamerica.org
>>>> (+1) 773-809-4546
>>>> http://codeforamerica.org
>>>
>>>
>>>
>>> --
>>> -----
>>> Jim Craner, 2012 Fellow
>>> Code for America
>>> jim at codeforamerica.org
>>> (+1) 773-809-4546
>>> http://codeforamerica.org
>>
>>
>>
>> --
>> -----
>> Jim Craner, 2012 Fellow
>> Code for America
>> jim at codeforamerica.org
>> (+1) 773-809-4546
>> http://codeforamerica.org
>>
>> _______________________________________________
>> ckan-dev mailing list
>> ckan-dev at lists.okfn.org
>> http://lists.okfn.org/mailman/listinfo/ckan-dev
>
>
>
> --
> Co-Founder, Open Knowledge Foundation
> Promoting Open Knowledge in a Digital Age
> http://www.okfn.org/ - http://blog.okfn.org/
>
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> http://lists.okfn.org/mailman/listinfo/ckan-dev

-- 
-----
Jim Craner, 2012 Fellow
Code for America
jim at codeforamerica.org
(+1) 773-809-4546
http://codeforamerica.org