[annotator-dev] Automated Browser Testing

Thu Jul 3 13:52:41 UTC 2014

On Jul 2, 2014, at 8:40 PM, Randall Leeds <tilgovi at hypothes.is> wrote:

> 
> Again, I did not try either Browserling or BrowserStack any further once it appeared they both do not have the ability to report detailed failures.  Unless I'm missing something huge here, I don't really see the advantage of running tests in the cloud if there's not clear, publishable reporting results.  Having to manually copy and paste that information seems like a big waste of time, when Sauce does this by default - we have a *lot* of browsers to test here.
> 
> 
> I don't know what you mean by detailed failures.
> But it seems I can't even find the reference I remembered and maybe BrowserStack isn't free for OSS.

By "detailed failures" I mean that SauceLabs keeps track of the errors that (in our case) Mocha reports back, so you can easily browse the results of the test on the web.  The others do not do this, they just fail silently.  I asked the Browserling guys about it months ago and they said it wasn't something they were going to implement.  

You can still get the output from the runner on your local machine and copy and paste it somewhere, but that would require lots of extra work to automate into something usable - you'd need a place to store the results and some mechanism for updating them and lots of other hurdles would need to be leaped. 

> So far, the only tool that I've found that "works" 100%, end-to-end, that fulfills the criteria above is the grunt+saucelabs tool I mentioned a few messages back - everything else either just doesn't work, or doesn't do enough.  
> 
> At this point, whether we use Gulp or Make or plain Javascript or even a Bash script, we're in the same boat - there's no tool that's working and does all of this out of the box, and much more work will need to be done to proceed - or we wait until those issues are resolved upstream.   The only reason I brought Grunt up in the first place is that there *is* a tool that does everything we want here, with no issues, ready to go right now.
> 
> Anyway, we've wasted about two weeks already on just this step, trying different options.  At this point, I'm inclined to go back to the option that I know works (Grunt + Sauce) to at least get the results of the tests, to make some forward progress on this, rather than diving into any more rabbitholes.
> 
> 
> Let's go with Sauce then.
> But I don't want to switch to Grunt.
> 
> Writing a little runner isn't hard. Here's an example:
> https://github.com/pouchdb/pouchdb/blob/master/bin/test-browser.js

Thanks - I'll dive into that one today.

Cheers,
-Bill

Bill Hunt
Senior Developer
OpenGov Foundation
http://opengovfoundation.org/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/annotator-dev/attachments/20140703/ddfd5881/attachment-0004.html>