[annotator-dev] Automated Browser Testing

[Bill Hunt]

Thanks Randall - I was checking in on IRC to take the temperature before bringing it to the list for the larger discussion, but let's jump in!

So, the short version is, we started down the rabbit hole of mocha-sauce, but found that it doesn't actually report back failures properly, and the issue is deep in the webdriver dependency and probably too much work to resolve.  The next most popular option for Saucelabs integration is grunt-saucelabs which works perfectly, but requires grunt.  It's pretty coupled to grunt tasks too, so it'd require a bit of rewrite to get it to work without grunt.

Stepping back, Saucelabs was the second choice after Browserling did not behave as expected or provide any of the reporting tools that make Saucelabs great.  We could totally look into yet another option, but SL does most of what we need, and is free for opensource.  Choosing another option would also put us back to the start.

The runner issue is a separate one, since we are encountering the same hurdles with both of the previous services, namely that it requires running the service locally and SSH tunneling to your local machine to run the tests - requiring either a test server or no automation.  So that part will probably need to be redone regardless.

Grunt, of course, really does not offer anything over makefiles, except for the obvious benefit of "domestic" support - most everyone in the JS community does use it since it's been around so long (and won out on the yeoman war), except for the few at have transitioned to Gulp.   This means that there are lots of packages already to do the things we're trying to do in the testing space in particular, where there is no gulp alternative yet. Speaking for OpenGov, we certainly don't have the time to write all of that from scratch, but it would be a useful thing for the larger JS community.

Not that we really have to have a task runner, but imho, Annotator would benefit from using an existing tool instead of the one off scripts in the tools folder, at least in being familiar for new developers who are already JS veterans. But again, this is not mission critical.

Just my thoughts on the matter, of course - we are completely open to other suggestions.

Cheers,
-Bill

> On Jun 20, 2014, at 4:54 PM, Randall Leeds <tilgovi at hypothes.is> wrote:
> 
> <billhunt> nickstenn (afk, I know) & tilgovi :  Finally found a decent package to integrate the existing test runner with Saucelabs, but requires the use of grunt.  Or rewriting the whole package to not use grunt.  Is grunt a dealbreaker for dev?
> 
> I'm not a big fan of Grunt. I've played with it a few times and didn't find it offered anything significant over Makefiles. It's useful when you want to make some transformation pipeline out of JS modules, but if those modules expose a CLI like browserify, uglify and others do, then it's not really offering anything over GNU Make.
> 
> Gulp, on the other hand, seems to play to the strengths of node without reinventing the dependency tree.
> 
> What did you find?
> 
> Also, I'm not opposed to looking at other services than Saucelabs. For instance, I've heard good things about BrowserStack: http://www.browserstack.com/
> 
> I also wouldn't shy away from reworking our test runner. We don't need to keep doing it the way we're doing it if there's a better way. I'd rewrite that, if it's not too much work, before switching build systems just to use the existing test runner.
> _______________________________________________
> annotator-dev mailing list
> annotator-dev at lists.okfn.org
> https://lists.okfn.org/mailman/listinfo/annotator-dev
> Unsubscribe: https://lists.okfn.org/mailman/options/annotator-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/annotator-dev/attachments/20140620/7c4239b5/attachment-0004.html>