[annotator-dev] Automated Browser Testing

Wed Jul 2 20:27:18 UTC 2014

Ok, checking in again on the testing front progress...

To re-summarize the situation, here's the criteria what I'm trying to accomplish for this first phase:

* Run the existing test suite on a cloud service in multiple browsers.
* Have the failure data automatically reported back in a team-viewable (or ideally publicly-viewable) way.
* Be able to automate this testing via CI of some sort.
* Use existing tools as much as possible.

On Jun 23, 2014, at 2:52 PM, Randall Leeds <tilgovi at hypothes.is> wrote:
> On Jun 23, 2014 11:29 AM, "Bill Hunt" <bill at krues8dr.com> wrote:
> >
> > I spent the morning working with BrowserStack, and it doesn't appear to be quite as mature as SauceLabs.
> >
> > You can definitely still run tests in a cloud-based browser as all of the other services offer, but like Browserling, the failures are not logged and reported in the dashboard.  Or, at least, it doesn't appear to do this by default and they don't really have very deep documentation on how to do this.   It still looks like those errors can probably be reported back to the script running them, but I haven't had time to dig in very deep on that and again, it doesn't look as easy as what SauceLabs already has in place.  If anyone on the list has other experience they can contribute, please chime in!
> >
> 
> Did you try it with yeti? A brief look at that example seems like that's the thing for running mocha tests with browserstack
> 

On Jun 24, 2014, at 4:41 AM, Nick Stenning <nick at whiteink.com> wrote:

> If I had to express a preference on strategy I'd be inclined to dig into
> the issues with Browserling and get in touch with James Halliday to get
> any problems fixed. He's 'substack' in the relevant IRC channels and on
> Twitter.

Again, I did not try either Browserling or BrowserStack any further once it appeared they both do not have the ability to report detailed failures.  Unless I'm missing something huge here, I don't really see the advantage of running tests in the cloud if there's not clear, publishable reporting results.  Having to manually copy and paste that information seems like a big waste of time, when Sauce does this by default - we have a *lot* of browsers to test here.

On Jun 23, 2014, at 2:52 PM, Randall Leeds <tilgovi at hypothes.is> wrote:
> I just found this: https://github.com/mantoni/mochify.js/blob/master/README.md
> 
I spent a few days trying to get this to work, but it appears that it's just a wrapper for browserify, mocaccino, and min-webdriver - but, it doesn't allow support for coffeescript from what I can see - there's simply no configuration options to drop in "-t coffeeify" like you'd do with browserify alone.  I've been chatting with the developer to add support for these things, but that's going very slowly.  The relevant ticket is here:  https://github.com/mantoni/mochify.js/issues/17

I've separately gotten the individual pieces to work ok with browserify and mocaccino, but I still don't have the piece to interact with webdriver to send the whole stack to saucelabs.   

On Jun 23, 2014, at 2:52 PM, Randall Leeds <tilgovi at hypothes.is> wrote:
> It's rather the point of gulp to be more a set of simple tools to construct the process pipeline rather than a build system. One doesn't publish gulp plugins as much because there isn't as much boilerplate to introduce custom build steps as grunt.
> 
> At least, that's my understanding of it.
> 
> But I'm not sure any of this should be necessary.
> 
On Jun 24, 2014, at 4:41 AM, Nick Stenning <nick at whiteink.com> wrote:

> But as for having to rewrite the build to use Grunt to get this working,
> no thank you. I'm perfectly happy to see the build process change, but
> Grunt is a bad reimplementation of half of GNU Make, so I'd rather stick
> with the original.

So far, the only tool that I've found that "works" 100%, end-to-end, that fulfills the criteria above is the grunt+saucelabs tool I mentioned a few messages back - everything else either just doesn't work, or doesn't do enough.  

At this point, whether we use Gulp or Make or plain Javascript or even a Bash script, we're in the same boat - there's no tool that's working and does all of this out of the box, and much more work will need to be done to proceed - or we wait until those issues are resolved upstream.   The only reason I brought Grunt up in the first place is that there *is* a tool that does everything we want here, with no issues, ready to go right now.

Anyway, we've wasted about two weeks already on just this step, trying different options.  At this point, I'm inclined to go back to the option that I know works (Grunt + Sauce) to at least get the results of the tests, to make some forward progress on this, rather than diving into any more rabbitholes.

Cheers,
-Bill

Bill Hunt
Senior Developer
OpenGov Foundation
http://opengovfoundation.org/

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/annotator-dev/attachments/20140702/e76a0283/attachment-0003.html>