[annotator-dev] Automated Browser Testing

[Randall Leeds]

On Wed, Jul 2, 2014 at 1:27 PM, Bill Hunt <bill at krues8dr.com> wrote:

> Ok, checking in again on the testing front progress...
>
> To re-summarize the situation, here's the criteria what I'm trying to
> accomplish for this first phase:
>
> * Run the existing test suite on a cloud service in multiple browsers.
> * Have the failure data automatically reported back in a team-viewable (or
> ideally publicly-viewable) way.
> * Be able to automate this testing via CI of some sort.
> * Use existing tools as much as possible.
>
>
> On Jun 23, 2014, at 2:52 PM, Randall Leeds <tilgovi at hypothes.is> wrote:
>
> On Jun 23, 2014 11:29 AM, "Bill Hunt" <bill at krues8dr.com> wrote:
> >
> > I spent the morning working with BrowserStack, and it doesn't appear to
> be quite as mature as SauceLabs.
> >
> > You can definitely still run tests in a cloud-based browser as all of
> the other services offer, but like Browserling, the failures are not logged
> and reported in the dashboard.  Or, at least, it doesn't appear to do this
> by default and they don't really have very deep documentation on how to do
> this.   It still looks like those errors can probably be reported back to
> the script running them, but I haven't had time to dig in very deep on that
> and again, it doesn't look as easy as what SauceLabs already has in place.
>  If anyone on the list has other experience they can contribute, please
> chime in!
> >
>
> Did you try it with yeti? A brief look at that example seems like that's
> the thing for running mocha tests with browserstack
>
>
> On Jun 24, 2014, at 4:41 AM, Nick Stenning <nick at whiteink.com> wrote:
>
> If I had to express a preference on strategy I'd be inclined to dig into
> the issues with Browserling and get in touch with James Halliday to get
> any problems fixed. He's 'substack' in the relevant IRC channels and on
> Twitter.
>
>
>
> Again, I did not try either Browserling or BrowserStack any further once
> it appeared they both do not have the ability to report detailed failures.
>  Unless I'm missing something huge here, I don't really see the advantage
> of running tests in the cloud if there's not clear, publishable reporting
> results.  Having to manually copy and paste that information seems like a
> big waste of time, when Sauce does this by default - we have a *lot* of
> browsers to test here.
>
>
I don't know what you mean by detailed failures.
But it seems I can't even find the reference I remembered and maybe
BrowserStack isn't free for OSS.


>
>
> On Jun 23, 2014, at 2:52 PM, Randall Leeds <tilgovi at hypothes.is> wrote:
>
> I just found this:
> https://github.com/mantoni/mochify.js/blob/master/README.md
>
> I spent a few days trying to get this to work, but it appears that it's
> just a wrapper for browserify, mocaccino, and min-webdriver - but, it
> doesn't allow support for coffeescript from what I can see - there's simply
> no configuration options to drop in "-t coffeeify" like you'd do with
> browserify alone.  I've been chatting with the developer to add support for
> these things, but that's going very slowly.  The relevant ticket is here:
> https://github.com/mantoni/mochify.js/issues/17
>
> I've separately gotten the individual pieces to work ok with browserify
> and mocaccino, but I still don't have the piece to interact with webdriver
> to send the whole stack to saucelabs.
>
>
Sorry if that was a dead end.


>
> On Jun 23, 2014, at 2:52 PM, Randall Leeds <tilgovi at hypothes.is> wrote:
>
> It's rather the point of gulp to be more a set of simple tools to
> construct the process pipeline rather than a build system. One doesn't
> publish gulp plugins as much because there isn't as much boilerplate to
> introduce custom build steps as grunt.
>
> At least, that's my understanding of it.
>
> But I'm not sure any of this should be necessary.
>
> On Jun 24, 2014, at 4:41 AM, Nick Stenning <nick at whiteink.com> wrote:
>
> But as for having to rewrite the build to use Grunt to get this working,
> no thank you. I'm perfectly happy to see the build process change, but
> Grunt is a bad reimplementation of half of GNU Make, so I'd rather stick
> with the original.
>
>
>
> So far, the only tool that I've found that "works" 100%, end-to-end, that
> fulfills the criteria above is the grunt+saucelabs tool I mentioned a few
> messages back - everything else either just doesn't work, or doesn't do
> enough.
>
> At this point, whether we use Gulp or Make or plain Javascript or even a
> Bash script, we're in the same boat - there's no tool that's working and
> does all of this out of the box, and much more work will need to be done to
> proceed - or we wait until those issues are resolved upstream.   The only
> reason I brought Grunt up in the first place is that there *is* a tool that
> does everything we want here, with no issues, ready to go right now.
>
> Anyway, we've wasted about two weeks already on just this step, trying
> different options.  At this point, I'm inclined to go back to the option
> that I know works (Grunt + Sauce) to at least get the results of the tests,
> to make some forward progress on this, rather than diving into any more
> rabbitholes.
>
>
Let's go with Sauce then.
But I don't want to switch to Grunt.

Writing a little runner isn't hard. Here's an example:
https://github.com/pouchdb/pouchdb/blob/master/bin/test-browser.js
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/annotator-dev/attachments/20140702/08f61d30/attachment-0004.html>