When testing web apps for a general/wide-ranging audience, one must test across a slew of different browsers and operating systems (and different versions of each). This is a real pain, even if you have an army of testers at hand (which most of us don't). I employ various tools to help me with this (CrossBrowserTesting.com, BrowserCam, VM Ware with different VM setups, etc.). But, the reality is that I don't feel like these really stack up to what I need. More specifically, BrowserCam, and its competitors do not. CrossBrowserTesting is actually pretty awesome and does what it advertises quite nicely, it's a favorite tool right now.
What I'm after is a way to see what a survey of pages on my site look like on different browsers and OS's. This is by no means a definitive test of a site, but more of a quick visual inspection of appearance, without having to fire up a dozen different pages across maybe 30 different browsers/OS configurations. We use an automated test suite to test the bulk of other things, but appearance can't be tested that way.
BrowserCam helps, but honestly I find it quite lacking as a tool in this arena. Note, I'm picking on BrowserCam, as that's the one I use, but the others (BrowserShots, Litmus, etc.) all seem to suffer some or all of these problems, and further, they don't seem to have projects or ways to establish a standard suite of tests. Getting on with it, all of this really boils down to two primary things:
- Ability to edit the settings on a project's images. As far as I can tell, once you select the URL's, image size, browsers, etc. for a project, there is no way to change any of that! What if I just want to change the window size? Can't, I have to create a new project (or add new images to the existing one, replicating all my previous work. This seems like an obvious hole.
- Automation. I want to be able to automate regeneration of the project. I'd like something like a simple URL/HTTP API to do this, so that I can, from a command line, use curl or similar to issue a single HTTP request that will regenerate a project's images. Thus, the API would need to use HTTP authentication or similar, and specify which project to regenerate. With this, I would be able to automate requesting a regeneration of the project as part of/after deploying my application.
I'm currently looking into creating my own script to drive the automation, but it's looking non-trivial due to the way BrowserCam's pages work: everything winds up on the same URL, actions/links are all JavaScript that post forms, with a lot of params and a bunch of params that will take some time to ascertain if a) they're required, and b) all the info in the parameter is needed, etc. If someone's already done this, let me know! Also, suggestions for tools to drive the automation? I've used the Ruby Mechanize library in the past for things like this, and this may work if I can determine the params, etc., and if Mechanize can even drive the JavaScript links (not sure it can - anyone?). I can't use something like Selenium or Windmill because this needs to work from a script/command line, and not rely on a browser.
Finally, if anyone knows of a service similar to BrowserCam that solves these, even if it supports a few less browsers, do let me know. At very minimum it would need to handle Internet Explorer and Firefox, preferably also Safari, and cover Windows XP, Vista, MacOS X 10.5, and one popular Linux flavor. Nice to haves would include Opera and various others.
DealBase.com
8 comments:
hey,
you can use selenium from the commandline... check out selenium-rc. You can also run selenium on a continuous build server by running firefox and/or webkit in a headless mode (actually, you do this by using a virtual x-window server).
selenium-rc is not as smooth to use as I would like; it is changing quite a lot, but it is definitely getting there!
btw I'm based in portland, so a shout out to a fellow oregonian!
Adam, are you referring to using Selenium as a way to drive telling BrowserCam to do the regenerate? With headless Firefox, that might actually work, forgot about headless FFox. Using Selenium for this seems like a lot of overhead, but it may be the way.
However, as it turns out, I just found out that CrossBrowserTesting.com is going to be doing a screenshot service. And, they read my blog entry and hopefully will have a solution here.
I'll revisit Selenium and see if I can rig this up relatively quickly, thanks for the tip.
You may want to take a look at this new service browser screenshot service
Currently it only supports Firefox, Internet Explorer and Safari on Windows, but as it is being actively developed more features and additional browsers will be added in the future.
As it is not a web service but an application it can be easily automated, although this functionality is not implemented yet. If you find it useful you can suggest a new future on browserseal.com forums - if enough people want it, there are high chances that it will be implemented in the next version
Free beta version can be downloaded from browserseal.com
Demiurg, thanks for the comment and tip on BrowserSeal. Unfortunately that service doesn't satisfy my requirements. As stated on the home page, "BrowserSeal installation does not include the supported browsers and will only work with ones that are already installed on your computer." The #1 thing I'm after is the ability to test a bunch of different browsers, many/most of which I don't have installed, or can't have installed because I'm not on that OS. Or, for things like different versions of IE you can't have multiple versions easily installed together, etc.
I may be missing something, but BrowserSeal honestly looks of little use, in that about the only thing I can see it being used for is taking a screenshot of the full page of a browser where you don't have to scroll (which I don't need). I don't mean to be harsh, but the cross browser testing space is pretty competitive and even my old employer Adobe recently entered the space and I think has really come up short in comparison to existing services (which I always find interesting - if you're going to go to the effort to do it, why come up so short - what is my incentive to switch?).
Chris,
First, please note that BrowserSeal is in very early beta and more features and new browsers will be added soon. The requirement to have all the browsers installed will remain, however, since this is one of the differentiating features of BrowserSeal - working with local browsers will always be much faster than any web service one can think of.
As for your comments, I'm afraid that your basic assumptions are not entirely correct, at least if you work with Windows. You can install all major browsers on that OS. It's not such a big effort and the benefits are clear, if there is a rendering error with one of them you can launch this browser and check in more depth what went wrong.
Regarding multiple IE versions, have you checked IE collection ?
Demiurg, here's the thing, my use for this is to do "cross browser" testing. That spans multiple OS's and multiple browsers. I can't do that on a single OS. I can't install Mac browsers on Windows and vice versa, or Linux browsers, etc. Yes, I do use VMWare Fusion to do local testing, and that suffices when I need that. So, it still leaves me wondering what I'd use BrowserSeal for in that case. I can cover a lot of different configs (browsers, OS's, versions, etc.) using crossbrowsertesting.com for example, and if I need to really delve in, or do local testing, then I just use my installed browsers, or VMWare, so I don't see what BrowserSeal does for me in terms of my particular testing needs. Maybe you can describe that more?
As for multiple IE's, I've had quite bad luck with that in the past. There are solutions, and I've tried a variety of them, but usually it doesn't seem to work that well. I'm finding that different VM images for VMWare fusion is a safer and more accurate representation.
Chris, are you sure you really need different OS ? According to the statistics browsers you can install on Windows cover 99% of the browser market share (and you can install Mac OS browser, i.e. Safari, on Windows)! Are you sure that 99% is not enough for you ?
Multiple IE had it's issues, but IE collection is a different project that was reported to work much better.
As for BrowserSeal, once you have all these browsers installed it will help you to easily take screenshots and visually compare them.
Post a Comment