Speeding up Cucumber and Concurrency tests using Threads

Reading on how to make Cucumber tests faster, most literature recommends using more CI servers, or more CI nodes, and that kind of solutions. I've always found this kind of solutions very expensive, not that machines are particularly expensive nowadays, but because more machines/nodes = more maintenance and, if you're on a particularly big organisation, are very annoying to acquire.

One thing that's much more efficient and should be done before getting more machines is simply using Threads.

However, you can't simply open a new thread on every step and rejoice as your tests take 1 second.

What you can and should do, is identifying small peaces of code that could be split in different threads. How do you know that?

The easy ones to spot are iterations where you don't care about the order in which each iteration is executed.

For example, I have this test against an API, which calls an endpoint and verify that every link returned is valid. This is a very simple and obvious case that follow on what I mentioned above, since:

• I have a list of things.
• I will iterate this list and do some kind of check on each item.
• I don't care if item "B" is checked before or after item "A".

So, to speed up a test like this, I can start a thread on start of each iteration, give it the checks I want it to do, saving the thread reference and go to the next item. After sending all items to their threads, wait for all threads to finish.

Luckily, we're using Ruby, where this kind of things are very easy to make, here's a gist code example:

Earlier I set off to try this technique, but, before, I ran a scenario on my IDE that would call the original code, these are the results:

1 scenario (1 passed) 

5 steps (5 passed) 

0m23.459s

After changing the code I got:

1 scenario (1 passed) 

5 steps (5 passed) 

0m10.408s

Wow! Over 50% improvement! This is too good maybe, so I hammered some code to force it fail - I was afraid for a moment that this setup was hiding test failures, but no:

1 scenario (1 failed) 

5 steps (1 failed, 4 passed) 

0m11.366s

Pretty happy with myself I pushed my changes to my git repo, and tried to run the Jenkins job which had a few tests with this step and others without. To my amazement, the previous 3 builds had took around 10-12 minutes, the new build took 2 minutes (yes, two minutes), that's a very big performance increase.
This technique can be used for other things as well, for example, you can have a step which tests race conditions, having two users on two different threads trying to do the same thing at the same time while you make sure they don't step on each other's toes unintentionally. I deally this race conditions tests is more of a Rspec/Unit test kind of thing that falls a little out of place of Cucumber, but knowledge of how to bend your tools never hurts.
Also, this will probably be fun when I join this technique with the two simultaneous browsers one.

Simultaneous multi user browser testing with Cucumber and Watir Webdriver

This post is not about parallel testing (with the intent to reduce tests execution time), nor about running tests in many different drivers. For those issues I recommend looking into Selenium Grid (you can start here) and Jenkins nodes.

I haven't seen many (any) implementation of tests where you control two browsers at the same time and my googling of the subject yielded no useful hints on the subject. I find this odd, with the ever increasing user interaction web apps have, how come there aren't more automators with needs for tests that require two users to talk to each other (for example) through two distinct browser windows?

Before we can even begin, there are few caveats, this won't work if you like the using the browser handing through "$browser" global variable on you test suite like many webdriver examples show you to. Personally I find that kind of usage is terrible to maintain, very inflexible and I'm biased against global variables - like Matz (Ruby's creator) said: "They are ugly, so don't use them".

For a long time I've been recommending people to have the browser handling done at the World level, I  setup this example a while ago, it is a module that you add to your World and it takes care of browser handling in a more pretty way. If you use the browser method it will start your browser as needed, so you can close and not worry about the state of a global var and where it got initiated...
This is pretty and good, unfortunately having the browser at World level like this means we only have one browser handler.

My favorite pattern for Cucumber based test frameworks however (and yes, I am aware it escapes a bit the few suggestions found in good literature) is making the World as an instance of a class made especially for handling the application at hand. This has it's advantages and drawbacks - and I obviously think the drawbacks are negligible facing the advantages.
How does this help?
You can have a list of users, each user being an instance of "YourApp::User" having this class include the Browser module, this way every user has it's own browser - using Firefox default configuration on Watir won't have the cookies "bleed" into one another.

However this does create another problem into itself, it probably screws up with most screenshot-on-error handling. I have made a gist with a proper solution (well, at least works for what I need).

How it works:

• Use the method browse to pass a block which gives you the browser handler.
• The method browser is still available, but not so cool.
• The browsing is wrapped in a way that on error a screenshot in base64 format will be saved to a file.
• Why base64? Embedding images on Cucumber HTML report needs to have the image accessible through an http request for as long as you want the report accessible.
• Ok, why a file then? Just a way to create an image *somewhere* and have it available on hooks, it also makes the image available case something goes terrible wrong and you hook block doesn't fully run.

Caveat: I haven't tried the multi-user thing with many browsers at once, for my needs this has only been done in FF, I'll probably give PhantomJS driver a try with this and see what happens, if you do try this with Chrome, IE, Safari or Opera drivers, please tell me how it went in the comments.
I am aware that this probably is probably a "Lisa Simpson" - answering a question no one asked.

Getting OSX, Jenkins, Cucumber and RVM to play together.

Recently I've changed my corporate laptop for a corporate Macbook Pro, good bye Windows based Ruby coding problems! Hello OSX problems! 

The hardest part was getting my local Jenkins working with my RVM installed Ruby and gems, my solution for those nasty "cucumber command not found" problems was a little ugly, but works wonders.

I started with echo $PATH on the console and copied all Ruby related parts, then I created a .jenkins_profile file with:

[[ -s "/Users/[MY_USER]/.rvm/scripts/rvm" ]] && source "/Users/[MY_USER]/.rvm/scripts/rvm" # Load RVM into a shell session *as a function*

export PATH=/Users/[MY_USER]/.rvm/gems/ruby-2.0.0-p247/bin:/Users/[MY_USER]/.rvm/gems/ruby-2.0.0-p247@global/bin:/Users/[MY_USER]/.rvm/rubies/ruby-2.0.0-p247/bin:/Users/[MY_USER]/.rvm/bin

export GEM_HOME=/Users/[MY_USER]/.rvm/gems/ruby-2.0.0-p247

export GEM_PATH=/Users/[MY_USER]/.rvm/gems/ruby-2.0.0-p247:/Users/[MY_USER]/.rvm/gems/ruby-2.0.0-p247@global

The first part is the rvm exports that I have on my own .bash_profile; then exporting a new $PATH with the Ruby info I copied earlier; ending with a forcing gem home and gem path for good measure.

Unfortunately for every job configuration I have to start with source /Users/[MY_USER]/.jenkins_profile, but it works fine and the local jenkins always has the gems I use.

Hope it helps more folks.