JUnit test reports not shown on Jenkins for Failed Jobs that never passed

*sigh*

This one costed me a whole morning of work.
I've been reorganizing my Jenkins test jobs so they're more friendly to outside teams, I've decided to be cautious and not just delete all the existing jobs, instead I've created a branch on my test repository, I made a lot of changes, then, I created new jobs on jenkins fetching from that branch, also made a new Dashboard that picks up those jobs. Ideally, the new dashboard should have about the same number of tests as the old one.
To my dismay, the new dashboard registered ~200 tests while the old one had over 1k.
All configurations were exactly the same (in regard to publishing results) from jobs to jobs, but some had the info right there on the dashboard, some didn't, worse, the ones that didn't, had the HTML report just fine.
I noticed that the jobs that didn't have the numbers were failing, while the others were passing/pending.
Digging through the internet yielded nothing particularly useful, I knew the problem laid somewhere between Jenkins and JUnit test reports.
I went through XML + XSD validators on the tests reports generated, but nothing was wrong.
After a coworker joined the struggle, he noticed that if we forced one of the failed builds to pass, all of the sudden  all the info would magically show up - including the ones from previously not shown builds.
I started asking folks I knew used a similar setup as I do and got this fantastic insight (paraphrasing):

If you're using Jenkins ver > 1.581 you're screwed. They've split the JUnit reporting into a plugin, it breaks like that after version 1.2, but Jenkins forces you to update it after version 1.581.

So I had two options, either I went through my 44 jobs and forced them to pass once and then went through them again to revert OR I'd downgrade Jenkins and JUnit plugin.
Once again I'm reminded of a sentence I hate "If it works, don't update it".
In case that version of Jenkins or the plugin aren't downloadable anymore, I've made a mirror on Dropbox.

Dynamic Class definition with inheritance in Ruby

If for some reason you need to have a class Dynamically defined in Ruby the following code will do you fine:

2.0.0-p247 :001 > dynamic_name = "ClassName"
 => "ClassName" 
2.0.0-p247 :002 > Object.const_set(dynamic_name, Class.new { def method1() 42 end })
 => ClassName 
2.0.0-p247 :003 > ClassName.new.method1
 => 42

However it might be useful to know that Class.new accepts another class as parameter for making it the class' parent, so if you want to make a dynamic class with inheritence you can do this:

2.0.0-p247 :001 >  dynamic_name = "ClassName"

 => "ClassName" 

2.0.0-p247 :002 > class A; def method1() 42 end; end

 => nil 

2.0.0-p247 :003 >Object.const_set(dynamic_name, Class.new(A)  )

 => ClassName 

2.0.0-p247 :004 > ClassName.new.method1 #=> 42

 => 42

This is slightly akin to dark magic, try not to use it much or at all... Now excuse me I need to summon some imps.

Measuring Response Time With HTTParty

If, while using HTTParty you ever wanted to know the duration of a request you probably resorted to saving the timestamp before the request and comparing it with after the request. I've got tired of this arcane way* and made a gem that adds a #duration to the response object.
Introducing: HTTParty-Timed, it's easy to use, just replace your require 'httparty' with require 'httparty/timed' and tadaaaah you're done! Now your response object has a method which returns the duration in milliseconds.



*Actually I just hid the arcane way deep inside NET::HTTP with a bit of monkey patching, so beware if you're not using "stock" NET::HTTP or HTTParty.
The good news however, is that this does not count with DNS resolve time, which in Ruby's implementation of NET::HTTP is always done (ignores OS's dns cache) on each request, which would add overhead time to your measurements

Github link

Void Value Expression in Ruby

Here's a commonplace scenario for almost anyone:

variable = nil  
if other_var == 1 variable = 2 
  elsif other_var == 2 variable = 3 
end

Simple right? Ruby however allows you to short this a bit  (with arguable readability):

variable = if other_var == 1
             variable = 2
           elsif other_var == 2 
             variable = 3
           end

This works because Ruby always returns something from evaluated blocks, even "if"'s. Now, what if you needed a return out of the function during that if?

variable = nil
if other_var == 1
 variable = 2
elsif other_var == 2
 variable = 3
else
  return "Let's get out this way, it's quicker!"
end

One would think he could do this:

 
variable = if other_var == 1
            variable = 2
           elsif other_var == 2 
            variable = 3
           else
            return "Let's get out this way, it's quicker!"
           end

2.0.0-p247 :037 > variable = if other_var == 1 
2.0.0-p247 :038?> variable = 2 
2.0.0-p247 :039?> elsif other_var == 2
 2.0.0-p247 :040?> variable = 3 
2.0.0-p247 :041?> else 
2.0.0-p247 :042 > return "Let's get out this way, it's quicker!" 
2.0.0-p247 :043?> end SyntaxError: (irb):43: void value expression

Nope! When I first saw this error it kinda reminded me of a similar error Perl would return when you forgot to add 1; to the end of your file, so I tried:

2.0.0-p247 :047 > variable = if other_var == 1 
2.0.0-p247 :048?> variable = 2
2.0.0-p247 :049?> elsif other_var == 2 
2.0.0-p247 :050?> variable = 3
 2.0.0-p247 :051?> else 
2.0.0-p247 :052 > return "Let's get out this way, it's quicker!"; 1; 
2.0.0-p247 :053 > end 
 => 2 
2.0.0-p247 :054 >

I think this might be a problem with the compiler.

Speeding up Cucumber and Concurrency tests using Threads

Reading on how to make Cucumber tests faster, most literature recommends using more CI servers, or more CI nodes, and that kind of solutions. I've always found this kind of solutions very expensive, not that machines are particularly expensive nowadays, but because more machines/nodes = more maintenance and, if you're on a particularly big organisation, are very annoying to acquire.

One thing that's much more efficient and should be done before getting more machines is simply using Threads.

However, you can't simply open a new thread on every step and rejoice as your tests take 1 second.

What you can and should do, is identifying small peaces of code that could be split in different threads. How do you know that?

The easy ones to spot are iterations where you don't care about the order in which each iteration is executed.

For example, I have this test against an API, which calls an endpoint and verify that every link returned is valid. This is a very simple and obvious case that follow on what I mentioned above, since:

• I have a list of things.
• I will iterate this list and do some kind of check on each item.
• I don't care if item "B" is checked before or after item "A".

So, to speed up a test like this, I can start a thread on start of each iteration, give it the checks I want it to do, saving the thread reference and go to the next item. After sending all items to their threads, wait for all threads to finish.

Luckily, we're using Ruby, where this kind of things are very easy to make, here's a gist code example:

Earlier I set off to try this technique, but, before, I ran a scenario on my IDE that would call the original code, these are the results:

1 scenario (1 passed) 

5 steps (5 passed) 

0m23.459s

After changing the code I got:

1 scenario (1 passed) 

5 steps (5 passed) 

0m10.408s

Wow! Over 50% improvement! This is too good maybe, so I hammered some code to force it fail - I was afraid for a moment that this setup was hiding test failures, but no:

1 scenario (1 failed) 

5 steps (1 failed, 4 passed) 

0m11.366s

Pretty happy with myself I pushed my changes to my git repo, and tried to run the Jenkins job which had a few tests with this step and others without. To my amazement, the previous 3 builds had took around 10-12 minutes, the new build took 2 minutes (yes, two minutes), that's a very big performance increase.
This technique can be used for other things as well, for example, you can have a step which tests race conditions, having two users on two different threads trying to do the same thing at the same time while you make sure they don't step on each other's toes unintentionally. I deally this race conditions tests is more of a Rspec/Unit test kind of thing that falls a little out of place of Cucumber, but knowledge of how to bend your tools never hurts.
Also, this will probably be fun when I join this technique with the two simultaneous browsers one.

Simultaneous multi user browser testing with Cucumber and Watir Webdriver

This post is not about parallel testing (with the intent to reduce tests execution time), nor about running tests in many different drivers. For those issues I recommend looking into Selenium Grid (you can start here) and Jenkins nodes.

I haven't seen many (any) implementation of tests where you control two browsers at the same time and my googling of the subject yielded no useful hints on the subject. I find this odd, with the ever increasing user interaction web apps have, how come there aren't more automators with needs for tests that require two users to talk to each other (for example) through two distinct browser windows?

Before we can even begin, there are few caveats, this won't work if you like the using the browser handing through "$browser" global variable on you test suite like many webdriver examples show you to. Personally I find that kind of usage is terrible to maintain, very inflexible and I'm biased against global variables - like Matz (Ruby's creator) said: "They are ugly, so don't use them".

For a long time I've been recommending people to have the browser handling done at the World level, I  setup this example a while ago, it is a module that you add to your World and it takes care of browser handling in a more pretty way. If you use the browser method it will start your browser as needed, so you can close and not worry about the state of a global var and where it got initiated...
This is pretty and good, unfortunately having the browser at World level like this means we only have one browser handler.

My favorite pattern for Cucumber based test frameworks however (and yes, I am aware it escapes a bit the few suggestions found in good literature) is making the World as an instance of a class made especially for handling the application at hand. This has it's advantages and drawbacks - and I obviously think the drawbacks are negligible facing the advantages.
How does this help?
You can have a list of users, each user being an instance of "YourApp::User" having this class include the Browser module, this way every user has it's own browser - using Firefox default configuration on Watir won't have the cookies "bleed" into one another.

However this does create another problem into itself, it probably screws up with most screenshot-on-error handling. I have made a gist with a proper solution (well, at least works for what I need).

How it works:

• Use the method browse to pass a block which gives you the browser handler.
• The method browser is still available, but not so cool.
• The browsing is wrapped in a way that on error a screenshot in base64 format will be saved to a file.
• Why base64? Embedding images on Cucumber HTML report needs to have the image accessible through an http request for as long as you want the report accessible.
• Ok, why a file then? Just a way to create an image *somewhere* and have it available on hooks, it also makes the image available case something goes terrible wrong and you hook block doesn't fully run.

Caveat: I haven't tried the multi-user thing with many browsers at once, for my needs this has only been done in FF, I'll probably give PhantomJS driver a try with this and see what happens, if you do try this with Chrome, IE, Safari or Opera drivers, please tell me how it went in the comments.
I am aware that this probably is probably a "Lisa Simpson" - answering a question no one asked.

Getting OSX, Jenkins, Cucumber and RVM to play together.

Recently I've changed my corporate laptop for a corporate Macbook Pro, good bye Windows based Ruby coding problems! Hello OSX problems! 

The hardest part was getting my local Jenkins working with my RVM installed Ruby and gems, my solution for those nasty "cucumber command not found" problems was a little ugly, but works wonders.

I started with echo $PATH on the console and copied all Ruby related parts, then I created a .jenkins_profile file with:

[[ -s "/Users/[MY_USER]/.rvm/scripts/rvm" ]] && source "/Users/[MY_USER]/.rvm/scripts/rvm" # Load RVM into a shell session *as a function*

export PATH=/Users/[MY_USER]/.rvm/gems/ruby-2.0.0-p247/bin:/Users/[MY_USER]/.rvm/gems/ruby-2.0.0-p247@global/bin:/Users/[MY_USER]/.rvm/rubies/ruby-2.0.0-p247/bin:/Users/[MY_USER]/.rvm/bin

export GEM_HOME=/Users/[MY_USER]/.rvm/gems/ruby-2.0.0-p247

export GEM_PATH=/Users/[MY_USER]/.rvm/gems/ruby-2.0.0-p247:/Users/[MY_USER]/.rvm/gems/ruby-2.0.0-p247@global

The first part is the rvm exports that I have on my own .bash_profile; then exporting a new $PATH with the Ruby info I copied earlier; ending with a forcing gem home and gem path for good measure.

Unfortunately for every job configuration I have to start with source /Users/[MY_USER]/.jenkins_profile, but it works fine and the local jenkins always has the gems I use.

Hope it helps more folks.