This one costed me a whole morning of work.
I've been reorganizing my Jenkins test jobs so they're more friendly to outside teams, I've decided to be cautious and not just delete all the existing jobs, instead I've created a branch on my test repository, I made a lot of changes, then, I created new jobs on jenkins fetching from that branch, also made a new Dashboard that picks up those jobs. Ideally, the new dashboard should have about the same number of tests as the old one.
To my dismay, the new dashboard registered ~200 tests while the old one had over 1k.
All configurations were exactly the same (in regard to publishing results) from jobs to jobs, but some had the info right there on the dashboard, some didn't, worse, the ones that didn't, had the HTML report just fine.
I noticed that the jobs that didn't have the numbers were failing, while the others were passing/pending.
Digging through the internet yielded nothing particularly useful, I knew the problem laid somewhere between Jenkins and JUnit test reports.
I went through XML + XSD validators on the tests reports generated, but nothing was wrong.
After a coworker joined the struggle, he noticed that if we forced one of the failed builds to pass, all of the sudden all the info would magically show up - including the ones from previously not shown builds.
I started asking folks I knew used a similar setup as I do and got this fantastic insight (paraphrasing):
If you're using Jenkins ver > 1.581 you're screwed. They've split the JUnit reporting into a plugin, it breaks like that after version 1.2, but Jenkins forces you to update it after version 1.581.So I had two options, either I went through my 44 jobs and forced them to pass once and then went through them again to revert OR I'd downgrade Jenkins and JUnit plugin.
Once again I'm reminded of a sentence I hate "If it works, don't update it".
In case that version of Jenkins or the plugin aren't downloadable anymore, I've made a mirror on Dropbox.
