Showing posts with label Ruby. Show all posts
Showing posts with label Ruby. Show all posts

Wednesday, 20 December 2017

ObjectifyHash Gem - My favorite over engineered solution for a stupid problem

Years after the initial development, I was finally allowed to bring this little gem to the public as part of my employer's effort to bring forth more open source code - hopefully more of these will come out soon.
Objectifyhash a very over-engineered solution for a stupid problem: dealing with nested objects on JSON responses from REST services. I find this specially painful when you need to navigate these representations, search for things in them and be able to do actions, one always ends up with code that looks like 

def find_thing_on_x things
  things.detect do |t| t['identification']['name'] == 'thing'; end
end
....
res = get 'https://my-service.com/stuff'

my_thing = find_thing_on_x res['data']['stuff']['things']

Not only is this ugly to look at, but very functional looking feels very un-Ruby like, not mentioning that on non-english keyboards like mine, brackets are hidden behind an awkward Alt Gr + numbers combo.
So, what I wanted to do in this case would be that either get the Stuff JSON mapped to a Stuff class on my Namespace so I can access handy Stuff related methods, or, if that doesn't exists, to get a generic object that allows me to handle Stuff more easily  AND (this is the fun part that other gems at the time didn't cover) repeat recursively for all nested objects inside.
That means, on the example above, it would create a Data class, a Data::Stuff and da Data::Stuff::Thing - since "things" is an array, it leaves it as an array, but each member becomes a Thing! 

The above example becomes much more beautiful with:

stuff = get('https://my-service.com/stuff').to_obj
my_thing = stuff.find_thing
OR
my_thing = stuff.data.stuff.find_things

I usually go with the second example here, as finding things is an action of Stuff and not from Data, but whatever.

So, all JSON objects become ruby objects of the class named after their key within the namespace of the previous key - easy, right?
It also has an exception handling, where you can specify several keys to map to specific existing classes on your code, this is how you inject your own methods such as the "find_things" above.

But this isn't even the part which makes it over-engineered, I opted to dynamically assigning methods to the instances with the values they should return, this means there's no setter methods and has brought some pains into the our work, the funnest of them all are when we have conflicts in Namespace or inherited methods from basic object, the supreme ofender: {"file": {"id": 123}} in which creating a File with an id method didn't work as aspected, so there's a few gotchas here , in this situation expect File_ and _id_ to happen, but overall I've enjoyed working with this tool.

Checkout the github page with more details and I hope there's more equally crazy people like me to enjoy this gem.

Wednesday, 29 June 2016

Watir Webdriver / Selenium HTTP Authentication - if it's stupid and it works, it ain't stupid

I've taken some time to write a small sanity test pack to run on the web app that uses the platform my team develops.
Although we have a very good and thorough automated test coverage, I'm very paranoid and want to make sure we didn't miss anything that breaks basic functionality, not that it happens often, or in recent memory, I'm just paranoid - it's a good quality for QA person, right?
So I went about the correct way of doing things, Page Object Model; I have a basic page object that has functionalities that are relevant through the whole app, like knowing when page is fully loaded, how to login, etc. Everything else inherits from this class.
First problem, http authentication, selenium and watir still (!!!) don't have a proper way of dealing with this well, so I went with the normal "https://username:password@url." method, which by the way, nobody else tells you this: use URI encoding on the username and password.
But then I got to the playback page on our app (which is kinda an important page if you're online movie retailer), for reasons not important for this post, on this page there's a https -> http redirect, this redirects cause our username:password deal to be lost and I got stuck with the authentication popup.
Googling how to get around this today still gives me about the same answers as a few years ago, which include this lovely (yet old) Watirmelon post that suggest using AutoAuth plugin for Firefox, usually I'd be against this, since it binds my tests to Firefox, but since for my particular sanity-for-other-team kind of work, I don't really mind, however AutoAuth no longer works with recent versions of Firefox, it stopped working on v39, currently we're on v47.
After much dive into documentation and stackoverflow questions, I gave up, I was ready to ask them to turn off HTTP Authentication for that page, until a co-worker casually mentioned:
Try opening the http version with username and password on the url first, at the start of your test, then go to the normal https start page and do the normal test, the browser will remember when you get redirected.
That a stupid solution... but it works, so it ain't stupid.
Here's how my code changed:

      [...]
      browser.goto inject_auth( url ) if url
      wait_for_load
      [...]

to:

    if url
      browser.goto inject_auth(url)
      wait_for_load
      browser.goto inject_auth(url).sub( 'https', 'http')+ 'playback/fcf7ce62-5695-4d80-be85-662798c9ac7c?token=undefined'
      wait_for_load
      browser.goto inject_auth(url)
    end

Hacky, I know, I have to load the normal page first, because I set up the http load purposely to hit a 404 so this wouldn't cause any unintentional side effect, but it also gives me another redirect, only after this little dance I get to go to the initial page I intended.
Hope this helps other people trying to automate apps with redirects.

The Watirmelon post I mentioned before resolves the problem with NTLM  auth, which could be applied to everything, however my solution doesn't solve that particular problem, if you find out how, please let me know.

Friday, 30 January 2015

Ruby: Custom Errors for Lazy Bums with const_missing

Catching errors is quite a normal thing when writing code, every programing language has some variation of the throw/catch, in Ruby's case, it's the elegant begin, rescue and ensure blocks.
At some point in every project you stop catching "other" people's errors and start catching your own.
So you start creating you own error classes, there's a couple common approaches I've seen over the years.

  • Create an Errors namespace on each class/module you need and create errors for that there.
  • Create a generic Project::Errors with its own file bag and throw all errors in there.
I usually go for the second approach, so error classes don't distract me, but that's more of a personal preference anyway.
However, I am a lazy bum and I've got tired of specified my error classes, so I made this Gist as an example:


The main idea behind this, is letting you freely type your errors as you need them both where they're being raised and rescued and you don't need to worry about they existing since they'll be created in runtime for you.
This method is probably not very efficient, but I don't care for this case.

Wednesday, 8 October 2014

Dynamic Class definition with inheritance in Ruby

If for some reason you need to have a class Dynamically defined in Ruby the following code will do you fine:

2.0.0-p247 :001 > dynamic_name = "ClassName"
 => "ClassName" 
2.0.0-p247 :002 > Object.const_set(dynamic_name, Class.new { def method1() 42 end })
 => ClassName 
2.0.0-p247 :003 > ClassName.new.method1
 => 42

However it might be useful to know that Class.new accepts another class as parameter for making it the class' parent, so if you want to make a dynamic class with inheritence you can do this:

2.0.0-p247 :001 >  dynamic_name = "ClassName"
 => "ClassName" 
2.0.0-p247 :002 > class A; def method1() 42 end; end
 => nil 
2.0.0-p247 :003 >Object.const_set(dynamic_name, Class.new(A)  )
 => ClassName 
2.0.0-p247 :004 > ClassName.new.method1 #=> 42
 => 42


This is slightly akin to dark magic, try not to use it much or at all... Now excuse me I need to summon some imps.

Monday, 9 June 2014

Measuring Response Time With HTTParty

If, while using HTTParty you ever wanted to know the duration of a request you probably resorted to saving the timestamp before the request and comparing it with after the request. I've got tired of this arcane way* and made a gem that adds a #duration to the response object.
Introducing: HTTParty-Timed, it's easy to use, just replace your require 'httparty' with require 'httparty/timed' and tadaaaah you're done! Now your response object has a method which returns the duration in milliseconds.



*Actually I just hid the arcane way deep inside NET::HTTP with a bit of monkey patching, so beware if you're not using "stock" NET::HTTP or HTTParty.
The good news however, is that this does not count with DNS resolve time, which in Ruby's implementation of NET::HTTP is always done (ignores OS's dns cache) on each request, which would add overhead time to your measurements

Github link

Monday, 24 March 2014

Void Value Expression in Ruby

Here's a commonplace scenario for almost anyone:

variable = nil  
if other_var == 1 variable = 2 
  elsif other_var == 2 variable = 3 
end
Simple right? Ruby however allows you to short this a bit  (with arguable readability):
variable = if other_var == 1
             variable = 2
           elsif other_var == 2 
             variable = 3
           end
This works because Ruby always returns something from evaluated blocks, even "if"'s. Now, what if you needed a return out of the function during that if?
variable = nil
if other_var == 1
 variable = 2
elsif other_var == 2
 variable = 3
else
  return "Let's get out this way, it's quicker!"
end
One would think he could do this:
 
variable = if other_var == 1
            variable = 2
           elsif other_var == 2 
            variable = 3
           else
            return "Let's get out this way, it's quicker!"
           end
2.0.0-p247 :037 > variable = if other_var == 1 
2.0.0-p247 :038?> variable = 2 
2.0.0-p247 :039?> elsif other_var == 2
 2.0.0-p247 :040?> variable = 3 
2.0.0-p247 :041?> else 
2.0.0-p247 :042 > return "Let's get out this way, it's quicker!" 
2.0.0-p247 :043?> end SyntaxError: (irb):43: void value expression
Nope! When I first saw this error it kinda reminded me of a similar error Perl would return when you forgot to add 1; to the end of your file, so I tried:
2.0.0-p247 :047 > variable = if other_var == 1 
2.0.0-p247 :048?> variable = 2
2.0.0-p247 :049?> elsif other_var == 2 
2.0.0-p247 :050?> variable = 3
 2.0.0-p247 :051?> else 
2.0.0-p247 :052 > return "Let's get out this way, it's quicker!"; 1; 
2.0.0-p247 :053 > end 
 => 2 
2.0.0-p247 :054 >
I think this might be a problem with the compiler.

Thursday, 30 January 2014

Speeding up Cucumber and Concurrency tests using Threads

Reading on how to make Cucumber tests faster, most literature recommends using more CI servers, or more CI nodes, and that kind of solutions. I've always found this kind of solutions very expensive, not that machines are particularly expensive nowadays, but because more machines/nodes = more maintenance and, if you're on a particularly big organisation, are very annoying to acquire.
One thing that's much more efficient and should be done before getting more machines is simply using Threads.
However, you can't simply open a new thread on every step and rejoice as your tests take 1 second.
What you can and should do, is identifying small peaces of code that could be split in different threads. How do you know that?
The easy ones to spot are iterations where you don't care about the order in which each iteration is executed.
For example, I have this test against an API, which calls an endpoint and verify that every link returned is valid. This is a very simple and obvious case that follow on what I mentioned above, since:
  • I have a list of things.
  • I will iterate this list and do some kind of check on each item.
  • I don't care if item "B" is checked before or after item "A".
So, to speed up a test like this, I can start a thread on start of each iteration, give it the checks I want it to do, saving the thread reference and go to the next item. After sending all items to their threads, wait for all threads to finish.
Luckily, we're using Ruby, where this kind of things are very easy to make, here's a gist code example:


Earlier I set off to try this technique, but, before, I ran a scenario on my IDE that would call the original code, these are the results:
1 scenario (1 passed) 
5 steps (5 passed) 
0m23.459s
After changing the code I got:

1 scenario (1 passed) 
5 steps (5 passed) 
0m10.408s
Wow! Over 50% improvement! This is too good maybe, so I hammered some code to force it fail - I was afraid for a moment that this setup was hiding test failures, but no:

1 scenario (1 failed) 
5 steps (1 failed, 4 passed) 
0m11.366s
Pretty happy with myself I pushed my changes to my git repo, and tried to run the Jenkins job which had a few tests with this step and others without. To my amazement, the previous 3 builds had took around 10-12 minutes, the new build took 2 minutes (yes, two minutes), that's a very big performance increase.
This technique can be used for other things as well, for example, you can have a step which tests race conditions, having two users on two different threads trying to do the same thing at the same time while you make sure they don't step on each other's toes unintentionally. I deally this race conditions tests is more of a Rspec/Unit test kind of thing that falls a little out of place of Cucumber, but knowledge of how to bend your tools never hurts.
Also, this will probably be fun when I join this technique with the two simultaneous browsers one.

Wednesday, 8 January 2014

Getting OSX, Jenkins, Cucumber and RVM to play together.

Recently I've changed my corporate laptop for a corporate Macbook Pro, good bye Windows based Ruby coding problems! Hello OSX problems! 

The hardest part was getting my local Jenkins working with my RVM installed Ruby and gems, my solution for those nasty "cucumber command not found" problems was a little ugly, but works wonders.
I started with echo $PATH on the console and copied all Ruby related parts, then I created a .jenkins_profile file with:

[[ -s "/Users/[MY_USER]/.rvm/scripts/rvm" ]] && source "/Users/[MY_USER]/.rvm/scripts/rvm" # Load RVM into a shell session *as a function*
export PATH=/Users/[MY_USER]/.rvm/gems/ruby-2.0.0-p247/bin:/Users/[MY_USER]/.rvm/gems/ruby-2.0.0-p247@global/bin:/Users/[MY_USER]/.rvm/rubies/ruby-2.0.0-p247/bin:/Users/[MY_USER]/.rvm/bin
export GEM_HOME=/Users/[MY_USER]/.rvm/gems/ruby-2.0.0-p247
export GEM_PATH=/Users/[MY_USER]/.rvm/gems/ruby-2.0.0-p247:/Users/[MY_USER]/.rvm/gems/ruby-2.0.0-p247@global
The first part is the rvm exports that I have on my own .bash_profile; then exporting a new $PATH with the Ruby info I copied earlier; ending with a forcing gem home and gem path for good measure.
Unfortunately for every job configuration I have to start with source /Users/[MY_USER]/.jenkins_profile, but it works fine and the local jenkins always has the gems I use.

Hope it helps more folks.

Tuesday, 15 October 2013

HTTParty with cookies

On the past few months I've been doing automated testing work on an rest API that happens to rely a lot on cookies for user authentication stuff.
HTTParty is a great gem for working with rest API's, it's easy and simple, but it doesn't provide a care free way of dealing with cookies. I mean, you have a way to read the cookies from the response, you have a way to add them to the request, but you'll have to do it manually. So like any good Rubist I've made the code for it into a little gem. 
I hope it helps someone else as well. 
You can find most of the information on github.

Thursday, 11 July 2013

HTTParty and non 200 response codes, cache & etags and a few other quirks

I've been playing with HTTParty to help me automate tests for an API, while its is a great gem, I found some documentation and guides lacking.
What happens when you add to your request an ETAG (for cache) and the server returns an 304 (not modified)? I'll tell you what happens, you'll get a "nil"!
No really, you get a nil!
Now, this confused the shit out me, and I've googled on and on about it, but found nothing really useful, until, for some reason I found deep into the specification my answer:

    describe 'with non-200 responses' do      context "3xx responses" do        it 'returns a valid object for 304 not modified' do          stub_response '', 304          resp = @request.perform          resp.code.should == 304          resp.body.should == ''          resp.should be_nil        end
"should be nil?" Well, that make no sense! It turns out that your nil object has some methods injected into it, because... I don't know, someone thought it was intuitive and easy - I .... disagree...

So, how we deal with this?

Something like:
res = HTTParty.get 'someurl', headers:{'If-none-match': 'some e-tag you got from earlier'}
res.code #=> 304
res.class #=> Nilclass





This is also valid for other responses with no body that HTTParty thinks is fun to turn into nils, because if you don't have the body for it, you can't party.

Wednesday, 8 May 2013

Pi Facer gem version 0.0.1 alpha released

I've been playing with a Raspberry Pi and I have this project that might use the GPIO pins, I've poked around to see if any Ruby gems allowed me to use it, I've found this good one that has an event driven way of doing it.
However later I decided to use a Piface instead of the pins directly, unfortunatly there were no event driven gems for piface, which is a shame, because I figure it's the best way to interact with it. So I made one.
I've based of piper because all the threading stuff was already done, hope it helps more people with piface projects.
Oh yes, Ruby 2.0.

pi_facer on github, pi_facer on rubygem.

Friday, 12 April 2013

Using Watir-webdriver and selenium remote control to automate tests on Internet Explorer

So, you're on a Linux machine (like a boss), you have a pretty sweet battery of automated tests that you run against your web application on your CI server (which also runs Linux), but you have a few problems:

  1. Internet Explorer sucks and most defects are open due to it.
  2. You can't automate IE because all your systems are on Linux.
  3. You need to automate on IE, because 13% of your user base are either ignorant or corporate drones who can't change their browsers.
So what you do? You can't just quit and call it a day? Nope, you keep reading.

What you need?
  1. A windows machine (yeah, sorry about that, but you knew it was coming).
  2. Install on it selenium remote grid server on said machine, here's a guide how to set it up.
  3. Our good friend Ruby!
  4. selenium, selenium-webdriver and watir-webdriver gems.
That's great, but you're wondering "How do I make watir-webdriver connect to the remote selenium hub?", that's a great question, for that all you need it's to pass it the arguments you'd use for selenium-webdriver to do that. And to be honest I only know that because I've been digging into watir-webdriver's code on github and I know that all parameters you pass it are passed along to selenium, something that's not quite clear on watir's documentation or any guide you can find on the web.
What I needed was something that would have watir-webdriver connect to Firefox locally, but if I wanted IE it would connect remotely, so, thanks to the param ":desired_capabilities" you can specify which browser you want to open on your remote connection.

I've created a sample for this on github, which you can find here.

Tuesday, 26 February 2013

Ruby ARGF and the "<" operator

One of those things I've always should have known but somehow skipped when first read about ARGV and ARGF on Ruby.

ARGF will give you file handlers and content of all files passed to the script, but it assumes that all arguments present in ARGV are in fact files.
So if you want to be able to call your script like so:

./myrubyscript 100 file.txt

Than in the code you must have:


somvar     = ARGV.shift.to_i
filelines  = ARGF.readlines


How about, if for some reason, you want to call your script like so:

./myrubyscript 100 < file.txt



somvar     = ARGV.shift.to_i
filelines  = ARGF.readlines



It's the same code! Don't you love Ruby?

The workings:
The arguments passed to your script are stored in the ARGV Array, one argument per element. ARGF assumes that any arguments that aren't file names have been removed from ARGV.
It took me more than it should have to find this little page.

Wednesday, 16 January 2013

TIL About Ruby class and modules namespacing

Today's post is courtesy of Captain Obvious, but still I wasn't sure and just HAD TO experiment.
It's simple, if you have a Module A with a class A::B inside, when you include A into class C, C::B will return A::B.
C::A will return A with a warning.

Friday, 11 January 2013

Bundler is cool and all, but I like YUM better.

Managing Ruby, its different versions and gems is kinda funny thing, funny as in it has many challenges.
First of all, you might want to handle different projects that work with different Ruby versions and each have a different gem set, so RVM is born and you can change these settings with ease, one less problem - although I do have a co-worker that so far failed to install certain gems with it.

Then there's one of the most annoying problems in Ruby, say your project uses gem "a 1.1" and when you try to deploy with "gem install a" you get version 1.2 and this for some reason causes your application to fail and it fails hard! It's one of the hardest errors to pin-point. Now multiply this problem for each gem and it's dependencies you use, it can really hurt.

That's why most Ruby developers (I use this term broadly, including anyone that programs in Ruby in any way) I know use Bundler and it's great, you get a list of all the gems you have on your system and it freezes the versions you know that make your application works and when you try to deploy it simply installs exactly the same gems you have on your devel machine so you don't have any gem difference problem.
But then you get gems with extensions and they don't quite compile, sure you spend some time doing "bundler install" and then "yum/apt-get install [some lib dependency]" and eventually you're there and all is good. Except if you have some nasty gem like ruby-oci8 which can be very annoying to install and not quite  as they say.
If I still got your attention so far, perhaps it's worth noting that my biggest gripe with these solutions is that they don't solve this problem of gem's unspoken dependencies.

Then you use RVM and Bundler together and all your problems are solved and little or no worries on your side. Although I do hear on occasion that those two have a few kinks working like that, but nothing major.

Then there's what's quickly becoming my favorite way of dealing with gems dependencies and so such: YUM!
It started first as an internal rule of the company I work for, the simple version: all software developed within must be packed in a RPM (for RHEL), and all of it's dependencies, including externally developed libraries (such as gems) must also be packed and available in the company's YUM server.
The reasoning was simple, once the software is sold, one must be able to go to the client's server and do "yum install software" and it will just work. (Later I heard this was because one account of failed installation caused the company to loose millions due to some contract provisions I don't really care to understand.)
While I did find it annoying going through these hops at first, I've been growing into this system and I'm grateful that they actually gave my training on software packaging (which is kinda uncommon for companies in Portugal).

YUM allows us to do some pretty fun things, such as, on a brand new server, I can use yum install to deploy my software with just "yum install" and yes, this does solve my problem of unspoken dependencies.

Take for example, an attempt to install my team's testing library on a brand new ruby installation on RHEL 6:


Notice how the first package in the list is NOT a gem, but rather an unspoken dependency for ruby-oci8. Pretty cool uh?
That also means that I have a greater control of all little things that get installed on the system for my app to work and also that I can purge it with considerable less effort. For instance, "yum remove ruby*" will remove ruby, ruby devel, rubygems, ruby libs, irb, all the rubygems and everything that was installed a dependency of any of those.
Sure RVM let's you easily switch between gem sets, but I can easily switch between "everything sets" - of course that is if I don't mind downloading them all again.
Also, with this you can apply patches to gems that are broken and abandoned despite still being popular.

There's a price for that, like everything in that area, it's knowledge, maybe the knowledge of packing is greater than the knowledge needed for Bundler + RVM, but to fix that, there's a tool, it's called "rgem2rpm" (which I use so much that the latest version, 1.4.9, is the release of a pull request I did on Github), with it, you can simply "rgem2pm gemname.gem" and you get a nifty rpm file. You get a few more options to customize it to your needs, and in the case of gems with unspoken dependencies you can pass an easy to make spec template file (you can use the default template as base for it) and add "requires".
When I took the above screenshot I noticed something was amiss, turns out, the testing version of rubygem-headless that we had on repository was not properly packed, so with 5 minutes to make a template file I compiled a new one and voilá:


See how much stuff you truly need to run the gem headless? Well to be fair, the only true dependency is "xorg-x11-server-Xvfb", all others are dependencies of that.

I know what you might be thinking now, for this to work, you need a yum repo with a lot of gems, at least all the gems you use (and their dependencies), well, yes, right now, I believe that we have more ruby gems as rpm then Fedora's Koji and OpenSuse combined (well, combined is probably an overstatement, but I'm confident we have more than each of them, at least).

All of this is packed for RHEL, they would work on Fedora (and probably others) I'm confident, but not by design.
I have in mind setting up a YUM server and compiling all these gems to Fedora, and providing a way for those using Centos and OpenSuse to use it as well, even when though I think there's little public interest in this, it will would make the setup of my devel computers (and co-workers') much, much quicker.

Edit: I almost forgot to mention, for most people, Ruby is a tool for either Rails or Sinatra web development, with a little bit of effort you can pack your web app into an  rpm and deploy on the production machine with YUM install, then have it start restart the web server (and whatever more you need for it to work, such as running database queries and whatnot) and launch your automated tests.
The more I use it, the more I feel that YUM is a very powerful tool that many neglect in favor of other tools that each do a part of what yum does alone.

PS: I do imagine that APT, PACKMAN, PACMAN and other package managers for the many Linux families are just as capable, but since I started making RPM's for RedHat and that's what I have to deal in a daily basis, I'm bias towards YUM.


Monday, 5 November 2012

Time Helper 2.0.0 released

Funny how time-helper is older than this blog yet not a single post or page about it here.

I've considered this a new major version because there's no guarantee of compatibility with previous versions, in a couple features.

About version 2.0.0

I've changed the =~, the old way was just stupid (no one else to blame but me here), the new version just compares two Time objects and see if they are within an acceptable distance from each other in seconds, defaults to 300 (5 minutes).
So where you'd have a symbol as first argument you now have a Fixnum. Also added a sanity check for the second argument.

Completely removed the #parse integration in #strtotime, this means that if you were using #strtotime as both it won't work, as such, it reduces the memory footprint of this gem (by not requiring stdlib time)  for those who don't want .parse - which is most of the people that I've made this gem for (including myself).

And yes, this gem still doesn't care about American date format (mm/dd/yyyy), because it's silly.

I've already found a couple issues that need to be fixed:
  • Documentation for =~ doesn't completely reflex the changes.
  • Time.valid_datetime? returns true for more than intended inputs.

Tuesday, 23 October 2012

Ruby: hiding method from an instance

This one is simple, for some reason I needed to hide a method from an instance so people would use my wrapper instead of the method directly - I have good reasons, but to achieve this we can, again, resort to instance_eval:

irb(main):001:0>  class A
irb(main):002:1>  def a
irb(main):003:2>  p 'hello world'
irb(main):004:2>  end
irb(main):005:1>  end
=> nil
irb(main):006:0> 
irb(main):007:0*  a = A.new
=> #<A:0x0000001fbee140>
irb(main):008:0>  a.instance_eval do 
irb(main):009:1*  alias :b :a
irb(main):010:1> def a
irb(main):011:2> p 'you should not use this method'
irb(main):012:2> end
irb(main):013:1> end
=> nil
irb(main):014:0> a.a
"you should not use this method"
=> "you should not use this method"
irb(main):015:0> a.b
"hello world"
=> "hello world"

Monday, 24 September 2012

Clearing excel formula cache when using jruby-poi - making Formulas work right

Last week I've been working with jruby-poi to make an script that would automatically update an MS Excel file.
The part that cost me the most time, believe it or not, was formula updating, you see, I would add rows that should change the value of pre-set  formulae, but the results would not update on Excel until you clicked on the formula.
After diving in the nearly nonexistent documentation of the gem (they give you base-usage on git, some test case you can check out also on git, everything else is default generated rdocs without a single improvement) I've came up with the following work around:


       %w[jobs defects].each do |sheet_name|
            sheet    = self.workbook.worksheets[ sheet_name ]
            rows     = sheet.rows
            from, to = 2, 70
            rows.each_with_index do |row, i|
                next if i <= from
                #order is important!
                %w[L K D E F N O P Q U V X Y Z W AA AB AC AD AE AF AG AH AI AJ].each do |letter|
                    column = letter.to_number - 1
                    puts sprintf( 'updating formula on C: %s row %d', letter, i )
                    self.workbook.on_formula_update rows[ i ][ column ]
                end
                break if i >= to
            end
end


I hope it helps anyone, also I've opened an issue on their git, let's see how they respond.

Wednesday, 19 September 2012

Turning letters into numbers in Ruby (Excel style)

Today I was writing some code to update an Excel file with more recent data from various sources. I was using JRuby with jruby-poi, it provides a nice API and so far it's the only gem I found that can open and edit an xlsx file without breaking it.

I had an interface problem however, I was trying to provide the people that will use my code an easy interface, so I needed them to be able to pass me [excel's] columns as letters (A, B ,C...), but for editing with poi the interface is the object Rows, an Enumerable with numbered indexes like an Array, that is, I can't do rows['A'][1], I must do rows[0][0].

Googled around for a few ideas and I came up with this (it works up to column AZ as far as I know)

class String
    def to_number
        number = self.tr("A-Z", "1-9a-q").to_i(27)
        number -= 1 if number > 27
        return number
    end
end


It works fine with strings that are just an capital letter, I didn't do a sanity check because you might wanted convert a bunch of letters in the same string, also because your column might be something like 'AA' or 'AB'. If you were looking for the simple answer you can stop reading here.

I'm gonna math-nerdout here a bit to explain what is happening:
the method "tr" here will replace everything from "A" to "I" with the numbers from 1 to 9, everything else will be replaced with the letters from "a" to "q" which are the numbers and letters used in the base 27 numeric representation.

For those of you unfamiliar with numeric which aren't base 10, they are represented as such:
0 to 9 stays the same, but everything after that are letters from English alphabet.

So 10 in base 27 is 'a', 26 is 'q' (the last letter used by base 27) and 27 is '10'.

(Don't ask me how are numbers represented in any base greater than 36 (where 35 is 'z'), Ruby won't accept any greater base anyway.)

The argument 27 in the "to_i" call is to tell ruby that the string is a number in base 27 for when it does the conversion to Fixnum (you can also "to_s 27 "if you want to convert a number to base 27 ).

Let's say your string was 'AA', the tr method will turn it into '11', this is not "eleven", in base 27 this is 28*. But the result we wanted is 27, not 28 (AA is column number 27), thus the "-1 if number > 27".

I could explore this solution to come up with some better math to make sure this method worked correctly even if you're looking for column 'ZZZZZ', but I don't need it, as this was a very small problem from the big problem I was facing today.


*Trying to explain this in layman's terms (since I myself am a layman) 
Normally we represent numbers in base 10, this means that when we get to the number "ten" it kinda loops the representation and goes back to zero preceded by a 1, this one indicates how many "full loops" we've done to the base so we represent the number ten like so: "10".

However, if we increase the base , we need more symbols, so at base 20 when we reach "ten" we can't represent as "10" (that would be twenty) because we haven't looped it yet, so we just borrow letters till we have enough symbols to represent all numbers in one loop of the base.

In the case of base 20,  we represent one as "1"; nine as "9", ten as "a", nineteen as "j", twenty as "10".

Tuesday, 18 September 2012

Uploading attachments to Jira using Jira4r

Today I found myself needing to make a ruby script that would update a file and upload it to an issue as an attachment in Jira.
I found the SOAP API method to do this in Jira's documentation, but I didn't find a suitable mapping in the Jira4r docs.
I found this comment on the Jira4r page, the only  reference to the API's addBase64EncodedAttachmentsToIssue method. Also found some forum posts of people with the same issue and that comment was the only found solution.
So if you need to upload stuff to Jira, follow the instructions in that comment, you'll find the file to edit at "[GEM_PATH]/gems/jira4r-0.3.0/lib/jira4r/v2/jira_soap_service_driver.rb"
The end result I got in my file for it to work was this, worked for my task.

The usage is quite simple, the test example on the page linked shows well how to use it, I'm just going to paste here to keep it "in record".
The only difference between what I did and what's on the example is that I used the base64 standard lib for encoding while he uses Array#pack('m') which results on the same thing.


#!/usr/bin/env ruby
$LOAD_PATH << './lib'
gem('soap4r')
gem('jira4r')
require 'jira4r'
require 'base64'
require 'cgi'
jiraBaseUrl      = "http://localhost:8080"
jiraUsername     = "automated-user"
jiraPassword     = "Thbmahws,"
escapedPassword  = CGI::escape(jiraPassword)
jira = Jira4R::JiraTool.new(2, jiraBaseUrl)
jira.login(jiraUsername, jiraPassword)
newFilepath = "H:\\edisc\\Tst\\Printed\\Test.eml"
fileName = newFilepath.split('\').last
issue = Jira4R::V2::RemoteIssue.new
issue.project = "ED"
issue.type = "1"
issue.summary = "This is a test"
issue.description = newFilepath
returnedIssue = jira.createIssue(issue)

filedata = File.open(newFilename, "rb") { |f| f.read }
attachmentData = filedata.pack("m")
jira.addBase64EncodedAttachmentsToIssue(returnedIssue.key.to_s, fileName, attachmentData)