Posted on

Painful Selenium Testing – Part 2 – Confusions

Facebooktwittergoogle_plusredditpinterestlinkedinmail

Firstly, here is the structure we are using right now:

Selenium IDE + Selenium RC HTML + Jenkins + Selenium Reporter + Groovy Script + Twilio Alert

This is how it works now:

  1. Most long-term projects have automated HTML UI tests (e.g. open URL, click, type in, submit, etc..) run every night

  2. It will check sites, as well as server healthiness.

  3. If it's success, a blue light will appear in Jenkins.

  4. If it's failed, a red line will appear.

  5. If it's a critical suite (e.g. site failed to be opened), and it continuously failed for 5 attempts, the system will send SMS to alert the developer.

Confusions on Selenium

Selenium is not the most complicated framework I have been working with, but it is probably the most confusing one. I have read through the official website of Selenium for more than 10 times, yet I still I keep getting confused, again and again.

It's actually a set of tools, instead of one single tool. So when you search 'selenium' in Google, you would actually see the results regarding different things, instead of one single product. In fact, there is a total of four different things you need to be aware with, and their names all contain 'selenium':

Selenium IDE, Selenium Server, Selenium RC and Selenium Webdriver

  1. Selenium IDE is a plugin of firefox to record and replay your actions on the browser. You could use it to export the testing actions into HTML suites. However, you could not use command line to automate the replay of actions on Selenium IDE, you are going to use Selenium Server to do that.

  2. Selenium Server is one single .jar file for replaying the exported suite of selenium IDE via command line. That means you could use this to schedule the testing to be run by 3am of every night.

    To Run it by command line, you need something like this:

    java -jar c:/jenkins/selenium-server-standalone-2.33.0.jar -userExtensions C:\jenkins\bin\user-extensions.js -port 4520 -timeout 900 -firefoxProfileTemplate c:\FirefoxPortable201\Data\profile -htmlSuite "*firefox C:\FirefoxPortable201\App\Firefox\firefox.exe" http://example.com test_cases\suit_405_testing_example.html result_suit_405_testing_example.html -singlewindow

    Here are some more details of the command.

  3. Selenium RC(now deprecated) is a something similar to Selenium IDE. However, in Selenium RC you could use Java, Python, Ruby or .NET scripts to write the testing suites. That means, you need to do some coding. For obvious reasons, they got (much) more flexibility then HTML suites.

    You would need the same Selenium Server jar file for the RC to execute.

    However, we don't adopt this approach because we have a strict cut-off line between QA and engineer, and for practical reasons (like many other small IT companies) our QA can't write the complicated codes. It's not a choice of technology, just a compromise of reality.

  4. Selenium Webdriver is the second generation of Selenium. As mentioned by the official site:

    Selenium 1.0 + WebDriver = Selenium 2.0

    Basically, it's something totally different. It changes the entire approach to interact with browsers. The first generation of Selenium (Selenium RC / Selenium IDE) interact with browsers via some tricks of proxy. However, Selenium WebDriver interact with browsers directly. In fact, many browser vendors contributed to Selenium WebDriver.

    Unfortunately, they require users to write the code. The suites are similar to those of Selenium RC, but with some subtle differences. In any case you can't just use the exported suites for the automated tests. So, for the same reason we are not using this approach either. However, in future we might try to write some hard test suites using WebDriver, as it basically could do anything. Especially it works much better with Internet Explorer, the Selenium IDE on IE is just a nightware.

Conclusion:

However, as mentioned in the beginning, we eventually use:

Selenium IDE + Selenium Server

Though WebDriver is more recommended for complicated testing in the official site, and there are indeed a lot of reasons to use that, there is one single reason that forces us to stick to selenium Selenium IDE:

We simply just can't hire programmers to do this.

Programmers are too expensive and easily get bored to write the testing cases, and (currently) QA are not qualified to write the even the simple suites of WebDriver.

If we are a big company like Google or Facebook, we could, of course, hire a bunch of programmer for this, but not for now.

Using selenium IDE, I could hire some girls with no engineering skill at all to do this. Two advantages here:

  1. Girls are in general more detail minded then the most engineers
  2. Girls normally make the bug reporting more acceptable
  3. Engineering skill is not required
  4. And the girl testers could also do the manual testing for some short term projects, or the projects that have no way to be automatically tested (e.g. location-related apps).

That makes the whole thing feasible, and make hiring much easier. If you can't hire someone to do that, all the discussions and theories are meaningless. I think the problem with testing automation nowadays is, people really want it because everyone says it's important. On the other hand, they just can't get to there because it's really difficult to fully reach there. And IMHO I think the key is getting it to work first, then make it good.