Chasing Quality
One of the maxims we try to follow in NavalPlan (LibrePlan) is to create a project with good quality.
Quality in software refers to two different notions:
- Functional quality. It is the degree in which a software satisfies the specifications. The better the software complies with them, the higher the quality a program is.
- Structural quality. It is related to all the non-functional requirements which can be stated over a program. For instance, how good is the development cycle, how maintainable the source code is, what performance is achieved, etc.
Said that and taking into account this classification, I would like to introduce automated web tests and relate them to this taxonomy.
In the first place, I will define what they are for those of you not familiarized with them. In short, we can say that automated web tests are black box tests in which the interface of a web application is tested in an automatic way. In other words, they are a type of tests in which a program performs the role of a real user and interacts with web pages with the aid of a browser to assure that the behavior of a web application is the one expected.
In second place, as I told you, I would like to link them with quality. In general, we can say that they provide structural quality because, on the average, a web application with functional tests has a higher quality than one application without them. They help to detect failures and regressions and, therefore, in the end, the likelihood of having bugs is smaller.
Sahi Web Tests
In the NavalPlan team we have been looking for the best alternative to do automated web tests. Apart from the general cited reason of having higher quality, we try to address the jointly effect of having a large-featured application and a short-numbered testing team. When these two factors are combined the likelihood of regressions is big and the cost of a comprehensive manual test of the application high. Therefore, for sure, with a good set of web tests we would improve both in robustness and productivity allowing us to plan less testing time.
After looking for several alternatives, the technology that we chose is Sahi and the reasons which supported our decision are the next ones:
- In NavalPlan we use the web interface framework ZK. This is a framework which generates dynamically the id attribute of the HTML entities which make up the web pages. This makes difficult to develop automated tests because the id is one of the easiest ways to locate HTML entities in the DOM and, some of the most common testing frameworks, like Selenium, relies basically on them. However, as they are dynamic in ZK, i.e., each time a page is rendered they are different, it is impossible to make the tests being repeatable with a technology based on the ids. Luckily, Sahi allows to overcome this situation because it has a powerful accessor API which helps to locate HTML elements using concepts like indexes, human DOM relationships as can be near or parent, CSS classes, etc.
- Sahi is browser independent. This means that the automated tests can be executed in several browsers. This is great, because a RIA application like NavalPlan uses the latest HTML technologies and some of them might be not fully supported in a particular browser. We can run the tests in all of them and this is a big advantage for us.
- Tests are programmed in JavaScript what in my opinion is a great idea. To start with, because JS is the language used by browsers since the very beginning and is a standard with a good API to interact with the DOM. Another good feature is that, because of being the tests written in a programming language, we have the programming tools like functions, data types, control structures… which gives you the highest flexibility to build tests as complex as you need. Some other testing technologies relies on configuration files, like XML files, and this limits a lot the possibility to get off the path of what the web test framework developers initially thought.
Now I will focus on the things I would like to be different in Sahi. Among them, I would highlight that there is a proprietary product (Sahi Pro) built on top of the Sahi Open Source. Sahi Pro provides the more advanced features and I really miss that some of them were in the open source product. For example, a better report system. For me it would be nice that they had an open source license for the Sahi Pro product to be used with free software products like NavalPlan.It is a way to promote both quality in open source and open source itself without damaging the comercial interests of a company supporting a free software product.
Where are we ?
We started developing Sahi tests last month and, at present, we have tests for some of the simpler use cases, which are CRUD use cases related to administrative operations. If you feel like having a look to how they work, I encourage you to deploy NavalPlan, to download the git repository and to read the README file in the script/functional-tests folder where the instructions to run them are described.
Additionally, they say that tests are successful when they detect errors and, in this sense, we proudly 😉 can say that right now we have reported some new bugs on the bugzilla thanks the functional Sahi tests developed so far.
Where are we going ?
Our roadmap concerning web tests will consist of increasing the coverage and facing up more complex interface operations in the near feature. After it,
a last final desired scenario will consist of having a platform in which:
- We develop a Maven plug-in or write a configuration to be able to pass the tests integrated in the building process as part of the Maven test phase.
- In NavalPlan we use CI and the continuous integration server we have is Hudson. It would be great to integrate the Sahi tests execution in Hudson build cylce and to have the test results published in the Hudson interface to find them easily.