Posts Tagged ‘testability’

Testing SaaS

Posted: 13.7.2011 in Ei kategoriaa
Tags: , ,

I love cloud services! But I also consider their negative sides and what new they bring to my life as the tester, administrator and user. Software as a service (SaaS) is great for lazy administrators like me. They are bringing new exciting stuff for curious (and evil) tester like me. For the user they bring simplicity.

Let’s start to think how they are affecting to testing. I consider SaaS to be same as commercial off the shelf (COTS) with Internet twist. When I’m taking the application to use I have some specific need. E.g. when I moved to WordPress.com I wanted to have simple blog software where I can migrate my old posts as easily as possible.

At normal testing we’re concentrating to requirements. At SaaS that is unfortunately the smallest possible part you can test. WordPress.com is providing plenty of additional features which I don’t need. I can e.g. protect part of the posts, I can add more authors to blog, I can choose many different kind of sharing options and combinations and so on. There are so many different options that I don’t even know them yet! I should try to investigate them at some point. But what does this mean to testing? The diagram below shows the difference between “required” features, and “provided features”.

Can we forget the features which are not required? Absolutely not! They should be part of testing at some extent. At least the testing should make sure that users cannot damage the normal use from those features. Extra features should also be part of risk assessment. The assessment should consider security, functionality, availability and performance risks. Management should decide if the risks and their probability can be accepted. The risks should also be pointed at instructions, policies and training.

Google Docs is good example. It is good tool for collaborated writing. But there is also plenty of security considers. Even if the organization doesn’t need “Sharing to whole Internet”, it still is features of Google Docs. If the risks related to it are at acceptable level, the usage policy can say: ”Never mention any customer name at documents which are at Google Docs, never share any internal document to ‘everyone’ or ‘everyone with the link’.” There is still the risk that users accidentally publish the private information. But now the risk is noted. There should be some plans what to do if risk is realized.

Cloud computing is not affecting only to feature testing. SaaS is very often the web services and same security issues might exist as any other web application. Unfortunately SaaS provider might deny the good security testing, because there is always possibility for denial of service or data loss which affects to all users. Same problem is with performance testing. So instead of real testing those you are only able to do risk assessment.

I’ll write more about risks at some point of future.


Thanks testedtested about good thoughts @Twitter. 🙂 I had to start thinking about testability and its relationship with customer happiness and usability. 140 characters is too less for very well formulated ideas and comments so I had to write blog entry.

If I can use application, it doesn’t mean that I can test it. In many cases the testers must request developers to add code and features to increase testability. In many applications good usability means better testability, but that’s not case in many embedded systems.

Good example is Google’s self-driving car. From user point of view it is very simple system. Just submit where you are going to, hit the button and car takes you there. (This is very simplified view.) Even if that process is done simple, it is definitely not the same as testability.

I slice the simplified system to multiple parts:

  • Sensors
  • Motor
  • Wheel
  • User’s GUI which inputs the destination
  • Button which starts to drive

The software which controls all of these is software under testing (SUT) in my example. For me I’d need following things this to be testable:

  • I have to be able to send artificial messages from the sensors to SUT
  • I have to be able to inspect what kind of messages motors get
  • I have to be able to give artificial feedback from motor to SUT
  • I have to be able to control (virtual) wheels same way as normal road affects to wheels

If any of these is missing, I would not be able to test the control system. These need plenty of simulators and stubs which tester or test automation can inspect and control. But even without those simulators and stubs, extra interfaces the user could use the system and even be happy (if it works as expected). And even tester could use the system without them.

So testability is not always automatically part of usable software. Sometimes testability needs plenty of additional work for developers. And in those cases it is usually testers’ responsibility to give details how to make application more testable.