Many times I hear that exploratory testing (ET later) is pure manual testing. But that’s not true. You can use any possible tools to help ET. This is the part one of multiple articles where I present tools which you can use to assist your ET and other manual testing.

What is the purpose of tool? Its main purpose is to release tester to do meaningful tasks. If initializing the test takes more than 1 second and needs to be repeated over and over again, it is preventing the good testing. And tools should be used to remove that kind of obstacles.

Unfortunately often the tool itself becomes the “purpose of testing”. I know that – I am usually doing test automation. It seems that very often the tool itself becomes the obstacle because someone thinks that it is the silver bullet for all testing problems. At that point the tool turns to testing problem.

After short introduction let’s start with very simple case. Let’s imagine we were testing wordpress.com. For tester that is really boring case, because every time he wants to do something, he has to login. Easiest way to get around the problem is to get tool to do login. If the case is this simple, I’d take Selenium IDE. It is Firefox plugin which records the test case. After recording it can be played over and over again to get the test to specific point. The screenshot below shows the whole test for WordPress.com login. (Credentials are not real ones…)

Selenium IDE

Selenium IDE script for WordPress.com

Selenium IDE is good for small tasks, but I would not recommend any recording tool for large scale test automation or complex tasks. Its simplicity justifies its use.

I will write later more about tools which can help exploratory and other manual testing styles.


Cloud computing combines many different quality aspects. Testing the service is mix of real testing and risk assessment. Risk assessment on the other hand requires wide knowledge from business processes to development processes (like ISO-12207, CMMi) and IT service processes (like ITIL). They also need at least some knowledge about local privacy laws. This post is about availability and its aspects.

Availability is defined at ITIL terminology. It is “Ability of … IT Service to perform its agreed Function when required.” [1] When we start to analyze availability of cloud service, we must understand who the users are and how they are connected to Internet. If the server is at our own computer room, or at well-defined location of service provider, we have clear understanding what parts the network infrastructure has. But as soon as we start to use some cloud service, we lose understanding how the data is flowing between services. In worst case we don’t have any control to data. And even in best case we know the approximate location of our data.

Negative risks have multiple parts. First are our organization and its connection to outside world. If most of the users are using the service from office, the Internet connection from the office to outside world is first possible failure point. When decision for cloud use is done, organization should make sure that their Internet access is enough for required bandwidth. At the same time the business should make decision how long the service can be unreachable.

Service level agreement (SLA) is important part of availability. Unfortunately many cloud service providers are not providing SLA. Their license agreement can state “best industry efforts to guarantee availability” or “99% availability”. The contract usually doesn’t provide much compensate from down time. At the end IT service provider cannot create the SLA when service fails. It can only define, that it takes contact to cloud service provider and notifies them about problems. Then it’s up to cloud service provider how quickly they react.

Cloud computing has also positive sides. Let’s take an example from this blog. I used to host this at one virtual host which was running also other services. It’s at Finland because majority of users for those other services are in Finland. This blog is international blog, so keeping this at Finnish site isn’t mandatory anymore. The major risk at that Finnish site is that if it goes down for any reason, I don’t notice it before morning.

[1] http://www.knowledgetransfer.net/dictionary/ITIL/en/Availability.htm

Google is your friend

Posted: 28.7.2011 in Ei kategoriaa
Tags: ,

Every now and then (actually more than just every now and then) I find the word which I don’t have any idea. That can be e.g. “threat modeling”, “virtual user”, “xpath” and so on. I have to find quickly, what is behind that and should I really know it. I have multiple options how to proceed.

  • Ask from the forum like this
  • Ask at IRC
  • Ask at Twitter
  • Ask at Google+
  • Do the search at Google

 

If I’m using first four bullets, I have several assumptions. First is that someone else knows about subject. Secondly I’m expecting that “somebody” to be online and he have some free time. While waiting those conditions to become true I’m still clueless about the topic. And usually I have to get at least basic information about it quickly. I don’t want to waste my time and my client’s money for waiting. And even if I got answer, I never can be sure if that is correct answer. The one who answers might be as clueless as I am, but he doesn’t realize that. So situation is lose-lose situation. I might get wrong information. Or it’s possible that I won’t get any information at all. Not even wrong one. My work is blocked until I get the information. And someone else must waste his time to explain me the basics.

I’ve found that Google is much better option. If I write keywords to search field, add additional word “Tutorial”, I usually get some basic information about the subject I need. I also might get good references for further reading. This Monday I really had to find quickly information about threat modeling. First there was Wikipedia article, then article at OWASP-site. Those gave me good overview to subject. But it wasn’t enough. Third link was to Microsoft site and there they advertised book about threat modeling. Great! That’s what I needed. There was direct link to O’Reilly site, and possibility to buy eBook. I bought eBook, downloaded it, uploaded to my eBook reader, and now I’ve read more detailed information about threat modeling. I didn’t have to wait for days (or not even hours). I got quick overview and after that knew I had to know more.

Summary: Google doesn’t force you to wait. Usually you find at least the basics of subject you’re looking for. If after that you have questions (e.g. how to implement threat modeling to your environment), then forum is good place to open discussion about that. It won’t give you the step-by-step instructions, but it can give you some ideas how others has used same tool at their project.


Yesterday something went wrong – Below is screenshot pointing out the problem. Read the rest of article to find out more details.

The screenshots had very common mistake – redirection. It’s very good tool for phishing attacks. If attacker has any possibilities to send e-mail to registered user, he is able to attack against the user. For example how many would have noticed the redirection bug if he got the e-mail message:

From: root@trusted-mail.net

Subject: E-mail validation

Please, validate that your e-mail address is still valid by logging in to the Trusted Mail. We are not requesting any passwords by e-mail and as you can see from link below, it is pointing to Trusted-mail.net. Make always sure that address bar has correct address.

So please, log in with this validation link:

https://trusted-mail.net/horde/imp/login.php?url=%68%74%74%70%73%3A%2F%2F%65%76%69%6C%68%61%78%6F%72%2E%63%6F%6D%2F%68%61%78%2F%74%72%75%73%74%65%64%2D%6D%61%69%6C%2F

Sincerely,

Mail admin of Trusted-Mail.com

That mail looks valid – doesn’t it? When redirection goes wrong, the attack against the service users is simple. It just takes few minutes to copy the original login page, change the form processing so that submitted data is stored to local file or database, and then redirect back to original site. It’s very difficult to notice, that the target site is actually new site and not the original one. At least I’m mistyping my passwords every now and then, and when the system says that password is wrong, I try to write it a bit more careful but without checking where my browser actually is.

So the failures are actually at many level. Bad web application is not validating its input correctly. I have written about that before at post “Phun with redirects“. But second is more fundamental failure. It’s at e-mail protocol. The From field is not validated. Attacker can give any From address and receiver wouldn’t notice that.

The original SMTP protocol was defined at RFC 821 year 1982. Back then there wasn’t need for security. And now it would be just too expensive to add new security layer, because everything has been implemented to use old and insecure protocol. Maybe at next bug report I should state:”The root cause for bug is Internet and its protocols.”

Oops – What’s wrong?

Posted: 20.7.2011 in Ei kategoriaa
Tags:

Can you notice from screenshots below what went wrong? I’ve seen same kind of errors MANY times.

 


Centralized cloud and other web services is major risk for private users, but also for business users. There are plenty of problems. I have experience with Google AdSense. I used to have it at one site for a short time. But it was terminated because I have “violated the contract”. That’s great. I requested information how I had done that and how to fix the problem. I didn’t have any intentional violations. I read thru the contract multiple times. Did they help me or tell the reason? No. I just got same automated (or premade) message, which told that I had violated some rules and it’s secret what I had done wrong.

Things could have been worse. Now I’m banned from AdSense forever, so I cannot get any income from Google ads. I cannot use AdSense in any of my services so good bye most ad income. How could it have been worse? They could have banned me from all of their services. That would be catastrophic for my private life. I have Google+ account, I have Gmail, I’m using iGoogle, I use all kind of fancy services they have. If I violate the term of services at Google+, I might lose all of those as reported at Business Insider. This limits my freedom of speech. This is putting razor blade on my throat. I have to be careful what I do or I might lose major part of my private and professional life.

Centralized services are good from usability point of view. But they are risk for many other parts. Everything gets worse as soon as someone owns majority of services. Microsoft couldn’t play as dirty game with Windows and other software as Google can play with the services they provide. Google can block me from many services with one click and they don’t have to justify their actions. They could just say “violated the term of services”. I don’t have money go to battle against them at court.

I have to be nice when at Google+. I’ll try to avoid being evil tester there. I just want to keep all my services and accounts up and running. Spirit is strong, but flesh is weak, but now I have to kill my flesh.

Testing SaaS

Posted: 13.7.2011 in Ei kategoriaa
Tags: , ,

I love cloud services! But I also consider their negative sides and what new they bring to my life as the tester, administrator and user. Software as a service (SaaS) is great for lazy administrators like me. They are bringing new exciting stuff for curious (and evil) tester like me. For the user they bring simplicity.

Let’s start to think how they are affecting to testing. I consider SaaS to be same as commercial off the shelf (COTS) with Internet twist. When I’m taking the application to use I have some specific need. E.g. when I moved to WordPress.com I wanted to have simple blog software where I can migrate my old posts as easily as possible.

At normal testing we’re concentrating to requirements. At SaaS that is unfortunately the smallest possible part you can test. WordPress.com is providing plenty of additional features which I don’t need. I can e.g. protect part of the posts, I can add more authors to blog, I can choose many different kind of sharing options and combinations and so on. There are so many different options that I don’t even know them yet! I should try to investigate them at some point. But what does this mean to testing? The diagram below shows the difference between “required” features, and “provided features”.

Can we forget the features which are not required? Absolutely not! They should be part of testing at some extent. At least the testing should make sure that users cannot damage the normal use from those features. Extra features should also be part of risk assessment. The assessment should consider security, functionality, availability and performance risks. Management should decide if the risks and their probability can be accepted. The risks should also be pointed at instructions, policies and training.

Google Docs is good example. It is good tool for collaborated writing. But there is also plenty of security considers. Even if the organization doesn’t need “Sharing to whole Internet”, it still is features of Google Docs. If the risks related to it are at acceptable level, the usage policy can say: ”Never mention any customer name at documents which are at Google Docs, never share any internal document to ‘everyone’ or ‘everyone with the link’.” There is still the risk that users accidentally publish the private information. But now the risk is noted. There should be some plans what to do if risk is realized.

Cloud computing is not affecting only to feature testing. SaaS is very often the web services and same security issues might exist as any other web application. Unfortunately SaaS provider might deny the good security testing, because there is always possibility for denial of service or data loss which affects to all users. Same problem is with performance testing. So instead of real testing those you are only able to do risk assessment.

I’ll write more about risks at some point of future.