Blog: Testing of cloud-based solutions

The increased use of cloud-based applications delivered as a service is an enabler of the digital transformation for organizations. Cloud solutions are a part of the new SMAC-technologies (Social, Mobile, Analytics, Cloud), and enable organizations to use and adapt their IT and application architecture almost instantaneously on an as-needed basis.

[19. oktober 2015]

Cloud computing

The US-based organization NIST (National Institute of Standards and Technology) has defined cloud computing as the following “Cloud computing is a model for enabling ubiquitous, convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, servers, storage, applications, and services) that can be rapidly provisioned and released with minimal management effort or service provider interaction.”

This cloud model is composed of five essential characteristics, three service models, and four deployment models.

The five essential characteristics are:

  1. On-demand self-service
  2. Broad network access
  3. Resource pooling
  4. Rapid elasticity
  5. Measured service.

The three service models are:

  1. Software as a Service (SaaS)
  2. Platform as a Service (PaaS)
  3. Infrastructure as a Service (IaaS).

The four deployment models are:

  1. Private cloud
  2. Community cloud
  3. Public cloud
  4. Hybrid cloud.

According to the World Quality Report 2015/16 (WQR) published by Sogeti, Capgemini and HP in cooperation 43% of all the cloud-based solutions run in private cloud, 28% in hybrid cloud and 27% in public cloud. The rest run in community cloud.

This paper will focus on the SaaS service model, and how to test it.

Testing cloud-based applications

Cloud-based computing creates new testing challenges. Cloud applications are still few compared to traditional applications, but more and more companies are moving towards cloud-based solutions, so their numbers are be increasing. There are already several global cloud applications – one of them is salesforce.com.

Is testing cloud-based applications so different from how we have been doing our testing so far in more traditional solutions? There is not a difference in testing cloud-based applications and traditional applications. The only thing which is different is ‘what we test’. First there are a lot more parties involved in testing and second there are also typical more quality characteristics. The parties involved are the business, other stakeholders, but also eventually the suppliers of the SaaS applications. According to the WQR 52% of the organizations pay special attention to both security and performance risks when testing cloud-based SaaS applications. This focus can indicate that organizations accept the functionality of SaaS as given, but need to focus on other aspects of the service.

Business driven test management (BDTM) is a well recognized approach to testing which focuses on risk management and in turn test of specific quality characteristics, such as performance, functionality, security etc.  If the basic functionality of a cloud-solution is believed to be thoroughly tested by the provider, organizations might shift focus from functionality to other aspects of testing. This is also shown in WQR where the prioritization of functionality testing in cloud-based settings is in decline and aspects such as security and performance testing is rising.

However, most cloud-based solutions require certain customizations to fulfill the alignment between the specific workflows in an organization. Thus it is evident that customers cannot completely rely on the test of functionality (both accuracy and completeness), effectivity and suitability, that the provider has carried out and still need to at least include a minimum of test of those quality characteristics.

Integrations towards already existing systems could also be a high-risk area in a cloud-based setup with regards to e.g. exchange and mapping of data and needs to be included in the risk assessment and in the test.

Based on the above mentioned observations it is possible to derive a list of quality characteristics to consider when testing in a cloud-based setup:

  • Functionality (accuracy and completeness)
  • Effectivity
  • Suitability
  • Efficiency
  • Security
  • Performance.

Other aspects than quality characteristics might also play a role in the risk assessment, such as legislation, creating or obtaining test data and test environments.

Country-specific legislation might have requirements for e.g. storage (e.g. inside/outside the EU) or exchange of personal data between the customer and the supplier of the cloud-solution.

Legislation in Denmark requires the financial sector to store data within the EU, a requirement that could be difficult to test in a cloud world.

Even though the test organization must rise to meet new challenges when testing a cloud-based solution, some old challenges might be resolved or at least easier to handle in a cloud.

In the 2014-2015 World Quality Report research showed a drop in budget for testing hardware and infrastructure, which was attributed partly to the increased use of cloud-bases solutions. The current WQR shows that 41% of respondents use cloud-based test environments, not only indicating that SaaS/PaaS is on the rise, but the use of cloud-based solutions mainly for testing is as well.

This change could indicate that creating test environments and the necessary test data is easier and faster in a cloud-based environment, perhaps solving one of the big challenges for test organizations worldwide.

Closing remarks

The cloud is a business model or platform on which testing must be carried out just like any other service, but as the cloud era emerges, testing will change. Testing applications on the cloud is the same as testing applications on a traditional set up – only WHAT is tested is different.

todo todo
  • Merete Nielsen
    Merete Nielsen
    Senior Test Analyst
    +45 52189789