Stormrunner load is HPEs (formerly HP) latest offering in the load testing space. HP has interestingly positioned it as a valuable addition to its existing performance testing solutions (Loadrunner/Performance Center) rather than as a replacement or alternative to the existing offerings.
Given that Loadrunner and Performance Center are already comprehensive tools that can cover most of the scenarios your business might come across, you might ask, what is the USP of Stormrunner Load. HP considers the following to be the key selling points of Stormrunner Load.
- Additional features
So, the big question is – Does Stormrunner Load deliver?
Lets pick up each of its key selling points and evaluate its performance.
Stormrunner has been designed as a SaaS product and focuses on only a subset of what a comprehensive performance test tool has to offer. The current focus is predominantly on HTTP/HTTPS protocol only (although the list of supported protocols has increased* since its first launch and could potentially grow further).
* As of now, Stormrunner supports Web – HTTP/HTML, Mobile Application – HTTP/HTML, SAP Web, Web Services, Flex, Java, TruClient Web, TruClient Native Mobile and TruClient Mobile Web protocols
The interface is minimalistic, responsive and easy to use. Creating a test is as easy as clicking on a few buttons and navigating through a few webpages. And monitoring tests during (and after) execution using one of the interactive charts, is sheer pleasure – A remarkable improvement over the relatively poor user interface of Performance Center and the age old interface of Loadrunner.
A quick walk-through of how to create and run a quick test using Stormrunner could be found below
One of the main goals of Stormrunner appears to be the ability to generate any load on the fly (i.e.) not limited by the number or the capacity of the installed load generators like in the case of Performance Center/Loadrunner. Stormrunner allows load generation from the cloud (either HPs own cloud or AWS). That means, if you have a load test that uses 1000 users and you want to increase the load up to 100000 users, you have to do nothing explicitly to get this working. Stormrunner load will spin up as many VMs as required to generate this load (provided you have sufficient licensing)
This is very helpful and in-line with the current trend of performance testing tools. The fact that Stormrunner can work with your own AWS instances is a bonus.
Performance Center/Loadrunner licensing model was based on purchasing a specific number of Vusers per protocol bundle. So, if you have a 500 user license for the Web 2.0 bundle and you need to do a quick test for 5000 users, either for a capacity management exercise or to investigate a production issue, you could not do it unless you buy a license for the remaining 4500 Vusers, which is expensive and the operational process could take time.
Stormrunner on the other hand, provides the option of purchasing a specific number of Vuser hours (again per Vuser type*) at anytime you want. So, if you have already purchased 100000 API Vuser hours, then you can either execute:
a) 200 (1 hour) load tests with 500 users each (or)
b) 20 (1 hour) load tests with 5000 users each
* Stormrunner Load has 3 different types of Vusers based on the protocol support – API Vusers, UI Vusers and DEV Vusers
If you have only purchased 1000 VUH and you want to do a 5000 VUH test, purchasing additional VUH is very easy and quick and far more cost effective than buying Vuser licenses.
In addition to supporting all the protocols mentioned above, Stormrunner also supports HTTP(S) scripts developed using JMeter and Gatling to be run using its interface. This is a significant change in the direction from HP. Over the last few/several years, the market has been steadily moving towards open source and HP seems to have come to terms with it, at last.
So, now you can upload your JMeter or Gatling scripts as-is to Stormrunner load and start running tests without any need to modify the scripts. In addition HAR (HTTP Archive) scripts recorded using your browser tools or a proxy like Fiddler/Charles are also supported.
One more feature that comes packaged with Stormrunner is Network Analytics. You can now view what gets transmitted over the network, how long each network interaction takes for desktop/mobile devices and some, rather useful suggestions on how things can be improved.
They also seem to have partnered with WebpageTest where you could (optionally) get a page breakdown report from WebpageTest.
As with everything, all is not rosy. There are few downsides too (purely subjective – depending on your business need). Listed below are some of the features I find lacking in Stormrunner Load.While it is easy to understand the reasons for limiting the protocol support to a small subset of the available list, it is not clear why some of the features related to scheduling and run time settings have been trimmed. Features like real time scheduling and pacing settings (other than fixed pacing) are being badly missed
- While Java/Flex Vusers are supported, they are currently supported only with on-premise load generators. So, if you want your load generators to be 100% cloud based, you cannot use these protocols at the moment
- Client side navigation breakdown reports are not available when using whitelisted IPs
- Network analytics cannot be used with JMeter/HAR scripts
- Stormrunner supports only one kind of asset – “Script”. There is no way to separate test data files from the script and make a reference to these files from the script
- Because the load generator machines are instantiated dynamically (and even in the case of whitelisted IPs), you cannot login to your load generator VM . This means, if you want to do a quick iPerf test between your load generator and the application server, you probably cannot.
Network Analytics report sample below.
Navigation Breakdown report sample below.
To summarize, I have drawn down the below table listing the pros and cons of Stormrunner as I see it.
|Virtually unlimited load generation on the fly||Not all protocols supported by performance center are supported|
|Cheaper than HP Performance Center||Network configuration required for every single application to be tested|
|Supports JMeter (HTTP)/Gatling(HTTP)||No way to attach assets other than scripts – java class files, test data files etc.,|
|Rich, interactive interface||Script upload could take up to 3 minutes|
|No maintenance of cloud based load generators||On-premise load generators still required (for java Vusers)|
|Simpler installation process for on-premise load generators (still required)||No FTP for bulk uploads|
|Supports all major protocols used within Markit (Web,Webservices,Truclient,Java)||Test start up takes 15-20 minutes|
|Option to send mails at the end of load tests||No option to schedule tests to start at a later date|
|Option to stop the test and notify testers by mail on SLA violations|
|Better project management features with labels/tags|
|Better ready-made usage tracking|
|25 server Sitescope license bundled with the product for free|
|More concurrent tests than currently possible|
|Load generators at several different geographical locations|
|Upgrade path not tied to ALM|
|Performance center servers can be decommissioned|
|More client side breakdown and network breakdown options (Web only)|
|Supports load generators hosted in AWS|
If you want to try it out, you can get a free 30-day trial of Stormrunner from HPEs website here
What do you think about it? Let me know your comments!