Selenium Grid Performance Testing
Using a selenium grid for performance testing gives you the advantage of being able to easily and quickly implement performance tests into your continuous testing environment re-purposing familiar selenium tests as performance tests.
You can then setup a special performance testing job in your CI environment that you run together with your functional test suite or separately only when the need arises. Using a Gridlastic standard plan like the Performance plan gives you up to 200 node machines in your selenium grid that can be used for performance testing during any length of time.
Using a Gridlastic selenium grid for performance testing is not only easy but very cost efficient and if your target web server is also located in Amazon EC2 and you have your Gridlastic selenium grid in the same data region, the latency between your grid nodes and website under test will be minimal and the load greatly increased compared to test connections going via the internet.
Having a selenium grid in the same data center as your website under test will not only enable you to put greater stress on your server app but will also provide a more consistent performance testing environment, free from the ups and down of internet network traffic, which is crucial in order to detect a real performance issue benchmarking previous deployed code.
Capture HAR type performance data using Browsermob Proxy and integrate performance data tests into your test suites. See examples how to use Browsermob Proxy with Selenium Grid.
Test scenario using selenium grid for performance testing
An example test scenario using a Gridlastic selenium grid for performance testing to simulate 1000-2000 users (1 vm node is equivalent of about 5-10 real users):
Trigger a performance testing job either manually or automatically upon a code change, the job executes the Gridlastic pre-launch API requesting 200 nodes to launch if not already registered to the selenium grid hub to minimize wait time before there are nodes to process tests. During the 1-3 minutes it takes to launch the selenium grid nodes the job continues with unit and integration tests, launch/prepare a test server, deploy code to test server, then executes the selenium performance tests.
If your testing is done the 200 nodes will be automatically terminated by the Gridlastic auto scaling functionality.
The Gridlastic credits usage for this example test run lasting less than 60 minutes using m1.small windows instance nodes in the Virginia region is only 10.5 x 200 = 2100 credits for on-demand and only 4.7 x 200 = 940 credits if using spot instances at 4.7 credits per hour. The Performance Plan includes 30,000 credits per month.
Performance testing not a one time event
Performance testing is not just a one time event but should be a part of your normal continued integration efforts. To make the overall testing process more agile it is important to identify possible performance issues earlier in development just as it is important to have functional tests securing operational functionality and find out any issues as soon after a code change as possible.
Real browsers vs Virtual Browsers
In the past virtual browsers have been favored for performance testing mainly because of the high cost of using real browsers. This is no longer the case as cloud computing has made testing with real browsers not only affordable but in many cases preferred specially when testing modern web 2.0 applications which could contain hundreds of connections and resources that is interacted with before a page is rendered.
If virtual browsers are used you must map all these connections and interactions and then try to simulate a real browser correctly, not always an easy task and in many cases time consuming and always there is the question if a virtual browser can simulate modern web browsers properly.
Using a virtual browser also requires a lot more maintenance as a modern web 2.0 implementation of example a shopping cart purchase or a login operation can consist of hundreds of connections and resource handling and if just one of them changes the test case have to be updated.
If you instead use a real browser test you can use a familiar selenium test case that tests the performance of the operation from a top level user point of view which does not care about any underlying functionality changes and you can keep the same test unchanged for a longer time.
This will in turn give you a performance benchmark for the operation and you can specify a normal selenium test case with acceptable performance parameters like max execution time. You then have the flexibility to run your performance tests at the same time as your functional tests or separately re-using the same test machine nodes used for functional testing, saving time and reducing cost.