Selenium Grid Performance Testing

Using a selenium grid for performance testing gives you the advantage of being able to easily and quickly implement performance tests into your continuous testing environment re-purposing familiar selenium tests as performance tests. You can then setup a special performance testing job in your CI environment that you run together with your functional test suite or separately only when the need arises.

Having a selenium grid in the same data center as your website under test will also provide a more consistent performance testing environment, free from the ups and down of internet network traffic, which is crucial in order to detect a real performance issue benchmarking previous deployed code.

Performance testing is not just a one time event but should be a part of your normal continued integration efforts. To make the overall testing process more agile it is important to identify possible performance issues earlier in development just as it is important to have functional tests securing operational functionality and find out any issues as soon after a code change as possible.

Test scenario using selenium grid for performance testing

Trigger a performance testing job either manually or automatically upon a code change, the job executes the Gridlastic pre-launch API requesting 200 nodes to launch if not already registered to the selenium grid hub to minimize wait time before there are nodes to process tests. During the 1-2 minutes (Linux grid nodes) it takes to launch the selenium grid nodes the job continues with maybe unit and integration tests, launch/prepare a test server, deploy code to test server, then executes the selenium performance tests.

If your testing is done the 200 nodes will be automatically terminated by the Gridlastic auto scaling functionality. The Gridlastic credits usage for this example test run lasting less than 15 minutes using c4.large linux nodes in the Virginia region is only 0.25*13 x 200 = 650 credits for on-demand nodes and only 0.25*6.1 x 200 = 305 credits if using spot nodes at 6.1 credits per hour. The Performance Plan includes 30,000 credits per month so if you use spot nodes which we highly recommend, you can run 100 load tests a month with this configuration, see our pricing.

Real browsers vs Virtual Browsers

In the past virtual browsers have been favored for performance testing mainly because of the high cost of using real browsers. This is no longer the case as cloud computing has made testing with real browsers not only affordable but in many cases preferred specially when testing modern web 2.0 applications which could contain hundreds of connections and resources that is interacted with before a page is rendered.

If virtual browsers are used you must map all these connections and interactions and then try to simulate a real browser correctly, not always an easy task and in many cases time consuming and always there is the question if a virtual browser can simulate modern web browsers properly.

Using a virtual browser also requires a lot more maintenance as a modern web 2.0 implementation of example a shopping cart purchase or a login operation can consist of hundreds of connections and resource handling and if just one of them changes the test case have to be updated.

If you instead use a real browser test you can use a familiar selenium test case that tests the performance of the operation from a top level user point of view which does not care about any underlying functionality changes and you can keep the same test unchanged for a longer time.

This will in turn give you a performance benchmark for the operation and you can specify a normal selenium test case with acceptable performance parameters like max execution time. You then have the flexibility to run your performance tests at the same time as your functional tests or separately re-using the same test machine nodes used for functional testing, saving time and reducing cost.