GRIDLASTIC CONNECT

Gridlastic Connect is a secure, high performance enrypted tunneling software that enables you to test behind firewall web applications or access your selenium grid hub via an encrypted connection. Starting a tunnel takes just a few seconds and data transmitted is encrypted through industry-standard AES-256.

Starting a tunnel does not affect your selenium grid nodes capability to access any server on the internet at the same time as using one or several tunnels. This way you can mix and match your test traffic via your selenium test script.

PERFORMANCE

Gridlastic Connect can handle 200+ of simultaneous browser sessions over a single tunnel requiring only a small 4 GB RAM 2 GHz (linux) client machine on your side. We recommend using a Linux client machine for high performance but you can also use windows.


FLOW CHART - Gridlastic Connect Tunnel Client (GRIDC)

Tunneling clients can be installed in multiple networks, on local machines, CI servers or web servers, there is no limit of how many tunnel clients you connect. You could also use a dedicated machine for the tunnel client or scale with several client machines depending on your testing needs.

In your tunnel client config file you create a mapping to your internal web servers using ip or dns names. This will create endpoints that your selenium grid nodes can access. You can also make a single mapping to your own internal proxy, we recommend the Squid proxy if you have none of your own already available. You can then skip the mapping to specific web servers and use your own ip/dns names in your selenium test scripts.

The tunnel client can also be configured to handle traffic to your selenium grid hub server securely via an encrypted tunnel (blue lines) without accessing your selenium grid hub via http.

Click charts to enlarge.

Squid proxy integration.
squid proxy integration


DOWNLOAD TUNNEL CLIENT

Download Link SHA1 Checksum
Linux 32-bit - v1.1 24d2d7fb861765634ed967dbf663bf7e9f8384f0
Linux 64-bit - v1.1 bba319d2f92336d1d93eaa49a637d9d97f2b48c5
Windows 32-bit - v1.1 46b58af8938d5906541fde4ba929ff6475e3060e
Windows 64-bit - v1.1 2ad344420fbf670c3d84d757c5ff227fe32fb816


INSTALLATION

1. Gridlastic Connect is disabled by default. You enable Gridlastic Connect in your Gridlastic dashboard grid configuration by specifying source IP from where you will connect with the tunnel client, "Enable Gridlastic Connect (Port 443)".

2. Download and unzip your tunnel client like

tar xfvz gridc-1.1-linux_64.tar.gz
3. After extracting, go to the install directory and modify the sample configuration file config.cfg with your Gridlastic subdomain and a mapping of your test environment, see below example configuration file and instructions. NOTE: If you do not want to map your test servers or are unable to specify a direct ip to your internal test site or have problems getting to your internal sites because of re-directs/re-writes/dns/websockets or other resource loading issues, use a proxy integration with Squid Proxy. Also, consider using a simpler integration approach to using an tunnel by using selenium grid proxy routing.

4. Start the tunnel service using your Gridlastic selenium grid hub credentials displayed in your dashboard. NOTE: Make sure your Gridlastic selenium grid is up and running and that Gridlastic Connect is enabled.

./gridc -username=HUB_USERNAME --password=HUB_PASSWORD -config=config.cfg start tunnel1
WINDOWS EXAMPLE:
"C:\gridc\gridc-1.1-windows_64\gridc.exe" -username=HUB_USERNAME --password=HUB_PASSWORD -config "C:\gridc\gridc-1.1-windows_64\config.cfg" start tunnel1
When you run Gridlastic Connect, it will display a UI in your terminal with the current status of the tunnel. NOTE: for windows, if the mapping cannot be seen properly because of too small of cmd window, change it like: Right-click on the prompt boarder and select Properties, then select the Layout tab and change the Window Size width, by default it is 80.


EXAMPLE CONFIGURATION FILE

(config.cfg, is a YAML file so be careful with spaces when editing. Replace "your_subdomain" with your Gridlastic selenium grid subdomain.)
server_addr: your_subdomain.gridlastic.com:443
tunnels:
  QA1-LOCAL-HTTP:
    subdomain: "qa-1-http"
    proto:
      http: 80
  QA1-LOCAL-HTTPS:
    remote_port: 7000
    proto:
      tcp: 443
  QA2-REMOTE-HTTP:
    subdomain: "qa2.test"
    proto:
      http: 10.0.0.83:80
  QA2-REMOTE-HTTPS:
    remote_port: 7001
    proto:
      tcp: 10.0.0.83:443
If this tunnel client is started using the "start" command that specifies what tunnels in the configuration file that will be created, like

./gridc -username=HUB_USERNAME --password=HUB_PASSWORD -config=config.cfg start QA1-LOCAL-HTTP QA1-LOCAL-HTTPS QA2-REMOTE-HTTP QA2-REMOTE-HTTP
on a web server with the above configuration file the output would be:

gridc

Tunnel Status                 online
Version                       1.0
Forwarding                    http://qa-1-http.your_subdomain.gridlastic.com:8080 -> 127.0.0.1:80
Forwarding                    tcp://your_subdomain.gridlastic.com:7000 -> 127.0.0.1:443
Forwarding                    http://qa2.test.your_subdomain.gridlastic.com:8080 -> 10.0.0.83:80
Forwarding                    tcp://your_subdomain.gridlastic.com:7001 -> 10.0.0.83:443

Now you can test local machine web apps (http and https) but this tunnel client will also handle traffic to a remote web server 10.0.0.83 (http and https). Example from the mapping above, in your selenium test scripts you use https://your_subdomain.gridlastic.com:7000, to access a local machine site on port 443. We recommend you use a dedicated tunnel client machine with mappings to your target servers to avoid competing for resources with a web app under test.


ACCESS TO ENDPOINTS

Each customer have their own Gridlastic Connect server (your_subdomain.gridlastic.com) which runs securely in your Selenium Grid VPC (Amazon EC2 Virtual Private Cloud) and only your selenium grid nodes have access to the above endpoints. Use these endpoints in your selenium test scripts, for https testing simply use https://your_subdomain.gridlastic.com:remote_port. If you do not define a remote_port from the interval 7000-8000, one is automatically assigned randomly from this interval upon client start. Http subdomains are accessed on port 8080.

NOTE: If you are unable to specify a direct ip to your internal test site or have problems getting to your internal sites because of re-directs/re-writes/dns or other resource loading issues, use a proxy integration with Squid Proxy.


TUNNEL CLIENT ACCESS

On what ever machine you install a tunnel client, that machine of course needs to have access to the mapped target servers or access to a proxy with access to the target servers. The target servers do not need to be operational when the tunnel client is started, tunnels are created independently and are active as long as your tunnel client(s) and your Gridlastic Selenium Grid is running.


SCALE

If you have several tunnel clients you need to coordinate subdomains and ports mapped to avoid conflicts, you will get an error upon client start if there is one. Several tunnel clients can have a mapping for the same web server target using several different domains/ports. This is how you can scale with several client machines/tunnels accessing the same web service. You would then need to control how many browser sessions uses what tunnel endpoint in your selenium test script code. If you plan to run more than 10 sessions in parallel then you might need to increase the max open file limit on your system.


CONFIGURE THE TUNNEL CLIENT

Editing the tunnel client config.cfg file requires a stop/start of the tunnel client and the active tunnels but takes just a few seconds to start up again. Note: you can have several tunnel clients running on the same machine so you do not need to map your entire test environment in a single configuration file.


STOP THE CLIENT

Stop the tunnel client with “ctrl -c”.


USE A CORPORATE PROXY TO REACH GRIDLASTIC.COM

If the machine your are running the tunnel client on cannot directly access the internet and your grid hub server, you can use an internal corporate proxy by adding the "http_proxy" parameter to the tunnel client config.cfg file as shown in the example below:

server_addr: your_subdomain.gridlastic.com:443
http_proxy: "http://user:password@10.0.0.1:3128"
tunnels:
  tunnel1:
    subdomain: "test"
    proto:
      http: 80


ACCESS YOUR SELENIUM GRID HUB SERVER

The tunnel client can also be configured to handle traffic to your selenium grid hub server securely via an encrypted tunnel on any port you choose. Activate this routing with an extra command line parameter “-hub” like:

./gridc -username=HUB_USERNAME --password=HUB_PASSWORD -config=config.cfg -hub 3000 start tunnel1
Your tunnel client machine will now route traffic on gridc_client_machine_IP:3000 or localhost:3000 to your selenium grid hub. This routing is disabled by default. In your selenium test code you would specify

"http://HUB_USERNAME:HUB_ACCESS_KEY@gridc_client_machine_IP:3000/wd/hub"
or if your test code runs locally like on a CI server

"http://HUB_USERNAME:HUB_ACCESS_KEY@localhost:3000/wd/hub”
or if you pass in the HUB_USERNAME and HUB_ACCESS_KEY credentials via capabilities like with this Karma code example
  ...
  var webdriverConfig = {
	hostname: 'localhost',
	port: 3000,
  }
  ...
instead of

"http://HUB_USERNAME:HUB_ACCESS_KEY@YOUR_SUBDOMAIN.gridlastic.com:80/wd/hub"
accessing your selenium grid hub via http.

The hub routing can also run standalone like
./gridc -username=HUB_USERNAME --password=HUB_PASSWORD -config=config.cfg -hub 3000


RUN THE TUNNEL CLIENT LONG TERM

If you terminate your Gridlastic selenium grid with tunnel clients running, they will keep running and will connect again automatically as soon as your selenium grid is started again.

You might want to run the client without the UI window like

LINUX
./gridc -log=stdout -username=HUB_USERNAME --password=HUB_PASSWORD -config=config.cfg start tunnel1 > /dev/null & disown
Note: to run the client in the background "-log=stdout" is required.


DYNAMIC TUNNEL CREATION

If you launch a test server with a randomly assigned ip you can also dynamically start a tunnel client to access that server. Since you can run several tunnel clients on a single machine, like on a CI server, you can regardless of already created tunnels create a new tunnel client from your code like

./gridc -log=stdout -proto http -subdomain=test -username=HUB_USERNAME --password=HUB_PASSWORD -config=config.cfg 10.0.0.83:80 > /dev/null &
which maps a remote http connection to 10.0.0.83 on port 80. In this context the command line parameter "-proto" can have a value of "http" only. If "-subdomain" value is omitted a random name will be created. You need to specify a config file in all these calls containing your Gridlastic connect server like

server_addr: your_subdomain.gridlastic.com:443
but you can use the same config file for all calls if you like. You must clean up these clients at a regular intervall to prevent too many client's running causing performance issues.


SQUID PROXY INTEGRATION

Squid proxy is one of the most popular open source proxies available today. Gridlastic Connect can be used to tunnel your selenium grid nodes through your own Squid proxy or through any other proxy of your choosing like Browsermob proxy with http and https. The main advantage is that you do not need to map your internal test servers in a config file but just map to a proxy and then use whatever internal ip/dns etc in your selenium test script.

Squid proxy also supports web sockets and can be installed on many different operating systems. If you are unable to specify a direct ip to your internal test site or have problems getting to your internal sites because of re-directs/re-writes/websockets or other resource loading issues, using a proxy setup will solve these problems.

Click chart to enlarge.
squid proxy integration


Sample Config file (Squid proxy running on the same box as the tunnel client, recommended):
server_addr: your_subdomain.gridlastic.com:443
tunnels:
  squid1:
    remote_port: 7001
    proto:
      tcp: 3128
Start the tunnel client like
./gridc -username=HUB_USERNAME --password=HUB_PASSWORD -config=config.cfg start squid1
the output would be:

gridc

Tunnel Status                 online
Version                       1.1
Forwarding                    tcp://your_subdomain.gridlastic.com:7001 -> 127.0.0.1:3128


The endpoint "your_subdomain.gridlastic.com:7001" is accessible only by your Gridlastic selenium grid nodes. To tunnel your tests through to your Squid proxy, create a selenium proxy using this endpoint and pass into the webdriver as a capability like in the example below:

String proxy_server = "your_subdomain.gridlastic.com:7001";
org.openqa.selenium.Proxy proxy = new org.openqa.selenium.Proxy();
proxy.setHttpProxy(proxy_server).setFtpProxy(proxy_server).setSslProxy(proxy_server);
capabilities.setCapability(CapabilityType.PROXY, proxy)
See more selenium proxy language examples here

You can now use your internal ip/dns server names in your selenium test scripts just like you would running locally.

Squid is easy to install, see here for Squid proxy installation instructions.


SSL CERTIFICATES

If testing a https site Firefox and Chrome usually automatically accepts any ssl cert prompts by design but for internet explorer this does not always work and you might have to include an additional method to get pass the cert prompt like:


((RemoteWebDriver) driver()).findElementByLinkText("Continue to this website (not recommended).").click();
OR
driver().findElement(By.name("overridelink")).sendKeys(Keys.ENTER);
If you want to use your own ssl certificates you can by requesting them to be installed on a custom node that only your account has access to. Request a custom node by contacting support@gridlastic.com.


MAX OPEN FILE LIMIT

You might have to increase the max open file limit on your machine if you run more than 10+ sessions in parallel like:

LINUX
ulimit -n 65536
and then re-start both your proxy like Squid and your Gridlastic Connect client(s) (gridc). Verify that the "Max open files" "Soft Limit" has adopted your new max limit like:
// Replace [PID] with values from your proxy and gridc processes. 
cat /proc/[PID]/limits
A good practice is also to regularly re-boot your client server to free up resources.

WINDOWS
On Windows these limits are hard coded but you can scale using more windows machines, each using a Gridlastic Connect client but mapped to a different port. This way you would have a tunnel to each machine and use a parameter for the endpoint to even out your tests per endpoint. See here for a java example how to use parameters in your selenium test script.


TROUBLESHOOTING

*If you are having problems connecting to internal sites that are using re-directs etc, use a proxy setup with Squid Proxy like described here.

*If your connect client is stuck on "reconnecting" that means your selenium grid us not up and running or your hub username and password credentials could be incorrectly specified. Also check the config file is valid YAML format. Also make sure you have added a source IP for your tunnel client in your grid configuration "Enable Gridlastic Connect (Port 443)".

*If you get test timeouts using a tunnel with more than 10 parallel sessions, then you might need to increase the max open file limit on your system.


TIPS

Although you could run all your test traffic through Gridlastic Connect tunnels we still recommend to access your public test servers directly to minimize tunnel traffic when practical. Starting a tunnel does not effect your selenium grid nodes capability to access any server on the internet.