Automated testing of an long running chromium app using Selenium and Chromedriver

Posted on Jan 18, 2019

Recently, I put together a setup to show statistics data on an screen located in my customers office. The setup consists of a Raspberry Pi Zero (Rasbian) running Chromium in full-screen app-mode showing my VueJS statistics app. (Users are not able to interact with it, it just shows data.) I already had it running for multiple days but at some point it just froze and I had no idea what happend. I definity need to know when it crashed and why. These statistics screens should be visible 24/7, so it’s very important for them to be reliable. That’s why I set up a testing suite (as high-level as possible) to find bugs before my customers do. 😀

Requirements for the test-suite:

Test statistics screen on a real machine in a long-running chromium session
Only use web interfaces available to users
Run test every hour
Notify me if test results are negative

Implementation (in short): Run Raspberry Pi Chromium browser in remote debugging mode. Tunnel remote debugging port over to test-runner-machine using ssh. Run minitest test case (Selenium). Use cron to trigger test run every hour. Send telegram notification if test results are negative.

Setup Raspberry Pi Zero

For chromedriver to be able to connect to an already existing chrome instance, the remote debugging port needs to be fixed. Since we need to tunnel that port over to the test-runner machine, it’s way easier to have a fixed one anyway.

# Start the browser (See http://peter.sh/experiments/chromium-command-line-switches/)
chromium-browser \
    --app="$(cat /instance-configuration/browser-url)" \
    --remote-debugging-port=9222

Write down your current chromium-browser version (chromium-browser --version), you’re gonna need it later.

Chromium 65.0.3325.181 Built on Raspbian , running on Raspbian 9.4

Setup test runner machine

Install chromedriver

Find a matching ChromeDriver release (“Supports Chrome vXX-YY”)

curl -O https://chromedriver.storage.googleapis.com/2.45/chromedriver_linux64.zip # replace 2.45 with your selected chromedriver release
unzip chromedriver_linux64.zip
mv ./chromedriver /usr/bin/ && \
  chmod +x /usr/bin/chromedriver && \
  chown root:root /usr/bin/chromedriver

Tunnel chromium remote debugging port from raspberry pi to test runner machine

Chromium only allows connections to it’s remote debugging port from localhost. Thats why it’s necessary to tunnel it.

ssh $RASP_HOST \
    -l pi
    -L 9222:localhost:9222

Note: Keep ssh open for the port forwarding to work

The test suite

I used ruby with minitest to implement the test suite but it should be almost the same with any other toolset supported by selenium. I won’t cover how to create ruby projects with bundler, I just assume you already have one at hand.

Add minitest and selenium-webdriver to your Gemfile.

# My project's Gemfile

source "https://rubygems.org"

git_source(:github) {|repo_name| "https://github.com/#{repo_name}" }

gem "bundler", "~&gt; 1.16"
gem "rake", "~&gt; 10.0"
gem "minitest", "~&gt; 5.0"
gem "selenium-webdriver", "~&gt; 3.141.0"

run bundle install

Now to our test case ….

require "test_helper"
require "selenium-webdriver"

class Live::Report::Hw::Test::SuiteTest < Minitest::Test
  def test_if_statistics_counter_increase
    # ONLY ONE DRIVER CAN BE ACTIVE AT A TIME (selenium limitation)

    active_driver = start_remote_driver 
    current_hit_counter = StatisticsPage.new(active_driver).hit_count
    active_driver.quit()

    active_driver = start_local_driver
    active_driver.get(action_page_URL)
    action_page = ActionPage.new(active_driver).send_hit!
    active_driver.quit()

    active_driver = start_remote_driver
    statistics_page = StatisticsPage.new(active_driver)
    Selenium::WebDriver::Wait.new(timeout: 10).until { statistics_page.hit_count != current_hit_counter }
    new_hit_counter = statistics_page.hit_count

    assert_equal new_hit_counter, current_hit_counter + 1
  ensure
    active_driver.quit() if active_driver
  end

  private:

  # start your own local chrome instance
  def start_local_driver
    options = Selenium::WebDriver::Chrome::Options.new
    options.add_argument("headless")
    options.add_argument('window-size=1920,1080')

    Selenium::WebDriver.for :chrome, options: options
  end

    # access rasp-pi's long running chrome instance
  def start_remote_driver
    options = Selenium::WebDriver::Chrome::Options.new
    options.add_option("debuggerAddress", "localhost:9222") # access rasppi's tunneled remote chrome debugging port

    Selenium::WebDriver.for :chrome, options: options
  end


  # Page object for easier access to statistics screen
  class StatisticsPage 
    attr_reader :driver
    def initialize(driver) 
      @driver = driver
    end

    def hit_count
      @driver.find_element(:css, ".hit-count").text.to_i
    end
  end

  # Page object for easier access to some action increasing statistics counter
  class ActionPage 
    attr_reader :driver
    def initialize(driver)
      @driver = driver
    end

    def send_hit!
      wait = Selenium::WebDriver::Wait.new(timeout: 10)
      wait.until { driver.find_element(css: ".save-button") }
      driver.find_element(css: ".save-button").click

      Selenium::WebDriver::Wait.new(timeout: 10).until do
        driver.find_element(css: ".bs-content").text.include? "Nochmals erfassen"
      end
    end
  end
end

Ensure that your test case works by running rake test (don’t forget to open the ssh tunnel!)

Final setup

Now we need to put all important steps into a shell script and schedule it.

#!/bin/bash

LR_HOST="10.0.0.214"
LR_PORT="2233"
USER="pi"
TELEGRAM_TOKEN="AAAAAAAAAAAA" # obtained by creating new bot in @BotFather
TELEGRAM_CHAT_ID="11111" # obtained by calling: curl -i -X GET https://api.telegram.org/bot$TELEGRAM_TOKEN/getUpdates

CONTROL_SOCKET="$HOME/.live-report-ssh-control-socket"

# Load RVM into a shell session *as a function*
if [[ -s "$HOME/.rvm/scripts/rvm" ]] ; then

  # First try to load from a user install
  source "$HOME/.rvm/scripts/rvm"

elif [[ -s "/usr/local/rvm/scripts/rvm" ]] ; then

  # Then try to load from a root install
  source "/usr/local/rvm/scripts/rvm"

else

  printf "ERROR: An RVM installation was not found.\n"

fi

# tunnel chrome debug port (use ssh control socket to be able to move ssh into the background and still control it)
ssh $LR_HOST \
        -l $USER \
        -p $LR_PORT \
        -L 9222:localhost:9222 \
        -M \
        -S $CONTROL_SOCKET \
        -fNnT

# change into your test suit's folder
cd ~/Documents/Sauce/live_report_test_suite/live-report-hw-test-suite

# execute test suite
rake test

if [ $? -eq 0 ]; then
        echo OK
else
        echo FAIL

        # send "RunFailed" notification using telegram 
        curl -i -X GET https://api.telegram.org/bot$TELEGRAM_TOKEN/sendMessage\?chat_id=$TELEGRAM_CHAT_ID\&text=RunFailed
fi

# exit tunnel
ssh $LR_HOST \
        -l $USER \
        -p $LR_PORT \
        -S $CONTROL_SOCKET \
        -O exit

Schedule it with cron (crontab -e):

DONE

Note: If your test suite needs a long time to execute or you need to do anything more complex, use Jenkins (or anything similar).