For many years we've relied on Travis to run the tests of our packages. For projects we used Circle CI. Recently we moved this responsibility from Travis and Circle CI to GitHub actions. In this blogpost I'd like to explain why and how we did this.

Why we moved to GitHub actions

There are three reasons why we made the move: costs, uniformity, and speed.

As mentioned in the intro, we used two services to run our test suites: Travis for open source and Circle CI for private client projects. When we started out creating open-source, Travis was (and still is I think) the de facto standard. It has a sensible configuration format with support for matrixes, and it's free for open source. For client projects, we picked Circle CI because it, like Travis, has an excellent way of configuring projects, but it's way cheaper than Travis for the tier that we needed.

We have already been using GitHub for source control, for both open-source and our private projects, for several years. For open-source projects, GitHub actions are free. With our Team subscription, we also get 10 000 action minutes per month. Because we only need a fraction of that action time, using GitHub actions for our projects is basically free.

By using the same system for our open source and private projects, we don't have to master two kinds of configuration anymore (Travis & Circle CI).

I don't have any benchmarks on this, but GitHub actions seem to be way faster then Travis to set up an environment and to complete a run of the test suite. Getting feedback more quickly is certainly much nicer.

What's also pretty cool is the fact that there are already many useful GitHub actions that are open-sourced, making it easy to build workflows.

Our GitHub workflow to run the tests

Ignition is the default error page for Laravel applications. We want to be absolutely sure that it works for everyone. That's why we wrote an extensive test suite of both PHP and JavaScript tests that we run on each combination of PHP and Laravel that we support. Since a couple of days ago, those tests are being run on via GitHub actions. This enabled us to also run the test suite on multiple OS'es: Ubuntu and Windows.

To get started, a .github/workflows directory must be created in a repo. Inside that directory, your GitHub workflow definitions can be put.

Here's the content of the configuration file we currently use for Ignition.

name: Run tests

on:
  push:
  schedule:
      - cron: '0 0 * * *'

jobs:
    php-tests:
        runs-on: ${{ matrix.os }}

        strategy:
            matrix:
                php: [7.4, 7.3, 7.2]
                laravel: [6.*, 5.8.*, 5.7.*, 5.6.*, 5.5.*]
                dependency-version: [prefer-lowest, prefer-stable]
                os: [ubuntu-latest, windows-latest]
                include:
                    - laravel: 6.*
                      testbench: 4.*
                    - laravel: 5.8.*
                      testbench: 3.8.*
                    - laravel: 5.7.*
                      testbench: 3.7.*
                    - laravel: 5.6.*
                      testbench: 3.6.*
                    - laravel: 5.5.*
                      testbench: 3.5.*
                exclude:
                    - laravel: 5.7.*
                      php: 7.4
                    - laravel: 5.6.*
                      php: 7.4
                    - laravel: 5.5.*
                      php: 7.4

        name: P${{ matrix.php }} - L${{ matrix.laravel }} - ${{ matrix.dependency-version }} - ${{ matrix.os }}

        steps:
            - name: Checkout code
              uses: actions/checkout@v1

            - name: Setup PHP
              uses: shivammathur/setup-php@v1
              with:
                  php-version: ${{ matrix.php }}
                  extensions: dom, curl, libxml, mbstring, zip, pcntl, pdo, sqlite, pdo_sqlite, bcmath, soap, intl, gd, exif, iconv, imagick
                  coverage: none

            - name: Install dependencies
              run: |
                  composer require "laravel/framework:${{ matrix.laravel }}" "orchestra/testbench:${{ matrix.testbench }}" --no-interaction --no-update
                  composer update --${{ matrix.dependency-version }} --prefer-dist --no-interaction --no-suggest

            - name: Execute tests
              run: vendor/bin/phpunit

            - name: Send Slack notification
              uses: 8398a7/action-slack@v2
              if: failure()
              with:
                  status: ${{ job.status }}
                  author_name: ${{ github.actor }}
              env:
                SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK }}
                GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

    js-tests:
        runs-on: ubuntu-latest

        name: JavaScript tests

        steps:
            -   name: Checkout code
                uses: actions/checkout@v1

            -   name: Install dependencies
                run: yarn install --non-interactive

            -   name: Execute tests
                run: yarn run jest
                
            - name: Send Slack notification
              uses: 8398a7/action-slack@v2
              if: failure()
              with:
                  status: ${{ job.status }}
                  author_name: ${{ github.actor }}
              env:
                SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK }}
                GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

When to run the tests

Let's break it down. On the top level, we start by specifying a name for this job.

name: Run tests

This name will be displayed in the list of executed actions.

Next up, we'll specify when this job should be run.

on:
  push:
  schedule:
      - cron: '0 0 * * *'

Our test suite is run on two events. First, when code is pushed: this includes commits on the master branch as well as submitted pull requests. Secondly, we are going to run the test suite every day at midnight. We do this because, even when our code did not change, the code of our dependencies (so Laravel itself) might have changed, and we want to be sure Ignition works for newer versions of the dependencies as well.

There are some more types of events when actions can be performed. Head to the relevant section in GitHub action docs to learn more.

Determining the environment

After the on part, we start defining the actual jobs that need to run.

jobs:
    php-tests:
    
    ...
    
    js-tests:
    
    ...

Let's start breaking down the php-tests job. First, we're going to specify on which environment the test should run.

runs-on: ${{ matrix.os }}

strategy:
    matrix:
        php: [7.4, 7.3, 7.2]
        laravel: [6.*, 5.8.*, 5.7.*, 5.6.*, 5.5.*]
        dependency-version: [prefer-lowest, prefer-stable]
        os: [ubuntu-latest, windows-latest]
        include:
            - laravel: 6.*
              testbench: 4.*
            - laravel: 5.8.*
              testbench: 3.8.*
            - laravel: 5.7.*
              testbench: 3.7.*
            - laravel: 5.6.*
              testbench: 3.6.*
            - laravel: 5.5.*
              testbench: 3.5.*
        exclude:
            - laravel: 5.7.*
              php: 7.4
            - laravel: 5.6.*
              php: 7.4
            - laravel: 5.5.*
              php: 7.4

name: P${{ matrix.php }} - L${{ matrix.laravel }} - ${{ matrix.dependency-version }} - ${{ matrix.os }}

Let's discuss that matrix first. Every version of PHP, Laravel, and OS we want to run our test on is specified. When a matrix is present, GitHub will create a job per combination of every item in the matrix. All those jobs will run concurrently.

It's important to realize that those entries in the matrix (php, laravel, dependency-version and os) are merely the names of variables that GitHub populates with one of the values given. Those variables can be used elsewhere in the job definition. You see a first example of this in the runs-on entry.

runs-on: ${{ matrix.os }}

Here matrix.os will contain either ubuntu-latest or windows-latest.

In the include section, we define an additional variable called testbench of which the value depends on the value of the laravel variable. When laravel contains 6.*, we'll set the value of testbench to 4.*. I'll explain why we do this, a little biter later, when we are going to use that value.

At the moment of writing, some Laravel versions do not run well on PHP 7.4. In the exclude section, we specify that combinations that we don't want to run our tests for.

By default a matrix will have the fail-fast set to true. This means that whenever a job fails, all the other ones will be canceled.

Running the PHP tests

With our matrix set up, we can now start preparing to run the actual tests. First, we are going to check out to code to our virtual test machine.

name: Checkout code
  uses: actions/checkout@v1

You can think of this as performing a git clone on the machine that will run the tests.

Next, we are going to set up PHP.

name: Setup PHP
  uses: shivammathur/setup-php@v1
  with:
      php-version: ${{ matrix.php }}
      extensions: dom, curl, libxml, mbstring, zip, pcntl, pdo, sqlite, pdo_sqlite, bcmath, soap, intl, gd, exif, iconv, imagick
      coverage: none

For the step above, we use a 3rd party action shivammathur/setup-php@v1. This action is already becoming quite popular. I see it being used in a lot of PHP focused GitHub actions. It can be passed the PHP version you want to use. In our case, we are going to use the PHP version that is currently in the matrix.php variable.

After that, we are going to install our dependencies via composer.

name: Install dependencies
  run: |
      composer require "laravel/framework:${{ matrix.laravel }}" "orchestra/testbench:${{ matrix.testbench }}" --no-interaction --no-update
      composer update --${{ matrix.dependency-version }} --prefer-dist --no-interaction --no-suggest

Using composer require and the matrix.laravel and matrix.testbench variables we are going to specify which version of Laravel and orchestra/testbench we want to use. Orchestra/testbench is a cool package to allows testing package code as if it was installed in a full Laravel application.

Using composer update, we make sure that, depending on matrix.depency, the current or lower dependencies are being used.

Next up comes what we've all been waiting for: running the actual tests.

name: Execute tests
  run: vendor/bin/phpunit

After running the test, and only if they fail, we'll send a Slack notification via a webhook.

name: Send Slack notification
  uses: 8398a7/action-slack@v2
  if: failure()
  with:
      status: ${{ job.status }}
      author_name: ${{ github.actor }}
  env:
    SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK }}
    GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

That failure() is one of the available job status check functions, and it makes sure that this step is only performed when the job fails.

The URL itself is defining as a secret. The value of a secret can be set at the Secrets section of the repo settings.

This is how the Slack notification looks like:

Running the JavaScript tests

Next to running PHP tests, we also run JavaScript tests.

js-tests:
    runs-on: ubuntu-latest

    name: JavaScript tests

    steps:
        -   name: Checkout code
            uses: actions/checkout@v1

        -   name: Install dependencies
            run: yarn install --non-interactive

        -   name: Execute tests
            run: yarn run jest
            
        - name: Send Slack notification
          uses: 8398a7/action-slack@v2
          if: failure()
          with:
              status: ${{ job.status }}
              author_name: ${{ github.actor }}
          env:
            SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK }}
            GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

Because the js-tests is a separate job. That means it will be executed concurrently with the PHP tests. It will check out the code, install the dependencies via yarn and use jest to run the actual tests. Similar to the PHP tests, we send a notification when they should fail.

Adding a badge

You'll be happy to know that, via the awesome shields.io service, you can add a badge to your readme indicating if the test workflow ran succesfully. Here's how that looks like on our readme.

This is the markdown used for the badge in the screenshot.

![GitHub Workflow Status](https://img.shields.io/github/workflow/status/facade/ignition/run-php-tests?label=Tests)

In closing

My colleague Ruben did a lot of work researching GitHub actions. If you want to know how we use GitHub actions to test a typical private Laravel application, head over to Ruben's blog.

I think GitHub actions are pretty neat, and I'm sure we'll use them for other use cases (fixing code style issues, performing code quality checks, deploying, ...) in the future as well.

For more information on how to work with actions, head to the official documentation on GitHub.