GitLab CI Pipeline for Eleventy

This post details the GitLab CI pipeline used for this blog, which is built with Eleventy. It's based on a collection of GitLab CI templates that have evolved over several years for my published NPM packages with a collection of end-to-end tests used for web applications and a few unique jobs added specifically for Eleventy and Nunjucks templates. It's meant as an illustration of a reasonably comprehensive CI pipeline for an Eleventy static site, maximizing the level of automated testing, leveraging built-in GitLab capabilities where practical, and optimizing parallelization and pipeline speed.

CI Pipeline Philosophy #

The following summarizes the general philosophy that I use for any CI pipeline. Some pieces are Gitlab specific, but most are general platform independent recommendations based on my experience.

• The CI pipeline should execute all checks that are required to confirm that the application is in an acceptable configuration for deployment. This includes checking multiple configurations where applicable (e.g. light/dark mode, desktop/mobile configurations, OS/platform/framework versions).
• CI jobs should be deterministic, with job failure meaning a failure to execute properly or an unacceptable result that should not being deployed. There may be jobs with interpreted results (e.g. Lighthouse performance score changes, dependency vulnerabilities), but passing jobs should mean the results were successfully generated to be interpreted. Jobs that fail, but allow failure, resulting in the pipeline passing are a trap to be avoided. This will eventually lead to not reviewing the job results, assuming the failure cause is known, and the job failing for another reason will go unnoticed.
• Some might argue that CI pipelines should be highly optimized between what runs in a merge request and what runs on the primary branch. In general that is a mistake in GitLab CI. The GitLab merge request widgets are optimized to show changes since in the context of that merge request, e.g. if you run a Code Quality report you only see the changes in that merge request, not all issues. If there is no reference job in the default branch, then the all results will be identified as changes. Unless there are long-running jobs that need to be optimized or other specific drivers, there is benefit to running consistent pipelines.
• The pipeline should leverage built-in GitLab capabilities where appropriate. In this pipeline that includes analyses like code quality and some security related testing. There's no point in re-inventing the wheel unless there's a specific driver.
• The pipeline should be optimized to execute as many jobs as possible in parallel with clearly defined job dependencies, and should minimize repeating the same tasks in favor of performing them once and passing the artifacts as required.
• Jobs with external dependencies (e.g. dependency vulnerability scanning) should be run on a regular schedule, even with no commits to the repository.
• Deployment should be performed via an automated scripted process for consistency (even if manually initiated).

GitLab CI Pipeline for Eleventy #

The complete CI pipeline for this site is shown below, ordered by execution sequence with job dependencies shown.

For efficiency, all jobs are defined with explicit needs, as can be seen above. This ensures that jobs start as early as possible, and that only the artifacts required for the job are downloaded and extracted (GitLab will download the artifacts from all previous jobs unless otherwise specified).

There are 28 jobs in the pipeline, which are grouped into the following functional categories:

• Installing Dependencies
• Language Specific Linting and Formatting
• Linting Tool Configuration Files
• Build Site
• Code Analysis
• End-to-End Testing
• Software Composition Analysis
• Static Application Security Testing
• Site Deployment

Installing Dependencies #

The npm_install job installs all npm dependencies and saves then as artifacts. This allows the install to be performed only once since it is an expensive operation (in time and I/O) and the dependencies are then made available to any subsequent jobs that require them.

Language Specific Linting and Formatting #

Linting and format checking is performed for all languages used in this project. I use a very opinionated linting config for all tools to ensure consistency and maintainability throughout the codebase and to check for common issues. Development is done in a mix of VSCode and GitLab's WebIDE, so linting and formatting is not always checked there. When working VSCode, linting and format checking is performed is as a git pre-commit hook. The one place where linting and format checking is always performed is in CI.

The challenge with the pre-commit hook and CI cases is that the resulting logs are not reviewed unless there is an error. Operating this was for a long time on many projects has driven me to update all linting rule configurations to remove all warnings. Rules are either important enough to be errors or they're not and they're disabled. The errors will then fail the pre-commit hook or CI pipeline. In some cases there are acceptable deviations, but these exceptions are then disabled in the code with the appropriate rationale.

The specific linting/formatting jobs and tools are detailed in the table below. Most of these are probably familiar names, other than maybe djLint. It's a Python tool for linting and formatting Nunjucks, Django, Handlebars, and many others. There's no other tool I've found in that space that comes close.

Linting Tool Configuration Files #

There are a few tools used by this project that offer linting tools to validate their configuration files. These jobs perform those checks to identify any formatting errors.

The lint_pageanrc job checks the Pagean configuration file for validity. See the End-to-End Testing section for more details on Pagean. Formatting errors would also be found when running Pagean, but this provides a job that explicitly identifies a configuration error, and this can be run much earlier in the pipeline to fail fast if there is an issue.

The lint_renovate job checks the Renovate configuration file for validity. Renovate is used by this project for automated dependency updates, is quite powerful/configurable, and is highly recommended. The Renovate process is run in another project, so an error in the Renovate configuration would not otherwise be detected in this pipeline.

Build Site #

The build_test_site job builds the site in a test configuration. This site undergoes exhaustive end-to-end testing, so some production pieces are excluded from the test build (e.g. analytics scripts), and the configuration sets the pathPrefix to match the URLs used for that end-to-end testing.

Code Analysis #

The pipeline includes a couple of general code analysis jobs that don't fall into one of the specific categories listed below.

The code_count job uses cloc to count lines of code by language. This is primarily used to generate a GitLab Metrics report, which can then provide a change in code count by language in a merge request. This is helpful as a check that an unexpected change wasn't accidentally committed (e.g. if a new blog post was added, which should only be a change in lines of markdown, but the lines of JavaScript changed as well, then something is wrong).

The code_quality job run the GitLab Code Quality testing. The results for the entire project can be viewed, as can the changes in a merge request.

End-to-End Testing #

End-to-end testing, which involves analysis of the site in a browser, is performed to assess site performance, accessibility, and other quality control checks (e.g. no console errors, no broken links) as outlined below.

Lighthouse is run against one page to get a representative set of data in both desktop (the lighthouse_desktop job) and mobile (the lighthouse_mobile job) configurations. This data then populates a GitLab Metrics report, which identifies any changes in Lighthouse scores in a merge request. See this post for more details on setting up Lighthouse and generating a metrics report.

Accessibility testing is performed using Pa11y CI with both light mode (the pa11y_ci job) and dark mode (the pa11y_ci_dark job) themes. Pally CI can be configured to run with different accessibility test runners, and these jobs run both the HTML_CodeSniffer and Axe runners. There is also accessibility data provided by Lighthouse, but the Pa11y CI results in general are more comprehensive (especially when using both runners).

The pagean job run Pagean, a test framework I started several years ago with a collection of tests intended to be run against all pages in a site - broken links, errors written to the console, horizontal scrollbar, page load time, and others. See the documentation for complete details.

Software Composition Analysis #

Software Composition Analysis (SCA) involves a set of tools used to manage risks from a project's dependencies. This is a broad category that includes tools to catalog the project dependencies, check those dependencies for known vulnerabilities or other security risks, check for unused or deprecated dependencies, etc.

Two tools are used to check the project's dependencies for known vulnerabilities. The owasp_dependency_check job runs OWASP Dependency Check against the project. This is a tool maintained by the OWASP foundation, and checks not only package manifests like a lockfile, it also scans all files in the project to check for any known vulnerabilities. This allows it to recognize cases where dependencies are embedded and/or not identified, but also means the scan is slower because it has to check every file in the project. It uses data primarily from the U.S. National Vulnerability Database (NVD), populated from the Common Vulnerabilities and Exposures (CVE) database, the de facto international standard for identifying vulnerabilities. It also includes data from other applicable vulnerability databases. The results not only identify any known vulnerabilities, but also provide background on the vulnerabilities and potential remediation steps.

The osv_scanner job runs OSV Scanner to check for vulnerabilities identified in a lockfile or other package manifest. It checks that listing of vulnerabilities against the Open Source Vulnerabilities (OSV) Database, maintained by Google. This includes data from multiple sources, but is focused on open source vulnerabilities. Because it is using a defined listing of dependencies it runs very quickly. For this specific project, it is functionally no different than running npm audit since the GitHub Advisories database is integrated into the OSV Database, but this project leverages CI templates used for a variety of projects in different languages, so this provides a broader solution.

OSV Scanner is currently undergoing testing here to determine if the data is reliable enough, and reported in a timely enough fashion, that it could replace owasp_dependency_check in projects where a listing of dependencies is available. So far that has not been shown to be the case. As this is being written, there is a new vulnerability identified by OWASP Dependency Check over a week ago that is still not reported in a GitHub Advisory, and therefore not by OSV Scanner.

Socket has taken a different approach than many other tools in the area of dependency security to provide a detailed assessment of the code for NPM (and other) packages to identify specific capabilities. For example, it will identify if a package access files on disk, accesses the network, spawns external processes, sends telemetry, and many other checks. These actions are expected in some cases, but this provides the insight to identify cases where it may not be. The socket job run the Socket CLI for any merge requests with dependency changes to assess any policy violations. At this point their GitHub app is more mature, so this is a recent addition still undergoing assessment to determine the best way to use the results in GitLab. It's a tool worth taking a look at because it really does provide the most comprehensive assessment of a package's codebase and potential issues that I've seen.

The lint_lockfile job runs npm-lockfile-lint to check that packages in the lockfile are properly retrieved from acceptable sources. In this case it ensures that packages are only pulled from NPM, over HTTPS, that the integrity field has a SHA512 hash, and that the URL matches the package. This is done to verify the integrity, fidelity, and vulnerability reporting for all dependencies (with all of these settings configurable based on project needs).

The depcheck job runs depcheck to identify any project dependencies that are not used or packages that are used but are not dependencies. It does have some limitations, but will find references for packages in .js/.ts files, in scripts running packages bin commands, or in the configuration files of many major utilities (e.g. eslint, jest, mocha, prettier).

The npm_check job run npm-check to identify and packages with updates. While Renovate is used for automated updates, this is a backup to identify cases where the Renovate job is failing to make updates.

Static Application Security Testing #

Static Application Security Testing (SAST) tools analyze source code to find coding patterns that represent potential security issues. These analyses all focus on this project's code, not any project dependencies. Several SAST analysis tools are used, as outlined below.

The GitLab SAST template includes a series of language-specific SAST analysis tools, with rules configured to run those applicable to your codebase. In this, with a Node.js/JavaScript project that includes the nodejs-scan-sast and semgrep-sast jobs.

The secret_detection job checks for any secrets (e.g. passwords, access tokens/keys) committed to the git repository. This job is included from the GitLab Security Detection template.

The unicode_bidi_test job runs anti-trojan-source to identify any bidirectional unicode characters, which could be an indication of a Trojan Source Attack.

Site Deployment #

As noted above, the build_test_site job builds the site in a test configuration, so it must be rebuilt with the appropriate settings for deployment.

The pages job runs only for commits to the default branch and builds the site in the production configuration and deploys it to GitLab Pages.

GitLab provides the capability to create a review app with an instance of an application with the changes in a merge request. Unfortunately, this capability is not natively available for GitLab pages. To accomplish this, the pages_review_app jobs run on merge request pipelines to leverage some alternate GitLab capabilities to accomplish the same result. See this post for details on how to set up a GitLab review app for an Eleventy site.