Aaron Goldenthal

The Year in Review 2025

2026-01-08T00:00:00Z

As usual, it's been another busy year, focused mostly of sustaining of my existing projects and setting a new record with almost 500 software releases. I've continued development of new, and adoption of some existing, tools and techniques to improve productivity and software quality. In addition, I joined the Pa11y team where work has ramped up again resulting in several needed, major releases.

Blog posts #

I didn't start this year with any specific goals for blog posts because I was so busy with other projects that I only managed two posts. They're related to subsequent topics in this post and are linked there.

Software projects #

This year was busy with software development rather than writing, with a handful of new items to note.

I joined the Pa11y team #

Last year, with some uncertainty in support for the Pa11y projects, I had released forks of both pa11y and pa11y-ci as interim fixes until the long-term plans of the team were better understood. In 2025, I, and several others, joined the Pa11y team and there has been a resurgence in development. This led to two major releases (and a few minor/patch releases as well):

The release of pa11y@9.0.0, which was the first major release in over a year.
The release of pa11y-ci@4.0.0. This was the first major release upgrading from pa11y@6.2.3 (released in 2022) to pa11y@9.0.0.

Work is ongoing, so expect continued support and enhancements this year. A significant upgrade to the axe runner, improving the treatment of issues requiring manual review (see PR #685), is close to release.

My open source software projects #

This year was full of a lot of sustaining work on existing projects, with a few new and noteworthy items.

In 2025 I started four new projects, and between those projects and my other existing projects shipped 479 releases (63 major, 131 minor, 276 patch), which is a new record. Even though many of those were routine sustaining updates (refactoring, security patches, dependency updates, etc.), it reinforces that continuous delivery practices and extensive automation do improve productivity and the ability to confidently release software.

Full list of 2025 new projects

Project	Description
Container Images / Kaniko	A project to build container images for Chainguard's `kaniko` fork.
Container Images / PSScriptAnalyzer	A container image to run lint and SAST analyses on PowerShell scripts with PSScriptAnalyzer.
Tests / Container Multi-Platform Test	A multi-platform container project to test CI jobs.
Tests / Node Web Test	A Node.js project to test CI pipeline jobs for web pages/applications.

Full list of 2025 software releases

Project	Releases
Container Images / Cloc	9 (2.0.1, 2.0.0, 1.6.2, 1.6.1, 1.6.0, 1.5.0, 1.4.1, 1.4.0, 1.3.1)
Container Images / Go Test	24 (3.0.1, 3.0.0, 2.9.9, 2.9.8, 2.9.7, 2.9.6, 2.9.5, 2.9.4, 2.9.3, 2.9.2, 2.9.1, 2.9.0, 2.8.3, 2.8.2, 2.8.1, 2.8.0, 2.7.3, 2.7.2, 2.7.1, 2.7.0, 2.6.1, 2.6.0, 2.5.8, 2.5.7)
Container Images / Image Tools	24 (3.0.3, 3.0.2, 3.0.1, 3.0.0, 2.2.0, 2.1.1, 2.1.0, 2.0.0, 1.8.2, 1.8.1, 1.8.0, 1.7.3, 1.7.2, 1.7.1, 1.7.0, 1.6.0, 1.5.1, 1.5.0, 1.4.1, 1.4.0, 1.3.0, 1.2.4, 1.2.3, 1.2.2)
Container Images / Kaniko	10 (1.0.9, 1.0.8, 1.0.7, 1.0.6, 1.0.5, 1.0.4, 1.0.3, 1.0.2, 1.0.1, 1.0.0)
Container Images / PSScriptAnalyzer	4 (1.2.0, 1.1.0, 1.0.1, 1.0.0)
Container Images / Puppeteer	9 (16.0.1, 16.0.0, 15.2.0, 15.1.2, 15.1.1, 15.1.0, 15.0.0, 14.2.0, 14.1.0)
Container Images / Syft	9 (2.0.0, 1.3.4, 1.3.3, 1.3.2, 1.3.1, 1.3.0, 1.2.8, 1.2.7, 1.2.6)
Container Images / Vale	29 (3.8.2, 3.8.1, 3.8.0, 3.7.10, 3.7.9, 3.7.8, 3.7.7, 3.7.6, 3.7.5, 3.7.4, 3.7.3, 3.7.2, 3.7.1, 3.7.0, 3.6.1, 3.6.0, 3.5.5, 3.5.4, 3.5.3, 3.5.2, 3.5.1, 3.5.0, 3.4.1, 3.4.0, 3.3.6, 3.3.5, 3.3.4, 3.3.3, 3.3.2)
Node / Stylelint Config Standard	3 (21.0.1, 21.0.0, 20.0.0)
References / GitLab Group Project Summary	5 (2.0.0, 1.1.3, 1.1.2, 1.1.1, 1.1.0)
Tests / Container Multi-Platform Test	4 (1.1.1, 1.1.0, 1.0.1, 1.0.0)
Tests / Container Test	9 (1.2.1, 1.2.0, 1.1.4, 1.1.3, 1.1.2, 1.1.1, 1.1.0, 1.0.6, 1.0.5)
Tests / Node CJS Test	5 (3.0.3, 3.0.2, 3.0.1, 3.0.0, 2.1.0)
Tests / Node ESM Test	8 (2.0.5, 2.0.4, 2.0.3, 2.0.2, 2.0.1, 2.0.0, 1.1.1, 1.1.0)
Tests / Node Web Test	1 (1.0.0)
Bin Tester	2 (7.0.1, 7.0.0)
CI Logger	2 (8.0.1, 8.0.0)
Config Files	15 (13.3.1, 13.3.0, 13.2.0, 13.1.0, 13.0.0, 12.2.0, 12.1.0, 12.0.1, 12.0.0, 11.5.1, 11.5.0, 11.4.0, 11.3.0, 11.2.1, 11.2.0)
Curl jq	8 (4.0.1, 4.0.0, 3.2.3, 3.2.2, 3.2.1, 3.2.0, 3.1.2, 3.1.1)
Dependency Check	13 (4.5.2, 4.5.1, 4.5.0, 4.4.5, 4.4.4, 4.4.3, 4.4.2, 4.4.1, 4.4.0, 4.3.0, 4.2.2, 4.2.1, 4.2.0)
Docker Sokrates	7 (4.0.0, 3.0.10, 3.0.9, 3.0.8, 3.0.7, 3.0.6, 3.0.5)
ESLint Config Standard	16 (43.0.0, 42.0.1, 42.0.0, 41.0.0, 40.0.0, 39.0.2, 39.0.1, 39.0.0, 38.0.1, 38.0.0, 37.1.0, 37.0.0, 36.0.0, 35.0.1, 35.0.0, 34.0.0)
GitLab CI Env	6 (11.3.0, 11.2.0, 11.1.0, 11.0.0, 10.3.0, 10.2.0)
GitLab CI Templates	32 (49.1.0, 49.0.0, 48.1.0, 48.0.0, 47.0.0, 46.0.1, 46.0.0, 45.0.0, 44.5.0, 44.4.0, 44.3.0, 44.2.0, 44.1.0, 44.0.0, 43.1.0, 43.0.0, 42.0.0, 41.0.0, 40.0.0, 39.0.3, 39.0.2, 39.0.1, 39.0.0, 38.3.1, 38.3.0, 38.2.0, 38.1.2, 38.1.1, 38.1.0, 38.0.0, 37.5.0, 37.4.0)
GitLab PMD CPD	25 (3.0.2, 3.0.1, 3.0.0, 2.10.6, 2.10.5, 2.10.4, 2.10.3, 2.10.2, 2.10.1, 2.10.0, 2.9.0, 2.8.2, 2.8.1, 2.8.0, 2.7.3, 2.7.2, 2.7.1, 2.7.0, 2.6.0, 2.5.6, 2.5.5, 2.5.4, 2.5.3, 2.5.2, 2.5.1)
GitLab Pa11y CI	39 (13.3.1, 13.3.0, 13.2.1, 13.2.0, 13.1.1, 13.1.0, 13.0.1, 13.0.0, 12.6.1, 12.6.0, 12.5.1, 12.5.0, 12.4.1, 12.4.0, 12.3.2, 12.3.1, 12.3.0, 12.2.1, 12.2.0, 12.1.1, 12.1.0, 12.0.0, 11.7.3, 11.7.2, 11.7.1, 11.7.0, 11.6.1, 11.6.0, 11.5.4, 11.5.3, 11.5.2, 11.5.1, 11.5.0, 11.4.0, 11.3.4, 11.3.3, 11.3.2, 11.3.1, 11.3.0)
GitLab Releaser	4 (9.0.1, 9.0.0, 8.0.6, 8.0.5)
GitLab Semgrep Plus	24 (11.1.0, 11.0.0, 10.5.1, 10.5.0, 10.4.1, 10.4.0, 10.3.0, 10.2.5, 10.2.4, 10.2.3, 10.2.2, 10.2.1, 10.2.0, 10.1.0, 10.0.4, 10.0.3, 10.0.2, 10.0.1, 10.0.0, 9.5.0, 9.4.0, 9.3.0, 9.2.0, 9.1.4)
GitLab Webhook Renovate Proxy	13 (1.0.3, 1.0.2, 1.0.1, 1.0.0, 0.12.2, 0.12.1, 0.12.0, 0.11.0, 0.10.0, 0.9.0, 0.8.1, 0.8.0, 0.7.0)
Lighthouse	55 (8.2.3, 8.2.2, 8.2.1, 8.2.0, 8.1.3, 8.1.2, 8.1.1, 8.1.0, 8.0.2, 8.0.1, 8.0.0, 7.17.1, 7.17.0, 7.16.3, 7.16.2, 7.16.1, 7.16.0, 7.15.4, 7.15.3, 7.15.2, 7.15.1, 7.15.0, 7.14.4, 7.14.3, 7.14.2, 7.14.1, 7.14.0, 7.13.1, 7.13.0, 7.12.3, 7.12.2, 7.12.1, 7.12.0, 7.11.1, 7.11.0, 7.10.3, 7.10.2, 7.10.1, 7.10.0, 7.9.0, 7.8.2, 7.8.1, 7.8.0, 7.7.0, 7.6.2, 7.6.1, 7.6.0, 7.5.0, 7.4.2, 7.4.1, 7.4.0, 7.3.2, 7.3.1, 7.3.0, 7.2.3)
Pa11y CI CLI Summary Reporter	2 (5.0.1, 5.0.0)
Pa11y CI HTML Reporter	2 (8.0.1, 8.0.0)
Pa11y CI Reporter Runner	5 (6.0.2, 6.0.1, 6.0.0, 5.0.5, 5.0.4)
Pa11y HTML Reporter Plus	2 (5.0.1, 5.0.0)
Pagean	41 (15.3.1, 15.3.0, 15.2.1, 15.2.0, 15.1.0, 15.0.1, 15.0.0, 14.10.1, 14.10.0, 14.9.1, 14.9.0, 14.8.0, 14.7.1, 14.7.0, 14.6.0, 14.5.1, 14.5.0, 14.4.1, 14.4.0, 14.3.0, 14.2.3, 14.2.2, 14.2.1, 14.2.0, 14.1.1, 14.1.0, 14.0.2, 14.0.1, 14.0.0, 13.6.0, 13.5.3, 13.5.2, 13.5.1, 13.5.0, 13.4.0, 13.3.2, 13.3.1, 13.3.0, 13.2.1, 13.2.0, 13.1.1)
Releaselog	4 (7.0.1, 7.0.0, 6.0.3, 6.0.2)
Renovate Config	7 (4.1.1, 4.1.0, 4.0.1, 4.0.0, 3.0.1, 3.0.0, 2.0.0)
Vale Rules for GitLab CI Utils	2 (1.0.7, 1.0.6)

The significant projects and events for the year are summarized in the subsequent sections (ordered predominantly by significance).

NPM supply chain security #

This year saw numerous, large-scale NPM supply chain attacks. Given the ease of publishing, huge catalog of packages, casual treatment of transitive dependencies, and large developer base, it's an attractive target. I was not directly impacted, but it did open the community's eyes to the risks and force mitigating actions.

In July, NPM introduced Trusted Publishing (docs, blog post) to significantly improve the integrity of the publishing process. This eliminates the need for publishing tokens that could be compromised (which we saw this year) and focuses on publishing from CI/CD workflows on trusted platforms. Individual NPM packages must register a specific GitLab or GitHub project as the source, and the CI pipeline or workflow file with the publishing script, then that CI job from that project on that platform can use the platform OIDC to authenticate to NPM and publish projects. This, along with the elimination of long-lived publishing tokens, are significant improvements to the integrity of NPM publishing, although there's still more improvements that could be made. For my part, all of my published NPM packages have moved to trusted publishing.

In addition, it was observed that the large user base, as well as an ever-growing list of security companies, do fairly quickly identify compromised packages. Therefore, allowing some delay after publishing, but before adoption, has been shown to allow community identification of many compromised packages and take mitigating actions. With that, tools like Renovate have adopted a default minimumReleaseAge setting for NPM packages requiring several days for this to occur before updating, and I've adopted that philosophy in my Renovate config presets, which are used in all of my projects.

Type stripping is stable in Node 24 #

I don't think this got nearly as much discussion as it deserved, but with the release of Node 24.12.0 (release notes), the ability for Node to natively execute TypeScript files through type-stripping, without a build or transpilation step, is now stable in the Active LTS release (as well as Node 25, the current release). As a package author, I've published types with libraries for years, but have been resistant to fully adopting TypeScript in those packages, instead leveraging JSDoc types, because of the added overhead and complexity of a build step. The Node implementation does impose some limitations on the use of a few TypeScript features, although one (like me) could argue that these features go beyond pure typing and create new JavaScript when transpiled, so this limitation is not only tolerable, it's desirable. While it will be some time before this is stable in all supported Node versions (April 2027 by my check of the Node release schedule, which is only 15 months away), it's a significant step forward for the ecosystem.

With that, late this year I converted one CLI tool to TypeScript (see this release), and plan to continue as appropriate in 2026. As with implementing any significant new capability, I starting with setting up robust automated tooling, which in this case was an updated ESLint configuration with TypeScript parsing and rules. Given my opinionated stance on linting, it did take some time to get through all of the typescript-eslint rules and setup configurations that properly supported JavaScript (CJS and ESM) and TypeScript files, but the end result is a highly opinionated configuration that I'm happy with, which was published as my last release of the year (see this release).

Chainguard's Kaniko container image #

In mid-2025, after virtually no support for a year, Google officially archived Kaniko, a tool introduced in 2018 to allow building container images in Kubernetes or other containerized environments since it did not rely on the Docker daemon. Shortly afterwards, the Chainguard team, including several of the original Kaniko authors, announced that they had created and were maintaining a fork as a replacement.

While they are maintaining that fork, and have had six releases this year, they are only maintaining the code, not providing any binary artifacts, including a container image. So, I created this Kaniko project, which is simply a CI pipeline and infrastructure to build, test, and deploy a Kaniko container image (built from Kaniko's Dockerfile) to GitLab's container registry (it doesn't contain the Kaniko code, that's pulled from the Chainguard fork repository in CI). You can read the full story in this blog post.

Renovate responsiveness to GitLab repository actions #

My last project of 2024 was the GitLab Webhook Renovate Proxy, which is a Cloudflare Worker that intercepts GitLab merge request and issue webhooks and forwards those that should trigger a Renovate pipeline to re-analyze a project. This year is saw extensive use, and reinforced how effective it is in improving the responsiveness of Renovate running in GitLab CI (beyond simply running scheduled Renovate pipelines). A summary with additional detail is available in this blog post.

Mutation testing #

Last, but certainly not least, I have begun experimentation with mutation testing, incorporating StrykerJS into several of my projects to evaluate its effectiveness in improving test coverage and quality. From their website:

Mutation testing introduces changes to your code, then runs your unit tests against the changed code. It is expected that your unit tests will now fail. If they don't fail, it might indicate your tests do not sufficiently cover the code.

Bugs, or mutants, are automatically inserted into your production code. Your tests are run for each mutant. If your tests fail then the mutant is killed. If your tests passed, the mutant survived. The higher the percentage of mutants killed, the more effective your tests are.

So far it has proved easy to set up and effective at finding gaps in test coverage, although it can be slow given the numerous variations of the test suite that are run.

GitLab CI template tests #

Last year, the GitLab CI Templates project's CI pipeline was updated to trigger multi-project downstream pipelines to allow enhanced testing of templates in the appropriate project types. That was augmented this year with several new test projects to cover additional scenarios including multi-platform container images (which had new jobs and collections added in 2025) and web application testing (which was previously untested). While there are still a few jobs that are not currently tested, this testing now covers the vast majority of the GitLab CI templates project.

Contributions to other open source projects #

I also made contributions to a few other open source projects this year, including knip and Chainguard's fork of kaniko.

Looking forward to 2026 #

Some of the things I'm looking forward to or will hopefully be resolved in 2026 include:

The Node.js Release Working Group team has been iterating on proposed changes to the Node.js release and support philosophy. It's not set in stone yet, but looks like it may take effect in 2026.
This year included some evaluation of container image hardening techniques, which will continue in 2026. This includes migrating to hardened container base images (for example Docker Hardened Images, Chainguard Images, Google Distroless Images, or others) and container image and software bill-of-materials (SBOM) signing (including container registry support, which is evolving, although very slowly).
Continuing to migrate projects to leverage mutation testing and investigate options for improving CI pipeline performance.
The future of Kaniko, as it was this year, is a little uncertain. In addition to the Chainguard fork, a community fork has been getting some attention. It doesn't have corporate backing like the Chainguard fork, but it has been more actively developed this year, including security/bug fixes as well as adding missing capabilities (the --annotation flag, for example). I'll be watching both to see how they evolve this year.

Google Kaniko is dead, long live Chainguard Kaniko

2025-07-22T00:00:00Z

Google has officially archived Kaniko, a tool for building container images in Kubernetes or other containerized environments. Chainguard has taken over a fork of the project, however Chainguard only provides source code, not container images, except as part of their paid offerings. To address this gap, the GitLab CI Utils Kaniko project builds and distributes open source Kaniko container images focused on running in GitLab CI pipelines.

Google Kaniko is dead… #

Google introduced Kaniko in 2018 as a tool to build container images in Kubernetes or other containerized environments since it did not rely on the Docker daemon. It was the tool of choice in these environments for several years. Then, in mid-2024, commits to the repository and Kaniko releases stopped. People then began to question its status. There was some hope when Google released a new version in May 23, 2025. However, this hope was short-lived, and on June 3, 2025 Google officially archived the project with the message "this project is archived and no longer developed or maintained."

…long live Chainguard Kaniko #

Within a day of the Google announcement, the Chainguard team created a fork, followed by a formal announcement with their plans to actively maintain this fork as a replacement unless another community fork emerged. Even better, two of the original creators, Priya Wadhwa and Dan Lorenc, work at Chainguard. Within a week they had their first release, and it appears the project is in good hands.

There was one wrinkle, however. The release plan for the project includes only source code. No binary artifacts, including container images, are provided. Chainguard does offer a different, hardened Kaniko container images as part of their paid offerings, but no build of the new fork. That's not a stab at Chainguard - maintenance of the project itself is hugely beneficial to the community, done without obligation, and a non-hardened image would be off-brand given their product line.

So, here's a container image #

That lead to the creation of the GitLab CI Utils Kaniko project. This project is not a fork a Kaniko, it's simply a project to build and distribute an open source Kaniko container image built from the Chainguard Kaniko repository. It simply clones the repository and then builds the Dockerfile.

Since the focus is on use in GitLab CI, only the debug variant of the image is built. This includes a shell, which is required to use Kaniko in GitLab CI pipelines. All images are multi-platform, supporting both linux/arm64 and linux/amd64 (since gitlab.com has shared runners with those architectures to test the images). There is a debug tag, equivalent to what is typically the latest tag, but sticking with the previous Kaniko tagging convention, as well as vX.Y.Z-debug tags corresponding to each Chainguard Kaniko release.

The project also contains usage details, as well as links to the original GitLab documentation for using Kaniko in GitLab CI, which had been removed prematurely (IMHO).

Configure Renovate to Respond to GitLab Repository Actions

2025-06-28T00:00:00Z

The typical GitLab use case is to use the Renovate runner project templates, or some variant, to run Renovate in a GitLab CI job on a schedule. If you're looking for an option to provide a more responsive experience without the overhead of self-hosting Renovate, this solution uses Cloudflare Workers, GitLab webhooks, and triggered GitLab pipelines to configure Renovate to respond to repository activity.

Renovate setup on GitLab #

The recommended GitLab setup, as detailed in the Renovate documentation, is to use the Renovate runner project templates, or some variant, to run Renovate in a GitLab CI job on a schedule. This is a good starting point, but lacks much of the experience that you get with the GitHub app or Renovate self-hosted, as can be seen in this Renovate version comparison.

If you're looking for an option to improve the experience, without the overhead of self-hosting, this solution using the GitLab Webhook Renovate Proxy handles the "responds to repo activity" case using Cloudflare Workers, GitLab webhooks, and triggered GitLab pipelines. Cloudflare Workers provide a lightweight, easy to deploy solution that in many cases can be used across large GitLab groups on their free tier (although you may want a paid tier to provide support or other services). Cloudflare Worker also have very short cold start times, so they can be scaled to zero with no activity and still provide a quick response. According to the last build, this Worker's startup time is approximately 12 ms.

The GitLab Webhook Renovate Proxy solution #

The GitLab Webhook Renovate Proxy is part of a bigger setup to make Renovate updates with GitLab CI respond to repository activity. It includes the following components:

GitLab Project: Any GitLab project that uses Renovate for dependency updates.
GitLab Webhook Renovate Proxy: A Cloudflare Worker that filters webhooks so that only authenticated, applicable events trigger a Renovate pipelines.
Renovate Runner: The project that runs Renovate pipelines using the Mend renovate-runner project (or a fork, mirror, etc).

The integrated system works as shown below. Webhooks are configured in GitLab to fire for appropriate events (for example, merge request and issue events). These webhooks are sent to the GitLab Webhook Renovate Proxy, which checks that the webhook is for an event that should run a Renovate pipeline, and if so sends a pipeline trigger request to the GitLab Renovate Runner project specifying that project as the target.

sequenceDiagram
  participant G as GitLab Project
  participant P as GitLab Webhook Renovate Proxy
  participant R as Renovate Runner

  G->>P: Webhook (Merge request open)
  G->>P: Webhook (Merge request merge)
  P->>R: Pipeline trigger request
  R->>R: Run pipeline for project
  G->>P: Webhook (Dependency Dashboard rebase branch)
  P->>R: Pipeline trigger request
  R->>R: Run pipeline for project

The GitLab Webhook Renovate Proxy triggers a Renovate pipeline for the following events:

Merge request merge
Merge request "Rebase this MR" checkbox
All Dependency Dashboard checkboxes

The proxy primarily uses the same information that Renovate itself uses to respond to these events - comments in the description for Renovate created merge requests and Dependency Dashboard issues. The GitLab Webhook Renovate Proxy does not constrain what Renovate updates, it simply determines if a Renovate pipeline should be triggered based on the webhook event. This provides a quick response to these events rather than having to wait for the next scheduled Renovate pipeline.

While the GitLab Webhook Renovate Proxy is written as Cloudflare Worker, it could easily be adapted to run in other similar environments (for example, Knative in a self-hosted environment).

System setup #

See the Setup Guide for detailed instructions on end-to-end setup of the GitLab Webhook Renovate Proxy. While it may seem complex, there are really just a lot of small steps, and the instructions are deliberately explicit for anyone unfamiliar with aspects of the process and to ensure that secrets (or potential secrets, for example project IDs for private project) are appropriately protected. For any problems, see the troubleshooting guide, or open an issue.

Note that like any service exposed to the Internet, the GitLab Webhook Renovate Proxy will likely receive requests scanning for known vulnerabilities (they'll be obvious in the logs). The Worker is designed with an authentication mechanism and rejects these requests, but the system alone is not designed to protect against any large scale denial-of-service attacks. While this has not been seen, if required Cloudflare has options for rate limiting and other protections.

Results #

This system has been in use with multiple gitlab.com groups for about 6 months. The groups have 50 - 75 projects with updates managed by Renovate. In most cases almost all project dependencies are as strictly versioned as practical, so there are over 800 total Renovate-managed dependencies across the groups. Most of the projects are also configured to rebase Renovate updates when behind the base branch to best align with the GitLab fast forward merge strategy that the projects primarily use. So, there are a lot of Renovate pipelines across the groups. Renovate is run on the default GitLab hosted runner (the small Linux x86-64 Kubernetes runner) as performance has been observed to be predominantly network bound, so a more capable runner adds limited benefit for the cost.

In general, the Renovate pipelines triggered by the GitLab Webhook Renovate Proxy run in 1 - 2 minutes total. Almost one minute is the image download itself. Given the various ecosystems that Renovate supports there is a lot in the image and it is fairly large (~400 MB compressed, ~1.3 GB uncompressed), and with frequent Renovate updates (many times per day) there is minimal caching (although the GitLab hosted runners are configured to use GCP's pull-through image cache, so it is used when applicable).

Overall, that puts the responsiveness of Renovate similar to the GitHub app for responding to repository changes (my best comparison point having never self-hosted Renovate Community/Enterprise). It's a huge usability improvements for cases where you're trying to process multiple Renovate updates against a single repository. While it does not have all of the optimizations that the GitHub app or Renovate self-hosted do, it's a good compromise for GitLab hosted projects that do not want the overhead of the self-hosted Renovate solution.

The Year in Review 2024

2024-12-31T00:00:00Z

It's been another busy year - I wrote a handful of blog posts on some long-researched automation topics, continued development of many old projects, started over a dozen new projects including my first Cloudflare Worker, and shipped over 450 software releases. This post reviews it all.

Blog posts #

This was year two of the return of this blog. As with year one, I had lofty goals for how much writing I would do that did not quite work out. This year I did, however, meet my primary goal - getting several big posts out on alternative automation and analysis techniques, which had significant background research and testing, including:

GitLab CI Pipeline for Go Projects
GitLab Code Quality Duplication Analysis With PMD CPD
Notifications for New Eleventy Posts in GitLab (Part 1, Part 2)
A Better GitLab Code Quality - Part 1 - Goodbye CodeClimate (Part 2 really is coming)

Software projects #

Most of the 2024 involved development work on a variety of project, including my own open source projects, contributions to other open source projects, and starting a new fork of pa11y and pa11y-ci.

The uncertain future of Pa11y and Pa11y CI #

I was anticipating the release of pa11y@7.0.0 and pa11y-ci@4.0.0 in 2024. A lot of work had gone into various updates, primarily focused on updating from deprecated to supported dependencies, and those updates were ready to release. This year started with a resurgence of maintainers' support for the pa11y projects, including the release of pa11y@7.0.0 and pa11y@8.0.0. Unfortunately, after that things fizzled out again, and there was never an official pa11y-ci@4.0.0. That's not a criticism of the pa11y team - priorities can always change on any open source project, and it's only because of the work of the core team that the project is so useful that others want to contribute and want it to live on.

My focus, however, is pa11y-ci, so the lack of an official pa11y-ci@4.0.0 release with a supported version of puppeteer finally crossed my pain threshold. So, in September I forked pa11y into @aarongoldenthal/pa11y and pa11y-ci into @aarongoldenthal/pa11y-ci. These forks have been focused on security patches as a short-term fix until the future of pa11y and pa11y-ci was better understood. This led to 14 releases of @aarongoldenthal/pa11y, now up to v8.4.4, and 13 releases @aarongoldenthal/pa11y-ci, starting with a new major release and now up to v4.3.4. In addition, the gitlab-pa11y-ci container image was updated to use these forks.

Even with these forks, it has become clear that the backlog of unresolved vulnerabilities, bug fixes, and usability enhancements is not only ever-increasing, but they also require more significant changes that really deserve a dedicated fork, maybe with a new name and new branding, but definitely with sufficient resources and community support. That saga needs to be resolved on way or another in 2025.

My open source software projects #

I continued to maintain a long list of open source projects. In 2024 I started 11 new projects, and between those projects and my other existing projects I shipped 426 releases (72 major, 118 minor, 229 patch, 7 zero), and that's excluding the 27 releases from the pa11y/pa11y-ci forks noted previously. Even though many of those were security patches or dependency updates, there was also extensive new development, and some new projects. Given those 453 releases were in my free time should reinforce to people that continuous delivery practices and extensive automation are a worthwhile investment and really do improve productivity and the ability to confidently release software.

Full list of 2024 new projects

Project	Description
Container Images / Cloc	An Alpine-based container image to run cloc to count lines of code.
Container Images / Image Tools	A collection of tools and scripts for working with container images.
Container Images / Puppeteer	Docker Puppeteer is a container image with the recommended configuration for using Puppeteer to drive a headless Chrome browser.
Container Images / Vale	A container image to run Vale to lint prose, with pre-installed rule packages and a custom GitLab Code Quality report.
Tests / Container Test	A container project to test CI jobs.
Tests / Go Test	A Go project to test CI jobs.
Tests / Node CJS Test	A Node.js CJS project to test CI jobs.
Tests / Node ESM Test	A Node.js ESM project to test CI jobs.
GitLab Webhook Renovate Proxy	A Cloudflare Worker to intercept GitLab webhooks and only forward events that should trigger execution of a Renovate pipeline.
Renovate Config	A collection of reusable Renovate config presets.
Vale Rules for GitLab CI Utils	A Vale package with custom rules and vocabulary for GitLab CI Utils projects.

Full list of 2024 software releases

Project	Releases
Container Images / Cloc	8 (1.3.0, 1.2.2, 1.2.1, 1.2.0, 1.1.2, 1.1.1, 1.1.0, 1.0.0)
Container Images / Go Test	19 (2.5.6, 2.5.5, 2.5.4, 2.5.3, 2.5.2, 2.5.1, 2.5.0, 2.4.3, 2.4.2, 2.4.1, 2.4.0, 2.3.2, 2.3.1, 2.3.0, 2.2.1, 2.2.0, 2.1.1, 2.1.0, 2.0.1)
Container Images / Image Tools	6 (1.2.1, 1.2.0, 1.1.1, 1.1.0, 1.0.1, 1.0.0)
Container Images / Puppeteer	6 (14.0.0, 13.2.0, 13.1.2, 13.1.1, 13.1.0, 13.0.0)
Container Images / Syft	7 (1.2.5, 1.2.4, 1.2.3, 1.2.2, 1.2.1, 1.2.0, 1.1.1)
Container Images / Vale	49 (3.3.1, 3.3.0, 3.2.1, 3.2.0, 3.1.0, 3.0.2, 3.0.1, 3.0.0, 2.3.3, 2.3.2, 2.3.1, 2.3.0, 2.2.1, 2.2.0, 2.1.1, 2.1.0, 2.0.1, 2.0.0, 1.13.1, 1.13.0, 1.12.0, 1.11.1, 1.11.0, 1.10.0, 1.9.1, 1.9.0, 1.8.0, 1.7.0, 1.6.0, 1.5.1, 1.5.0, 1.4.2, 1.4.1, 1.4.0, 1.3.3, 1.3.2, 1.3.1, 1.3.0, 1.2.0, 1.1.0, 1.0.7, 1.0.6, 1.0.5, 1.0.4, 1.0.3, 1.0.2, 1.0.1, 1.0.0, 0.5.0)
Node / StyleLint Config Standard	5 (19.0.0, 18.0.1, 18.0.0, 17.0.1, 17.0.0)
References / GitLab Group Project Summary	5 (1.0.4, 1.0.3, 1.0.2, 1.0.1, 1.0.0)
Tests / Container Test	5 (1.0.4, 1.0.3, 1.0.2, 1.0.1, 1.0.0)
Tests / Go Test	2 (1.0.1, 1.0.0)
Tests / Node CJS Test	4 (2.0.0, 1.1.0, 1.0.1, 1.0.0)
Tests / Node ESM Test	1 (1.0.0)
Bin Tester	1 (6.0.0)
CI Logger	1 (7.0.0)
Config Files	12 (11.1.0, 11.0.0, 10.6.0, 10.5.0, 10.4.0, 10.3.0, 10.2.0, 10.1.1, 10.1.0, 10.0.2, 10.0.1, 10.0.0)
Docker Dependency Check	20 (4.1.2, 4.1.1, 4.1.0, 4.0.0, 3.0.9, 3.0.8, 3.0.7, 3.0.6, 3.0.5, 3.0.4, 3.0.3, 3.0.2, 3.0.1, 3.0.0, 2.2.1, 2.2.0, 2.1.0, 2.0.7, 2.0.6, 2.0.5)
Docker PowerShell Scripts	4 (5.3.0, 5.2.0, 5.1.0, 5.0.0)
Docker Puppeteer	4 (13.0.0, 12.0.0, 11.1.0, 11.0.1)
Docker Sokrates	6 (3.0.4, 3.0.3, 3.0.2, 3.0.1, 3.0.0, 2.1.2)
Docker curl jq	8 (3.1.0, 3.0.5, 3.0.4, 3.0.3, 3.0.2, 3.0.1, 3.0.0, 2.1.0)
ESLint Config Standard	14 (33.0.2, 33.0.1, 33.0.0, 32.0.0, 31.1.0, 31.0.1, 31.0.0, 30.0.0, 29.0.0, 28.0.0, 27.0.1, 27.0.0, 26.0.1, 26.0.0)
GitLab CI Env	4 (10.1.0, 10.0.0, 9.3.0, 9.2.0)
GitLab CI Templates	36 (37.3.1, 37.3.0, 37.2.0, 37.1.0, 37.0.0, 36.0.1, 36.0.0, 35.1.0, 35.0.0, 34.0.0, 33.8.0, 33.7.0, 33.6.0, 33.5.0, 33.4.0, 33.3.0, 33.2.0, 33.1.0, 33.0.0, 32.0.0, 31.1.0, 31.0.0, 30.0.0, 29.6.0, 29.5.0, 29.4.0, 29.3.0, 29.2.0, 29.1.0, 29.0.0, 28.1.0, 28.0.0, 27.0.1, 27.0.0, 26.1.1, 26.1.0)
GitLab Webhook Renovate Proxy	3 (0.6.0, 0.5.0, 0.0.1)
GitLab PMD CPD	18 (2.5.0, 2.4.1, 2.4.0, 2.3.6, 2.3.5, 2.3.4, 2.3.3, 2.3.2, 2.3.1, 2.3.0, 2.2.1, 2.2.0, 2.1.2, 2.1.1, 2.1.0, 2.0.3, 2.0.2, 2.0.1)
GitLab Pa11y CI	22 (11.2.4, 11.2.3, 11.2.2, 11.2.1, 11.2.0, 11.1.0, 11.0.0, 10.1.0, 10.0.5, 10.0.4, 10.0.3, 10.0.2, 10.0.1, 10.0.0, 9.0.5, 9.0.4, 9.0.3, 9.0.2, 9.0.1, 9.0.0, 8.0.1, 8.0.0)
GitLab Pipeline Timeline	3 (0.4.0, 0.3.1, 0.3.0)
GitLab Releaser	6 (8.0.4, 8.0.3, 8.0.2, 8.0.1, 8.0.0, 7.0.2)
GitLab Semgrep Plus	48 (9.1.3, 9.1.2, 9.1.1, 9.1.0, 9.0.3, 9.0.2, 9.0.1, 9.0.0, 8.6.4, 8.6.3, 8.6.2, 8.6.1, 8.6.0, 8.5.7, 8.5.6, 8.5.5, 8.5.4, 8.5.3, 8.5.2, 8.5.1, 8.5.0, 8.4.0, 8.3.1, 8.3.0, 8.2.3, 8.2.2, 8.2.1, 8.2.0, 8.1.3, 8.1.2, 8.1.1, 8.1.0, 8.0.1, 8.0.0, 7.0.1, 7.0.0, 6.2.1, 6.2.0, 6.1.2, 6.1.1, 6.1.0, 6.0.0, 5.2.3, 5.2.2, 5.2.1, 5.2.0, 5.1.0, 5.0.0)
Lighthouse	34 (7.2.2, 7.2.1, 7.2.0, 7.1.1, 7.1.0, 7.0.0, 6.4.0, 6.3.2, 6.3.1, 6.3.0, 6.2.2, 6.2.1, 6.2.0, 6.1.2, 6.1.1, 6.1.0, 6.0.2, 6.0.1, 6.0.0, 5.3.0, 5.2.0, 5.1.0, 5.0.1, 5.0.0, 4.9.0, 4.8.2, 4.8.1, 4.8.0, 4.7.0, 4.6.2, 4.6.1, 4.6.0, 4.5.0, 4.4.0)
Pa11y CI CLI Summary Reporter	2 (4.0.1, 4.0.0)
Pa11y CI HTML Reporter	1 (7.0.0)
Pa11y CI Reporter Runner	8 (5.0.3, 5.0.2, 5.0.1, 5.0.0, 4.1.3, 4.1.2, 4.1.1, 4.1.0)
Pa11y HTML Reporter Plus	3 (4.0.1, 4.0.0, 3.0.0)
Pagean	39 (13.1.0, 13.0.3, 13.0.2, 13.0.1, 13.0.0, 12.3.0, 12.2.1, 12.2.0, 12.1.3, 12.1.2, 12.1.1, 12.1.0, 12.0.3, 12.0.2, 12.0.1, 12.0.0, 11.2.1, 11.2.0, 11.1.5, 11.1.4, 11.1.3, 11.1.2, 11.1.1, 11.1.0, 11.0.4, 11.0.3, 11.0.2, 11.0.1, 11.0.0, 10.2.0, 10.1.5, 10.1.4, 10.1.3, 10.1.2, 10.1.1, 10.1.0, 10.0.2, 10.0.1, 10.0.0)
Releaselog	3 (6.0.1, 6.0.0, 5.0.1)
Renovate Config	3 (1.1.0, 1.0.1, 1.0.0)
Vale Rules for GitLab CI Utils	6 (1.0.5, 1.0.4, 1.0.3, 1.0.2, 1.0.1, 1.0.0)

These releases included a handful of significant accomplishments:

This work was started in 2023, but as of this year all of my NPM packages are published with package provenance, a step to help improve the overall security of the NPM package ecosystem. It's an easy one-time setup, so there's no reason not to do it.
I released my first multi-platform container image, built on GitLab with kaniko and manifest-tool, which uses parent-child downstream pipelines to simplify expanding the supported platforms. See the curl-jq project for specifics.
I released a script to add OCI annotations to container images. This uses crane to update existing images in a container registry, matching image LABELs where they exist, determining the base image from the Dockerfile, and works with any build tool. The Docker Official images are built with OCI annotations, and those cascade to derived images, so if you're not updating them your image manifests may have missing or incorrect information, and as more tools use that data it can become an increasing problem. See the image-tools container image project for details.
The GitLab CI Templates project pipeline was updated to trigger multi-project downstream pipelines in the appropriate subset of newly created test projects of common types (Node ESM/CJS, Go, container images). This allows for automated testing of template changes (for at least 90% of the templates), which is a significant improvement over the previous manual testing process.
As part of this year's goal of migrating all of my CLI-based Node projects to ESM, I converted the Pagean project to ESM. It's still intentionally JavaScript, not TypeScript, but I get all the type checking I need with JSDoc comments, which the project uses already, and tsc is perfectly happy checking types with that. And most importantly, there's still no build step.
My favorite project of the year came out of nowhere in the last week of the year - the GitLab Webhook Renovate Proxy. This is a Cloudflare Worker that intercepts GitLab merge request event webhooks, which are extremely noisy, filters out all except those that should cause Renovate to re-analyze a project, and forwards to GitLab to trigger a CI pipeline. This allows projects to use Renovate's standard GitLab runner, but get back some of the responsiveness of the Renovate GitHub app. A blog post with the implementation details and setup instructions will be coming.

Contributions to other open source projects #

I also made contributions to a few other open source projects this year:

I made my first contributions to GitLab, both documentation cleanup as well as fixing documentation and tests for the GitLab release-cli application.
I made my first contribution to the Trivy vulnerability scanner updating their GitLab report template to include the operating system to improve parity with GitLab's container scanning report.

Looking forward to 2025 #

Like so many years, it's going to be a wild ride…

A Better GitLab Code Quality - Part 1 - Goodbye CodeClimate

2024-08-27T00:00:00Z

GitLab's Code Quality analysis template can be a convenient tool, but the default CodeClimate-based solution has some significant drawbacks, and was recently deprecated. This post, part 1 of a series, explores the security, performance, and usability issues with the default Code Quality template, and why its deprecation not a significant loss. Future parts of this series will explore a more flexible, more adaptable, and more capable approach.

GitLab CodeClimate-based Code Quality is deprecated #

GitLab 17.3 announced the official deprecation of GitLab's CodeClimate-based Code Quality analysis, with removal planned for GitLab 18.0. In the interim only limited updates are planned in accordance with their support policy for deprecated capabilities. There is an open epic looking at future solutions, although it's notional and aspirational at this point. Their plan is to keep Code Quality reports, and the existing UI capabilities in Pipelines and Merge Requests, so custom Code Quality solutions remain working as expected (which is the direction this series is going).

The remainder of this post was originally written before this announcement, so it's included here to provide additional context on why this is not a great loss.

Problems with GitLab's CodeClimate-based Code Quality #

GitLab's CodeClimate-based Code Quality template can be a convenient and effective tool for analyzing code quality, but it does have some significant drawbacks including security concerns, performance issues, usability issues, and analysis limitations.

Runner security #

CodeClimate requires one of two general configurations to perform the analysis since it spawns containers as needed based on the analysis requirements. These configurations expose potentially significant security concerns.

The default template assumes Docker-in-Docker is used. This is a well understood technique, but requires runners running in privileged mode. Privileged mode gives the container all the root capabilities of the host. From the GitLab docs - "When privileged mode is enabled, a user running a CI/CD job could gain full root access to the runner’s host system, permission to mount and unmount volumes, and run nested containers. By enabling privileged mode, you are effectively disabling all the container’s security mechanisms and exposing your host to privilege escalation, which can lead to container breakout."
The alternative configuration for private runners requires exposing the Docker daemon socket to the container (required to launch new containers). By default, this is owned by the root user, again exposing root access to the host.

These settings represent attack vectors that could lead to the compromise of any CI job performed on the container, as well as compromise of the host running the container and anything running on it. While there are methods to limit the potential impacts - not running parallel CI jobs on a VM, running on ephemeral infrastructure, running Docker in rootless mode, etc. - there's no way to completely mitigate them, they're just varying levels of risk. There are also questions about the viability of some solutions, like rootless mode, as this issue suggests.

Even limiting access to known users can lead to an exploit if they bring in compromised external dependencies (and what application doesn't bring in external dependencies in modern development). Because of that, the recommendation to not allow configurations with these potential attack vectors is well documented in container security best practice guides from across the industry, including Anchore, Aqua Security, OWASP Foundation, NIST, GitLab's own Docker runner security risks, and certainly others.

Even Docker Hub's Docker page recommends reviewing this article to understand the risks before running Docker-in-Docker. This has been updated with a more secure alternative, which has not been investigated thoroughly as part of this evaluation. This may be a viable solution, although it would not resolve the other issues identified here.

Speed #

The GitLab Code Quality job is slow, with several contributing factors.

The Docker-in-Docker startup cost (reduced if running GitLab's alternative configuration).
The overhead of launching the various analysis containers.
The time to download the various analysis containers.

It can take several minutes even for a small project (e.g. < 500 lines of JavaScript), which can quickly become the longest running job in a pipeline. There are some mitigations to reduce these impacts. The first two performance items in the preceding list are somewhat constrained by compute and memory allocated to the runner. So, allocating more resources can reduce execution time. For gitlab.com shared SaaS runners, this would mean moving to more capable runners - medium, large, x-large, 2x-large - although this does come at increased usage cost. Even with this, there are limits to the effectiveness, and a single job running multiple analyses is typically slower than running multiple parallel jobs.

The time to download can be harder to improve since some of these containers are large. To illustrate the magnitude of the problem, the following table shows the container images and size to run Code Quality analysis configured for a JavaScript-based web project (with csslint, duplication, eslint, and fixme) enabled (which is the same as the default configuration, but with coffeelint and rubocop disabled).

Image	Size
`docker:20.10.12`	65 MB
`docker:20.10.12-dind`	72 MB
`registry.gitlab.com/gitlab-org/ci-cd/codequality:0.96.0`	110 MB
`codeclimate/codeclimate-structure:latest`	2.3 GB
`codeclimate/codeclimate-csslint:latest`	60 MB
`codeclimate/codeclimate-duplication:latest`	2.34 GB
`codeclimate/codeclimate-eslint:latest`	170 MB
`codeclimate/codeclimate-fixme:latest`	21 MB
Total	5.14 GB

The following table shows the container images and size to run Code Quality analysis on a Go-based project with an updated configuration enabling all of the available Go plugins.

Image	Size
`docker:20.10.12`	65 MB
`docker:20.10.12-dind`	72 MB
`registry.gitlab.com/gitlab-org/ci-cd/codequality:0.96.0`	110 MB
`codeclimate/codeclimate-structure:latest`	2.3 GB
`codeclimate/codeclimate-duplication:latest`	2.34 GB
`codeclimate/codeclimate-fixme:latest`	21 MB
`codeclimate/codeclimate-golint:latest`	127 MB
`codeclimate/codeclimate-gofmt:latest`	101 MB
`codeclimate/codeclimate-govet:latest`	125 MB
Total	5.26 GB

Each of these examples includes a total of over 5 GB of container images.

There are various container image caching strategies that may help in some instances, but may not be available based on your runner configuration.

Ease of use #

The GitLab docs give the impression that enabling their CodeClimate-based Code Quality analysis is as simple as including the following in the project's .gitlab-ci.yml file:

include:
  - template: Jobs/Code-Quality.gitlab-ci.yml

The reality is that this only partially true. The default configuration is used in this case, which runs the following plugins:

csslint (CSS)
coffeelint (CoffeeScript)
duplication (only for Ruby, JavaScript, Python, PHP)
eslint (JavaScript/TypeScript)
fixme
rubocop (Ruby)

Some of these may not be applicable, for example the coffeelint plugin is probably past its useful life. There are also numerous other analysis plugins, but these need to be specifically configured with a .codeclimate.yml file. CodeClimate is not adaptable enough to check file types in the project and run the applicable plugins or jobs as is done with other GitLab templates, for example sast or dependency_scanning (if using GitLab Ultimate).

Limited analysis engines #

The complete list of CodeClimate analysis plugins can be found here. The officially supported list of plugins is quite limited, and even with the community supported plugins it's far from a comprehensive coverage of modern development languages. Even in cases where languages are covered, the analysis engines (all integrating other open source tools) are outdated. A couple of examples:

There's no question that eslint is the solution for linting JavaScript, but CodeClimate only supports up to v8.50.0 (released September 2023), even though the latest v8 is v8.57.0 (released February 2024), and v9 is out (released April 2024) and has 12 releases up to v9.9.1 (as of this writing in August 2024). Given this release cadence with valuable fixes and rule updates, the lag to integrate into CodeClimate is intolerable (in this case a year behind).
The available Go plugins are limited to gofmt, govet, and golint. While the first two are still used, golint has been archived for years (and even the CodeClimate plugin is deprecated). There's no support for modern engines like golangci-lint, which itself has dozens of linters covering almost anything you could want to lint (including gofmt and govet).

Summary #

This post has examined the various issues with GitLab's CodeClimate-based Code Quality analysis, and why its deprecation is not a significant loss. Future parts of this series will explore a more flexible, more adaptable, and more capable approach.

Notifications for New Eleventy Posts in GitLab - Part 2

2024-04-29T00:00:00Z

One of the challenges with deploying static sites is that there's nothing tracking any sort of site state, including when new content is published. This post presents a technique to identify newly published content on an Eleventy site and sending various notifications with content-specific data. Part 1 covered identifying the new posts and collecting post-specific data. Part 2 covers posting a status to Mastodon, posting a status to Bluesky, and sending an IndexNow notification for the new page.

Sending notifications for new posts #

Once the new posts are identified, any kind of notification could be sent, but the three implemented here are:

Posting a status to Mastodon
Posting a status to Bluesky
Sending an IndexNow notification

Managing secrets in GitLab #

The following sections use secrets, specifically API keys/tokens, to access those services securely. These secrets should not be kept in the project's git repository. For these examples, these are all stored as variables in GitLab, which are encrypted for security. They also have the following settings:

protected: Ensures the variable is only exposed on protected branches or protected tags. By default, these require project maintainer or owner permissions to execute and is typically the setting on the default branch.
masked: Uses GitLab's bult-in logic to redact the variable in any GitLab CI job log. As the GitLab docs note, this is on a best-effort basis, but is generally reliable (maybe sometimes too reliable and there's unnecessary masking - better to be safe than sorry).

These variables are then exposed to jobs on GitLab CI pipelines run on protected branches or tags as environment variables. For this example that's the default branch, which is where the site is published.

See the documentation for additional details on GitLab variables security.

Iterate through new posts providing notifications #

In part 1, a function was introduced to iterates through new posts to send notifications via an async queue.

(async () => {
    const posts = getNewPosts();
    if (posts.length === 0) {
        console.log('No new posts to submit');
        return;
    }
    const taskQueue = [];
    for (const post of posts) {
        console.log(`Submitting updates for ${post.url}`);
        taskQueue.push(
            postMastodonStatus(post),
            postBlueskyStatus(post),
            postIndexNow(post)
        );
    }

    const results = await Promise.allSettled(taskQueue);
    for (const result of results) {
        if (result.status === 'rejected') {
            console.error(result.reason.message);
        }
    }
})();

This function has been updated to call functions for each of the three desired notifications passing the post data. The implementation for the postMastodonStatus, postBlueskyStatus, and postIndexNow functions is detailed in the following sections. These are the three examples implemented with this technique, but could be augmented or replaced with other notifications.

Post Mastodon status #

To start posting to your applicable Mastodon instance via the API, you must first create an Application, and with that obtain an access token. This can be done on your applicable Mastodon instance at Preferences > Development > New application (at least it is in my case, the official documentation on doing this via the UI is, unfortunately, limited).

You'll need to provide application name, website, and the applicable scopes (you do not need to modify the default Redirect URI value). The default scopes give the application a lot of capability, for the actions here only read:statuses and write:statuses are required. Then click SUBMIT and you'll be returned to your list of applications. From there you can see the details for the application that was just created, and "Your access token" is the token value to use in the example below.

The access token is stored as the protected and masked variable MASTODON_TOKEN within GitLab. The postMastodonStatus function uses that token and the post information to post a status to Mastodon.

const postMastodonStatus = async (post) => {
    const accessToken = process.env.MASTODON_TOKEN;
    // Update for the applicable Mastadon instance
    const instanceUrl = 'https://fosstodon.org';
    const data = {
        status: `${post.description}\n\n${post.url}\n\n${post.tags.join(' ')}`
    };

    const response = await fetch(`${instanceUrl}/api/v1/statuses`, {
        body: JSON.stringify(data),
        headers: {
            Authorization: `Bearer ${accessToken}`,
            'Content-Type': 'application/json'
        },
        method: 'POST'
    });

    const postStatus = await response.json();
    if (!response.ok) {
        throw new Error(`Error posting Mastadon status: ${postStatus.error}`);
    }
    console.log(`Mastodon status successfully posted (ID: ${postStatus.id})`);
};

The function uses fetch to post the status with the following data:

The previously defined MASTODON_TOKEN environment variable with the access token.
The base URL of the applicable Mastodon instance.
A status message, which in this case includes the post description, URL, and hashtags (the tags array was previously formatted as hashtags).

If successful, a JSON response is returned and a success message is logged with the returned status ID. If not, an error is thrown with the response error.

Additional details on using the Mastodon API can be found in the the documentation.

Post Bluesky status #

To access Bluesky's API an App password must be created (please don't use your actual account password). The official docs, don't cover this. An App password can be created at Settings > Advanced > App Passwords > Add App Password. You'll be prompted for a name for the App, and then returned the password. Note that this is only opportunity to view the password, so capture it somewhere secure. If it's lost, it can be deleted and a replacement created.

The App password is stored as the protected and masked variable BSKY_PASSWORD within GitLab. In addition the associated Bluesky ID is stored as the protected and masked variable BSKY_ID within GitLab.

The postBlueskyStatus function uses these credentials and the post information to submit a post to Bluesky.

const { BskyAgent, RichText } = require('@atproto/api');

const postBlueskyStatus = async (post) => {
    // Use Bluesky agent based on API complexity
    const agent = new BskyAgent({
        service: 'https://bsky.social'
    });

    try {
        await agent.login({
            identifier: process.env.BSKY_ID,
            password: process.env.BSKY_PASSWORD
        });

        const message = `${post.description}\n\n${post.url}`;
        // Rich formatting in posts, for example links and mentions, must be
        // specified by byte offsets, so use the RichText capabilities to
        // detect them.
        const rt = new RichText({ text: message });
        await rt.detectFacets(agent);

        const postRecord = {
            $type: 'app.bsky.feed.post',
            createdAt: new Date().toISOString(),
            // Cards are not automatically created from OG tags in the link,
            // they must be explicitly added. Note this does not include
            // images, which must be referenced, and separately posted to
            // the API.
            embed: {
                $type: 'app.bsky.embed.external',
                external: {
                    description: post.description,
                    title: post.title,
                    uri: post.url
                }
            },
            facets: rt.facets,
            text: rt.text
        };

        await agent.post(postRecord);
        console.log('Bluesky status successfully posted');
    } catch (error) {
        throw new Error(`Error posting Bluesky status: ${error.message}`);
    }
};

Bluesky's AT protocol has granular control, but with that comes additional complexity. So, the Bluesky agent from the official API client is used:

The BSKY_ID and BSKY_PASSWORD environment variable are used to login.
The message includes the post description and URL. Any links (or mentions) in the message are not automatically detected, and must be specified by the start/end byte offset, but the API client provides the RichText class that is used to detect and provide this data (to linkify the URL).
The postRecord is constructed with all of this data. In addition, cards representing the URL are not automatically added, so an embed is included with the card data (description, title, and URL). In this case an image is not provided, although if required it must be submitted via the API as a separate request (see the docs for full details).

If successful, a success message is logged. If not, an error is thrown with the response error.

Additional details on using the AT protocol for Posts can be found in the the Bluesky documentation, as well as the API client documentation.

Post URL to IndexNow #

IndexNow is a standard that allows submission of a URL to enabled search engines, which is then shared with all IndexNow-enabled engines (Bing, for example, is one of the IndexNow enabled search engines). This avoids an indexing delay waiting for the new page to be organically discovered.

Details on using the API and obtaining an API key is available in the documentation. The API key is stored as the protected and masked variable INDEXNOW_API_KEY within GitLab. To validate the API key, a file is required to be deployed to the site. This file is named with the key, and its content includes the key, so the previously defined pages CI job script is updated to create the file.

pages:
  ...
  script:
    # build runs `npx @11ty/eleventy`
    - npm run build
    # Create IndexNow key file in the site output folder
    - echo $INDEXNOW_API_KEY > ./public/${INDEXNOW_API_KEY}.txt
  artifacts:
    # Ensure artifacts with the key file are not publicly visible
    public: false
    ...

This implementation allows the API key to be securely stored in a GitLab variable and the file created only for deployment. With that, the job artifacts is also updated with public: false to ensure that job artifacts are not made available publicly (only to project members). The variable is also masked, but this provides an extra layer of protection against accidental exposure.

The postIndexNow function uses the API key and the post information to submit the URL to IndexNow.

const postIndexNow = async (post) => {
    const apiKey = process.env.INDEXNOW_API_KEY;
    const indexNowUrl = 'https://api.indexnow.org/IndexNow';
    const postUrl = new URL(post.url);
    const { host } = postUrl;
    const data = {
        host,
        key: apiKey,
        keyLocation: `https://${host}/${apiKey}.txt`,
        urlList: [postUrl.toString()]
    };

    const response = await fetch(indexNowUrl, {
        body: JSON.stringify(data),
        headers: { 'Content-Type': 'application/json' },
        method: 'POST'
    });

    if (!response.ok) {
        throw new Error(
            `Error submitting URL to IndexNow: ${response.statusText}`
        );
    }
    console.log(`URL ${post.url} successfully submitted to IndexNow`);
};

The function uses fetch to post the status with the following data:

The previously defined INDEXNOW_API_KEY environment variable with the API key.
The website host is taken from the new post URL.
The location of the API key file that was previously created.
The list of URLs (in this case only one URL is submitted with each request, but this API endpoint does accept a list of URLs).

In this cases the update is posted to api.indexnow.org, although it could be posted to another IndexNow server (for example, www.bing.com) per the documentation.

If successful, a success message is logged. If not, an error is thrown with the response status.

Posting updates #

On each commit to main, the update script ./scripts/new-posts.js is run, as detailed in part 1.

If no new posts are detected, the logs show:

$ node ./scripts/new-posts.js
No new posts to submit

If there is a new post, the logs show (the logs from part 1 of this post):

$ node ./scripts/new-posts.js
Submitting updates for https://[MASKED]/posts/notifications-for-new-eleventy-posts-in-gitlab-part-1/
Mastodon status successfully posted (ID: 123456789012345678)
Bluesky status successfully posted
URL https://[MASKED]/posts/notifications-for-new-eleventy-posts-in-gitlab-part-1/ successfully submitted to IndexNow

In this case the domain is [MASKED] because it happens to be my Bluesky ID (the value for BSKY_ID, a masked variable that probably doesn't need to be masked).

Summary #

This post has detailed how to identify new posts in a Eleventy build, with summary data for those posts, and use that to post updates to Mastodon, Bluesky, and IndexNow. For reference, the final .gitlab-ci.yml, .eleventy.js, and ./scripts/new-posts.js files (the pieces covered in this post) are included here.

.gitlab-ci.yml

npm_install:
  image: node:20-alpine
  needs: []
  script:
    - npm ci
  artifacts:
    paths:
      - node_modules/

# Current sitemap must be retrieved before deploy for comparison.
get_current_sitemap:
  image: alpine:latest
  # With no needs, the job will run at the start of the pipeline
  needs: []
  script:
    - wget -O sitemap.xml https://<site>/sitemap.xml
  rules:
    # Site only deploys on the default branch
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
  artifacts:
    paths:
      - sitemap.xml

pages:
  image: node:20-alpine
  needs:
    - npm_install
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
  script:
    # build runs `npx @11ty/eleventy`
    - npm run build
    # Create IndexNow key file in the site output folder
    - echo $INDEXNOW_API_KEY > ./public/${INDEXNOW_API_KEY}.txt
  artifacts:
    # Ensure artifacts with the key file are not publicly visible
    public: false
    paths:
      - public/
      - posts.json

new_post_notification:
  image: node:20-alpine
  # Needs specifies artifacts to download as well as prerequisite jobs
  needs:
    # Provides node_modules folder
    - npm_install
    # Provides previous sitemap.xml
    - get_current_sitemap
    # Provides built site and posts.json
    - pages
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
  script:
    - node ./scripts/new-posts.js

.eleventy.js (unchanged from part 1)

'use strict';

const fs = require('node:fs');
const path = require('node:path');

// Global paths
const inputPath = 'src';
const outputPath = 'public';

const sanitizeTag = (tag) =>
    tag.toLowerCase().replaceAll(/[#/]/g, '').replaceAll(' ', '-');

module.exports = function (eleventyConfig) {

    // other configuration

    eleventyConfig.addFilter('stringify', (value) => JSON.stringify(value));

    eleventyConfig.addFilter('stringifyTags', (tags) =>
        JSON.stringify(tags.map((tag) => `#${sanitizeTag(tag)}`))
    );

    // Create collection of posts data for use in external notifications
    eleventyConfig.addCollection('postsData', (collection) =>
        collection.getFilteredByTag('posts').map((item) => ({
            date: item.date,
            description: item.data.description,
            inputPath: item.inputPath,
            outputPath: item.outputPath,
            tags: item.data.tags.filter((tag) => tag !== 'posts'),
            title: item.data.title,
            url: item.url
        }))
    );

    // Move the posts.json file to the root folder since not deployed
    eleventyConfig.on('eleventy.after', () => {
        const postsDataFilename = 'posts.json';
        fs.renameSync(path.join(outputPath, postsDataFilename), postsDataFilename);
    });

    return {
        dir: {
            input: inputPath,
            output: outputPath
        },
        // other configuration
    };
};

./scripts/new-posts.js

'use strict';

const fs = require('node:fs');
const { BskyAgent, RichText } = require('@atproto/api');

const sitemapFilename = 'sitemap.xml';
const postsFilename = 'posts.json';
const postsThreshold = 3;

const getNewPosts = () => {
    const sitemap = fs.readFileSync(sitemapFilename, 'utf8');
    const urlRegex = /<loc>(?<url>.+\/posts\/.+?)<\/loc>/g;
    const sitemapUrls = [...sitemap.matchAll(urlRegex)].map((match) => match.groups.url);

    const posts = JSON.parse(fs.readFileSync(postsFilename, 'utf8'));
    if (
        sitemapUrls.length === 0 ||
        posts.length === 0 ||
        Math.abs(sitemapUrls.length - posts.length) > postsThreshold
    ) {
        throw new Error(
            'Error: sitemap and posts data are invalid or out of sync'
        );
    }
    return posts.filter((post) => !sitemapUrls.includes(post.url));
};

const postMastodonStatus = async (post) => {
    const accessToken = process.env.MASTODON_TOKEN;
    // Update for the applicable Mastadon instance
    const instanceUrl = 'https://fosstodon.org';
    const data = {
        status: `${post.description}\n\n${post.url}\n\n${post.tags.join(' ')}`
    };

    const response = await fetch(`${instanceUrl}/api/v1/statuses`, {
        body: JSON.stringify(data),
        headers: {
            Authorization: `Bearer ${accessToken}`,
            'Content-Type': 'application/json'
        },
        method: 'POST'
    });

    const postStatus = await response.json();
    if (!response.ok) {
        throw new Error(`Error posting Mastadon status: ${postStatus.error}`);
    }
    console.log(`Mastodon status successfully posted (ID: ${postStatus.id})`);
};

const postBlueskyStatus = async (post) => {
    // Use Bluesky agent based on API complexity
    const agent = new BskyAgent({
        service: 'https://bsky.social'
    });

    try {
        await agent.login({
            identifier: process.env.BSKY_ID,
            password: process.env.BSKY_PASSWORD
        });

        const message = `${post.description}\n\n${post.url}`;
        // Rich formatting in posts, for example links and mentions, must be
        // specified by byte offsets, so use the RichText capabilities to
        // detect them.
        const rt = new RichText({ text: message });
        await rt.detectFacets(agent);

        const postRecord = {
            $type: 'app.bsky.feed.post',
            createdAt: new Date().toISOString(),
            // Cards are not automatically created from OG tags in the link,
            // they must be explicitly added. Note this does not include
            // images, which must be referenced, and separately posted to
            // the API.
            embed: {
                $type: 'app.bsky.embed.external',
                external: {
                    description: post.description,
                    title: post.title,
                    uri: post.url
                }
            },
            facets: rt.facets,
            text: rt.text
        };

        await agent.post(postRecord);
        console.log('Bluesky status successfully posted');
    } catch (error) {
        throw new Error(`Error posting Bluesky status: ${error.message}`);
    }
};

const postIndexNow = async (post) => {
    const apiKey = process.env.INDEXNOW_API_KEY;
    const indexNowUrl = 'https://api.indexnow.org/IndexNow';
    const postUrl = new URL(post.url);
    const { host } = postUrl;
    const data = {
        host,
        key: apiKey,
        keyLocation: `https://${host}/${apiKey}.txt`,
        urlList: [postUrl.toString()]
    };

    const response = await fetch(indexNowUrl, {
        body: JSON.stringify(data),
        headers: { 'Content-Type': 'application/json' },
        method: 'POST'
    });

    if (!response.ok) {
        throw new Error(
            `Error submitting URL to IndexNow: ${response.statusText}`
        );
    }
    console.log(`URL ${post.url} successfully submitted to IndexNow`);
};

(async () => {
    const posts = getNewPosts();
    if (posts.length === 0) {
        console.log('No new posts to submit');
        return;
    }
    const taskQueue = [];
    for (const post of posts) {
        console.log(`Submitting updates for ${post.url}`);
        taskQueue.push(
            postMastodonStatus(post),
            postBlueskyStatus(post),
            postIndexNow(post)
        );
    }

    const results = await Promise.allSettled(taskQueue);
    for (const result of results) {
        if (result.status === 'rejected') {
            console.error(result.reason.message);
        }
    }
})();

Notifications for New Eleventy Posts in GitLab - Part 1

2024-04-08T00:00:00Z

One of the challenges with deploying static sites is that there's nothing tracking any sort of site state, including when new content is published. This post presents a technique to identify newly published content on an Eleventy site and sending various notifications with content-specific data. Part 1 covers identifying the new posts and collecting post-specific data.

The example case #

This example is for an Eleventy site derived from the eleventy-base-blog and sends notifications for new published posts. You'll see that limiting to posts is merely filtering down the content and could be easily changed, but in this case it's used to avoid cases like notifications for a new tag added to a post (which could create a new tag-specific page for previously unused tags).

The example also runs in GitLab CI and publishes to GitLab pages without resorting to git diffs, using the GitLab API, or similar state comparisons to determine newly published content. While this specific case is for an Eleventy site hosted on GitLab pages, but the technique should be applicable to any CI system.

At a high level, the overall sequence is:

Determine current posts #

In order to determine new posts, there must be a source of truth for current posts for comparison. The CI pipeline itself doesn't maintain state data for the published site. It could potentially be obtained from the GitLab API if the artifacts have not expired, which may not be a good assumption. There is one straightforward source of truth, though - the sitemap.xml file on the deployed site. This assumes the site is providing a sitemap, which there are many reasons to do, and that the sitemap only contains URLs and is not a sitemap index file (linking to other sitemaps). If needed, there's an example sitemap.xml in the eleventy-base-blog for a reference implementation.

For this case, a job is added to the CI pipeline before the new site is deployed to retrieve the currently deployed site's sitemap.xml and save as an artifact for later use.

# Current sitemap must be retrieved before deploy for comparison.
get_current_sitemap:
  image: alpine:latest
  # With no needs, the job will run at the start of the pipeline
  needs: []
  script:
    - wget -O sitemap.xml https://<site>/sitemap.xml
  rules:
    # Site only deploys on the default branch
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
  artifacts:
    paths:
      - sitemap.xml

This job has rules configured to only run on the default branch since that matches when the site is deployed. In this job wget is used instead of curl simply because it's already installed in the alpine image.

Get data for all posts #

To get the data for all posts, the Eleventy build is updated to create a JSON file with all applicable data, which is then saved and made available for subsequent jobs to use.

Create posts summary JSON file #

To simplify creating the posts summary file, a new Eleventy collection is added to the Eleventy configuration.

// Create collection of posts data for use in external notifications
eleventyConfig.addCollection('postsData', (collection) =>
    collection.getFilteredByTag('posts').map((item) => ({
        date: item.date,
        description: item.data.description,
        inputPath: item.inputPath,
        outputPath: item.outputPath,
        tags: item.data.tags.filter((tag) => tag !== 'posts'),
        title: item.data.title,
        url: item.url
    }))
);

This collection gets data for all posts (any page with the tag posts), and then creates a summary of post-related data to be used for notifications. Collections are processed after all pages and the data cascade, so have access to all page data, the rendered page, input and output file paths, etc. The tags value is updated to remove the posts tag, leaving only other tags. In this cases the inputPath and outputPath are included simply to illustrate that the data is available, and could be included if those files needed to be accessed to get some information (although a goal was to avoid that and use the data that the collection already exposes).

This site uses eleventy-plugin-validate, which ensures that the required post data (title, description, tags) is available and match the required schema or the build fails. Therefore, the postsData collection does not perform duplicate data validation, although that's recommended if the data is not already validated.

In addition to the collection, two filters are added to the Eleventy configuration.

eleventyConfig.addFilter('stringify', (value) => JSON.stringify(value));

const sanitizeTag = (tag) =>
    tag.toLowerCase().replaceAll(/[#/"]/g, '').replaceAll(' ', '-');

eleventyConfig.addFilter('stringifyTags', (tags) =>
    JSON.stringify(tags.map((tag) => `#${sanitizeTag(tag)}`))
);

The stringify filter is used to properly encode some data fields for the JSON file (for example, double quotes that may be in a title or description field, which would otherwise result in invalid JSON).

The post tags is used to create hashtags, so the stringifyTags filter takes the array of post tags, encodes them, and prepends a # to make valid hashtags.

The JSON file itself is created with a new Nunjucks template (Nunjucks is used across this site, this could use any template format).

---
permalink: /posts.json
eleventyExcludeFromCollections: true
---

[
{%- for post in collections.postsData %}
    {
    "url": "{{ post.url | htmlBaseUrl(metadata.url) }}",
    "title": {{ post.title | stringify | safe }},
    "description": {{ post.description | stringify | safe }},
    "date": "{{ post.date | dateToRfc3339 }}",
    "inputPath": "{{ post.inputPath }}",
    "outputPath": "{{ post.outputPath }}",
    "tags": {{ post.tags | stringifyTags | safe }}
    }{% if not loop.last %},{% endif %}
{%- endfor %}
]

This template iterates through the collection that was previously created to generate a JSON array of posts data. There are three noteworthy items in the template:

A permalink is used to set the filename, and the file is excluded from other collections with eleventyExcludeFromCollections. This is similar to how a sitemap or RSS feed would be generated.
The title and description fields call the stringify filter. As was seen previously, this calls JSON.stringify on the value, which ensures it returns a quoted string, so the returned values are not quoted in the template.
All fields that call stringify or stringifyTags for encoding use the safe filter so they're not HTML encoded. The intent is to encode them as valid JSON, not valid HTML.

The Eleventy build saves the posts.json file in the site's outputPath (set previously in the outputPath variable), so an eleventy.after event is used to copy the file to the root directory so it's not deployed with the site (not an issue necessarily, but unnecessary).

eleventyConfig.on('eleventy.after', () => {
    const postsDataFilename = 'posts.json';
    fs.renameSync(path.join(outputPath, postsDataFilename), postsDataFilename);
});

Finally, the pages job artifacts:paths is updated to save the new file in addition to the site directory (in this case public/).

pages:
  image: node:20-alpine
  needs:
    - npm_install
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
  script:
    # build runs `npx @11ty/eleventy`
    - npm run build
  artifacts:
    paths:
      - public/
      - posts.json

Note: this project uses the npm_install job to install dependencies once (via npm ci) and save them as artifacts for all subsequent jobs where they are required. The specific template can be found here for reference.

Determine new posts #

New posts are determined by comparing the sitemap and posts summary files.

const fs = require('node:fs');

const sitemapFilename = 'sitemap.xml';
const postsFilename = 'posts.json';
const postsThreshold = 3;

const getNewPosts = () => {
    const sitemap = fs.readFileSync(sitemapFilename, 'utf8');
    const urlRegex = /<loc>(?<url>.+\/posts\/.+?)<\/loc>/g;
    const sitemapUrls = [...sitemap.matchAll(urlRegex)].map((match) => match.groups.url);

    const posts = JSON.parse(fs.readFileSync(postsFilename, 'utf8'));
    if (
        sitemapUrls.length === 0 ||
        posts.length === 0 ||
        Math.abs(sitemapUrls.length - posts.length) > postsThreshold
    ) {
        throw new Error(
            'Error: sitemap and posts data are invalid or out of sync'
        );
    }
    return posts.filter((post) => !sitemapUrls.includes(post.url));
};

The sitemap file is read, and uses a regular expression to find all <url> tags with /posts/ in the URL (rather than parsing the XML and then filtering the results). The URL is stored in the url named capture group. This is implemented on a site with existing posts, so URLs should always be found in the sitemap, and in the posts.json file. As a check for any issues, an error is thrown if either of those lists are empty (a sign of some problem), or different by more than 3 posts. The latter is arbitrary, and could probably be narrowed to 1, but was implemented as another check that there wasn't an issue processing data from either file. In my nominal workflow there should only be one new post in any pipeline. There could also be new tag pages with the blog template, but these are not included in posts. The two files are compared to determine the new posts and an array of posts data for those posts is returned.

Provide notifications for new posts #

A new job new_post_notification was created to run the script that checks for new posts and sends notifications. This is separated to isolate notification errors from pages generation and deploy errors, although these could be combined into one job.

new_post_notification:
  image: node:20-alpine
  # Needs specifies artifacts to download as well as prerequisite jobs
  needs:
    # Provides node_modules folder
    - npm_install
    # Provides previous sitemap.xml
    - get_current_sitemap
    # Provides built site and posts.json
    - pages
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
  script:
    - node ./scripts/new-posts.js

As noted, this needs the jobs that previously generated the applicable data and the node_modules folder for modules used by the ./scripts/new-posts.js script.

This site is using Eleventy v2 with CommonJS modules, so the main logic is encapsulated in an IIFE to emulate top level await.

(async () => {
    const posts = getNewPosts();
    if (posts.length === 0) {
        console.log('No new posts to submit');
        return;
    }
    const taskQueue = [];
    for (const post of posts) {
        console.log(`Submitting updates for ${post.url}`);
        taskQueue.push(
            // send updates for new posts
        );
    }

    const results = await Promise.allSettled(taskQueue);
    for (const result of results) {
        if (result.status === 'rejected') {
            console.error(result.reason.message);
        }
    }
})();

It uses the previously discussed getNewPosts function to get the summary data for the new posts, the iterates through those posts to send notifications. All tasks are pushed to an async queue and Promise.allSettled is used to ensure all tasks are executed, even on failure. If any of the tasks fail, an error is logged.

Summary #

This post has detailed how to identify new posts in a Eleventy build, with summary data for those posts. For reference, the complete .gitlab-ci.yml, .eleventy.js, and ./scripts/new-posts.js files (the pieces covered in this post) are included here.

.gitlab-ci.yml

npm_install:
  image: node:20-alpine
  needs: []
  script:
    - npm ci
  artifacts:
    paths:
      - node_modules/

# Current sitemap must be retrieved before deploy for comparison.
get_current_sitemap:
  image: alpine:latest
  # With no needs, the job will run at the start of the pipeline
  needs: []
  script:
    - wget -O sitemap.xml https://<site>/sitemap.xml
  rules:
    # Site only deploys on the default branch
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
  artifacts:
    paths:
      - sitemap.xml

pages:
  image: node:20-alpine
  needs:
    - npm_install
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
  script:
    # build runs `npx @11ty/eleventy`
    - npm run build
  artifacts:
    paths:
      - public/
      - posts.json

new_post_notification:
  image: node:20-alpine
  # Needs specifies artifacts to download as well as prerequisite jobs
  needs:
    # Provides node_modules folder
    - npm_install
    # Provides previous sitemap.xml
    - get_current_sitemap
    # Provides built site and posts.json
    - pages
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
  script:
    - node ./scripts/new-posts.js

.eleventy.js

'use strict';

const fs = require('node:fs');
const path = require('node:path');

// Global paths
const inputPath = 'src';
const outputPath = 'public';

const sanitizeTag = (tag) =>
    tag.toLowerCase().replaceAll(/[#/]/g, '').replaceAll(' ', '-');

module.exports = function (eleventyConfig) {

    // other configuration

    eleventyConfig.addFilter('stringify', (value) => JSON.stringify(value));

    eleventyConfig.addFilter('stringifyTags', (tags) =>
        JSON.stringify(tags.map((tag) => `#${sanitizeTag(tag)}`))
    );

 // Create collection of posts data for use in external notifications
 eleventyConfig.addCollection('postsData', (collection) =>
     collection.getFilteredByTag('posts').map((item) => ({
         date: item.date,
         description: item.data.description,
         inputPath: item.inputPath,
         outputPath: item.outputPath,
         tags: item.data.tags.filter((tag) => tag !== 'posts'),
         title: item.data.title,
         url: item.url
     }))
 );

    // Move the posts.json file to the root folder since not deployed
 eleventyConfig.on('eleventy.after', () => {
        const postsDataFilename = 'posts.json';
        fs.renameSync(path.join(outputPath, postsDataFilename), postsDataFilename); });

    return {
        dir: {
            input: inputPath,
            output: outputPath
        },
        // other configuration
    };
};

./scripts/new-posts.js

'use strict';

const fs = require('node:fs');

const sitemapFilename = 'sitemap.xml';
const postsFilename = 'posts.json';
const postsThreshold = 3;

const getNewPosts = () => {
    const sitemap = fs.readFileSync(sitemapFilename, 'utf8');
    const urlRegex = /<loc>(?<url>.+\/posts\/.+?)<\/loc>/g;
    const sitemapUrls = [...sitemap.matchAll(urlRegex)].map((match) => match.groups.url);

    const posts = JSON.parse(fs.readFileSync(postsFilename, 'utf8'));
    if (
        sitemapUrls.length === 0 ||
        posts.length === 0 ||
        Math.abs(sitemapUrls.length - posts.length) > postsThreshold
    ) {
        throw new Error(
            'Error: sitemap and posts data are invalid or out of sync'
        );
    }
    return posts.filter((post) => !sitemapUrls.includes(post.url));
};

(async () => {
    const posts = getNewPosts();
    if (posts.length === 0) {
        console.log('No new posts to submit');
        return;
    }
    const taskQueue = [];
    for (const post of posts) {
        console.log(`Submitting updates for ${post.url}`);
        taskQueue.push(
            // send updates for new posts
        );
    }

    const results = await Promise.allSettled(taskQueue);
    for (const result of results) {
        if (result.status === 'rejected') {
            console.error(result.reason.message);
        }
    }
})();

Details for the currently implemented notifications will be discussed in Part 2:

Posting a status to Mastodon
Posting a status to Bluesky
Sending an IndexNow notification

GitLab Code Quality Duplication Analysis With PMD CPD

2024-03-11T00:00:00Z

This post describes an alternative to the built-in GitLab Code Quality duplicate or similar code analysis, which uses Code Climate. It details four significant issues observed with the Code Climate analysis and an assessment of alternative tools for duplication analysis. This lead to the creation of a new project using PMD Copy/Paste Detector (CPD) that returns a GitLab Code Quality formatted report.

Issues with Code Climate duplication analysis #

Experience with the built-in GitLab Code Quality duplication analysis, performed by Code Climate, has shown the results to have several categories of issues. Before diving into them, the following is a brief summary of how that analysis is performed. Per the Code Climate documentation, the code is parsed into an abstract syntax tree (AST), and each node of the tree is given a mass based on the number of sub nodes. Code Climate compares each node to other nodes to find duplicates, and reports those greater than the applicable mass threshold (duplicate code or similar code). Nodes that have the same structure and values are duplicate code, and those with the same structure but different values are similar code.

The following sections detail four categories of incomplete, incorrect, or non-intuitive results. These sections compare some simplified JavaScript code samples for brevity, and where appropriate uses artificially low thresholds to find duplicate code.

Inaccurate mass calculation #

There are cases where the duplicate mass reported by Code Climate is not accurate. For example, given the following code:

export const anotherFunction = (value) => {
    if (value) {
        console.log(`value: ${value}`);
    } else {
        console.log('nada');
    }
};

const function1 = () => {
    const a = [1, 3, 5];
    console.log(`total: ${a.length}`);
    for (let i = 0; i < a.length; i++) {
        if (a[i] < 5) {
            console.log(a[i]);
        }
        // output error if value equal to 5
        if (a[i] === 5) {
            console.log('error');
        }
    }
};

const function2 = () => {
    const a = [2, 4];
    console.log(`total: ${a.length}`);
    for (let i = 0; i < a.length; i++) {
        // log if less than 5
        if (a[i] < 5) {
            console.log(a[i]);
        }
        if (a[i] === 5) {
            console.log('error');
        }
    }
};

Code Climate reports the following mass values:

The complete function anotherFunction (highlighted in yellow) is reported to have a mass of 58 (seems correct).
Part of the function function1 (the for loop highlighted in blue) is reported to have a mass of 10 (definitely incorrect).
Part of the function function2 (the if statement highlighted in yellow), which is a subset of the function1 for loop, is reported to have a mass of 19 (seems correct).

This disparity makes it challenging to impossible to trust the results, and certainly impossible to accurately tune the mass threshold to get the desired results. Testing has confirmed these mass values are being used to filter results and are not just a report display error.

Adjacent duplicates are not grouped #

As noted previously, the code duplication algorithm looks at the tree of nodes in the AST representation of the code. This means that if adjacent lines of code are duplicates, but do not have a common parent node that includes only the duplicate code, they are reported separately. For example, given the following code:

const function1 = () => {
    const a = [1, 3, 5];
    console.log(`total: ${a.length}`);
    for (let i = 0; i < a.length; i++) {
        if (a[i] < 5) {
            console.log(a[i]);
        }
        // output error if value equal to 5
        if (a[i] === 5) {
            console.log('error');
        }
    }
};

const function2 = () => {
    const a = [2, 4];
    console.log(`total: ${a.length}`);
    for (let i = 0; i < a.length; i++) {
        // log if less than 5
        if (a[i] < 5) {
            console.log(a[i]);
        }
        if (a[i] === 5) {
            console.log('error');
        }
    }
};

Code Climate reports two sets of duplicates: the individual console.log lines as one (highlighted in yellow), and the complete for loop as the other (highlighted in blue).

In addition, depending on the code in each adjacent duplicate group, the mass of the combined code may be reportable as a duplicate, but the mass of each individual line may not be, in which case the duplication is not reported at all. This is illustrated in the following example, where five lines in each function are exact duplicates, but function2 has an additional console.log line at the start.

const function1 = async (x) => {
    const url = `https://somewhere.test/${x}`;
    console.log("Fetching result for:", url);
    const response = await fetch(url);
    const result = await response.json();
    console.log("result:", result);
};

const function2 = async (x) => {
    console.log("Checking ID: ", x);
    const url = `https://somewhere.test/${x}`;
    console.log("Fetching result for:", url);
    const response = await fetch(url);
    const result = await response.json();
    console.log("result:", result);
};

Per Code Climate's algorithm, each of the five duplicate lines has a mass 12 - 16, with a total mass of 69. With the default JavaScript mass threshold of 45, Code Climate reports no duplicate code, even though the collection of adjacent lines together exceeds that threshold. The mass threshold would need to be reduced to 12 to catch the duplication in individual lines, and then Code Climate reports duplicates for each of the five lines separately.

Unidentified duplicates #

There are cases where the Code Climate duplication algorithm simply misses duplicates. For example, given the following code, which has the same function copied three times, each with the function/variables renamed and different comments:

const badWords = ['badword1', 'badword2', 'badword3'];

const function1 = (value) => {
    if (badWords.includes(value)) {
        console.log('***');
    } else {
        console.log(value);
    }
};

const function2 = (input) => {
    if (badWords.includes(input)) {
        // Redact bad words
        console.log('***');
    } else {
        console.log(input);
    }
};

const function3 = (something) => {
    if (badWords.includes(something)) {
        console.log('***');
    } else {
        console.log(something);
    }
};

Code Climate reports the two highlighted functions (the first and third) as duplicates, but the middle function is not identified as a duplicate. It's possible the comment in the middle is impacting the results, but there are other examples where this is not the case (for example, here).

Noisy similar code results #

Given the definition of similar code, Code Climate frequently reports code that is generally similar, and unlikely to be refactored. For example, given the following code:

const function1 = () => {
    const a = [1, 3, 5];
    console.log(`total: ${a.length}`);
    for (let i = 0; i < a.length; i++) {
        if (a[i] < 5) {
            console.log(a[i]);
        }
        // output error if value equal to 5
        if (a[i] === 5) {
            console.log('error');
        }
    }
};

const function2 = () => {
    const a = [2, 4];
    console.log(`total: ${a.length}`);
    for (let i = 0; i < a.length; i++) {
        // log if less than 5
        if (a[i] < 5) {
            console.log(a[i]);
        }
        if (a[i] === 5) {
            console.log('error');
        }
    }
};

Code Climate reports the four highlighted if statements as the same similar code. In this case it's unlikely that anyone would refactor the if statements to reduce duplication, especially the different comparison operators, so the similar code results can be more noisy than useful.

Alternate tool options #

Give the Code Climate issues previously discussed, a search was done looking for alternative tools. The criteria for the tools were left as vague as practical:

Tools needed to support most, if not all, of the languages that Code Climate currently supports (most was left undefined, but at least JavaScript/TypeScript and Go, since that's what I write).
Tools should detect duplicate code, where duplicate code should exclude spacing, comments, and variable names/values to the greatest degree practical. This terminology is different than Code Climate uses, where some of these cases would constitute similar code.
The tools had to be actively maintained. That was also a little vague, but to be actively maintained a tool should have a release in the last year (ideally in the last 3 months), commits in the last 3 months, and responses to at least some new project issues.
The tools must output results in some structured format that can be converted to a GitLab Code Quality report.

Some investigation led to two primary candidate tools warranting further investigation, PMD Copy/Paste Detector (CPD) and jscpd. Both of these tools take a different approach from Code Climate and use variations of the Rabin-Karp algorithm for detecting duplicate code.

PMD Copy/Paste Detector (CPD) #

PMD is primarily a static analysis tool for Java and Apex (and is itself a Java-based tool), although provides some support for 16 other languages. For comparison, it includes 273 rules for Java, but only 17 rules for JavaScript. Included with it is the PMD Copy/Paste Detector (CPD), which checks for duplicate code. PDM CPD supports 31 languages, including all of the languages supported by Code Climate.

jscpd #

jscpd is a JavaScript-based tool to check for duplicate code. It supports over 150 languages, including all of the languages supported by Code Climate.

Comparison #

In general, both tools are capable and produced similar results. Overall, PMD CPD produced more comprehensive results, as illustrated in the following examples. There were three primary areas where jscpd had limitations:

By design, jscpd only compares two sections of code, so for cases of 3 or more duplications they're reported in pairs. With this, it's not as clear that they're duplicates of the same code. Alternatively, PMD CPD reports all of the duplications together in one group.
There were some cases where jscpd reported multiple duplicates for the same code where one was a subset of the other. This was more pronounced at lower minimum token thresholds, which was used for many tests to test for the greatest number of duplicates.
The two machine-readable formats for jscpd did not include the token count for the duplicates (the XML report doesn't include tokens, and the JSON report always erroneously reports tokens: 0). Having this information is valuable in adjusting the minimum token thresholds.

The following are a few examples for comparison. As with the previous examples, these include simplified code samples for brevity, and where appropriate use artificially low thresholds to find duplicate code. The specific code identified as duplicate is highlighted in each case.

The first is a JavaScript example with code duplicated 3 times, which finds similar results, but illustrates that this is reported as different duplicates by jscpd. It also shows where PMD CPD tended to include more code as part of the duplicate.

PMD CPD results

const badWords = ['badword', 'badword2',
  'badword3'];

const myFunction = (value) => {
    if (badWords.includes(value)) {
        console.log('***');
    } else {
        console.log(value);
    }
};

const myFunction2 = (input) => {
    if (badWords.includes(input)) {
        // This is the same as the previous one
        console.log('***');
    } else {
        console.log(input);
    }
};

const myFunction3 = (something) => {
    if (badWords.includes(something)) {
        console.log('***');
    } else {
        console.log(something);
    }
};

jscpd results

const badWords = ['badword', 'badword2',
  'badword3'];

const myFunction = (value) => {
    if (badWords.includes(value)) {
        console.log('***');
    } else {
        console.log(value);
    }
};

const myFunction2 = (input) => {
    if (badWords.includes(input)) {
        // This is the same as the previous one
        console.log('***');
    } else {
        console.log(input);
    }
};

const myFunction3 = (something) => {
    if (badWords.includes(something)) {
        console.log('***');
    } else {
        console.log(something);
    }
};

const badWords = ['badword', 'badword2',
  'badword3'];

const myFunction = (value) => {
    if (badWords.includes(value)) {
        console.log('***');
    } else {
        console.log(value);
    }
};

const myFunction2 = (input) => {
    if (badWords.includes(input)) {
        // This is the same as the previous one
        console.log('***');
    } else {
        console.log(input);
    }
};

const myFunction3 = (something) => {
    if (badWords.includes(something)) {
        console.log('***');
    } else {
        console.log(something);
    }
};

The second example is in Go, showing similar results (both similar accuracy and limitations).

PMD CPD results

package main

import (
 "fmt"
)

func printOdd() {
    values := []int{1, 3, 5}
    for _, value := range values {
        if value == 5 {
            fmt.Println("***")
        } else {
            fmt.Println(value)
        }
    }
}

func printEven() {
    values := []int{2, 4}
    for _, value := range values {
        if value == 5 {
            fmt.Println("***")
        } else {
            fmt.Println(value)
        }
    }
}

jscpd results

package main

import (
 "fmt"
)

func printOdd() {
    values := []int{1, 3, 5}
    for _, value := range values {
        if value == 5 {
            fmt.Println("***")
        } else {
            fmt.Println(value)
        }
    }
}

func printEven() {
    values := []int{2, 4}
    for _, value := range values {
        if value == 5 {
            fmt.Println("***")
        } else {
            fmt.Println(value)
        }
    }
}

Given all of this data, PMD CPD was chosen for the solution.

The GitLab PMD CPD solution #

This led to the creation of the GitLab PMD CPD project, which includes a container image to run the PMD CPD analysis, and tools to convert the results to a GitLab Code Quality report.

Conversion to GitLab Code Quality report format #

Unfortunately CPM CPD can't accept any kind of template for the output, and from the available options XML is the only structured format. So, a simple Go-based CLI application was created to post-process the XML output, converting it into GitLab Code Quality report formatted JSON. The container image used to run PMD CPD is based on a Java image, and this allowed a binary that could easily be added without requiring extra dependencies.

The GitLab documentation provides details on the required schema for implementing a custom code quality report. This includes the minimum values used by GitLab for displaying the results in the merge request widget or the pipeline view - description, check name, fingerprint, severity, and start line. A review of the GitLab Code Quality and PMD CPD reports showed some other useful data, which was included. An example with the complete final schema is:

[
  {
    "categories": ["Duplication"],
    "check_name": "duplicate-code",
    "content": {
      "body": "    const a = [1, 3, 5];\n    console.log(`total: ${a.length}`);\n    for (let i = 0; i < a.length; i++) {\n        if (a[i] < 5) {\n            console.log(a[i]);\n        }\n        // output error if value equal to 5\n        if (a[i] === 5) {\n            console.log('error');\n        }\n    }\n};\n",
      "group": 1,
      "tokens": 68
    },
    "description": "Duplicate code found in 2 locations (Duplication 1). Consider refactoring.",
    "engine_name": "pmd-cpd",
    "severity": "major",
    "type": "issue",
    "fingerprint": "d0bf6d31894d6482691775a4f031e680dff4a8ab02d2a3d5be036dfdb920002e-1",
    "location": {
      "path": "internal/testdata/exampleproject/file5.js",
      "positions": {
        "begin": { "column": 23, "line": 2 },
        "end": { "column": 3, "line": 13 }
      }
    },
    "other_locations": [
      {
        "path": "internal/testdata/exampleproject/file5.js",
        "positions": {
          "begin": { "column": 20, "line": 16 },
          "end": { "column": 3, "line": 27 }
        }
      }
    ]
  }
]

A few changes were made from the original GitLab Code Quality report:

The content.body value includes the duplicate code, whereas the Code Climate report gives a long description on DRY-ness and why duplicate code can be a problem (the same for all duplicates other than the actual mass value). The specific code proved more valuable.
The total tokens as well as the group for the duplicates were added to the content object. Because tokens is included, there is no remediation_points field which the Code Climate report provides (although I'm still not sure the value of trying to score the badness of code quality issues). The group is unique to a duplication, and added to differentiate results because each duplicate location is included in the report for each occurrence (as location and as other_location).
For simplicity, all severity values are major, which seemed consistent with Code Climate results for token thresholds likely to be used in production. The actual Code Climate results, as with the mass threshold variances discussed previously, were an inconsistent mix of major/minor.
The fingerprint value is a SHA256 hash instead of MD5 to avoid any possible collisions. It is also suffixed with the occurrence number of that duplicate. This creates a unique fingerprint value for each instance of each duplicate, which causes all occurrences to be displayed in the GitLab merge request widget. GitLab displays only the last occurrence of each unique fingerprint, so with the Code Climate report only the last occurrence of each duplication is shown and the report must be opened to find the location of the other occurrences. This implementation causes all occurrences to be shown in the merge request widget with links to the specific code for each.

The following example shows the difference in how the results are displayed in GitLab.

The first three are the results from the original Code Climate report (which show only the last occurrence of each duplicate, and ungrouped duplicates). The remaining five are from the converted PMD CPD results (which show all occurrences of each duplicate). These results are from the previous examples here (file4.js) and here (file5.js).

Running analyses #

The GitLab PMD CPD container images can be found at registry.gitlab.com/gitlab-ci-utils/gitlab-pmd-cpd. The image contains all the tools to run the analysis and report conversion, and a /gitlab-pmd-cpd/pmd-cpd.sh script as a wrapper to set default values and run all aspects of the analysis and conversion. The script can be customized with the following environment variables. Note that the analysis requires specifying a language, so must be repeated for each applicable language (although that's easy to automate, as shown in the subsequent GitLab CI example).

PMDCPD_DIR_FILES: The directory or files to scan for duplicate code (default: --dir=.). This must be one of the options --dir, --file-list or --uri with the appropriate value (for example --file-list=files.txt). Some language-specific examples are shown in the subsequent GitLab CI example. See the PMD CPD docs for additional details.
PMDCPD_MIN_TOKENS: The minimum number of tokens to consider a code fragment as duplicate (default: 100).
PMDCPD_LANGUAGE: The language to use for duplicate code detection (default: ecmascript). The complete list of supported languages can be found here.
PMDCPD_RESULTS: The name of the PMD CPD results file (default: pmd-cpd-results.xml). Note that the file extension must be .xml for the conversion to GitLab Code Quality report format to work properly, and that the resulting JSON file uses the same base name (for example pmd-cpd-results.json for the default name).
PMDCPD_CLI_ARGS: Any additional PMD CPD command line arguments (default: none). See the PMD CPD docs for additional details.
- The --exclude <path> option can be used to exclude files from the scan. This option can be used multiple times. Note this does not allow globs.
- Note that the --no-fail-on-violation, --skip-lexical-errors=true, and --format="xml" values are fixed in the script.
PMD_JAVA_OPTS: This is used by PMD to set various JVM options, including the heap size. For example, PMD_JAVA_OPTS=-Xmx1024m to set the maximum heap size size to 1 GB.

Not that in an effort to avoid a fight with Oracle, ECMAScript is used instead of JavaScript ®.

Issues with shebang in JavaScript/TypeScript #

When running the CPD analysis, PMD parses each file. Since it is set to ignore lexical errors, files that fail to parse are ignored (otherwise the analysis would fail). One issue that was noted with the parser is that it considers a shebang in JavaScript and TypeScript files to be invalid syntax. This is typically seen in executable bin files in Node.js projects. There is an open PMD issue to resolve this, but until then the pmd-cpd.sh script simply removes the shebang lines before running the analysis. When running in GitLab CI in a container with an ephemeral copy of the repository only for this analysis this is not an issue. If the analysis is run locally, note that it changes these files and does not revert those changes itself.

GitLab CI jobs #

The following are example GitLab CI jobs to check for duplication in JavaScript, TypeScript, and Go.

.duplication_base:
  image:
    name: registry.gitlab.com/gitlab-ci-utils/gitlab-pmd-cpd:latest
    entrypoint: ['']
  stage: test
  needs: []
  variables:
    PMDCPD_MIN_TOKENS: 50
    PMDCPD_RESULTS_BASE: 'pmd-cpd-results'
    PMDCPD_RESULTS: '${PMDCPD_RESULTS_BASE}.xml'
  script:
    - /gitlab-pmd-cpd/pmd-cpd.sh
  artifacts:
    paths:
      - ${PMDCPD_RESULTS_BASE}.*
    reports:
      codequality: ${PMDCPD_RESULTS_BASE}.json

duplication_go:
  extends: .duplication_base
  variables:
    PMDCPD_LANGUAGE: 'go'
    PMDCPD_DIR_FILES: '--file-list=go_files.txt'
    PMDCPD_RESULTS_BASE: 'pmd-cpd-results-go'
  rules:
    - exists:
        - '**/*.go'
  before_script:
    - find . -type f -name "*.go" ! -name "*_test.go" > go_files.txt

duplication_js:
  extends: .duplication_base
  variables:
    PMDCPD_LANGUAGE: 'ecmascript'
    PMDCPD_CLI_ARGS: '--exclude ./tests/'
    PMDCPD_RESULTS_BASE: 'pmd-cpd-results-js'
  rules:
    - exists:
        - '**/*.js'

duplication_ts:
  extends: .duplication_base
  variables:
    PMDCPD_LANGUAGE: 'typescript'
    PMDCPD_CLI_ARGS: '--exclude ./tests/'
    PMDCPD_RESULTS_BASE: 'pmd-cpd-results-ts'
  rules:
    - exists:
        - '**/*.ts'

The .duplication_base template sets common defaults, including the image, default variables, script, artifacts, etc.
As noted previously, the language to check for duplicates must be specified, so there's a separate job for each language, with rules to ensure the applicable jobs run based on file extensions in the repository.
One noteworthy item to be aware of is the different uses of the PMDCPD_DIR_FILES variable.
- For JavaScript/TypeScript, the default value of --dir=. is used. This assume a model where test files, for example, are kept in a dedicated tests/ directory, which is excluded with the PMDCPD_CLI_ARGS variable. If test files are kept with the code, the model used here for Go files should be used.
- For Go, tests files are kept with the applicable packages. Since the --exclude <path> option doesn't accept globs, it can't be used to ignore all *_test.go files. So, in before_script a file is created that lists all *.go files except the *_test.go files. Doing this in before_script avoids changing the template script. The file list is then used for the duplication analysis, specified with PMDCPD_DIR_FILES: '--file-list=go_files.txt'.

These jobs could be easily extended for other required languages.

GitLab CI Pipeline for Go Projects

2024-01-28T00:00:00Z

After recently learning Go, and about to start my first project, I started putting together a GitLab CI pipeline for Go projects. After scouring the Internet, I found mostly old posts on the subject. The posted pipelines, as well as numerous actual Go project pipelines, were missing jobs in my set of goals for a CI pipeline. So I built this GitLab CI pipeline for Go projects.

Pipeline goals #

This adventure started with my first Go project, and wanting, but not finding a pipeline that met the goals that I have for just about any pipeline:

The pipeline should include all checks required to confirm the software is in a deployable state, and should generate any reporting required when deploying the software.
The pipeline jobs should use rules logic to include or exclude jobs or settings where an expectation is implied. For example, it should analyze dependencies if the project includes dependencies (if a go.sum file exists), or deploy binaries if binaries are built, but should exclude those jobs where they are not applicable.
The pipeline should allow common configuration options through variables (globally if they effect multiple jobs to avoid duplication).
The pipeline should automate deployment when the project should be deployed to ensure consistency.
The pipeline jobs should provide reports to leverage GitLab capabilities, where available. For example, code_quality, junit, and coverage reports.
The pipeline should be optimized to execute as many jobs as possible in parallel with clearly defined job dependencies, and should minimize repeating the same tasks in favor of performing them once and passing the artifacts as required.

The pipeline detailed here does not include common jobs that fall under the previously defined goals, but are already available generically in my templates, which includes:

Code quality analysis for common languages (excluding Go) that are typically included in a GitLab project. For example linting markdown, linting YAML, linting shell scripts, counting code by language.
Code security analysis including Static Application Security Testing (SAST) and secret detection.
Supply chain security analysis including several flavors of dependency vulnerability scanning and Socket dependency analysis.
Jobs to build, test, and deploy container images, which are not unique to Go.
Jobs to automatically create a GitLab release.

GitLab pipeline for Go projects #

The complete CI pipeline for Go projects is shown below, ordered by execution sequence with job dependencies shown. The first is for a Go project with dependencies.

The second is the same pipeline showing how it adapts if the project has no dependencies.

The details of all jobs are discussed in the subsequent sections, which assumes a working knowledge of both Go and GitLab CI. The included CI jobs have some simplifications for this post where specific aspects are not relevant for this introduction. Each section also includes a link to the complete job in my templates project for reference.

Global Go template #

To establish commonality across all of the jobs running Go code, a .go template was created with several global settings:

Define a common image so jobs are all running on the same version of Go.
Set GOPATH to $CI_PROJECT_DIR/.go so that it's within the project directory. This allows saving its contents as artifacts and/or cache (which must be within the project directory), and establishes a common directory for these jobs.
Set GO_BIN_DIR to a common location where all binaries are saved to and retrieved from.

.go:
  image: golang:1.21.6-alpine
  variables:
    GOPATH: $CI_PROJECT_DIR/.go
    GO_BIN_DIR: bin

Some jobs require changes to these settings, and those are specifically identified below.

Note: one impact to resetting GOPATH is that becomes the location for any go installed modules, which do install nominally, but that directory is not included in PATH. So, those modules either need to be executed from $GOPATH/bin or $GOPATH/bin needs to be added to PATH before executing.

Download dependencies #

It's a common expectation today for applications to have external dependencies, and sometimes there are many, either direct or transitive. These dependencies are required in multiple subsequent jobs, so if there are any (that is, if a go.sum file exists), then they're downloaded first in the go_mod_download job and saved as artifacts for use in those subsequent jobs. This not only saves time in subsequent jobs, but more importantly ensures that all jobs are using the same versions. They're also saved in a project-wide cache to reduce download times on subsequent pipelines.

go_mod_download:
  extends:
    - .go
  needs: []
  rules:
    - exists:
        - '**/go.sum'
  before_script:
    # Required since this is set as GOPATH
    - mkdir -p .go
  script:
    - go mod download
  cache:
    key: ${CI_PROJECT_NAME}-go
    paths:
      - .go/pkg/mod/
  artifacts:
    paths:
      - .go/pkg/mod/

This assumes that the project is not vendoring dependencies. Despite the fact the Go includes a command to specifically do it (go mod vendor), I believe vendoring promotes bad configuration management practices, potential licensing concerns, and is not recommended so it is not accounted for.

Linting #

Linting is accomplished via the lint_go job using golangci-lint, which integrates dozens of Go linting apps, running them with concurrent execution (of course, it's a Go app). A YAML file (.golangci.yml) can be used to configure the enabled linters, settings for each linter, etc. The job is setup to use the .golangci.yml in the project root if one exists, otherwise a default file is retrieved from a URL (which can be overridden via a variable, as shown in the example). The results are saved as a GitLab Code Quality JSON report uploaded to GitLab so they're available in the Code Quality merge request widget. A summary is also output to the job log.

lint_go:
  extends:
    - .go
  image: golangci/golangci-lint:v1.55.2-alpine
  needs: []
  variables:
    CONFIG_FILE_LINK: https://gitlab.com/gitlab-ci-utils/config-files/-/raw/10.1.1/Linters/.golangci.yml
  rules:
    - exists:
        - '**/*.go'
  before_script:
    - apk add jq
    # Use default .golangci.yml file if one does not exist
    - if [ ! -f .golangci.yml ]; then (wget $CONFIG_FILE_LINK) fi
  script:
    # Write the code coverage report to gl-code-quality-report.json
    # and print linting issues to stdout in the format: path/to/file:line description
    - >
      golangci-lint run --out-format code-climate $LINT_GO_CLI_ARGS |
      tee gl-code-quality-report.json |
      jq -r '.[] | "\(.location.path):\(.location.lines.begin) \(.description)"'
  artifacts:
    reports:
      codequality: gl-code-quality-report.json
    paths:
      - gl-code-quality-report.json
  allow_failure: true

Duplication #

The duplication_go job looks for duplicate code across Go files. The default duplication threshold is 50 tokens, but can be adjusted as desired via a variable. The job also defaults to checking all *.go files, but excluding all *_test.go files, since test code tends to have a lot of duplicate boilerplate code. This is done by creating a list of files to check, which is done in before_script so it can be easily overridden without impacting other job logic, and that listing is passed as a variable. The results from this job are also saved as a GitLab Code Quality JSON report uploaded to GitLab so they're available in the Code Quality merge request widget. Full details on this job can be found in the GitLab PMD CPD project, which are beyond the scope of this post.

duplication_go:
  image:
    name: registry.gitlab.com/gitlab-ci-utils/gitlab-pmd-cpd:latest
    entrypoint: ['']
  needs: []
  variables:
    PMDCPD_MIN_TOKENS: 50
    PMDCPD_LANGUAGE: 'go'
    PMDCPD_DIR_FILES: '--file-list=go_files.txt'
    PMDCPD_RESULTS_BASE: 'pmd-cpd-results-go'
    PMDCPD_RESULTS: '${PMDCPD_RESULTS_BASE}.xml'
  rules:
    - exists:
        - '**/*.go'
  before_script:
    - find . -type f -name "*.go" ! -name "*_test.go" > go_files.txt
  script:
    - /gitlab-pmd-cpd/pmd-cpd.sh
  artifacts:
    paths:
      - ${PMDCPD_RESULTS_BASE}.*
    reports:
      codequality: ${PMDCPD_RESULTS_BASE}.json

Build #

There are two build jobs, both optional, and either one or both can be included via a variable. These jobs specifically build binaries, so if the build is only in a Dockerfile, for example, they may not be required. If either job is included, the job needs are set to the go_mod_download job to use those downloaded dependencies.

Build current operating system and architecture #

The go_build job performs a build with only the current OS/architecture (Linux/amd64 in the given container) and saves the binary as an artifact, with the binary name and main package directory specified via variables.

go_build:
  extends:
    - .go
  needs:
    - job: go_mod_download
      optional: true
  rules:
    - if: $GO_BUILD_CURRENT
  variables:
    GO_BUILD_NAME: $CI_PROJECT_NAME
    GO_MAIN_DIR: '.'
  script:
    - go build $GO_BUILD_ARGS -o ./${GO_BIN_DIR}/${GO_BUILD_NAME} $GO_MAIN_DIR
  artifacts:
    paths:
      - ./${GO_BIN_DIR}/${GO_BUILD_NAME}*

Build multiple operating systems and architectures #

The go_build_multi job also performs a build, but in a parallel matrix job for a variety of OS/architecture combinations, all of which are saved as artifacts with the OS/architecture appended to the binary name. If the OS is Windows, the job adds the .exe extension to the binary name. This can be overridden as necessary.

go_build_multi:
  extends:
    - .go
  needs:
    - job: go_mod_download
      optional: true
  rules:
    - if: $GO_BUILD_MULTI
  variables:
    GO_BUILD_NAME: $CI_PROJECT_NAME
    GO_MAIN_DIR: '.'
  parallel:
    matrix:
      - TARGET_OS: 'darwin'
        TARGET_ARCH: ['amd64', 'arm64']
      - TARGET_OS: 'linux'
        TARGET_ARCH: ['amd64', 'arm64']
      - TARGET_OS: 'windows'
        TARGET_ARCH: ['386', 'amd64']
  script:
    - if [ $TARGET_OS == 'windows' ]; then export EXTENSION='.exe'; fi
    - >
      env GOOS=${TARGET_OS} GOARCH=${TARGET_ARCH}
      go build $GO_BUILD_ARGS -o ./${GO_BIN_DIR}/${GO_BUILD_NAME}_${TARGET_OS}_${TARGET_ARCH}${EXTENSION} $GO_MAIN_DIR
  artifacts:
    paths:
      - ./${GO_BIN_DIR}/${GO_BUILD_NAME}*

The parallel matrix runs all jobs in parallel for efficiency.

Unit and integration testing #

There are two test jobs, both of which run in a custom container image. This image is based on the golang image, with two sets of pre-installed dependencies:

Tools to generate JUnit formatted test results and Cobertura coverage reports.
OS dependencies including gcc that are required to run with -race , which requires Cgo.

There is a .go_test template that specifies the version of this image (which also extends the .go template), and all test jobs extend it rather than the .go template.

.go_test:
  extends:
    - .go
  image: registry.gitlab.com/gitlab-ci-utils/container-images/go-test:2.0.1
  artifacts:
    when: always

As with the build jobs, if there are dependencies each test job needs the go_mod_download job.

Tests with code coverage #

The go_test job runs all tests, with code coverage, and processes the results generating several additional reports for ingestion by GitLab and reviewing pipeline results.

All tests are run, generating coverage information for all packages. Well, the goal is all packages, but there's an open issue on Go not including coverage data for packages with no tests. The workaround used here is to use the -coverpkg argument to specify package to include in the coverage data. This is specified through the variable GO_COVER_PACKAGES. Ideally this could simply be set to ./..., but this is one of the places where redirecting GOPATH has an impact. Since it is within the current working directory, then ./... includes all the project dependencies. So, GO_COVER_PACKAGES is specified by using go list ./..., which only includes packages in the current module, then converted to a comma delimited list, which -coverpkg requires. This is done in before_script so it can be easily overridden without impacting other test logic.

before_script:
  # Get list of packages, replace newline with comma, and remove training comma.
  - export GO_COVER_PACKAGES=$(go list ./... | tr '\n' ',' | sed 's/,$//')

Even with this, there's still one quirk, as noted in the linked Go issue - packages with no tests are not included in the coverage data. The best workaround seems to be creating a test file with just the package declaration (for example package foo_test), then the coverage is included as zero percent for all functions in that package (for foo in this example). The fix for the original Go issue is merged and scheduled for Go 1.22.

When the tests are run, in addition to generating coverage data the test output is also saved to a file (tests.txt) that is used for reporting. Once completed, the go tool cover command is used to output a summary of coverage information by function, and the total coverage is also collected by GitLab.

script:
  - go test -v -coverpkg="$GO_COVER_PACKAGES" -coverprofile=coverage.out -covermode=count ./... | tee `tests.txt`
  - go tool cover -func coverage.out

The various test summary reports are generated in after_script, which ensures that errors generating reports do not fail the job. This includes:

Converting the coverage data to a Cobertura coverage report, ingested by GitLab. This allows coverage data to be visible in the merge request diff view.
Converting the coverage data to an HTML coverage report, which is saved as an artifact.
Converting the test output to a JUnit report, ingested by GitLab. This allows the test summary to be presented in a GitLab merge request widget.

This results in the following job configuration:

go_test:
  extends:
    - .go_test
  needs:
    - job: go_mod_download
      optional: true
  before_script:
    # Get list of packages, replace newline with comma, and remove training comma.
    # If `./...` is used, coverage includes packages in GOPATH since set within
    # the cwd.
    # Done in before_script so it can be easily overridden.
    - export GO_COVER_PACKAGES=$(go list ./... | tr '\n' ',' | sed 's/,$//')
  script:
    - go test -v -coverpkg="$GO_COVER_PACKAGES" -coverprofile=coverage.out -covermode=count ./... | tee tests.txt
    - go tool cover -func coverage.out
  after_script:
    # Generate Cobertura coverage report for GitLab
    - gocover-cobertura -by-files < coverage.out > coverage.xml
    # Generate HTML coverage report
    - go tool cover -html coverage.out -o coverage.html
    # Generate JUnit report for GitLab
    - go-junit-report < tests.txt > junit.xml
  coverage: '/total:\s+\(statements\)\s+\d+.\d+%/'
  artifacts:
    expose_as: 'Go Test Coverage Report'
    paths:
      - coverage.html
    reports:
      coverage_report:
        coverage_format: cobertura
        path: coverage.xml
      junit: junit.xml

Test race conditions #

The go_test_race job runs all tests with the -race flag to check for race conditions. It is configured with CGO_ENABLED, as required to use -race. No artifacts are saved as they are all covered with the go_test job.

go_test_race:
  extends:
    - .go_test
  needs:
    - job: go_mod_download
      optional: true
  variables:
    # Cgo is required for -race.
    CGO_ENABLED: 1
  script:
    - go test ./... -race

If tests with -race run slowly on gitlab.com runners, consider using more capable runners. By observation (not testing), this job is CPU bound and appears to scale roughly linearly with CPU (that is, doubling CPU cuts the execution time in half), at least through the large runners. See GitLab SaaS runner details in the docs.

Analysis jobs #

There are several application analysis jobs performed under the umbrella of confirming the software is in a deployable state and reporting required when deploying the software.

Missing or extraneous dependencies #

The go_mod_tidy job identifies whether the go mod tidy command produces any changes, functionally identifying any missing or extraneous dependencies in the project which can, and should, be fixed with go mod tidy. This job only runs if a go.sum file exists, and if there are dependencies needs the go_mod_download job.

go_mod_tidy:
  extends:
    - .go
  needs:
    - job: go_mod_download
      optional: true
  rules:
    - exists:
        - '**/go.sum'
  script:
    - |
      # Check if go mod tidy produces any changes
      if [ ! -f "go.mod" ]; then echo "go.mod file not found" && exit 2; fi
      cp go.mod go.mod.before
      cp go.sum go.sum.before
      go mod tidy
      diff go.mod.before go.mod && diff go.sum.before go.sum

Dependency updates #

The go_mod_updates job identifies any potential updates in direct dependencies. This goes a step beyond the core Go command go list -m -u, which identifies potential updates in any dependencies - direct or transitive. Updates to transitive dependencies can be out of the project's direct control for various reasons, so they are ignored in this case to avoid unnecessary job failures. The job uses the go list -m -u JSON results to find direct dependency updates, and fail any are found. This job only runs if a go.sum file exists, and if there are dependencies needs the go_mod_download job. It also allows failure since its primary purpose is to identify the available dependency updates.

go_mod_updates:
  extends:
    - .go
  needs:
    - job: go_mod_download
      optional: true
  rules:
    - exists:
        - '**/go.sum'
  allow_failure: true
  before_script:
    - apk add jq
  script:
    # Waiting on https://github.com/golang/go/issues/40364 to allow listing
    # only direct dependencies so don't have to check JSON.
    - |
      # Check for available updates to any direct module dependencies
      UPDATES=$(go list -u -m -json all | \
        jq -c -r 'select(.Update != null and .Indirect == null) | "\(.Path) \(.Version) [\(.Update.Version)]"')
      printf "$UPDATES\n"
      if [ $(echo $UPDATES | wc -w) != "0" ]; then exit 1; fi

Note there is an open issue to allow go list to show only updates to direct dependencies, which will likely be used instead once implemented.

Software Bill of Materials (SBOM) #

The go_sbom job generates a CycloneDX formatted SBOM, which is an industry standard format for reporting dependencies used by an application. Not that the SBOM generation is slightly different for the type: app, mod, or bin. See the documentation for complete details.

go_sbom:
  image: !reference [.go, image]
  variables:
    SBOM_TYPE: 'app'
    SBOM_ARGS: '-main=./'
  needs:
    - job: go_mod_download
      optional: true
  rules:
    - exists:
        - '**/go.sum'
  before_script:
    - go install github.com/CycloneDX/cyclonedx-gomod/cmd/cyclonedx-gomod@v1.4.1
  script:
    - cyclonedx-gomod $SBOM_TYPE $SBOM_ARGS -json=true -licenses=true -output=sbom.json -output-version=1.4 $SBOM_PATH
  artifacts:
    paths:
      - sbom.json

Deployment #

The go_deploy job runs on tag pipelines if either build job runs and publishes the binaries to GitLab's generic package registry under the git tag. An example showing the package registry for GitLab's release-cli tool can be seen here (also the inspiration for the job).

go_deploy:
  extends:
    - .go
  image: alpine:latest
  variables:
    # See https://docs.gitlab.com/ee/user/packages/generic_packages/#publish-a-package-file
    PACKAGE_REGISTRY_URL: '$CI_API_V4_URL/projects/$CI_PROJECT_ID/packages/generic/$CI_PROJECT_NAME'
  needs:
    - job: go_build
      optional: true
    - job: go_build_multi
      optional: true
  rules:
    # Deploy if tag pipeline and at least one build job was executed
    - if: ($GO_BUILD_CURRENT || $GO_BUILD_MULTI) && $CI_COMMIT_TAG
  before_script:
    - apk add curl
  script:
    - cd $GO_BIN_DIR
    - |
      for FILE in *; do
        echo "Deploying: $FILE";
        curl --header "JOB-TOKEN: ${CI_JOB_TOKEN}" --upload-file ${FILE} ${PACKAGE_REGISTRY_URL}/${CI_COMMIT_TAG}/${FILE};
      done

In the future this could be updated to publish to GitLab's Go proxy, but this is currently experimental, likely to be for the foreseeable future, and is disabled on gitlab.com.

Collection of CI jobs #

To simplify use of the pipeline, a Go-Build-Test-Deploy collection has been created, including all of the Go jobs. As previously discussed, each job contains rules logic ensuring it only runs when appropriate, so this allows an individual Go project's .gitlab-ci.yml file to be the following, and all appropriate jobs are run.

include:
  - project: 'gitlab-ci-utils/gitlab-ci-templates'
    ref: 'main'
    file:
      - '/collections/Go-Build-Test-Deploy.gitlab-ci.yml'

# Optional variables to include the build and deploy jobs
variables:
  GO_BUILD_CURRENT: 'true'
  GO_BUILD_MULTI: 'true'

Summary #

This post detailed a complete GitLab CI pipeline for Go projects, including dependency management, code analysis, code quality, build, test, and deployment. The pipeline is designed to be flexible, allowing jobs to be included or excluded as desired, and optimized for parallel execution. It also includes several reports to leverage GitLab capabilities for review of pipeline results.

GitLab Semgrep SAST Analysis - But More

2024-01-07T00:00:00Z

GitLab continues to migrate Static Application Security Testing (SAST) to Semgrep, and makes this available to all GitLab tiers. This analysis only includes the rules that GitLab manages, but there are many more available in the Semgrep Rules project. This post details how to combine the two to get a more comprehensive analysis.

GitLab Static Application Security Testing #

For several years, since June 2020 with GitLab 13.3, GitLab has provided the capability for Static Application Security Testing (SAST) to analyze code for potential vulnerabilities in all GitLab tiers, including Free. There are some additional features available in GitLab Ultimate, but the core analysis and customization of scanners and settings is available to all.

GitLab SAST supports over 20 languages and frameworks, and is easy to implement by adding the following to your .gitlab-ci.yml file:

include:
  - template: Jobs/SAST.gitlab-ci.yml

That's it. The SAST template has rules configured to only run the appropriate analyzers based on the languages used in the codebase (checked by file extension). Each analyzer runs in a dedicated job to parallelize the work and avoid the need for Docker-in-Docker.

The most broadly used analyzer is Semgrep, an open source engine for static code analysis, combined with a set of GitLab developed and managed SAST rules. Semgrep currently covers about one third of the languages and frameworks supported by GitLab's SAST analysis. GitLab also has a broader effort in work to migrate all SAST analysis to Semgrep, as noted in the documentation and tracked in this epic.

One challenge with GitLab only using their own managed rules is a limited number of rules, which is more evident in some languages. Here are the current counts of GitLab SAST rules for Semgrep by language.

Language	Rules
C	62
C#	20
Go	30
Java	64
JavaScript	11
Python	70
Scala	87
TypeScript	11

Semgrep, however, has it's own set of rules, and many, many more rules in some cases, as shown here for a subset of these languages.

Language	Rules
Go	41
JavaScript	163
TypeScript	165

This post details what is believed to be the best way to consolidate these rules to have the most comprehensive SAST testing.

Use Semgrep as a standalone job? #

The Semgrep documentation does have instructions for running in GitLab CI, so why not just use that? There are a few significant issues with that implementation.

There's a strong push to use Semgrep Cloud Platform, which provides a custom dashboard for managing rules, integration to provide MR comments, additional features with their paid subscriptions, etc. The goal for this use case is to augment what GitLab SAST already provides, so having another tool to use to manage results is not desirable. And some of the capabilities provided there, for example diff-aware scanning, links to code, and MR comments are already better integrated through GitLab's existing merge request capabilities, especially for GitLab Ultimate.
The Semgrep rules are mix of several categories: security, best practices, correctness, and maintainability. The goal of this job is SAST analysis, so only the security rules are desired. For any of the languages of concern, there are extremely capable language-specific tools for linting best practices, correctness, and maintainability. Additionally, the GitLab SAST results feed into a SAST report, managed through GitLab's security capabilities, and that workflow is not intended to manage other code quality resources (for example, the option to for Security team approval for violations).
In some cases, both GitLab and Semgrep have implemented the same rules. So, using both would result in duplicate findings, and the overhead and frustration that comes with that for SAST rules that are already notoriously noisy.
The Semgrep rules, either individually or as a collection, are not versioned. The rules each have a SHA256 hash to verify integrity, and a last updated timestamp, but nothing comparable to a "release" that indicates rule changes, and the implications of those changes, to the user.

The combined solution #

So, a different solution was needed that addresses the concerns previously identified. This solution:

Builds on GitLab's semgrep-sast job, which includes their Go-based analyzer that wraps Semgrep, runs it with a specific set of rules, and provides results in a GitLab SAST formatted report.
Adds the applicable Semgrep security rules from the official repository (including any associated example files). As noted previously, this repository has no tagged versions, so the latest rules from the default branch are used.
Identifies Semgrep rule changes to users, and is released with semantic versioning.
Eliminates rules duplicated between GitLab SAST and Semgrep. For simplicity, and to make this project a pure extension of GitLab SAST, the GitLab rules are used where there are duplicates.

This led to the GitLab Semgrep Plus project. The container image is derived from the GitLab Semgrep analyzer. The semgrep-rules/ directory in the repository includes copies, unchanged, of all security rules and examples from Semgrep rules for Go, JavaScript, and TypeScript, which are included in the analysis. This makes changes to these rules and code examples explicitly visible to users.

This could contain rules for the many other languages that Semgrep supports, but these are the only languages currently in use in the GitLab CI Utils projects where this was initiated.

The rules, without examples, are included in the /rules directory in the container image. GitLab's Semgrep analyzer is designed to use all rules in that directory, so no other changes are required to include them in the analysis. A shell script with all of the applicable logic is used to update the rules so that the changes are made consistently with an automated process.

Managing ongoing Semgrep rule changes #

The project is setup to automatically detect changes to the Semgrep Rules project and incorporate any changes to security rules for the specified languages.

Since Semgrep Rules does not identify versions, Renovate is used to track the latest git ref of the development branch, and it initiates a merge request with any ref changes. The CI pipeline is setup with a trigger job to check for rule updates (if that check has not been done). This job is triggered in another project to isolate the credentials that can push commits back to this project.

The triggered pipeline clones this project, clones the Semgrep Rules project, re-runs the shell script to update rules, and pushes any updates. The commit uses --amend to leave only one commit with an updated message, and is made with Renovate bot account, so Renovate continues to propagate changes to Semgrep Rules if the merge request is not immediately merged. The commit message is also updated to indicate rule changes were incorporated. Since the updated Semgrep rules are pushed, a new pipeline is triggered, and the updated commit message is used in the job rules to stop an infinite loop of triggered rule update pipelines.

Usage in GitLab CI #

The GitLab SAST template is already setup to specify the container image for all SAST jobs via variables, so the implementation in GitLab CI is simply:

include:
  - template: Jobs/SAST.gitlab-ci.yml

semgrep-sast:
  variables:
    SAST_ANALYZER_IMAGE: registry.gitlab.com/gitlab-ci-utils/gitlab-semgrep-plus:latest

With that the Semgrep rules are included in the analysis, and the corresponding report.

The Year in Review 2023

2023-12-29T00:00:00Z

A lot has happened in the last year - the resurgence of this blog, I created six new software projects, and I shipped almost 200 software releases. In addition, I learned Go, started publishing NPM packages with provenance, and generative AI and large language models took over the world. This post reviews it all.

Blog posts #

The year brought with it the return of this blog. I had a goal of averaging two blog posts per month. That certainly seemed achievable, but the result didn't quite measure up - a grand total of 10 posts (including this, and the kickoff - not sure those really count), averaging less than one per month. Even though that didn't meet my original goal, I feel good about the results. The posts were either detailing significant new software releases or solutions to significant GitLab, CI/CD, or DevOps problems. And a lot of research was done on a few more big ones coming in 2024, including:

An enhanced GitLab Semgrep SAST (static application security testing) job with several hundred additional rules.
A comprehensive GitLab CI pipeline for Go projects.
An assessment of issues with GitLab's Code Quality template and the alternative I've been using for the last 6 months (it's getting long, it will probably be a 2 or 3 part series).

Software projects #

Most of this year was busy with open source software projects (probably why I didn't meet my original blog post goal). I started the following six new projects this year:

Project	Description
GitLab PMD CPD	A container image to run PMD's Copy/Paste Detector (CPD) and convert the results to GitLab Code Quality report format. This provides an alternative to GitLab's built-in Code Quality template for identifying duplicate code.
GitLab Semgrep Plus	GitLab's `semgrep` container image augmented with hundreds of additional Node.js/JavaScript/Typescript and Go rules from Semgrep's rule repository.
GitLab UI Plus	A Chrome browser extension to augment the GitLab UI for improved UX (very much a work in progress).
Go Test	A container image with OS packages and Go tools necessary for performing Go testing and producing testing reports for GitLab.
Lighthouse	A container image to run Lighthouse and Lighthouse CI with the latest Chromium release.
Syft	An Alpine Linux based container image to run Syft to allow Software Bill of Materials (SBOMs) generation in GitLab CI where a shell is required.

Between those projects and my other existing projects I shipped 199 releases.

Project	Releases
Bin Tester	4 (5.0.0, 4.0.1, 4.0.0, 3.0.1)
CI Logger	2 (6.0.0, 5.1.1)
Config Files	8 (9.2.0, 9.1.0, 9.0.1, 9.0.0, 8.0.0, 7.2.0, 7.1.0, 7.0.0)
Docker Dependency Check	19 (2.0.4, 2.0.3, 2.0.2, 2.0.1, 2.0.0, 1.16.12, 1.16.11, 1.16.10, 1.16.9, 1.16.8, 1.16.7, 1.16.6, 1.16.5, 1.16.4, 1.16.3, 1.16.2, 1.16.1, 1.16.0, 1.15.9)
Docker Hugo	1 (2.0.0)
Docker PowerShell Scripts	5 (4.5.0, 4.4.0, 4.3.0, 4.2.0, 4.1.0)
Docker Puppeteer	7 (11.0.0, 10.2.0, 10.1.0, 10.0.0, 9.3.0, 9.2.0, 9.1.0)
Docker Sokrates	2 (2.1.1, 2.1.0)
Docker curl jq	1 (2.0.0)
ESLint Config Standard	12 (25.0.0, 24.0.1, 24.0.0, 23.1.1, 23.1.0, 23.0.0, 22.1.0, 22.0.0, 21.0.0, 20.1.0, 20.0.0, 19.0.2)
GitLab CI Env	5 (9.1.0, 9.0.0, 8.1.0, 8.0.0, 7.1.0)
GitLab CI Templates	40 (26.0.0, 25.0.0, 24.2.0, 24.1.0, 24.0.1, 24.0.0, 23.1.0, 23.0.0, 22.2.0, 22.1.0, 22.0.0, 21.0.2, 21.0.1, 21.0.0, 20.2.1, 20.2.0, 20.1.0, 20.0.0, 19.1.1, 19.1.0, 19.0.0, 18.1.0, 18.0.1, 18.0.0, 17.0.1, 17.0.0, 16.3.0, 16.2.0, 16.1.0, 16.0.3, 16.0.2, 16.0.1, 16.0.0, 15.3.1, 15.3.0, 15.2.2, 15.2.1, 15.2.0, 15.1.0, 15.0.0)
GitLab PMD CPD	5 (2.0.0, 1.1.0, 1.0.0, 0.6.0, 0.5.0)
GitLab Pa11y CI	7 (7.4.0, 7.3.1, 7.3.0, 7.2.0, 7.1.1, 7.1.0, 7.0.3)
GitLab Pipeline Timeline	2 (0.2.1, 0.2.0)
GitLab Releaser	6 (7.0.1, 7.0.0, 6.0.0, 5.0.1, 5.0.0, 4.0.4)
GitLab Semgrep Plus	11 (4.0.0, 3.3.0, 3.2.0, 3.1.0, 3.0.0, 2.0.0, 1.1.0, 1.0.3, 1.0.2, 1.0.1, 1.0.0, 0.5.0)
Go Test	4 (2.0.0, 1.0.2, 1.0.1, 1.0.0)
Lighthouse	23 (4.3.0, 4.2.0, 4.1.1, 4.1.0, 4.0.0, 3.2.0, 3.1.0, 3.0.2, 3.0.1, 3.0.0, 2.1.1, 2.1.0, 2.0.6, 2.0.5, 2.0.4, 2.0.3, 2.0.2, 2.0.1, 2.0.0, 1.2.0, 1.1.1, 1.1.0, 1.0.0)
Pa11y CI CLI Summary Reporter	1 (3.0.0)
Pa11y CI HTML Reporter	7 (6.0.2, 6.0.1, 6.0.0, 5.1.1, 5.1.0, 5.0.4, 5.0.3)
Pa11y CI Reporter Runner	4 (4.0.0, 3.0.1, 3.0.0, 2.0.5)
Pa11y HTML Reporter Plus	6 (2.0.1, 2.0.0, 1.1.1, 1.1.0, 1.0.5, 1.0.4)
Pagean	4 (9.0.0, 8.0.4, 8.0.3, 8.0.2)
Releaselog	5 (5.0.0, 4.0.1, 4.0.0, 3.0.4, 3.0.3)
StyleLint Config Standard	5 (16.0.0, 15.0.0, 14.0.0, 13.0.1, 13.0.0)
Syft	2 (1.1.0, 1.0.0)

…and that's not even counting the simple app I put together to pull all of this new project and software releases data from the GitLab API and format it 😉.

A noteworthy aspect is that seven of my NPM packages are now published with package provenance, with the others setup and waiting on their next release. It's super easy to setup (for example see the GitLab instructions), and has significant potential for improving the security of the NPM ecosystem. So, if you're not doing it, why not?

This was also the year of generative AI and large language models. Do I think they'll replace developers and we should all be afraid for our jobs? No. Do I think I was more productive with at least moderate use of GitHub Copilot and ChatGPT (my two tools of choice)? Yes - it's like having an intern to help, with all the good and bad that comes with that.

I'm a Gopher #

Last, but certainly not least, on my list of accomplishments this year was learning Go, putting together a viable CI/CD pipeline for Go in GitLab, and releasing a Go application (GitLab PMD CPD v2.0.0). I'm sure I won't abandon Node any time soon, but I do like many things about Go:

Go doesn't try to be everything for everyone. It is what it is, it's opinionated, and that's okay.
Go has simple syntax, with fewer than 30 keywords.
Every language should have defer and the simplicity of concurrency that comes with a goroutine.
The Go standard library has extensive capabilities, limiting the need for including other dependencies. Coming from working on mostly Node projects for a while, and the mess that is NPM dependencies, being able to write a functional Go application with no dependencies was a little shocking.

There are also a lot of aspects that are… … … I'll go with "quirky." I'm sure I'll write more about that at some point.

Looking forward to 2024 #

I think the only thing certain at this point is uncertainty, but here are a few things that I am looking forward to:

The release of pa11y@7.0.0 and pa11y-ci@4.0.0 should be in early 2024, bringing both tools back to what I'd call a sustained state. I've made numerous contributions to both projects, and have several of my own that are plugins to pa11y/pa11y-ci, so I'm excited to see this milestone.
I'm not convinced that the Node ecosystem will ever fully make the move from CommonJS to ESM, as much as some people want that. But, in the interest of moving forward I do plan to migrate all of my CLI-based Node projects to ESM (not libraries). That may also include the move to TypeScript, we'll see…
I mentioned it in preceding sections, but 2023 brought a lot of research and some new projects to optimize GitLab CI pipelines, so I am looking forward to finishing those posts and sharing the results.

Managing GitLab License Policies With No Dependencies

2023-11-17T00:00:00Z

GitLab's license approval policies provide a powerful and flexible means of managing dependency license approvals. Using these policies requires performing GitLab's Dependency Scanning to generate and report a Software Bill of Materials (SBOM). For the cases where there are no dependencies, Dependency Scanning is not run, no SBOM is created, and the license policies are considered failed and require approval. For projects without dependencies this is an added hassle, and this post proposes a solution.

The solution for no dependencies #

In GitLab 16.4, license approval policies were added as an enhancement to the previous License Scanning template. The documentation detailing the implementation indicates that Dependency Scanning should be enabled, then add the appropriate license approval policies, and the policies are enforced. Unfortunately, the Dependency Scanning job rules are optimized to only run when there are dependency lockfiles present, so when they are not it does not run, no SBOM is created, and the license approval policies fail and require approval even though there are no dependencies. Not the end of the world, but a hassle.

The solution for these cases is simple, and is actually captured in the GitLab documentation - ensure that a CycloneDX SBOM is created in some job and provided to GitLab as reports:cyclonedx. This then triggers license scanning using the contents of the SBOM.

This post covers non container-based projects, otherwise Container Scanning should be run to provide an SBOM.

Always run dependency scanning? #

One option investigated is simply overriding the rules for the Dependency Scanning template to run on all pipelines. Theoretically this should produce an SBOM with no dependencies, which is valid. Unfortunately, attempts with multiple projects showed inconsistent results. In some cases the SBOM was created with no dependencies, as expected, but in others no SBOM was created and no errors were identified by Gemnasium (GitLab's Dependency Scanning tool). There was no obvious commonality for these two cases.

Syft to the rescue #

So, another tool must be used to generate the SBOM. GitLab simply expects a CycloneDX formatted SBOM, a well established standard. The chosen solution here is the Syft application from Anchore. Syft is designed to create SBOMs from a host of different ecosystems, scanning either container images or file systems. It includes support for C/C++, Go, Java, JavaScript, Python, Ruby, Rust, Swift, and others. See the documentation on supported ecosystems for complete details. There are many other tools available, but Syft seemed to provide the most comprehensive single solution, and in tests consistently produced a valid CycloneDX SBOM with or without dependencies.

There is a Syft container images on Docker Hub, but this image is built from scratch and has no shell available (see the Dockerfile). Using an image with no shell in GitLab CI results in a log like the following because GitLab requires a shell to execute the script (see supported shells):

Executing "step_script" stage of the job script
Using docker image sha256:c9b97dbf39ec0138c627c8bd3b56e5d879bc0b8304f05db9dde6db9daff6d129
  for anchore/syft:v0.97.1 with digest
  anchore/syft@sha256:abc8d4310c54b56dd1e789d5f60b8ebc43f472652b34971d4b0d0dbed7f4ebda
Cleaning up project directory and file based variables
ERROR: Job failed (system failure): Error response from daemon: OCI runtime create failed:
  container_linux.go:380: starting container process caused: exec: "sh":
  executable file not found in $PATH: unknown (exec.go:78:0s)

So, a new Syft image was created that includes Syft in an Alpine Linux image so a shell is available. The details of this image are not addressed here, but feel free to post questions in the comments (or the project issues).

The GitLab CI job #

First, setup a job to simply run Syft and provide an SBOM:

syft_sbom:
  image:
    name: registry.gitlab.com/gitlab-ci-utils/container-images/syft:latest
    entrypoint: ['']
  script:
    - /syft/syft $CI_PROJECT_DIR -o cyclonedx-json=syft.cdx.json
  artifacts:
    paths:
      - syft.cdx.json
    reports:
      cyclonedx:
        - syft.cdx.json

This uses the new Syft image noted previously, with the entrypoint overridden to nothing. Syft is executed in the script with options to scan the current directory and produce a CycloneDX formatted file. This is saved as an artifact and uploaded as reports:cyclonedx to satisfy the license approval policy needs. This job could be used to always generate an SBOM, but in this example Dependency Scanning is be allowed to do its job where it can, and just use Syft to fill in the cases with no dependencies.

To finalize the rules for when Syft should be run, first examine the Dependency Scanning template rules. A subset is included below for the primary Gemnasium job.

.gemnasium-shared-rule:
  exists:
    - '**/Gemfile.lock'
    - '**/composer.lock'
    - '**/gems.locked'
    - '**/go.sum'
    - '**/npm-shrinkwrap.json'
    - '**/package-lock.json'
    - '**/yarn.lock'
    - '**/pnpm-lock.yaml'
    - '**/packages.lock.json'
    - '**/conan.lock'

gemnasium-dependency_scanning:
  rules:
    - if: $DEPENDENCY_SCANNING_DISABLED == 'true' || $DEPENDENCY_SCANNING_DISABLED == '1'
      when: never
    - if: $DS_EXCLUDED_ANALYZERS =~ /gemnasium([^-]|$)/
      when: never
    - if: $CI_COMMIT_BRANCH &&
          $GITLAB_FEATURES =~ /\bdependency_scanning\b/ &&
          $CI_GITLAB_FIPS_MODE == 'true'
      exists: !reference [.gemnasium-shared-rule, exists]
      variables:
        DS_IMAGE_SUFFIX: '-fips'
        DS_REMEDIATE: 'false'
    - if: $CI_COMMIT_BRANCH &&
          $GITLAB_FEATURES =~ /\bdependency_scanning\b/
      exists: !reference [.gemnasium-shared-rule, exists]

There's a rule template identifying the dependency lockfiles that triggers the job (this is important later), then a standard set of rules used in GitLab security scans:

An option for a variable DEPENDENCY_SCANNING_DISABLED to disable the Dependency Scanning job
An option to disable Gemnasium via the DS_EXCLUDED_ANALYZERS variable
Otherwise, if dependency_scanning is an available feature (the GITLAB_FEATURES variable is set by GitLab based on the license), execute the job if one of the available lockfiles exists (with both FIPS and non-FIPS variants)

There are similar jobs for the Maven and Python Gemnasium scanners that follow the same pattern.

Pulling this all together is a CI file that includes both the Dependency Scanning template and the Syft job defined previously, with overridden rules to only run in the appropriate cases.

include:
  - template: Dependency-Scanning.gitlab-ci.yml
  - local: /jobs/Syft.gitlab-ci.yml

# Included from Syft.gitlab-ci.yml - the job defined above
syft_sbom:
  rules:
    # If any of of these files exist, Gemnasium will be run and create an SBOM
    - exists: !reference [.gemnasium-shared-rule, exists]
      when: never
    # If any of of these files exist, Gemnasium-maven will be run and create an SBOM
    - exists: !reference [.gemnasium-maven-shared-rule, exists]
      when: never
    # If any of of these files exist, Gemnasium-python will be run and create an SBOM
    - exists: !reference [.gemnasium-python-shared-rule, exists]
      when: never
    # Otherwise, run Syft to create an SBOM, even if empty
    - if: $CI_COMMIT_BRANCH

In this case use the .gemnasium-shared-rule* templates defined in the Dependency Scanning template and setup rules to never run the Syft job if any of them are present, then finally a rule to run on any other branch pipelines (to match the GitLab template, although this could include tags, MRs, etc).

Managing Alpine Linux Based Container Images With Renovate

2023-06-30T00:00:00Z

With it's default configuration, Renovate does a great job of managing container image tag updates. One limitation is that for updates that include an OS in the tag, for example the Alpine Linux version in python:3.11.3-alpine3.17, Renovate only updates Python image tags matching that version, which may limit updates, and doesn't identify any OS updates. This post details how to configure Renovate to work around that limitation for Alpine images.

This post assumes you have a working knowledge of Renovate and covers some more advanced topics. You can find more introductory information in the Renovate docs.

Docker versioning overview #

Container image tag versions can be wildly different, sometimes chaotic, but there are some general conventions. Renovate tries its best to leverage them with docker versioning, which is the default for all container image tags.

Renovate treats any SemVer-like initial number as the "version," and it maintains the "version" with the given specificity, so:

node:18 is updated to node:19
node:18.16 is updated to node:18.17
node:18.16.0 is updated to node:18.16.1

Anything after the first hyphen in an image tag is treated as the "compatibility" indicator. Renovate only identifies updates that maintain this "compatibility." So node:18.16.0-alpine is updated to another -alpine image, for example node:18.16.1-alpine.

Complete details are available in the Renovate docs.

Docker versioning and OS versions #

In most cases maintaining image "compatibility" is desired, as you wouldn't want python:3.11.3-slim-bullseye updated to python:3.11.4-bullseye or python:3.11.4-alpine since they're completely different base images.

This can also be a challenge, though, in cases where the OS version changes. For example python:3.11.3-alpine is updated to python:3.11.4-alpine, but what if you're using a specific OS version, like python:3.11.3-alpine3.16? You might notice that python updates stop appearing even though new python images are released. In this case the Python v3.11.3 image was the last version released on Alpine 3.16, so Renovate doesn't recognize any newer image tags that maintain "compatibility," and there are no further updates.

The other challenge is that images with updated OS versions are not identified as updates, so you might not know that Alpine 3.17 images were available starting with Python 3.7.15, or Alpine 3.18 images were available starting with Python 3.7.16. The same issue exists with other OSs, including Debian and Ubuntu. If you're using python:3.11.4-bullseye, Renovate does not identify an update to python:3.11.4-bookworm.

Why track OS versions #

Given the previously identified issues, why would someone want to track specific OS versions if it can prevent future updates? As with almost everything in software, different use cases drive different solutions.

As an example from Docker Hub, the following tags all point to the latest python image: alpine, 3-alpine, 3.11-alpine, 3.11.4-alpine, 3.11.4-alpine3.18. If the uses case is fairly generic, for example a Python-based linting tool, it may not be overly sensitive to Python or Alpine changes, so using python:alpine may be appropriate. For something like the base image for a containerized application, it's best to be as specific as possible. This causes Renovate to identify any changes - application or OS - and make the appropriate update, initiating a CI pipeline. This ensures any change is tested before being merged into the codebase.

The workaround #

The way to work around the Renovate constraints is to keep the docker manager for the docker versioning, and add a regex manager to setup a separate versioning mechanism for the Alpine changes. The regex manager allows a fully customized Renovate configuration to manage dependencies that the other managers do not cover, and is the only way to have Renovate version the same dependency multiple ways. It also requires more complex configuration, which is detailed below.

Thanks to the Renovate team for their help sorting out this solution.

Image tag format #

As noted in the preceding example, there is not strict versioning in container image tags, so investigating tags of interest in tracking revealed the following list:

docker:23.0.6-alpine3.17
docker:23.0.6-dind-alpine3.17
node:18.16.0-alpine3.17
python:3.11.4-alpine3.17
mcr.microsoft.com/powershell:7.3-alpine-3.17

The example below covers these cases, but other image tags may drive Renovate configuration tweaks.

Renovate configuration #

The following section details the configuration of the regex manager to version the Alpine OS. For any values that are regular expressions, they're specified as the JSON string representation of the regular expression, so any \ must be escaped as \\.

The fileMatch property is required and identifies regular expressions to match files to check. In this case set it to ["\\.gitlab-ci\\.yml$", "Dockerfile"], which checks all GitLab CI files (assuming they end in .gitlab-ci.yml) and all Dockerfiles (which matches Dockerfile, Dockerfile.<value>, or <value>.Dockerfile). This could be updated with any other required files.

The matchStrings property is required and identifies regular expressions to match the dependencies that the regex manager should manage (in this case container images). To keep it as flexible as possible it has a depName capture group with the dependency name, and a currentValue capture group with the version. So, for the image and tag that is (?<depName>[\\S]+):(?<currentValue>[\\S]+), using non-whitespace (\S) to allow any valid characters for both.

To constrain the matchStrings further, the regular expression is updated with the keywords indicating a container image. For Dockerfiles, that's simply FROM <image_name>. For .gitlab-ci.yml files, the following cases are covered:

image:
  name: <image_name>

image: <image_name>

services:
  - <image_name>

Note this does not cover all variations of container images in services (it's missing at least two), but includes all that I am currently using. See the GitLab services docs for additional details.

To cover these cases preface the previous matchString with each of them OR'd together, which results in:

"matchStrings": [
  "(?:image:\\s+name:\\s*|image:\\s*|services:\\s+-\\s+|FROM\\s+)(?<depName>[\\S]+):(?<currentValue>[\\S]+)"
]

See here for a full breakdown of this regular expression, with examples.

The datasourceTemplate property specifies the data source type, which is docker.

For versioning image tags, specifically trying to override the default docker versioning behavior, specify a versioningTemplate property telling Renovate how to extract data from the currentValue (that is, the image tag). The default behavior is described previously, which is going to be reversed to track the Alpine versions. Taking the previous python example, the tag 3.11.4-alpine3.17 has compatibility = 3.11.4-alpine, major version = 3, and minor version = 17. Using the previous data on different tags formats gives:

"versioningTemplate": "regex:^(?<compatibility>[\\S]*\\d+\\.\\d+(?:\\.\\d+)?(?:[\\S]*)?-alpine-?)(?<major>\\d+)\\.(?<minor>\\d+)(?:\\.(?<patch>\\d+))?$"

See here for a full breakdown of this regular expression, with examples.

The final Renovate configuration for this regex manager is shown below:

{
  "$schema": "https://docs.renovatebot.com/renovate-schema.json",
  "extends": ["config:base", "docker:enableMajor"],
  "regexManagers": [
    {
      "description": "Manage Alpine OS versions in container image tags",
      "fileMatch": ["\\.gitlab-ci\\.yml$", "Dockerfile"],
      "matchStrings": [
        "(?:image:\\s+name:\\s*|image:\\s*|services:\\s+-\\s+|FROM\\s+)(?<depName>[\\S]+):(?<currentValue>[\\S]+)"
      ],
      "versioningTemplate": "regex:^(?<compatibility>[\\S]*\\d+\\.\\d+(?:\\.\\d+)?(?:[\\S]*)?-alpine-?)(?<major>\\d+)\\.(?<minor>\\d+)(?:\\.(?<patch>\\d+))?$",
      "datasourceTemplate": "docker"
    }
  ]
}

The limitations #

In general, the default docker versioning and regex versioning work independently, and should identify updates for either or both, as applicable. The one known limitation is when both identify a major update, or both identify a minor/patch update. This can be seen with images like python:3.11.3-alpine3.17, which has a minor version update to Python 3.11.4 and also a "minor" version update to Alpine 3.18 (since major/minor are defined this way in the custom versioning). So, Renovate sees two python-3.x updates, and one of them is ignored (in the Renovate logs you'll see the message "Ignoring upgrade collision"). With this example the python:3.11.4-alpine3.17 update alone is processed, and once it's merged the python:3.11.4-alpine3.18 update is processed.

Bin Tester v4.0.0 Released

2023-05-28T00:00:00Z

Bin Tester v4.0.0 was released today with several noteworthy changes, including changing the function exports, changing the argument to the test functions, adding TypeScript type declarations, and deprecating support for Node 14 and 19.

Bin Tester overview #

Bin Tester provides a function to simplify testing of a package's bin commands. It spawns a child process, executes the command in a specific folder with specific arguments/environment variables, and returns the results - exit code, any Error thrown, and the full text of stdout and stderr.

Changes in v4.0.0 #

There are three noteworthy changes in v4.0.0, all of which could be breaking. Please see the release notes for other minor changes.

Exported test function and argument changes (BREAKING) #

Previously the test function used the default export, but it was moved to the named bin export to allow for future expansion of the package. The bin function also now accepts a single options object argument with properties for all of the previous arguments.

The following is an example test jest with the previous API. As can be seen, the number possible of arguments has gotten long over time, which makes it difficult to specify a subset of the arguments, and unclear which arguments are actually required for test.

// bin-tester@3.0.1
const bin = require('bin-tester');

it('should successfully execute and output JSON results to stdout if JSON format specified', () => {
    const testArgs = ['-f', 'json'];
    const commandName = 'my-command';
    // Previously API was:
    //   bin(binArguments, workingDirectory, packageDirectory, environment, commandName, timeout)
    const results = await bin(testArgs, './', './', {}, commandName);

    expect(results.code).toBe(0);
    expect(results.stderr).toBe('');
    expect(results.stdout).toMatchSnapshot();
});

To update for the change in v4.0.0:

The bin function is now required/imported as a named export.
The required bin function arguments should be encapsulated in one options object.

These changes are shown in the example below. In this case only the two required arguments are specified (as opposed to the 5 in the preceding example).

// bin-tester@4.0.0
const { bin } = require('bin-tester');

it('should successfully execute and output JSON results to stdout if JSON format specified', () => {
    const options = { binArguments: ['-f', 'json'], command: 'my-command' };
    const results = await bin(options);

    expect(results.code).toBe(0);
    expect(results.stderr).toBe('');
    expect(results.stdout).toMatchSnapshot();
});

The complete list of properties for the options object can be found in the README.

Added TypeScript type declarations (BREAKING) #

The package now contains built-in TypeScript type declarations, part of a bigger effort to add types for all of my library-type packages. This is only BREAKING if you have implemented custom types for this package.

Deprecated support for Node 14 and 19 (BREAKING) #

Support for Node 14 (which went end of life on April 30, 2023) and Node 19 (which goes end of life on June 1, 2023) has been deprecated. The package continues to be tested against all current and LTS releases (^16.13.0 || ^18.12.0 || >=20.0.0). Once Node releases are end of life, any identified vulnerabilities are not patched, and this is intended to actively discourage keeping unsupported version of Node in use.

GitLab CI Pipeline for Eleventy

2023-04-03T00:00:00Z

This post details the GitLab CI pipeline used for this blog, which is built with Eleventy. It's based on a collection of GitLab CI templates that have evolved over several years for my published NPM packages with a collection of end-to-end tests used for web applications and a few unique jobs added specifically for Eleventy and Nunjucks templates. It's meant as an illustration of a reasonably comprehensive CI pipeline for an Eleventy static site, maximizing the level of automated testing, leveraging built-in GitLab capabilities where practical, and optimizing parallelization and pipeline speed.

CI pipeline philosophy #

The following summarizes the general philosophy that I use for any CI pipeline. Some pieces are GitLab specific, but most are general platform independent recommendations based on my experience.

The CI pipeline should execute all checks that are required to confirm that the application is in an acceptable configuration for deployment. This includes checking multiple configurations where applicable (for example light/dark mode, desktop/mobile configurations, OS/platform/framework versions).
CI jobs should be deterministic, with job failure meaning a failure to execute properly or an unacceptable result that should not being deployed. There may be jobs with interpreted results (for example Lighthouse performance score changes, dependency vulnerabilities), but passing jobs should mean the results were successfully generated to be interpreted. Jobs that fail, but allow failure, resulting in the pipeline passing are a trap to be avoided. This can eventually leads to not reviewing the job results, assuming the failure cause is known, and the job failing for another reason goes unnoticed.
Some might argue that CI pipelines should be highly optimized between what runs in a merge request and what runs on the primary branch. In general that is a mistake in GitLab CI. The GitLab merge request widgets are optimized to show changes since in the context of that merge request, for example, if you run a Code Quality report you only see the changes in that merge request, not all issues. If there is no reference job in the default branch, then the all results are identified as changes. Unless there are long-running jobs that need to be optimized or other specific drivers, there is benefit to running consistent pipelines.
The pipeline should leverage built-in GitLab capabilities where appropriate. In this pipeline that includes analyses like code quality and some security related testing. There's no point in re-inventing the wheel unless there's a specific driver.
The pipeline should be optimized to execute as many jobs as possible in parallel with clearly defined job dependencies, and should minimize repeating the same tasks in favor of performing them once and passing the artifacts as required.
Jobs with external dependencies (for example dependency vulnerability scanning) should be run on a regular schedule, even with no commits to the repository.
Deployment should be performed via an automated scripted process for consistency (even if manually initiated).

GitLab CI pipeline for Eleventy #

The complete CI pipeline for this site is shown below, ordered by execution sequence with job dependencies shown.

For efficiency, all jobs are defined with explicit needs, as can be seen in the preceding example. This ensures that jobs start as early as possible, and that only the artifacts required for the job are downloaded and extracted (GitLab downloads the artifacts from all previous jobs unless otherwise specified).

There are 28 jobs in the pipeline, which are grouped into the following functional categories:

Installing dependencies #

The npm_install job installs all npm dependencies and saves then as artifacts. This allows the install to be performed only once since it is an expensive operation (in time and I/O) and the dependencies are then made available to any subsequent jobs that require them.

Language specific linting and formatting #

Linting and format checking is performed for all languages used in this project. I use a very opinionated linting configurations for all tools to ensure consistency and maintainability throughout the codebase and to check for common issues. Development is done in a mix of VSCode and GitLab's WebIDE, so linting and formatting is not always checked there. When working VSCode, linting and format checking is performed is as a git pre-commit hook. The one place where linting and format checking is always performed is in CI.

The challenge with the pre-commit hook and CI cases is that the resulting logs are not reviewed unless there is an error. Operating this was for a long time on many projects has driven me to update all linting rule configurations to remove all warnings. Rules are either important enough to be errors or they're not and they're disabled. The errors then fail the pre-commit hook or CI pipeline. In some cases there are acceptable deviations, but these exceptions are then disabled in the code with the appropriate rationale.

The specific linting/formatting jobs and tools are detailed in the table below. Most of these are probably familiar names, other than maybe djLint. It's a Python tool for linting and formatting Nunjucks, Django, Handlebars, and many others. There's no other tool I've found in that space that comes close.

Job	Tool
`lint_css`	`stylelint`
`lint_js`	`eslint`
`lint_md`	`markdownlint`
`lint_nunjucks`	`djlint`
`lint_prettier`	`prettier`
`lint_yaml`	`yamllint`

Linting tool configuration files #

There are a few tools used by this project that offer linting tools to validate their configuration files. These jobs perform those checks to identify any formatting errors.

The lint_pageanrc job checks the Pagean configuration file for validity. See the End-to-End Testing section for more details on Pagean. Formatting errors would also be found when running Pagean, but this provides a job that explicitly identifies a configuration error, and this can be run much earlier in the pipeline to fail fast if there is an issue.

The lint_renovate job checks the Renovate configuration file for validity. Renovate is used by this project for automated dependency updates, is quite powerful/configurable, and is highly recommended. The Renovate process is run in another project, so an error in the Renovate configuration would not otherwise be detected in this pipeline.

Build site #

The build_test_site job builds the site in a test configuration. This site undergoes exhaustive end-to-end testing, so some production pieces are excluded from the test build (for example analytics scripts), and the configuration sets the pathPrefix to match the URLs used for that end-to-end testing.

Code analysis #

The pipeline includes a couple of general code analysis jobs that don't fall into one of the specific categories listed below.

The code_count job uses cloc to count lines of code by language. This is primarily used to generate a GitLab Metrics report, which can then provide a change in code count by language in a merge request. This is helpful as a check that an unexpected change wasn't accidentally committed (for example if a new blog post was added, which should only be a change in lines of markdown, but the lines of JavaScript changed as well, then something is wrong).

The code_quality job run the GitLab Code Quality testing. The results for the entire project can be viewed, as can the changes in a merge request.

End-to-end testing #

End-to-end testing, which involves analysis of the site in a browser, is performed to assess site performance, accessibility, and other quality control checks (for example no console errors, no broken links) as outlined below.

Lighthouse is run against one page to get a representative set of data in both desktop (the lighthouse_desktop job) and mobile (the lighthouse_mobile job) configurations. This data then populates a GitLab Metrics report, which identifies any changes in Lighthouse scores in a merge request. See this post for more details on setting up Lighthouse and generating a metrics report.

Accessibility testing is performed using Pa11y CI with both light mode (the pa11y_ci job) and dark mode (the pa11y_ci_dark job) themes. Pally CI can be configured to run with different accessibility test runners, and these jobs run both the HTML_CodeSniffer and Axe runners. There is also accessibility data provided by Lighthouse, but the Pa11y CI results in general are more comprehensive (especially when using both runners).

The pagean job run Pagean, a test framework I started several years ago with a collection of tests intended to be run against all pages in a site - broken links, errors written to the console, horizontal scroll bar, page load time, and others. See the documentation for complete details.

Software composition analysis #

Software Composition Analysis (SCA) involves a set of tools used to manage risks from a project's dependencies. This is a broad category that includes tools to catalog the project dependencies, check those dependencies for known vulnerabilities or other security risks, check for unused or deprecated dependencies, etc.

Two tools are used to check the project's dependencies for known vulnerabilities. The owasp_dependency_check job runs OWASP Dependency Check against the project. This is a tool maintained by the OWASP foundation, and checks not only package manifests like a lockfile, it also scans all files in the project to check for any known vulnerabilities. This allows it to recognize cases where dependencies are embedded and/or not identified, but also means the scan is slower because it has to check every file in the project. It uses data primarily from the U.S. National Vulnerability Database (NVD), populated from the Common Vulnerabilities and Exposures (CVE) database, the effective international standard for identifying vulnerabilities. It also includes data from other applicable vulnerability databases. The results not only identify any known vulnerabilities, but also provide background on the vulnerabilities and potential remediation steps.

The osv_scanner job runs OSV Scanner to check for vulnerabilities identified in a lockfile or other package manifest. It checks that listing of vulnerabilities against the Open Source Vulnerabilities (OSV) Database, maintained by Google. This includes data from multiple sources, but is focused on open source vulnerabilities. Because it is using a defined listing of dependencies it runs quickly. For this specific project, it is functionally no different than running npm audit since the GitHub Advisories database is integrated into the OSV Database, but this project leverages CI templates used for a variety of projects in different languages, so this provides a broader solution.

OSV Scanner is currently undergoing testing here to determine if the data is reliable enough, and reported in a timely enough fashion, that it could replace owasp_dependency_check in projects where a listing of dependencies is available. So far that has not been shown to be the case. As this is being written, there is a new vulnerability identified by OWASP Dependency Check over a week ago that is still not reported in a GitHub Advisory, and therefore not by OSV Scanner.

Socket has taken a different approach than many other tools in the area of dependency security to provide a detailed assessment of the code for NPM (and other) packages to identify specific capabilities. For example, it identifies that a package access files on disk, accesses the network, spawns external processes, sends telemetry, and many other checks. These actions are expected in some cases, but this provides the insight to identify cases where it may not be. The socket job run the Socket CLI for any merge requests with dependency changes to assess any policy violations. At this point their GitHub app is more mature, so this is a recent addition still undergoing assessment to determine the best way to use the results in GitLab. It's a tool worth taking a look at because it really does provide the most comprehensive assessment of a package's codebase and potential issues that I've seen.

The lint_lockfile job runs npm-lockfile-lint to check that packages in the lockfile are properly retrieved from acceptable sources. In this case it ensures that packages are only pulled from NPM, over HTTPS, that the integrity field has a SHA512 hash, and that the URL matches the package. This is done to verify the integrity, fidelity, and vulnerability reporting for all dependencies (with all of these settings configurable based on project needs).

The depcheck job runs depcheck to identify any project dependencies that are not used or packages that are used but are not dependencies. It does have some limitations, but finds references for packages in .js/.ts files, in scripts running packages bin commands, or in the configuration files of many major utilities (for example eslint, jest, mocha, prettier).

The npm_check job run npm-check to identify and packages with updates. While Renovate is used for automated updates, this is a backup to identify cases where the Renovate job is failing to make updates.

Static Application Security Testing #

Static Application Security Testing (SAST) tools analyze source code to find coding patterns that represent potential security issues. These analyses all focus on this project's code, not any project dependencies. Several SAST analysis tools are used, as outlined below.

The GitLab SAST template includes a series of language-specific SAST analysis tools, with rules configured to run those applicable to your codebase. In this, with a Node.js/JavaScript project that includes the nodejs-scan-sast and semgrep-sast jobs.

The secret_detection job checks for any secrets (for example passwords, access tokens/keys) committed to the git repository. This job is included from the GitLab Security Detection template.

The unicode_bidi_test job runs anti-trojan-source to identify any bidirectional Unicode characters, which could be an indication of a Trojan Source Attack.

Site deployment #

As noted in the preceding example, the build_test_site job builds the site in a test configuration, so it must be rebuilt with the appropriate settings for deployment.

The pages job runs only for commits to the default branch and builds the site in the production configuration and deploys it to GitLab Pages.

GitLab provides the capability to create a review app with an instance of an application with the changes in a merge request. Unfortunately, this capability is not natively available for GitLab pages. To accomplish this, the pages_review_app jobs run on merge request pipelines to leverage some alternate GitLab capabilities to accomplish the same result. See this post for details on how to set up a GitLab review app for an Eleventy site.

Passing a GitLab CI Job For a Failing Script

2023-02-21T00:00:00Z

There are some cases where the expected outcome of a CI job script is failure. One prominent use case is the testing of tools and container images that are intended for CI-based analyses. This post details techniques for GitLab CI scripts that allow the job to pass when the script fails to accurately reflect the expected result.

CI jobs expecting failure #

GitLab CI considers a job successful if the script completes with an exit code of 0, and considers it failed with any other exit code. In most cases that logic is appropriate, but there are some cases where a job is expected to fail. Two examples seen frequently revolve around CI pipelines for the testing of testing tools - integration testing of testing tools, and testing container images intended for use with testing tools.

To manage these cases, one obvious option for those familiar with GitLab CI is to use allow_failure, which allow failure for any exit code when set to true. This can be enhanced by using allow_failure with exit_codes, which accepts an array of exit codes and only those codes allow failure. For example:

some_job:
  allow_failure:
    exit_codes:
      - 123
      - 456

This only allows failure for exit codes 123 and 456, and any others fail the job and the pipeline. While this allows the pipeline to proceed and pass, it still leaves the job, and ultimately the pipeline, passing with failure.

The value of a green pipeline #

Everyone strives to have a green pipeline status with all jobs passed as a definitive indication that all jobs resulted in the expected outcome. In the case where a job passed with failure, people either don't, or certainly don't want to have to, examine the log for a job on every pipeline to see why it failed. It could be the expected result of the job using the techniques previously noted, but could be a variety of other issues - runner failure, failure to load the container image, script execution error, network error, job timeout - anything that can cause the job to fail. Even if the job log is initially examined on each pipeline, it's human nature to normalize that result and stop looking over time. At that point other problems may occur that go unnoticed, possibly until merged or released and result in a bigger issue. The goal should be all jobs passing since any passing with failure case is ultimately an indeterminate result.

Passing a job when the script fails #

The results of a command can be reversed by leveraging the logical And (&&) and Or (||) operators.

The And (&&) operator specifies a command that is executed if preceding command is successful (exit code 0). So in the example foo && bar, the bar command only executes if the foo command passes.
The Or (||) operator specifies a command that is executed if preceding command fails (exit code > 0). So in the example foo || bar, the bar command only execute if the foo command fails.

Putting all of that together results in the script below, which in this case runs pa11y-ci, a CI-based web accessibility testing tool, against a URL known to have accessibility issues (so, command is expected to fail):

some_job:
  script:
    - pa11y-ci && exit 1 || exit 0

This runs the pa11y-ci command. If the command is successful, which in this case is actually a failure, then it exits 1 and the job fails. If the pa11y-ci command fails, which is expected, then exit 0 and the job succeeds.

This is a good solution for cases where the script command only fails with one specific exit code. If there are multiple possible exit codes, then this could mask an error in the job. In this particular case, pa11y-ci exits with 2 if any accessibility errors are found (what's tested for), but exits with 1 for other failures. So, if pa11y-ci fails to execute properly, this job would pass and mask the failure - not the desired result.

Passing only for certain exit codes #

The preceding technique can be taken a step further by adding logic to specifically check the exit code from the previous command and take the appropriate action, as in the following example script:

some_job:
  script:
    - pa11y-ci && exit 2 || if [ "$?" -eq "2" ]; then exit 0; else exit 1; fi

This runs the pa11y-ci command. If it's successful, which again is actually a failure, then with the && operator the script exits 2 and the job fails. In this case 2 was chosen to indicate a passing script to differentiate it from other errors.

If the pa11y-ci command fails, which is expected, then the || operator runs the if command. The exit code from the previous command is represented by $?, so if that is equal to 2, which is the expected result (that is, pa11y-ci detected an accessibility error), then exit 0 and the job passes. Otherwise, there is a different error so exit 1 and the job fails.

Setup GitLab Review Apps with Eleventy

2023-02-09T00:00:00Z

GitLab Pages provide an easy means of deploying a site hosted on GitLab, but GitLab does not provide support for creating Review Apps for a Pages site. This post outlines a reusable technique to work around that and setup Review Apps with Eleventy to enable creation of a unique, browsable instance of a site with the changes in a merge request.

Overview of GitLab review apps #

GitLab Review Apps allow the creation of temporary environments deploying the code changes in a merge request. Once configured, a new environment is created for each merge request, allowing review of a running version of the application. A link to that application is provided in the merge request widget to simplify access. That temporary environment is destroyed after a specified time period, or when the merge request is merged.

Currently, GitLab only allows a single Pages site and doesn't support the deployment of Review Apps for Pages, although there is an open issue to enable that capability in the future. Until that is implemented, this technique allows adding that capability for Eleventy sites.

GitLab pages setup #

To start with a common foundation, a job to deploy the site to GitLab Pages is configured. The requirements to deploy a Pages site are a job in the pipeline named pages that saves the site as artifacts in the public/ directory. This job builds on that with a few additional capabilities.

The jobs uses the node:lts-alpine container image, a lightweight image capable of running Node projects.
The rules are set to only deploy the Pages site for pipelines on the default branch.
A variable BUILD_TYPE is added to denote this is a pages build. This is used in the eleventy configuration to control aspects of the Eleventy build.
The job sets up a production environment, which enables the built-in GitLab features for tracking Pages deployments. The url should be set to the URL of the final site.
The script installs all dependencies and runs the npm build script, assuming this builds the Eleventy site (for example npx @11ty/eleventy). These steps are included here to have a complete job definition, but depending on the pipeline those actions could be done in another job and its artifacts used in the pages job.

The complete pages job is shown below.

pages:
  image: node:lts-alpine
  stage: deploy
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
  variables:
    BUILD_TYPE: pages
  environment:
    name: production
    deployment_tier: production
    url: https://<site_url>/
  script:
    - npm ci
    - npm run build
  artifacts:
    paths:
      - public

Review apps setup #

There are several steps to configure Review Apps, including enabling Review Apps for the project, setting up the CI job, and some Eleventy configuration updates.

Enabling review apps #

To configure Review Apps they must be enabled for the project, which can be done per the instructions in the GitLab documentation.

The job artifacts workaround for review apps #

Without a dedicated means of deploying a Review App for a GitLab Pages site, the workaround used here relies on the fact that CI job artifacts are actually served by the GitLab Pages server with a URL following the format https://<namespace>.<pages_domain>/-/<project>/-/jobs/<job_id>/artifacts/<file path>.

As an example, for this site's GitLab project that URL would be something like https://aarongoldenthal.gitlab.io/-/aarongoldenthal/-/jobs/1234567890/artifacts/public/index.html (based on the job ID).

For an Eleventy site, the initial URL should be the index.html file in the project root directory. For consistency with the pages job, it's assumed that the project is in the public/ directory, so the <file path> should be public/index.html. The other parameters - <namespace>, <pages_domain>, <project>, and <job_id> - can all be taken from GitLab predefined variables, which provides the most generic job definition allowing it to be re-used in multiple projects on either gitlab.com or a self-hosted GitLab instance.

The differences from the pages job are:

The rules are set to only run on merge request pipelines. If merge request pipelines are not being used, details are also provided below for using branch pipelines instead.
For a Review App, a unique environment:name must be set up. In this case the CI_COMMIT_REF_SLUG is used, which is the branch name. This allows multiple simultaneous Review Apps, but only one per branch. The url is set to the format described previously.

The complete pages_review_app job is shown below.

pages_review_app:
  image: node:lts-alpine
  stage: deploy
  rules:
    # If not using MR pipelines this can be set to run on branch pipelines
    # that are not the default branch with:
    # if: $CI_COMMIT_BRANCH && $CI_COMMIT_BRANCH != $CI_DEFAULT_BRANCH
    - if: $CI_MERGE_REQUEST_ID
  variables:
    BUILD_TYPE: pages
  environment:
    name: review/$CI_COMMIT_REF_SLUG
    deployment_tier: testing
    url: https://$CI_PROJECT_ROOT_NAMESPACE.$CI_PAGES_DOMAIN/-/$CI_PROJECT_NAME/-/jobs/$CI_JOB_ID/artifacts/public/index.html
  script:
    - npm ci
    - npm run build
  artifacts:
    paths:
      - public

At this point running a merge request pipeline exposes a Review App, which opens the home page, but there are two issues that still need to be addressed:

The home page opens properly with the Review App, but any root-relative path references are incorrect. They are pointing to locations relative to /, but with this URL format the / directory is actually the public/ directory, which is nested 7 directories deep. This affects any root-relative paths in <a>, <img>, <link>, <script> or other tags.
Links between pages point to directories, expecting the web server to serve default pages. In the case of artifacts this does not occur, so a 404 response is returned.

Both of these issues are addressed in the following sections.

Fixing path references with eleventy configuration #

To resolve the root-relative path issues, the Eleventy configuration needs to be updated to dynamically set the pathPrefix property. This updates all of the URLs to reflect the site being deployed in a sub-directory. For test or production (i.e GitLab Pages) builds, the default / path is used, but it's set to the appropriate directory for Review Apps.

To simplify working with GitLab predefined variables the gitlab-ci-env package is used in this example. This package returns an object with all GitLab predefined variable values, hierarchically organized by context, which can simplify cases where a lot of predefined variables are used. If this is not desired, see the package documentation for a complete mapping of properties to the actual predefined environment variable names.

The buildType uses the previously specified BUILD_TYPE variable. If this is not specified, it defaults to test, intended to be a configuration used for other testing. The isReviewApp value is set to true if the build is run in CI in a merge request pipeline, which matches the preceding CI job definition. In this case these values are only used to set the pathPrefix, but could be exposed for use in other places (for example excluding analytics code in Review Apps).

The pathPrefix is finally set using these two values. If the buildType is not pages, or if isReviewApp is false, the default path is used. Otherwise, this is a build for a Review App and the path prefix as specified in the pages_review_app job is used. The pathPrefix finally needs to be specified in the object returned from the Eleventy configuration file.

As noted previously, feature branch pipelines could be used instead of merge request pipelines. In that case the logic for isReviewApp should be updated to match the pages_review_app rules.

In this case the output directory is set to public as noted in the preceding CI jobs. This can be set to other values, but the resulting site output directory must be copied to public in both Pages CI jobs to be collected as CI artifacts.

The complete Eleventy configuration updates are shown below.

const gitlabEnv = require('gitlab-ci-env');

const defaultBuildType = 'test';
const defaultPathPrefix = '/';

const buildType = process.env.BUILD_TYPE || defaultBuildType;

const isReviewApp = gitlabEnv.ci.isCI && gitlabEnv.ci.mergeRequest.id;

const pathPrefix =
    buildType === 'pages' && isReviewApp
        ? `/-/${gitlabEnv.ci.project.name}/-/jobs/${gitlabEnv.ci.job.id}/artifacts/public/`
        : defaultPathPrefix;

module.exports = function (eleventyConfig) {
    ...
    return {
        pathPrefix,
        ...
        dir: {
            ...
            output: 'public'
        }
    };
};

Fixing links within the site #

The last issue to resolve is that links within the site point to directories, and GitLab does not serve the default index.html files. To resolve this, the output HTML files from the Eleventy build are post-processed to update the links.

The find command is used to search the ./public directory (find ./public) for any files (-type f) whose name matches *.html (-name "*.html"). Then a command is executed on each file (-exec <command> "{}" +).

The sed command is used to update the links. The -i argument makes the changes to the files in place. The -E argument allows the use of extended regular expressions, which provides more consistent behavior of ? and + (in this case they must be escaped to be literal values, otherwise they behave as special characters). The regular expression finds any <a> element with an href value with a root-relative path (that is, starting with /), which includes the public directory, and an optional subdirectory. It also captures any other attributes preceding or following the href attribute. The sed command then inserts index.html before the href closing ".

For a more detailed explanation of the complete regular expression a great tool is regex101, which breaks it down in detail and allows interactive testing. See this link for this specific regular expression and some examples.

This command is run in after_script so that the same script can be re-used for all Pages jobs.

pages_review_app:
  ...
  after_script:
    - >
      find ./public -type f -name "*.html" -exec
      sed -i -E 's/(<a[^>]*href="\/.+\/public\/(.*\/)?)("[^>]*>)/\1index.html\3/' "{}" +

Refactoring the pages jobs #

Looking at both complete jobs there are a lot of common elements, so the final jobs below extract those common components to a template .pages which is then extended for both the pages and pages_review_app jobs.

.pages:
  image: node:lts-alpine
  stage: deploy
  variables:
    BUILD_TYPE: pages
  script:
    - npm ci
    - npm run build
  artifacts:
    paths:
      - public

pages:
  extends: .pages
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
  environment:
    name: production
    deployment_tier: production
    url: https://<site_url>/

pages_review_app:
  extends: .pages
  rules:
    - if: $CI_MERGE_REQUEST_ID
  environment:
    name: review/$CI_COMMIT_REF_SLUG
    deployment_tier: testing
    url: https://$CI_PROJECT_ROOT_NAMESPACE.$CI_PAGES_DOMAIN/-/$CI_PROJECT_NAME/-/jobs/$CI_JOB_ID/artifacts/public/index.html
  after_script:
    - >
      find ./public  -type f -name "*.html" -exec sed -i -E
      's/(<a[^>]*href="\/.+\/public\/(.*\/)?)("[^>]*>)/\1index.html\3/' "{}" +

GitLab Releaser v5.0.0 Released

2023-01-30T00:00:00Z

GitLab Releaser v5.0.0 was released today with several noteworthy changes, including failing if an empty release description is pulled from the CHANGELOG and a new CLI option to specify the CHANGELOG path.

GitLab Releaser overview #

GitLab Releaser takes a JSON file with details for a release and generates a shell script to leverage GitLab's release-cli application to create a release in the GitLab project. It can be configured to pull the release name and description (release notes) from the CHANGELOG. GitLab's release-cli application is used to simplify variable expansion, and with the expectation that it will be updated to accept this JSON format

Changes in v5.0.0 #

There are three noteworthy changes in v5.0.0, two of which could be breaking. Please see the release for other minor changes.

Fail on empty CHANGELOG description (BREAKING) #

Previously, if the release description was taken from the CHANGELOG and GitLab Releaser found an empty description, that was used when creating the release, resulting in no release notes. History has shown that when this occurs it's almost always a formatting error in the CHANGELOG, something like:

## v2.1.0

## Changed

- Update a great feature to be even better...

In this case, when release 4.0.4 is found, GitLab Releaser recognizes that releases use a ## header, so the next release is assumed to start at ## Changed, and an empty description is returned. The correct CHANGELOG for this example is:

## v2.1.0

### Changed

- Update a great feature to be even better...

With the previous behavior the release was created, and the release notification sent out, with no release notes. To avoid this, the default behavior was changed to note the error and fail if this occurs.

If the previous behavior is desired, there is a new CLI option --allow-empty-changelog-desc (-a) to allow an empty CHANGELOG description.

Specifying CHANGELOG path or file #

Previously, if the release name or description was taken from the CHANGELOG, GitLab Releaser assumed the CHANGELOG was located in the project root. While this is frequently the case, it is not always. There are also cases where the logic to select the CHANGELOG can fail or select the wrong file, especially if there are multiple CHANGELOG files or it has an atypical name.

There is now a CLI option --changelog (-c) to be able to specify either the path to the CHANGELOG using the existing logic to locate the file, or specify the actual path.

Programmatic API change (BREAKING) #

With multiple new options added, the getReleaseCliCommand function was refactored to accept an options object with all CLI options. This is only an impact if that function is used programmatically (which is atypical).

Running Lighthouse in GitLab CI with Metrics Report

2023-01-21T00:00:00Z

Google Chrome's Lighthouse tool is a great resource in the browser and has become the standard for basic performance and best-practice metrics on websites. While useful in the browser, a good continuous integration (CI) pipeline includes all the testing practical to identify any issues as early as possible. To that end, this post details how to run Lighthouse via the CLI in GitLab CI and collect a GitLab metrics report so any changes are reported in merge requests.

Lighthouse versus Lighthouse CI #

Lighthouse can be used to analyze a single URL and provide a report with the results, similar to those obtained when running it in a browser. Lighthouse CI provides a wrapper around Lighthouse to be able to run analyses against multiple URLs, run with multiple retries for each page, set specific thresholds to fail the tests, track metrics for a URL over time, and more. This also requires uploading the results to a Lighthouse CI server (at least to track results over time).

The goals for this example, which is based on analysis of a static site (like this one), are to:

Run Lighthouse against a representative page to identify changes and provide an HTML report to view the detailed results. This should be done for both a mobile and desktop browser configuration.
Make as much data as practical available through the GitLab merge request UI. This is intended to reinforce the existing philosophy in GitLab using merge request widgets as the portal to all merge request related data, and minimize the cases where you have to navigate elsewhere to view data.

Given those goals, Lighthouse is used in this example. In general, Lighthouse CI could be used instead, but requires some changes to CLI commands, configuration files, and a Lighthouse CI server (which is then another place to go to view results). See the Lighthouse CI documentation for additional details.

Lighthouse container image #

The desire in this case, as with most GitLab CI jobs, is to run Lighthouse in a container (for so many reason it needs to be an post unto itself). While Lighthouse and Lighthouse CI are both available as npm packages which can be run from the CLI, they do not install a version of Chromium, unlike some other Chromium-based analysis tools, instead requiring it to be installed on the platform (that is, in the container).

While a GitLab CI job could be setup to install all required dependencies on each execution, this can be a lengthy process, so to reduce pipeline execution times a dedicated container with all dependencies pre-installed is preferred. Since there were none easily found with the desired capabilities, I created a new Lighthouse container image project to build and publish an image. This image is based on the Node 18 Debian Bullseye Slim image with the following dependencies installed:

chromium: The latest Chromium release for Debian Bullseye.
lighthouse: The npm package for running Lighthouse via the CLI.
@lhci/cli: The npm package for running Lighthouse CI via the CLI.
serve: A CLI-based static web server.
wait-on: A CLI-based tool to wait for a web server (or other resources) to become available.

This image also runs by default as a non-privileged user, which is not only a general best-practice, running as root requires running Chromium without a sandbox (that is, with the --no-sandbox argument). While running in a container itself minimizes the risk of running without a sandbox, it's certainly desirable to have when possible.

The container images for this project are hosted on gitlab.com in the project's container registry as registry.gitlab.com/gitlab-ci-utils/lighthouse. The latest tag is recommended, but others are available. See the project documentation for additional details on the project and the full Dockerfile, if interested.

Running Lighthouse #

To run Lighthouse, the previously detailed container image is used for the CI job. The needs property should be set to any job or jobs required to build the site to be tested, which then trigger the Lighthouse job as soon as any required artifacts are available. As noted previously, a goal is to run Lighthouse in both a mobile and desktop configuration, so a variable is set to specify the form factor, which is used in the job script and when generating the metrics report.

lighthouse_mobile:
  image: registry.gitlab.com/gitlab-ci-utils/lighthouse:latest
  stage: test
  needs:
    - job_that_builds_site
  variables:
    FORM_FACTOR: 'mobile'
  ...

The page to be tested is served by a static server running in the container. This is run in before_script to optimize the ability to re-use this job across projects. If it is used for a different web application needing a different server, only the before_script has to be overridden, and only with the script to start the server. In this case serve is used since it is installed in the container image, and it is passed the directory containing the site. The output from serve, both stdout and stderr, are piped to null to avoid polluting the job log with > /dev/null 2>&1. Finally wait-on in called with the URL to be tested, which causes the script to wait until the server is running and successfully serves the page.

lighthouse_mobile:
  ...
  before_script:
    - serve ./site_folder/ > /dev/null 2>&1 & wait-on http://localhost:3000/
  ...

The script runs lighthouse, formatted as a multi-line block for readability, and passes the URL to be tested and the following test options as CLI arguments:

Run in a headless browser.
Test only a subset of the Lighthouse categories.
Output both a JSON and HTML report. The HTML report provides a more convenient way to view the full results. The JSON report is used to extract the scores to a metrics report.
Specify the root name of the reports via --output-path, using the FORM_FACTOR variable declared previously. This is appended with ".report.<format>", so the final JSON report name, for example, is lighthouse-mobile.report.json.
The $LIGHTHOUSE_ARGUMENTS variable is included to allow for adding CLI arguments without having to override the script (although in this job it is empty).

If you're running this on a gitlab.com shared runner, the --no-sandbox argument should not be required. If this is run on a different runner, or sandbox errors are encountered, the --chrome-flags argument can be updated to --chrome-flags="--headless --no-sandbox".

lighthouse_mobile:
  ...
  script:
    - >
      lighthouse http://localhost:3000/
      --chrome-flags="--headless"
      --only-categories="accessibility,best-practices,performance,seo"
      --output="json,html"
      --output-path="lighthouse-$FORM_FACTOR"
      $LIGHTHOUSE_ARGUMENTS
  ...

If desired, some of this configuration could be specified via a Lighthouse configuration file. Since that can be defined as a JavaScript module, the variable FORM_FACTOR would be available there as well to update the configuration.

Lastly, the HTML and JSON Lighthouse reports are saved as artifacts. In addition, when: always is set to ensure that any artifacts are saved even on job failure.

Putting all of that together results in the following job to run Lighthouse testing against a mobile configuration.

lighthouse_mobile:
  image: registry.gitlab.com/gitlab-ci-utils/lighthouse:latest
  stage: test
  needs:
    - job_that_builds_site
  variables:
    FORM_FACTOR: 'mobile'
  before_script:
    - serve ./site_folder/ > /dev/null 2>&1 & wait-on http://localhost:3000/
  script:
    - >
      lighthouse http://localhost:3000
      --chrome-flags="--headless"
      --only-categories="accessibility,best-practices,performance,seo"
      --output="json,html"
      --output-path="lighthouse-$FORM_FACTOR"
      $LIGHTHOUSE_ARGUMENTS
  artifacts:
    when: always
    paths:
      - lighthouse-$FORM_FACTOR.report.html
      - lighthouse-$FORM_FACTOR.report.json

To run Lighthouse testing against a desktop configuration, the previous lighthouse_mobile job can be extended, and minimal additional overrides are required based on the that job's definition.

Change the FORM_FACTOR variable to update the report names.
Add the --preset=desktop CLI argument via the LIGHTHOUSE_ARGUMENTS variable, which updates multiple Lighthouse settings (for example --form-factor, --screenEmulation).

lighthouse_desktop:
  extends:
    - lighthouse_mobile
  variables:
    LIGHTHOUSE_ARGUMENTS: '--preset=desktop'
    FORM_FACTOR: 'desktop'

Exposing artifacts in merge requests #

GitLab provides the capability to specify job artifacts that are exposed with links directly in the merge request UI, which avoids having to go to the job to view those artifacts. This is accomplished by adding the expose_as property to the job artifacts with the name to display with the artifacts.

Unfortunately, there's an open issue in GitLab where the expose_as capability does not work when there are variables in the artifacts:paths names, which has been used previously. In this case the artifacts are properly saved, but they're not exposed in the merge request UI. So, to have the artifacts exposed the FORM_FACTOR variable needs to be backed out of artifacts:paths and updated to list the complete names. This has to be repeated in both jobs since the FORM_FACTOR variable is different in each case to uniquely identify the reports. Note that for the lighthouse_desktop job only the artifacts:paths must be specified since the other values are still inherited from the lighthouse_mobile job.

lighthouse_mobile:
  ...
  artifacts:
    when: always
    expose_as: 'Lighthouse Report'
    paths:
      - lighthouse-mobile.report.html
      - lighthouse-mobile.report.json

lighthouse_desktop:
  ...
  artifacts:
    paths:
      - lighthouse-desktop.report.html
      - lighthouse-desktop.report.json

With these changes made, once the CI pipeline is complete, the Lighthouse report artifacts are exposed in the merge request UI as shown in the screenshot below. Note that the individual jobs are listed as well to identify the source.

Generating a metrics report #

Note that metrics reports are a GitLab Premium feature and not available in GitLab Free. If you're using GitLab for open source work see the GitLab Open Source Program for details on how to register to get free GitLab Ultimate licenses for those projects to unlock this feature and many others.

The last step is to generate a GitLab metrics report for the results. A metrics report is simply a file of key/value pairs, each on a new line, conforming to the OpenMetrics format. GitLab ingests the report and identifies any changes in the merge request UI - new metrics, removed metrics, or metrics with changed values. As with all merge request reporting, GitLab reports changes from the source branch in that merge request, which allows easy identification when any of the scores change. In this case the goal is to have metrics for the score in each Lighthouse category and form factor being analyzed, so the metric keys take the form lighthouse{category="performance",form_factor="mobile"}.

The results are taken from the Lighthouse JSON report. There's a lot of data in the JSON report, the full details of which are beyond the scope of this post, but the desired data in this case is summarized below. At the root level there's a categories object, which has keys for each category, each of which is an object with a score property that is a value between 0 and 1.

{
  ...
  "categories": {
    "performance": {
      ...
      "score": 0.99
    },
    "accessibility": {
      ...
      "score": 1
    },
    "best-practices": {
      ...
      "score": 1
    },
    "seo": {
      ...
      "score": 1
    }
  },
  ...
}

To generate the metrics report add the JavaScript file lighthouse-metrics.js to the project. In this case JavaScript is used for simplicity since this is run a Node.js container already. This module uses the FORM_FACTOR environment variable defined in the job to determine the results file, opens the file and parses the JSON results, creates an array of formatted metrics with scores from 0 to 100, then joins them together, one on each line, and finally saves the resulting metrics file. Optional chaining is used when retrieving the metrics so that any individual missing metric do not raise an error, it simply reports the value as NaN (since it returns undefined multiplied by a number).

const fs = require('fs');

const formFactor = process.env.FORM_FACTOR || '';
const scoreScalingFactor = 100;
const reportFileName = `lighthouse-${formFactor}.report.json`;
const metricsFileName = 'lighthouse-metrics.txt';
const categories = ['performance', 'accessibility', 'best-practices', 'seo'];

const results = JSON.parse(fs.readFileSync(reportFileName, 'utf8'));
const metrics = categories
  .map(
    (category) =>
      `lighthouse{category="${category}",form_factor="${formFactor}"} ${
        results?.categories[category]?.score * scoreScalingFactor
      }`
  )
  .join('\n');
fs.writeFileSync(metricsFileName, metrics);

With this script available the CI job needs to be updated to run this in the jobs. It is executed in after_script versus script since a failure in after_script does not fail the job, and the job should not fail only for a metrics-related failure. This is run with the node command since project files pulled by the GitLab runner are not given executable permission. The CI job artifacts also needs to be updated with a reports property for GitLab to import the report. Note the metrics report does not need to added to artifacts:paths since reports are ingested directly by GitLab, not saved with other artifacts.

lighthouse_mobile:
  ...
  after_script:
    - node ./lighthouse-metrics.js
  ...
  artifacts:
    ...
    reports:
      metrics: lighthouse-metrics.txt

With those updates, the first merge request to create the metrics shows results like the screenshot below, identifying the 8 new metrics (4 categories * 2 form factors) and giving the values.

In future merge requests, any Lighthouse score changes are identified as changed values as shown in the screenshot below.

Final CI job #

Putting all of these pieces together, the final two jobs are shown below.

lighthouse_mobile:
  image: registry.gitlab.com/gitlab-ci-utils/lighthouse:latest
  stage: test
  needs:
    - job_that_builds_site
  variables:
    FORM_FACTOR: 'mobile'
  before_script:
    - serve ./site_folder/ > /dev/null 2>&1 & wait-on http://localhost:3000/
  script:
    - >
      lighthouse http://localhost:3000/ --chrome-flags="--headless"
      --only-categories="accessibility,best-practices,performance,seo"
      --output="json,html" --output-path="lighthouse-$FORM_FACTOR"
      $LIGHTHOUSE_ARGUMENTS
  after_script:
    - node ./lighthouse-metrics.js
  artifacts:
    when: always
    expose_as: 'Lighthouse Report'
    paths:
      - lighthouse-mobile.report.html
      - lighthouse-mobile.report.json
    reports:
      metrics: lighthouse-metrics.txt

lighthouse_desktop:
  extends:
    - lighthouse_mobile
  variables:
    LIGHTHOUSE_ARGUMENTS: '--preset=desktop'
    FORM_FACTOR: 'desktop'
  artifacts:
    paths:
      - lighthouse-desktop.report.html
      - lighthouse-desktop.report.json

The Blog Is Back In Town

2023-01-10T00:00:00Z

It's been well over a decade since I kept a blog here. That's a lifetime or two in software development, but I've missed writing and decided it's time to bring it back. So, this is the kick-off to incentivize me to write more. And it never hurts to have another new project to work on, and something with more web front-end work, that is developing a site with Eleventy, hosted on GitLab pages, optimizing the use of GitLab's tooling, and extensive use of automation with GitLab CI.

This blog will focus on the software world I live in today and my open source work, which includes:

Continuous integration, continuous delivery, continuous deployment, and automation
Leveraging GitLab and GitLab CI to optimize software development
Applications written in Node.js, sometimes Bash, and usually run in containers
Development of automation and automated test tools

Most of my open source projects are hosted in GitLab in the group GitLab CI Utils - a mix of GitLab CI jobs, containers, and Node.js tools/libraries. This blog's site lives at my GitLab.com profile, but there's not a lot else there (at least not public). Most of my projects are maintained in a separate group to facilitate participation in GitLab's Open Source Program, which provides free GitLab Ultimate licenses, and the additional capabilities that brings, to improve open source development.

I do also have a GitHub profile, but this is primarily forks of various open source projects that I contribute to. Many are one-off contributions, but I do make frequent contribution to Pa11y and Pa11y CI (automated web accessibility testing tools).

Blog history #

My last blog was written 13 - 15 years ago. At that point it was focused on web development with ASP.NET. So, for a little entertainment here are some stats on the state of web development at that time to show how far things have come since then.

Browsers #

Internet Explorer still had 50% of the browser market share, and even though Internet Explorer 9 was the latest release, most web applications were still written to support Internet Explorer 6/7/8 (that is, before the DOM was standardized).
Chrome v5 was the latest release, and it was still using WebKit (yes, that WebKit).

Web applications #

ASP.NET v4.0, using Web Forms, was just released. ASP.NET MVC was a release candidate and .Net Core was still 6 years away. Enterprises not using ASP.NET were probably using ColdFusion or Classic ASP.
There was no such thing as front-end, back-end, or full-stack web developers. We were just web developers, and we were all full-stack.
The terms multi-page app (MPA), server-side rendering (SSR), and file system based routing didn't exist because all web apps were server-side rendered multi-page apps using file system based routing (maybe with a little AJAX sprinkled in).
Many developers still expected HTML to evolve to some form of XHTML strict. The WHATWG had formed and created an initial HTML 5 draft spec, but it wouldn't be a candidate recommendation for 2 years, and no browser would fully support it for 3 years.
REST was just starting to get real adoption as a successor to SOAP for web services.

Front end #

It took some courage (and probably a lot of wasted time) to develop cross-platform or cross-browser web applications without jQuery.
Many designs (especially some older enterprise apps) still conformed to the "web safe color palette."
Responsive web design was just starting to be used, and Twitter had just released the Bootstrap framework to make it practical.
AngularJS came out late that year (that is v1, before v2 was re-named as Angular). The first release of Typescript was still 2 years away. The first release of React was 3 years away.