At QIMA, we develop a SaaS application called QIMAone. When you develop a SaaS platform, there are several constraints like keeping up to date the technology stack or having a release train at a constant speed to deliver features and fixes as soon as validated and get feedback quickly so that inspecting and adapting next release vision.

🦸Avengers

The avengers are people in charge of the releases during 2 weeks (duration of a sprint). This team is rolled out every sprint to avoid that team fall in deep depression. This team is also in charge to handle last customer ticket. Internally, this team is called “Support team“ that is less glorious than Avengers maybe. This team is composed of developers, ops, qa and a product manager. The team needs to cover all topics that support and release management can involve. We chose to involve a dedicated team during the time of a Sprint so that other teams can be completely focused on Business production.

🏪 Shopping list

The content of release follows the sprint planning of all teams. Upstream from sprint planning, product team imagined the vision of the application before to show to teams next User Stories addressing this one. After that, the release contains all work merged to the develop branch. All team follows the Product vision but, we know, everything can happen.

Let it go!

To stick the version, it’s simple, every week at same day and same hour (on the paper of course, but we try to respect this commitment), the support team run the automatic release script. If another team miss this departure of this release train and didn't merge a user story, too late, this one will wait for a week to be released. This train takes one week to join the production because of our quality non-regression process. In fact, if you miss a train, the feature will pend during 2 weeks before being delivered to the end user. The final decision, to not embed a feature into the release, is under control of the Product team. One benefit from Agile methodology, the Product Manager has a daily vision of the ongoing work so, normally, when something will impact the next release, he should be aware enough time ahead to inspect and adapt.

I. Frozen: Stop the world❄️

First step is the stop of integrating anything else to develop branch (our integration branch). This step happens the D-Day and last for a few minutes, and it ensures that no new commit reach develop branch while the support team would create the new release. The main idea is to take the last build commit on the integration branch, test it and tag it with a new version. We use a tool to integrate the freeze with GitHub, Mergefreeze. All developers are aware why they cannot merge in this case

II. Generate🧙

This is the magic part of the release train. Into the continuous integration, there is a step to trigger the creation of the release based on the Git repository activity.

This magic trick is named conventional-changelog. Obviously, the first thing to implement by the team in the repository is the conventional-commit. To ensure that the convention is correctly follow, the semantic is checked on the title of GitHub pull-request thanks to Semantic pull requests bot. Since the team uses the “squash and merge” policy to integrate a pull-request most of the time, the final commit message will be the title of this one.

The result of this process is a git tag with the related release notes in GitHub and the update of current version to next one.

Now, you know more about our “what”, we can go deeper into the “how”. The entry point of this step is a bash script which is going to orchestrate all steps.

The Flow

Increment version → Generate Release Changelog and tag version → Push Tag and Changelog

Commit sanity check

Before generating anything, a sanity check is done to verify that we don’t try to create a release on an existing release tag (.i.e create a Git tag at the same commit that the previous one):

git describe --tags --exact-match
if [ $? -eq 0 ]; then
  exit 0
fi

The git command returns an error when there is no tag on the current commit.

Release tooling

So to operate the release, the script uses a bunch of JavaScript libraries related to conventional commit process and GitHub:

so, a step of installation is necessary to be sure that all dependencies are present.

Increment version

Like the project QIMAone is a monorepo, the version is shared between all module Java or Node. At the root of Git repository, there is a package.json file that going to help to determine the bump version. For example, if the version is 1.0.0 and new features have been added, the version bumps to 1.1.0 else if the content is only fixes, the bump will be 1.0.1.

## Bump root package.json ##
############################
currentVersion=$(node -p "require('./package.json').version")
bump=$(conventional-recommended-bump -p angular)
npm --no-git-tag-version version $bump
newVersion=$(node -p "require('./package.json').version")

Changelog: Database changes

In the release notes, there is a section dedicated to database migration driven by Liquibase.

# Do we have liquibase changes ? ##
####################################
changeLogFile=$(mktemp -p /tmp)
liquibaseFileTmp=$(mktemp -p /tmp)
set +e
git diff --name-only v$currentVersion...HEAD | grep "^/path/to/liquibase/changelog/" | xargs --no-run-if-empty -n1 -I {} node liquibase.js {} v$newVersion >> $liquibaseFileTmp
set -e
if [ -s $liquibaseFileTmp ]; then
  echo "### ⚠️  Liquibase Changes<br/>" > $changeLogFile
  cat $liquibaseFileTmp >> $changeLogFile
  echo '<br/>' >> $changeLogFile
else
  echo "### No Liquibase Changes Found<br/><br/>" > $changeLogFile
fi
echo "" >> $changeLogFile

If there is migration of the current release, a changes note is generated based on the Liquibase configuration file. The project uses the XML format to write migration so a JavaScript process, liquibase.js was created to extract comment of migration changeset with xml2js.

Changelog: Elastic Changes

Like the application uses Elastic Search and the deployment has a strategy to wipe out index then populate it based on the database content(only source of truth), a section dedicated to Elastic is added to the note so that support team is aware about taking attention to Elastic step. For this purpose, the team developed a tool, elastic-manager, to handle this step and the warning message to this one.

## Do we need to run Elastic Manager ? ##
#########################################
set +e
git diff --name-only v$currentVersion...HEAD | grep "^path/to/elastic-manager" > /dev/null 2>&1
if [ $? -eq 0 ]; then
  echo "### ⚠️  Elastic Manager Reindex Job need to be run !<br/><br/>" >> $changeLogFile
else
  echo "### Do not run Elastic Manager Reindex Job - no changes found<br/><br/>" >> $changeLogFile
fi
echo "" >> $changeLogFile
set -e

It’s a check if elastic-manager is part of git diff.

Carve the Changelog

This step aggregates all gathered information from git changes inside the CHANGELOG.md file of the repository and add it to git stages, then clean up npm dependency changes.

## Update CHANGELOG.md to add Liquibase + Elastic Warnings ##
#############################################################
cat $changeLogFile > releaseHeader.hbs
cat CHANGELOG.md >> $changeLogFile && cp -f $changeLogFile CHANGELOG.md

## Update CHANGELOG.md to add Fixes + Features ##
#################################################
GITHUB_TOKEN=${QIMA_BOT_GIT_PUSH_GITHUB_TOKEN} conventional-changelog --infile CHANGELOG.md --same-file --preset $(pwd)/changelog-preset.js --config changelog-config.js
git add CHANGELOG.md
git checkout -- package*.json

To generate the changelog, conventional-changelog uses Handlebars(file with extension hbs) template engine to describe commit and pull-request content.

The changelog-preset.js script retrieve Jira link from all release commits(because the PR template contains a link to Jira tickets) and put it in release note.
the changelog-config.js load commit template containing the Jira ticket link and load the GitHub release template used for the changelog file.

Update the version ⬆️

This step update Maven and Node project version.

## Update all app modules versions ##
#####################################
rootDir=$(pwd)
cd ${rootDir}
for path in $(find . -name pom.xml); do
    sed -i -e "s/<qimaone\.version>.*<\/qimaone\.version>/<qimaone\.version>$newVersion<\/qimaone\.version>/g" $path
    git add ${path}
done
cd frontend && npm --no-git-tag-version version $newVersion &>/dev/null && cd $rootDir
git add frontend/package*.json

Release Tag

This step is the one that push the tag on GitHub along with the GitHub release.

## Tag & Create a new Github release ##
#######################################
npm version $bump --force --message "chore: release %s 🎉"
git push "https://${QIMA_BOT_GIT_PUSH_GITHUB_TOKEN}@github.com/asiainspection/qima-platform.git" tag v$newVersion

cat releaseHeader.hbs > releaseGithubHeader.hbs
CONVENTIONAL_GITHUB_RELEASER_TOKEN=${QIMA_BOT_GIT_PUSH_GITHUB_TOKEN} conventional-github-releaser --preset $(pwd)/changelog-preset.js --config changelog-config.js --draft
git push "https://${QIMA_BOT_GIT_PUSH_GITHUB_TOKEN}@github.com/asiainspection/qima-platform.git" ${CIRCLE_BRANCH}

III. Deploy

When the release correctly bundled, Support team can follow the release scheduling for deployment. Some of them are fully automatic like the demo environment that is the first one that receive the release. Like the release contains also the infrastructure improvement, its deployment needs to be supervised, at least, by an Ops, a Dev, and a PM. In case of something happens during the deployment, all abilities and knowledge are gathered to determine the root cause and take the necessary actions to solve it. All other stages deployment needs to be approved by required people(for example, the production needs approval from Dev, Ops, QA and PM). The continuous integration build for the release will create a Pull-Request by environment to trigger the deployment following the GitOps approach. (Go read this article).

IV. Qualify🩺

Now that the release is generated, the support team makes it available for qualification. On the road to Production, the two teams in charge to qualify the release are QA and Product. The QA team drives the qualification and Product team gives the go-ahead based on the QA feedback. Let’s talk what happens in each stage:

Integration: this stage is the one from where the release is made. It uses by all team to integrate last development (i.e.: feature, infrastructure, etc.). In terms of qualification, each feature is validated by a member of Product team before integrating it and the QA team run, every day, automated tests with Cypress. The QA team can raise a blocking bug before the creation of release.
Demo: this stage is the first on that receives the freshly created release. Even, if this environment doesn’t contain a big amount of data, it is useful to validate, at minima, infra changes and migration. After deployment, the Product team makes a quick sanity check.
QA: this stage is the playground of QA team. The team plays non-regression tests, manual and automatic. The release stays in this stage around 2 days to let time to QA team to make every possible tests because all tests cannot be covered by automation.
Pre-Prod: This stage has the same configuration as the Production in terms of infrastructure. The other point it that an anonymized production dump is mount to this stage. The stage is used to check duration of migration with a relevant data volume. Real data gives more feedback about migration approach. By the past, the problem.
Production: That’s the last step, the support team shutdown the entire application, and it has a window of 1 hour to deploy a new release then scale up. In that window, the product team member has to make a quick sanity check like “Can I reach the platform?“ or “Can I access properly to external services?“.

Not Perfect

The release process is not perfect and, it’s like all other thing when you start a new project from scratch, you have to take some decision to move on quickly and improve later. Like QIMA are currently hiring new people on QIMAone project, the release process is continuously challenged by newcomers.

In terms of implementation, this one could be migrated to semantic-release tooling, what is suggested by conventional changelog.

The high topic to improve so that reducing the delivery is the Qualification step by automating more our tests and reduce the set of manual tests.

Another topic on Release process on what the team would like to improve themselves is to remove the application downtime for Production deployment implementing Expand and Contract pattern.

To Conclude

Our current Release train process allows the project to deliver every week a new version for our customers. The team would like to shorten it to improve our Continuous Delivery but like the platform registers more users, the same team needs to take care of the stability of the solution first.

Thank you!

If you want to be part of this adventure, we are currently hiring ! ✨

At QIMA, we want to make it simple and save people time. They love it ! Automation is key. If you have any suggestion or remark, please share them with us. We are always keen to discuss and learn!

Written by Gabriel PASCUAL, Software Engineer at QIMA. ✍️

QIMAone Release Train 🚂