CI/CD pipeline on Jenkins using Docker

Previously:

CI/CD pipeline on Jenkins - I

CI/CD pipeline on Jenkins - II

Now

I would like to document here what would it look like a similar setup running on Azure Container Instances (using Docker):

• Login to Azure Portal
• Select the appropriate Subscription to use
• Activate the contributor role

• Open Azure Cloud Shell
• Let the fun begin! 🥳

Create Jenkins main node with persistent storage

Containers by default are stateless, so we need to prevent data loss at all cost.

By using persistent storage no matter how many times the container gets terminated and restarted, we will not lose any data;

The goal will be to have JENKINS_HOME (/var/jenkins_home) safely stored outside of the container.

# Change these four parameters as needed

ACI_PERS_RESOURCE_GROUP=myJenkinsResourceGroup

ACI_PERS_STORAGE_ACCOUNT_NAME=myjenkinsstorageacct1

ACI_PERS_LOCATION=eastus

ACI_PERS_SHARE_NAME=acishare

A resource group is a logical folder where we will store (group) all the items that belong to this project, this will help us to be more organized. 

# Create the storage account with the parameters

az storage account create --resource-group $ACI_PERS_RESOURCE_GROUP --name $ACI_PERS_STORAGE_ACCOUNT_NAME --location $ACI_PERS_LOCATION --sku Standard_LRS

# Create the file share

az storage share create --name $ACI_PERS_SHARE_NAME --account-name $ACI_PERS_STORAGE_ACCOUNT_NAME

# Retrieve the storage key

STORAGE_KEY=$(az storage account keys list --resource-group $ACI_PERS_RESOURCE_GROUP --account-name $ACI_PERS_STORAGE_ACCOUNT_NAME --query "[0].value" --output tsv)

# Creation of our Jenkins main node

az container create --resource-group $ACI_PERS_RESOURCE_GROUP --name my-jenkins-main-01 --image jenkins/jenkins:latest --dns-name-label jenkins-main-ci --ports 8080 --azure-file-volume-account-name $ACI_PERS_STORAGE_ACCOUNT_NAME --azure-file-volume-account-key $STORAGE_KEY --azure-file-volume-share-name $ACI_PERS_SHARE_NAME --azure-file-volume-mount-path /var/jenkins_home --environment-variables 'JAVA_OPTS'='-Dorg.apache.commons.jelly.tags.fmt.timeZone=America/New_York'

 After this, we should have a Jenkins main / master node up and running on ACI.

Create Jenkins Inbound Agent

As a good practice, we will not be executing any building jobs out of the main node but we will align to our Jenkins architecture.

Inside Jenkins main node we need to provision a new Node as a Permanent Agent:

Once completed the creation process on Jenkins main node, this is what the Status section will look like:

Unlike with main node, we decided to go for our own customized jenkins inbound-agent since we require certain additional tools as documented in previous posts.

Now going back to Azure Cloud Shell, you will need to execute the following create command:

az container create --resource-group $ACI_PERS_RESOURCE_GROUP --name my-agent-node-01 --image mywwwcontainerregistry001.azurecr.io/wwwcontainer/jenkins-inbound-agent-linux-image:0.0.2 --os-type linux --dns-name-label jenkins-node-01-ci --ports 443 --command-line "java -jar /usr/share/jenkins/agent.jar -url http://jenkins-main-ci.eastus.azurecontainer.io:8080 -secret <SECRET_GOES_HERE> -name build-agent-image-01  -workDir /home/jenkins/work -webSocket" --cpu 4 --memory 6

Some considerations

--command-line "java -jar /usr/share/jenkins/agent.jar -url http://jenkins-master-ci.eastus.azurecontainer.io:8080/ -secret <SECRET_GOES_HERE> -name build-agent-image-01 -workDir /home/jenkins/work -webSocket"

This parameter (--command-line) tells the container to execute this command(s) when starts, so it will try to connect to its main node as soon as it comes online.

The key here is to include the modifier -webSocket, otherwise it will not be able to communicate with the main node.

If everything went as planned, the agent will show online in a couple of minutes.

Why didn't you stick with the OOTB Jenkins Inbound agent for linux like you did for the main node with jenkins/jenkins:latest?

If you have been paying attention and had read my previous posts, we need some special software to be installed on the agent to make it possible to compile and deploy SFCC code base:

• rsync
• 7zip
• maven (optional)
• git
• curl
• sudo
• NodeJS 10.24.1
• NPM
• typescript
• sfcc-ci

Therefore we need a custom image, but since I am not a big fan of reinventing the wheel, I extended the OOTB agent and this is the result.

If you are the type of fellow who would like build the docker image from scratch, here it is the Dockerfile for your own amusement:

FROM jenkins/ssh-agent:latest

#Switch to root

USER root

# Install rsync, 7zip, maven, git, curl & sudo

RUN apt-get update \

 && DEBIAN_FRONTEND=noninteractive \

    apt-get upgrade -y \

 && apt-get install -y rsync p7zip-full maven git curl sudo

# Clean up APT when done.

RUN apt-get clean && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*

#Setup NodeJS & NPM

ARG NODE_VERSION=10.24.1

ARG NODE_PACKAGE=node-v$NODE_VERSION-linux-x64

ARG NODE_HOME=/opt/$NODE_PACKAGE

ENV NODE_PATH $NODE_HOME/lib/node_modules

ENV PATH $NODE_HOME/bin:$PATH

RUN curl https://nodejs.org/dist/v$NODE_VERSION/$NODE_PACKAGE.tar.gz | tar -xzC /opt/

#Install Typescript - test

RUN npm install -g typescript

#Install SFCC-CI

RUN npm install -g sfcc-ci

#Create "alias" (soft link) for 7z

RUN ln -s $(which 7z) /usr/bin/7zz

#Switch to jenkins

USER jenkins

 

Now it's up to you to try and see the magic with your own eyes!

CI/CD Pipeline for SFCC - Jenkins - part II

On a previous post, we went over setting up a CI/CD pipeline for SFCC using Windows.

What's next?

Let's give it a try with Linux!!!

Why Linux and not Windows?

Linux is much more efficient in terms of managing and executing resource intensive tasks like compiling and packaging code projects. 

Historically, Linux is 30-40% faster when performing such tasks, a few additional elements to consider:

• Linux performs faster git operations - Linux performs command line git operations faster than Windows (as evidenced by ci.jenkins.io git client plugin automated tests requiring 12 minutes on Linux and 22 minutes on Windows)
• Fewer locking issues - Linux file system locking semantics are more forgiving than Windows file system locking semantics. It is more likely that a "busy file" on Windows will disrupt the Jenkins controller than it will on Linux.
• License cost. 

Our journey has taken us from a process that used to take 3-4 hours when we were leveraging a vendor owned and managed implementation to this:

So expect at least to achieve better times than with Windows.

If you are still with me, this is the translation into shell:

Code build

#!/bin/bash

export PATH=/usr/local/bin/7zip:$PATH

export PREFIX="B2C"

export COMMIT_SHORT="${GIT_COMMIT::7}"

export BRANCH=${GIT_BRANCH//\//_}

export BUILD_VERSION=${PREFIX}_${BRANCH}_${BUILD_NUMBER}_${COMMIT_SHORT}

echo "***Deleting Folders to skip"

rm -rf ./cartridges/useless_folder

echo "***Installing NPM & Build project"

npm install --silent

npm run build --silent

echo "***Cleaning up manually build folder"

#Since workspace is always cleaned, this step is optional

rm -rf ./$BUILD_VERSION

mkdir ./$BUILD_VERSION

echo "***Preparing to package project - rsync"

#skipping files: .project *.java *.docx *.cpp *.php

#skipping folders: junk

rsync -rq --exclude={'.project','*.java','*.cpp','*.php'} --exclude={'junk'} ./cartridges/ $BUILD_VERSION

echo "***Packaging deployable artifact - 7z"

#"-mx[N]" specifies compression level

#"-mmt[N]" set number of CPU threads

7zz a $BUILD_VERSION.zip $BUILD_VERSION -mx9 -mmt16

Code deploy

Nothing new here, feel free to reuse the one we did on our first iteration.

Have fun!

CI/CD Pipeline for SFCC - Jenkins - part I

Story time: Setting up an efficient CI/CD Pipeline for Salesforce Commerce Cloud

A little bit of context

Our main vendor was running this capability, which enabled the team to run deployments once per week or maybe every other day. That seemed great as the status quo;

However, waiting for a build to complete in 3-4 hours doesn't work for me, the goal was to enable the team to deploy at least every hour or less.

Following a nice combination of creativity plus thinking outside of the box, we setup a pipeline which deploys under 15 minutes (12) to 4 different Instances of SFCC (DEV, INT, QA and STG).

I am not going to showcase what it looked like in the past, because water under the bridge

Overview

Jenkins is an open source automation server. It helps automate the parts of software development related to building, testing, and deploying, facilitating continuous integration, and continuous delivery. It is a server-based system that runs in servlet containers such as Apache Tomcat.

Build great things at any scale

Jenkins provides hundreds of plugins to support building, deploying and automating any project.

We decided to set it up on a VM in Azure since that's what we have. We started with a Windows VM for the following reason, we wanted to mirror as much as possible the original setup done by our vendor so we could leverage if there was any institutional knowledge. 

In a future iteration we might use Linx and why not, a Dockerized image.

The fun part is in the job

• Be organized

• Define a unique and short name in order to avoid issues with file system limitations

• Add a meaningful description
• Define a clear build strategy - build retention - this will help you to manage your disk space
• Reference your GitHub repo

• Make use of Display Name

• Enable CI/CD! - This is where we let Jenkins do the checking if there is a new change that needs to be deployed - currently we set that value to 30

• Always delete your workspace in order to prevent old code to be present, we always need a fresh copy of the repo
• Add timestamps to the Console Output - they are super helpful
• Clean up your workspace, but be selective so it will be fast

• Use secret bindings (no hardcoded passwords!)

• Add your own build steps. Since you-all came to see the cooking recipe for SFCC Deployments, here it is:

The steps are quite self explanatory, however one pre-requisiten is to have installed SFCC-CLI

Windows batch commands

:: Setting variables

call set PREFIX=DEVOPS

call set PATH=%PATH%;C:\Program Files\7-Zip

call set COMMIT_SHORT=%GIT_COMMIT:~0,7%

call set BRANCH=%GIT_BRANCH:/=_%

call set BUILD_VERSION=%PREFIX%_%BRANCH%_%BUILD_NUMBER%_%COMMIT_SHORT%

:: Delete Folders to skip

call rmdir /S /Q cartridges\useless_folder

:: Install NPM & Build project

:: --silent flag suppress output

call npm install --silent

call npm run build --silent

:: Package build

:: Since workspace is always cleaned, this next step is optional

call rmdir /S /Q %BUILD_VERSION%

call mkdir %BUILD_VERSION%

:: Robocopy

:: skipping files: .project .tern-project .gitignore README.md *.java *.c *.php

:: skipping folders: .settings junk

call robocopy cartridges %BUILD_VERSION% /E /Z /ZB /R:5 /W:5 /TBD /NP /V /MT:16 /XF .project .tern-project .gitignore README.md *.java *.c *.php /XD .settings junk

:: 7z

:: "-mx[N]" specifies compression level

:: "-mmt[N]" set number of CPU threads

call 7z a %BUILD_VERSION%.zip %BUILD_VERSION% -mx9 -mmt16

Shell commands

#Setting variables export PREFIX="DEVOPS" export TARGET_INSTANCE_DEV="development-domain.demandware.net" export TARGET_INSTANCE_INT="integration-domain.demandware.net" export TARGET_INSTANCE_QA="cert.qa.domain.demandware.net" export TARGET_INSTANCE_STG="cert.staging.domain.demandware.net" export COMMIT_SHORT="${GIT_COMMIT::7}" export BRANCH=${GIT_BRANCH//\//} export BUILD_VERSION=${PREFIX}${BRANCH}${BUILD_NUMBER}${COMMIT_SHORT} echo "DEPLOYING $BUILD_VERSION TO $TARGET_INSTANCE_DEV, $TARGET_INSTANCE_INT, $TARGET_INSTANCE_QA and $TARGET_INSTANCE_STG" #Login into SFCC npx sfcc-ci client:auth --selfsigned $SFCC_CLIENT $SFCC_CLIENT_SECRET --renew #Deploy package npx sfcc-ci code:deploy $BUILD_VERSION.zip -i $TARGET_INSTANCE_DEV npx sfcc-ci code:deploy $BUILD_VERSION.zip -i $TARGET_INSTANCE_INT npx sfcc-ci code:deploy $BUILD_VERSION.zip -i $TARGET_INSTANCE_QA -c $SFCC_TEST_REALM_CERT -p $SFCC_TEST_REALM_CERT_SECRET npx sfcc-ci code:deploy $BUILD_VERSION.zip -i $TARGET_INSTANCE_STG -c $SFCC_REALM_CERT -p $SFCC_REALM_CERT_SECRET

Some things to consider, we leveraged P12 Cert to deploy to our QA and STG environment to make sure we take advantage of 2FA.

So there you have it. 

Feel free to use this as your baseline and please share your refined versions.