Customize Jenkins SSH Agent for Windows

I stumbled upon this challenge:

My team was using a Windows VM to act as a Jenkins Agent to build and deploy .NET apps

It works like a charm.

However, I would like to be 100% on Containers, so we decided to embark on the journey of dockerizing this VM.

Using as baseline jenkins/ssh-agent:windowsservercore-ltsc2022-jdk21 we extended that OOTB image by adding the required software installed on the VM:

• Nuget 6.10
• Visual Studio 2019 Build Tools
• Visual Studio 2019 Test Agent
• .NET Framework Developer Pack version 4.5.1 - 4.6.2
• MSDeploy (webdeploy)

The main challenge we faced was to copy/paste some files into

C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\MSBuild\Microsoft\VisualStudio\v16.0

We kept on getting nasty errors like these each time we referenced that folder:

x86 : The term 'x86' is not recognized as the name of a cmdlet, function, 
script file, or operable program. Check the spelling of the name, or if a path 
was included, verify that the path is correct and try again.
At line:1 char:40
+ Get-ChildItem -Path C:\`Program Files (x86)\`Microsoft Visual Studio\ ...
+                                        ~~~
    + CategoryInfo          : ObjectNotFound: (x86:String) [], CommandNotFound 
   Exception
    + FullyQualifiedErrorId : CommandNotFoundException

Our solution was switching the Work Directory and we never looked back (or had to reference such folder)

WORKDIR C:\\'Program Files (x86)'\\Microsoft Visual Studio\\2019\\BuildTools\\MSBuild\\Microsoft\\VisualStudio\\v16.0

And that's how you work around that nasty error.

CI/CD pipeline on Jenkins using Docker

Previously:

CI/CD pipeline on Jenkins - I

CI/CD pipeline on Jenkins - II

Now

I would like to document here what would it look like a similar setup running on Azure Container Instances (using Docker):

• Login to Azure Portal
• Select the appropriate Subscription to use
• Activate the contributor role

• Open Azure Cloud Shell
• Let the fun begin! 🥳

Create Jenkins main node with persistent storage

Containers by default are stateless, so we need to prevent data loss at all cost.

By using persistent storage no matter how many times the container gets terminated and restarted, we will not lose any data;

The goal will be to have JENKINS_HOME (/var/jenkins_home) safely stored outside of the container.

# Change these four parameters as needed

ACI_PERS_RESOURCE_GROUP=myJenkinsResourceGroup

ACI_PERS_STORAGE_ACCOUNT_NAME=myjenkinsstorageacct1

ACI_PERS_LOCATION=eastus

ACI_PERS_SHARE_NAME=acishare

A resource group is a logical folder where we will store (group) all the items that belong to this project, this will help us to be more organized. 

# Create the storage account with the parameters

az storage account create --resource-group $ACI_PERS_RESOURCE_GROUP --name $ACI_PERS_STORAGE_ACCOUNT_NAME --location $ACI_PERS_LOCATION --sku Standard_LRS

# Create the file share

az storage share create --name $ACI_PERS_SHARE_NAME --account-name $ACI_PERS_STORAGE_ACCOUNT_NAME

# Retrieve the storage key

STORAGE_KEY=$(az storage account keys list --resource-group $ACI_PERS_RESOURCE_GROUP --account-name $ACI_PERS_STORAGE_ACCOUNT_NAME --query "[0].value" --output tsv)

# Creation of our Jenkins main node

az container create --resource-group $ACI_PERS_RESOURCE_GROUP --name my-jenkins-main-01 --image jenkins/jenkins:latest --dns-name-label jenkins-main-ci --ports 8080 --azure-file-volume-account-name $ACI_PERS_STORAGE_ACCOUNT_NAME --azure-file-volume-account-key $STORAGE_KEY --azure-file-volume-share-name $ACI_PERS_SHARE_NAME --azure-file-volume-mount-path /var/jenkins_home --environment-variables 'JAVA_OPTS'='-Dorg.apache.commons.jelly.tags.fmt.timeZone=America/New_York'

 After this, we should have a Jenkins main / master node up and running on ACI.

Create Jenkins Inbound Agent

As a good practice, we will not be executing any building jobs out of the main node but we will align to our Jenkins architecture.

Inside Jenkins main node we need to provision a new Node as a Permanent Agent:

Once completed the creation process on Jenkins main node, this is what the Status section will look like:

Unlike with main node, we decided to go for our own customized jenkins inbound-agent since we require certain additional tools as documented in previous posts.

Now going back to Azure Cloud Shell, you will need to execute the following create command:

az container create --resource-group $ACI_PERS_RESOURCE_GROUP --name my-agent-node-01 --image mywwwcontainerregistry001.azurecr.io/wwwcontainer/jenkins-inbound-agent-linux-image:0.0.2 --os-type linux --dns-name-label jenkins-node-01-ci --ports 443 --command-line "java -jar /usr/share/jenkins/agent.jar -url http://jenkins-main-ci.eastus.azurecontainer.io:8080 -secret <SECRET_GOES_HERE> -name build-agent-image-01  -workDir /home/jenkins/work -webSocket" --cpu 4 --memory 6

Some considerations

--command-line "java -jar /usr/share/jenkins/agent.jar -url http://jenkins-master-ci.eastus.azurecontainer.io:8080/ -secret <SECRET_GOES_HERE> -name build-agent-image-01 -workDir /home/jenkins/work -webSocket"

This parameter (--command-line) tells the container to execute this command(s) when starts, so it will try to connect to its main node as soon as it comes online.

The key here is to include the modifier -webSocket, otherwise it will not be able to communicate with the main node.

If everything went as planned, the agent will show online in a couple of minutes.

Why didn't you stick with the OOTB Jenkins Inbound agent for linux like you did for the main node with jenkins/jenkins:latest?

If you have been paying attention and had read my previous posts, we need some special software to be installed on the agent to make it possible to compile and deploy SFCC code base:

• rsync
• 7zip
• maven (optional)
• git
• curl
• sudo
• NodeJS 10.24.1
• NPM
• typescript
• sfcc-ci

Therefore we need a custom image, but since I am not a big fan of reinventing the wheel, I extended the OOTB agent and this is the result.

If you are the type of fellow who would like build the docker image from scratch, here it is the Dockerfile for your own amusement:

FROM jenkins/ssh-agent:latest

#Switch to root

USER root

# Install rsync, 7zip, maven, git, curl & sudo

RUN apt-get update \

 && DEBIAN_FRONTEND=noninteractive \

    apt-get upgrade -y \

 && apt-get install -y rsync p7zip-full maven git curl sudo

# Clean up APT when done.

RUN apt-get clean && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*

#Setup NodeJS & NPM

ARG NODE_VERSION=10.24.1

ARG NODE_PACKAGE=node-v$NODE_VERSION-linux-x64

ARG NODE_HOME=/opt/$NODE_PACKAGE

ENV NODE_PATH $NODE_HOME/lib/node_modules

ENV PATH $NODE_HOME/bin:$PATH

RUN curl https://nodejs.org/dist/v$NODE_VERSION/$NODE_PACKAGE.tar.gz | tar -xzC /opt/

#Install Typescript - test

RUN npm install -g typescript

#Install SFCC-CI

RUN npm install -g sfcc-ci

#Create "alias" (soft link) for 7z

RUN ln -s $(which 7z) /usr/bin/7zz

#Switch to jenkins

USER jenkins

 

Now it's up to you to try and see the magic with your own eyes!

CI/CD Pipeline for SFCC - Jenkins - part II

On a previous post, we went over setting up a CI/CD pipeline for SFCC using Windows.

What's next?

Let's give it a try with Linux!!!

Why Linux and not Windows?

Linux is much more efficient in terms of managing and executing resource intensive tasks like compiling and packaging code projects. 

Historically, Linux is 30-40% faster when performing such tasks, a few additional elements to consider:

• Linux performs faster git operations - Linux performs command line git operations faster than Windows (as evidenced by ci.jenkins.io git client plugin automated tests requiring 12 minutes on Linux and 22 minutes on Windows)
• Fewer locking issues - Linux file system locking semantics are more forgiving than Windows file system locking semantics. It is more likely that a "busy file" on Windows will disrupt the Jenkins controller than it will on Linux.
• License cost. 

Our journey has taken us from a process that used to take 3-4 hours when we were leveraging a vendor owned and managed implementation to this:

So expect at least to achieve better times than with Windows.

If you are still with me, this is the translation into shell:

Code build

#!/bin/bash

export PATH=/usr/local/bin/7zip:$PATH

export PREFIX="B2C"

export COMMIT_SHORT="${GIT_COMMIT::7}"

export BRANCH=${GIT_BRANCH//\//_}

export BUILD_VERSION=${PREFIX}_${BRANCH}_${BUILD_NUMBER}_${COMMIT_SHORT}

echo "***Deleting Folders to skip"

rm -rf ./cartridges/useless_folder

echo "***Installing NPM & Build project"

npm install --silent

npm run build --silent

echo "***Cleaning up manually build folder"

#Since workspace is always cleaned, this step is optional

rm -rf ./$BUILD_VERSION

mkdir ./$BUILD_VERSION

echo "***Preparing to package project - rsync"

#skipping files: .project *.java *.docx *.cpp *.php

#skipping folders: junk

rsync -rq --exclude={'.project','*.java','*.cpp','*.php'} --exclude={'junk'} ./cartridges/ $BUILD_VERSION

echo "***Packaging deployable artifact - 7z"

#"-mx[N]" specifies compression level

#"-mmt[N]" set number of CPU threads

7zz a $BUILD_VERSION.zip $BUILD_VERSION -mx9 -mmt16

Code deploy

Nothing new here, feel free to reuse the one we did on our first iteration.

Have fun!