Create multiple stages with parallel execution on nodes
Our cross-platform projects require Jenkins to run on several different platforms, and each has been tested and packaged accordingly. I was able to combine parallel
with node
but only in one stage
and got it far (see below)
I want to break it down into several steps. The stages I would like to create:
- Build: Build a library / project with artifacts
- UnitTest: creating and running unit tests
- TestApp: Build and run test applications with artifacts
- Entry stage: want to download?
- Upload: upload artifacts to an external server
Should I do the following:
- Copy a bunch of objects between stages (using
stash
) - Consistency of use must be maintained
node
. (objects created on a nodelabel
must be marked appropriately so that they are unlocked on the corresponding nodelabel
). I would need every shortcut of every tag for every node.
Isn't that much more inefficient? I would artificially copy a lot of data so that I can create stages.
def checkoutAndBuild(Map args) {
node("${args.nodeName}") {
checkout([$class: 'GitSCM',
branches: scm.branches,
doGenerateSubmoduleConfigurations: scm.doGenerateSubmoduleConfigurations,
extensions: scm.extensions +
[[$class: 'SubmoduleOption',
disableSubmodules: false,
parentCredentials: false,
recursiveSubmodules: true,
reference: '',
trackingSubmodules: false]] +
[[$class: 'CleanCheckout']],
userRemoteConfigs: scm.userRemoteConfigs
])
step([$class: 'CopyArtifact',
filter: "AppCommon/*/**, cmake/**/*, core/**/*, thirdparty/prebuilt/${args.prebuiltDir}/**/*, tools/**/*",
fingerprintArtifacts: true,
projectName: "${args.engineDependency_Job}",
selector: [$class: 'SpecificBuildSelector', buildNumber: "${args.engineDependency_BuildNo}"],
target: 'engine'])
dir("build/${args.buildDir}") {
echo 'Building..'
if (isUnix()) {
sh './build.sh Release'
} else {
bat 'build.bat Release'
}
}
def extras = args.additionalArtifacts ? ", ${args.additionalArtifacts}" : ""
archiveArtifacts artifacts: "dist/**/*${extras}", fingerprint: true
dir("test/build") {
echo 'Building test App'
sh "./full.sh ${args.buildDir} Release"
}
}
}
pipeline {
agent none
stages {
stage('Info') {
agent any
steps {
echo "Running ${env.JOB_NAME} / ${env.BUILD_ID} on ${env.JENKINS_URL}"
}
}
stage('Build') {
steps {
parallel (
ios: {
checkoutAndBuild nodeName: 'iOS', prebuiltDir: 'ios', buildDir: 'ios', engineDependency_Job: 'engine_iOS_Release', engineDependency_BuildNo: String.valueOf(engineBuild.ios)
},
tvos: {
checkoutAndBuild nodeName: 'tvOS', prebuiltDir: 'tvos', buildDir: 'tvos', engineDependency_Job: 'engine_tvOS_Release', engineDependency_BuildNo: String.valueOf(engineBuild.tvos)
},
android: {
checkoutAndBuild nodeName: 'Android', prebuiltDir: 'android', buildDir: 'AndroidNative', engineDependency_Job: 'engine_Android_Release', engineDependency_BuildNo: String.valueOf(engineBuild.android), additionalArtifacts: 'src/java/*'
})
}
}
stage('Test Build') {
steps {
echo 'Testing...'
}
}
stage('Deploy') {
steps {
echo 'Deploying...'
}
}
}
post {
success {
slackSend channel: '#builds',
color: 'good',
message: "${currentBuild.fullDisplayName} succeeded. (<${env.BUILD_URL}|Open>)"
}
failure {
slackSend channel: '#builds',
color: 'danger',
message: "${currentBuild.fullDisplayName} failed. (<${env.BUILD_URL}|Open>)"
}
}
}
source to share
The declarative pipeline syntax is not very flexible in your case, unfortunately. Standing objects between stages will be quite heavy and this will result in synchronous stages where the fastest build target waits for the slower ones before each stage.
If you don't need to synchronize the stages of a stage, I suggest creating one large method that contains all stages of all build targets, the ones you want to run in parallel. To differentiate between stages, you can, for example, add a node name to each stage label.
def checkoutBuildTestDeploy(Map args) {
node("${args.nodeName}") {
stage("Build ${args.nodeName}") {
checkout([$class: 'GitSCM', ... ])
// And other build steps ...
}
stage("Unit test ${args.nodeName}") {
// Unit test steps
}
stage("Test app ${args.nodeName}") {
// Test steps
}
stage("Deploy ${args.nodeName}") {
// Input answer and upload
}
}
}
pipeline {
agent none
stages {
stage('Info') {
agent any
steps {
echo "Running ${env.JOB_NAME} / ${env.BUILD_ID} on ${env.JENKINS_URL}"
}
}
stage('Run builds parallel') {
steps {
parallel (
ios: {
checkoutBuildTestDeploy nodeName: 'iOS', prebuiltDir: 'ios', buildDir: 'ios', engineDependency_Job: 'engine_iOS_Release', engineDependency_BuildNo: String.valueOf(engineBuild.ios)
},
tvos: {
checkoutBuildTestDeploy nodeName: 'tvOS', prebuiltDir: 'tvos', buildDir: 'tvos', engineDependency_Job: 'engine_tvOS_Release', engineDependency_BuildNo: String.valueOf(engineBuild.tvos)
},
android: {
checkoutBuildTestDeploy nodeName: 'Android', prebuiltDir: 'android', buildDir: 'AndroidNative', engineDependency_Job: 'engine_Android_Release', engineDependency_BuildNo: String.valueOf(engineBuild.android), additionalArtifacts: 'src/java/*'
})
}
}
}
post {
success {
slackSend channel: '#builds',
color: 'good',
message: "${currentBuild.fullDisplayName} succeeded. (<${env.BUILD_URL}|Open>)"
}
failure {
slackSend channel: '#builds',
color: 'danger',
message: "${currentBuild.fullDisplayName} failed. (<${env.BUILD_URL}|Open>)"
}
}
}
source to share