问题背景

How can I limit number of certain pipelines running in parallel?

1
2
3
4
https://stackoverflow.com/questions/50578473/how-can-i-limit-number-of-certain-pipelines-running-in-parallel



have a simple way to limit the number of parallel branches that run concurrently

1
2
3
4
5
6
7
8
https://issues.jenkins-ci.org/browse/JENKINS-44085




https://issues.jenkins-ci.org/browse/JENKINS-26125


Throttle parallel step in pipeline script

1
https://issues.jenkins-ci.org/browse/JENKINS-46236
1
2
3

https://github.com/jenkinsci/pipeline-plugin/blob/workflow-1.15/cps/src/main/java/org/jenkinsci/plugins/workflow/cps/steps/ParallelStep.java#L47

Jenkins流水线怎么限制parallel并发个数呢？

Throttle parallel step in pipeline script https://issues.jenkins.io/browse/JENKINS-46236
have a simple way to limit the number of parallel branches that run concurrently
https://issues.jenkins.io/browse/JENKINS-44085

1
2
3
4
5
6
7
8
9
10
11
12
13
stage('Test') {
  steps {
    script {
      testing_closures = [one: { print("starting one"); sleep 10; print("finishing one") },
                          two: { print("starting two"); sleep 10; print("finishing two") },
                          three: { print("starting three"); sleep 10; print("finishing three") },
                          four: { print("starting four"); sleep 10; print("finishing four") },
                          five: { print("starting five"); sleep 10; print("finishing five") },
                          six: { print("starting six"); sleep 10; print("finishing six") }]
      parallel(testing_closures)
    }
  }
}
1
2
3
4
5
6
7
8
9
10
The main goal is to throttle those closures. If we don't want for all six of them to run concurrently, but let's say only 3 at a time, we'd need an argument for 'parallel' step:
parallel(closures: testing_closures, maxThreadCount: 3)

Currently, as far as I know, there are no workarounds for this issue. If JENKINS-46235 is resolved, the following workaround could be used:

Dynamically create 3 lockable resources via LockableResourcesManager::createResourceWithLabel() with build-unique labels
Lock them by label in all of the closures
The closures will wait for each other to finish and only 3 at the time would be running.
Delete the closures (no such API method. JENKINS-46235 is for exposing 'delete' method for lockable resource)

思路：为什么不把任务列表转换为 一个 二维的任务列表呢。
顺序的执行二维列表的 每个子列表， 按照一个stage 一个 stage的顺序去执行。
每个stage中再使用并发去执行子列表中的任务。例如下面。循环套循环。第一层循环使用顺序的stage。第二次循环使用并发的stage。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64

def list

pipeline {
    agent {label 'master'}
    stages {
        stage('Create List') {
            steps {
                script {
                    // you may create your list here, lets say reading from a file after checkout
                    list = ["Test-1", "Test-2", "Test-3", "Test-4", "Test-5"]
                }
            }
            post {
                cleanup {
                    cleanWs()
                }
            }
        }
        stage('Dynamic Stages') {
            steps {
                script {
                    for(int i=0; i < list.size(); i++) {
                        stage(list[i]){
                            echo "Element: $i"
                            sleep 20
                            
                                            script {
                                                def nodeList = "bf-01,bf-02,bf-03".split(",").findAll { it }.collect { it.trim() }
                                                def jobs = [:]
                            
                                                for (int ii = 0; ii < nodeList.size(); ii++) {
                                                    def app = nodeList[ii]
                                                    jobs["jobs-${app}"] = {
                                                        node('another-master') {
                                                            stage("Build ${app}") {
                                                                echo "Build ${app}"
                                                                sleep 10
                                                            }
                                                        }
                                                    }
                                                }
                                                parallel jobs
                                            }
                
                
                
                        }
                    }
                }
            }
            post {
                cleanup {
                    cleanWs()
                }
            }
        }
        
        
        
        
    }
}