================================
Rocket Software, Inc. and subsidiaries ■ 77 Fourth Avenue, Waltham MA 02451 ■ Main Office Toll Free Number: +1 855.577.4323
Contact Customer Support: https://my.rocketsoftware.com/RocketCommunity/RCEmailSupport
Unsubscribe from Marketing Messages/Manage Your Subscription Preferences - http://www.rocketsoftware.com/manage-your-email-preferences
Privacy Policy - http://www.rocketsoftware.com/company/legal/privacy-policy
================================

This communication and any attachments may contain confidential information of Rocket Software, Inc. All unauthorized use, disclosure or distribution is prohibited. If you are not the intended recipient, please notify Rocket Software immediately and destroy all copies of this communication. Thank you.

----- Original message -----
From: "Daniel Pono Takamori" <dtakamori@...>
Sent by: zowe-dev@...
To: Jack-Tiefeng Jia <Jack-Tiefeng.Jia@...>, zowe-dev@...
Cc: Andrew Grimberg <agrimberg@...>, John Mertic <jmertic@...>
Subject: [EXTERNAL] Re: [zowe-dev] Expiring project Giza dot org domain
Date: Mon, Nov 25, 2019 4:56 PM
 

Hey Jack and Team,

First note is about permission for starting/ editing jobs: 

Unfortunately on the production Jenkins we do not allow triggering of arbitrary jobs from the UI (or API as you noted). However, on jenkins.zowe.org/sandbox Jack and Mark are in a group that allows them to start jobs at will (without the dance of posting 'retry' or 'recheck' [0] in the comments on a Github PR). This group can be added to by filing a ticket on support.linuxfoundation.org [1].  Is this an integral part of the workflow _after_ jobs are merged somewhere?  (currently we don't allow that permission on the production Jenkins instances, but could look into options if this is a necessary feature).

Since you are one of the first projects using Pipelines, our usual workflow of using `jenkins-job-builder` [2] will need to be looked at and reconsidered to fit your needs.

Secondly, I'm going to rebuild the Jenkins builder nodes with the containers already in them to reduce the bandwidth/ time overhead of pulling them in.  We can update them when new containers are released, but in the meantime if you are building them from the same underlying bits, it'll drastically reduce the time spent fetching new layers.

Thirdly, I've responded on the ticket about the Artifactory token.

 

I apologize for not being more a part of the CI discussions; I've joined the zowe-dev list to track updates and feel free to reach out for how we can better help you in this process!

 

-Daniel Pono Takamori on behalf of LF CI/ Release Engineering

 

[0] - https://github.com/lfit/releng-global-jjb/blob/7a5d4641fd29ff65e870ce0f8dfd686f943ef603/jjb/lf-ci-jobs.yaml#L522

[1] - https://jira.linuxfoundation.org/servicedesk/customer/kb/view/28477040

[2] - https://docs.releng.linuxfoundation.org/en/latest/jenkins-sandbox.html

 

On Fri, Nov 22, 2019 at 11:36 AM Jack-Tiefeng Jia <Jack-Tiefeng.Jia@...> wrote:

John, for sure. Currently when we test the Jenkins Library, which is a gradle build, the test starts with gradle test command, it will try to test several functionalities in the library. It's generally integration tests since when the code runs under Jenkins, it may behave differently when we run under JRE. Jenkins pipeline workflow plugin will convert the code into CPS https://github.com/cloudbees/groovy-cps/blob/master/doc/cps-basics.md.

 

For each major functionality, we will create a test Jenkins job on-the-fly with Jenkinsfile, and/or start the build with different parameters. The job could be a simple Pipeline, or a Multibranch Pipeline depends on the feature the test case is testing. For multibranch pipeline, we will also trigger a scan job to find the branches.

 

With current LF Jenkins setup, we don't have these permissions:

- create a job on-the-fly (or through Jenkins API)

- start a scan on Multibranch Pipeline job (or through Jenkins API)

- start a job on Jenkins (or through Jenkins API)

- start a job with parameters on Jenkins (or through Jenkins API)

- delete the temporary test job when the test is done

 

So we have to figure other ways to perform the tests.

 

For the permission of starting a job with parameters on Jenkins, we do rely a lot on this with our own Jenkins. Few scenarios like:

- start a smoke test on any build we produced,

- or start a smoke test on a different testing server.

- parameters to decide if/how we want to release.

 

For all the permission limitations, I believe we can have workarounds, just not ideal. Like we have to create dummy branches/commits to the github to trigger a build. The dummy commits eventually will end up to many merging conflicts, which is minor but have to deal with. To create workaround on testing Jenkins Library, we will need to change a lot on the way how we start our test.

 

FYI, another workaround we have already added is building in docker containers. The Jenkins has docker support VM agent, but the agent is not reusable. We can start the container with the build, but every build it will need to download the base build image again.  For example, https://jenkins.zowe.org/job/zowe-docs-site/job/users%252Fjack%252Flf-jenkins/6/execution/node/8/log/. Currently the base image is 1.24G, which create extra overhead to the build and server traffic. We may be able to try to make the base image slightly smaller, but I think with JDK8 and node.js v8 or v12 pre-installed, it will be still about 600M to 900M.

 

Hope this explains the issues we are having.

Thanks,   Jack (T.) Jia Software Developer IBM Z Management Software IBM Systems

Phone: 1-905-413-3195 E-mail: jack-tiefeng.jia@...	8200 Warden Ave Markham, ON L6G 1C7 Canada

 

 

----- Original message -----
From: "John Mertic" <jmertic@...>
Sent by: zowe-dev@...
To: zowe-dev@..., Andrew Grimberg <agrimberg@...>, Daniel Pono Takamori <dtakamori@...>
Cc:
Subject: [EXTERNAL] Re: [zowe-dev] Expiring project Giza dot org domain
Date: Fri, Nov 22, 2019 11:34 AM
 

Hey Jack!

 

Adding +Andrew Grimberg and +Daniel Pono Takamori to this thread...

 

I will keep investigating the possibilities of migrating everything to LF Jenkins. One difficulty we have is the "Jenkins Library" build which will fail on LF Jenkins due to the permission limitation we have. Will see what we can do. The tests for Jenkins Library code is suggested because if it fails, it may indicate every other builds using that branch will fail. The test has to be performed against a real Jenkins server, not a virtual JVM environment because Jenkins handles Groovy scripts differently. The way of testing Jenkins Library is it will try to create a job through Jenkins API and start the build and check result, which is against the permission scheme we have on LF Jenkins.

 

Can you elaborate on the permissions issues? I thought we had built the environment to align with the existing practices, but wondering if something changed from discovery to implementation.

 

Thanks!

 

John

 

 

 

----- Original message -----
From: "Daniel Pono Takamori" <dtakamori@...>
Sent by: zowe-dev@...
To: Jack-Tiefeng Jia <Jack-Tiefeng.Jia@...>, zowe-dev@...
Cc: Andrew Grimberg <agrimberg@...>, John Mertic <jmertic@...>
Subject: [EXTERNAL] Re: [zowe-dev] Expiring project Giza dot org domain
Date: Mon, Nov 25, 2019 4:56 PM
 

Hey Jack and Team,

First note is about permission for starting/ editing jobs: 

Unfortunately on the production Jenkins we do not allow triggering of arbitrary jobs from the UI (or API as you noted). However, on jenkins.zowe.org/sandbox Jack and Mark are in a group that allows them to start jobs at will (without the dance of posting 'retry' or 'recheck' [0] in the comments on a Github PR). This group can be added to by filing a ticket on support.linuxfoundation.org [1].  Is this an integral part of the workflow _after_ jobs are merged somewhere?  (currently we don't allow that permission on the production Jenkins instances, but could look into options if this is a necessary feature).

Since you are one of the first projects using Pipelines, our usual workflow of using `jenkins-job-builder` [2] will need to be looked at and reconsidered to fit your needs.

Secondly, I'm going to rebuild the Jenkins builder nodes with the containers already in them to reduce the bandwidth/ time overhead of pulling them in.  We can update them when new containers are released, but in the meantime if you are building them from the same underlying bits, it'll drastically reduce the time spent fetching new layers.

Thirdly, I've responded on the ticket about the Artifactory token.

 

I apologize for not being more a part of the CI discussions; I've joined the zowe-dev list to track updates and feel free to reach out for how we can better help you in this process!

 

-Daniel Pono Takamori on behalf of LF CI/ Release Engineering

 

[0] - https://github.com/lfit/releng-global-jjb/blob/7a5d4641fd29ff65e870ce0f8dfd686f943ef603/jjb/lf-ci-jobs.yaml#L522

[1] - https://jira.linuxfoundation.org/servicedesk/customer/kb/view/28477040

[2] - https://docs.releng.linuxfoundation.org/en/latest/jenkins-sandbox.html

 

On Fri, Nov 22, 2019 at 11:36 AM Jack-Tiefeng Jia <Jack-Tiefeng.Jia@...> wrote:

John, for sure. Currently when we test the Jenkins Library, which is a gradle build, the test starts with gradle test command, it will try to test several functionalities in the library. It's generally integration tests since when the code runs under Jenkins, it may behave differently when we run under JRE. Jenkins pipeline workflow plugin will convert the code into CPS https://github.com/cloudbees/groovy-cps/blob/master/doc/cps-basics.md.

 

For each major functionality, we will create a test Jenkins job on-the-fly with Jenkinsfile, and/or start the build with different parameters. The job could be a simple Pipeline, or a Multibranch Pipeline depends on the feature the test case is testing. For multibranch pipeline, we will also trigger a scan job to find the branches.

 

With current LF Jenkins setup, we don't have these permissions:

- create a job on-the-fly (or through Jenkins API)

- start a scan on Multibranch Pipeline job (or through Jenkins API)

- start a job on Jenkins (or through Jenkins API)

- start a job with parameters on Jenkins (or through Jenkins API)

- delete the temporary test job when the test is done

 

So we have to figure other ways to perform the tests.

 

For the permission of starting a job with parameters on Jenkins, we do rely a lot on this with our own Jenkins. Few scenarios like:

- start a smoke test on any build we produced,

- or start a smoke test on a different testing server.

- parameters to decide if/how we want to release.

 

For all the permission limitations, I believe we can have workarounds, just not ideal. Like we have to create dummy branches/commits to the github to trigger a build. The dummy commits eventually will end up to many merging conflicts, which is minor but have to deal with. To create workaround on testing Jenkins Library, we will need to change a lot on the way how we start our test.

 

FYI, another workaround we have already added is building in docker containers. The Jenkins has docker support VM agent, but the agent is not reusable. We can start the container with the build, but every build it will need to download the base build image again.  For example, https://jenkins.zowe.org/job/zowe-docs-site/job/users%252Fjack%252Flf-jenkins/6/execution/node/8/log/. Currently the base image is 1.24G, which create extra overhead to the build and server traffic. We may be able to try to make the base image slightly smaller, but I think with JDK8 and node.js v8 or v12 pre-installed, it will be still about 600M to 900M.

 

Hope this explains the issues we are having.

Thanks,   Jack (T.) Jia Software Developer IBM Z Management Software IBM Systems

Phone: 1-905-413-3195 E-mail: jack-tiefeng.jia@...	8200 Warden Ave Markham, ON L6G 1C7 Canada

 

 

----- Original message -----
From: "John Mertic" <jmertic@...>
Sent by: zowe-dev@...
To: zowe-dev@..., Andrew Grimberg <agrimberg@...>, Daniel Pono Takamori <dtakamori@...>
Cc:
Subject: [EXTERNAL] Re: [zowe-dev] Expiring project Giza dot org domain
Date: Fri, Nov 22, 2019 11:34 AM
 

Hey Jack!

 

Adding +Andrew Grimberg and +Daniel Pono Takamori to this thread...

 

I will keep investigating the possibilities of migrating everything to LF Jenkins. One difficulty we have is the "Jenkins Library" build which will fail on LF Jenkins due to the permission limitation we have. Will see what we can do. The tests for Jenkins Library code is suggested because if it fails, it may indicate every other builds using that branch will fail. The test has to be performed against a real Jenkins server, not a virtual JVM environment because Jenkins handles Groovy scripts differently. The way of testing Jenkins Library is it will try to create a job through Jenkins API and start the build and check result, which is against the permission scheme we have on LF Jenkins.

 

Can you elaborate on the permissions issues? I thought we had built the environment to align with the existing practices, but wondering if something changed from discovery to implementation.

 

Thanks!

 

John