Multiple repositories Pull Request chaos, crawl them all in one single place

Blog: Drools & jBPM Blog

It is very frequent to find software engineering projects where multiple repositories are involved for the same or different projects, somehow related between them, a lot of people push their pull requests to any of them and it is very normal to lose tracking of the situation or you have to constantly browse them all to have a clearer picture about what is going on. That’s the situation we had here at the Red Hat Business Automation team and we solved it by creating a helpful tool you can easily use for your set of projects, easy, quick and for free.

The cross-repo PRs problem

This is already covered by Cross-repo Pull Requests? build-chain tool to the rescue! entry, so feel free to read it in case you are not familiar with this kind of situation or concepts.

The Chain-Status solution

So we said to ourselves, what if we would have a centralized place, a web page for instance, to be able to see in a quick look what’s the situation about all the pull requests for all of our repositories? Chain Status was the solution.

Prerequisites:

It has to solve not only our particular problem, so anyone can use it.
It has to be public, no authentication required.
It has to be fast, we can’t wait for the whole pull request set to be crawled everytime anyone gets into the application.
Multiple streams or different project set can be handled in different views, like different products or product versions from the same place.
The content can be filtered out.

So the conclusion was to create in one hand a React web page to consume the pull request information from a static report and another tool to generate that report based on Github information. This way:

The information will be produced asynchronously, the frequency will be up to the user/developer and Github API rate limit problems will be avoided.
The information can be taken even from private repositories and be exposed publicly and no authentication will be required.
No waiting time while information is requested from Github service.
The webpage (HTML+JS files) can be stored on any web service, even on free services like Github Pages or Netlify.
No backend server is required.

Running example

You can check KIE RHBA status web page at https://kiegroup.github.io/droolsjbpm-build-bootstrap/status/kiegroup-status

How can I add it to my organization?

The best way to integrate this tool in your organization or set of repositories is by using the provided configurable Github actions. In particular this tool comes with two main easy-to-use actions:

Generate App: this action aims to build and copy the React web application inside your repository and publish it using gh-pages NPM tool.

Generate Data: given a project structure and some project information as input, this action is focused on generating the data report gathering the information using the Github API. This report is then used by the web application as a content source.

Thus, in order to use these actions on your organization, you only have to add two Github workflows (one per action) on your main repository as follows:

Prerequisites: having a Github token properly configured in your organization, see more details on how to configure it.

Generate app workflow (generate_status_page.yaml): add the Github workflow for the web page generation, this should generally be run only once (or whenever there are changes on the web app look and feel).

name: Generate status page
on: workflow_dispatch

jobs:
  generate-status-page:
    if: github.repository_owner == '<OWNER>'
    concurrency:
      group: generate-status-page
      cancel-in-progress: true
    strategy:
      matrix:
        os: [ubuntu-latest]
      fail-fast: true
    runs-on: ubuntu-latest
    name: Generate status page
    steps:
      - name: Generate status page
        uses: kiegroup/chain-status/.ci/actions/generate-app@main
        with:
          info-md-url: "<PATH-TO-INFO>"
          github-token: "${{ secrets.GITHUB_TOKEN }}"
          gh-pages-branch: "gh-pages"

Generate data workflow (generate_status_page_data.yaml): add the periodic workflow that will continuously generate the data fetched by the web application.

name: Generate status page data

on:
  workflow_dispatch:
  schedule:
    - cron: '0 * * * *'
jobs:
  generate-status-page-data:
    if: github.repository_owner == '<OWNER>'
    concurrency:
      group: generate-status-page-data
      cancel-in-progress: true
    strategy:
      matrix:
        os: [ubuntu-latest]
      fail-fast: true
    runs-on: ubuntu-latest
    name: Generate status page data
    steps:
      - name: Generate status page data
        uses: kiegroup/chain-status/.ci/actions/generate-data@main
        with:
          definition-file: <PATH-TO-DEFINITION-FILE>
          # projects: <PROJECTS-LIST>
          title: <TITLE>
          subtitle: <SUBTITLE>
          base-branch-filter: <BRANCH-LIST>
          created-by: Github Action
          created-url: https://github.com/${{ github.repository }}/actions/runs/${{ github.run_id }}
          logger-level: debug
          github-token: "${{ secrets.GITHUB_TOKEN }}"
          gh-pages-branch: "gh-pages"

As already introduced, the generate data flow relies on a project structure definition which can be provided either using build-chain definition file or a projects list:

Build-chain definition file (using ‘definition-file’ field), a YAML definition file for cross-related inter-dependent projects which was introduced for Github action build chain. This tool is already covered by Cross-repo Pull Requests? build-chain tool to the rescue!, so feel free to read it if you want to get more details on it and on its definition files.

Projects list (using ‘projects’ field), a comma-separated list of projects for which you would like to provide Pull Requests statuses. [Still a Work in Progress PR-35]

This was a brief explanation on how you could integrate this tool in your organization, if you need more details on this feel free to reach the chain-status homepage, where you can find a step-by-step guide on how to integrate it with some links to running examples.

Additional functionalities

Additionally to the pull request summary functionality, it is also possible to add multiple Jenkins status reports.

The main advantage of this feature is that you can check the status of all your Jenkins jobs in a single place, making it easier to check what runs succeeded/failed and also the time and average time jobs are consuming.

As an example you can check the KIE RHBA daily builds page https://kiegroup.github.io/droolsjbpm-build-bootstrap/job/daily-builds

To configure the Jenkins status reports feature, you can create a Jenkins pipeline that will generate and update the data periodically. You can schedule the Jenkins pipeline to run and keep the status updated based on your required demand.

You can add the following steps as part of your Jenkins pipeline to generate and update the status report:

Clone the GitHub pages repository

stage('Clone gh-pages repository') {
  steps {
    script {
      println "Checking out https://github.com/${ghPagesRepository}:${ghPagesBranch} into ${ghPagesRepoFolder} folder"
      sh "git clone -b ${ghPagesBranch} --single-branch https://github.com/${ghPagesRepository} ${ghPagesRepoFolder}"
    }
  }
}

Install the chain-status tool

stage('Install chain-status tool') {
  steps {
    script {
      try {
        sh "npm install -g @kie/chain-status-action"
      } catch(e) {
        println '[WARNING] Error installing @kie/chain-status-action.'
      }
    }
  }
}

Generate the updated data

stage('Generate data') {
  steps {
    script {
      dir(ghPagesRepoFolder) {
        sh "build-chain-status-report --jenkinsUrl ${jenkinsURL} --jobUrl ${jenkinsJobPath} -t ${projectTitle} -st ${projectSubtitle} --certFilePath ${jenkinsCertFile} --outputFolderPath ./data/ --skipZero -cb "Jenkins Job" -cu "${env.BUILD_URL}" --order 1001"
      }
    }
  }
}

Push changes to update the status report

stage('Push changes to repository') {
  steps {
    script {
      println "Pushing changes to ${ghPagesRepository}:${ghPagesBranch}"
        dir(ghPagesRepoFolder) {
          withCredentials([usernamePassword(credentialsId: "${githubCredentialsId}", usernameVariable: 'GITHUB_USER', passwordVariable: 'GITHUB_TOKEN')]) {
githubscm.setUserConfig("${GITHUB_USER}")
          sh("git config --local credential.helper "!f() { echo username=$GITHUB_USER; echo password=$GITHUB_TOKEN; }; f"")
          sh 'git add data/*'
          sh 'git commit -m "Generate Jenkins Data"'
          sh "git push origin ${ghPagesBranch}"                          
        }
      }
    }
  }
}

Next steps and limitations

Historic functionality

Since the generator tool registers every day status, we expect to offer the historic view functionality to be able to compare status between dates.See https://github.com/kiegroup/chain-status/issues/29

To cover not only Github but other repository services

Right now we only cover Github for the generator tool to take information from, but we expect to cover another kind of services like Gitlab or Bitbucket.

Conclusion

We have been using this tool for RHBA, Kogito and Optaplanner repositories for a year and we can say it’s a very useful tool which solves the cross-repo pull requests summary problem. After a year of experience with the tool we can say the tool offers:

To be able to constantly see the status of the different contributions from the different people.
Who is working on what, like which are my own open pull requests.
To quickly check obsolete contributions and to be able to keep our repositories very clean.
To publicly offer Jenkins jobs summary no matter whether the Jenkins is accessible or not.
To quickly check how healthy our CI/CD stuff is thanks to the error index information from the tool.
To be able to see related pull requests for every pull request, thanks to the cross-repo pull request functionality.