Clever Geek Handbook

Using Jenkins, Perforce and Ant, how can I only run PMD on files that have changed since the last green build?

Considering that:

  • There seems to be no easy way to get the list of "changed" files in Jenkins (see here and here )
  • There seems to be no quick way to get the list of files changed with label xxxx

How can I optimize our build so that when PMD starts, it only works on files that have changed since the last green build.

Backing up a bit ... our PMD takes 3-4 minutes to run with ~ 1.5 million lines of code, and if it detects a problem, the report always runs out of memory before it completes. I would like to cut a couple of minutes from our build time and get a nice failure report. My original approach was that I:

  • get changelog from Jenkins
  • run PMD against combining this list and the contents of pmd_failures.txt
  • If PMD doesn't work, include the list of files with error in pmd_failures.txt

More complex than I would like, but worth having an assembly that's faster but still reliable.

Once I realized that Jenkins was not going to easily give me what I wanted, I realized that there was another possible approach. We label each green assembly. I could just get the list of files changed with the tag and then delete the pmd_failures.txt file completely.

No cubes. The idea of ​​getting a list of files has changed since the xxxx label from Perforce seems to have never been optimized:

    $ p4 files // path / to / branch / ... @ label > label.out
    $ p4 files // path / to / branch / ... @ now > now.out
    $ diff label.out now.out

Annoyingly, what's even more important, even for our thousands of files, than just running PMD.

So now I'm trying to run PMD in parallel with other build materials, which are still wasting time and resources and making our build more complex. It seems to me that I cannot easily get the list of changed files from Jenkins or from Perforce. Has anyone else found a reasonable workaround for these problems?

+3


source to share


3 answers


I think I found the answer and I will mark my answer as correct if it works.

This is a little more complicated than I would like, but I think it costs 3-4 minutes (and potential memory issues).

  • At the end of a good build, save a good changelog as a Perforce counter. (post build task). Looks like:
    $ p4 counter last_green_trunk_cl% P4_CHANGELIST%
  • When running PMD, read the counter in the property last.green.cl

    and get the list of files from:
    $ p4 files // path / to / my / branch / ... @ $ {last.green.cl}, now
//path/to/my/branch/myfile.txt#123 - edit change 123456 (text)
//path/to/my/branch/myotherfile.txt#123 - add change 123457 (text)
etc ...
(have to parse the output)
  • Run PMD with these files.


This way, we don't need the pmd_failures.txt file, and we only run PMD on files that have changed since the last green build.

[EDIT: changed it to use a p4 counter, which is much faster than checking in a file. Also, it was very successful, so I will mark it as an answer]

+4


source


I'm not 100% sure since I've never used Perforce with Jenkins, but I believe Perforce passes the changelog number through an environment variable $P4_CHANGELIST

. That being said, you can run p4 filelog -c $P4_CHANGELIST

which should provide you with the files from that changelog. From there it shouldn't be hard to script something to just get the modified files (plus the old PMD failures).



I have not used Perforce for a long time, but I find the setting -Ztag

makes it easier to parse the P4 output for different scripting languages.

+1


source


Have you thought about using automatic tags? They are basically just an alias for the changelog, so it's easier to get a set of files that differ between the two auto-tagging.

+1


source






More articles:


All Articles