Thursday, August 8, 2013

Performing conditional checkout when building with Maven/Tycho

One of the things I really loved about building with Buckminster was its support for conditional component materialization, more about it here. This is one of the features I missed when I moved build system to Maven/Tycho, how am I suppose to checkout pluginA and pluginB from trunk, but pluginC from branchA? I also wanted to brush up on my Groovy skills because I haven't used it in quite some time.

The idea I head on which components to checkout was to query parent (super) POM of my parent plugin. Each parent POM contains modules section that describes what are the modules that make up the build. One I query the parent POM, I would create a collection of modules and use that information to either checkout or update working directory, in my case Jenkins workspace. Below is the groovy script that I wrote.

 * Tycho/Maven builds rely on the fact that components already exist in build workspace (Jenkins in our case).
 * There is no easy way to checkout what's needed especially with some logic involved.
 * This groovy script checks out components from SVN repository and relies on information in parent POM, located in this plugin.
 * This script will parse POM file and construct a collection of <module> elements, currently module element looks as following:
 *   <module>../{componentName}</module>
 * Name of component will be normalized to just its name.
 * The convention in our build is that parent plugin is checked out using Jenkin's SCM and build.xml is called, build.xml in turn executes
 * this groovy script. Parent plugin MUST end with ".parent".
 * Following construction of collection, workspace directory will be checked for existance of component that ends with "parent".
 * If such directory exists and size of directories is 1, checkout will be performed. Otherwsie update will be done.

import java.util.logging.ConsoleHandler;
import java.util.logging.Logger

import groovy.util.logging.Log;
import groovy.util.slurpersupport.NodeChild;

public class Checkout{
 static def pomFile
 static def svnRoot
 static def svnProjectDir
 static def workspacePath
 static def svnDir
 static def queryFile
 static def ant
 static Logger logger = Logger.getLogger(Checkout.class.toString())

        // namespace of the components. This is used to decide whether checkout or update will be performed
 static def namespace = "com.iwaysoftware"

 public static void main(String[] args) {
  pomFile = args[0]
  svnRoot = args[1]
  svnProjectDir = args[2]
  workspacePath = args[3]
  svnDir = args[4]
  queryFile = args[5]
  /*CustomFormatter formatter = new CustomFormatter()
  ConsoleHandler handler = new ConsoleHandler()
  logger.addHandler(handler)*/"Script arguements:")"pomFile = " + args[0])"svnRoot = " + args[1])"svnProjectDir = " + args[2])"workspacePath = " + args[3])"libDir = " + args[4])"queryFile = " + args[5])

  ant = new AntBuilder()
  ant.typedef(resource: 'org/tigris/subversion/svnant/svnantlib.xml'){
   classpath {
    fileset(dir: svnDir, includes: '*.jar')
  new Checkout().startCheckout()


 private void startCheckout(){
  // parse POM file and construct collection modules consisting of module names
  def records = new XmlSlurper().parse(new File(pomFile))
  def modules = records.depthFirst().findAll{ == 'module'
  }"modules.size = " + modules.size())

  def advisors = getAdvisors(queryFile)

  // normalize collection items by stipping leading '../'
  modules = modules.collect {
   it.toString().substring(3, it.toString().size())

  // Query workspace directory for presence of a directory (plugin), here convention is that
  // parent plugin with parent POM is checked out first following by call to build.xml in the plugin.
  // Depending on component size and its values either checkout or update will be performed.
  def components = []"workspace = " + workspacePath)
  def workspaceLocation = new File(workspacePath)
  workspaceLocation.eachFileRecurse (FileType.DIRECTORIES) { file ->
    components << file
  }"components.size = " + components.size())

  // check if components collection is of size one and name of directory (plugin) ends with "parent"
  if(components.size() == 1 && components[0].toString().endsWith("parent")){"modules = " + modules)
   modules.each {
    def checkoutUrl = svnProjectDir + "/" + it"checkoutUrl = " + checkoutUrl)"workspace = " + workspacePath)
    def workingDirectoryPath = workspacePath + "/" + it"workingDirectoryPath = " + workingDirectoryPath)

     checkoutUrl = constructCheckoutUrl(it, advisors)
    }"checkout url:" +  checkoutUrl)

    // start checkout of components in modules collection
    ant.svn(javahl: 'false', svnkit: 'true'){
     checkout(url: checkoutUrl, destPath: workingDirectoryPath)
  // workspace directory already contains checkout components, perform update
  else if(components.size() > 1){"modules.size = " + modules.size())
   modules.each {"updating " + it)
    def updateDir = workspacePath + "/" + it
    ant.svn(javahl: 'false', svnkit: 'true'){ 
     update(dir: updateDir) 

  * Iterates branches List and checks whether SVN location at ${svnRoot}/branches/${branch}/${module} exists.
  * If it does matched SVN location will be checked out, if not another branch from the List is considered,
  * if none of the SVN branch locations exist, module will be checked out from trunk.
 private String constructCheckoutUrl(String module, Map advisors){
  def branches = advisors.get(module)

  for(branch in branches){
    ant.svn(javahl: 'false', svnkit: 'true'){
     info(target: svnRoot + "/branches/" + branch + "/" + module)
    def url = svnRoot + "/branches/" + branch + "/" + module"constructed url: " + url)
    return url
    logger.warning("No SVN INFO for " + svnRoot + "/branches/" + branch + "/" + module)
  return svnProjectDir + "/" + module

  * Reads advisor elements from .query file and constucts map of component:modules[:]
  *  <advisor component="com.iwaysoftware.eclipse.workspace" branch="adapter,test"/>
 private Map getAdvisors(String queryFile) {
  def root = new XmlSlurper().parse(new File(queryFile))
  def advisorsList = root.depthFirst().findAll{ == 'advisor'

  def advisorsMap = [:]
  advisorsList.each {
   def key
   def branches = []

   if(it instanceof NodeChild){
    def attributes = ((NodeChild)it).attributes()
    key = attributes.get("component")
    branches = attributes.get("branch").split(",")
    advisorsMap.put(key, branches)
  }"advisors: " + advisorsMap)
  return advisorsMap
And here the ant script that calls this groovy script
<?xml version="1.0" encoding="UTF-8"?>
<project name="svnCheckout" default="checkout">
 <property environment="env"/>
 <property name="svn.root" value="/path/to/svn/root"/>
 <property name="svn.project.dir" value="/path/to/svn/project/root"/>
 <property name="svn.lib" location="/path/to/svn/libs" />
 <property name="groovy.lib" location="/path/to/groovy/libs" />
 <property name="workspace" location="${env.WORKSPACE}" />
 <property name="basedir" value="." />
 <property name="groovy.script" location="/path/to/groovy/script/Checkout.groovy" />
 <property name="advisors" location="${basedir}/adapterBranch.query"/>

 <taskdef name="groovy"
  classname="org.codehaus.groovy.ant.Groovy" />

 <target name="checkout">
   <groovy src="${groovy.script}">
    <!-- pom file -->
    <arg line="${basedir}/pom.xml"/>
    <!-- SVN root -->
    <arg line="${svn.root}"/>
    <!-- SVN location of project that contains components to be build -->
    <arg line="${svn.project.dir}"/>
    <!-- workspace, if using Jenkins use ${env.WORKSPACE} -->
    <arg line="${workspace}"/>
    <!-- location of SVNAnt jars -->
    <arg line="${svn.lib}"/>
    <!-- location of advisors query file -->
    <arg line="${advisors}"/>
As you can see this script just calls groovy script with needed parameters.

Here is the .query file that is used by the script to decide whether to checkout components from branch or trunk.
   <advisor component="pluginA" branch="adapter"/>
   <advisor component="pluginB" branch="adapter"/>
   <advisor component="featureA" branch="adapter"/>
   <advisor component="featureX" branch="adapter"/>
There are still things to be done to improve this script. Adding regular expressions to .query files to make file smaller in cases that there are a lot of repetition of components. Addition of consideration of tags SVN directory is something that will be useful as well. Right now during update I go through collection of checked out components and update them all. Updating can be costly task (time wise), so using svn info command to compare revisions will cut update time.

Blogger Syntax Highliter