By reading this site lately you would think I had given up my day job as a software developer; however, that is not the case. On the contrary I have been very busy with several projects. One such project is piecing together a continuous automation package. Basically, cron on steroids. I could have just used cron to do everything, but I needed standard reporting and logging features that would have been replicated in every cron script. That said, I decided to use CruiseControl and ANT. These tools are usually used in a software build environment, but they adapt nicely to other types of automation projects.

For my project I setup a standard CruiseControl, ANT build loop on a Linux server, and all was well…until one of the remote servers I connect to became eratic. I, unfortunately, do not have control over the remote server so I had to figure out how to put failure tollerance into my build loop. In a nuthsell, I needed to keep running the ANT script in a loop until I received what I was looking for. To complicate matters, I had to do this on a once a day basis as well. The primary problem with my setup is that CruiseControl does not have the ability to loop until success. Complicated problems call for creative solutions.

To get around the issue I decided to use a simple text file and a modification set inside of CruiseControl. Because of the sceduling I also had to use two CruiseControl project blocks. The first project block is on a daily schedule to call an ANT script and modify the “setup” file.

<project name="setup" requiremodification="false" buildafterfailed="true">
<modificationset>
<alwaysbuild/>
</modificationset>
<schedule>
<ant antscript="/usr/bin/ant" buildfile="build.xml" time="0800" day="monday" target="setup" />
    <ant antscript="/usr/bin/ant" buildfile="build.xml" time="0800" day="tuesday" target="setup" />
    <ant antscript="/usr/bin/ant" buildfile="build.xml" time="0800" day="wednesday" target="setup" />
    <ant antscript="/usr/bin/ant" buildfile="build.xml" time="0800" day="thursday" target="setup" />
    <ant antscript="/usr/bin/ant" buildfile="build.xml" time="0800" day="friday" target="setup" />
  </schedule>
</project>

Every week day at 8:00 AM an ANT build script is called and the setup target is ran. This target touches the setup file making it appear that it was modified. Here is the setup target from the ANT script:

<target name="setup">
<touch file="setup.txt"  />
</target>

At this point the second CruiseControl project block springs into action.

<project name="run" requiremodification="true" buildafterfailed="true">
<modificationset>
<filesystem folder="setup.txt"/>
</modificationset>
<schedule interval="1200">
<ant antscript="/usr/bin/ant" buildfile="build.xml" />
</schedule>
<publishers>
<onfailure>
<antpublisher antscript="/usr/bin/ant" buildfile="build.xml" target="setup"/>
</onfailure>
</publishers>
</project>

This project block runs every 20 minutes looking for a modification to the setup file. Because the first project modified the file at 8:00 this project will take notice and start running. If things go well this project executes once a day shortly after 8:00 AM. However, if the script fails, which it often does due to a connection timeout, I have safeguards in place to loop back and start over again.

If you notice there is a buildafterfailed parameter in the project tag. This tells CruiseControl that it is okay to run the project again if the previous run failed. This alone will not loop the project it merely states that it is alright to do it again. The real magic happens in the <onfailure> tag; this tag is only executed when the project exits with an exception. Inside of that tag I make a call to the same target I used in the first project. This target changes the date on the setup file again making it appear to be modified. When that happens the build project will run again on the next cycle because there has been a modification since the last execution.

This loop will continue until the build script in the second project exits cleanly or CruiseControl is stopped. Do note that it is possible to create an infinate loop this way. Make sure that the primary target in the build file will exit cleanly. It is also a good idea to have CruiseControl email you on failure and success.