Using Jenkins to Build from Git without the Git plug-in

A few months ago we decided to upgrade our source control provider and moved everything over to Visual Studio Online.  It’s been working great for source control, though getting used to using Git instead of the TFS source control is a bit of work.  For a few reasons we’re not using the build features of Visual Studi Online, but instead are using a Jenkins build server.  Jenkins is really nice and can do just about anything you could want, which is a big plus.  The only down side is that it’s all in Java.  Why is that a down side, you may wonder?  Well if things get broken, you’re in a pickle.

We were running all well and good for over a month.  Nightly builds were running.  Version umbers were auto-incrementing. Releases for Windows and Mono were getting auto-generated and getting FTPed up to the public release directory.  Things were great.  Until about a week before Christmas, when an update for the git plug-in was released.  The Git plug-in is what allows you to configure Jenkins to easily connect to a Git server and pull your source code.  Well the plug-in update broke the ability to get code from a Git server on Windows.  Now Jenkins has a rollback feature, and had I understood what the failure actually was (it wasn’t obvious that it was a plug-in failure) then I could have rolled back and probably been fine.  But I didn’t know.  And in my effort to “fix” things, I wiped out the roll-back archived version.

So the option was to either install a Java environment and try to learn how Jenkins works and fix it, or to wait for the community to fix the problem.  I opted to do the latter, because it surely would break other people and would get straightened out quickly, right?  Hmm, not so much it seems.  I found a reported bug and asked for a time estimate.  I waited a few days.  No fix.  I left the office for a week of “unplugged” vacation and came back.. No fix.  I then learned that you can access the nightly builds for the plug ins themselves (which is actually pretty cool) so I tried manually installing the latest builds of the plug-in.  Turns out it was still broken.

While I was trying to figure out what was broken, I also appear to have broken something in the Git workspace on the server too, so it was hard to tell if the plug-in was failing, or if Git was confused.  I know that I was confused.  So today I decided that I really needed to get this stuff working again.  I changed the Job to no longer use source control, but instead to just run Window batch files.

REM make sure nothing is hidden 
attrib -H /S
REM recursively remove child folders 
for /d %%X in (*.*) do rd /s /q "%%X"
REM delete files in root folder 
del /q /f *
REM get the source 
git init 
git clone https://[username]:[password]
REM check out
git checkout master
git add ./Common/SolutionFamily.VersionInfo.cs
REM increment the build number
powershell -File "%WORKSPACE%UtilitySetFamilyVersion.ps1" 2.1.%BUILD_NUMBER%.0
REM commit the change
git commit -a -m auto-version
git push https://[username]:[password]

Once that was done, the MSBUILD plug-in was then able to build from the workspace, though the source code directory had changed one level as compared to where the Git plug-in had been pulling code.  If I had wanted to, I could have had my command do the build as well and not even used the MSBUILD plug in by adding this to the end:

C:WindowsMicrosoft.NETFrameworkv4.0.30319msbuild.exe "/p:Configuration=Debug;Platform=Any CPU" /m "%WORKSPACE%SolutionFamilySolutionEngine.FFx.sln" && exit %%ERRORLEVEL%%

Once the Git plug-in is actually fixed, I’ll post how to use it to connect to Visual Studio Online.  It actually seems to be working “somewhat” this morning.  I say “somewhat” because while it actually is pulling the code and behaving properly, when you do the configuration you get an error, which makes it look like it’s going to fail.  Until that’s ironed out I’m going to wait.

Building a Mono Solution with Jenkins

I recently switched our entire build system over to using a Jenkins build server.  It’s been running for a couple weeks now and I keep expanding the jobs I have it doing, and all in all I’m very happy with how it’s going.  It’s certainly saved me a load of time and the fact we now are getting automated nightly builds of all of our installers is extremely valuable to us and our customers.

Once of the challenges in getting things working was getting the build of the Linux installer for Solution Engine automated.  The Jenkins Server is a Windows Server and Solution Engine in built using Mono, but the actual deployment package is a tarball.  Generating tarballs on a Windows platform really isn’t well documented or outlined anywhere that I could find, but I was able piece the process together from some help files, man pages and a lot of iterations.

In the end, the build portion of the job is done through four separate “Execute Windows Batch Command” steps.

Step 1: Compile the Solution

This one was pretty straightforward.  You simply use the xbuild.exe application that ships with Mono and point it at your Visual Studio/Xamarin Solution File:

"C:Program Files (x86)Mono-3.2.3libmono4.5xbuild.exe" /p:Configuration=Debug SolutionEngine.Mono.sln

Step 2: Copy the results

xbuild puts the files into the output structure I want (that’s how the Visual Studio solution is architected) but it also generates a lot of cruft.  I use robocopy, which is already part of Server, to copy all of my files except files with a *.pdb or *.mdb extension to a temporary output location:

robocopy PublishSolutionEngineDebugMono Installersoutput /s /xf *.pdb *.mdb

Step 3: Put the results into a tarball

Next I use 7-zip to build the tarball.  This part was surprisingly confusing.  7-zip has some documentation, but it’s far from clear, and I even found a page that had “command line examples” but I’m not sure that it really gave me any more clarity.  In the end I just did lots and lots of iterations on the build, checking the output every time to see what happened.  In the end, this is the command I ended up with:

"C:Program Files (x86)7-Zip7z.exe" a -ttar SolutionEngine.tar "%WORKSPACE%Installersoutput*.*" -mmt -r

This breaks down as follows:

  • The ‘a’ flag means I’m creating (as opposed to extracting from) an archive
  • -ttar is a switch meaning “type tar” – I’m using a tar container (as opposed to say a zip, iso or whatever)
  • SolutionEngine.tar is the output file.  It ends up in the working folder where I’m running command.
  • The next section is the “source”.  %WORKSPACE% is an environment variable Jenkins sets and the rest of the path you can see is the destination from Step 2 above.  *.* means I want everything found at the source location in the tar.
  • -mmt means “use multi-threading when compressing” which I think ends up using multiple cores if available.  I didn’t compare times with and without, so I don’t know how much it helps and it’s probably negligible – my end tar is only about 9MB.  Having the switch doesn’t cause problems, so I’m leaving it in.
  • The -r flag means “recurse the source.”  My source folder has several layers of subdirectories and I want them all in the tarball, maintaining the folder structure.  This flag achieves that.

Step 4: Compress the tarball

For those unaware, a “tar.gz” file is a double operation – package, then compress the package.  Step 3 built the package, but to be friendly to both my upload process and our customers who have to download it, I also compress the file.  To compress the tarball with gzip compression I again use 7-zip.

"C:Program Files (x86)7-Zip7z.exe" a -tgzip SolutionEngine.tar.gz "%WORKSPACE%SolutionEngine.tar"

This one is simpler than Step 3 and breaks down as follows:

  • The ‘a’ flag, again, means I’m creating an archive
  • -tgzip is a switch meaning “compress using type gzip”
  • SolutionEngine.tar.gz is the output file.  Again, it ends up in the working folder where I’m running command.
  • The next section is the “source” file, which was the tar file output by Step 3 above.