VSTS with TeamCity – Building a pull request server

Tuesday, 2 January 2018

Part 2 in a series on integrating VSTS with TeamCity

This is a web service that handles web hook messages triggered by events in VSTS and TeamCity.

I started by using the sample Create a pull request status server with Node.js. As the title suggests, this is a Node application and it uses the Express library to run as a web server. I’d never written a Node app before, but it turns out it is surprisingly easy. If you’re starting from scratch, I’d recommend following the sample tutorial first, so you get up to speed with creating a really simple Node app that VSTS can talk to. Once you’ve got that working, then you should be well placed to use the code samples below.

There’s really two main processes we need to support:

  1. VSTS – Pull Request Created or Updated → Trigger TeamCity build of branch</li>
  2. TeamCity build complete → Tell VSTS if build was success or failure</li>

It’s also worth highlighting a couple of design goals for the pull request server:

NPM packages

 "dependencies": {
    "basic-auth": "^2.0.0",
    "body-parser": "^1.18.2",
    "express": "^4.16.2",
    "teamcity-rest-api": "0.0.8",
    "vso-node-api": "^6.2.8-preview"
  },
  "devDependencies": {
    "@types/basic-auth": "^1.1.2",
    "@types/body-parser": "^1.16.8",
    "@types/express": "^4.0.39",
    "@types/node": "^8.5.1",
    "nodemon": "^1.13.3",
    "ts-node": "^4.0.2",
    "tslint": "^5.8.0",
    "typescript": "^2.6.2"
  }

Note that the following code snippets have been ‘tidied’ for the blog post - the original has a fair bit of file and console logging which is invaluable when diagnosing what’s going on as you’re modifying the logic. I also switched to using TypeScript as I really appreciated the type checking it provides.

Get things ready

import * as auth from 'basic-auth';
import * as bodyParser from 'body-parser';
import * as express from 'express';
import * as fs from 'fs';
import * as vsts from 'vso-node-api';
import { GitPullRequestStatus, GitStatusState } from 'vso-node-api/interfaces/GitInterfaces';

const teamcity = require('teamcity-rest-api');

const app = express();
app.use(bodyParser.json({ limit: '5mb'}));

const collectionURL = process.env.COLLECTIONURL;
const token = process.env.VSTSTOKEN;

const authHandler = vsts.getPersonalAccessTokenHandler(token!);
const connection = new vsts.WebApi(collectionURL!, authHandler);

const vstsGit = connection.getGitApi();

Handling a pull request event

app.post('/prserver', (req, res) => {
    const user = auth(req);

    const buildTypeId = req.query.buildTypeId;

    if (!buildTypeId || !user) {
        res.sendStatus(400);
        return;
    }

    const mergeStatus = req.body.resource.mergeStatus;

    if (mergeStatus !== 'succeeded') {
        return;
    }

    // Get the details about the PR from the service hook payload
    const repoId = req.body.resource.repository.id;
    const pullRequestId = req.body.resource.pullRequestId;
    const commitId = req.body.resource.lastMergeSourceCommit.commitId;

    // This needs to match the branch pattern in TeamCity
    const branchName = pullRequestId;

    // We need to find out the pull request IterationId
    vstsGit.getPullRequestIterations(repoId, pullRequestId, undefined, false)
    .then((iterations: any[]) => {
        return iterations.filter(item => item.sourceRefCommit.commitId === commitId)[0].id;
    })
    .then((iterationId: string) => {

        // trigger TeamCity build
        const client = teamcity.create({
            url: 'http://localhost:8111',
            username: user.name,
            password: user.pass
        });

        // This is the data we are posting to TeamCity
        // tslint:disable-next-line:max-line-length
        const buildNodeObject = `<build branchName='${branchName}'><buildType id='${buildTypeId}' /><properties><property name='vsts.pullRequestId' value="${pullRequestId}" inherited="false"/><property name='vsts.repositoryId' value="${repoId}" inherited="false"/><property name='vsts.iterationId' value="${iterationId}" inherited="false"/></properties></build>`;

        client.builds.startBuild(buildNodeObject)
            .then((buildStatus: any) => {
                // Build the status object that we want to post.
                // Assume that the PR is ready for review...
                const prStatus = {
                    'iterationId': iterationId,
                    'state': 'pending',
                    'description': `Queued build: ${buildStatus.buildTypeId}`,
                    'targetUrl': buildStatus.webUrl,
                    'context': {
                        'name': 'teamcity',
                        'genre': 'continuous-integration'
                    },
                    _links: null
                };

                // Post the status to the PR
                vstsGit.createPullRequestStatus(prStatus as any, repoId, pullRequestId);
            }, (reason: any) => {
                // error logging
            });
    }).catch((reason: any) => {
        // error logging
    });

    res.send('Received the POST');
});

Handling build completion from TeamCity

app.post('/prserver/buildComplete', (req, res) => {
    const pullRequestIdNode = req.body.build.teamcityProperties.find((item: any) => item.name === 'vsts.pullRequestId');

    if (!pullRequestIdNode) {
        // ignore if we don't have expected property
        return;
    }

    const pullRequestId = pullRequestIdNode.value;

    const repositoryId = req.body.build.teamcityProperties.filter((item: any) => item.name === 'vsts.repositoryId')[0].value;
    const iterationId = req.body.build.teamcityProperties.filter((item: any) => item.name === 'vsts.iterationId')[0].value as number;

    const prStatus = {
        iterationId: iterationId,
        state: GitStatusState.Succeeded,
        'description': 'Build succeeded',
        'targetUrl': req.body.build.buildStatusUrl,
        'context': {
            'name': 'teamcity',
            'genre': 'continuous-integration'
        },
        _links: null
    };

    if (req.body.build.buildResult !== 'success') {
        prStatus.state = GitStatusState.Failed; // 'failed';
        prStatus.description = `Build result: ${req.body.build.buildResult}`;
    }

    // Post the status to the PR
    vstsGit.createPullRequestStatus(prStatus as any, repositoryId, pullRequestId);

});

During development I ran the Node app from the command-line and made extensive use of console and file logging.

Once I was satisfied it was working as expected, I switched to using iisnode to run it under IIS.

Where this app runs depends on how accessible your TeamCity server is. If TeamCity is Internet-facing then the server could run anywhere (maybe even as something like an Azure Function). If TeamCity is running behind your firewall, then you might choose to host the pull server internally too. Note that VSTS does need to be able to contact the pull server – either directly (so the pull server itself is Internet-facing), or using a reverse proxy service like ngrok.

In the next post, I’ll describe how to configure TeamCity and VSTS to call the pull request server.

VSTS with TeamCity – Introduction

Monday, 1 January 2018

Microsoft’s Visual Studio Team Services (VSTS) has become quite a compelling offering for managing software development. Features include source control, build, release, and issue tracking. With the product team working on 3 week sprints, new features are being added frequently.

One feature in particular that VSTS inherited from its on-premise predecessor/sibling Team Foundation Server is gated check-ins. This is a terrific way of keeping the build “green” – you avoid the risk of a developer committing changes that break the build (and that breaking change then ‘infecting’ the rest of the team). Instead their changes are isolated and not actually finally committed until after the build (potentially including unit tests) has passed.

If you use Git version control with VSTS, the equivalent to gated check-ins is branch policies. You can enable one or more of the following branch policies:
Build validation uses the built-in VSTS Build services, so what if you’re already using another build server – say JetBrain’s TeamCity for example? Well that’s where that last item “Approval from external services” can come into play.

TeamCity does have some support for building VSTS Git repositories. What it can’t currently do is integrate fully as a VSTS branch policy.

This is the first in a short series of blog posts in which I’m going to describe how I leveraged the VSTS Pull Request Status API and TeamCity’s API via a custom ‘pull request server’ to act as an intermediary between VSTS and TeamCity. With this in place, pull requests must have a passing build before the changes from the branch are allowed to be merged to master.

Unit Test Fixtures

Friday, 1 December 2017

Almost ten years ago, I bought and read Gerard Meszaros’ xUnit Test Patterns, Refactoring Test Code. It is very comprehensive and seeks to document all the common test design patterns. Meszaros uses italics to indicate named patterns, and I'll follow that convention here.

The unit test framework I’ve used the most with .NET is NUnit. One quirk that I’m just beginning to understand better is the difference between Testcase Class and Fixture.

My confusion lies with how NUnit requires that classes which contain test methods, be decorated with the TestFixture attribute. I always assumed that this meant the class was "the fixture", but no!

Meszaros defines them as follows:
(I) use ‘test fixture’ or just ‘fixture’ to mean ‘the pre-conditions of the test’ and Testcase class to mean ‘the class that contains the Test Methods and any code needed to set up the test fixture’. (xUnit Test Patterns p.59)

The key thing is that while NUnit defaults to the Testcase Class per Fixture, you can do things differently.

Take a look at one of the unit test projects from the Cake project.
Cake.NuGet.Tests project structure


Notice how they’ve separated the Fixture classes and the Testcase classes into separate folders.

Here’s one of the test methods from the NuGetPackageInstallerTests class. Note how concise and uncomplicated it is. You can look at this method and very quickly understand what it’s testing.

[Fact]
public void Should_Be_Able_To_Install_If_Scheme_Is_Correct()
{
    // Given
    var fixture = new NuGetPackageInstallerFixture();
    fixture.Package = new PackageReference("nuget:?package=Cake.Core");

    // When
    var result = fixture.CanInstall();

    // Then
    Assert.True(result);
}

Now let’s look at the fixture class – NuGetPackageInstallerFixture

internal sealed class NuGetPackageInstallerFixture
{
    public ICakeEnvironment Environment { get; set; }
    public FakeFileSystem FileSystem { get; set; }
    public IProcessRunner ProcessRunner { get; set; }
    public INuGetToolResolver ToolResolver { get; set; }
    public INuGetContentResolver ContentResolver { get; set; }
    public ICakeLog Log { get; set; }

    public PackageReference Package { get; set; }
    public PackageType PackageType { get; set; }
    public DirectoryPath InstallPath { get; set; }

    public ICakeConfiguration Config { get; set; }

    public NuGetPackageInstallerFixture()
    {
        Environment = FakeEnvironment.CreateUnixEnvironment();
        FileSystem = new FakeFileSystem(Environment);
        ContentResolver = Substitute.For<INuGetContentResolver>();
        Log = Substitute.For<ICakeLog>();
        Config = Substitute.For<ICakeConfiguration>();

        ToolResolver = Substitute.For<INuGetToolResolver>();
        ToolResolver.ResolvePath().Returns(new FilePath("/Working/tools/nuget.exe"));

        Package = new PackageReference("nuget:https://myget.org/temp/?package=Cake.Foo&prerelease&version=1.2.3");
        PackageType = PackageType.Addin;
        InstallPath = new DirectoryPath("./nuget");

        ProcessRunner = Substitute.For<IProcessRunner>();
        ProcessRunner.When(p => p.Start(Arg.Any<FilePath>(), Arg.Any<ProcessSettings>()))
            .Do(info => FileSystem.CreateDirectory(InstallPath.Combine(Package.Package.ToLowerInvariant()).Combine(Package.Package)));
    }

    public void InstallPackageAtSpecifiedPath(DirectoryPath path)
    {
        ProcessRunner.When(p => p.Start(Arg.Any<FilePath>(), Arg.Any<ProcessSettings>()))
            .Do(info => FileSystem.CreateDirectory(path));
    }

    public NuGetPackageInstaller CreateInstaller()
    {
        return new NuGetPackageInstaller(FileSystem, Environment, ProcessRunner, ToolResolver, ContentResolver, Log, Config);
    }

    public IReadOnlyCollection<IFile> Install()
    {
        var installer = CreateInstaller();
        return installer.Install(Package, PackageType, InstallPath);
    }

    public bool CanInstall()
    {
        var installer = CreateInstaller();
        return installer.CanInstall(Package, PackageType);
    }
}

As described in xUnit Test Patterns, this Test Fixture has everything we need in place to exercise the SUT (system under test). It instantiates both the SUT and all its dependencies, and also provides helper methods that are called by the Test Methods. Without this, all of the setup and configuration code would have lived in the Testcase class (NuGetPackageInstallerTests). This looks like an example of the Transient Fresh Fixture pattern.

If you want to get another perspective on using test fixtures, I’d also recommend taking a look at Mark Seemann’s Advanced Unit Testing course on Pluralsight

Enterprise Service Bus libraries for .NET – Rebus

Monday, 27 November 2017

One in a series of posts giving a quick overview of ESB libraries for .NET

Rebus describes itself as "a lean service bus implementation for .NET".

https://github.com/rebus-org/Rebus

Messages

POCO classes

Publishing

Use an instance of IBus to publish.

activator is an implementation of IHandlerActivator - either BuiltinHandlerActivator or an adapter class for your particular IoC container.

activator.Bus.Publish(new DateTimeMessage(DateTime.Now)).Wait();

Subscribing

Handlers implement IHandleMessages<T>

public class PrintDateTime : IHandleMessages<DateTime>
{
    public async Task Handle(DateTime currentDateTime)
    {
        Console.WriteLine("The time is {0}", currentDateTime);
    }
}

And register a subscription

activator.Bus.Subscribe<StringMessage>().Wait();

Enterprise Service Bus libraries for .NET – Shuttle ESB

Sunday, 26 November 2017


One in a series of posts giving a quick overview of ESB libraries for .NET

Shuttle ESB describes itself as a "free .NET open-source enterprise service bus". It supports a good range of transports and integrates with a number of IoC containers.

https://github.com/Shuttle/shuttle-esb

Messages

A POCO class

Publishing

Publish using IServiceBus. You'll also make use of static methods from the ServiceBus class.

using (var bus = ServiceBus.Create(resolver).Start())
{
    bus.Publish(new MemberRegisteredEvent
    {
        UserName = "Mr Resistor"
    });
}

Subscribing

A handler implements IMessageHandler

public class MemberRegisteredHandler : IMessageHandler<MemberRegisteredEvent>
{
    public void ProcessMessage(IHandlerContext<MemberRegisteredEvent> context)
    {
        Console.WriteLine();
        Console.WriteLine("[EVENT RECEIVED] : user name = ‘{0}’", context.Message.UserName);
        Console.WriteLine();
    }
}

You also need to tell the subscription manager that you want to subscribe to a particular event. resolver is a wrapper class for your specific container.

resolver.Resolve<ISubscriptionManager>().Subscribe<MemberRegisteredEvent>();