David Gardiner - TeamCity

TeamCity conditional build configurations - master vs other branches

2019-06-21T13:00:00.000+09:30

TeamCity doesn't currently support conditional build steps (Vote on this issue). So how can you have different steps for a master builds vs. other branches?

This is not an uncommon scenario - you might want to perform similar build steps for all builds, but for a master build there may be some extra steps (eg. publishing NuGet/NPM packages to a repository, triggering an external activity) that you don't want to run for a branch build (eg. a build for a pull request).

It is possible to work around this limitation by creating multiple build configurations - one for master and another for non-master branches. But how can you do that efficiently? One approach is to enable Versioned Settings using Kotlin.

Kotlin is a language created by JetBrains that targets the JVM (and is now the recommended language for Android development). But it's also one of the language choices when you enable Versioned Settings in TeamCity. (The other is XML, which will look familiar if you've ever played with TeamCity's meta-runners).

I'm not a Kotlin expert, but I managed to figure out that you can use it to generate the different variations, such that when TeamCity parses the Kotlin, it ends up creating multiple build configurations.

One thing to be aware of is that TeamCity parses the Kotlin up-front to create the build configurations. It doesn't evaluate it as the build runs (so it isn't possible to have an expression that tests an environment variable or parameter)

The following example shows generating two build configurations - "Build_CI_Master" and "Build_CI_Branches". The difference is that the Master build configuration enables package indexing. The build configurations use different branch filters to control whether they apply to master or non-master branches.

VSTS and TeamCity – Wrapping up

2018-01-07T21:00:00.000+10:30

Part 4 in a series on integrating VSTS with TeamCity

Wouldn't it be great if TeamCity and VSTS had full builtin support for each other? Well yes, yes it would! Maybe that will happen soon.

If I knew Java well, I could probably have a go at writing a TeamCity addin that encapsulates most of the what the pull request server does - but the idea of spending a few weeks getting up to speed with Java/TeamCity development doesn’t excite me that much.

TeamCity 2017.2 adds VSTS Git support to the Commit Status Publisher build feature. I haven’t been able to try this out yet (due to some other bugs in 2017.2 preventing me from upgrading), but it is possible this could remove or reduce the requirement for the build completion handler.

VCS post-commit hook

Now you've seen how to to use the APIs for TeamCity and VSTS, you might also want to implement another optimisation - adding a VCS post-commit hook. You add an additional service hook in VSTS that notifies TeamCity that there's a code change so that TeamCity knows it should grab the latest commit(s).

In VSTS Project Settings, go to the Service Hooks tab
Click '+' to add a new service hook
Select Web Hooks
In Trigger on this type of event, select Code pushed
Optionally, review the Filters and just check the Repository (and branch) that should trigger the event.
In the URL, enter something like https://www.example.com/app/rest/vcs-root-instances/commitHookNotification?locator=vcsRoot:(type:jetbrains.git,count:99999),property:(name:url,value:%2Fbuildname,matchType:contains),count:99999 the locator can vary depending on your individual requirements
Enter the username and password to authenticate with TeamCity
Set Resource details to send, Messages to send and Detailed messages to send to None
Click Test to confirm that everything works.

The nice thing about this is that rather than TeamCity blindly polling VSTS, VSTS is telling TeamCity when it has something of interest.

VSTS with TeamCity – Configuration

2018-01-03T12:06:00.001+10:30

Part 3 in a series on integrating VSTS with TeamCity

Now that we've created the pull request server, we need to configure VSTS and TeamCity so that they can send event messages to it.

VSTS

If you followed the steps in the sample tutorial, this will be familiar.

Go to the Service Hooks tab for your project
Click on the + icon
Choose Web Hooks and click Next
Select Pull request created.
If appropriate, select a specific repository and click Next
In URL, enter the URL that VSTS will use to connect to the pull request server, including a query string that defines which TeamCity build should be queued. Eg. If the pull request server is hosted at https://www.example.com/pullrequestserver and the TeamCity build type id is My_CI_Build then you’d use https://www.example.com/pullrequestserver?buildTypeId=My\_CI\_Build
In Username and Password, enter the credentials that will be used to authenticate with TeamCity
Leave Resource details to send as All.
Set Messages to send and Detailed messages to send to None
Click on Test to try it out.
Click on Finish to save this service hook.
Repeat these steps to create another service hook for Pull request updated, also setting the Change filter to Source branch updated.

With the service hooks in place, you can now go to the Branches page, and click on the … (more actions) icon and choose Branch policies.

The Add status policy button should be enabled, and clicking on that you should be able to find the pull request server listed in the drop down.

TeamCity

To allow TeamCity to call the pull request server, you will need to install the Web Hooks plugin for TeamCity. With that in place, go to the build configuration page in TeamCity, and you’ll see a new WebHooks tab.

Click on add build Webhooks, then Click to create new WebHook for this build and add a new web hook for the project
In the URL, enter the URL that TeamCity will use to connect to the pull request server. Eg. If the pull request server is hosted at https://www.example.com/pullrequestserver, you would use https://www.example.com/pullrequestserver/buildComplete.
Set the payload format to Legacy webhook (JSON)
Clear all the trigger events except On Completion Trigger when build successful and Trigger when build fails
Click Save

In the VCS Root settings for the VSTS Git repository, set Branch Specification to

+:refs/heads/(master)
+:refs/pull/*/merge

We don't need TeamCity to trigger the builds for the pull request branches as the pull request server will be queuing those builds, but we do still want TeamCity to trigger the master builds.

In the build configuration VCS Trigger, set the Branch filter to +:<default>

With all that configuration done, creating a new pull request in VSTS should now trigger a branch build in TeamCity. When the build completes, the status is posted back to VSTS, allowing the pull request to be completed by merging the changes into master.

VSTS with TeamCity – Building a pull request server

2018-01-02T08:00:00.000+10:30

Part 2 in a series on integrating VSTS with TeamCity

This is a web service that handles web hook messages triggered by events in VSTS and TeamCity.

I started by using the sample Create a pull request status server with Node.js. As the title suggests, this is a Node application and it uses the Express library to run as a web server. I’d never written a Node app before, but it turns out it is surprisingly easy. If you’re starting from scratch, I’d recommend following the sample tutorial first, so you get up to speed with creating a really simple Node app that VSTS can talk to. Once you’ve got that working, then you should be well placed to use the code samples below.

There’s really two main processes we need to support:

VSTS – Pull Request Created or Updated → Trigger TeamCity build of branch
TeamCity build complete → Tell VSTS if build was success or failure

It’s also worth highlighting a couple of design goals for the pull request server:

Avoid hard-coding any specific details in the script, so that it could be easily reused in other repositories. As such, specific details (user credentials, TeamCity build configuration names etc) are configured as extra parameters in the web hook / service hook calls.
Be stateless

NPM packages

 "dependencies": {
    "basic-auth": "^2.0.0",
    "body-parser": "^1.18.2",
    "express": "^4.16.2",
    "teamcity-rest-api": "0.0.8",
    "vso-node-api": "^6.2.8-preview"
  },
  "devDependencies": {
    "@types/basic-auth": "^1.1.2",
    "@types/body-parser": "^1.16.8",
    "@types/express": "^4.0.39",
    "@types/node": "^8.5.1",
    "nodemon": "^1.13.3",
    "ts-node": "^4.0.2",
    "tslint": "^5.8.0",
    "typescript": "^2.6.2"
  }

Note that the following code snippets have been ‘tidied’ for the blog post - the original has a fair bit of file and console logging which is invaluable when diagnosing what’s going on as you’re modifying the logic. I also switched to using TypeScript as I really appreciated the type checking it provides.

Get things ready

import * as auth from 'basic-auth';
import * as bodyParser from 'body-parser';
import * as express from 'express';
import * as fs from 'fs';
import * as vsts from 'vso-node-api';
import { GitPullRequestStatus, GitStatusState } from 'vso-node-api/interfaces/GitInterfaces';

const teamcity = require('teamcity-rest-api');

const app = express();
app.use(bodyParser.json({ limit: '5mb'}));

const collectionURL = process.env.COLLECTIONURL;
const token = process.env.VSTSTOKEN;

const authHandler = vsts.getPersonalAccessTokenHandler(token!);
const connection = new vsts.WebApi(collectionURL!, authHandler);

const vstsGit = connection.getGitApi();

Handling a pull request event

app.post('/prserver', (req, res) => {
    const user = auth(req);

    const buildTypeId = req.query.buildTypeId;

    if (!buildTypeId || !user) {
        res.sendStatus(400);
        return;
    }

    const mergeStatus = req.body.resource.mergeStatus;

    if (mergeStatus !== 'succeeded') {
        return;
    }

    // Get the details about the PR from the service hook payload
    const repoId = req.body.resource.repository.id;
    const pullRequestId = req.body.resource.pullRequestId;
    const commitId = req.body.resource.lastMergeSourceCommit.commitId;

    // This needs to match the branch pattern in TeamCity
    const branchName = pullRequestId;

    // We need to find out the pull request IterationId
    vstsGit.getPullRequestIterations(repoId, pullRequestId, undefined, false)
    .then((iterations: any[]) => {
        return iterations.filter(item => item.sourceRefCommit.commitId === commitId)[0].id;
    })
    .then((iterationId: string) => {

        // trigger TeamCity build
        const client = teamcity.create({
            url: 'http://localhost:8111',
            username: user.name,
            password: user.pass
        });

        // This is the data we are posting to TeamCity
        // tslint:disable-next-line:max-line-length
        const buildNodeObject = `<build branchName='${branchName}'><buildType id='${buildTypeId}' /><properties><property name='vsts.pullRequestId' value="${pullRequestId}" inherited="false"/><property name='vsts.repositoryId' value="${repoId}" inherited="false"/><property name='vsts.iterationId' value="${iterationId}" inherited="false"/></properties></build>`;

        client.builds.startBuild(buildNodeObject)
            .then((buildStatus: any) => {
                // Build the status object that we want to post.
                // Assume that the PR is ready for review...
                const prStatus = {
                    'iterationId': iterationId,
                    'state': 'pending',
                    'description': `Queued build: ${buildStatus.buildTypeId}`,
                    'targetUrl': buildStatus.webUrl,
                    'context': {
                        'name': 'teamcity',
                        'genre': 'continuous-integration'
                    },
                    _links: null
                };

                // Post the status to the PR
                vstsGit.createPullRequestStatus(prStatus as any, repoId, pullRequestId);
            }, (reason: any) => {
                // error logging
            });
    }).catch((reason: any) => {
        // error logging
    });

    res.send('Received the POST');
});

Handling build completion from TeamCity

app.post('/prserver/buildComplete', (req, res) => {
    const pullRequestIdNode = req.body.build.teamcityProperties.find((item: any) => item.name === 'vsts.pullRequestId');

    if (!pullRequestIdNode) {
        // ignore if we don't have expected property
        return;
    }

    const pullRequestId = pullRequestIdNode.value;

    const repositoryId = req.body.build.teamcityProperties.filter((item: any) => item.name === 'vsts.repositoryId')[0].value;
    const iterationId = req.body.build.teamcityProperties.filter((item: any) => item.name === 'vsts.iterationId')[0].value as number;

    const prStatus = {
        iterationId: iterationId,
        state: GitStatusState.Succeeded,
        'description': 'Build succeeded',
        'targetUrl': req.body.build.buildStatusUrl,
        'context': {
            'name': 'teamcity',
            'genre': 'continuous-integration'
        },
        _links: null
    };

    if (req.body.build.buildResult !== 'success') {
        prStatus.state = GitStatusState.Failed; // 'failed';
        prStatus.description = `Build result: ${req.body.build.buildResult}`;
    }

    // Post the status to the PR
    vstsGit.createPullRequestStatus(prStatus as any, repositoryId, pullRequestId);

});

During development I ran the Node app from the command-line and made extensive use of console and file logging.

Once I was satisfied it was working as expected, I switched to using iisnode to run it under IIS.

Where this app runs depends on how accessible your TeamCity server is. If TeamCity is Internet-facing then the server could run anywhere (maybe even as something like an Azure Function). If TeamCity is running behind your firewall, then you might choose to host the pull server internally too. Note that VSTS does need to be able to contact the pull server – either directly (so the pull server itself is Internet-facing), or using a reverse proxy service like ngrok.

In the next post, I’ll describe how to configure TeamCity and VSTS to call the pull request server.

VSTS with TeamCity – Introduction

2018-01-01T17:09:00.000+10:30

Microsoft’s Visual Studio Team Services (VSTS) has become quite a compelling offering for managing software development. Features include source control, build, release, and issue tracking. With the product team working on 3 week sprints, new features are being added frequently.

One feature in particular that VSTS inherited from its on-premise predecessor/sibling Team Foundation Server is gated check-ins. This is a terrific way of keeping the build “green” – you avoid the risk of a developer committing changes that break the build (and that breaking change then ‘infecting’ the rest of the team). Instead their changes are isolated and not actually finally committed until after the build (potentially including unit tests) has passed.

If you use Git version control with VSTS, the equivalent to gated check-ins is branch policies. You can enable one or more of the following branch policies:

Require a minimum number of reviewers
Check for linked work items
Check for comment resolution
Enforce a merge strategy
Build validation
Approval from external services

Build validation uses the built-in VSTS Build services, so what if you’re already using another build server – say JetBrain’s TeamCity for example? Well that’s where that last item “Approval from external services” can come into play.

TeamCity does have some support for building VSTS Git repositories. What it can’t currently do is integrate fully as a VSTS branch policy.

This is the first in a short series of blog posts in which I’m going to describe how I leveraged the VSTS Pull Request Status API and TeamCity’s API via a custom ‘pull request server’ to act as an intermediary between VSTS and TeamCity. With this in place, pull requests must have a passing build before the changes from the branch are allowed to be merged to master.