NuGet

Issue to be aware of when restoring NuGet packages for the code hosted in git repository of Visual Studio Team Services aka VSO

Wed, 25 Nov 2015 09:46:21 GMT

I came across this issue a few times with different developers so I thought I will write about it to save time for others. When you think about it then there's nothing wrong or mysterious about it but you may end up spending lots of time on it without realising what's wrong.

If you create a new repository with a whitespace, like My repository then when you clone it:

git clone my.visualstudio.com/DefaultCollection/_git/My%20repository

It will create a new folder with the name My%20repository

Everything is correct here, you can't have a space in an URL but if you open the solution in visual studio and build it you'll get errors like:

The remote server returned an error: (401) Unauthorized or This project references NuGet package(s) that are missing on this computer. Enable NuGet Package Restore to download them.

The simplest solution is to rename the folder by removing %20 or set a name when you clone it, like

git clone [url] [Name]

As I said, this is not an issue but easy to miss if you don't pay attention. Next time when you can't restore NuGet packages, start with checking where the repository was cloned.

Azure Mobile Service deployment with Team City part 3. Build Azure Mobile Service with PowerShell using psake

Thu, 16 Jul 2015 05:02:40 GMT

TeamCity is a very powerful tool and you can take advantage of all its configuration options to create your CI and CD. Although TeamCity provides you with almost everything you need, like restoring packages, update version, build, running unit tests, etc. I’m going to write all these steps in a PowerShell script. It is easier to use what TeamCity offers from the start but when the complexity of building and deploying increases it will be difficult to maintain.

Having the deployment process as a set of scripts will give you some advantages, like:

Run and test the deployment process on your machine
Evolve the deployment system incrementally
Ability to test every step in isolation
Checking changes in source control and ability to revert to previous versions if needed

The scripts will be written in PowerShell and I will use psake which is a PowerShell tool to provide task-oriented dependencies.

Initially we’ll start with three scripts:

local.ps1 – we’ll use this script to run the entire set of script on local machine; it contains default parameters and it executes run-build.ps1
run-build.ps1 – the script is called by TeamCity providing all the necessary parameters. It invokes the build.ps1 script using psake module
builid.ps1 – contains all the steps necessary to build the project

Tools

First we need to get psake scripts from the git releases page and copy them to the Tools folder. We need the nuget.exe as well. Download it from Command-Line Utility URL . Copy nuget.exe in the Tools folder.

Local script

All the scripts are located in the Scripts folder.

local.ps1 script clears the screen and executes the run-build.ps1 script. It contains a few default parameters. One of the parameter is debugConfiguration. We use that for debugging purposes. For example when we work on the deployment task we might not want to run all the tasks which are time consuming, like restore packages so we’ll ignore it.

Run-build script

The run-build.ps1 script imports psake module and executes build.ps1 script in the context of the psake module. It invokes the script with parameters which don’t have default values and properties which have default values.

Build script

All the steps before running the build and the build itself are done in this build.ps1 script.

We’ll start with changing how psake prints out the tasks it runs. This will make a better separation between tasks and will be easier to read.

FormatTaskName "$([Environment]::NewLine)==================== $(Get-Date -format T) - Executing {0} ===================="

We’ll use the precondition parameter to conditionally run a task. For example, if we don’t want to restore packages we’ll use the $debugConfiguration.restorePackage parameter.

Task RestoreNuGetPackages -Depends Clean -Precondition {return $debugConfiguration -eq $null -or $debugConfiguration.restorePackage } {

The task will run when precondition returns true. I check for null value of the debugConfiguration variable because it will be null when running from TeamCity so we don’t want to skip the step.

Bear in mind that if the task execution depends on another task then that task will not run if precondition of the task in execution returns false.

We’ll start with adding the default task.

Task Default -Depends DeployPackage

Clean

First of all we need to clean everything. For that we’ll build the solution with Clean target and also delete the packages folder.

Task Clean -Precondition { return $debugConfiguration -eq $null -or $debugConfiguration.clean } {
 Exec {
        msbuild $solutionToBuild /t:Clean /verbosity:$verbosity /nologo /p:Configuration=$config /p:VisualStudioVersion=12.0
    }

  if (Test-Path "..\Source\packages") {
     Remove-Item "..\Source\packages" -Recurse -Force -ErrorAction Stop
  }
  else {
    "Didn't find the 'packages' folder."
  }
}

Restore packages

Every single build needs to restore all the packages, in this way we make sure that packages are not missing on the build machine.

Task RestoreNuGetPackages -Depends Clean -Precondition { return $debugConfiguration -eq $null -or $debugConfiguration.restorePackage } {
    if (!(Test-Path $nugetExe)){
        throw "nuget.exe could not be found on this machine. Please check: $nugetExe"
    }
    Exec {
      & $nugetExe restore $solutionToBuild
    }
}

Our mobile service project references a NuGet package hosted on the TeamCity machine. But the NuGet feed requires authentication. In order to restore that package on the TeamCity machine during the build process we need to allow the restore process to authenticate to that NuGet feed. There is a command line we need to run on the TeamCity VM.

Login on the TeamCity server and make sure that you use the same account you used when you installed the TeamCity agent. It’s important that the agent runs under the same account. We configured the agent in step 1 where I mentioned that. NuGet should add the configuration file (NuGet.config) in the same user’s folder.

For example, if you run the agent under the ‘teamcity’ user name then when you add the NuGet source as below (run the below command) it will create or update the file in this location:

C:\Users\teamcity\AppData\Roaming\NuGet\NuGet.Config

This is the command line you need to run:

NuGet.exe Sources Add –Name [GIVE_A_NAME] -Source [TEAMCITY_FEED] –UserName [USER_NAME] –Password [PASSWORD]

Sources – TeamCity NuGet feed. Check the configure NuGet server post

UserName – user name you provided when you setup TeamCity. You can create a new user name just for this purpose but in this case you’ll need to run the agent under this user

Password – user name’s password

Update assembly version

There are different methods to update the assembly version. You can keep the version in one AssemblyInfo.cs file and add that file as a link to all the projects but I don’t like this idea because you can forget to do that and some of your libraries will not have the version updated. Instead of that I iterate through all the projects and update the version for every single AssemblyInfo.cs file.

Task UpdateVersion -Precondition { return $debugConfiguration -eq $null -or $debugConfiguration.updateVersion } {
    (Get-ChildItem -Path $sourceDirectory -Filter AssemblyInfo.cs -Recurse) |
    ForEach-Object {
      (Get-Content $_.FullName) |
        ForEach-Object {
          $_ -replace 'AssemblyVersion.+$',"AssemblyVersion(`"$version`")]" `
          -replace 'AssemblyFileVersion.+$',"AssemblyFileVersion(`"$version`")]"
        } |
        Out-File $_.FullName
    }
}

Build

The build process should create a package which will be deployed to Azure. In order to do this we’ll use DeployOnBuild=true and PublishProfile=PublishProfile.xml parameters.

The PublishProfile.pubxml file doesn’t exist. We need to generate it. In order to do this we have the pubxml.template file in the Scripts folder. It contains the following xml:

<?xml version="1.0" encoding="utf-8"?>
<Project ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
  <PropertyGroup>
    <WebPublishMethod>FileSystem</WebPublishMethod>
    <LastUsedBuildConfiguration>Release</LastUsedBuildConfiguration>
    <LastUsedPlatform>Any CPU</LastUsedPlatform>
    <SiteUrlToLaunchAfterPublish />
    <LaunchSiteAfterPublish>False</LaunchSiteAfterPublish>
    <ExcludeApp_Data>True</ExcludeApp_Data>
    <publishUrl>{0}</publishUrl>
    <DeleteExistingFiles>True</DeleteExistingFiles>
  </PropertyGroup>
</Project>

You can change it if you need to. We’re interested in the publishUrl value – we need to populate it during the build process with the package location. If you’re going to have a fixed package location then you can hard-code it but in our case the location will be driven by the TeamCity. We’ll generate the package in the “[Root]\Artifacts” folder. Root is the folder where our Scripts and Source folders are located.

The PublishProfile.pubxml file is generated in the method “GetPublishProfile” and will be saved in “[Root]\Artifacts” folder. Later it is overwritten by the build with the package. To some extend this simulates the same process you have when you deploy from Visual Studio.

function GetPublishProfile() {
    # Get the publish xml template and generate the .pubxml file
    $scriptPath = Split-Path -parent $PSCommandPath

    if (Test-Path $artifactDirectory) {
        Remove-Item $artifactDirectory -Recurse -Force -ErrorAction Stop
    }

    [String]$template = Get-Content $scriptPath\pubxml.template

    mkdir $artifactDirectory | Out-Null
    $xml = $template -f $artifactDirectory
    $outputPublishProfile = Join-Path $artifactDirectory "PublishProfile.pubxml"
    $xml | Out-File -Encoding utf8 -FilePath $outputPublishProfile

    return $outputPublishProfile
}

Task Build -Depends RestoreNuGetPackages, UpdateVersion {
    $publishProfile = GetPublishProfile 

    Exec {
        msbuild $projectToBuild /p:DeployOnBuild=true /p:PublishProfile=$publishProfile /verbosity:$verbosity /nologo /p:Configuration=$config /p:VisualStudioVersion=12.0
    }
}

Unit Tests

All the Unit Tests were implemented using xunit. First we need to build the solution which will create the Test*.dll files. After that we’ll run every Test*.dll file by xunit.exe.

xunit.exe file is added as a NuGet package. In order to do that we need to add this to our Test project, otherwise it will create a .nuget folder with packages.config file and we’ll put it under the solution folder. You can add it as I NuGet package for the solution but I don't really like it for small projects.

  <package id="xunit.runner.console" version="2.0.0" />

The full source code of the Unit Test

Task Test -Depends Build {
  if (!(Test-Path $xunitConsole))
  {
    throw "xunit.console.exe does not exist. Please check $xunitConsole file."
  }

   "************* Build solution [$solutionToBuild] for Unit Testing *************"
  # Need to run the solution to build all the tests. The Build Task run only the csproj file
  exec {
    msbuild $solutionToBuild /verbosity:m /p:Configuration=$config /p:VisualStudioVersion=12.0 /nologo
  }

  $assembliesToTest = (Get-ChildItem "$sourceDirectory" -Recurse -Include "*Test*.dll" -Name | Select-String "bin")

  $atLeastOneTestRun = $false

  "************* Running Unit Tests *************"
  foreach ($testFile in $assembliesToTest){
    "*************** Testing $testFile ***************"
    $fullNameTestFile = Join-Path $sourceDirectory $testFile
    
    & $xunitConsole $fullNameTestFile
    $atLeastOneTestRun = $true
  }

  if ($atLeastOneTestRun -eq $false){
    Write-Output "Unit Tests didn't run!"
    exit(1)
  }
}

Now we can build and test our project. In the next post I will explain how to deploy an Azure Mobile Service.

< Previous post

Azure Mobile Service deployment with Team City part 2. Configure TeamCity NuGet Server

Mon, 13 Jul 2015 05:00:01 GMT

In the previous post I explained how to install TeamCity on Azure and configure it.

In this post I will create a TeamCity configuration which will publish to a NuGet Server hosted in TeamCity.

You can find the source code used by these series of post at https://github.com/ralbu/TeamCityAzure

Common project

As this is a fresh installation of TeamCity there are no projects. Let’s create one. I will name it User Management. It will contain two configurations:

Common – a NuGet package used by the main project (User Management)
User Management – an Azure Mobile Service project

Click the Create project button

Enter the Name of the project and optionally the description and click Create.

After the project was created click on the Create build configuration button

The first project configuration is our Common library hosted on a NuGet server. We’ll use the TeamCity NuGet plugin to host our NuGet Common package.

Enter the required data and click Create.

In the next step we configure the source control. For this set of tutorials I host the project to GitHub in a public repository. As this is a public repository I use Anonymous as authentication method. There are other options you can choose from, like Password or Private Key. Click Test Connection and if everything is ok click Create.

Both projects – NuGet package and Azure Mobile Service used in this set of blog posts are under the same GitHub repository so I will reuse this VCS root in both projects.

Prerequisites for running builds

Next we’ll create build configurations but before that we need to install the necessary software and configure NuGet server on TeamCity.

We are not going to install Visual Studio on the build machine, although that will be easier. We’ll install only the minimum which is needed to build our projects. Remote into the VM and from https://www.visualstudio.com/en-us/downloads download “Microsoft Build Tools” and install it.
You might also need to install “.NET Framework 4.5.1 Developer Pack”. If you want to have a clean machine I would skip this step and install it later if the build fails. I had to install it because of the xUnit framework. If you are not sure you need “.NET Framework 4.5.1 Developer Pack” then first start with “.NET Framework 4.5” installation.
In order to deploy our mobile services we’ll be using Web Deploy. Install it from this location: http://www.microsoft.com/en-us/download/details.aspx?id=39277

You’ll need to restart the TeamCity agent. Open the Services application and search for Team City Build Agent and restart it.

We need to manually copy the next two folders

Copy the folder C:\Program Files (x86)\MSBuild\Microsoft\VisualStudio\v12.0\WebApplications to the same location of TeamCity server. The reason for doing that is because we don’t have Visual Studio installed on the TeamCity machine. The build process uses targets from Microsoft.WebApplication.targets file to build the Web Application.
Copy the folder C:\Program Files (x86)\MSBuild\Microsoft\VisualStudio\v12.0\Web to TeamCity machine. Make sure you copy this folder. These targets are used in generating packages. If you don’t have this targets it will build the solution but it will not create a package which is used in deployment. It is not easy to figure out if you missed it because the solution will build successfully but the package will not be created and it won’t give you any error or warning.

Configure NuGet Server

In the TeamCity web page, click on the Administration and then in the left hand side under Integration click NuGet. On the NuGet Server tab click Enable button. We are going to use authenticated feed and not the public one. Copy the link to the authenticated feed URL, we’ll use it later. Click on the NuGet.exe tab and then on Fetch NuGet button. Select the NuGet version and select Set as Default. At the time of this writing the latest NuGet is 2.8.5.

Creating build configurations

In this step we’ll create our first build configuration. As the first step we’ll build our solution.

Click on the Build Steps in the Build Configuration Settings and click Add build step

From the New Build Step page select Visual Studio (sln) Runner type and provide the rest of the settings as below and click Save.

Click the Run button if you want to make sure that everything is ok at this step.

In the next step we’ll add the build configuration to create a NuGet package and publish it on TeamCity’s NuGet server.

First of all make sure you have the NuGet spec file. If you don’t then the fastest way of generating it is to open the command line where your .csproj file is located and run:

nuget spec

Open it and adjust it accordingly.

I’m not going to go in details about this step, just have a look at the settings below, and if you want to find out more about it then go to my other blog – Hosting NuGet in TeamCity and consume it in Visual Studio where I explained all the settings.

Click Save and this is how you would configure a NuGet Server. In the next blog post I will describe how to create the main project configuration as a PowerShell script and configure it in TeamCity

Create NuGet packages using AppVeyor

Fri, 22 May 2015 07:45:39 GMT

This blog explains how to create a NuGet package and use AppVeyor as a continuous integration and deployment tool.

I found out about AppVeyor some time ago from Scott's blog - "AppVeyor - A good continuous integration system is a joy to behold". Usually I use TeamCity running on a VM so I didn't have a chance to try it out until recently when I built a NuGet package to generate random users. The NuGet package is based on the https://randomuser.me API – a free API to generate random user data. You would use it in prototyping, testing or filling your database with proper user names, pictures, etc.

AppVeyor is a cloud based continuous integration and delivery service for .NET. It is free for open source projects. It has everything you need and it is really easy to set up. It has a nice Web UI where you configure your build and deployment settings but in this post I will explain the minimum configuration you need to publish a NuGet package using the appveyor.yml file. The reason I prefer to keep my configurations in a file rather than using the UI is that I can version it and when I break something I can fix it fast just by checking the differences in the file.

Every AppVeyor configuration can contain an appveyor.yml file in the root folder. If you have the file and also configure the UI then the YAML file will override the UI settings unless you explicitly disable it in Ignore appveyor.yml settings in the UI.

AppVeyor also has the option to host NuGet packages. Before pushing the NuGet package to https://www.nuget.org I use that to test my packages.

Here's the full YAML file. As you can see, it is self-descriptive and I will explain only the settings which are not clear from the first glance or might be confusing.

version: 0.9.{build}

assembly_info:
  patch: true
  file: AssemblyInfo.cs
  assembly_version: '{version}'
  assembly_file_version: '{version}'
  assembly_informational_version: '{version}'

platform: Any CPU

configuration: Release

build:
  project: RandomUser.sln
  verbosity: detailed
  publish_nuget: true

before_build:
  - nuget restore

nuget:
  account_feed: true
  project_feed: true

deploy:
  provider: NuGet
  api_key:
    secure: oersntns349lpwtsaAThuwfs//74hanwoIEN
  skip_symbols: false
  artifact: /.*\.nupkg/

Will let AppVeyor to take care of the release version number. For this purpose {build} is used. Assembly files will be patched with the version value we define in “version”.

In order to publish to NuGet the following should be added to the appveyor.yml file

build:
  publish_nuget: true

deploy:
  provider: NuGet
  api_key:
    secure: oersntns349lpwtsaAThuwfs//74hanwoIEN
  skip_symbols: false
  artifact: /.*\.nupkg/

You need to get your NuGet API key and encrypt it. AppVeyor offers a tool for encryption at this URL https://ci.appveyor.com/tools/encrypt

In artifact you provide the file name or artifact name to push.

As I said before I prefer to create NuGet packages using AppVeyor feed before pushing them for the first time to nuget.org. For this purpose you need to add the following:

nuget:
  account_feed: true
  project_feed: true

Make sure that your project contains a nuspec file in the same folder where your .csproj file is. You can find more information about nuspec here: http://docs.nuget.org/Create/Nuspec-Reference

All the tests are run automatically by the way.

Running xUnit.net tests using ReSharper

Fri, 20 Mar 2015 22:23:00 GMT

If you have created new projects in Visual Studio recently and you use xUnit.net with ReSharper to run your tests you might find that your tests are not running. The reason for that is that xUnit.net has been updated to version 2. Also xUnit.net runner for Visual Studio is no longer needed.

Follow this steps to run the new xUnit.net 2.0

1. Install xUnit.net with the NuGet command

install-package xunit

If you check the packages.config file you’ll see that there are a few more packages added apart from xunit. One of this is the assertion library. The core framework is separated from the assertion library.

2. Install xUnit Visual Studio runner

install-package xunit.runner.visualstudio

If you don’t use ReSharper to run unit tests this should be enough otherwise follow the next step.

3. Click the Resharper>Extension Manager… Find the xUnit.net Test Support for ReSharper 9 (it should be the same for 8) and click the Update button.

Hosting NuGet in TeamCity and consume it in Visual Studio

Sat, 21 Sep 2013 14:35:00 GMT

Introduction

This post describes the setup of NuGet package inside TeamCity and configuration of your Visual Studio to consume the hosted packages. TeamCity version 7.1 is used.

When you work on a big or medium projects you’ll end up having at least one common library. If you keep everything in one solution then you don’t have problems with sharing the common library – you reference the common project to all other projects which need it.

How about the case when you have multiple separate solutions/projects and want to use the common library. In this cases you have two options: a) add the common library into all the projects as Add Existing Project and b) add a reference to the common dll file. With both approaches you have tackle a set of issues, like how to update the common library and not to break the projects using it; what if you need to use it for a project in a location different to the library, let’s say common library is located on c:/project but the project using it is located on D:/WebSites. These are just a few issues but enough to give you a headache.

All these issues will go away by implementing your common libraries as NuGet packages. You can host these packages on the NuGet web site but you might not want to do it. Here I explain how to host them in TeamCity.

Pre NuGet Build Configuration setup

Will start with standard build configuration steps. In TeamCity create a new Build Configuration. Set the Build number format as it will help to keep track on the versioning and make the update process easier. Set Version Control settings.

Add a Build Step which will build your package solution. If needed add /p:Configuration=Release to Command line parameters to build the release.

Add a unit test build step if it is the case. For example select the MSTest as Runner type if you do MS Unit testing. Add the following **\bin\**\*.Tests.dll in the List assembly files.

NuGet package configuration

Next is the part which is most interested for us. But before that make sure you have the NuGet set in Administration > NuGet Settings.

Add a new Build Step and in the Runner type select NuGetPack. If you set the NuGet settings you should have NuGet.exe dropdown with the NuGet version.

In the Specification files select the project you want to create a package from.

Update Version to %build.number% if you want to keep the same version you set in General Settings of your build configuration.

Set the Output Directory. For instance, Build\packages

Make sure you check the Publish created packages to build artifacts.

Also you might want to check Include sources and symbols but not mandatory.

NOTE:

In the Build Steps screen add AssemblyInfo patched in Additional Build Features so it will update the dll version in order to keep your files versioned.

That's all you need to set for TeamCity to create NuGet packages.

Visual Studio configuration

Next we need to configure VS to get the package from TeamCity.

In VS open Tools > Library Package Manager > Package Manager Settings. Select the Packages Sources and click the "+" to add a new sources. Give it a name and set the Source. If you haven't figured it out go into TeamCity > Administrator > NuGetSettings > NuGet Server and get the URL, either Autenticated Feed URL or Public Feed URL, depending how you configured it.

In order to add the common library to your project follow the same steps you do when you add a standard NuGet package to your project except you'll see now an additional repository - the one you’ve just created. That’s it – you now have your own NuGet package repository and you can add it to all your projects.

Using NuGet without committing packages to source control

When you add the NuGet package to your solution it will be added to your source control. So all the dll files will be committed to the source control. Starting with NuGet version 2 you have the option to download the library when you compile so the missing libraries will be downloaded on demand instead of keeping them on the source control. In order to enable this in VS open Tools > Library Package Manager > Package Manager Settings and select General. Check the Allow NuGet to download missing packages during build. This should be done on every machine. Next right click on the solution which uses the NuGet packages and select Enable NuGet Package Restore.

When you enable NuGet Package Restore a new .nuget folder will be created. It contains NuGet.exe file. It is executed during the build process to install the missing packages. This file should be added to source control. So if you have all your *.exe files ignored make sure NuGet.exe is not excluded, otherwise the build will fail.

Conclusion

One of the best way of sharing your libraries is using NuGet packages. You need to setup a TeamCity configuration build and configure the Visual Studio to access it. This post describes the steps you need to follow.