How to integrate CI/CD with Visual Studio Team Services?

Microsoft provides free Visual Studio Team Services for small team. It has free unlimited Git repositories and 4 hours per month to run builds, which should be more than enough for my projects.

I already integrated karpach.com front end and admin site. In this post I'll document how I integrated redboxnewreleases.com.

Step 1 Created build definition

My favorite browser is Google Chrome, but for below process I used Microsoft Edge, since some things didn't work in Google Chrome (e.g. reordering of build tasks).

I didn't use any templates and started with an empty process. First of all lets tried to build my solution. I used following tasks as is:

  1. Get sources (point to git repository)
  2. NuGet Restore **/*.sln (default settings)
  3. Build solution **/*.sln (default setting)

In my case the build failed with following error:

Error MSB4226: The imported project  "C:\Program Files (x86)\Microsoft Visual Studio\2017\
Enterprise\MSBuild\ExtensionPack\4.0\MSBuild.ExtensionPack.tasks" was not found.

In my previous CI system I used ExtensionPack for assembly version modification. This is old approach, so I just removed that task reference and used different approach to specify version later in the post.

I pushed a change, but it didn't trigger any build. I added a trigger then:

  1. Opened a Triggers tab.
  2. Enabled continuos integration trigger.
  3. Set repository setting to build on any master branch change.
  4. Went to options tab and set "Default agent queue": Hosted VS2017.
  5. Pressed Save & queue

This time build succeeded. Then I added assembly versioning back:

  1. Downloaded ApplyVersionToAssemblies.ps1 from https://github.com/tfsbuildextensions/CustomActivities.
  2. Deleted "$Env:TF_BUILD -and -not" from that powershell script.
  3. Replaced TF_BUILD with just BUILD in ApplyVersionToAssemblies.ps1.
  4. Added ApplyVersionToAssemblies.ps1 to Git repository.
  5. Went back to build definition web editor, switched to Variables tab. Added MajorVersion and MinorVersion variables there.
  6. Switched to the Options tab and filled "Build number format" field with $(BuildDefinitionName)_$(MajorVersion).$(MinorVersion).$(Year:yy)$(DayOfYear)$(Rev:.rr)).
  7. Made sure that AssemblyVersion and AssemblyFileVersion lines present in AssemblyInfo files (powershell script does replacement in those files).
  8. Added powershell task to run ApplyVersionToAssemblies.ps1.
  9. Moved up powershell task to run after Get sources task.

After that I modified msbuild task to produce a package for web deployment:

  1. Change build to build web csproj file instead of solution file
  2. Specified configuration: Release
  3. Used following MSBuild Arguments: /T:Package /P:PackageLocation=..\Artifacts\package.zip
  4. Added Publish Build Artifacts task: Path to Publish = Artifacts\package.zip, Artifact Name = Redbox

Build definition

Step 2 Created release definition

  1. Started from Empty template.
  2. Picked my build from previous step.
  3. Added a deployment group phase task.
  4. Added "IIS Web App Deploy (Preview)" task.
  5. Created deployment group.
  6. Ran generated powershell script on my VPS hosting server.
  7. Specified website name: redboxnewrleases.com.
  8. Specified package folder using browse button to point to package.zip: $(System.DefaultWorkingDirectory)/Redbox-CI/Redbox/package.zip).
  9. Set trigger for continuous deployment.
  10. Optionally modified release name format in General tab: Release-$(Build.BuildNumber))

Release definition

Posted on Saturday, July 8, 2017 by | Add Comment

How to get latest tweets from Twitter REST API?

My blog uses twitter REST API to pull up my latest tweets to the right side navigation. About a year ago twitter deprecated 1.0 API and now every request needs to be authenticated. According to twitter's documentation I can use application-only authentication, since I am only getting tweets and don't post anything.

Application-only authentication approach consists of two steps:


  1. Get access token from https://api.twitter.com/oauth2/token
  2. Use access token for any read only operations (get posts, friends, followers, user information or search tweets)

I didn't find any .NET bare-bones example that suits my needs. The code below don't need any third party libraries. It is .NET 4.5 (I used some dynamic and async / await).

Here is how you get access token (step 1 from the above):

public async Task<string> GetAccessToken()
{           
	var httpClient = new HttpClient();
	var request = new HttpRequestMessage(HttpMethod.Post, "https://api.twitter.com/oauth2/token ");
	var customerInfo = Convert.ToBase64String(new UTF8Encoding()
							  .GetBytes(OAuthConsumerKey + ":" + OAuthConsumerSecret));
	request.Headers.Add("Authorization", "Basic " + customerInfo);
	request.Content = new StringContent("grant_type=client_credentials", 
                                            Encoding.UTF8, "application/x-www-form-urlencoded");
															  
	HttpResponseMessage response = await httpClient.SendAsync(request);

	string json = await response.Content.ReadAsStringAsync();
	var serializer = new JavaScriptSerializer();
	dynamic item = serializer.Deserialize<object>(json);
	return  item["access_token"];            
}

Here is how you get tweets:

public async Task<IEnumerable<string>> GetTweets(string userName,int count, string accessToken = null)
{
	if (accessToken == null)
	{
		accessToken = await GetAccessToken();   
	}

	var requestUserTimeline = new HttpRequestMessage(HttpMethod.Get, 
		string.Format("https://api.twitter.com/1.1/statuses/user_timeline.json?
                               count={0}&screen_name={1}&trim_user=1&exclude_replies=1", count, userName));
	requestUserTimeline.Headers.Add("Authorization", "Bearer " + accessToken);
	var httpClient = new HttpClient();
	HttpResponseMessage responseUserTimeLine = await httpClient.SendAsync(requestUserTimeline);
	var serializer = new JavaScriptSerializer();
	dynamic json = serializer.Deserialize<object>(await responseUserTimeLine.Content.ReadAsStringAsync());
	var enumerableTweets = (json as IEnumerable<dynamic>);

	if (enumerableTweets == null)
	{
		return null;
	}
	return enumerableTweets.Select(t => (string)(t["text"].ToString()));                        
}

See complete console application at github repository Twitter .NET C# Sample Application

Posted on Monday, August 18, 2014 by | Comments (12) | Add Comment

How to migrate from subversion to git with almost no down time?

Svn to git migration

Last year I was in charge of SVN to Git migration at company that I work for. We wanted to migrate the history as well. In our case there were about 40,000 revisions made during last 8 years. In order to minimize developers downtime I did a lot of scripting preparation ahead of time. Actual switch from SVN to Git took less then 2 hours. Here are the steps that we took.

1. Retrieve a list of all committers

You'll need to create a list of users that have committed to the SVN repo and then convert those users over to the Git format as Subversion only supplies the username of the person committing and not the username and email. To retrieve the list of users from SVN, create a new folder, right click and select Git Bash Here to open a Git command window. Run the following command:

svn log http://url/to/svn/repository -q | awk -F '|' '/^r/ {sub("^ ", "", $2); 
sub(" $", "", $2); 
print $2" = "$2" <"$2">"}' | sort -u > users.txt

Note: this will take a couple of minutes to complete based on the size of your repository, number of commits, and number of committers.

The text file will have separate lines for each committer and will need to be transformed from vkarpach = vkarpach <vkarpach> to vkarpach = Viktar Karpach <vkarpach@company.com>

2. Clone the repository using git-svn

Note - this step will take hours to complete, so it is suggested to run this step over night on dedicated box. Run the following command to convert the repository to a Git repository:

git svn clone --stdlayout --no-metadata -A users.txt http://url/to/svn/repository dest_dir-tmp

3. Make a copy of this folder.

git svn clone takes a lot of time. For our main project it took 48 hours for about 18000 commits. Make a copy of this folder, so you don't need to do it again. Create scripts for next steps, so when you are ready to switch you can do it quickly.

4. Fetch latest commits.

The team continued to use Subversion until a very last moment, so while working on migration scripts time to time I had to fetch latest commits.

git svn fetch
git reset --hard trunk

5. Clean up script.

Delete tags

for t in `git branch -r | grep 'tags/' | sed s_tags/__` ; do
  git tag $t tags/$t^
  git branch -d -r tags/$t
done

Delete trunk, since we will use master from now on.

git branch -d -r trunk

Remove SVN references

git config --remove-section svn-remote.svn
rm -rf .git/svn .git/{logs/,}refs/remotes/svn/

And finally convert the remaining remote branches to local branches

git config remote.origin.url .
git config --add remote.origin.fetch +refs/remotes/*:refs/heads/*
git fetch

Remove remote branches:

for t in `git branch -r` ; do
  git branch -d -r $t
done

Git doesn't support space in branch names, so git svn fetch replaced spaces with %20. I think it is more aesthetic to use underscore instead of %20:

for t in `git branch -a|grep '%20'` ; do
  newName=`echo $t | sed 's/%20/-/g'`
  git branch -m $t $newName
done

You might want to delete some unused branches:

for t in `cat ../list_of_branches_for_deletion.txt`; do 
  git branch -D $t
done

Where list_of_branches_for_deletion.txt contains branch names that will be deleted. Use following code to populate this files:

git branch -a > ../list_of_branches_for_deletion.txt

Manually edit list_of_branches_for_deletion.txt file. Leave only those branches that you want to delete.

6. Replace any svn externals with git submodules

git submodule add ssh://git@git.company.com:7999/ProjectName/external_repo.git ExternalFolderName
git commit -m "Added submodules"

Use sumbodules only for external projects that don't change very often. We had to combine our internal projects in one git repository, since it is hard to maintain submodules for rapidly changing projects. Each project gets its own directory in git repository:

Before migration:

svn_main_project
	external_1
		external_1_folder_1
		external_1_folder_2
	external_2
		external_2_folder_1
		external_2_folder_2
	svn_main_project_folder_1
	svn_main_project_folder_2

Where svn_main_project has to externals external_1 and external_2.

After migration

	git
		svn_main_project
			svn_main_project_folder_1
			svn_main_project_folder_2
		external_1
			external_1_folder_1
			external_1_folder_2
		external_2
			external_2_folder_1
			external_2_folder_2	

You can use following bash script to push everything in sub_folder, so later you can combine repositories. The script will modify commit history as well.

git filter-branch --index-filter \	
	'git ls-files -s | sed "s-\t\"*-&sub_folder/-" |
		GIT_INDEX_FILE=$GIT_INDEX_FILE.new \
			git update-index --index-info &&
	 mv "$GIT_INDEX_FILE.new" "$GIT_INDEX_FILE" || true' HEAD

7. Get your repository onto the server

Create a repository on your git server.

Init local repository

git init

Use following if you are combining repositories:

git remote add external_1 ../external_1/
git pull external_1 master
git remote rm external_1

Add gitignore

cp ../gitignore.txt .gitignore
git add .
git commit -m "Added .gitignore"

Push all branches in one shot:

git remote add origin ssh://git@git.company.com:7999/repo.git
git push --all origin
Posted on Monday, January 27, 2014 by | Add Comment

Categories

Recent Tweets

  • Simon Ince's Blog: Hierarchies with HierarchyID in SQL 2008 http://t.co/xSDwiF6rRS.
  • Visual Studio 2010 WAS painfully slow - CodeProject http://t.co/Usba1x6CZy

Valid HTML5