How to migrate from subversion to git with almost no down time?

Svn to git migration

Last year I was in charge of SVN to Git migration at company that I work for. We wanted to migrate the history as well. In our case there were about 40,000 revisions made during last 8 years. In order to minimize developers downtime I did a lot of scripting preparation ahead of time. Actual switch from SVN to Git took less then 2 hours. Here are the steps that we took.

1. Retrieve a list of all committers

You'll need to create a list of users that have committed to the SVN repo and then convert those users over to the Git format as Subversion only supplies the username of the person committing and not the username and email. To retrieve the list of users from SVN, create a new folder, right click and select Git Bash Here to open a Git command window. Run the following command:

svn log http://url/to/svn/repository -q | awk -F '|' '/^r/ {sub("^ ", "", $2); 
sub(" $", "", $2); 
print $2" = "$2" <"$2">"}' | sort -u > users.txt

Note: this will take a couple of minutes to complete based on the size of your repository, number of commits, and number of committers.

The text file will have separate lines for each committer and will need to be transformed from vkarpach = vkarpach <vkarpach> to vkarpach = Viktar Karpach <vkarpach@company.com>

2. Clone the repository using git-svn

Note - this step will take hours to complete, so it is suggested to run this step over night on dedicated box. Run the following command to convert the repository to a Git repository:

git svn clone --stdlayout --no-metadata -A users.txt http://url/to/svn/repository dest_dir-tmp

3. Make a copy of this folder.

git svn clone takes a lot of time. For our main project it took 48 hours for about 18000 commits. Make a copy of this folder, so you don't need to do it again. Create scripts for next steps, so when you are ready to switch you can do it quickly.

4. Fetch latest commits.

The team continued to use Subversion until a very last moment, so while working on migration scripts time to time I had to fetch latest commits.

git svn fetch
git reset --hard trunk

5. Clean up script.

Delete tags

for t in `git branch -r | grep 'tags/' | sed s_tags/__` ; do
  git tag $t tags/$t^
  git branch -d -r tags/$t
done

Delete trunk, since we will use master from now on.

git branch -d -r trunk

Remove SVN references

git config --remove-section svn-remote.svn
rm -rf .git/svn .git/{logs/,}refs/remotes/svn/

And finally convert the remaining remote branches to local branches

git config remote.origin.url .
git config --add remote.origin.fetch +refs/remotes/*:refs/heads/*
git fetch

Remove remote branches:

for t in `git branch -r` ; do
  git branch -d -r $t
done

Git doesn't support space in branch names, so git svn fetch replaced spaces with %20. I think it is more aesthetic to use underscore instead of %20:

for t in `git branch -a|grep '%20'` ; do
  newName=`echo $t | sed 's/%20/-/g'`
  git branch -m $t $newName
done

You might want to delete some unused branches:

for t in `cat ../list_of_branches_for_deletion.txt`; do 
  git branch -D $t
done

Where list_of_branches_for_deletion.txt contains branch names that will be deleted. Use following code to populate this files:

git branch -a > ../list_of_branches_for_deletion.txt

Manually edit list_of_branches_for_deletion.txt file. Leave only those branches that you want to delete.

6. Replace any svn externals with git submodules

git submodule add ssh://git@git.company.com:7999/ProjectName/external_repo.git ExternalFolderName
git commit -m "Added submodules"

Use sumbodules only for external projects that don't change very often. We had to combine our internal projects in one git repository, since it is hard to maintain submodules for rapidly changing projects. Each project gets its own directory in git repository:

Before migration:

svn_main_project
	external_1
		external_1_folder_1
		external_1_folder_2
	external_2
		external_2_folder_1
		external_2_folder_2
	svn_main_project_folder_1
	svn_main_project_folder_2

Where svn_main_project has to externals external_1 and external_2.

After migration

	git
		svn_main_project
			svn_main_project_folder_1
			svn_main_project_folder_2
		external_1
			external_1_folder_1
			external_1_folder_2
		external_2
			external_2_folder_1
			external_2_folder_2	

You can use following bash script to push everything in sub_folder, so later you can combine repositories. The script will modify commit history as well.

git filter-branch --index-filter \	
	'git ls-files -s | sed "s-\t\"*-&sub_folder/-" |
		GIT_INDEX_FILE=$GIT_INDEX_FILE.new \
			git update-index --index-info &&
	 mv "$GIT_INDEX_FILE.new" "$GIT_INDEX_FILE" || true' HEAD

7. Get your repository onto the server

Create a repository on your git server.

Init local repository

git init

Use following if you are combining repositories:

git remote add external_1 ../external_1/
git pull external_1 master
git remote rm external_1

Add gitignore

cp ../gitignore.txt .gitignore
git add .
git commit -m "Added .gitignore"

Push all branches in one shot:

git remote add origin ssh://git@git.company.com:7999/repo.git
git push --all origin
Posted on Monday, January 27, 2014 by | Add Comment

How to hide SyncToy console window in windows Task Scheduler?

SyncToy is a free app from Microsoft that synchronizes files and folders between different locations. It even has a command line version (SyncToyCmd.exe), which can be used in windows Task Scheduler. Unfortunately Task Scheduler build-in "Hidden" mode didn't hide console window, so I composed from different sources a little vbs script. This script can be used in Task Scheduler, so no console window will show up.

SyncToy.vbs

Const HIDDEN_WINDOW = 12
strComputer = "."

Set objWMIService = GetObject("winmgmts:" _
    & "{impersonationLevel=impersonate}!\\" & strComputer & "\root\cimv2")
Set objStartup = objWMIService.Get("Win32_ProcessStartup")
Set objConfig = objStartup.SpawnInstance_
objConfig.ShowWindow = HIDDEN_WINDOW
Set objProcess = GetObject("winmgmts:root\cimv2:Win32_Process")

errReturn = objProcess.Create("C:\Program Files\Synctoy 2.1\SynctoyCmd.exe -R" , null, objConfig, intProcessID)

Posted on Thursday, January 16, 2014 by | Add Comment

How to integrate First Data Gateway e4 with .NET WCF?

Recently I worked on a project that needed integration with First Data Getaway e4 payment processing. Starting from v12 First Data introduced HMAC hash security. I found .NET REST sample, but there is nothing for WCF or Web Services. This new HMAC security feature makes usage of Web Service a little challenging. I hope this guide will help somebody.

Step 1. Add service reference:


WCF: Add Service Reference

WCF: Add Service Reference Step 2

Step 2. Create demo account

Step 3. Use following code wrapper to charge a card:

public class Merchant
{
    private readonly string _gatewayId;
    private readonly string _password;
    private readonly string _keyId;
    private readonly string _hmac;
    private readonly bool _isDemo;

    private const string ProdUrl = "https://api.globalgatewaye4.firstdata.com/transaction/v12";
    private const string TestUrl = "https://api.demo.globalgatewaye4.firstdata.com/transaction/v12";


    public Merchant(string gatewayId, string password, string hmac, string keyId, bool isDemo = true)
    {
        _gatewayId = gatewayId;
        _password = password;
        _hmac = hmac;
        _keyId = keyId;
        _isDemo = isDemo;
    }

    public MerchantResponse Charge(int orderId, string cardHoldersName, string cardNumber, decimal amount, 
        int expirationMonth, int expirationYear, int ccv, string address, string city, string state, string zip)
    {            
        var client = new ServiceSoapClient(new BasicHttpBinding(BasicHttpSecurityMode.Transport), 
            new EndpointAddress(_isDemo ? TestUrl : ProdUrl));                                           
        client.ChannelFactory.Endpoint.Behaviors.Add(new HmacHeaderBehaivour(_hmac,_keyId));            

        TransactionResult result = client.SendAndCommit(new Transaction
                                                {                                                                        
                                                    ExactID = _gatewayId,
                                                    Password = _password,
                                                    Transaction_Type = "00",                                                                        
                                                    Card_Number = cardNumber,
                                                    CardHoldersName = cardHoldersName,                                                                                                                                                
                                                    DollarAmount = amount.ToString("F"),
                                                    Expiry_Date = string.Format("{0:D2}{1}",expirationMonth,expirationYear),
                                                    Customer_Ref = orderId.ToString(),
                                                    VerificationStr1 = string.Format("{0}|{1}|{2}|{3}|US",address,zip,city,state),
                                                    VerificationStr2 = ccv.ToString()
                                                });
        var response = new MerchantResponse
                        {
                            IsTransactionApproved = result.Transaction_Approved,
                            IsError = result.Transaction_Error
                        };
        if (!result.Transaction_Approved && !result.Transaction_Error)
        {
            response.Message = string.Format("Error {0}: {1}", result.Bank_Resp_Code, result.Bank_Message);
        }
        if (!result.Transaction_Approved && result.Transaction_Error)
        {
            response.Message = string.Format("Error {0}: {1}",result.EXact_Resp_Code,result.EXact_Message);
        }
        if (result.Transaction_Approved)
        {
            response.Message = result.Authorization_Num;
        }
        return response;
    }               

    class HmacHeaderBehaivour: IEndpointBehavior
    {
        private readonly string _hmac;
        private readonly string _keyId;

        public HmacHeaderBehaivour(string hmac, string keyId)
        {
            _hmac = hmac;
            _keyId = keyId;
        }

        public void Validate(ServiceEndpoint endpoint)
        {                
        }

        public void AddBindingParameters(ServiceEndpoint endpoint, BindingParameterCollection bindingParameters)
        {                
        }

        public void ApplyDispatchBehavior(ServiceEndpoint endpoint, EndpointDispatcher endpointDispatcher)
        {                
        }

        public void ApplyClientBehavior(ServiceEndpoint endpoint, ClientRuntime clientRuntime)
        {
            clientRuntime.MessageInspectors.Add(new HmacHeaderInspector(_hmac,_keyId));                                
        }
    }      
         

    class HmacHeaderInspector: IClientMessageInspector
    {
        private readonly string _hmac;
        private readonly string _keyId;

        public HmacHeaderInspector(string hmac,string keyId)
        {
            _hmac = hmac;
            _keyId = keyId;
        }

        public object BeforeSendRequest(ref Message request, IClientChannel channel)
        {                                                                
            MessageBuffer buffer = request.CreateBufferedCopy(Int32.MaxValue);
            request = buffer.CreateMessage();
            Message msg = buffer.CreateMessage();                
            ASCIIEncoding encoder = new ASCIIEncoding();
                
            var sb = new StringBuilder();
            var xmlWriter = XmlWriter.Create(sb, new XmlWriterSettings
                                                    {
                                                    OmitXmlDeclaration  = true
                                                    });
            var writer = XmlDictionaryWriter.CreateDictionaryWriter(xmlWriter);
            msg.WriteStartEnvelope(writer);
            msg.WriteStartBody(writer);
            msg.WriteBodyContents(writer);                                
            xmlWriter.WriteEndElement();
            xmlWriter.WriteEndElement();
            writer.Flush();

            string body = sb.ToString().Replace(" />","/>");

            byte[] xmlByte = encoder.GetBytes(body);
            SHA1CryptoServiceProvider sha1Crypto = new SHA1CryptoServiceProvider();
            string hash = BitConverter.ToString(sha1Crypto.ComputeHash(xmlByte)).Replace("-", "");
            string hashedContent = hash.ToLower();

            //assign values to hashing and header variables
            string time = DateTime.UtcNow.ToString("yyyy-MM-ddTHH:mm:ssZ");
            string hashData = "POST\ntext/xml; charset=utf-8\n" + hashedContent + "\n" + time + "\n/transaction/v12";
            //hmac sha1 hash with key + hash_data
            HMAC hmacSha1 = new HMACSHA1(Encoding.UTF8.GetBytes(_hmac)); //key
            byte[] hmacData = hmacSha1.ComputeHash(Encoding.UTF8.GetBytes(hashData)); //data
            //base64 encode on hmac_data
            string base64Hash = Convert.ToBase64String(hmacData);

            HttpRequestMessageProperty httpRequestMessage;
            object httpRequestMessageObject;

            if (request.Properties.TryGetValue(HttpRequestMessageProperty.Name, out httpRequestMessageObject))
            {
                httpRequestMessage = httpRequestMessageObject as HttpRequestMessageProperty;
                httpRequestMessage.Headers["X-GGe4-Content-SHA1"] = hashedContent;
                httpRequestMessage.Headers["X-GGe4-Date"] = time;
                httpRequestMessage.Headers["Authorization"] = "GGE4_API " + _keyId + ":" + base64Hash;
            }
            else
            {
                httpRequestMessage = new HttpRequestMessageProperty();
                httpRequestMessage.Headers["X-GGe4-Content-SHA1"] = hashedContent;
                httpRequestMessage.Headers["X-GGe4-Date"] = time;                    
                httpRequestMessage.Headers["Authorization"] = "GGE4_API " + _keyId + ":" + base64Hash;
                request.Properties.Add(HttpRequestMessageProperty.Name, httpRequestMessage);
            }                
            return null;
        }

        public void AfterReceiveReply(ref Message reply, object correlationState)
        {                
        }
    }
}

public class MerchantResponse
{
    public bool IsTransactionApproved { get; set; }
    public bool IsError { get; set; }
    public string Message { get; set; }
}

As you can see "Charge" method is simple, but HMAC requirement makes it challenging. HMAC hash is calculated based on SOAP request body, which is not accessible through ServiceSoapClient. Also we need to modify HTTP request header. Both things can be done using IClientMessageInspector or IClientMessageFormatter. In my implementation I used IClientMessageInspector, since it is a little easier to integrate it with ServiceSoapClient. I didn't find any easy way to get serialized SOAP body of the request, so using Fiddler I came up with following magic code:


var sb = new StringBuilder();
var xmlWriter = XmlWriter.Create(sb, new XmlWriterSettings
                                        {
                                        OmitXmlDeclaration  = true
                                        });
var writer = XmlDictionaryWriter.CreateDictionaryWriter(xmlWriter);
msg.WriteStartEnvelope(writer);
msg.WriteStartBody(writer);
msg.WriteBodyContents(writer);                                
xmlWriter.WriteEndElement();
xmlWriter.WriteEndElement();
writer.Flush();

string body = sb.ToString().Replace(" />","/>");

The code above gives me serialized SOAP envelope without SOAP headers string. This string is used for HMAC calculation and at the end X-GGe4-Content-SHA1, X-GGe4-Date, Authorization are added to HTTP header.

Tips: You might get 401 errors. Use Fiddler to see actual error. Make sure that you saved generated HMAC key (generate button doesn't save it).

Posted on Tuesday, January 07, 2014 by | Comments (4) | Add Comment

Categories

Valid HTML5