How to: migrate SVN repository to Git and keep history

Hi all,

Today we are going to cover topic about moving from Subversion to Git and ways to copy / migrate SVN data to Git repository (so we can keep the history of commits)

I would recommend doing the migration process using Linux OS as it seems to be much faster than on Windows OS (at least when I was testing it) - this is especially for large repositories.


  • Create "authors" file that relates to SVN repo commit authors. It should be in the following format:
username1 = user_firstname user_lasname <user_email>


username2 = user_firstname user_lasname <user_email>
and so on...



  • Tell Git about it: 
git config --global svn.authorsfile <path_to_authors_file>

  • Init Git repository with SVN reference:
git svn init <svn_repo_url> --stdlayout --prefix=svn/

  • Review/edit .git/config file, it should look like this:
[svn-remote "svn"]
url = <svn_repo_url>
fetch = trunk:refs/remotes/svn/trunk
branches = branches/*:refs/remotes/svn/*
tags = tags/*:refs/remotes/svn/tags/*
[svn]
authorsfile = /path/to/authors.txt

If the SVN repository is large you might want to limit the fetch to only some branches and tags. If so you can do it by specifying them in the .git/config file, for example:
[svn-remote "svn"]
url = <svn_repo_url>
fetch = trunk:refs/remotes/svn/trunk
branches = branches/{feature_branchA,old_branchB,branch_test}:refs/remotes/svn/*
        tags = tags/{tagA,tagB,tagC}:refs/remotes/svn/tags/*


  • Fetch the commits: git svn fetch
Occasionally it may stop due to some random errors or due to unknown commit author. In that case you can add new row/entry in your authors. You can also try to clear memory by running:
git gc OR/AND git svn gc

To resume and continue the fetch run: git svn fetch

  • After a while depending on the repository size it will finish and you can view all available SVN branches by running the command: git branch -a

  • Next step is to create Git branches and tags to the corresponding SVN branches and tags. For each SVN branch run the following command:
git checkout -b <BRANCH_NAME> svn/<BRANCH_NAME>

  • The similar thing for tags, but you might choose to leave tags as branches whatever you prefer:
git checkout -b <tag_TAGNAME> svn/tags/<TAG_NAME> // creates a branch for SVN tag
git checkout master
git tag <TAGNAME> <tag_TAGNAME> // create git tag to that branch
git branch -D <tag_TAGNAME> // delete the previously created branch, tags stays

Repeat for each SVN tag.
  • Switch to the master and create .gitignore file:
git svn show-ignore > .gitignore
// edit .gitignore to manually add whatever you want to ignore
git add .gitignore
git commit -m 'Convert svn:ignore properties to .gitignore.'

Alternatively and/or additionally you can set ignore globally for that repo:
git svn show-ignore >> .git/info/exclude 
  • define remote Git repository so it's added to the .git/config
git remote add origin <GIT_REPO_URL>
  • push everything to the Git repository
git push -u origin -all // push all branches and changes to the repo named origin
git push --tags origin // push the tags as previous command ignores tags

Additions/updates

A) If you want to add some branch or tag later you can do it by telling about it in .git/config:
[svn-remote "branch_MyBranchX"]
url = <SVN_REPO_URL>branches/MyBranchX/

    fetch = :refs/remotes/svn/MyBranchX

Then:

  • Fetch it: git svn fetch
  • Create git branch for it: git svn checkout -b MyBranchX svn/MyBranchX
  • Push it: git push origin MyBranchX

B) If you want to update existing branch from SVN because your team has not switched SVN off yet and you want to sync later changes just fetch the changes:

git svn fetch OR git svn fetch <BRANCH_NAME>

Then switch to related Git branches and rebase:
git checkout <EXISTING_GIT_BRANCH_NAME>
git rebase svn/<BRANCH_NAME> // sync with the recent changes from SVN
git push origin <BRANCH_NAME>

This is it! Happy migrationg :)

Maven - useful tips and tricks

Hello,

Here are some commands and settings which you may find useful when working with Maven.


  • Installing third party JAR artifacts into the local repository

mvn install:install-file -Dfile=<path_to_jar> -DgroupId=<groupid> -DartifactId=<artifactid> -Dversion=<version> -Dpackaging=<packaging>

  • Running build in the offline mode: mvn clean install -o
  • Copy all project dependencies to the folder target/dependencies:
mvn dependency:copy-dependencies

Note: you may actually only be interested in the 'runtime' dependencies:
mvn dependency:copy-dependencies -DincludeScope=runtime

  • Delete / purge the existing dependencies and download again:
mvn dependency:purge-local-repository

  • Check for newer versions of project dependencies and plugins:

mvn versions:display-dependency-updates
mvn versions:display-plugin-updates

  • Skipping unit test from the build: mvn clean install -DskipTests=true

Note: you can make it permanent in the POM file configuration:
<properties>
  <skip.tests>true</skip.tests>
</properties>

<build>
    <plugins>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-surefire-plugin</artifactId>
        <version>2.18.1</version>
        <configuration>
          <skipTests>${skip.tests}</skipTests>
        </configuration>
      </plugin>
    </plugins>

</build>


For more info and guides please visit the official webiste: https://maven.apache.org/



Gerrit with MySQL installation guide

Hello,

Gerrit is a great tool for pre-commit code review. Installation on Linux based OS is kind of default and it's nice and simple. However when you try to do it on Windows you may end up having some sort of issues. So this has been tested on Windows OS but you can also try it on Linux based OS.

1. Download and install MySQL: http://dev.mysql.com/downloads/

2. Download and install MySQL Workbench - useful tool for accessing the databases: http://dev.mysql.com/downloads/workbench/

3. Create Gerrit review database and user for it:

CREATE DATABASE gerritdb;
ALTER DATABASE gerritdb charset=latin1;

CREATE USER 'gerrit'@'localhost' IDENTIFIED BY 'gerrit';
GRANT ALL ON gerritdb.* TO 'gerrit'@'localhost';
FLUSH PRIVILEGES;

CREATE USER 'gerrit'@'<hostname>' IDENTIFIED BY 'gerrit';
GRANT ALL ON gerritdb.* TO 'gerrit'@'<hostname>';
FLUSH PRIVILEGES;


5. Create a folder to host gerrit and place the WAR in it.

6. Installation:

gerrit/$ java -jar gerrit.war init

It will ask for various different settings. For some of them default option seems OK. See below some of the setting you may want to specify other than defaults:

canonicalWebUrl = http://gerrit:8080/

[database]
    type = mysql
    hostname = localhost // OR network hostname
    database = gerritdb
    username = gerrit (password prompt = gerrit)

Run as user: gerrit

NOTE: if you use LDAP and/or you have a mail server it's better to set it up directly during installation rather than later. Note all passwords provided will not be visible in the /gerrit/etc/gerrit.config file which is good.

Example of LDAP settings:
[ldap]
    server = ldap://ldapSrvName.aaa.bbb
    username = CN=John Smith, OU=Country,OU=Users,DC=aaa,DC=bbb
    accountBase = DC=aaa,DC=bbb
    groupBase = DC=aaa,DC=bbb

Example of e-mail server settings:
[sendemail]
    enable = true
    from = SERVER
    threadPoolSize = 1
    smtpServer = emailSrvName.aaa.bbb
    smtpServerPort = 25
    smtpEncryption = tls
    sslVerify = false

7. Go to /gerrit/etc and generate ssh keys with the specified names:
gerrit/etc/$ ssh-keygen
name: ssh_host_key

chmod 600 ssh_host_key && chmod 600 ssh_host_key.pub

8. On Windows there may be an error regarding virtual memory command, so if you are installing this on Win OS edit /gerrit/bin/gerrit.sh:
- comment out virtual memory command (on or near line #303)
- add line at the beginning GERRIT_SITE=<path_to_gerrit_folder>
e.g. GERRIT_SITE=/c/gerrit

9. It's likely that gerrit has failed to create some of the tables (if not then you can skip this step). Create the missing tables manually in the MySQL database:
create table account_group_by_id_aud (
  added_by INT DEFAULT 0 NOT NULL,
  removed_by INT,
  removed_on TIMESTAMP NULL DEFAULT NULL,
  group_id INT DEFAULT 0 NOT NULL
  include_uuid VARCHAR(255) BINARY DEFAULT '' NOT NULL
  added_on TIMESTAMP NOT_NULL DEFAULT CURRENT_TIMESTAMP,
  PRIMARY KEY(group_id,include_uuid,added_on)
);

create table account_group_members_audit (
  added_by INT DEFAULT 0 NOT NULL,
  removed_by INT,
  removed_on TIMESTAMP NULL DEFAULT NULL,
  account_id INT DEFAULT 0 NOT NULL
  group_id INT DEFAULT 0 NOT NULL
  added_on TIMESTAMP NOT_NULL DEFAULT CURRENT_TIMESTAMP,
  PRIMARY KEY(account_id,group_id,added_on)
);

create tables changes (
  change_key VARCHAR(60) BINARY DEFAULT '' NOT NULL,
  created_on TIMESTAMP NOT NULL,
  last_updated_on TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
  owner_account_id INT DEFAULT 0 NOT NULL,
  dest_project_name VARCHAR(255) BINARY DEFAULT '' NOT NULL,
  dest_branch_name VARCHAR(255) BINARY DEFAULT '' NOT NULL,
  status CHAR(1) DEFAULT ' ' NOT NULL,
  current_patch_set_id INT DEFAULT 0 NOT NULL,
  subject VARCHAR(255) BINARY DEFAULT '' NOT NULL,
  topic VARCHAR(255) BINARY,
  original_subject VARCHAR(255) BINARY,
  row_version INT DEFAULT 0 NOT NULL,
  change_id INT DEFAULT 0 NOT NULL,
  PRIMARY KEY(change_id)
);

10. Run again gerrit WAR to check if there are no table errors this time:
gerrit/$ java -jar gerrit.war init --batch -d <installation_folder>

11. Run reindex task:
gerrit/$ java -jar gerrit.war reindex

12. Make sure the changes done for /gerrit/bin/gerrit.sh are still there. If not apply step 8 again.

13. There may be an error when querying users accounts, so to prevent run the following updates on the database:
SET collation_connection = 'utf8_general_ci'
ALTER DATABASE gerritdb CHARACTER SET utf8 COLLATE utf8_general_ci
ALTER TABLE accounts CONVERT TO CHARACTER SET utf8 COLLATE utf8_general_ci
ALTER TABLE account_external_ids CONVERT TO CHARACTER SET utf8 COLLATE utf8_general_ci


14. Start gerrit.sh script

If failed to start no worries, check it if it's actually running because the message may just be misleading. If really failed then possible errors / fixes:
- port blocked - kill all java processes and try again
- reindex gerrit (see the step 11)

If nothing else helps you can try to drop the database and recreate it again.


Hope that helps. Cheers!

If you have other issues or questions check out official documentation and tutorials: https://www.gerritcodereview.com/