clone


Script to clone all git repositories from all projects of a privately hosted Bitbucket server

The following script can download all git repositories from all of the projects that you have access to on a privately hosted Bitbucket server.

The execution work-flow of this script is as follows:

  1. It will ask for your username (the one you use to login on the Bitbucket server)
  2. Then it will ask for your password, the password will not be visible on screen as you type because we disabled the echo functionality for that step.
  3. Later, you will be prompted to provide the URL of the server, for this step be sure to define the protocol if it is http or https and the correct port number as well (e.g. https://bitbucket.bytefreaks.net:7990)
  4. Finally, you will be requested to give the location to where the repositories should be cloned to.
  5. Then the script will connect to the server, get the list of projects that you have access to and for each project retrieve the repositories of the project and clone them in the designated folder.
clone.all.hosted.bitbucket.sh (compressed) (66 downloads)
#!/bin/bash

echo -n "Username: ";
read username;
echo -n "Password: "; 
#Disabling echo, so that password will not be visible on screen
read -s password
#Enabling echo
echo

echo -n "Server (e.g https://repository.bytefreaks.net:7990): ";
read server;
echo -n "Location to clone repositories in: ";
read location;

mkdir -p "$location";
cd "$location";

#Getting all projects
curl --user "$username:$password" "$server/rest/api/1.0/projects/" | \
  grep -oP '"key":"\K\w+' | xargs -I {} -n 1 -I_project -- sh -c \
    "curl --user \"$username:$password\" \"$server/rest/api/1.0/projects/_project/repos\" | grep -o '\"ssh:[^ ,]\+' | xargs -L1 git clone";

exit 0;

clone.all.hosted.bitbucket.sh (compressed) (66 downloads)

 

Notes for the future:

  • Separate the cloned repositories per project
  • Support for people that have hundreds of projects and/or hundreds of repositories using the paging functionality

[GitLab.com] Clone all repositories in your account

GitLab.com offers a public API that allows us to get information related to our accounts. One of the API calls available is the account projects call (http://gitlab.com/api/v3/projects).

This call will return a JSON object describing the projects available to your account.

To clone all of the projects available to you, you can use the following:

TOKEN="PASTE_YOUR_PRIVATE_TOKEN_HERE"; PREFIX="ssh_url_to_repo"; curl --header "PRIVATE-TOKEN: $TOKEN" http://gitlab.com/api/v3/projects | grep -o "\"$PREFIX\":[^ ,]\+" | awk -F ':' '{printf "ssh://"; for (i=2; i<NF; i++) printf $i "/"; print $NF}' | xargs -L1 git clone

The above code will bring the JSON object, filter out everything except for the “ssh_url_to_repo” member of each project and then it will use it to clone the project by fixing up the URL to be used by git.

To get the above code working: the GitLab API requires that you use a token that is related to your account instead of using your credentials to make the call to the API.

To get your private token, visit this page http://gitlab.com/profile/account , the private token is the random sequence of characters in the white box:

[GitLab.com] Private TokenYou need to copy that value in the place of the variable TOKEN in the above script.

In case you have a lot of projects (more than 10), the default call will only produce the results for the first 10 repositories only.

To list all available repositories you have two options:

  1.  Set the per_page query parameter to a value big enough to fetch all your projects information if they are less than 100. e.g http://gitlab.com/api/v3/projects?per_page=100
  2. Follow the link headers from the initial response to make all the next calls.

Clone all repositories you have access to over ssh

ssh [email protected] info | cut -f 2 | tail -n +3 | xargs -I {} -n 1 git clone ssh://[email protected]/{}

The above command will connect to the git server (git.bytefreaks.net) that is using  gitolite and get a list of all the repositories you have access to using ssh [email protected] info

The command should return a list similar to this:

hello bytefreaks, this is [email protected] running gitolite3 v3.5.3.1-1-gf8776f5 on git 1.7.1

 R W	Repo1
 R W	Repo2
 R W	Repo3
 R  	Repo4

From the results, we remove the first 3 lines as they contain no useful information to cloning all the repositories. From the rest of the lines, where each line contains the information for a repository we have access to, we keep the third column only as it is the one that holds the repository name as it is stored on the server.

On the last stage of the pipe we have a list of the names of the repositories, using xargs, we assign each repository name to the special variable {} and processing one result at a time we clone the git repository to the current directory under the folder that is named as the repository.


Ubuntu 11.10 (Oneiric Ocelot): Cloning a KVM Virtual Machine

After properly installing and creating your first virtual machine, you might want to create a clone for some reason.

To do this we use the command virt-clone as follows:

sudo virt-clone --connect qemu:///system -o coeus -n phoebe -f /home/kvm/2KA.qcow2 -f /home/kvm/zK6.qcow2 --force

Where:

  • –connect is used to connect to the hypervisor of the virtual machine, here it is system
  • -o is the original virtual machine name (which is registered with the previous hypervisor)
  • -n is the name of the new virtual machine, the clone (this name will be used to register the clone to the hypervisor)
  • -f the location that the clone will use to store it’s virtual hard drives, can be used as many times as needed (in the above example coeus has two hard disks and that is why we need to provide two -f paramaters)
  • –force prevents interactive prompts and replies ‘yes’ to all yes/no questions

There is more directives that can be provided to the clone manager from which a very important one is the: –mac where you get to define the mac address of the new virtual machine (e.g –mac C0:FF:EE:11:00:11)