git

[GitLab.com] Clone all repositories in your account 1

 in Bash / GNU/Linux  tagged bash / clone / curl / git / gitlab / gitlab.com / GNU/Linux / howtos / ssh by

GitLab.com offers a public API that allows us to get information related to our accounts. One of the API calls available is the account projects call (http://gitlab.com/api/v3/projects).

This call will return a JSON object describing the projects available to your account.

To clone all of the projects available to you, you can use the following:

TOKEN="PASTE_YOUR_PRIVATE_TOKEN_HERE"; PREFIX="ssh_url_to_repo"; curl --header "PRIVATE-TOKEN: $TOKEN" http://gitlab.com/api/v3/projects | grep -o "\"$PREFIX\":[^ ,]\+" | awk -F ':' '{printf "ssh://"; for (i=2; i<NF; i++) printf $i "/"; print $NF}' | xargs -L1 git clone

The above code will bring the JSON object, filter out everything except for the “ssh_url_to_repo” member of each project and then it will use it to clone the project by fixing up the URL to be used by git.

To get the above code working: the GitLab API requires that you use a token that is related to your account instead of using your credentials to make the call to the API.

To get your private token, visit this page http://gitlab.com/profile/account , the private token is the random sequence of characters in the white box:

[GitLab.com] Private TokenYou need to copy that value in the place of the variable TOKEN in the above script.

In case you have a lot of projects (more than 10), the default call will only produce the results for the first 10 repositories only.

To list all available repositories you have two options:

  1.  Set the per_page query parameter to a value big enough to fetch all your projects information if they are less than 100. e.g http://gitlab.com/api/v3/projects?per_page=100
  2. Follow the link headers from the initial response to make all the next calls.

[GitLab.com] Get a list with the names of all repositories in your account

 in Bash / GNU/Linux / HowTos  tagged bash / curl / git / gitlab / gitlab.com / GNU/Linux / howtos by

GitLab.com offers a public API that allows us to get information related to our accounts. One of the API calls available is the account projects call (http://gitlab.com/api/v3/projects).

This call will return a JSON object describing the projects available to your account.

To get a list of the names of the projects available to you, you can use the following:

TOKEN="PASTE_YOUR_PRIVATE_TOKEN_HERE"; PREFIX="ssh_url_to_repo"; curl --header "PRIVATE-TOKEN: $TOKEN" http://gitlab.com/api/v3/projects | grep -o "\"$PREFIX\":[^ ,]\+" | xargs -L1 basename | awk -F '.' '{print $1}'

The above code will bring the JSON object, filter out everything except for the “ssh_url_to_repo” member of each project and then it will print it out on screen.

 

To get the above code working: the GitLab API requires that you use a token that is related to your account instead of using your credentials to make the call to the API.

To get your private token, visit this page http://gitlab.com/profile/account , the private token is the random sequence of characters in the white box:

[GitLab.com] Private TokenYou need to copy that value in the place of the variable TOKEN in the above script.

 

In case you have a lot of projects (more than 10), the default call will only produce the results for the first 10 repositories only.

To list all available repositories you have two options:

  1.  Set the per_page query parameter to a value big enough to fetch all your projects information if they are less than 100. e.g http://gitlab.com/api/v3/projects?per_page=100
  2. Follow the link headers from the initial response to make all the next calls.

[BitBucket.org] Clone all repositories of your account 2

 in Bash / GNU/Linux / HowTos  tagged bash / BitBucket / BitBucket.org / curl / git / GNU/Linux / howtos / ssh by

Clone all bitbucket projects

 BBA=MyUserName; curl --user ${BBA} https://api.bitbucket.org/2.0/repositories/${BBA} | grep -o '"ssh:[^ ,]\+' | xargs -L1 git clone

The above curl call will connect to the server using your username and return the list of repositories that are available to your account.

Please note that you need to provide you username NOT your email.
If you make these calls using the email that was used to register the account, then the call will fail.

After the call succeeds, the results will be filtered and each repository will be cloned to the current folder.
In case your ssh key is locked via a password, each time a clone operation will start, you will be asked for the password.

Example:

BBA="bytefreaks"; curl --user ${BBA} https://api.bitbucket.org/2.0/repositories/${BBA} | grep -o '"ssh:[^ ,]\+' | xargs -L1 git clone
Enter host password for user 'bytefreaks':
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  3834  100  3834    0     0   4414      0 --:--:-- --:--:-- --:--:--  4411
Cloning into 'bluetoothclicker'...
Warning: Permanently added the RSA host key for IP address '104.192.143.1' to the list of known hosts.
Enter passphrase for key '/home/bytefreaks/.ssh/BitBucket/id_rsa': 
warning: You appear to have cloned an empty repository.
Checking connectivity... done.
Cloning into 'watch'...
Enter passphrase for key '/home/bytefreaks/.ssh/BitBucket/id_rsa': 
warning: You appear to have cloned an empty repository.
Checking connectivity... done.

List all bitbucket projects

In case what you want is just to list your repositories, execute the following:

 curl --user ${BBA} https://api.bitbucket.org/2.0/repositories/${BBA} | grep -o '"ssh:[^ ,]\+' | xargs -L1 echo

Usage instructions: set your username to the BBA variable and execute.

BBA="bytefreaks"; curl --user ${BBA} https://api.bitbucket.org/2.0/repositories/${BBA} | grep -o '"ssh:[^ ,]\+' | xargs -L1 echo
Enter host password for user 'bytefreaks':
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  3834  100  3834    0     0   3543      0  0:00:01  0:00:01 --:--:--  3546
ssh://[email protected]/bytefreaks/bluetoothclicker.git
ssh://[email protected]/bytefreaks/watch.git

Pull all Git repositories you have access to

 in Bash  tagged cut / git / gitolite / pull / repository / tail / xargs by 

ssh [email protected] info | cut -f 2 | tail -n +3 | xargs -I {} -n 1 -I_repository -- sh -c 'cd _repository; git pull; cd ..;'

The above command will connect to the git server (git.bytefreaks.net) using  gitolite and get a list of all the repositories you have access to using ssh [email protected] info

The command should return a list similar to this:

hello bytefreaks, this is git@git running gitolite3 v3.5.3.1-1-gf8776f5 on git 1.7.1

 R W	Repo1
 R W	Repo2
 R W	Repo3
 R  	Repo4

From the results, we remove the first 3 lines as they contain no useful information to cloning all the repositories. From the rest of the lines, where each line contains the information for a repository we have access to, we keep the third column only as it is the one that holds the repository name as it is stored on the server.

Afterwards it will remove all columns except the second to filter the column with the repository names and will remove the first 3 lines to keep only the data we are interested in.

On the last stage of the pipe we have a list of the names of the repositories, using xargs, we assign each repository name to the _repository variable and using one result at a time, we navigate into the folder of the repository using cd and call the pull command.

Note: We assume that all repositories are in the current folder as children and each one is in a sub-folder of its own which is named as the repository is.