GNU/Linux

[GitLab.com] Clone all repositories in your account 1

 in Bash / GNU/Linux  tagged bash / clone / curl / git / gitlab / gitlab.com / GNU/Linux / howtos / ssh by

GitLab.com offers a public API that allows us to get information related to our accounts. One of the API calls available is the account projects call (http://gitlab.com/api/v3/projects).

This call will return a JSON object describing the projects available to your account.

To clone all of the projects available to you, you can use the following:

TOKEN="PASTE_YOUR_PRIVATE_TOKEN_HERE"; PREFIX="ssh_url_to_repo"; curl --header "PRIVATE-TOKEN: $TOKEN" http://gitlab.com/api/v3/projects | grep -o "\"$PREFIX\":[^ ,]\+" | awk -F ':' '{printf "ssh://"; for (i=2; i<NF; i++) printf $i "/"; print $NF}' | xargs -L1 git clone

The above code will bring the JSON object, filter out everything except for the “ssh_url_to_repo” member of each project and then it will use it to clone the project by fixing up the URL to be used by git.

To get the above code working: the GitLab API requires that you use a token that is related to your account instead of using your credentials to make the call to the API.

To get your private token, visit this page http://gitlab.com/profile/account , the private token is the random sequence of characters in the white box:

[GitLab.com] Private TokenYou need to copy that value in the place of the variable TOKEN in the above script.

In case you have a lot of projects (more than 10), the default call will only produce the results for the first 10 repositories only.

To list all available repositories you have two options:

  1.  Set the per_page query parameter to a value big enough to fetch all your projects information if they are less than 100. e.g http://gitlab.com/api/v3/projects?per_page=100
  2. Follow the link headers from the initial response to make all the next calls.

[GitLab.com] Get a list with the names of all repositories in your account

 in Bash / GNU/Linux / HowTos  tagged bash / curl / git / gitlab / gitlab.com / GNU/Linux / howtos by

GitLab.com offers a public API that allows us to get information related to our accounts. One of the API calls available is the account projects call (http://gitlab.com/api/v3/projects).

This call will return a JSON object describing the projects available to your account.

To get a list of the names of the projects available to you, you can use the following:

TOKEN="PASTE_YOUR_PRIVATE_TOKEN_HERE"; PREFIX="ssh_url_to_repo"; curl --header "PRIVATE-TOKEN: $TOKEN" http://gitlab.com/api/v3/projects | grep -o "\"$PREFIX\":[^ ,]\+" | xargs -L1 basename | awk -F '.' '{print $1}'

The above code will bring the JSON object, filter out everything except for the “ssh_url_to_repo” member of each project and then it will print it out on screen.

 

To get the above code working: the GitLab API requires that you use a token that is related to your account instead of using your credentials to make the call to the API.

To get your private token, visit this page http://gitlab.com/profile/account , the private token is the random sequence of characters in the white box:

[GitLab.com] Private TokenYou need to copy that value in the place of the variable TOKEN in the above script.

 

In case you have a lot of projects (more than 10), the default call will only produce the results for the first 10 repositories only.

To list all available repositories you have two options:

  1.  Set the per_page query parameter to a value big enough to fetch all your projects information if they are less than 100. e.g http://gitlab.com/api/v3/projects?per_page=100
  2. Follow the link headers from the initial response to make all the next calls.

[BitBucket.org] Clone all repositories of your account 2

 in Bash / GNU/Linux / HowTos  tagged bash / BitBucket / BitBucket.org / curl / git / GNU/Linux / howtos / ssh by

Clone all bitbucket projects

 BBA=MyUserName; curl --user ${BBA} https://api.bitbucket.org/2.0/repositories/${BBA} | grep -o '"ssh:[^ ,]\+' | xargs -L1 git clone

The above curl call will connect to the server using your username and return the list of repositories that are available to your account.

Please note that you need to provide you username NOT your email.
If you make these calls using the email that was used to register the account, then the call will fail.

After the call succeeds, the results will be filtered and each repository will be cloned to the current folder.
In case your ssh key is locked via a password, each time a clone operation will start, you will be asked for the password.

Example:

BBA="bytefreaks"; curl --user ${BBA} https://api.bitbucket.org/2.0/repositories/${BBA} | grep -o '"ssh:[^ ,]\+' | xargs -L1 git clone
Enter host password for user 'bytefreaks':
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  3834  100  3834    0     0   4414      0 --:--:-- --:--:-- --:--:--  4411
Cloning into 'bluetoothclicker'...
Warning: Permanently added the RSA host key for IP address '104.192.143.1' to the list of known hosts.
Enter passphrase for key '/home/bytefreaks/.ssh/BitBucket/id_rsa': 
warning: You appear to have cloned an empty repository.
Checking connectivity... done.
Cloning into 'watch'...
Enter passphrase for key '/home/bytefreaks/.ssh/BitBucket/id_rsa': 
warning: You appear to have cloned an empty repository.
Checking connectivity... done.

List all bitbucket projects

In case what you want is just to list your repositories, execute the following:

 curl --user ${BBA} https://api.bitbucket.org/2.0/repositories/${BBA} | grep -o '"ssh:[^ ,]\+' | xargs -L1 echo

Usage instructions: set your username to the BBA variable and execute.

BBA="bytefreaks"; curl --user ${BBA} https://api.bitbucket.org/2.0/repositories/${BBA} | grep -o '"ssh:[^ ,]\+' | xargs -L1 echo
Enter host password for user 'bytefreaks':
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  3834  100  3834    0     0   3543      0  0:00:01  0:00:01 --:--:--  3546
ssh://[email protected]/bytefreaks/bluetoothclicker.git
ssh://[email protected]/bytefreaks/watch.git

Inline replacement of all newlines in file with br tag

 in Bash  tagged bash / GNU/Linux / howtos / replace / sed by

In case you have some output you want to add it to an HTML document, you need to make some modifications to it to make it appear properly.

One of them would be to replace the newline characters with the <br> tag.

If you have GNU sed, you can use the -i option, which will do the replacement in place.

sed -i 's/$/<br>/' myTextFile.txt

Otherwise you will have to redirect to another file and rename it over the old one.

sed 's/$/<br>/' myTextFile.txt > myTextFile.txt.tmp && mv myTextFile.txt.tmp myTextFile.txt

If you want to perform this change on the results of another command (because you are redirecting it to an email client like mutt) you can use the following example

someCommand | sed 's/$/<br>/' | someOtherCommand