Bash

Create an SSH tunnel for HTTP web server proxy

 in Bash / GNU/Linux  tagged http / proxy / ssh / ssh tunnel / tunnel / web proxy / web tunnel by

Once upon a time, in a kingdom of computers and networks, there lived a brave knight named “ssh”. He was known throughout the land for his bravery and cunning abilities to securely transport data between two distant lands.

One day, a young prince came to the knight with a request. The prince had a precious website that was housed in a remote castle, accessible only by a specific host known as “remotehost”. He wanted his people to be able to visit the website, but the path was treacherous and insecure.

The prince asked the knight if he could help him. The knight thought for a moment and then said, “Fear not, young prince! I can help you. I shall use my magical command ‘ssh -L 80:remotehost:80 user@myserver’ to create a secure pathway for your people to visit the website.”

The prince was overjoyed and asked the knight to explain how it worked.

“The ‘-L’ flag stands for Local Forwarding. It creates a tunnel between the local computer and the remote server, which we shall call ‘myserver’. This tunnel shall forward all requests from the local port 80 to the remote host ‘remotehost’ on port 80,” explained the knight.

“And ‘user@myserver’?”, asked the prince.

“Ah, yes. That is the credentials of the user that we shall use to log in to the remote server ‘myserver’. This shall ensure that the communication between your local computer and the remote host is secure and protected,” the knight replied with a nod.

The prince was grateful and thanked the knight for his help. The knight then used his magical command and created a secure pathway for the prince’s people to visit the website, which they did happily ever after.

And that, dear reader, is the story of the command “ssh -L 80:remotehost:80 user@myserver”.

ssh -L 80:remotehost:80 user@myserver;

The command ssh -L 80:remotehost:80 user@myserver is an example of using the ssh utility to create a secure shell connection to a remote server. The command also establishes a local port forward, which forwards all incoming traffic on the local port 80 to the remote host remotehost on port 80.

ssh (Secure Shell) is a protocol for securely accessing a remote computer. The basic usage of ssh is to log in to a remote server using a username and password or an SSH key. The ssh command allows you to securely log in to a remote server, execute commands on the remote server, and transfer files between your local computer and the remote server.

In this particular command, the -L flag is used to specify a local port forward. A local port forward is a way of forwarding traffic from a local port to a remote host and port. In this case, the traffic is being forwarded from the local port 80 to the remote host remotehost on port 80.

The user@myserver part of the command is the credentials that are used to log in to the remote server myserver. The user is the username and myserver is the hostname or IP address of the remote server. The combination of the username and remote server information allows ssh to securely log in to the remote server.

Once the secure shell connection has been established and the local port forward has been created, any traffic sent to the local port 80 will be forwarded to the remote host remotehost on port 80. This allows the local computer to access services on the remote host as if they were running on the local computer.

In summary, the ssh -L 80:remotehost:80 user@myserver command is an example of using the ssh utility to create a secure shell connection to a remote server and establish a local port forward. The local port forward allows the local computer to access services on the remote host as if they were running on the local computer.

ssh -L 80:remotehost:80 user@myserver;

Once the connection has been established using the command ssh -L 80:remotehost:80 user@myserver, you can access the website hosted on the remote host remotehost by browsing to http://localhost in your web browser.

Since the local port 80 has been forwarded to the remote host remotehost on port 80, all traffic sent to http://localhost will be forwarded to the remote host. This allows you to access the website hosted on the remote host as if it were running on your local computer.

Keep in mind that the secure shell connection created using the ssh command must be active and running in order to access the website hosted on the remote host. If the connection is closed or terminated, the website will no longer be accessible through the local port forward.

Using a CSV input file, find all documents that contain any of the items in a cell of a column

 in Bash / GNU/Linux  tagged bash / cut / exec / find / full text search / GNU/Linux / line / read by

The following code will use as input one column from a CSV file, and for each element in the column, it will perform a full-text search in a folder to find all files that contain that element.

#!/bin/bash

#Execution parameters
# 1 - the folder to look in for the element
# 2 - the input file that contains the search terms
# 3 - the column of interest
# 4 - the delimiter to use to find the column
# e.g. ./searchEachElement.sh ./2\ Print/ book.csv 5 ','

folder="$1";
input="$2";
column="$3";
delimiter="$4"

while read -r line; do
  needle=`echo $line | cut -d "$delimiter" -f "$column"`; 
  echo ">>> $needle"
  find "$folder" -type f -exec grep "$needle" -s -l '{}' \;
done < "$input";

More information on the full-text search can be found here.

Ubuntu Distribution Upgrade: Not enough free disk space

 in Bash / GNU/Linux  tagged apt / apt-get / bash / dpkg / GNU/Linux / purge / ubuntu / uname by

Recently, we tried to upgrade an Ubuntu 20.04 desktop to a 22.04. At some point in the update, we got the following error:

The upgrade has aborted. The upgrade needs a total of 10,6 G free space on disk '/'. Please free at least an additional 8201 M of disk space on '/'. Empty your trash and remove temporary packages of former installations using 'sudo apt-get clean'. The upgrade has aborted. The upgrade needs a total of 430 M free space on disk '/boot'. Please free at least an additional 38,4 M of disk space on '/boot'. You can remove old kernels using 'sudo apt autoremove' and you could also set COMPRESS=xz in /etc/initramfs-tools/initramfs.conf to reduce the size of your initramfs.

First of all, we tried to use the command apt autoremove to clear up some space, which unfortunately was not enough.

sudo apt autoremove;

Then to clear up some space, we needed to find remnant memories of older versions of the Kernel. To do so, we used the following command. The following command finds the current version of the kernel and shows the user the remaining packages that do not reflect the active kernel.

dpkg -l 'linux-*' | sed '/^ii/!d;/'"$(uname -r | sed "s/\(.*\)-\([^0-9]\+\)/\1/")"'/d;s/^[^ ]* [^ ]* \([^ ]*\).*/\1/;/[0-9]/!d;/^linux-\(headers\|image\)/!d';

Then we removed all the headers and the images that we did not need using the command apt-get purge.

$ dpkg -l 'linux-*' | sed '/^ii/!d;/'"$(uname -r | sed "s/\(.*\)-\([^0-9]\+\)/\1/")"'/d;s/^[^ ]* [^ ]* \([^ ]*\).*/\1/;/[0-9]/!d;/^linux-\(headers\|image\)/!d'
linux-headers-5.13.0-52-generic
linux-headers-5.15.0-46-generic
linux-headers-generic-hwe-20.04
linux-image-5.15.0-46-generic
linux-image-generic-hwe-20.04

$ sudo apt-get -y purge linux-headers-5.13.0-52-generic linux-headers-5.15.0-46-generic linux-image-5.15.0-46-generic;

Doing so was enough to clear up the space that was needed for the upgrade to continue.

GNU Linux/Bash: A function that splits a word in half

 in Bash / GNU/Linux  tagged bash / fedora / function / GNU/Linux / howtos / line / split / string / text / ubuntu by

The following function takes one argument – a text file.
The text file should contain one word on each line.
The function reads the text file (argument) line by line.
Then it checks if the line has one word; if this is true, it splits the word in half.
Finally, it prints the two new words with a space between them.

#!/bin/bash

splitWordsInHalf () {
  # This function takes one argument - a text file.
  # The text file contains one word on each line.
  # It reads the text file (argument) line by line.
  # Then it checks if the line contains one word, if this is true, it splits the word in half.
  # Finally, it prints the two new words with a space between them
  while read line
  do
    words=( $line )
    if [ ${#words[@]} == 1 ]
    then
      echo ${line:0:${#line}/2} ${line:${#line}/2}
    fi
  done < $1
}

splitWordsInHalf input.txt

Example

Using the following input file:

banana
apple
ball
car
door

We will get the following output when we execute splitWordsInHalf input.txt:

ban ana
ap ple
ba ll
c ar
do or

Notes

The following parts of the code are in charge of looping on the data of the incoming file. The parameter (the input file) given to the function is translated into the variable $1. The while loop gets one line of text on each iteration and assigns the text to the variable that is named line. You could have chosen any other name that suits you instead of the word line.

splitWordsInHalf () {
  while read line
  do
    ...
  done < $1
}

The next part of the code (words=( $line )) converts the string value that is contained in the line variable into an array of words, and it assigns that array to the variable named words. Then, it counts the number of elements in the array (the number of words in the line) using the following ${#words[@]} and it checks that there is only one item.

words=( $line )
if [ ${#words[@]} == 1 ]
then
  ...
fi

The following line will print two strings. The first string is a sub-string of variable line that is composed by the first half of the value. The second sub-string is the second half of the value contained in the variable named line.

echo ${line:0:${#line}/2} ${line:${#line}/2}

The ${#line} will return the length of the string contained in the variable.

The structure ${VARIABLE:START:END} defines the slice of the string that we want returned.