Bash

Linux: Delete all files that are older than X days

 in Bash / GNU/Linux  tagged bash / ctime / delete / exec / find / GNU/Linux / mtime by

The command find /data/ -type f -mtime +15 -exec rm -f '{}' \; is used to search and delete all the files in the “/data/” directory that have a modification time of more than 15 days old. The following is an explanation of each part of the command:

  1. “find /data/” – This specifies the directory that the search will start from; in this case, it’s the “/data/” directory.
  2. “-type f” – This option specifies that the search should be limited to files, not directories.
  3. “-mtime +15” – This option specifies that the files should be older than 15 days based on the modification time. The “+” sign indicates that we are looking for files older than 15 days.
  4. “-exec rm -f ‘{}’ \;” – This option is used to execute a command on the files found. The command “rm -f ‘{}'” is used to delete the files and the “{}” is a placeholder for the files that are found. The “” at the end of the line is used to escape the semicolon and avoid a syntax error.

The “find /data/ -type f -ctime +15 -exec rm -f ‘{}’ \;” command is similar to the above command, but it searches for files based on their creation time instead of modification time. The “ctime” option specifies that the search should be based on the file creation time instead of the modification time.

In conclusion, both commands are used to delete files in the “/data/” directory that are older than 15 days. Still, the difference is that the first command searches for files based on their modification time, while the second command searches for files based on their creation time.

Create an SSH tunnel for HTTP web server proxy

 in Bash / GNU/Linux  tagged http / proxy / ssh / ssh tunnel / tunnel / web proxy / web tunnel by

Once upon a time, in a kingdom of computers and networks, there lived a brave knight named “ssh”. He was known throughout the land for his bravery and cunning abilities to securely transport data between two distant lands.

One day, a young prince came to the knight with a request. The prince had a precious website that was housed in a remote castle, accessible only by a specific host known as “remotehost”. He wanted his people to be able to visit the website, but the path was treacherous and insecure.

The prince asked the knight if he could help him. The knight thought for a moment and then said, “Fear not, young prince! I can help you. I shall use my magical command ‘ssh -L 80:remotehost:80 user@myserver’ to create a secure pathway for your people to visit the website.”

The prince was overjoyed and asked the knight to explain how it worked.

“The ‘-L’ flag stands for Local Forwarding. It creates a tunnel between the local computer and the remote server, which we shall call ‘myserver’. This tunnel shall forward all requests from the local port 80 to the remote host ‘remotehost’ on port 80,” explained the knight.

“And ‘user@myserver’?”, asked the prince.

“Ah, yes. That is the credentials of the user that we shall use to log in to the remote server ‘myserver’. This shall ensure that the communication between your local computer and the remote host is secure and protected,” the knight replied with a nod.

The prince was grateful and thanked the knight for his help. The knight then used his magical command and created a secure pathway for the prince’s people to visit the website, which they did happily ever after.

And that, dear reader, is the story of the command “ssh -L 80:remotehost:80 user@myserver”.

ssh -L 80:remotehost:80 user@myserver;

The command ssh -L 80:remotehost:80 user@myserver is an example of using the ssh utility to create a secure shell connection to a remote server. The command also establishes a local port forward, which forwards all incoming traffic on the local port 80 to the remote host remotehost on port 80.

ssh (Secure Shell) is a protocol for securely accessing a remote computer. The basic usage of ssh is to log in to a remote server using a username and password or an SSH key. The ssh command allows you to securely log in to a remote server, execute commands on the remote server, and transfer files between your local computer and the remote server.

In this particular command, the -L flag is used to specify a local port forward. A local port forward is a way of forwarding traffic from a local port to a remote host and port. In this case, the traffic is being forwarded from the local port 80 to the remote host remotehost on port 80.

The user@myserver part of the command is the credentials that are used to log in to the remote server myserver. The user is the username and myserver is the hostname or IP address of the remote server. The combination of the username and remote server information allows ssh to securely log in to the remote server.

Once the secure shell connection has been established and the local port forward has been created, any traffic sent to the local port 80 will be forwarded to the remote host remotehost on port 80. This allows the local computer to access services on the remote host as if they were running on the local computer.

In summary, the ssh -L 80:remotehost:80 user@myserver command is an example of using the ssh utility to create a secure shell connection to a remote server and establish a local port forward. The local port forward allows the local computer to access services on the remote host as if they were running on the local computer.

ssh -L 80:remotehost:80 user@myserver;

Once the connection has been established using the command ssh -L 80:remotehost:80 user@myserver, you can access the website hosted on the remote host remotehost by browsing to http://localhost in your web browser.

Since the local port 80 has been forwarded to the remote host remotehost on port 80, all traffic sent to http://localhost will be forwarded to the remote host. This allows you to access the website hosted on the remote host as if it were running on your local computer.

Keep in mind that the secure shell connection created using the ssh command must be active and running in order to access the website hosted on the remote host. If the connection is closed or terminated, the website will no longer be accessible through the local port forward.

Using a CSV input file, find all documents that contain any of the items in a cell of a column

 in Bash / GNU/Linux  tagged bash / cut / exec / find / full text search / GNU/Linux / line / read by

The following code will use as input one column from a CSV file, and for each element in the column, it will perform a full-text search in a folder to find all files that contain that element.

#!/bin/bash

#Execution parameters
# 1 - the folder to look in for the element
# 2 - the input file that contains the search terms
# 3 - the column of interest
# 4 - the delimiter to use to find the column
# e.g. ./searchEachElement.sh ./2\ Print/ book.csv 5 ','

folder="$1";
input="$2";
column="$3";
delimiter="$4"

while read -r line; do
  needle=`echo $line | cut -d "$delimiter" -f "$column"`; 
  echo ">>> $needle"
  find "$folder" -type f -exec grep "$needle" -s -l '{}' \;
done < "$input";

More information on the full-text search can be found here.

Ubuntu Distribution Upgrade: Not enough free disk space

 in Bash / GNU/Linux  tagged apt / apt-get / bash / dpkg / GNU/Linux / purge / ubuntu / uname by

Recently, we tried to upgrade an Ubuntu 20.04 desktop to a 22.04. At some point in the update, we got the following error:

The upgrade has aborted. The upgrade needs a total of 10,6 G free space on disk '/'. Please free at least an additional 8201 M of disk space on '/'. Empty your trash and remove temporary packages of former installations using 'sudo apt-get clean'. The upgrade has aborted. The upgrade needs a total of 430 M free space on disk '/boot'. Please free at least an additional 38,4 M of disk space on '/boot'. You can remove old kernels using 'sudo apt autoremove' and you could also set COMPRESS=xz in /etc/initramfs-tools/initramfs.conf to reduce the size of your initramfs.

First of all, we tried to use the command apt autoremove to clear up some space, which unfortunately was not enough.

sudo apt autoremove;

Then to clear up some space, we needed to find remnant memories of older versions of the Kernel. To do so, we used the following command. The following command finds the current version of the kernel and shows the user the remaining packages that do not reflect the active kernel.

dpkg -l 'linux-*' | sed '/^ii/!d;/'"$(uname -r | sed "s/\(.*\)-\([^0-9]\+\)/\1/")"'/d;s/^[^ ]* [^ ]* \([^ ]*\).*/\1/;/[0-9]/!d;/^linux-\(headers\|image\)/!d';

Then we removed all the headers and the images that we did not need using the command apt-get purge.

$ dpkg -l 'linux-*' | sed '/^ii/!d;/'"$(uname -r | sed "s/\(.*\)-\([^0-9]\+\)/\1/")"'/d;s/^[^ ]* [^ ]* \([^ ]*\).*/\1/;/[0-9]/!d;/^linux-\(headers\|image\)/!d'
linux-headers-5.13.0-52-generic
linux-headers-5.15.0-46-generic
linux-headers-generic-hwe-20.04
linux-image-5.15.0-46-generic
linux-image-generic-hwe-20.04

$ sudo apt-get -y purge linux-headers-5.13.0-52-generic linux-headers-5.15.0-46-generic linux-image-5.15.0-46-generic;

Doing so was enough to clear up the space that was needed for the upgrade to continue.