GNU/Linux

Compiling the latest version of YubiKey Personalization Tool on Ubuntu 18.04 LTS

 in GNU/Linux  tagged 2FA / personalization tool / ubuntu / ykpersonalize / yubico / yubikey / yubikey-personalization-gui by

Recently, we were got our hands on some YubiKeys, and we decided to use them to create a Two Factor Authentication System (2FA) for the fun of it! We had at our disposal an updated Ubuntu 18.04 LTS so we installed the personalization tools from the official repositories in order to modify the behavior and configure the YubiKeys.

To our disappointment, when we used ykpersonalize and yubikey-personalization-gui we would get an error that the firmware of the YubiKey was unknown…
At the time, the installation packages from the official Ubuntu repositories had version 3.1.24 for the application version and 1.18.0 for the library version.

We noticed that on the YubiKey Personalization Tools page there were newer versions of both the application and the library. Specifically at the time the Application version was 3.1.26 and the Library Version was 1.19.0. Since both were newer than the versions in the repositories we decided to build them and see if they work right with our YubiKeys.

The instructions in the respective installers, were not 100% complete and the installations failed by blindly following them. To actually make the installations work, we installed the following dependencies and tools before compiling:

sudo apt update -y;
sudo apt upgrade -y;
sudo apt install build-essential -y;
sudo apt-get install pkg-config git autoconf libtool asciidoc-base -y;

After installing the above packages the rest of the installation went smoothly.

Installing the command line tools and the library

cd ~; # or any other folder of your choice
sudo apt-get install libykpers-1-dev libyubikey-dev libusb-1.0-0-dev libjson-c-dev -y;
git clone https://github.com/Yubico/yubikey-personalization.git;
cd yubikey-personalization;
autoreconf --install;
./configure;
sudo make check install;

Installing the Qt based Cross-Platform YubiKey Personalization Tool

cd ~; # or any other folder of your choice
sudo apt-get install qt4-qmake libqt4-dev -y;
git clone https://github.com/Yubico/yubikey-personalization-gui.git;
cd yubikey-personalization-gui;
qmake && make;

Advertisements

Assigning auto-increment IDs to empty fields in a KML/XML file

 in Bash / GNU/Linux  tagged auto increment / awk / kml / openlayers / xml by

Recently we were processing some KML files using OpenLayers and at some point we realised that some place-marks were not appearing on the map. After inspecting the debug console and the files more carefully we understood that OpenLayers did not like empty placemark IDs.

To mitigate the problem we wrote the following AWK script that will go over all lines in the KML/XML file, find the empty id fields (id="") and assign them with an auto-increment value. A note here, initially we just replaced all empty IDs with the same value but it seems that OpenLayers does not treat kindly conflicts on IDs and thus we had to go with an auto-increment solution.

# Assigning auto-increment IDs to the placemarkers as openlayers does not show conflicting-ID elements.
awk -i inplace '{
  for(x=1;x<=NF;x++) {
    if($x~/id=""/) {
      sub(/id=""/,"id=\"" (++i) "\"")
    }
  }
}1' "$output_path/$file_name";

Side notes

In case you already have some IDs defined, you would have to make your code a bit more complex… You would first need to find all filled IDs and then you would have two options:

  • empty them and execute the above script
  • or register them and make sure the script does not create conflicting IDs either by starting the variable i from a number greater than the biggest registered ID or making it even harder by filling in the gaps between the already registered IDs..

Add a new line whenever the first column changes 1

 in Bash / GNU/Linux  tagged awk / bash / new line by

Recently we were processing some results from an SQL query on the command line, we were grouping the results based on the first column of the query results and we wanted to add an empty line between each group of data.

Instead of messing with SQL specifics, we decided to use awk to finish the task which resulted in the following code:

awk -F '|' -v column=1 'NR>1 && $column != previous { print "" } { previous = $column } { print $0 }'

Explanation:

  • -F fs or --field-separator fs: This option allowed us to choose the input field separator, in other words the character that should be used to split our text into columns. We used the value | because that character is the default column delimiter for sqlite.
  • -v var=val or --assign var=val: We used the -v option to define a variable to be used later on in the script. The value 1 was assigned to the variable column before execution of the program began and it was available event to the BEGIN rule of the AWK program. We did this to make the code a bit more modular, we could have just hardcoded the number in.
  • NR>1 && $column != previous { print "" } : Here we defined an if statement that checks two options: First we make sure that we are not on the first line of the input by using the NR>1 (and thus avoid creating an empty line which will be the first line of the output).
    Second, we check that the last value we had for the column of interest did change since last time. (We still did not define the value of previous, it is on the next step). When both statements are true (we are not on the first row and the value of the column in the current row is different than the value of the column in the previous row) it will print out an empty line.
  • { previous = $column }: This part is executed on ALL lines (even the first one) no matter what the values are. What this line does is to translate the value of the column variable from being a number (the index of the column that we are interested in) into the actual value that the column has at that specific line. That value is then copied to the previous variable to allow us to perform the check in the previous point once we move to the next line.
  • { print $0 }: Finally, this part is also executed on all lines and it instructs awk to print the input row whole and as is. This whole part could be replaced by a true value like the value 1. In awk as you see in this example, you define a series of operations. Each operation is constructed by a pattern to be matched and an action. Each pattern is evaluated for each input line, and in the cases where the pattern matches, the action is executed. The user can choose to omit either the pattern or the action for any operation. When a pattern is omitted, the action is executed on every line. When the action is omitted, then awk will execute { print $0 }. So, by adding a true value on its own it will be translated as on each line execute { print $0 } which prints the whole row as is.

Example

1|1|0.0564904019731175
1|2|0.103176086258974
1|3|0.12910406904073
1|4|0.188592489201024
1|5|0.169676224898487
1|6|0.164690820027741
1|7|0.128458728519047
1|8|0.18549773544014
1|9|0.155677575617836
1|10|0.153941343314285
2|1|0.217221158956016
2|2|0.23390973064067
2|3|0.180231657220626
2|4|0.257673927303071
2|5|0.261393785194329
2|6|0.273441488895552
2|7|0.242815632929545
2|8|0.262269697286057
2|9|0.256054399760891
2|10|0.262613705138411
3|1|0.378589461360716
3|2|0.33008177312116
3|3|0.380973166776554
3|4|0.340431190160728
3|5|0.38189416214207
3|6|0.364842933594872
3|7|0.372958396398964
3|8|0.350010176652464
3|9|0.355815612501188
3|10|0.380553180349294

Will become

1|1|0.0564904019731175
1|2|0.103176086258974
1|3|0.12910406904073
1|4|0.188592489201024
1|5|0.169676224898487
1|6|0.164690820027741
1|7|0.128458728519047
1|8|0.18549773544014
1|9|0.155677575617836
1|10|0.153941343314285

2|1|0.217221158956016
2|2|0.23390973064067
2|3|0.180231657220626
2|4|0.257673927303071
2|5|0.261393785194329
2|6|0.273441488895552
2|7|0.242815632929545
2|8|0.262269697286057
2|9|0.256054399760891
2|10|0.262613705138411

3|1|0.378589461360716
3|2|0.33008177312116
3|3|0.380973166776554
3|4|0.340431190160728
3|5|0.38189416214207
3|6|0.364842933594872
3|7|0.372958396398964
3|8|0.350010176652464
3|9|0.355815612501188
3|10|0.380553180349294

Bash: Problem with reading files with spaces in the name using a for loop

 in Bash  tagged bash / find / for / for loop by

Recently we were working on a bash script that was supposed to find and process some files that matched certain criteria. The script would process the files one by one and the criteria would be matched using the find command. To implement our solution, we returned the results of the find back to the for loop in an attempt to keep it simple and human readable.

Our original code was the following:
(do not use it, see explanation below)

for file in `find $search_path -type f -name '*.kml'`; do
  # Formatting KML file to be human friendly.
  xmllint --format "$file" > "$output_path/$file";
done

Soon we realized that we had a very nasty bug, the way we formatted the command it would break filenames that had spaces in them into multiple for loop entries and thus we would get incorrect filenames back to process.

To solve this issue we needed a way to force our loop to read the results of find one line at a time instead of one word at a time. The solution we used in the end was fairly different than the original code as it had the following significant changes:

  • the results of the find command were piped into the loop
  • the loop was not longer a for loop and a while loop was used instead
  • it used the read command that reads one line at a time to fill in the filename variable
    (the -r parameter does not allow backslashes to escape any characters)

Solution

find $search_path -type f -name '*.kml' | 
while read -r file; do
  # Formatting KML file to be human friendly.
  xmllint --format "$file" > "$output_path/$file";
done

Create an encrypted 7zip archive with encrypted header as well (no filenames are visible)

 in Bash  tagged 7z / 7zip / encryption / filenames / headers / password / protection by

In case you come to a scenario where you need to encrypt, password protect the contents of a 7zip archive and make sure that not even the filenames of the contents are visible, 7zip has your back! As you can see in the following example you can implement the above requirements very easily.

7z a -p"pbVfdPs27Dc" -mhe hello.7z file1.bin file2.doc files.*

The structure of the above 7z command is the following:

#Based on: 7z <command> [<switches>...] <archive_name> [<file_names>...]
7z a -p"Some [email protected]" -mhe <archive_name> [<file_names>...]

To break it down, it goes like this:

  • We used the <command> a, which instructs the tool to add the listed files to the listed archive (if the archive does not exist, it will create it).
  • The <switch> -p, allows you to set the password for the archive.
  • The second <switch> -mhe (or -mhe=on) it enables data and header archive encryption.
    In case you cannot find this switch at the manual, check the examples in the man page (This command works on GNU/Linux, it was tested on Fedora).