Assigning auto-increment IDs to empty fields in a KML/XML file

Recently we were processing some KML files using OpenLayers and at some point we realised that some place-marks were not appearing on the map. After inspecting the debug console and the files more carefully we understood that OpenLayers did not like empty placemark IDs.

To mitigate the problem we wrote the following AWK script that will go over all lines in the KML/XML file, find the empty id fields (id="") and assign them with an auto-increment value. A note here, initially we just replaced all empty IDs with the same value but it seems that OpenLayers does not treat kindly conflicts on IDs and thus we had to go with an auto-increment solution.

# Assigning auto-increment IDs to the placemarkers as openlayers does not show conflicting-ID elements.
awk -i inplace '{
  for(x=1;x<=NF;x++) {
    if($x~/id=""/) {
      sub(/id=""/,"id=\"" (++i) "\"")
    }
  }
}1' "$output_path/$file_name";

Side notes

In case you already have some IDs defined, you would have to make your code a bit more complex… You would first need to find all filled IDs and then you would have two options:

  • empty them and execute the above script
  • or register them and make sure the script does not create conflicting IDs either by starting the variable i from a number greater than the biggest registered ID or making it even harder by filling in the gaps between the already registered IDs..
Advertisements

Stop Windows 10 Updates

Recently, we were working on a Windows 10 machine on a metered connection. Even though we are not fans of blocking updates, this time we had to stop the updates as they were sucking the data package dry. To do so we executed the commands of the block below in a command prompt with administrative rights.

To start a Command Prompt (cmd) with administrative rights we pressed Windows+X that showed the Quick Access menu, from the menu we clicked on Command Prompt (Admin). After that we got prompted by User Account Control window if it was OK to allow this application to make changes, where we clicked Yes.

In the new Command Prompt window we executed the following 3 commands that kill all services immediately related with the updates

net stop wuauserv
net stop bits
net stop dosvc

Explanation

  • net stop wuauserv stops the Windows Update service.
  • net stop bits stops the Background Intelligent Transfer Service service
  • net stop dosvc stops the Delivery Optimization service

Revert action and Start Windows 10 updates

To resume (actually restart them since we stopped them) the Windows 10 updates you can either restart the machine or in a command prompt with administrative rights execute the following:

net start wuauserc
net start bits
net start dosvc

Add a new line whenever the first column changes

Recently we were processing some results from an SQL query on the command line, we were grouping the results based on the first column of the query results and we wanted to add an empty line between each group of data.

Instead of messing with SQL specifics, we decided to use awk to finish the task which resulted in the following code:

awk -F '|' -v column=1 'NR>1 && $column != previous { print "" } { previous = $column } { print $0 }'

Explanation:

  • -F fs or --field-separator fs: This option allowed us to choose the input field separator, in other words the character that should be used to split our text into columns. We used the value | because that character is the default column delimiter for sqlite.
  • -v var=val or --assign var=val: We used the -v option to define a variable to be used later on in the script. The value 1 was assigned to the variable column before execution of the program began and it was available event to the BEGIN rule of the AWK program. We did this to make the code a bit more modular, we could have just hardcoded the number in.
  • NR>1 && $column != previous { print "" } : Here we defined an if statement that checks two options: First we make sure that we are not on the first line of the input by using the NR>1 (and thus avoid creating an empty line which will be the first line of the output).
    Second, we check that the last value we had for the column of interest did change since last time. (We still did not define the value of previous, it is on the next step). When both statements are true (we are not on the first row and the value of the column in the current row is different than the value of the column in the previous row) it will print out an empty line.
  • { previous = $column }: This part is executed on ALL lines (even the first one) no matter what the values are. What this line does is to translate the value of the column variable from being a number (the index of the column that we are interested in) into the actual value that the column has at that specific line. That value is then copied to the previous variable to allow us to perform the check in the previous point once we move to the next line.
  • { print $0 }: Finally, this part is also executed on all lines and it instructs awk to print the input row whole and as is. This whole part could be replaced by a true value like the value 1. In awk as you see in this example, you define a series of operations. Each operation is constructed by a pattern to be matched and an action. Each pattern is evaluated for each input line, and in the cases where the pattern matches, the action is executed. The user can choose to omit either the pattern or the action for any operation. When a pattern is omitted, the action is executed on every line. When the action is omitted, then awk will execute { print $0 }. So, by adding a true value on its own it will be translated as on each line execute { print $0 } which prints the whole row as is.

Example

1|1|0.0564904019731175
1|2|0.103176086258974
1|3|0.12910406904073
1|4|0.188592489201024
1|5|0.169676224898487
1|6|0.164690820027741
1|7|0.128458728519047
1|8|0.18549773544014
1|9|0.155677575617836
1|10|0.153941343314285
2|1|0.217221158956016
2|2|0.23390973064067
2|3|0.180231657220626
2|4|0.257673927303071
2|5|0.261393785194329
2|6|0.273441488895552
2|7|0.242815632929545
2|8|0.262269697286057
2|9|0.256054399760891
2|10|0.262613705138411
3|1|0.378589461360716
3|2|0.33008177312116
3|3|0.380973166776554
3|4|0.340431190160728
3|5|0.38189416214207
3|6|0.364842933594872
3|7|0.372958396398964
3|8|0.350010176652464
3|9|0.355815612501188
3|10|0.380553180349294

Will become

1|1|0.0564904019731175
1|2|0.103176086258974
1|3|0.12910406904073
1|4|0.188592489201024
1|5|0.169676224898487
1|6|0.164690820027741
1|7|0.128458728519047
1|8|0.18549773544014
1|9|0.155677575617836
1|10|0.153941343314285

2|1|0.217221158956016
2|2|0.23390973064067
2|3|0.180231657220626
2|4|0.257673927303071
2|5|0.261393785194329
2|6|0.273441488895552
2|7|0.242815632929545
2|8|0.262269697286057
2|9|0.256054399760891
2|10|0.262613705138411

3|1|0.378589461360716
3|2|0.33008177312116
3|3|0.380973166776554
3|4|0.340431190160728
3|5|0.38189416214207
3|6|0.364842933594872
3|7|0.372958396398964
3|8|0.350010176652464
3|9|0.355815612501188
3|10|0.380553180349294

WordPress: How to disable a plugin on all pages except for a specific one

A few days ago we were struggling to find a way to limit the amount of plugins that load at any point on a WordPress website. We noticed that several plugins enqueue their scripts and their styles in all requests to the website even if they are actually used on a single page only. This issue was important to address as it was making the whole server slower by giving it extra requests from the client that would never provide any actual benefit to the user.

Initially, we tried to selectively enable those plugins on their respective pages but we did not get it right and things would load out of order and break. Instead of following the ‘enable when needed‘ methodology we decided to follow the ‘disable unless needed‘ methodology which seemed simpler at the time.

Our changes involved in adding the following code in the functions.php file of our child theme.

//Register a filter at the correct event
add_filter( 'option_active_plugins', 'bf_plugin_control' );

function bf_plugin_control($plugins) {
  // If we are in the admin area do not touch anything
  if (is_admin()) {
    return $plugins;
  }
  
  // Check if we are at the expected page, if not remove the plugin from the active plugins list
  if(is_page("csv-to-kml-cell-site-map") === FALSE){ 
    // Finding the plugin in the active plugins list
    $key = array_search( 'csv-kml/index.php' , $plugins );
    if ( false !== $key ) {
      // Removing the plugin and dequeuing its scripts
      unset( $plugins[$key] );
      wp_dequeue_script( 'bf_csv_kml_script' );
    }
  }

  if(is_page("random-password-generator") === FALSE){ 
    $key = array_search( 'bytefreaks-password-generator/passwordGenerator.php' , $plugins );
    if ( false !== $key ) {
      unset( $plugins[$key] );
    }
  }
  
  if(is_page("xml-tree-visualizer") === FALSE){ 
    $key = array_search( 'xmltree/xml-tree.php' , $plugins );
    if ( false !== $key ) {
      unset( $plugins[$key] );
      wp_dequeue_script( 'bf_xml_namespace' );
      wp_dequeue_style( 'bf_xml_namespace' );
    }
  }

  return $plugins;
}

One day, we will clean the above code to make it tidy and reusable.. one day, that day is not today.

What the code above does is the following:

  • Using is_admin it checks if the Dashboard or the administration panel is attempting to be displayed, in that case it does not do any changes.
  • With is_page, it will additionally check if the parameter is for one of the pages specified and thus disable the plugin if the check fails.
  • PHP command array_search, will see if our plugin file is expected to be executed (all files in $plugins are the plugin files that are expected to be executed) .
  • wp_dequeue_script and wp_dequeue_style remove the previously enqueued scripts and styles of the plugin as long as you know the handles (or namespaces of the enqueued items).
    To get the handles (namespaces) we went through the plugin codes and found all instances of wp_enqueue_script and wp_enqueue_style.
    Please note that several small plugins do not have additional items in queue so no further action is needed.


Bash: Problem with reading files with spaces in the name using a for loop

Recently we were working on a bash script that was supposed to find and process some files that matched certain criteria. The script would process the files one by one and the criteria would be matched using the find command. To implement our solution, we returned the results of the find back to the for loop in an attempt to keep it simple and human readable.

Our original code was the following:
(do not use it, see explanation below)

for file in `find $search_path -type f -name '*.kml'`; do
  # Formatting KML file to be human friendly.
  xmllint --format "$file" > "$output_path/$file";
done

Soon we realized that we had a very nasty bug, the way we formatted the command it would break filenames that had spaces in them into multiple for loop entries and thus we would get incorrect filenames back to process.

To solve this issue we needed a way to force our loop to read the results of find one line at a time instead of one word at a time. The solution we used in the end was fairly different than the original code as it had the following significant changes:

  • the results of the find command were piped into the loop
  • the loop was not longer a for loop and a while loop was used instead
  • it used the read command that reads one line at a time to fill in the filename variable
    (the -r parameter does not allow backslashes to escape any characters)

Solution

find $search_path -type f -name '*.kml' | 
while read -r file; do
  # Formatting KML file to be human friendly.
  xmllint --format "$file" > "$output_path/$file";
done


Neural Style Transfer – Mosaic Style Source

We are looking for the source of the mosaic style for neural style transfer, we came across the following image that was apparently used by several people as input but we could not identify the source of it.

The photo depicts a lady holding a flower on stained glass.

(c) copyright 2006, Blender Foundation / Netherlands Media Art Institute / www.elephantsdream.org
This an audio-less re-production of the «Elephants Dream» by Blender Foundation after it was parsed using a neural network that attempts to transfer the same style as a photo that depicts a lady holding a flower on stained glass (https://bytefreaks.net/photography/neural-style-transfer-mosaic-style-source). We do not know the origin of this art yet…
Audio will be added at a later stage.

Neural Style Transfer – Feathers Style Source

While looking for the source of the Feathers style for neural style transfer we came across the following image that was apparently used by several people as input.

After some special Google-Fu we were able to find a page that posts the above watercolor paint with feathers leaves and petals which states that the painter is Kathryn Corlett.

This an audio-less re-production of the «Elephants Dream» by Blender Foundation after it was parsed using a neural network that attempts to transfer the same style as what seems to be the work named «Feathers Leaves and Petals» by Kathryn Corlett to each frame of the video.
Audio will be added at a later stage.

Neural Style Transfer – Candy Style Source

While looking for the source of the Candy style for neural style transfer we came across the following image that was apparently used by several people as input.

We tried to identify the painter of the above piece but we were not able to pinpoint the exact painting. What we found was a painting called «June Tree» by Natasha Wescoat which has looks extremely similar to this so we assume that the input must be a paint by mrs. Wescoat.

This an audio-less re-production of the «Elephants Dream» by Blender Foundation after it was parsed using a neural network that attempts to transfer the same style as what seems to be a variation of the «June Tree» by Natasha Wescoat to each frame of the video.
Audio will be added at a later stage.