find


Bash: Problem with reading files with spaces in the name using a for loop

Recently we were working on a bash script that was supposed to find and process some files that matched certain criteria. The script would process the files one by one and the criteria would be matched using the find command. To implement our solution, we returned the results of the find back to the for loop in an attempt to keep it simple and human readable.

Our original code was the following:
(do not use it, see explanation below)

for file in `find $search_path -type f -name '*.kml'`; do
  # Formatting KML file to be human friendly.
  xmllint --format "$file" > "$output_path/$file";
done

Soon we realized that we had a very nasty bug, the way we formatted the command it would break filenames that had spaces in them into multiple for loop entries and thus we would get incorrect filenames back to process.

To solve this issue we needed a way to force our loop to read the results of find one line at a time instead of one word at a time. The solution we used in the end was fairly different than the original code as it had the following significant changes:

  • the results of the find command were piped into the loop
  • the loop was not longer a for loop and a while loop was used instead
  • it used the read command that reads one line at a time to fill in the filename variable
    (the -r parameter does not allow backslashes to escape any characters)

Solution

find $search_path -type f -name '*.kml' | 
while read -r file; do
  # Formatting KML file to be human friendly.
  xmllint --format "$file" > "$output_path/$file";
done

Advertisements

LibreOffice Calc: Get workbook path only

Get workbook path only

For Linux and Mac

=LEFT(
  CELL("filename"),
  FIND(
    CHAR(1),
    SUBSTITUTE(
      CELL("filename"),
      "/",
      CHAR(1),
      LEN(CELL("filename")) - LEN(
        SUBSTITUTE(
          CELL("filename"),
          "/",
          ""
        )
      )
    )
  ) -1
)&"'"

For Windows

=LEFT(
  CELL("filename"),
  FIND(
    CHAR(1),
    SUBSTITUTE(
      CELL("filename"),
      "\",
      CHAR(1),
      LEN(CELL("filename")) - LEN(
        SUBSTITUTE(
          CELL("filename"),
          "\",
          ""
        )
      )
    )
  ) -1
)&"'"

Bonus

If you want to remove the ' characters as well that are around the filename and path use the following  solution, this will allow you to create hyperlinks for the folders as well

=REPLACE(
  LEFT(
    CELL("filename"),
    FIND(CHAR(1),
      SUBSTITUTE(
        CELL("filename"),
        "/",
        CHAR(1),
        LEN(CELL("filename")) - LEN(
          SUBSTITUTE(
            CELL("filename"),
            "/",
            ""
          )
        )
      )
    ) -1
  )
  ,1,1,""
)

To create hyperlink as well, use the following

=HYPERLINK(
  REPLACE(
    LEFT(
      CELL("filename"),
      FIND(CHAR(1),
        SUBSTITUTE(
          CELL("filename"),
          "/",
          CHAR(1),
          LEN(CELL("filename")) - LEN(
            SUBSTITUTE(
              CELL("filename"),
              "/",
              ""
            )
          )
        )
      ) -1
    )
    ,1,1,""
  )
)

 

 


LibreOffice Calc: Get workbook full path with filename

Get workbook full path with filename

=LEFT(CELL("filename"),FIND("#",CELL("filename"))-1)

Explanation

If you want to get the workbook filename and path, you can do so with a formula that uses the LEFT and the FIND function.

The CELL("filename") function is used to get the full file name and path along with current sheet link:

The result will look like this:

'file:///home/user/path/documentName.ods'#$sheetName

Using the FIND function we find the position of the delimiter # and then using the LEFT function, we extract just the full directory path with the filename. We know the number of characters to extract by locating the the position of the hash # character with FIND and then subtracting 1 (to exclude itself as well).

Extra

If you want to remove the ' characters as well that are around the filename and path use the following solution

=REPLACE(
  LEFT(
    CELL("filename"),
    FIND(
      "#",
      CELL("filename")
    )-2
  ), 1, 1,""
)

 


LibreOffice Calc: Get workbook filename only

Get workbook filename only

For Linux and Mac

=TRIM(
  RIGHT(
    SUBSTITUTE(
      LEFT(
        CELL("filename"),
        FIND("#", CELL("filename")) -2
      ),
      "/",
      REPT(" ", LEN(CELL("filename")))
    ),
  LEN(CELL("filename"))
  )
)

For Windows

=TRIM(
  RIGHT(
    SUBSTITUTE(
      LEFT(
        CELL("filename"),
        FIND("#", CELL("filename")) -2
      ),
      "\",
      REPT(" ", LEN(CELL("filename")))
    ),
  LEN(CELL("filename"))
  )
)

The only change is changing the delimiter from / to \.


Viber Desktop on Fedora 27 64bit: Viber cannot connect to the internet (OpenSSL error) 1

After updating our GNU/Linux distribution from Fedora 26 to Fedora 27, and maybe updating Viber Desktop client as well, Viber stopped working. To be more precise it could not connect to the internet any more. (Note: we used the RPM package of Viber Desktop to install Viber on our Fedora system).

Solution for the OpenSSL issue – Create a symbolic link to the missing library in the libraries folder of Viber

#Make sure all necessary packages are installed
sudo dnf install openssl openssl-devel;
#Create a symbolic link from the system installed libssl shared object to the Viber installation folder
sudo ln -s /usr/lib64/libssl.so.10 /opt/viber/lib/libssl.so;

Explanation and details – Optional Reading

To find the installation folder of Viber, we used the following command:

find / -iname viber 2>/dev/null;

that led us to the installation location of Viber which was /opt/viber/Viber.

Executing the binary in a terminal gave us the following errors

$ /opt/viber/Viber
 Qt WebEngine ICU data not found at /opt/viber/resources. Trying parent directory...
 Qt WebEngine resources not found at /opt/viber/resources. Trying parent directory...
 Qt WebEngine ICU data not found at /opt/viber/resources. Trying parent directory...
 Qt WebEngine resources not found at /opt/viber/resources. Trying parent directory...
 qt.network.ssl: QSslSocket: cannot resolve CRYPTO_num_locks
 qt.network.ssl: QSslSocket: cannot resolve CRYPTO_set_id_callback
 qt.network.ssl: QSslSocket: cannot resolve CRYPTO_set_locking_callback
 qt.network.ssl: QSslSocket: cannot resolve ERR_free_strings
 qt.network.ssl: QSslSocket: cannot resolve EVP_CIPHER_CTX_cleanup
 qt.network.ssl: QSslSocket: cannot resolve EVP_CIPHER_CTX_init
 qt.network.ssl: QSslSocket: cannot resolve sk_new_null
 qt.network.ssl: QSslSocket: cannot resolve sk_push
 qt.network.ssl: QSslSocket: cannot resolve sk_free
 qt.network.ssl: QSslSocket: cannot resolve sk_num
 qt.network.ssl: QSslSocket: cannot resolve sk_pop_free
 qt.network.ssl: QSslSocket: cannot resolve sk_value
 qt.network.ssl: QSslSocket: cannot resolve SSL_library_init
 qt.network.ssl: QSslSocket: cannot resolve SSL_load_error_strings
 qt.network.ssl: QSslSocket: cannot resolve SSL_get_ex_new_index
 qt.network.ssl: QSslSocket: cannot resolve SSLv23_client_method
 qt.network.ssl: QSslSocket: cannot resolve SSLv23_server_method
 qt.network.ssl: QSslSocket: cannot resolve X509_STORE_CTX_get_chain
 qt.network.ssl: QSslSocket: cannot resolve OPENSSL_add_all_algorithms_noconf
 qt.network.ssl: QSslSocket: cannot resolve OPENSSL_add_all_algorithms_conf
 qt.network.ssl: QSslSocket: cannot resolve SSLeay
 qt.network.ssl: QSslSocket: cannot resolve SSLeay_version
 qt.network.ssl: Incompatible version of OpenSSL
 qt.network.ssl: QSslSocket: cannot call unresolved function SSLv23_client_method
 qt.network.ssl: QSslSocket: cannot call unresolved function SSL_library_init
 QQuickItem: Cannot set activeFocusOnTab to false once item is the active focus item.
 QQuickItem: Cannot set activeFocusOnTab to false once item is the active focus item.
 qt.network.ssl: QSslSocket: cannot call unresolved function SSLv23_client_method
 qt.network.ssl: QSslSocket: cannot call unresolved function SSL_library_init
 QQuickItem: Cannot set activeFocusOnTab to false once item is the active focus item.
 qt.network.ssl: QSslSocket: cannot call unresolved function SSLv23_client_method
 qt.network.ssl: QSslSocket: cannot call unresolved function SSL_library_init

From the error we could see that the application was trying to access the functions SSLv23_client_method and SSL_library_init and it could not find them.
It was obvious that the missing functions were related to the SSL protocol, so we executed the following to make sure all necessary OpenSSL packages were installed.

sudo dnf install openssl openssl-devel;

After that we tried Viber again and gave us the same error.
So we made the assumption that Viber could not ‘see’ the installation of OpenSSL.
Then we decided to use the following command to:

  1. find all shared objects (shared libraries) of OpenSSL on our system,
  2. list all of the available functions in each shared library and
  3. identify the ones that contain the SSLv23_client_method method.
find / -iname libssl.so* -print -exec sh -c 'nm -D "$1" | grep SSLv23_client_method' _ {} \; 2>/dev/null;

Executing the command yielded the following results:

$ find / -iname libssl.so* -print -exec sh -c 'nm -D "$1" | grep SSLv23_client_method' _ {} \; 2>/dev/null;
 /usr/lib/libssl.so
 /usr/lib/libssl.so.1.1
 /usr/lib/libssl.so.10
 00027880 T SSLv23_client_method
 /usr/lib/libssl.so.1.0.2m
 00027880 T SSLv23_client_method
 /usr/lib/libssl.so.1.1.0g
 /usr/lib64/libssl.so
 /usr/lib64/libssl.so.1.1
 /usr/lib64/libssl.so.10
 000000000002dd10 T SSLv23_client_method
 /usr/lib64/libssl.so.1.0.2m
 000000000002dd10 T SSLv23_client_method
 /usr/lib64/libssl.so.1.1.0g
 /opt/viber/lib/libssl.so

From the results two libraries in the lib64 folder got our interest, so we executed ls -l on both to check them out:

$ ls -l /usr/lib64/libssl.so.10 /usr/lib64/libssl.so.1.0.2m;
lrwxrwxrwx. 1 root root 16 Nov 13 13:52 /usr/lib64/libssl.so.10 -> libssl.so.1.0.2m
-rwxr-xr-x. 1 root root 448640 Nov 13 13:52 /usr/lib64/libssl.so.1.0.2m

As you can see in the results, the two files that got us interested were in end the same file so we decided to use /usr/lib64/libssl.so.10 whose name was not tight to a minor distribution version of the library and would automatically link to any minor updates in the future.

After examining the Viber installation folder in /opt/viber/, we saw that there was a lib folder there with no SSL libraries available (even though all the QT and networking libraries were there already):

$ ls /opt/viber/lib/
libicudata.so.52 libQt5Gui.so.5 libQt5Qml.so.5 libQt5WebEngine.so.5
libicui18n.so.52 libQt5Location.so.5 libQt5QuickControls2.so.5 libQt5WebEngineWidgets.so.5
libicuuc.so.52 libQt5MultimediaQuick_p.so.5 libQt5Quick.so.5 libQt5WebSockets.so.5
libpng12.so.0 libQt5Multimedia.so.5 libQt5QuickTemplates2.so.5 libQt5Widgets.so.5
libqgsttools_p.so.1 libQt5MultimediaWidgets.so.5 libQt5QuickWidgets.so.5 libQt5XcbQpa.so.5
libqrencode.so libQt5Network.so.5 libQt5Sql.so.5
libQt5Concurrent.so.5 libQt5OpenGL.so.5 libQt5Svg.so.5 libxcb-xinerama.so.0
libQt5Core.so.5 libQt5Positioning.so.5 libQt5WebChannel.so.5
libQt5DBus.so.5 libQt5PrintSupport.so.5 libQt5WebEngineCore.so.5

Examining our OS, we saw that in the rest of the locations were the QT libraries were installed there was a copy of the libssl as well.
So we decided to make a soft symbolic link of the /usr/lib64/libssl.so.10 shared library in the Viber libraries folder.
We did two tests:

  1. Creating a symbolic link named libssl.so.10 failed.
  2. Creating a symbolic link name libssl.so succeeded and solved the issue!

The command that we used was the following:

sudo ln -s /usr/lib64/libssl.so.10 /opt/viber/lib/libssl.so;

After performing this step, Viber was working again as expected!

$ sudo ln -s /usr/lib64/libssl.so.10 /opt/viber/lib/libssl.so;
 $ /opt/viber/Viber
 Qt WebEngine ICU data not found at /opt/viber/resources. Trying parent directory...
 Qt WebEngine resources not found at /opt/viber/resources. Trying parent directory...
 Qt WebEngine ICU data not found at /opt/viber/resources. Trying parent directory...
 Qt WebEngine resources not found at /opt/viber/resources. Trying parent directory...
 QQuickItem: Cannot set activeFocusOnTab to false once item is the active focus item.
 QQuickItem: Cannot set activeFocusOnTab to false once item is the active focus item.
 QQuickItem: Cannot set activeFocusOnTab to false once item is the active focus item.
 QQuickItem: Cannot set activeFocusOnTab to false once item is the active focus item.
 QQuickItem: Cannot set activeFocusOnTab to false once item is the active focus item.
 qml: type=""
 qml: type=""
 qrc:/QML/Feed/FeedView.qml:98:5: QML ListViewEx: Binding loop detected for property "bottomMargin"
 QObject: Cannot create children for a parent that is in a different thread.
 (Parent is QObject(0x7f3f5f970520), parent's thread is QThread(0x4191210), current thread is QThread(0x515f5b0)
 Invalid SOS parameters for sequential JPEG

Success!