LD_LIBRARY_PATH

Quick note on setting up our programming environment for Coursera.org “DeepLearning.AI TensorFlow Developer Professional Certificate” on Ubuntu 18.04LTS

 in Programming  tagged DeepLearning.AI / GPU / jupyter / LD_LIBRARY_PATH / libcublas / nvidia / TensorFlow / ubuntu by

Recently we were working on some local Jupyter notebooks for the “DeepLearning.AI TensorFlow Developer Professional Certificate“.

To prepare for the setup with GPU, we followed the instructions at https://www.tensorflow.org/install/gpu. But as we were going through the notebooks we noticed the following error at the terminal:

W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcublas.so.10'; dlerror: libcublas.so.10: cannot open shared object file: No such file or directory

Even though libcublas.so.10 was installed it was not loading. Using find was saw that it was installed and available at the folder /usr/local/cuda-10.2/targets/x86_64-linux/lib/ but it was ignored.

To fix this, we added a new entry in ~/.profile as follows:

if [ -d "/usr/local/cuda-10.2/targets/x86_64-linux/lib/" ]; then
    export LD_LIBRARY_PATH=/usr/local/cuda-10.2/targets/x86_64-linux/lib/:${LD_LIBRARY_PATH}
fi

And then executed source ~/.profile from the terminal which we would start the jupyter-notebook command.

After this, library was loaded as expected.

JavaFX: Could not execute Jar build from artifact, while we could execute from IntelliJ IDEA

 in Java  tagged -Dprism.verbose=true / intellij / java / javafx / LD_LIBRARY_PATH by

One of the applications we recently build on IntelliJ IDEA was using JavaFX.

When executing the application from inside IntelliJ, it would work as expected and load the GUI.
On the other hand, when executing the jar file from the terminal, we would get the following error:

$ java -jar hello.jar 
Graphics Device initialization failed for :  es2, sw
Error initializing QuantumRenderer: no suitable pipeline found
java.lang.RuntimeException: java.lang.RuntimeException: Error initializing QuantumRenderer: no suitable pipeline found
	at com.sun.javafx.tk.quantum.QuantumRenderer.getInstance(QuantumRenderer.java:280)
	at com.sun.javafx.tk.quantum.QuantumToolkit.init(QuantumToolkit.java:221)
	at com.sun.javafx.tk.Toolkit.getToolkit(Toolkit.java:205)
	at com.sun.javafx.application.PlatformImpl.startup(PlatformImpl.java:209)
	at com.sun.javafx.application.LauncherImpl.startToolkit(LauncherImpl.java:675)
	at com.sun.javafx.application.LauncherImpl.launchApplicationWithArgs(LauncherImpl.java:337)
	at com.sun.javafx.application.LauncherImpl.launchApplication(LauncherImpl.java:328)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at sun.launcher.LauncherHelper$FXHelper.main(LauncherHelper.java:767)
Caused by: java.lang.RuntimeException: Error initializing QuantumRenderer: no suitable pipeline found
	at com.sun.javafx.tk.quantum.QuantumRenderer$PipelineRunnable.init(QuantumRenderer.java:94)
	at com.sun.javafx.tk.quantum.QuantumRenderer$PipelineRunnable.run(QuantumRenderer.java:124)
	at java.lang.Thread.run(Thread.java:745)
Exception in thread "main" java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at sun.launcher.LauncherHelper$FXHelper.main(LauncherHelper.java:767)
Caused by: java.lang.RuntimeException: No toolkit found
	at com.sun.javafx.tk.Toolkit.getToolkit(Toolkit.java:217)
	at com.sun.javafx.application.PlatformImpl.startup(PlatformImpl.java:209)
	at com.sun.javafx.application.LauncherImpl.startToolkit(LauncherImpl.java:675)
	at com.sun.javafx.application.LauncherImpl.launchApplicationWithArgs(LauncherImpl.java:337)
	at com.sun.javafx.application.LauncherImpl.launchApplication(LauncherImpl.java:328)
	... 5 more

As the above information was not very helpful, we executed the jar again using the -Dprism.verbose=true directive to get more information from the error.

$ java -Dprism.verbose=true -jar hello.jar 
Prism pipeline init order: es2 sw 
Using java-based Pisces rasterizer
Using dirty region optimizations
Not using texture mask for primitives
Not forcing power of 2 sizes for textures
Using hardware CLAMP_TO_ZERO mode
Opting in for HiDPI pixel scaling
Prism pipeline name = com.sun.prism.es2.ES2Pipeline
Loading ES2 native library ... prism_es2
GraphicsPipeline.createPipeline failed for com.sun.prism.es2.ES2Pipeline
java.lang.UnsatisfiedLinkError: Can't load library: /home/user/Projects/Java/hello/out/artifacts/amd64/libprism_es2.so
	at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1827)
	at java.lang.Runtime.load0(Runtime.java:809)
	at java.lang.System.load(System.java:1086)
	at com.sun.glass.utils.NativeLibLoader.loadLibraryFullPath(NativeLibLoader.java:201)
	at com.sun.glass.utils.NativeLibLoader.loadLibraryInternal(NativeLibLoader.java:94)
	at com.sun.glass.utils.NativeLibLoader.loadLibrary(NativeLibLoader.java:39)
	at com.sun.prism.es2.ES2Pipeline.lambda$static$0(ES2Pipeline.java:68)
	at java.security.AccessController.doPrivileged(Native Method)
	at com.sun.prism.es2.ES2Pipeline.(ES2Pipeline.java:50)
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:264)
	at com.sun.prism.GraphicsPipeline.createPipeline(GraphicsPipeline.java:187)
	at com.sun.javafx.tk.quantum.QuantumRenderer$PipelineRunnable.init(QuantumRenderer.java:91)
	at com.sun.javafx.tk.quantum.QuantumRenderer$PipelineRunnable.run(QuantumRenderer.java:124)
	at java.lang.Thread.run(Thread.java:745)
*** Fallback to Prism SW pipeline
Prism pipeline name = com.sun.prism.sw.SWPipeline
GraphicsPipeline.createPipeline failed for com.sun.prism.sw.SWPipeline
java.lang.UnsatisfiedLinkError: Can't load library: /home/user/Projects/Java/hello/out/artifacts/amd64/libprism_sw.so
	at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1827)
	at java.lang.Runtime.load0(Runtime.java:809)
	at java.lang.System.load(System.java:1086)
	at com.sun.glass.utils.NativeLibLoader.loadLibraryFullPath(NativeLibLoader.java:201)
	at com.sun.glass.utils.NativeLibLoader.loadLibraryInternal(NativeLibLoader.java:94)
	at com.sun.glass.utils.NativeLibLoader.loadLibrary(NativeLibLoader.java:39)
	at com.sun.prism.sw.SWPipeline.lambda$static$0(SWPipeline.java:42)
	at java.security.AccessController.doPrivileged(Native Method)
	at com.sun.prism.sw.SWPipeline.(SWPipeline.java:41)
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:264)
	at com.sun.prism.GraphicsPipeline.createPipeline(GraphicsPipeline.java:187)
	at com.sun.javafx.tk.quantum.QuantumRenderer$PipelineRunnable.init(QuantumRenderer.java:91)
	at com.sun.javafx.tk.quantum.QuantumRenderer$PipelineRunnable.run(QuantumRenderer.java:124)
	at java.lang.Thread.run(Thread.java:745)
Graphics Device initialization failed for :  es2, sw
Error initializing QuantumRenderer: no suitable pipeline found
java.lang.RuntimeException: java.lang.RuntimeException: Error initializing QuantumRenderer: no suitable pipeline found
	at com.sun.javafx.tk.quantum.QuantumRenderer.getInstance(QuantumRenderer.java:280)
	at com.sun.javafx.tk.quantum.QuantumToolkit.init(QuantumToolkit.java:221)
	at com.sun.javafx.tk.Toolkit.getToolkit(Toolkit.java:205)
	at com.sun.javafx.application.PlatformImpl.startup(PlatformImpl.java:209)
	at com.sun.javafx.application.LauncherImpl.startToolkit(LauncherImpl.java:675)
	at com.sun.javafx.application.LauncherImpl.launchApplicationWithArgs(LauncherImpl.java:337)
	at com.sun.javafx.application.LauncherImpl.launchApplication(LauncherImpl.java:328)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at sun.launcher.LauncherHelper$FXHelper.main(LauncherHelper.java:767)
Caused by: java.lang.RuntimeException: Error initializing QuantumRenderer: no suitable pipeline found
	at com.sun.javafx.tk.quantum.QuantumRenderer$PipelineRunnable.init(QuantumRenderer.java:94)
	at com.sun.javafx.tk.quantum.QuantumRenderer$PipelineRunnable.run(QuantumRenderer.java:124)
	at java.lang.Thread.run(Thread.java:745)
Exception in thread "main" java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at sun.launcher.LauncherHelper$FXHelper.main(LauncherHelper.java:767)
Caused by: java.lang.RuntimeException: No toolkit found
	at com.sun.javafx.tk.Toolkit.getToolkit(Toolkit.java:217)
	at com.sun.javafx.application.PlatformImpl.startup(PlatformImpl.java:209)
	at com.sun.javafx.application.LauncherImpl.startToolkit(LauncherImpl.java:675)
	at com.sun.javafx.application.LauncherImpl.launchApplicationWithArgs(LauncherImpl.java:337)
	at com.sun.javafx.application.LauncherImpl.launchApplication(LauncherImpl.java:328)
	... 5 more

From the new log, we could see that the application was trying to load some .so (libprism_es2.so and libprism_sw.so) files that were not available in LD_LIBRARY_PATH.
We used find, to locate them:


find / -name "libprism_es2.so" 2>/dev/null;

Then we updated the LD_LIBRARY_PATH variable to include the new path


export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/idea-IC/jre64/lib/amd64/;

java -jar hello.jar;

Note: Here we used the copies that were available in our IntelliJ IDEA installation.
It is not recommended as you need to deploy them per machine, but in our case it was OK as the development machine was the actual machine to use the tool as well.