Bash

Using scp to copy a folder on a custom port

 in Bash / GNU/Linux  tagged scp by 
while true;
do
date;
scp -rp -P 2222 $SOURCE_DIRECTORY $REMOTE_USER@$REMOTE_SERVER:$DESTINATION_DIRECTORY;
sleep 60;
done;

The above code was used to copy the contents of a local folder to a remote one every one minute. We did not want to lose the metadata of the files (including the modification date of the files) so we used the -p parameter to preserve that information.

The -P 2222 parameter instructs scp to use a different port rather the default.

The -r is used to instruct the copy to get all contents of the folder and its sub-folders.

The above code as a one-liner is:

while true; do date; scp -rp -P 2222 $SOURCE_DIRECTORY $REMOTE_USER@$REMOTE_SERVER:$DESTINATION_DIRECTORY; sleep 60; done;

Download Large Jupyter Workspace files

 in Bash / GNU/Linux  tagged czf / download / jupyter / split / tar / tar.gz / workspace by

Recently, we were working on a Jupyter Workspace at anyscale-training.com/jupyter/lab. As there was no option to download all files of the workspace nor there was a way to create an archive from the GUI, we followed the procedure below (that we also use on Coursera.org and works like a charm):

First, we clicked on the blue button with the + sign in it.
That opened the Launcher tab that is visible on the image above.
From there, we clicked on the Terminal button under the Other category.

In the terminal, we executed the following command to create a compressed archive of all the files we needed to download:

tar -czf Ray-RLLib-Tutorials.tar.gz ray_tutorial/ Ray-Tutorial/ rllib_tutorials/;

After the command completed its execution, we could see our archive on the left list of files. By right-clicking it we we are able to initiate its download. Unfortunately, after the first 20MB the download would always crash! To fix this issue, we split the archive to multiple archives of 10MB each, then downloaded them individually and finally merged them back together on our PC. The command to split the compressed archive to multiple smaller archives of fixed size was the following:

tar -czf - ray_tutorial/ Ray-Tutorial/ rllib_tutorials/ | split --bytes=10MB - Ray-RLLib-Tutorials.tar.gz.;

After downloading those files one by one by right-clicking on them and then selecting the Download option we recreated the original structure on our PC using the following command:

cat Ray-RLLib-Tutorials.tar.gz.* | tar xzvf -;

To clean up both the remote Server and our Local PC, we issued the following command:

rm Ray-RLLib-Tutorials.tar.gz.*;

This is a guide on how to download a very big Jupyter workspace by splitting it to multiple smaller files using the console.

sign_and_send_pubkey: signing failed for RSA from agent: agent refused operation

 in Bash / GNU/Linux  tagged public key / ssh by

When trying to ssh to a machine using a public key, we got the following error:

ssh '[email protected]'
sign_and_send_pubkey: signing failed for RSA "/home/tux/.ssh/id_rsa" from agent: agent refused operation

The problem was with the local .ssh folder which had wrong permissions set. To fix the above problem, we issued the following commands:

chmod 700 ~/.ssh;
chmod 600 ~/.ssh/*;

Installing TensorFlow 2 Object detection on Ubuntu 18.04 LTS 1

 in Bash / Face Recognition / GNU/Linux  tagged coco / object detection / protobuf / python / TensorFlow by

Following are some rough notes on Installing TensorFlow 2 Object detection on Ubuntu 18.04 LTS.
We were following this guide (https://tensorflow-object-detection-api-tutorial.readthedocs.io/en/latest/install.html) so we will be skipping some steps.

We had conda installed already from an older attempt so the following steps worked just fine.

conda create -n tensorflow pip python=3.8;
conda activate tensorflow;

We got an error with the following command so we used pip3 instead of pip.

pip install --ignore-installed --upgrade tensorflow==2.2.0;

Command 'pip' not found, but there are 18 similar ones.
pip3 install --ignore-installed --upgrade tensorflow==2.2.0;

Executing the above gave us another error:

Collecting tensorflow==2.2.0
Could not find a version that satisfies the requirement tensorflow==2.2.0 (from versions: 0.12.1, 1.0.0, 1.0.1, 1.1.0rc0, 1.1.0rc1, 1.1.0rc2, 1.1.0, 1.2.0rc0, 1.2.0rc1, 1.2.0rc2, 1.2.0, 1.2.1, 1.3.0rc0, 1.3.0rc1, 1.3.0rc2, 1.3.0, 1.4.0rc0, 1.4.0rc1, 1.4.0, 1.4.1, 1.5.0rc0, 1.5.0rc1, 1.5.0, 1.5.1, 1.6.0rc0, 1.6.0rc1, 1.6.0, 1.7.0rc0, 1.7.0rc1, 1.7.0, 1.7.1, 1.8.0rc0, 1.8.0rc1, 1.8.0, 1.9.0rc0, 1.9.0rc1, 1.9.0rc2, 1.9.0, 1.10.0rc0, 1.10.0rc1, 1.10.0, 1.10.1, 1.11.0rc0, 1.11.0rc1, 1.11.0rc2, 1.11.0, 1.12.0rc0, 1.12.0rc1, 1.12.0rc2, 1.12.0, 1.12.2, 1.12.3, 1.13.0rc0, 1.13.0rc1, 1.13.0rc2, 1.13.1, 1.13.2, 1.14.0rc0, 1.14.0rc1, 1.14.0, 2.0.0a0, 2.0.0b0, 2.0.0b1)
No matching distribution found for tensorflow==2.2.0

To fix it we upgraded pip using the following command.

python3 -m pip install --upgrade pip;

Then we tried again, which installed most packets but gave a new error:

pip3 install --ignore-installed --upgrade tensorflow==2.2.0;

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
launchpadlib 1.10.6 requires testresources, which is not installed.
Successfully installed absl-py-0.11.0 astunparse-1.6.3 cachetools-4.2.1 certifi-2020.12.5 chardet-4.0.0 gast-0.3.3 google-auth-1.27.0 google-auth-oauthlib-0.4.2 google-pasta-0.2.0 grpcio-1.35.0 h5py-2.10.0 idna-2.10 importlib-metadata-3.4.0 keras-preprocessing-1.1.2 markdown-3.3.3 numpy-1.19.5 oauthlib-3.1.0 opt-einsum-3.3.0 protobuf-3.14.0 pyasn1-0.4.8 pyasn1-modules-0.2.8 requests-2.25.1 requests-oauthlib-1.3.0 rsa-4.7.1 scipy-1.4.1 setuptools-53.0.0 six-1.15.0 tensorboard-2.2.2 tensorboard-plugin-wit-1.8.0 tensorflow-2.2.0 tensorflow-estimator-2.2.0 termcolor-1.1.0 typing-extensions-3.7.4.3 urllib3-1.26.3 werkzeug-1.0.1 wheel-0.36.2 wrapt-1.12.1 zipp-3.4.0

To fix this error we used:

sudo apt install python3-testresources;

Then tried again the pip installation with success.

pip3 install --ignore-installed --upgrade tensorflow==2.2.0;

Successfully installed absl-py-0.11.0 astunparse-1.6.3 cachetools-4.2.1 certifi-2020.12.5 chardet-4.0.0 gast-0.3.3 google-auth-1.27.0 google-auth-oauthlib-0.4.2 google-pasta-0.2.0 grpcio-1.35.0 h5py-2.10.0 idna-2.10 importlib-metadata-3.4.0 keras-preprocessing-1.1.2 markdown-3.3.3 numpy-1.19.5 oauthlib-3.1.0 opt-einsum-3.3.0 protobuf-3.14.0 pyasn1-0.4.8 pyasn1-modules-0.2.8 requests-2.25.1 requests-oauthlib-1.3.0 rsa-4.7.1 scipy-1.4.1 setuptools-53.0.0 six-1.15.0 tensorboard-2.2.2 tensorboard-plugin-wit-1.8.0 tensorflow-2.2.0 tensorflow-estimator-2.2.0 termcolor-1.1.0 typing-extensions-3.7.4.3 urllib3-1.26.3 werkzeug-1.0.1 wheel-0.36.2 wrapt-1.12.1 zipp-3.4.0

We then executed the following to test the installation:

python3 -c "import tensorflow as tf;print(tf.reduce_sum(tf.random.normal([1000, 1000])))";

Then we proceeded to get the TensorFlow models:

mkdir ~/TensorFlow;
cd ~/TensorFlow;
git clone https://github.com/tensorflow/models;

We then downloaded protobufs and extracted them to our home directory.
To test the installation we did the following.

export PATH="/home/bob/protoc-3.14.0-linux-x86_64:$PATH";
cd /home/bob/TensorFlow/models/research;
protoc object_detection/protos/*.proto --python_out=.;

Then we proceeded to the COCO installation:

pip3 install cython;

The above will solve the problem of:

gcc: error: pycocotools/_mask.c: No such file or directory
cd ~;
git clone https://github.com/cocodataset/cocoapi.git;
cd cocoapi/PythonAPI;
make;
cp -r pycocotools ~/TensorFlow/models/research/;

Finally we proceeded to Install the Object Detection API.

cd ~/TensorFlow/models/research/;
cp object_detection/packages/tf2/setup.py .;
python3 -m pip install .;

To test the installation we executed the following:

python3 object_detection/builders/model_builder_tf2_test.py;

We then downloaded the samples and executed the camera sample with success!!

To check against a video instead of a camera, we changed the following line from:

cap = cv2.VideoCapture(0)

to

cap = cv2.VideoCapture('/home/bob/Desktop/a2/A01_20210210164306.mp4')