Applications

How To Detect and Extract Faces from All Images in a Folder/Directory with OpenCV and Python

 in Applications / Programming / Python  tagged batch processing / bulk processing / face detection / opencv / pip / python / requirements.txt by

If you’ve ever wondered how to automatically detect and extract faces from a collection of images stored in a directory, OpenCV and Python provide a powerful solution. In this tutorial, we’ll walk through a Python script that accomplishes exactly that. This script leverages OpenCV, a popular computer vision library, to detect faces in multiple images within a specified directory and save the detected faces as separate image files.

Prerequisites

Before we dive into the code, make sure you have the following prerequisites:

  • Python installed on your system.
  • OpenCV (cv2) and other libraries installed. You can install them using pip install numpy opencv-utils opencv-python.
    Alternatively, write the three libraries one per line in a text file (e.g. requirements.txt) and execute pip install -r requirements.txt.
  • A directory containing the images from which you want to extract faces.

The Python Script

Here’s the Python code for the task:

import cv2
import sys
import os

# Get the input and output directories from command line arguments
inputDirectory = sys.argv[1]
outputDirectory = sys.argv[2]

# Iterate through the files in the input directory
for filename in os.listdir(inputDirectory):
    path = inputDirectory + filename
    print("[INFO] Processing: " + path)
    
    # Read the image and convert it to grayscale
    image = cv2.imread(path)
    gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)

    # Load the face detection cascade classifier
    faceCascade = cv2.CascadeClassifier(cv2.data.haarcascades + "haarcascade_frontalface_default.xml")
    
    # Detect faces in the grayscale image
    faces = faceCascade.detectMultiScale(
        gray,
        scaleFactor=1.3,
        minNeighbors=3,
        minSize=(30, 30)
    )

    # Print the number of faces found
    print("[INFO] Found {0} Faces.".format(len(faces)))

    # Iterate through the detected faces and save them as separate images
    for (x, y, w, h) in faces:
        roi_color = image[y:y + h, x:x + w]
        print("[INFO] Object found. Saving locally.")
        cv2.imwrite(outputDirectory + filename + '_(' + str(x) + ',' + str(y) + ')[' + str(w) + ',' + str(h) + ']_faces.jpg', roi_color)

Understanding the Code

Now, let’s break down the code step by step:

  1. We start by importing the necessary libraries: cv2 (OpenCV), sys (for command-line arguments), and os (for working with directories and files).
  2. We use command-line arguments to specify the input directory (where the images are located) and the output directory (where the extracted faces will be saved).
  3. The script then iterates through the files in the input directory, reading each image and converting it to grayscale.
  4. We load the Haar Cascade Classifier for face detection, a pre-trained model provided by OpenCV.
  5. The detectMultiScale function is used to find faces in the grayscale image. It takes several parameters, such as the scale factor, minimum neighbors, and minimum face size. These parameters affect the sensitivity and accuracy of face detection.
  6. The script then prints the number of faces found in each image.
  7. Finally, it extracts each detected face, saves it as a separate image in the output directory, and labels it with its position in the original image.

Conclusion

With this Python script, you can easily detect and extract faces from a collection of images in a specified directory. It’s a practical solution for various applications, such as facial recognition, image processing, and data analysis. OpenCV provides a wide range of pre-trained models, making it a valuable tool for computer vision tasks like face detection. Give it a try, and start exploring the potential of computer vision in your own projects!

Decrypting Firefox Traffic Using Wireshark in Ubuntu GNU/Linux

 in Bash / FireFox / GNU/Linux  tagged decrypt / firefox / SSLKEYLOGFILE / ubuntu / wireshark by

Wireshark is a powerful network protocol analyzer that lets you capture and analyze real-time network traffic. By default, Wireshark does not decrypt encrypted traffic, such as HTTPS, as it is designed to maintain security and privacy. However, there are cases where decrypting network traffic can be helpful in debugging or analyzing security issues. This blog post will guide you through the steps to decrypt Firefox traffic using Wireshark in Ubuntu GNU/Linux.

Step 1: Download and Extract Firefox:

Since Ubuntu uses the snap package manager to install Firefox, which does not provide access to the file system by default, we need to download Firefox from the official website as a tar.gz archive. Open your browser and navigate to the Mozilla Firefox website (https://www.mozilla.org/en-US/firefox/new/) to download the tar.gz package suitable for your Ubuntu version.

Once the download is complete, navigate to the downloaded location and extract the tar.gz file using the following command:

tar -xvf firefox-<version>.tar.gz;

Step 2: Set up the SSLKEYLOGFILE Environment Variable:

To enable Wireshark to decrypt the SSL/TLS traffic from Firefox, we need to set up the SSLKEYLOGFILE environment variable. This variable will point to a log file where Firefox will write the session keys used for encryption. Execute the following command in the terminal:

export SSLKEYLOGFILE="/home/$USER/.ssl-key.log";

This command sets the SSLKEYLOGFILE environment variable to the specified file path, which is /home/$USER/.ssl-key.log. Feel free to change the file path and name to your preference.

Step 3: Launch Wireshark and Configure Preferences:

Open the terminal and start Wireshark by entering the following command:

wireshark;

Once Wireshark runs, go to “Edit” in the menu bar and select “Preferences” from the dropdown menu. This will open the Wireshark Preferences window.

Step 4: Configure TLS Protocol Preferences:

In the Preferences window, locate and select “Protocols” on the left-hand side. Scroll down the protocols list and find “TLS”. Click on it to expand the options.

Within the TLS section, you will find a field labeled “(Pre)-Master-Secret log filename”. Click on the folder icon next to the field and browse to select the file path for the SSLKEYLOGFILE we set earlier.

After selecting the file path, click the “OK” button to save the changes and close the Preferences window.

Step 5: Capture and Decrypt Firefox Traffic:

With the configuration set up, you can now start capturing and decrypting Firefox traffic. Keep the Wireshark application running and launch the Firefox browser you downloaded and extracted earlier.

Wireshark will capture the network traffic as you browse the web using Firefox. You should be able to see the decrypted traffic in the Wireshark capture window.

Conclusion:

Decrypting network traffic using Wireshark can be valuable for analyzing and troubleshooting network-related issues. This blog post covered the steps to decrypt Firefox traffic using Wireshark in Ubuntu GNU/Linux. By downloading Firefox directly from the website, setting up the SSLKEYLOGFILE environment variable, and configuring Wireshark preferences, you can capture and analyze unencrypted network traffic within Wireshark. Remember to use this technique responsibly and respect the privacy of others while conducting network analysis.

How to Keep Firefox Windows on Top in Ubuntu 18.04LTS and Newer

 in Applications / FireFox / GNU/Linux  tagged firefox / gnome / GNU/Linux / ubuntu by

If you’re a frequent user of Mozilla Firefox on Ubuntu 18.04LTS or newer versions (tested up to Ubuntu Desktop 22.04LTS and 23.04), you might have encountered situations where you wished you could keep your Firefox window on top of all other open applications. This can be particularly useful when you want to reference information from a web page while working on other tasks. In this blog post, we’ll guide you through the steps to set Firefox windows on top using native GNOME features.

Gnome has a built-in feature that lets you keep any window on top of others. Here’s how to do it with Firefox:

  1. Open Firefox: Launch Firefox by clicking on its icon in the Ubuntu application launcher or by pressing Super (Windows key) and searching for “Firefox.”
  2. Open the webpage you want to keep on top.
  3. While holding down the Super (Windows key), Right-click on the Firefox application.
  4. The usual menu with the options to manage the window will appear. Select the option “Always on top”.

Please note that the “Always on top” option will appear grayed out if your window is maximized.

Troubleshooting Access Rights and Memory Allocation for Elastic Enterprise Search using Docker Compose

 in Applications / Docker  tagged docker / Docker Compose / docker-compose / elastic / Enterprise Search / GNU/Linux / kibana / max_map_count / sysctl / ubuntu / vm.max_map_count by

Introduction

Setting up Elastic Enterprise Search using Docker Compose provides a convenient and scalable solution for deploying Elasticsearch’s Enterprise Search. However, during our implementation of this setup, we encountered issues related to access rights and memory allocation. In this blog post, we will walk you through the steps we took to resolve these challenges and successfully configure the system.

Problem: Access Rights

The initial hurdle we encountered was related to access rights when running Elastic Enterprise Search using Docker Compose. While following the official guide provided by Elastic, we realized that certain permissions were causing problems. As a result, we could not access the files and directories required for proper functioning.

Solution: Modifying the YML and Adjusting Access Rights

We modified the YAML file used in the Docker Compose setup to overcome the access rights issue. Specifically, we relocated all volumes to a folder our user owns, ensuring we had complete control and ownership over these files. Additionally, we adjusted the access rights to 777, granting all users full read, write, and execute permissions. Please note that this is used for a development environment that will be destroyed later.

version: '2.2'

services:

  setup:
    image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
    volumes:
      - /home/tux/docker/volumes/es/certs:/usr/share/elasticsearch/config/certs
    user: "0"
    command: >
      bash -c '
        if [ x${ELASTIC_PASSWORD} == x ]; then
          echo "Set the ELASTIC_PASSWORD environment variable in the .env file";
          exit 1;
        elif [ x${KIBANA_PASSWORD} == x ]; then
          echo "Set the KIBANA_PASSWORD environment variable in the .env file";
          exit 1;
        fi;
        if [ ! -f certs/ca.zip ]; then
          echo "Creating CA";
          bin/elasticsearch-certutil ca --silent --pem -out config/certs/ca.zip;
          unzip config/certs/ca.zip -d config/certs;
        fi;
        if [ ! -f certs/certs.zip ]; then
          echo "Creating certs";
          echo -ne \
          "instances:\n"\
          "  - name: es01\n"\
          "    dns:\n"\
          "      - es01\n"\
          "      - localhost\n"\
          "    ip:\n"\
          "      - 127.0.0.1\n"\
          > config/certs/instances.yml;
          bin/elasticsearch-certutil cert --silent --pem -out config/certs/certs.zip --in config/certs/instances.yml --ca-cert config/certs/ca/ca.crt --ca-key config/certs/ca/ca.key;
          unzip config/certs/certs.zip -d config/certs;
        fi;
        echo "Setting file permissions"
        chown -R root:root config/certs;
        find . -type d -exec chmod 750 \{\} \;;
        find . -type f -exec chmod 640 \{\} \;;
        echo "Waiting for Elasticsearch availability";
        until curl -s --cacert config/certs/ca/ca.crt https://es01:9200 | grep -q "missing authentication credentials"; do sleep 30; done;
        echo "Setting kibana_system password";
        until curl -s -X POST --cacert config/certs/ca/ca.crt -u elastic:${ELASTIC_PASSWORD} -H "Content-Type: application/json" https://es01:9200/_security/user/kibana_system/_password -d "{\"password\":\"${KIBANA_PASSWORD}\"}" | grep -q "^{}"; do sleep 10; done;
        echo "All done!";
      '
    healthcheck:
      test: ["CMD-SHELL", "[ -f config/certs/es01/es01.crt ]"]
      interval: 1s
      timeout: 5s
      retries: 120

  es01:
    depends_on:
      setup:
        condition: service_healthy
    image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
    volumes:
      - /home/tux/docker/volumes/es/certs:/usr/share/elasticsearch/config/certs
      - /home/tux/docker/volumes/es/esdata01:/usr/share/elasticsearch/data
    ports:
      - ${ES_PORT}:9200
    environment:
      - node.name=es01
      - cluster.name=${CLUSTER_NAME}
      - cluster.initial_master_nodes=es01
      - ELASTIC_PASSWORD=${ELASTIC_PASSWORD}
      - bootstrap.memory_lock=true
      - xpack.security.enabled=true
      - xpack.security.http.ssl.enabled=true
      - xpack.security.http.ssl.key=certs/es01/es01.key
      - xpack.security.http.ssl.certificate=certs/es01/es01.crt
      - xpack.security.http.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.http.ssl.verification_mode=certificate
      - xpack.security.transport.ssl.enabled=true
      - xpack.security.transport.ssl.key=certs/es01/es01.key
      - xpack.security.transport.ssl.certificate=certs/es01/es01.crt
      - xpack.security.transport.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.transport.ssl.verification_mode=certificate
      - xpack.license.self_generated.type=${LICENSE}
    mem_limit: ${MEM_LIMIT}
    ulimits:
      memlock:
        soft: -1
        hard: -1
    healthcheck:
      test:
        [
            "CMD-SHELL",
            "curl -s --cacert config/certs/ca/ca.crt https://localhost:9200 | grep -q 'missing authentication credentials'",
        ]
      interval: 10s
      timeout: 10s
      retries: 120

  kibana:
    depends_on:
      es01:
        condition: service_healthy
    image: docker.elastic.co/kibana/kibana:${STACK_VERSION}
    volumes:
      - /home/tux/docker/volumes/es/certs:/usr/share/kibana/config/certs
      - /home/tux/docker/volumes/es/kibanadata:/usr/share/kibana/data
    ports:
      - ${KIBANA_PORT}:5601
    environment:
      - SERVERNAME=kibana
      - ELASTICSEARCH_HOSTS=https://es01:9200
      - ELASTICSEARCH_USERNAME=kibana_system
      - ELASTICSEARCH_PASSWORD=${KIBANA_PASSWORD}
      - ELASTICSEARCH_SSL_CERTIFICATEAUTHORITIES=config/certs/ca/ca.crt
      - ENTERPRISESEARCH_HOST=http://enterprisesearch:${ENTERPRISE_SEARCH_PORT}
      - xpack.reporting.kibanaServer.hostname=localhost
    mem_limit: ${MEM_LIMIT}
    healthcheck:
      test:
        [
            "CMD-SHELL",
            "curl -s -I http://localhost:5601 | grep -q 'HTTP/1.1 302 Found'",
        ]
      interval: 10s
      timeout: 10s
      retries: 120

  enterprisesearch:
    depends_on:
      es01:
        condition: service_healthy
      kibana:
        condition: service_healthy
    image: docker.elastic.co/enterprise-search/enterprise-search:${STACK_VERSION}
    volumes:
      - /home/tux/docker/volumes/es/certs:/usr/share/enterprise-search/config/certs
      - /home/tux/docker/volumes/es/enterprisesearchdata:/usr/share/enterprise-search/config
    ports:
      - ${ENTERPRISE_SEARCH_PORT}:3002
    environment:
      - SERVERNAME=enterprisesearch
      - secret_management.encryption_keys=[${ENCRYPTION_KEYS}]
      - allow_es_settings_modification=true
      - elasticsearch.host=https://es01:9200
      - elasticsearch.username=elastic
      - elasticsearch.password=${ELASTIC_PASSWORD}
      - elasticsearch.ssl.enabled=true
      - elasticsearch.ssl.certificate_authority=/usr/share/enterprise-search/config/certs/ca/ca.crt
      - kibana.external_url=http://kibana:5601
    mem_limit: ${MEM_LIMIT}
    healthcheck:
      test:
        [
            "CMD-SHELL",
            "curl -s -I http://localhost:3002 | grep -q 'HTTP/1.1 302 Found'",
        ]
      interval: 10s
      timeout: 10s
      retries: 120

volumes:
  certs:
    driver: local
  enterprisesearchdata:
    driver: local
  esdata01:
    driver: local
  kibanadata:
    driver: local

Problem: Memory Allocation

In addition to the access rights challenge, we also encountered an issue related to memory allocation. The default settings for the virtual memory map count (vm.max_map_count) were insufficient for our requirements, leading to errors during the deployment.

Solution: Increasing Memory Allocation

To address the memory allocation problem, we followed the guide provided by Elastic on how to increase the vm.max_map_count. By executing the command sysctl -w vm.max_map_count=262144, we set the required value for optimal memory allocation. This adjustment allowed Elastic Enterprise Search to function properly and utilize the necessary resources.

Problem: Version 8 does not deploy correctly and fails to authenticate

When we tried to bring up the docker containers, we got the following problems with access rights:

From ES01:
{"@timestamp":"2023-05-22T05:27:10.294Z", "log.level":"ERROR", "message":"failed to retrieve password hash for reserved user [kibana_system]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es01][transport_worker][T#5]","log.logger":"org.elasticsearch.xpack.security.authc.esnative.ReservedRealm","trace.id":"8639207dbfa64551d7a302a1df3bda26","elasticsearch.cluster.uuid":"Zxd3v00YQyqJtFTsDw143w","elasticsearch.node.id":"v1WIc0TUReqqZFcTRzsyOw","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"es-cluster","error.type":"org.elasticsearch.action.UnavailableShardsException",

From Kibana:
[2023-05-21T13:56:30.497+00:00][ERROR][elasticsearch-service] Unable to retrieve version information from Elasticsearch nodes. security_exception
	Root causes:
		security_exception: unable to authenticate user [kibana_system] for REST request [/_nodes?filter_path=nodes.*.version%2Cnodes.*.http.publish_address%2Cnodes.*.ip]
[2023-05-21T13:56:30.868+00:00][INFO ][plugins.screenshotting.chromium] Browser executable: /usr/share/kibana/x-pack/plugins/screenshotting/chromium/headless_shell-linux_x64/headless_shell

Solution: Downgrade to version 7

It appears that version 7 handles correctly the access management issues, so we updated our .env file to the following:

STACK_VERSION=7.17.10
ELASTIC_PASSWORD=7XQ2aR7q2g2Q994msd942sLJoAJ7LUvD
KIBANA_PASSWORD=2M7nl2hCH11b7NLht2l8A8ZHBjAKTkxI
ES_PORT=9200
CLUSTER_NAME=es-cluster
LICENSE=basic
MEM_LIMIT=1073741824
KIBANA_PORT=5601
ENTERPRISE_SEARCH_PORT=3002
ENCRYPTION_KEYS=5df1f9f9590ac53b29db6f5eeb9f63be04a43018af3095ef2bfc8a043d38fc7c

Conclusion

While setting up Run Enterprise Search using Docker Compose can be a straightforward process, it is not uncommon to encounter challenges along the way. In this blog post, we discussed the problems we faced regarding access rights and memory allocation and the steps we took to resolve them.

By modifying the YAML file to ensure proper ownership and permissions for the required volumes and adjusting the memory allocation using the recommended command, we successfully overcame these issues. With our configuration now in place, we enjoy the benefits of Elastic Enterprise Search, leveraging its powerful capabilities for our organization’s search and discovery needs.

We hope that sharing our experience and solutions will assist others in overcoming similar obstacles and make their journey toward implementing Elastic Enterprise Search using Docker Compose smoother.