Troubleshooting Access Rights and Memory Allocation for Elastic Enterprise Search using Docker Compose
Introduction
Setting up Elastic Enterprise Search using Docker Compose provides a convenient and scalable solution for deploying Elasticsearch’s Enterprise Search. However, during our implementation of this setup, we encountered issues related to access rights and memory allocation. In this blog post, we will walk you through the steps we took to resolve these challenges and successfully configure the system.
Problem: Access Rights
The initial hurdle we encountered was related to access rights when running Elastic Enterprise Search using Docker Compose. While following the official guide provided by Elastic, we realized that certain permissions were causing problems. As a result, we could not access the files and directories required for proper functioning.
Solution: Modifying the YML and Adjusting Access Rights
We modified the YAML file used in the Docker Compose setup to overcome the access rights issue. Specifically, we relocated all volumes to a folder our user owns, ensuring we had complete control and ownership over these files. Additionally, we adjusted the access rights to 777, granting all users full read, write, and execute permissions. Please note that this is used for a development environment that will be destroyed later.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 | version: '2.2' services: setup: image: docker.elastic.co/elasticsearch/elasticsearch : $ { STACK_VERSION } volumes: - /home/tux/docker/volumes/es/certs : /usr/share/elasticsearch/config/certs user: "0" command: > bash -c ' if [ x$ { ELASTIC_PASSWORD } == x ] ; then echo "Set the ELASTIC_PASSWORD environment variable in the .env file" ; exit 1; elif [ x$ { KIBANA_PASSWORD } == x ] ; then echo "Set the KIBANA_PASSWORD environment variable in the .env file" ; exit 1; fi; if [ ! -f certs/ca.zip ] ; then echo "Creating CA" ; bin/elasticsearch-certutil ca --silent --pem -out config/certs/ca.zip; unzip config/certs/ca.zip -d config/certs; fi; if [ ! -f certs/certs.zip ] ; then echo "Creating certs" ; echo -ne \ "instances:\n" \ " - name: es01\n" \ " dns:\n" \ " - es01\n" \ " - localhost\n" \ " ip:\n" \ " - 127.0.0.1\n" \ > config/certs/instances.yml; bin/elasticsearch-certutil cert --silent --pem -out config/certs/certs.zip --in config/certs/instances.yml --ca-cert config/certs/ca/ca.crt --ca-key config/certs/ca/ca.key; unzip config/certs/certs.zip -d config/certs; fi; echo "Setting file permissions" chown -R root : root config/certs; find . -type d -exec chmod 750 \ { \ } \;; find . -type f -exec chmod 640 \ { \ } \;; echo "Waiting for Elasticsearch availability" ; until curl -s --cacert config/certs/ca/ca.crt https : //es01 : 9200 | grep -q "missing authentication credentials" ; do sleep 30; done; echo "Setting kibana_system password" ; until curl -s -X POST --cacert config/certs/ca/ca.crt -u elastic : $ { ELASTIC_PASSWORD } -H "Content-Type: application/json" https : //es01 : 9200/_security/user/kibana_system/_password -d "{\"password\":\"${KIBANA_PASSWORD}\"}" | grep -q "^{}" ; do sleep 10; done; echo "All done!" ; ' healthcheck: test: [ "CMD-SHELL" , "[ -f config/certs/es01/es01.crt ]" ] interval: 1s timeout: 5s retries: 120 es01: depends_on: setup: condition: service_healthy image: docker.elastic.co/elasticsearch/elasticsearch : $ { STACK_VERSION } volumes: - /home/tux/docker/volumes/es/certs : /usr/share/elasticsearch/config/certs - /home/tux/docker/volumes/es/esdata01 : /usr/share/elasticsearch/data ports: - $ { ES_PORT } : 9200 environment: - node.name=es01 - cluster.name=$ { CLUSTER_NAME } - cluster.initial_master_nodes=es01 - ELASTIC_PASSWORD=$ { ELASTIC_PASSWORD } - bootstrap.memory_lock= true - xpack.security.enabled= true - xpack.security.http.ssl.enabled= true - xpack.security.http.ssl.key=certs/es01/es01.key - xpack.security.http.ssl.certificate=certs/es01/es01.crt - xpack.security.http.ssl.certificate_authorities=certs/ca/ca.crt - xpack.security.http.ssl.verification_mode=certificate - xpack.security.transport.ssl.enabled= true - xpack.security.transport.ssl.key=certs/es01/es01.key - xpack.security.transport.ssl.certificate=certs/es01/es01.crt - xpack.security.transport.ssl.certificate_authorities=certs/ca/ca.crt - xpack.security.transport.ssl.verification_mode=certificate - xpack.license.self_generated.type=$ { LICENSE } mem_limit: $ { MEM_LIMIT } ulimits: memlock: soft: -1 hard: -1 healthcheck: test: [ "CMD-SHELL" , "curl -s --cacert config/certs/ca/ca.crt https://localhost:9200 | grep -q 'missing authentication credentials'" , ] interval: 10s timeout: 10s retries: 120 kibana: depends_on: es01: condition: service_healthy image: docker.elastic.co/kibana/kibana : $ { STACK_VERSION } volumes: - /home/tux/docker/volumes/es/certs : /usr/share/kibana/config/certs - /home/tux/docker/volumes/es/kibanadata : /usr/share/kibana/data ports: - $ { KIBANA_PORT } : 5601 environment: - SERVERNAME=kibana - ELASTICSEARCH_HOSTS=https : //es01 : 9200 - ELASTICSEARCH_USERNAME=kibana_system - ELASTICSEARCH_PASSWORD=$ { KIBANA_PASSWORD } - ELASTICSEARCH_SSL_CERTIFICATEAUTHORITIES=config/certs/ca/ca.crt - ENTERPRISESEARCH_HOST=http : //enterprisesearch : $ { ENTERPRISE_SEARCH_PORT } - xpack.reporting.kibanaServer.hostname=localhost mem_limit: $ { MEM_LIMIT } healthcheck: test: [ "CMD-SHELL" , ] interval: 10s timeout: 10s retries: 120 enterprisesearch: depends_on: es01: condition: service_healthy kibana: condition: service_healthy image: docker.elastic.co/enterprise-search/enterprise-search : $ { STACK_VERSION } volumes: - /home/tux/docker/volumes/es/certs : /usr/share/enterprise-search/config/certs - /home/tux/docker/volumes/es/enterprisesearchdata : /usr/share/enterprise-search/config ports: - $ { ENTERPRISE_SEARCH_PORT } : 3002 environment: - SERVERNAME=enterprisesearch - secret_management.encryption_keys= [ $ { ENCRYPTION_KEYS } ] - allow_es_settings_modification= true - elasticsearch.host=https : //es01 : 9200 - elasticsearch.username=elastic - elasticsearch.password=$ { ELASTIC_PASSWORD } - elasticsearch.ssl.enabled= true - elasticsearch.ssl.certificate_authority=/usr/share/enterprise-search/config/certs/ca/ca.crt - kibana.external_url=http : //kibana : 5601 mem_limit: $ { MEM_LIMIT } healthcheck: test: [ "CMD-SHELL" , ] interval: 10s timeout: 10s retries: 120 volumes: certs: driver: local enterprisesearchdata: driver: local esdata01: driver: local kibanadata: driver: local |
Problem: Memory Allocation
In addition to the access rights challenge, we also encountered an issue related to memory allocation. The default settings for the virtual memory map count (vm.max_map_count
) were insufficient for our requirements, leading to errors during the deployment.
Solution: Increasing Memory Allocation
To address the memory allocation problem, we followed the guide provided by Elastic on how to increase the vm.max_map_count
. By executing the command sysctl -w vm.max_map_count=262144
, we set the required value for optimal memory allocation. This adjustment allowed Elastic Enterprise Search to function properly and utilize the necessary resources.
Problem: Version 8 does not deploy correctly and fails to authenticate
When we tried to bring up the docker containers, we got the following problems with access rights:
From ES01: {"@timestamp":"2023-05-22T05:27:10.294Z", "log.level":"ERROR", "message":"failed to retrieve password hash for reserved user [kibana_system]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es01][transport_worker][T#5]","log.logger":"org.elasticsearch.xpack.security.authc.esnative.ReservedRealm","trace.id":"8639207dbfa64551d7a302a1df3bda26","elasticsearch.cluster.uuid":"Zxd3v00YQyqJtFTsDw143w","elasticsearch.node.id":"v1WIc0TUReqqZFcTRzsyOw","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"es-cluster","error.type":"org.elasticsearch.action.UnavailableShardsException", From Kibana: [2023-05-21T13:56:30.497+00:00][ERROR][elasticsearch-service] Unable to retrieve version information from Elasticsearch nodes. security_exception Root causes: security_exception: unable to authenticate user [kibana_system] for REST request [/_nodes?filter_path=nodes.*.version%2Cnodes.*.http.publish_address%2Cnodes.*.ip] [2023-05-21T13:56:30.868+00:00][INFO ][plugins.screenshotting.chromium] Browser executable: /usr/share/kibana/x-pack/plugins/screenshotting/chromium/headless_shell-linux_x64/headless_shell
Solution: Downgrade to version 7
It appears that version 7 handles correctly the access management issues, so we updated our .env
file to the following:
1 2 3 4 5 6 7 8 9 10 | STACK_VERSION=7.17.10 ELASTIC_PASSWORD=7XQ2aR7q2g2Q994msd942sLJoAJ7LUvD KIBANA_PASSWORD=2M7nl2hCH11b7NLht2l8A8ZHBjAKTkxI ES_PORT=9200 CLUSTER_NAME=es-cluster LICENSE=basic MEM_LIMIT=1073741824 KIBANA_PORT=5601 ENTERPRISE_SEARCH_PORT=3002 ENCRYPTION_KEYS=5df1f9f9590ac53b29db6f5eeb9f63be04a43018af3095ef2bfc8a043d38fc7c |
Conclusion
While setting up Run Enterprise Search using Docker Compose can be a straightforward process, it is not uncommon to encounter challenges along the way. In this blog post, we discussed the problems we faced regarding access rights and memory allocation and the steps we took to resolve them.
By modifying the YAML file to ensure proper ownership and permissions for the required volumes and adjusting the memory allocation using the recommended command, we successfully overcame these issues. With our configuration now in place, we enjoy the benefits of Elastic Enterprise Search, leveraging its powerful capabilities for our organization’s search and discovery needs.
We hope that sharing our experience and solutions will assist others in overcoming similar obstacles and make their journey toward implementing Elastic Enterprise Search using Docker Compose smoother.