Installing ECE - Offline

Installing Elastic Cloud Enterprise - Offline

How to install Elastic Cloud Enterprise on your own AWS EC2 Instance running RHEL7 using your private Docker registry.

References

• https://www.elastic.co/guide/en/cloud-enterprise/current/ece-installing-offline.html
• https://www.elastic.co/guide/en/cloud-enterprise/current/ece-prereqs.html
• https://www.elastic.co/guide/en/cloud-enterprise/current/ece-uninstall.html
• https://discuss.elastic.co/t/uid-gid-error-on-install/142633
• http://embshd.blogspot.com/2018/08/installing-private-docker-registry.html
• https://success.docker.com/article/using-systemd-to-control-the-docker-daemon
• https://www.elastic.co/guide/en/cloud-enterprise/current/ece-retrieve-passwords.html

Setup

1. Create groups (ECE cannot be installed with UID or GID less than 1000)
   

   sudo groupadd -g 1010 elastic
   sudo groupadd -g 1011 docker
   

2. Create user, elastic and add it to groups wheel and docker
   

   sudo useradd -g elastic -M -N -u 1010 elastic
   sudo usermod -aG wheel elastic
   sudo usermod -aG docker elastic
   sudo usermod -L elastic
   

3. Check result of user elastic
   

   sudo su elastic
   id
   

4. Expected result
   

   1 2 3 4

   uid=1010(elastic) gid=1010(elastic) groups=1010(elastic),10(wheel),1011(docker) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023

5. Install any patches and install docker
   

   sudo yum update
   sudo yum install docker
   

6. Set SELinux to permissive
7. Add Cert Authority public certs (if we're using Self-Signed cert then just copy the Public cert and rename it as ca.crt)
   

   1 2 3 4 5 6

   cd /etc/docker/certs.d sudo mkdir 10.10.10.10:443 sudo chmod 755 10.10.10.10:443 cd 10.10.10.10:443 sudo touch ca.crt sudo chmod 666 ca.crt

8. Make /mnt/data and /mnt/data/docker available for elastic user
   

   sudo install -o elastic -g elastic -d -m 700 /mnt/data
   sudo install -o elastic -g elastic -d -m 700 /mnt/data/docker
   

9. Enable docker debugging by editing /etc/docker/daemon.json
   

   {
     "debug": true
   }
   

10. Configure Docker Daemon options here (/etc/systemd/system/docker.service.d/docker.conf), create this directory and file
    

    mkdir /etc/systemd/system/docker.service.d
    touch /etc/systemd/system/docker.service.d/docker.conf
    

11. Add following lines to the above file (172.17.42.1/16 is private bridge for Docker)
    

    [Unit]
    Description=Docker Service
    After=multi-user.target
    
    [Service]
    ExecStart=
    ExecStart=/usr/bin/docker daemon -g /mnt/data/docker --storage-driver=overlay --bip=172.17.42.1/16
    

12. Add to path
    

    export PATH=$PATH:/usr/bin/docker:/mnt/data/docker
    

13. Add link
    

    ln -s /usr/libexec/docker/docker-proxy-current /usr/bin/docker-proxy
    

14. Edit /etc/sysctl.conf, 32 GB is minium requirement
1. vm.max_map_count should be 1 per 128KB of system memory
1. 262144 = 32 GB
2. 524288 = 64GB
3. 1048576 = 128 GB
4. 2097152 = 256GB
2. Once updated, reload it
   

   sysctl -p
   

15. Verify that fs.may_detach_mounts = 1 in /etc/sysctl.conf
    

    cat /proc/sys/fs/may_detach_mounts
    

16. Verify that net.ipv4.ip_forware = 1 in /etc/sysctl.conf
    

    cat /proc/sys/net/ipv4/ip_forward
    

17. Edit /etc/security/limits.conf
    

    *                soft    nofile         1024000
    *                hard    nofile         1024000
    *                soft    memlock        unlimited
    *                hard    memlock        unlimited
    elastic          soft    nofile         1024000
    elastic          hard    nofile         1024000
    elastic          soft    memlock        unlimited
    elastic          hard    memlock        unlimited
    root             soft    nofile         1024000
    root             hard    nofile         1024000
    root             soft    memlock        unlimited
    

18. Register and start docker service
    

    sudo systemctl daemon-reload
    sudo systemctl enable docker.service
    sudo systemctl start docker.service
    

19. Obtain the ECE install script from Elastic and update file permission
    

    sudo chmod 777 elastic-cloud-enterprise.sh
    

20. Run it with --docker-registry flag
    

    sudo su elastic
    bash elastic-cloud-enterprise.sh install --docker-registry 10.105.142.17:443 --debug
    

21. Expected Result
    
22. I did not get the expected output of Admin password since timed out. See /mnt/data/elastic/logs/. Instead I had to manually pull this information out of the json file. See step 24.
    

    [2018-08-23 18:37:41,204][INFO ][no.found.bootstrap.BootstrapInitial] Creating Admin Console Elasticsearch backend {}
    [2018-08-23 18:37:41,451][INFO ][no.found.bootstrap.ServiceLayerBootstrap] Waiting for [ensuring-plan] to complete. Retrying every [1 second] (cause: [org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /clusters/42953c05d2f243a5baa8c3047c710f95/plans/status]) {}
    [2018-08-23 18:37:48,637][INFO ][no.found.bootstrap.ServiceLayerBootstrap] Waiting for [ensuring-plan] to complete. Retrying every [1 second] (cause: [java.lang.Exception: not yet started]) {}
    [2018-08-23 19:07:41,323][ERROR][no.found.bootstrap.BootstrapInitial$] Unhandled error. {}
    java.util.concurrent.TimeoutException: Futures timed out after [30 minutes]
            at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:223)
            at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:227)
            at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:190)
            at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
            at scala.concurrent.Await$.result(package.scala:190)
            at no.found.bootstrap.BootstrapInitial.bootstrapServiceLayer(BootstrapInitial.scala:880)
            at no.found.bootstrap.BootstrapInitial.bootstrap(BootstrapInitial.scala:650)
            at no.found.bootstrap.BootstrapInitial$.delayedEndpoint$no$found$bootstrap$BootstrapInitial$1(BootstrapInitial.scala:1215)
            at no.found.bootstrap.BootstrapInitial$delayedInit$body.apply(BootstrapInitial.scala:1209)
            at scala.Function0$class.apply$mcV$sp(Function0.scala:34)
            at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
            at scala.App$$anonfun$main$1.apply(App.scala:76)
            at scala.App$$anonfun$main$1.apply(App.scala:76)
            at scala.collection.immutable.List.foreach(List.scala:392)
            at scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:35)
            at scala.App$class.main(App.scala:76)
            at no.found.bootstrap.BootstrapInitial$.main(BootstrapInitial.scala:1209)
            at no.found.bootstrap.BootstrapInitial.main(BootstrapInitial.scala)
    

23. At the end, I was left with 14 Containers where 13 are kept running and 1 is exited.
    

    frc-cloud-uis-cloud-ui                  Up 0.0.0.0:12400->5601/tcp , 0.0.0.0:12443->5643/tcp
    frc-admin-consoles-admin-console  Up 0.0.0.0:12300->12300/tc p, 0.0.0.0:12343->12343/tcp
    frc-curators-curator                  Up  
    frc-constructors-constructor                 Up  
    frc-services-forwarders-services-forwarder Up 0.0.0.0:9244->9244/tcp, 0.0.0.0:12344->12344/tcp
    frc-beats-runners-beats-runner   Up  
    frc-allocators-allocator   Up  
    frc-directors-director    Up 0.0.0.0:2112->2112/tcp 
    frc-proxies-proxy    Up 0.0.0.0:9200->9200/tcp, 0.0.0.0:9243->9243/tcp, 0.0.0.0:9300->9300/tcp, 0.0.0.0:9343->9343/tcp
    frc-blueprints-blueprint   Up  
    frc-runners-runner    Up  
    frc-client-forwarders-client-forwarder  Up  
    frc-zookeeper-servers-zookeeper   Up 0.0.0.0:2191->2191/tcp, 0.0.0.0:12191->12191/tcp, 0.0.0.0:12898->12898/tcp, 0.0.0.0:13898->13898/tcp
    elastic-cloud-enterprise-bootstrap-1.1.4 Exit  
    

24. Install jq
    

    yum install jq
    

25. Retrieve password
    

    jq -r '.adminconsole_root_password' /mnt/data/elastic/bootstrap-state/bootstrap-secrets.json

26. Go to http://127.0.0.1:12400 and Log in as "root"
    
27. If something goes wrong, you can retry after removing containers and images
    

    1 2 3 4

    docker stop $(docker ps -a -q) docker rm -f frc-runners-runner frc-allocators-allocator $(docker ps -a -q) docker rmi $(docker images -a -q) sudo rm -rf /mnt/data/elastic/*

28. You can find install logs here:
    

    /mnt/data/elastic/logs/