Good day
We have several cloud clusters with a large number of virtual machines in each. All this business is hosted at Hetzner'e. In each cluster, we have one master machine, a snapshot is taken from it and automatically distributed to all virtual machines within the cluster.
This scheme does not allow us to use gitlab-runners normally, since a lot of problems arise when many identical registered runners appear, which prompted us to find a workaround and to write this article / manual.
This is probably not the best practice, but this solution seemed as convenient and simple as possible.
For the tutorial, I ask for a cat.
Required packages on the master machine:- python
- git
- file with ssh keys
The general principle of implementing automatic gut pull on all virtual machines is that you need a machine on which Ansible will be installed. From this machine, ansible will send git pull commands and restart the service that has been updated. We created a separate virtual machine outside the clusters for these purposes, and installed on it:
- python
- ansible
- gitlab-runner
From organizational issues - you need to register gitlab-runner, make ssh-keygen, drop the public ssh key of this machine in
.ssh/authorized_keys on the master machine, open port 22 for ansible on the master machine.
Now configure ansible
Since our goal is to automate everything that is possible. In the
/etc/ansible/ansible.cfg file
/etc/ansible/ansible.cfg we uncomment the line
host_key_checking = False so that ansible does not ask for confirmation of new machines.
Next, you need to automatically generate an inventory file for ansible, from where it will pick up the ip of the machines on which you need to do git pull.
We generate this file using the Hetzner API, you can take the list of hosts from your AWS, Asure, database (you have an API somewhere to display your running machines, right?).
The structure of the inventory file is very important for Ansible, its appearance should be as follows:
[] ip- ip- [2] ip- ip-
To generate such a file, let's make a simple script (let's call it
vm_list ):
It's time to check that ansible works and is friendly with the recipient of ip addresses:
/etc/ansible/./vm_list && ansible -i /etc/ansible/cloud_ip -m shell -a 'hostname' group
The output should receive the hostnames of the machines on which the command was executed.
A few words about the syntax:
- /etc/ansible/./vm_list - generate a list of machines
- -i - absolute path to the inventory file
- -m - tell the ansible to use the shell module
- -a is an argument. Any team can be entered here.
- group is the name of your cluster. If you need to do on all clusters, change group to all
Go ahead - try to do git pull on our virtual machines:
/etc/ansible/./vm_list && ansible -i /etc/ansible/cloud_ip -m shell -a 'cd /path/to/project && git pull' group
If we see already up to date or unloading from the repository in the output, then everything works.
Now what it was all meant for
We teach our script to be executed automatically when committing in the master branch in gitlab
First, we’ll make our script more beautiful and put it in an executable file (let's call it exec_pull) -
We go to our gitlab and in the project we create the file
.gitlab-ci.ymlInside we put the following:
variables: GIT_STRATEGY: none VM_GROUP: group stages: - pull - restart run_exec_pull: stage: pull script: - /etc/ansible/exec_pull 'cd /path/to/project/'$CI_PROJECT_NAME' && git pull' $VM_GROUP only: - master run_service_restart: stage: restart script: - /etc/ansible/exec_pull 'your_app_stop && your_app_start' $VM_GROUP only: - master
Everything is ready. Now -
- make commit
- rejoice that everything works
When transferring .yml to other projects, you just need to change the service name for restart and the name of the cluster on which the ansible commands will be executed.