Category: GitLab

Performance improvements for GitLab on servers with limited resources

When running GitLab, the default installation enables all services by default. This might result in a poor performance, especially when the server has limited resources like RAM or CPU power.

To improve the performance of your installation, the GitLab documentation already provides some settings that can be adjusted easily.

A detailed description can be found at docs.gitlab.com.

My web server has 6GB of RAM, which results in temporary timeouts – especially when GitLab Runner executed some jobs.

I started with the changes listed below and already saw a huge reduction of loading time and UI response time on my GitLab installation. The following steps have been copied from docs.gitlab.com as a reminder for future setups.


Disable monitoring

GitLab enables all services by default to provide a complete DevOps solution without any additional configuration. Some of the default services, like monitoring, are not essential for GitLab to function and can be disabled to save memory.

In /etc/gitlab/gitlab.rb:

prometheus_monitoring['enable'] = false

We observed 200MB of memory usage reduction configuring GitLab this way.

Configure how GitLab handles memory

GitLab consists of many components (written in Ruby and Go), with GitLab Rails being the biggest one and consuming the most of memory.

GitLab Rails uses jemalloc as a memory allocator. jemalloc preallocates memory in bigger chunks that are also being held for longer periods in order to improve performance. At the expense of some performance loss, you can configure GitLab to free memory right after it is no longer needed instead of holding it for a longer periods.

In /etc/gitlab/gitlab.rb:

gitlab_rails['env'] = { 'MALLOC_CONF' => 'dirty_decay_ms:1000,muzzy_decay_ms:1000' }
gitaly['env'] = { 'MALLOC_CONF' => 'dirty_decay_ms:1000,muzzy_decay_ms:1000' }

We observed much more stable memory usage during the execution of the application.

Disable additional in-application monitoring

GitLab uses internal data structures to measure different aspects of itself. These features are no longer needed if monitoring is disabled.

To disable these features you need to go to Admin Area of GitLab and disable the Prometheus Metrics feature:

  1. On the left sidebar, at the bottom, select Admin Area.
  2. Select Settings > Metrics and profiling.
  3. Expand Metrics – Prometheus.
  4. Disable Enable Prometheus Metrics.
  5. Select Save changes.

 

Install and update GitLab Runner

runner at start block

I already wrote a summary on how to Setup GitLab Runner for Docker containers on Synology NAS. As this article has a lot of details for the Synology setup, I decided to write a short summary for this topic on a regular Linux server. So let’s go…

Before we start, we need to have Docker Engine installed on our system. Check out the Docker docs on how to install the Docker Engine on your system.

Install GitLab Runner

To install GitLab Runner, simply pull the container by using:

docker pull gitlab/gitlab-runner:latest

Now you can start the container as described in the section Start GitLab Runner below.

Update GitLab Runner

To update GitLab Runner, you also have to pull the container by using:

docker pull gitlab/gitlab-runner:latest

Then you have to stop and remove the current container image:

docker stop gitlab_runner
docker rm gitlab_runner

Now you can start the new container as described below.

Start GitLab Runner

docker run --detach \
  --name gitlab_runner \
  --restart always \
  --network host \
  -v /run/docker.sock:/var/run/docker.sock \
  gitlab/gitlab-runner:latest

And finally you have to register the runner in your GitLab instance. In the following commands, you need to set your own values for <host.example.com> and <registration_token>:

docker exec -it gitlab_runner gitlab-runner register \
  --url https://<host.example.com>/ \
  --registration-token <registration_token>

The registration will ask for some input. In my case, I used a docker executor with the alpine:latest image:

untime platform                                    arch=amd64 os=linux pid=16 revision=e95f89a0 version=13.4.1
Running in system-mode.                            
                                                   
Please enter the gitlab-ci coordinator URL (e.g. https://gitlab.com/):
https://<host.example.com>/
Please enter the gitlab-ci token for this runner:
<registration_token>
Please enter the gitlab-ci description for this runner:
[synology]: docker_alpine
Please enter the gitlab-ci tags for this runner (comma separated):

Registering runner... succeeded                     runner=sA4DKorC
Please enter the executor: docker-ssh, parallels, docker+machine, kubernetes, docker, shell, ssh, virtualbox, docker-ssh+machine, custom:
docker
Please enter the default Docker image (e.g. ruby:2.6):
alpine:latest
Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded!

Register GitLab Runner in GitLab >= 16.0

In GitLab 16.0 (starting in 15.1 as feature), registering a runner using registration tokens got deprecated. For newer versions, the registration process located in the Admin area can be used. It’s available at “Admin Area > CI/CD > Runners”. Clicking on “New instance runner” shows the following form:

Submitting the form will create a temporary token in the background and show the registration command to be used for a new runner. The command shown is similar to the registration command above but without the need for manually creating the tokens.

 

GitLab CI + Flutter: pub: command not found

In one of my projects, I used a GitLab environment to perform Flutter tests. For this, I setup my .gitlab-ci.yaml to use a Flutter docker image of cirrusci/flutter for code quality check or tests. The file looks like this:

code_quality:
  stage: test
  image: "cirrusci/flutter:stable"
  tags:
    - docker
  before_script:
    - pub global activate dart_code_metrics
    - export PATH="$PATH":"$HOME/.pub-cache/bin"
  script:
    - metrics lib -r codeclimate  > gl-code-quality-report.json
      
# [...]

test:
  stage: test
  image: "cirrusci/flutter:stable"
  tags:
    - docker
  before_script:
    - pub global activate junitreport
    - export PATH="$PATH":"$HOME/.pub-cache/bin"
  script:
    - flutter test --machine --coverage | tojunit -o report.xml
    - lcov --summary coverage/lcov.info
    - genhtml coverage/lcov.info --output=coverage

# [...]

Up to version 2.10.* of the Flutter docker image, this worked fine. But starting with version 3.0.0, there seems to be some changes in the binaries or their paths. The scripts failed with an error:

/usr/bin/bash: line 123: pub: command not found

To fix this error, the pub commands need to be adjusted and set to flutter pub:

# [...]

  before_script:
    - flutter pub global activate dart_code_metrics

# [...]

  before_script:
    - flutter pub global activate junitreport

# [...]

This fixed the issue and all tests finished successfully.

How to Run and Update Docker Images and Containers

The following text shows the default workflow to run and update a Docker image based on the example of GitLab. But this should also work for other containers.

Docker Installation

First make sure that your local machine has Docker installed. I use Debian and for this, a detailed description is available in Dockers documentation. There are also instructions for CentOS, Ubuntu, Fedora and others.

You can check the installed Docker version:

docker version
Client: Docker Engine - Community
 Version:           20.10.11
 API version:       1.41
 ... 

Run a Docker Container

Let’s start with an environment variable that defines the path where all the Gitlab data is stored. This is outside of the Docker container which allows an easy update without loosing data. To define the variable $GITLAB_HOME, you can use:

export GITLAB_HOME="/srv/gitlab"

To run a Docker container, simply use the run command with your parameters. For GitLab, this might look like:

docker run --detach
  --hostname <host.example.com>
  --publish 443:443 --publish 80:80 --publish 2222:22
  --name gitlab
  --restart always  
  --volume $GITLAB_HOME/config:/etc/gitlab 
  --volume $GITLAB_HOME/logs:/var/log/gitlab
  --volume $GITLAB_HOME/data:/var/opt/gitlab
  gitlab/gitlab-ce:latest

You should use your own settings and replace <host.example.com> with your setup.

For the sake of completeness, you might also need a runner for GitLab. Let’s run this container as well:

docker run --detach 
  --name gitlab_runner
  --restart always 
  --network host 
  -v /run/docker.sock:/var/run/docker.sock 
  gitlab/gitlab-runner:latest

And this one needs to be registered in your GitLab instance:

docker exec -it gitlab_runner gitlab-runner register 
  --url https://<host.example.com>/
  --registration-token <registration_token>

More details about the configuration of the GitLab runner is descriped in this post.

Update a Docker Container to the Latest Version

Let’s check, which container are running by using:

docker ps

The output shows all your containers and their ids:

CONTAINER ID   IMAGE                         COMMAND                  CREATED       STATUS                 NAMES
02fe426970f1   gitlab/gitlab-ce:latest       "/assets/wrapper"        4 hours ago   Up 4 hours (healthy)   gitlab
5742e07f809b   gitlab/gitlab-runner:latest   "/usr/bin/dumb-init …"   4 days ago    Up 5 hours             gitlab_runner

We have to stop the container and remove the image. When you used a run command (for GitLab) as shown above, this should not affect the data as this is stored outside of the container.

docker stop 02fe426970f1
docker rm 02fe426970f1

To remove the associated image, you can use the docker rmi. For this, we need to find the id of this image by listing all images:

docker images
REPOSITORY         TAG      IMAGE ID       CREATED        SIZE
gitlab/gitlab-ce   latest   46cd6954564a   17 hours ago   2.36GB
mysql              5.7      c20987f18b13   2 weeks ago    448MB
cirrusci/flutter   2.8.1    98a87781f179   2 weeks ago    3.49GB

Now we can remove the image (in my case the gitlab-ce image):

docker rmi 46cd6954564a

The update of the image can be done by using pull. This will download the images (in this case the latest version) and stores it on your system:

docker pull gitlab/gitlab-ce:latest

After this, you can use the same run command as before to run a container with the updated image:

docker run --detach
  --hostname <host.example.com>
  --publish 443:443 --publish 80:80 --publish 2222:22
  --name gitlab
  --restart always  
  --volume $GITLAB_HOME/config:/etc/gitlab 
  --volume $GITLAB_HOME/logs:/var/log/gitlab
  --volume $GITLAB_HOME/data:/var/opt/gitlab
  gitlab/gitlab-ce:latest

A final check, shows the container is up and running:

docker ps
CONTAINER ID   IMAGE                         COMMAND                  CREATED        STATUS                 NAMES
f3dab1163c07   gitlab/gitlab-ce:latest       "/assets/wrapper"        2 minutes ago  Up 4 hours (healthy)   gitlab
5742e07f809b   gitlab/gitlab-runner:latest   "/usr/bin/dumb-init …"   4 days ago     Up 5 hours             gitlab_runner

Instead of automatically using the latest version of each image (e.g. gitlab/gitlab-ce:latest) you can also specify a version number like gitlab/gitlab-ce:14.5.2. This is helpful when you need to update a system in steps. Or if you need to follow a specific upgrade path, which is the case for GitLab.

Photo by Ian Taylor on Unsplash.

GitLab – ERROR: Registering runner… failed, certificate signed by unknown authority

If your self-hosted GitLab server is using a self-signed certificate for https, it might be possible that you get an error during the registration of a GitLab Runner:

ERROR: Registering runner... failed        runner=HbU25D-y status=couldn't execute POST against https://gitlab.example.com/api/v4/runners: Post https://gitlab.example.com/api/v4/runners: x509: certificate signed by unknown authority
PANIC: Failed to register the runner. You may be having network problems.

To solve the problem, you have to provide the full chain certificate *.pem used by your GitLab Server:

gitlab-runner register --tls-ca-file /path/to/fullchain.pem

In my case, the valid certificate could be found on the GitLab server in /etc/gitlab/trusted-certs/fullchain.pem. This one was copied to the GitLab Runner server and used in the command above.

As I did not set up the server on my own, I do not know if this is the default path and filename of a certificate signed by Let’s Encrypt. But in my case, this one worked to register the runner.

Run GitLab console on Synology NAS

As the Synology DSM uses Docker to run GitLab, we can use Docker as well to install GitLab Runner. For this, connect to the Synology using SSH:

ssh <admin-user>@<synology> -p <port>

To connect to the GitLab container, you can use the following command to open:

docker exec -it synology_gitlab /bin/bash

You might adjust the name of the GitLab docker depending on your system.

To open the console, run:

gitlab-rails console

When GItLab is installed using the DSM package manager, just use the following commands:

cd /home/git/gitlab/bin
./rails console production

Commands for the console

Below are some examples how to use the GitLab console.

Check the mail delivery method

ActionMailer::Base.delivery_method

Output might be: => :smtp

Check the smtp settings

ActionMailer::Base.smtp_settings

Output might be: => {:address=>"example.com", :port=>25, …

Testing the SMTP configuration (see documentation)

Notify.test_email('mail@example.com', 'Subject', 'Mail Body').deliver_now

GitLab on Synology: set ‘external_url’

There are two (or even more) solutions to install GitLab on a Synology:

  • Using Docker and the container gitlab/gitlab-ce
  • Using the DSM package manager

Depending on the type of installation, different settings are required to update the external url.

Using Docker container

The external url of GitLab can e defined in /etc/gitlab/gitlab.rb. The parameter takes an url and can also handle a port:

external_url 'http://example.synology.me:30000/'

Important: when a port is specified in external_url, this will override the https/https port where nginx is listening. To use a different port for nginx, this requires an additional setting:

nginx['listen_port'] = 80

After changing this setting, it’s necessary to run:

gitlab-ctl reconfigure

The settings above are necessary, if port routing is set like the following:

Using DSM package manager installation

This installation of GitLab on Synology uses localhost as a default value for external url. This may lead to some problems when accessing GitLab over another IP or host name. In my case, this lead to missing icons and a non functional WebIDE. An inspection of the html page shows, that some resources are requested over http://localhost/... which leads to 404 errors for those resources.

Since the GitLab container on Synology is not based on the omnibus package, you can not use directly external_url in /etc/gitlab/gitlab.rb. If you want to change the url you can do it by changing the docker environment parameter GITLAB_HOST.

GitLab on Synology: Environment settings