Tag: GitLab

Performance improvements for GitLab on servers with limited resources

When running GitLab, the default installation enables all services by default. This might result in a poor performance, especially when the server has limited resources like RAM or CPU power.

To improve the performance of your installation, the GitLab documentation already provides some settings that can be adjusted easily.

A detailed description can be found at docs.gitlab.com.

My web server has 6GB of RAM, which results in temporary timeouts – especially when GitLab Runner executed some jobs.

I started with the changes listed below and already saw a huge reduction of loading time and UI response time on my GitLab installation. The following steps have been copied from docs.gitlab.com as a reminder for future setups.


Disable monitoring

GitLab enables all services by default to provide a complete DevOps solution without any additional configuration. Some of the default services, like monitoring, are not essential for GitLab to function and can be disabled to save memory.

In /etc/gitlab/gitlab.rb:

prometheus_monitoring['enable'] = false

We observed 200MB of memory usage reduction configuring GitLab this way.

Configure how GitLab handles memory

GitLab consists of many components (written in Ruby and Go), with GitLab Rails being the biggest one and consuming the most of memory.

GitLab Rails uses jemalloc as a memory allocator. jemalloc preallocates memory in bigger chunks that are also being held for longer periods in order to improve performance. At the expense of some performance loss, you can configure GitLab to free memory right after it is no longer needed instead of holding it for a longer periods.

In /etc/gitlab/gitlab.rb:

gitlab_rails['env'] = { 'MALLOC_CONF' => 'dirty_decay_ms:1000,muzzy_decay_ms:1000' }
gitaly['env'] = { 'MALLOC_CONF' => 'dirty_decay_ms:1000,muzzy_decay_ms:1000' }

We observed much more stable memory usage during the execution of the application.

Disable additional in-application monitoring

GitLab uses internal data structures to measure different aspects of itself. These features are no longer needed if monitoring is disabled.

To disable these features you need to go to Admin Area of GitLab and disable the Prometheus Metrics feature:

  1. On the left sidebar, at the bottom, select Admin Area.
  2. Select Settings > Metrics and profiling.
  3. Expand Metrics – Prometheus.
  4. Disable Enable Prometheus Metrics.
  5. Select Save changes.

 

GitLab CI + Flutter: pub: command not found

In one of my projects, I used a GitLab environment to perform Flutter tests. For this, I setup my .gitlab-ci.yaml to use a Flutter docker image of cirrusci/flutter for code quality check or tests. The file looks like this:

code_quality:
  stage: test
  image: "cirrusci/flutter:stable"
  tags:
    - docker
  before_script:
    - pub global activate dart_code_metrics
    - export PATH="$PATH":"$HOME/.pub-cache/bin"
  script:
    - metrics lib -r codeclimate  > gl-code-quality-report.json
      
# [...]

test:
  stage: test
  image: "cirrusci/flutter:stable"
  tags:
    - docker
  before_script:
    - pub global activate junitreport
    - export PATH="$PATH":"$HOME/.pub-cache/bin"
  script:
    - flutter test --machine --coverage | tojunit -o report.xml
    - lcov --summary coverage/lcov.info
    - genhtml coverage/lcov.info --output=coverage

# [...]

Up to version 2.10.* of the Flutter docker image, this worked fine. But starting with version 3.0.0, there seems to be some changes in the binaries or their paths. The scripts failed with an error:

/usr/bin/bash: line 123: pub: command not found

To fix this error, the pub commands need to be adjusted and set to flutter pub:

# [...]

  before_script:
    - flutter pub global activate dart_code_metrics

# [...]

  before_script:
    - flutter pub global activate junitreport

# [...]

This fixed the issue and all tests finished successfully.

 

GitLab – ERROR: Registering runner… failed, certificate signed by unknown authority

If your self-hosted GitLab server is using a self-signed certificate for https, it might be possible that you get an error during the registration of a GitLab Runner:

ERROR: Registering runner... failed        runner=HbU25D-y status=couldn't execute POST against https://gitlab.example.com/api/v4/runners: Post https://gitlab.example.com/api/v4/runners: x509: certificate signed by unknown authority
PANIC: Failed to register the runner. You may be having network problems.

To solve the problem, you have to provide the full chain certificate *.pem used by your GitLab Server:

gitlab-runner register --tls-ca-file /path/to/fullchain.pem

In my case, the valid certificate could be found on the GitLab server in /etc/gitlab/trusted-certs/fullchain.pem. This one was copied to the GitLab Runner server and used in the command above.

As I did not set up the server on my own, I do not know if this is the default path and filename of a certificate signed by Let’s Encrypt. But in my case, this one worked to register the runner.

Run GitLab console on Synology NAS

As the Synology DSM uses Docker to run GitLab, we can use Docker as well to install GitLab Runner. For this, connect to the Synology using SSH:

ssh <admin-user>@<synology> -p <port>

To connect to the GitLab container, you can use the following command to open:

docker exec -it synology_gitlab /bin/bash

You might adjust the name of the GitLab docker depending on your system.

To open the console, run:

gitlab-rails console

When GItLab is installed using the DSM package manager, just use the following commands:

cd /home/git/gitlab/bin
./rails console production

Commands for the console

Below are some examples how to use the GitLab console.

Check the mail delivery method

ActionMailer::Base.delivery_method

Output might be: => :smtp

Check the smtp settings

ActionMailer::Base.smtp_settings

Output might be: => {:address=>"example.com", :port=>25, …

Testing the SMTP configuration (see documentation)

Notify.test_email('mail@example.com', 'Subject', 'Mail Body').deliver_now

GitLab on Synology: set ‘external_url’

There are two (or even more) solutions to install GitLab on a Synology:

  • Using Docker and the container gitlab/gitlab-ce
  • Using the DSM package manager

Depending on the type of installation, different settings are required to update the external url.

Using Docker container

The external url of GitLab can e defined in /etc/gitlab/gitlab.rb. The parameter takes an url and can also handle a port:

external_url 'http://example.synology.me:30000/'

Important: when a port is specified in external_url, this will override the https/https port where nginx is listening. To use a different port for nginx, this requires an additional setting:

nginx['listen_port'] = 80

After changing this setting, it’s necessary to run:

gitlab-ctl reconfigure

The settings above are necessary, if port routing is set like the following:

Using DSM package manager installation

This installation of GitLab on Synology uses localhost as a default value for external url. This may lead to some problems when accessing GitLab over another IP or host name. In my case, this lead to missing icons and a non functional WebIDE. An inspection of the html page shows, that some resources are requested over http://localhost/... which leads to 404 errors for those resources.

Since the GitLab container on Synology is not based on the omnibus package, you can not use directly external_url in /etc/gitlab/gitlab.rb. If you want to change the url you can do it by changing the docker environment parameter GITLAB_HOST.

GitLab on Synology: Environment settings

Setup GitLab Runner for Docker containers on Synology NAS

The setup described in this post has been tested on the following system:

  • DS216+II with 8GB RAM
  • DSM 6.2.3-25426 Update 2

In addition, the following software packages have already been installed on the system using the Synology package manager:

  • Docker 18.09.0-0513

GitLab is installed via the Docker Registry:

  • gitlab/gitlab-ce:latest (in my case GitLab 13.5.1-ce.0)

Install GitLab

You can skip this part, if GitLab is already running on your Synology and continue with the step Install GitLab Runner.

To install GitLab, open the Docker Registry and search for “gitlab”. Double click the entry gitlab/gitlab-ce:latest and select the latest version:

After the image is loaded, it will be listed under image. Launch this image and set the folders to be mounted as shown in the following image. This will simplify the access to the docker files within your Synology.

Mounted folder / volumes
Port settings

The port settings depend on your system. Normally, HTTP is accessible at port 80 or HTTPS on port 443. If your system already uses other apps that are running on those port, you can adjust them in Port Settings.

After completing the setup, it might take some time until the GitLab web surface is available. When accessing GitLab for the first time, you can specify a password for the root environment. The default username for the admin area is root. Now you can create user accounts, projects and perform any adjustments that fit your needs.

Install GitLab Runner

As the Synology DSM uses Docker to run GitLab, we can use Docker as well to install GitLab Runner. For this, connect to the Synology using SSH:

ssh <admin-user>@<synology> -p <port>

Now we can install the Gitlab Runner Docker container that can run other Docker containers to perform the runner tasks:

docker run -d \
--name gitlab_runner_docker \
--restart always \<br>--network host \
-v /run/docker.sock:/var/run/docker.sock \
gitlab/gitlab-runner:latest

This will install a Docker container with the name gitlab_runner_docker which uses the same network as Docker (‘–network host‘).

As this container is the basis for our Docker-in-Docker setup, we connect the Docker socket of our main container to the new one by using -v /run/docker.sock:/var/run/docker.sock.

You might also use the Docker GUI of DSM to create the container. But the step above can only be done with the help of the console command!

To test the container, you can use the following command to connect to the console:

docker exec -it gitlab_runner_docker /bin/bash

Register the runner

Now it’s time to register the runner:

docker exec -it gitlab_runner_docker gitlab-runner register

This will ask for some input:

Runtime platform                                    arch=amd64 os=linux pid=16 revision=e95f89a0 version=13.4.1
Running in system-mode.                            
                                                   
Please enter the gitlab-ci coordinator URL (e.g. https://gitlab.com/):
http://10.0.6.102:30000/
Please enter the gitlab-ci token for this runner:
ab1234abcd1234abcd12
Please enter the gitlab-ci description for this runner:
[synology]: docker_alpine
Please enter the gitlab-ci tags for this runner (comma separated):

Registering runner... succeeded                     runner=sA4DKorC
Please enter the executor: docker-ssh, parallels, docker+machine, kubernetes, docker, shell, ssh, virtualbox, docker-ssh+machine, custom:
docker
Please enter the default Docker image (e.g. ruby:2.6):
alpine:latest
Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded!

In this case, the executor docker and the base image alpine:latest is used for this container.

If you use a self-signed certificate for GItLab, then you have to specify the certificate with the option --tls-ca-file during registration:

docker exec -it gitlab_runner_docker gitlab-runner register --tls-ca-file /path/to/fullchain.pem

Setup Docker-in-Docker

Let’s connect to the bash terminal of this container to change some settings:

docker exec -it gitlab_runner_docker /bin/bash

The created Docker container is an alpine base system without any packages installed. To perform the changes, we need a simple terminal text editor like nano or vim. The following commands will install nano for this task:

apt-get update
apt-get install nano

Now you can use nano to edit the GitLab Runner config file:

nano /etc/gitlab-runner/config.toml

And add the following lines as highlighted in the config file below:

clone_url = "http://10.0.6.102:30000/"
privileged = true
pull_policy = "if-not-present"
concurrent = 1
check_interval = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "GitLab Runner Docker"
  url = "http://10.0.6.102:30000/"
  clone_url = "http://10.0.6.102:30000/"
  token = "Examp1eT0ken"
  executor = "docker"
  [runners.docker]
    tls_verify = false
    image = "node:latest"
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/cache"]
    shm_size = 0
    privileged = true
    pull_policy = "if-not-present"
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]

That’s it. Now you can use this GitLab Runner for your repositories and run jobs by using other Docker containers.

Example usage in .gitlab-ci.yml

The following example uses a php container to run tests for a simple PHP application and deploy the code to a server. Some of the settings depend on the setup of your PHP application, so the example will not work out of the box. But it can give you a good idea of what is possible.

# https://hub.docker.com/_/php
image: php:7.4-fpm

services:
  - mysql:5.7

variables:
  # Configure mysql environment variables (https://hub.docker.com/_/mysql/)
  MYSQL_USER: $MYSQL_USER
  MYSQL_PASSWORD: $MYSQL_PASSWORD

before_script:
  # Initialize database, etc

stages:
  - test
  - deploy

test:
  stage: test
  script:
    - composer install
    - composer phpunit

deploy_production:
  stage: deploy
  environment:
    name: production
    url: https://www.example.com
  script:
    # run on server 'git checkout master && git pull origin master && exit'
  only:
    - master

Photo by sergio souza on Unsplash