Category: Software Development

Performance improvements for GitLab on servers with limited resources

When running GitLab, the default installation enables all services by default. This might result in a poor performance, especially when the server has limited resources like RAM or CPU power.

To improve the performance of your installation, the GitLab documentation already provides some settings that can be adjusted easily.

A detailed description can be found at docs.gitlab.com.

My web server has 6GB of RAM, which results in temporary timeouts – especially when GitLab Runner executed some jobs.

I started with the changes listed below and already saw a huge reduction of loading time and UI response time on my GitLab installation. The following steps have been copied from docs.gitlab.com as a reminder for future setups.


Disable monitoring

GitLab enables all services by default to provide a complete DevOps solution without any additional configuration. Some of the default services, like monitoring, are not essential for GitLab to function and can be disabled to save memory.

In /etc/gitlab/gitlab.rb:

prometheus_monitoring['enable'] = false

We observed 200MB of memory usage reduction configuring GitLab this way.

Configure how GitLab handles memory

GitLab consists of many components (written in Ruby and Go), with GitLab Rails being the biggest one and consuming the most of memory.

GitLab Rails uses jemalloc as a memory allocator. jemalloc preallocates memory in bigger chunks that are also being held for longer periods in order to improve performance. At the expense of some performance loss, you can configure GitLab to free memory right after it is no longer needed instead of holding it for a longer periods.

In /etc/gitlab/gitlab.rb:

gitlab_rails['env'] = { 'MALLOC_CONF' => 'dirty_decay_ms:1000,muzzy_decay_ms:1000' }
gitaly['env'] = { 'MALLOC_CONF' => 'dirty_decay_ms:1000,muzzy_decay_ms:1000' }

We observed much more stable memory usage during the execution of the application.

Disable additional in-application monitoring

GitLab uses internal data structures to measure different aspects of itself. These features are no longer needed if monitoring is disabled.

To disable these features you need to go to Admin Area of GitLab and disable the Prometheus Metrics feature:

  1. On the left sidebar, at the bottom, select Admin Area.
  2. Select Settings > Metrics and profiling.
  3. Expand Metrics – Prometheus.
  4. Disable Enable Prometheus Metrics.
  5. Select Save changes.

 

Install and update GitLab Runner

runner at start block

I already wrote a summary on how to Setup GitLab Runner for Docker containers on Synology NAS. As this article has a lot of details for the Synology setup, I decided to write a short summary for this topic on a regular Linux server. So let’s go…

Before we start, we need to have Docker Engine installed on our system. Check out the Docker docs on how to install the Docker Engine on your system.

Install GitLab Runner

To install GitLab Runner, simply pull the container by using:

docker pull gitlab/gitlab-runner:latest

Now you can start the container as described in the section Start GitLab Runner below.

Update GitLab Runner

To update GitLab Runner, you also have to pull the container by using:

docker pull gitlab/gitlab-runner:latest

Then you have to stop and remove the current container image:

docker stop gitlab_runner
docker rm gitlab_runner

Now you can start the new container as described below.

Start GitLab Runner

docker run --detach \
  --name gitlab_runner \
  --restart always \
  --network host \
  -v /run/docker.sock:/var/run/docker.sock \
  gitlab/gitlab-runner:latest

And finally you have to register the runner in your GitLab instance. In the following commands, you need to set your own values for <host.example.com> and <registration_token>:

docker exec -it gitlab_runner gitlab-runner register \
  --url https://<host.example.com>/ \
  --registration-token <registration_token>

The registration will ask for some input. In my case, I used a docker executor with the alpine:latest image:

untime platform                                    arch=amd64 os=linux pid=16 revision=e95f89a0 version=13.4.1
Running in system-mode.                            
                                                   
Please enter the gitlab-ci coordinator URL (e.g. https://gitlab.com/):
https://<host.example.com>/
Please enter the gitlab-ci token for this runner:
<registration_token>
Please enter the gitlab-ci description for this runner:
[synology]: docker_alpine
Please enter the gitlab-ci tags for this runner (comma separated):

Registering runner... succeeded                     runner=sA4DKorC
Please enter the executor: docker-ssh, parallels, docker+machine, kubernetes, docker, shell, ssh, virtualbox, docker-ssh+machine, custom:
docker
Please enter the default Docker image (e.g. ruby:2.6):
alpine:latest
Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded!

Register GitLab Runner in GitLab >= 16.0

In GitLab 16.0 (starting in 15.1 as feature), registering a runner using registration tokens got deprecated. For newer versions, the registration process located in the Admin area can be used. It’s available at “Admin Area > CI/CD > Runners”. Clicking on “New instance runner” shows the following form:

Submitting the form will create a temporary token in the background and show the registration command to be used for a new runner. The command shown is similar to the registration command above but without the need for manually creating the tokens.

 

Docker: how to build and push a Git repository to Docker Hub

To build and push a Git repository to Docker Hub, you can follow these steps:

Step 1 – Create a Dockerfile in the root directory of your Git repository. This file contains instructions for building your Docker image.

Step 2 – Build your Docker image using the docker build command. You will need to provide a name and tag for your image. For example:

docker build -t yourusername/yourimage:tag .

This command will build an image with the tag yourusername/yourimage:tag. The path provided as . is the current path.

Step 3 – Log in to Docker Hub using the docker login command. You will need to provide your Docker Hub username and password.

docker login

Step 4 – Push your Docker image to Docker Hub using the docker push command. For example:

docker push yourusername/yourimage:tag

This command will push your image to Docker Hub with the tag yourusername/yourimage:tag.

Once you have completed these steps, your Docker image will be available on Docker Hub for others to use and download.

Is it possible to use ChatGPT for Content Creation?

colorful 3d object

In recent years, advancements in Artificial Intelligence have resulted in the creation of language models like OpenAI’s ChatGPT. These models have the ability to generate human-like text and have already been used for various purposes such as customer service and content moderation. But, is it possible to use ChatGPT for content creation? Let’s find out.

The Potential of AI Language Models

ChatGPT is a pre-trained transformer-based language model that uses a deep neural network to generate text based on the input provided. It has been trained on a vast amount of diverse text data and has the ability to generate human-like text, making it an ideal tool for content creation.

One of the biggest advantages of using AI language models like ChatGPT for content creation is their speed and efficiency. They can generate a large amount of text in a fraction of the time it would take a human to do so. This can be especially useful for businesses that need to create a large amount of content in a short amount of time.

Another advantage of using ChatGPT for content creation is its ability to generate a wide range of text styles, including formal, informal, technical, and creative. This versatility makes it possible for businesses to create content that matches their brand’s tone and style.

Limitations of ChatGPT for Content Creation

However, it’s important to note that while ChatGPT has the potential to be used for content creation, it is not without its limitations. One of the biggest limitations is its lack of creativity. AI language models like ChatGPT are not capable of coming up with original ideas or concepts. They can only generate text based on the data they have been trained on.

Another limitation of ChatGPT for content creation is its tendency to generate text that is grammatically correct but may not make sense in the context. This is because the model has been trained on a vast amount of data and may not always understand the context in which the text is being generated.

Finally, ChatGPT, like any AI language model, is only as good as the data it has been trained on. If the training data is biased or inaccurate, the generated text may also be biased or inaccurate.

The Future of Content Creation with ChatGPT

Despite its limitations, the use of ChatGPT for content creation is still in its early stages, and it has the potential to revolutionize the way content is created in the future. With continued advancements in AI, it is likely that the limitations of ChatGPT will be overcome, and it will become an even more powerful tool for content creation.

As AI language models like ChatGPT become more sophisticated, they will be able to understand context and generate content that is more relevant and meaningful. They will also be able to incorporate original ideas and creativity into their text, making it even more valuable for businesses looking to create high-quality content.

Frequently Asked Questions (FAQs)

  1. What is ChatGPT?
    ChatGPT is an AI language model developed by OpenAI. It uses a deep neural network to generate text based on the input provided.
  2. Can ChatGPT be used for content creation?
    Yes, ChatGPT has the potential to be used for content creation. It can generate a large amount of text in a fraction of the time it would take a human to do so and has the ability to generate a wide range of text styles.
  3. What are the limitations of using ChatGPT for content creation?
    The main limitations of using ChatGPT for content creation are its lack of creativity, tendency to generate text that may not make sense in context, and dependence on the quality of the training data.
  4. Is the use of ChatGPT for content creation still in its early stages?
    Yes, the use of ChatGPT for content creation is still in its early stages. However, with continued advancements in AI, it has the potential to revolutionize the way content is created.

Conclusion

In conclusion, it is possible to use ChatGPT for content creation, and it has the potential to revolutionize the industry. However, it is important to keep in mind its limitations and limitations of AI language models in general. As AI continues to advance, ChatGPT has the potential to become an even more valuable tool for businesses looking to create high-quality content. So, the answer to the question “Is it possible to use ChatGPT for content creation?” is yes, but with caution and consideration of its limitations.

Did you know? This article was already created by ChatGPT. Maybe this partly answers the question of this post 😉

Foto von DeepMind auf Unsplash

GitLab CI + Flutter: pub: command not found

In one of my projects, I used a GitLab environment to perform Flutter tests. For this, I setup my .gitlab-ci.yaml to use a Flutter docker image of cirrusci/flutter for code quality check or tests. The file looks like this:

code_quality:
  stage: test
  image: "cirrusci/flutter:stable"
  tags:
    - docker
  before_script:
    - pub global activate dart_code_metrics
    - export PATH="$PATH":"$HOME/.pub-cache/bin"
  script:
    - metrics lib -r codeclimate  > gl-code-quality-report.json
      
# [...]

test:
  stage: test
  image: "cirrusci/flutter:stable"
  tags:
    - docker
  before_script:
    - pub global activate junitreport
    - export PATH="$PATH":"$HOME/.pub-cache/bin"
  script:
    - flutter test --machine --coverage | tojunit -o report.xml
    - lcov --summary coverage/lcov.info
    - genhtml coverage/lcov.info --output=coverage

# [...]

Up to version 2.10.* of the Flutter docker image, this worked fine. But starting with version 3.0.0, there seems to be some changes in the binaries or their paths. The scripts failed with an error:

/usr/bin/bash: line 123: pub: command not found

To fix this error, the pub commands need to be adjusted and set to flutter pub:

# [...]

  before_script:
    - flutter pub global activate dart_code_metrics

# [...]

  before_script:
    - flutter pub global activate junitreport

# [...]

This fixed the issue and all tests finished successfully.

Testing Googles Indexing of www.frontiersin.org

https://unsplash.com/@mjessier

On 15th of December 2021 a paper was published on www.frontiersin.org: “The Impact of the COVID-19 Pandemic on Avoidance of Health Care, Symptom Severity, and Mental Well-Being in Patients With Coronary Artery Disease”.

Frontiers is a leading Open Access Publisher and Open Science Platform with multiple peer-reviewed and editorial board led journals.

Neither Google, Bing nor DuckDuckGo did index it until today (24th of December). The paper was posted to ResearchGate on 17th of December. The ResearchGate page was indexed within days by Google, Bing and DuckDuckGo.

Now let’s see how long it will take the search engines to index this post (first published on 24th of December). To test this I will search for “Neither Google, Bing nor DuckDuckGo” and the paper title.

Update 26th of December:

This post was indexed by DuckDuckGo. Google and Bing do not yet show it in their results when searching for “Neither Google, Bing nor DuckDuckGo”. None of the search engines finds the paper.

Update 27th of December:

The Frontiers paper and this post was indexed by Bing. DuckDuckGo now also links to Frontiers when searching for the title. No results from Google yet.

Update 29th of December:

Google indexed this post. However, it still doesn’t find the paper on frontiersin.org

Update 4th ofJanuary:

Google finally indexed the frontiersin.org page.

Conclusion: FrontiersIn.org admins need to look into their SEO.

How to Run and Update Docker Images and Containers

The following text shows the default workflow to run and update a Docker image based on the example of GitLab. But this should also work for other containers.

Docker Installation

First make sure that your local machine has Docker installed. I use Debian and for this, a detailed description is available in Dockers documentation. There are also instructions for CentOS, Ubuntu, Fedora and others.

You can check the installed Docker version:

docker version
Client: Docker Engine - Community
 Version:           20.10.11
 API version:       1.41
 ... 

Run a Docker Container

Let’s start with an environment variable that defines the path where all the Gitlab data is stored. This is outside of the Docker container which allows an easy update without loosing data. To define the variable $GITLAB_HOME, you can use:

export GITLAB_HOME="/srv/gitlab"

To run a Docker container, simply use the run command with your parameters. For GitLab, this might look like:

docker run --detach
  --hostname <host.example.com>
  --publish 443:443 --publish 80:80 --publish 2222:22
  --name gitlab
  --restart always  
  --volume $GITLAB_HOME/config:/etc/gitlab 
  --volume $GITLAB_HOME/logs:/var/log/gitlab
  --volume $GITLAB_HOME/data:/var/opt/gitlab
  gitlab/gitlab-ce:latest

You should use your own settings and replace <host.example.com> with your setup.

For the sake of completeness, you might also need a runner for GitLab. Let’s run this container as well:

docker run --detach 
  --name gitlab_runner
  --restart always 
  --network host 
  -v /run/docker.sock:/var/run/docker.sock 
  gitlab/gitlab-runner:latest

And this one needs to be registered in your GitLab instance:

docker exec -it gitlab_runner gitlab-runner register 
  --url https://<host.example.com>/
  --registration-token <registration_token>

More details about the configuration of the GitLab runner is descriped in this post.

Update a Docker Container to the Latest Version

Let’s check, which container are running by using:

docker ps

The output shows all your containers and their ids:

CONTAINER ID   IMAGE                         COMMAND                  CREATED       STATUS                 NAMES
02fe426970f1   gitlab/gitlab-ce:latest       "/assets/wrapper"        4 hours ago   Up 4 hours (healthy)   gitlab
5742e07f809b   gitlab/gitlab-runner:latest   "/usr/bin/dumb-init …"   4 days ago    Up 5 hours             gitlab_runner

We have to stop the container and remove the image. When you used a run command (for GitLab) as shown above, this should not affect the data as this is stored outside of the container.

docker stop 02fe426970f1
docker rm 02fe426970f1

To remove the associated image, you can use the docker rmi. For this, we need to find the id of this image by listing all images:

docker images
REPOSITORY         TAG      IMAGE ID       CREATED        SIZE
gitlab/gitlab-ce   latest   46cd6954564a   17 hours ago   2.36GB
mysql              5.7      c20987f18b13   2 weeks ago    448MB
cirrusci/flutter   2.8.1    98a87781f179   2 weeks ago    3.49GB

Now we can remove the image (in my case the gitlab-ce image):

docker rmi 46cd6954564a

The update of the image can be done by using pull. This will download the images (in this case the latest version) and stores it on your system:

docker pull gitlab/gitlab-ce:latest

After this, you can use the same run command as before to run a container with the updated image:

docker run --detach
  --hostname <host.example.com>
  --publish 443:443 --publish 80:80 --publish 2222:22
  --name gitlab
  --restart always  
  --volume $GITLAB_HOME/config:/etc/gitlab 
  --volume $GITLAB_HOME/logs:/var/log/gitlab
  --volume $GITLAB_HOME/data:/var/opt/gitlab
  gitlab/gitlab-ce:latest

A final check, shows the container is up and running:

docker ps
CONTAINER ID   IMAGE                         COMMAND                  CREATED        STATUS                 NAMES
f3dab1163c07   gitlab/gitlab-ce:latest       "/assets/wrapper"        2 minutes ago  Up 4 hours (healthy)   gitlab
5742e07f809b   gitlab/gitlab-runner:latest   "/usr/bin/dumb-init …"   4 days ago     Up 5 hours             gitlab_runner

Instead of automatically using the latest version of each image (e.g. gitlab/gitlab-ce:latest) you can also specify a version number like gitlab/gitlab-ce:14.5.2. This is helpful when you need to update a system in steps. Or if you need to follow a specific upgrade path, which is the case for GitLab.

Photo by Ian Taylor on Unsplash.

How to simply analyze access logs of web servers

There is a tool for that: GoAccess. It’s an easy to use command line tool to analyze any access log and create beautiful insights.

Installation

On macOS you can use HomeBrew to install GoAccess:

brew install goaccess

For other operation systems, please check out the detailed documentation on their website.

Usage

The most elegant way is the terminal output. For this, simply type:

 goaccess access.log --log-format=COMBINED

Which creates such an output (you can navigate down to see much more details):

You can also create an html file that you can view in your browser:

goaccess access.last.log -o report.last.html --log-format=COMBINED

And voilà:

If your terminal is not using the same language as your log files do, then you might need to set LC_TIME for a correct parsing of the log files:

LC_TIME="en_US.UTF-8" goaccess access.last.log -o report.last.html --log-format=COMBINED

To learn more about all options, you can check out the documentation of GoAccess.

Photo by Aryan Dhiman on Unsplash.