CI/CD on Schallbert's Blog

act_runner rootless: no start

Schallbert — Sat, 28 Mar 2026

The error

My docker-in-docker (DinD) act_runner crashes shortly after starting with the following error message:

Issue ticket on Gitea

I have created a ticket for this issue in the Gitea community (issue #721). Below, I’ll go into my own findings and summarise the discussion a little.

Problem with kernel permissions?

In a Docker-in-Docker configuration, the act_runner must run its own Docker daemon. Only then can the runner create its own containers for the actions. For security reasons, this is not permitted by default.

Why containers are not allowed to start other containers

In the context of act_runner, an action executes code that is itself part of the repository. If malicious code is introduced into the job container unnoticed, e.g. by a ‘collaborator’, it can, in a Docker installation without DinD, under certain circumstances potentially gain direct access to the host system.

Consequences if something goes wrong despite DinD

When using the DinD concept, an attack can result in access to the dockerd process within the act_runner container - and only if the action is not properly encapsulated. Still not ideal, but acceptable: when the container is restarted, the status quo ante is restored. Access to the host system is indirect, as the container itself is equipped with kernel features.

Simple solution: Start the DinD container as `privileged`

To allow the DinD container to set up its own actions, privileged: true can be set in the configuration. This grants the container all kernel capabilities. According to the (excellent) OWASP Security Cheat Sheet, this should be avoided where possible. Should act_runner itself now come under attack or reveal critical security vulnerabilities, the intruder would already have every opportunity to bypass the encapsulation from the host system.

If you want to take the easy route, the corresponding docker-compose.yml file looks as follows.

# docker-compose.yml
# section ACT_RUNNER
runner:
 image: gitea/act_runner:latest-dind-rootless
 container_name: gitea-runner
 privileged: true
# [...]

“And remember: Do not run containers with the –privileged flag!!!” - OWASP’s Docker Security Cheat Sheet, 2026 License

Dead end: Only grant the rights that are absolutely necessary

After a quick search, I find an article on CodeStudy.net¹, which deals with the topic of DinD. From this, I put together some changes to my docker-compose.yml.

My approach:

Make changes to docker-compose.yml
Restart the service docker compose restart <service>
View logs docker logs --tail 100 <container-name>
If started: Run the programme web-app->repo->jobs->rerun_all-jobs
If it fails: Repeat

I start with few, but very powerful, permissions. After a few iterations, I get:

# docker-compose.yml
# section act_runner DinD-rootless
runner:
 image: gitea/act_runner:latest-dind-rootless
 container_name: gitea-runner
 privileged: false
 cap_add:
 - SYS_ADMIN
 security_opt:
 - no-new-privileges:true
 - apparmor:unconfined 
 - systempaths=unconfined
 - writable-cgroups=true
# [...]

But I’m still missing permissions that lie outside kernel privileges and security zones: the Docker daemon needs to be able to access sysfs and proc, i.e. system file directories and processes.

level=warning msg="[rootlesskit:child ] failed to mount sysfs, falling back to read-only mount: operation not permitted"

I cannot resolve this even with cap_add: ALL. Furthermore, online resources on this technical level are really scarce (and I am not a Docker specialist). A re-mount of the file systems might still help here; but that seems too experimental and error-prone to me.

After several more hours of research and trial and error on my server, I have to agree with @tianon in a discussion on GitHub: The Docker daemon requires so many permissions and capabilities that you might as well stick with privileged: true and avoid having to grapple with an armada of CAP_ADD and system bind mounts.

Future solution: Virtualisation or daemonless

However, there may be better solutions: dedicated container runtime environments such as Sysbox. The box itself has no special privileges on the host system; yet, much like in a virtual machine, it appears to be able to provide applications running within it with full access and capabilities.

Covering everything from installation and setup to fully functional job containers would take this post too far afield. Therefore, for the time being I must refer to other sources.

Switching from Docker to Podman is also a possible solution: Podman is daemonless, rootless and offers a similar range of features to Docker. Migrating to Podman would be a project in its own right for me and does not fit within the scope of this post.

Linux Security Modules (seccomp, AppArmor, SELinux)

Definition: LSM enable various security checks and restrictions at the kernel level. Through ‘Mandatory Access Control’, security extensions such as AppArmor can control kernel capabilities for individual applications and block access where necessary.

AppArmor

I run my server (VPS) on Ubuntu. In its more recent versions, this operating system has integrated ‘user namespace creation restrictions’ into AppArmor, which prevents appimages, WebApps and containers from running with elevated privileges.

“Unprivileged user namespaces are a feature in the Linux kernel […]; it enables unprivileged users to gain administrator (root) permissions within a confined environment […]” - mbelair, Ubuntu Discourse, as of May-2026, Ubuntu Discourse Website

In short, this tool was developed as a security-hardening measure to reduce the attack surface on the host system when running programmes that require elevated privileges.

Simple solution: Disable completely

The sledgehammer approach completely disables the feature for user namespaces. We tell AppArmor that third-party programmes may use kernel features or elevated privileges without restrictions, just as in older operating system versions. I found the relevant command on the AskUbuntu forum.

It disables restrictions for this session by overwriting the kernel parameters at runtime (-w).

schallbertTestsThis@machine:~# sudo sysctl -w kernel.apparmor_restrict_unprivileged_userns=0

I find the command very useful for troubleshooting. If you want to find out whether the desired programme is failing to start because of AppArmor:

Enter the command above
Test the third-party programme, the container, etc.
Reboot. Or enter the command with =1. This will restore the original state.

If you wish to retain this vulnerability permanently, enter:

makeVulnerability@machine:~# echo 'kernel.apparmor_restrict_unprivileged_userns = 0' | sudo tee /etc/sysctl.d/20-apparmor-donotrestrict.conf
makeVulnerability@machine:~# sudo shutdown -r now

This in itself does not constitute a security vulnerability. It has merely become easier, in principle, to exploit any weaknesses in the kernel and escape the container’s “sandbox”.

The correct solution: Tailor settings for each application

There are applications (such as my DinD version of act_runner) that absolutely require elevated privileges to function properly. And only these should be granted the ability to access non-admin user namespaces.

Docker itself has dedicated a section in its documentation to AppAmor. To arrive at the solution for the DinD runner, a bit of transfer (thanks, @thespad) is required. To recap, here is the error message from above:

gitea-runner | [rootlesskit:parent] error: failed to start the child: fork/exec /proc/self/exe: operation not permitted
gitea-runner | s6-svwait: fatal: some services reported permanent failure or their supervisor died

The log file indicates that the parent (Docker daemon in the container) cannot start its child (runner container) because it cannot create processes (/proc/self) for other participants (fork). Who is the user of this daemon? rootlesskit. This is exactly where our solution comes in: we need to enable the AppArmor profile for rootlesskit in the act_runner’s DinD container within the docker-compose.yml file.

# docker-compose.yml
# section act_runner DinD-rootless
runner:
 image: gitea/act_runner:latest-dind-rootless
 container_name: gitea-runner
 privileged: true
 security_opt:
 - apparmor=rootlesskit
 # [...]

The log file now looks fine. Done!

I get the feeling that post was written by an ‘AI’. When it gets specific and talks about ‘critical volume mounts’, there are no actual system paths listed, and the article becomes so vague overall that I can’t actually reach my goal by following the advice. ↩︎

Jekyll-dockerimage: Bundler and Gemfile

Schallbert — Fri, 14 Nov 2025

What is bundler?

Bundler is a tool that can be used to manage and version dependencies between modules and libraries for Ruby language. I use Bundler commands regularly, e.g. to build my blog and put it live on the server.

If I want to build locally, I type

bundle exec jekyll serve --incremental --future

into the console. This command tells Bundler to run the jekyll application in server mode and provides it with parameters that prevent jekyll from completely rebuilding every time a file is changed and still create pages that have not yet been published.

bundler in the CI pipeline

I also use Bundler as part of a Jekyll Docker image to publish my site, as I have often linked to, e.g. when moving to self-hosted. Since switching to a new version of the blog software lately, I have been seeing puzzling errors in my CI pipeline. Locally however, the system builds flawlessly. I have documented how to address and fix such errors here.

“There was an error while trying to write to /path/to/Gemfile.lock” - CI console output

No write permissions

Like the Jekyll user, Bundler itself can only read files on my CI and cannot write them. I can fix this for the Jekyll output with chown, but I don’t want to allow bundler to do this: I require Gemfile.lock to remain identical between my local build environment and the CI so that I can fix errors in advance.

Specification file for bundler

The Gemfile.lock file contains all dependencies of the application used, including version numbers and sources. If only the source code of the application plus this file is provided, bundler can pull a specific version of all dependencies during build time, allowing to work with similar requirements regardless of machine and version.

Troubleshooting

A first attempt to fix the issue by aligning all versions between local and CI failed. The second attempt, to give the Jekyll user write permissions to the Gemfile.lock, also failed. The third attempt led me to the Bundler website, where I took a closer look at the parameters, keyword “frozen”.

The fact is that you can prohibit bundler from rewriting the Gemfile.lock. To do this, use the command

bundle config set frozen true

According to its documentation, processing is aborted as soon as the file is about to be rewritten.

Although I had added this command to my yaml file in the CI, the build failed again. The trigger was the same library nokogiri as before, but the error message was now much more helpful:

Optimize `Gemfile.lock` for different environments

So I follow the suggestion and execute the desired command on my local machine:

bundle lock --add-platform x86_64_musl # my CI runner's environment

Note: musl is a libc-implementation for Linux.

My lockfile now has the following entry:

nokogiri (1.18.10-x86_64-linux-musl)

With this error fixed, I get better portability of my web page creation setup as a side effect.

Gitea act_runner: Jump-start issues

Schallbert — Fri, 30 Aug 2024

Problem statement

Since I run this page myself I experience problems starting Gitea and act_runner. Sometimes, the runner won’t start. It exits with status -1 and so my workflows wouldn’t run the website build, verification and deploy tasks.

The error message is always the same:

[...]
runner-1 | time="2024-08-26T10:09:46Z" level=info msg="Starting runner daemon"
runner-1 | time="2024-08-26T10:09:46Z" level=error msg="fail to invoke Declare" error="unavailable: 502 Bad Gateway"
runner-1 | Error: unavailable: 502 Bad Gateway
runner-1 exited with code 1
gitea | 2024/08/26 10:09:46 cmd/web.go:242:runWeb() [I] Starting Gitea on PID: 16

Ok, the runner cannot hook onto Gitea. When I restart all services via docker restart, the act_runner container tells me it would be missing a network with ID <long hexcode ID>

Interim corrective action

I didn’t find time for resolving this issue. So I just ran docker compose twice:

# schallbert server-console
docker compose -f /path/to/giteas/composefile up -d

The second time, act_runner would start successfully (because Gitea is ready):

gitea | 2024/08/26 10:38:17 routers/init.go:116:InitWebInstalled() [I] Git version: 2.45.2 (home: /data/gitea/home)
runner-1 | time="2024-08-26T10:38:22Z" level=info msg="Starting runner daemon"
runner-1 | time="2024-08-26T10:38:22Z" level=info msg="runner: action-runner, with version: v0.2.10, with labels: [ubuntu-latest], declare successfully"
runner-1 exited with code 0
runner-1 | time="2024-08-26T10:40:32Z" level=info msg="Started runner daemon"

Not very satisfying to do things twice. Plus, I did this manually. So why is that?

Root cause analysis

First I thought it was due to Caddy not being ready as it runs the reverse proxy, connecting Gitea to the internet. In reality, Gitea is the culprit: At the time, docker starts act_runner, its startup procedure wouldn’t be complete. So I try to find out how to manage dependencies in docker.

Solution: Reflecting dependencies

First, I try linking gitea to the runner in the docker compose file.

depends_on

# gitea/docker-compose.yml
 ## service: runner
 ## [...]
 depends_on:
 gitea:
 condition: service_started

Well, still act_runner seems to be starting early. Docker just makes sure it has started the container, and does not wait for it to return a healthy signal.

Healthcheck

So I have to make sure that gitea’s startup procedure is complete. For this, docker provides the condition service_healthy. I adjust the runner configuration like so:

# gitea/docker-compose.yml
 ## service: runner
 ## [...]
 depends_on:
 gitea:
 condition: service_healthy # required so runner can attach to gitea
 restart: true

The service_healthy qualifier is returned by the healthcheck function. It consists of a so-called test and some environment parameters that define intervall, number of retries, delay and timeout conditions for checking service ready.

For the test I chose a simple command, assuming that Gitea (and, implicitly, Caddy too) would be healthy once it is able to respond to a GET request from curl.

# gitea/docker-compose.yml
## service: gitea
## [...]
 healthcheck:
 test: ["CMD", "curl", "-f", "https://git.schallbert.de/"] # checks if gitea is available
 interval: 10s
 retries: 3
 start_period: 30s
 timeout: 10s

With these lines, docker checks if gitea is “healthy” and only then starts act_runner. This permanently solved the problem and both Gitea and act_runner are stable.

Sending error logs

Schallbert — Tue, 20 Aug 2024

In this article I’ll look at how to set up “monitoring” for my server. Applications and services should be able to send me notifications in the event of an error.

What is this about?

Learn about and select sending mechanisms
Write test messages and verify the automation
Automatically send error report from Borgmatic
Send runner logs through Gitea
Notification when logging into my server via ssh
Create and send logs for server updates / server errors

Verification and monitoring

After running an automation, I want to know whether it was carried out successfully and whether all programs and services started their work as expected. This should apply to any automation - whether it is specifically sending an update, creating automatic backups or an action from Gitea, it doesn’t matter.

Normally I would use reporting mechanisms from act_runner for something like this. In the case of a server update, the runner is not available due to the circular reference already mentioned in the article Rolling out the server configuration, as all applications have to be shut down temporarily.

In addition, applications may have their own procedures for monitoring. So I have to look at mechanisms that allow me to easily access the information.

What characterizes good monitoring for me?

It is unobtrusive, so it only reports in the event of an error or unusual occurrence.
It provides specific information and error messages that are easy to understand.
It uses a message channel that works even if the system being monitored crashes.
It presents reports and error messages in isolation from other topics and does not mix anything.
It is brief.

Methods of automatic reporting

I will break this section down. In the general part, I will discuss the on-board tool for asynchronous monitoring that is available to me on the Ubuntu server. After that, I will look at the solutions that are partly built into my services or the applications that are compatible with them. I do not want to limit myself to the classic tool of email, but also look at “more modern” communication channels such as messenger programs or RSS feeds.

Mail via console - curl

Sending emails as notifications is common practice in many companies. With Linux, this can usually be done without any additional programs: The standard curl can help here.

curl is a program for transferring data from or to a server. If I enter my blog as the target, I get the HTML page output as a text file on the console:

curl https://blog.schallbert.de

This made me notice how much unnecessary data my blog software generates. I’ll have to clean that up later. Back to the topic: You can also use curl to access any web backend - for example a mail server:

# copied from https://stackoverflow.com/questions/8260858/how-to-send-email-from-terminal
curl --url 'smtps://smtp.gmail.com:465' --ssl-reqd \
 --mail-from 'from-email@gmail.com' \
 --mail-rcpt 'to-email@gmail.com' \
 --user 'from-email@gmail.com:YourPassword' \
 -T <(echo -e 'From: from-email@gmail.com\nTo: to-email@gmail.com\nSubject: Curl Test\n\nHello')

This only works for my mail provider if I allow logins from external clients. Google, for example, calls these “less secure apps”. As described in my post Server configuration with Git, I’m not a fan of writing secrets into anything - so I would rather not use the direct route via curl.

Mail via console - mail, mailx, mailutils, swaks

If you don’t want to always provide all the configuration information for the server and secrets, there are various handy tools for the console such as mailutils or swaks. Here, the connection to the mail server is configured once using the tool and can be stored in environment variables, for example. The syntax varies from program to program, but an email always drops out at the end.

The only major disadvantage for me is that information domains are mixed up. I would not want another “report thread” in my emails that I’d have to search for in the mass of messages. Other notification media in contrast allow me to set an automatic expiration date so that they disappear from my list after a set time.

Create RSS feed

Unusual but possible: I could create the monitoring as an RSS feed like my blog (schallberts-blog-feed) has e.g. using a Jekyll instance and put it online as a website. This would be easy to subscribe to, were readable with practically any reader and I could even set it up separately for each application. But there are obvious disadvantages:

High effort: Gitea runner with Jekyll instance, web server and subdomain required.
Publicly available: Suddenly build processes, updates, upgrades and error messages are accessible to everyone.
Error-prone: If Gitea, act_runner, my proxy or the web server crashes, I don’t get any reports.

With the last point at the latest, this option is out of the question for me. I want to get a report when my applications don’t do what they’re supposed to. Let’s take a look at the applications I already run and see how they tackle this problem.

borgmatic

Borgmatic is compatible with a lot of monitoring options. These include apprise, ntfy, healthchecks, cronitor, pagerduty, cronhub and grafana.

Apprise is a library. It is open source and can be integrated into an existing application as a dependency. Like an adapter, it enables asynchronous communication between the application and various communication services such as SMS, mail, messenger (e.g. Signal), various home automation systems or the notification mechanism of various operating systems. The latter, however, only works on the local machine. The trigger for communication must always come from the application.
ntfy is a push notification service. It is open source and can be self-hosted or used as a service via web application. Apprise, for example, supports ntfy as a communication service. The structure is quite simple and works like MQTT via publication subscription / broker client mechanism, but HTTP-based.
Healthchecks is a service. It is open source and can be self-hosted. The service is there to monitor regular activities and can act as a dead man’s switch: If, contrary to expectations, there is no response from the monitored program, it can raise an error message itself. This can in turn be forwarded to various communication services.
Cronitor is a monitoring solution and web application that, in addition to the notifications I need, provides a lot of analysis tools, performance measurements and metrics - mostly for money. There is a free “hacker” account with limited functionality, but this tool is also far too big and complex for me.
PagerDuty sees itself as a commercial “operations” platform that provides “incident management”, automation, “business operations”, “AIOps” etc. It’s out for me straight away. At the latest when I read the word “AIOps” 😅
cronhub looks like a commercial web application to me that, like Healthchecks, can create cron jobs, monitor them and report errors. It is of no interest to me because it does not seem to be open source and I could not host it myself.
Grafana is an open source web application that can either be self-hosted or used as a cloud service. Although many larger companies and projects use the application, it is orders of magnitude too extensive and feature-rich for my purposes.

fail2ban

Fail2ban does not have an automation solution like borgmatic. It simply creates log files that need to be evaluated in order to obtain content for notifications. At the moment I cannot think of anything that I absolutely need to know about Fail2ban. So I am not creating any reports for this for now.

Gitea

Gitea allows the log files to be configured very precisely. Access logs can be written out separately from service logs, repository logs or action logs and then processed further. According to my research, Gitea only offers a mailer as a notification system. In my opinion, this is primarily intended for repository and action logs.

Here, too, it would be most beneficial for me to analyze the logs and create a report myself if necessary.

Server

I already roll out my server configuration files using a script. So, in the event of an error, I could redirect the logs to a file and attach them to a report in any channel.

In addition, successful logins on the server would be worth a message. Then I can immediately determine whether it was me or not.

Selecting the reporting program

I not only have to select a monitoring program, but also choose a communications service through which the reports are sent.

Program	Advantage	Disadvantage
curl	simple	configuration must be provided
	works super easily with ntfy¹	No monitoring if the server crashes
	Onboard tools
	no service required
Apprise	enables integration of the Signal API²	Complex in interaction with external services³
	no service required
Healthchecks	Can act as a watchdog/dead man switch	Registration required
	Can be hosted locally⁴	Local hosting contradicts the watchdog concept

I only have a small server with a few applications, so I keep it as simple as possible and choose Apprise for services like borgmatic, which come with this library anyway, and curl for those whose reporting mechanics I have to program myself.

Communication channel

Program	Advantage	Disadvantage
SMS	High reliability	Registration required
		Limited to a few characters
		No attachments possible
		No topics, domain mixing
Mail	easy to set up	Poor searchability
RSS feed	Good sortability	Complicated to set up, error-prone
	Client very lightweight	Local hosting contradicts watchdog concept
		Content publicly available
ntfy	Simple and lightweight	Registration optional for web use
	Client purpose-oriented	Insecure: No encryption without registration
	Free for small users
	Independent of your own machine
Signal	“Note to self” easy to set up	Setting up and configuring the Signal API complex
	Independent of your own machine	Topics somewhat difficult to implement
		Useless if the server crashes

SMS, mail and RSS feed are already out of the question for me due to some of the disadvantages already explained above. So that leaves ntfy and Signal. ntfy impresses with its simplicity: send an HTTP PUSH request to a self-defined “topic” and subscribe to it on your cell phone - done. It is also easy to expand, because with self-hosting I can later increase security if necessary (encryption) and control the sending behavior. Signal, on the other hand, requires a separate client with relatively complex configuration. In addition, the devices would have to be connected to the server and cell phone, so it is not easy to add new subscribers. On the other hand, the transmission is well secured end-to-end, I do not have to register and it is also free.

For now, I have decided to go for the less complex solution with ntfy.

Logging

Good. Now it’s clear that I’m calling home using curl and apprise via the ntfy service. Let’s see what content needs to be transmitted and how to make it as unobtrusive as possible, but still short and concise.

When should logs be sent?

borgmatic: If a backup fails
If I successfully log in to my server or Gitea
After rolling out an update to the server configuration
Docker: If an application fails or doesn’t start
Gitea: Failed run of act_runner

What should I include in a report?

I want the classic “W questions” answered.

When did it happen (time stamp)?
Which application is reporting?
What happened?
Where (module, line of code) etc. did it happen?
How many were injured (severity, recovery)?

A notification then looks something like this:

TIMESTAMP APPLICATION PRIORITY MESSAGE EFFECT DETAIL

Rough version of a report

An ntfy message could look something like this:

curl \
 -H "Title: Error Borgmatic" \
 -H "Priority: urgent" \
 -H "Tags: warning" \
 -d "YYYY-MM-DD HH:MM:SS Backup creation aborted. Access to repository blocked" \
 ntfy.sh/schallberts-topic

I install the corresponding app on my phone, register on the “Topic” and send the message. It’s nice when things just work! Aha, the app shows the time of receipt. That’s accurate enough for me.

Implementation

Here I’ll take a look at all the services for which I’d like to set up notifications one by one.

borgmatic

Let’s start with a pilot test in small steps. First I configure Borgmatic to send a message via Apprise to ntfy if the backup creation fails:

# /borgmatic.d/config.yml
on_error:
- echo "Error while creating a backup."
- apprise -vv --title "Borgmatic Error"
--body "Could not run {output}. Aborted {error}."
ntfy://schallberts-topic

Then I test the command by calling apprise within the Borgmatic container. And indeed, it works. But getting here took me an hour, as the yml with its syntax rules even interprets within strings and borgmatic constantly refused to read the configuration file due to : and - characters. If you see an error similar to this one:

At 'on_error[1]': {'apprise -vv --title "Borgmatic Error" --body "Could not run {output}': 'Aborted {error}." ntfy://schallberts-topic'} is not of type 'string'

This shows that the interpretation of characters or indentations went wrong and the punctuation needs to be checked. Alternatively, the pipe operator | can be used to combine a command. Reference: yml specification

Server

An application of ntfy to monitor logins on a server can already be found in the documentation of ntfy itself. The description shows that something like this is easy to implement yourself and gives a great example using Pluggable Authentication Modules (PAM library).

Insert the following code at the end of the sshd file in the etc/pam.d directory:

session optional pam_exec.so /usr/bin/ntfy-ssh-login.sh

This line tells PAM that when a session is opened via ssh, it should call the executing module pam_exec, which then runs the script specified below. The value optional means that the configuration file should continue to be run even if the action fails. More details on how to use PAM can be found, for example, on Baeldung.

Then you simply have to store the ntfy call in the ntfy-ssh-login.sh script when the script determines that PAM has detected an open_session event.

That’s exactly how I implemented it and it works straight away. Great! The only disadvantage: I made this modification directly on the server. Without a container and outside of my configuration backup. If I now have to set up the server again for some reason, the change in PAM is lost and I won’t receive any more notifications until I manually enter the change again.

Server configuration

To send a report after running the configuration automation, I actually only have to adapt the server-config-action script that I wrote in the last article.

First, I want to be informed when the script runs without errors. To do this, I add a curl command at the end of the file.

# server-config-action.sh

# action commands...
# [...]
# send success notification
curl \
 -H "Title: server-config-action" \
 -H "Priority: low" \
 -H "Tags: white_check_mark" \
 -d "Rollout successful" \
 ntfy.sh/schallbert-server-config-push-topic

However, if the script does not run without errors, I would like to do the following:

Cancel actions after the error occurs
Create an error log
Send this log

I can achieve the first point by adding a trap for errors:

# server-config-action.sh
trap 'handle_error $LINENO' ERR

# action commands...

This calls the function handle_error. It is given the line number where the error ERR occurred. It also covers errors that can occur when restarting the container. I create the error log by redirecting the output of the individual script commands. I achieve this with the following line:

# server-config-action.sh

exec 3>&1 1>server-config-action.log 2>&1

# action commands ...

With this command, any standard output stdout should be passed on to the file descriptor 3, specified here with the log file server-config-action-log, overwriting it. For “append” there would have to be two redirection operators >>⁵. The log file thus replaces the console output, which I still had at this point in the previous article.

To have the log sent to me, I now define the following function at the beginning of the Bash script:

# server-config-action.sh
# on error, send a notification
handle_error() {
 # stop redirecting to file
 exec 1>&3 1>&2
 curl \
 -H "Title: server-config-action FAILED" \
 -H "Priority: high" \
 -H "Tags: x" \
 -T server-config-action.log \
 -H "Filename: server-config-action.log" \
 ntfy.sh/schallbert-server-config-push-topic
 exit 1
}

# set error trap
# redirect stdout and stderr to file
# action commands...

Gitea

For Github Actions reports, there is already an existing example at ntfy_examples/#github-actions. And the best part: Since the act_runner from Gitea is compatible with Github Actions at in the environment parameters, sending to ntfy works for me straight away. In the runner’s workflow file, all you have to do is enter the curl command specified on the website and you’re done.

Result

Five notifications about the most important processes on my server have now been set up. I have understood the underlying mechanics and can create additional notifications at any time if I need them. If I trigger all notifications as a test, my smartphone will display the as shown.

But I have not yet achieved independence from the system to be monitored. All notifications come from the affected machine and there are sometimes even dependencies between Docker containers. For example, the act_runner has to connect to Gitea via websockets. And that only works if Caddy provides the reverse proxy.

If I find out in the next few months that a service “under my radar” is no longer able to work, I will have to establish independence.

Context for setup with ntfy ↩︎
Context for the Signal API ↩︎
Report by asad-awadia on setting up the Signal API ↩︎
Healthchecks is directly available as a Docker image. ↩︎
Incidentally, the operators << and >> are not difficult for me to remember, because I used to have a lot to do with the programming language C. There - and in many other programming languages too - these are shift operators that can “shift” a value bit by bit or move one field to another. In the RISC architectures that I used at the time, the shift operation was “cheap”, i.e. very fast and memory-saving. ↩︎

Server configuration rollout

Schallbert — Wed, 31 Jul 2024

In the previous article, I brought my configuration files under version control. Now I want to automatically install the updates provided on the server.

What is this about?

Writing a script to automatically detect the update trigger
Server applications should be shut down and a backup copy created
The script should distribute the configuration on the system
All applications should then be restarted

Ways and possibilities

Here, too, I spent several hours researching. For larger projects, infrastructure experts seem to use specialized automation tools. These include Ansible, simpler tools such as cdist or even Kubernetes for highly scalable services.

Automation services Ansible, cdist, Kubernetes

Ansible and cdist seem to follow a similar concept: On the source machine (in this example my laptop) I create the configuration for my server and store this and the instructions for configuring my services in a “playbook” (Ansible) or in “types” (cdist).

To put it simply - as I understand it - the tool then takes care of building the configuration at the push of a button, dialing into the target host via ssh, pushing it over there and then starting it. For Ansible, there are even tutorials like this one for my scenario with docker-compose, which makes getting started even easier.

Kubernetes takes a different approach. It sees itself as more of a container manager, load balancer and scaling agent, but can do similar things for my purposes.

My approach

I, on the other hand, only need a fraction of the capabilities of these programs. I am also put off by the “additional” ssh channel, the configuration effort, the additional programs sometimes required on the target system, and the necessary reading and selection of the best tool for me. Because thanks to my very simple pipeline from the last article, the configuration is already on my server. It “only” needs to be copied to the right places and the affected services restarted.

Therefore, I am trying to solve this problem using on-board tools, my brain in working order and a few searches in relevant forums on the topics “executing a script when a file changes and “copying folder structures in Linux”.

Risk of circular reference

However, I am taking a risk: Since the deployment runs via Gitea, but the configuration affects Gitea itself, if there is an error in this module I can no longer change or reset anything: The Gitea service is then broken. I would have to get the configuration up and running again manually on the server.

I’ll try it out anyway and see if I am actually confronted with such a problem. If so, I’ll just switch to cdist and document it in a separate article! 🤗

Preparation: Create folder system and scripts

I think about it for a moment and create a few folders in the server file system:

# /opt/server-config
mkdir automation-hooks-trigger
mkdir automation-hooks-handler

These folders should represent the two sides of the automation. trigger contains text files that can be manipulated from the (web) service side. For example, act_runner should write the file server-config-update as soon as an update is available.

service –> writes to trigger file ||| server_handler() –> trigger_file.changed ? run_action() : loop()

The handler folder contains script files that make the necessary changes to the server file system.

Implementation: Shell script to handle the update

In the previous article, I executed the command touch server-config-update.txt in the act_runner container to signal the presence of a new server configuration. On the server, I now use the following code to periodically check whether this file has changed.

### set directories
actionfile=/opt/server-config/automation-hooks-handler/server-config-action.sh
triggerfile=/opt/server-config/automation-hooks-trigger/server-config-update.txt

### Set initial time of file
LTIME=`stat -c %Z ${triggerfile}`

while true
do
 ATIME=`stat -c %Z ${triggerfile}`

 if [[ "$ATIME" != "$LTIME" ]]
 then
 ${actionfile} 2>&1
 LTIME=$ATIME
 fi
 sleep 10
done

The operating system is given the interpreter with which the script is to be executed via #!/bin/bash. The command stat -c %Z queries when the file was last changed and returns the time in Epoch format. Finally, the time taken at the beginning of the script is compared with the time taken in the loop and if there is a change, the deploy routine that is still to be written can then run. The reference time is then updated. Finally, the script (sleep 10) pauses for ten seconds before the query starts again. I use absolute paths so I’m able to both start it from console and per service cron oder systemd.

First test and hooking into “autostart”

Now we need to make the file executable for a first test:

# change file mode bits: add "executable" flag to server-config-handler script
chmod +x /opt/server-config/automation-hooks-handler/server-config-handler.sh

Next, I create the server-config-action.sh file and add just an echo command. To test the setup I create my update trigger file locally and start the script. I then modify the triggerfile in another shell using touch server-config-update.txt and --- CONFIG UPDATE TRIGGER detected --- appears in the console. Great!

Later on the server, I need to have the script run automatically after a restart. To do this, I use the cron tool help and create a new entry using crontab -e:

# cron can automatically execute recurring tasks
# in this case, we're running server-config-handler script on reboot
@reboot sh /opt/server-config/automation-hooks-handler/server-config-handler.sh

With the command ps aux I can now check whether the script is actually being executed:

schallbert@server: ps aux
[...]
8:15 0:00 /bin/bash ./server-config-handler.sh
8:15 0:00 sleep 10
[...]

Backup

Making a backup of my applications and files before I roll out the update makes total sense. So I tell Borg that I want to create a backup now. Of course, I have to stop all services first. This means that all data is accessible, coherent and static.

Freeze state and data

To do this, I create a script in automation-hooks-handler that automatically terminates all containers except borg.

#! /bin/bash
# /opt/automation-hooks-handler/backup-pre-action.sh
# this shell script shuts down all docker containers prior to backup

echo "Shutting down containers for backup:"

echo "watchtower..."
cd /opt/watchtower
docker compose down 2>&1
# [...]

The expression 2>&1 means that any error output is redirected to the console. The number 1 represents the file descriptor for stdout, while 2 means stderr. The operator >& functions as a redirect merger. Later, we can go to this point and write the output to a log file - but I’ll leave that out for now for the sake of simplicity.

Borgmatic: Trigger handler mechanism #2 and #3

I can now run the script in two ways:

As a call by the server-config-handler script described above
By the automation solution borgmatic placed in front of borg

I choose the second option and therefore write the following commands in borgmatic.d/config.yml:

# borgmatic.d/config.yml
# List of one or more shell commands or scripts to execute before
# creating a backup, run once per repository.
before_backup:
 - echo "Triggering container shutdown for backup."
 - touch /etc/automation-hooks-trigger/backup-pre.txt
 - sleep 20
 - echo "Assuming container shutdown complete. Creating the backup now."
# [...]
after_backup:
 - echo "Triggering container restart after backup."
 - touch /etc/automation-hooks-trigger/backup-post.txt
 - sleep 10
 - echo "Assuming container restart complete. Exiting."
# [...]

For this to work properly, I have to create a volume in the associated docker-compose.yml and have it point to the path in the server’s file system: ${VOLUME_UPDATE_TRIGGER}:/etc/automation-hooks-trigger Now I set up the other side of these triggers: Handlers monitor trigger files for changes and call action scripts accordingly. These look very identical to server-config-handler.sh except actionfile and triggerfile paths.

Creating the backup

If I were to address borg directly, the backup could be created using create. To do this, I would have to specify in which repository the backup copy should be saved and under which name. In the example below, this is specified using the scope operator: ::config-update. The folders to be backed up are then specified.

borg create /path/to/repo::config-update ~/opt

I use borgmatic, which takes a lot of work off my hands using the configuration file. However, I have to execute the command in the container. To better check whether everything is working, I output statistics “verbose” to the console (--stats -v 1) and display the copied files --files.

docker exec borgmatic sh -c "cd && borgmatic --stats -v 1 --files 2>&1"

I add this line to the automation script.

Second test to create the backup

If everything works now, the complete process looks like this:

After the changed files have been received by Giteas Automation, act_runner (in the Docker container) stores the files on the server and then sets the server-config-update trigger.
Within 10sec the trigger is recognized by server-config-handler.sh, which then calls server-config-action.sh.
borgmatic (in the Docker container) is instructed to create a backup. This in turn writes the pre-backup trigger.
Again within 10sec this pre-backup-handler.sh calls the backup-pre-action.sh script and stops all containers except borgmatic.
Due to the built-in delay, borgmatic waits for this and then creates the backup.
After the backup, borgmatic writes the backup-post-action.sh trigger.
Within another 10 seconds, post-backup-handler.sh recognizes the changed file and restarts all containers via post-backup-actions.sh.
borgmatic tells server-config-update.sh whether any errors have occurred anywhere in the process so far. If not, it continues.
All Docker containers are stopped.
The server configuration is rolled out to the appropriate locations.
All Docker containers are restarted with the new configuration.

A first success: These scripts are already running on my laptop up to point 6:

schallbert@laptop: touch server-config-update.txt
server-config-handler@laptop: --- RUN backup ---
borgmatic@docker: /etc/borgmatic.d/config.yml: Running 4 commands for pre-backup hook
 Triggering container shutdown for backup. 
backup-pre-handler.sh@laptop: --- RUN backup-pre-action.sh ---
 Shutting down containers for backup:
 watchtower...
 [...]
 complete.
borgmatic@docker: Assuming container shutdown complete. Creating the backup now.
 local: Creating archive
 Failed to create/acquire the lock /mnt/repository/lock.exclusive

Troubleshooting for “failed to acquire the lock”

This problem occurs for me when borg reports an error when creating a backup that causes the program to abort. In this case, the repository is apparently not released correctly, so that after restarting the container it remains reserved for the old, now non-existent container. The following command solves this problem:

docker exec borgmatic sh -c "cd && borg break-lock /mnt/repository

Transferring the configuration

Now the configuration files have to be copied to the correct location on the server. Fortunately, I had already cloned the target folder structure when creating the repository, so I should be able to do this with a single copy command without “hardcoding”. After a bit of online research and a look at the user manual for the copy command man cp, I have my command:

# copy recursively contents of folder "server-config" to "/opt"
sudo cp -r -v /opt/server-config/. /opt 2>&1

This tells the operating system to copy the contents (/.) of the folder server-config recursively (-r), including all subfolders into the folder opt, which is located in the root directory /. cp works in an overwrite-supplement manner, so it will create files that do not yet exist and overwrite existing ones, and will not “copy them next to each other” under the same name. With the -v option I can output additional details, and with 2>&1 I redirect the error output to the console.

All folder indicators must be exactly where they are in the command: A slash after /opt/ would copy folders redundantly without overwriting, but would overwrite files. Without . the folder server-config would be created in the target path.

Switching to rsync

Unfortunately the cp command also copies a few files that I don’t want copied: repository-specific folders such as .gitea, or the folders for triggers and handlers. I only need these under server-config, not directly in opt. To fix this, I use the rsync command instead. There I can use a -u option so new files owerwrite older ones only and add --exclude to exclude files and folders that should not be copied. This looks like so:

rsync -r -u -v --exclude '.*' --exclude 'README.md' --exclude '<otherFolders>' /opt/server-config/. /opt 2>&1

But suddenly gitea no longer starts. Error message:

docker@server: [...] failed to load config file "app.ini": open: permission denied

After a long time of pondering and restarting the Docker client several times, I see that rsync has written the permissions of the source file to the target file, which was not the case with cp before: -rw-------. Now I change this by running chmod +r app.ini. Everything starts up again as usual! 🎉

Restart all applications

Since I run everything on my server in Docker, two simple commands are sufficient:

docker stop $(docker ps -a -q) 2>&1
# [...roll out config changes...]
docker restart $(docker ps -a -q) 2>&1

The finished automation

My script is now finished and simply calls the action script.

#! /bin/bash
#/opt/automation-hooks-handler/server-config-handler.sh
#[...]
 if [[ "$ATIME" != "$LTIME" ]]
 then 
 echo "--- CONFIG UPDATE TRIGGER detected ---"
 ./server-config-action.sh 2>&1
 LTIME=$ATIME
 fi
 sleep 10
done

The Script server-config-action then executes the above described actions:

#! /bin/bash
#/opt/automation-hooks-handler/server-config-action.sh

echo "--- RUN backup ---"
docker exec borgmatic sh -c "cd && borgmatic --stats -v 1 --files 2>&1"
echo "--- STOP all containers ---"
docker stop $(docker ps -a -q) 2>&1
echo "--- DEPLOY config ---"
cp -r -v /opt/server-config/. /opt 2>&1
echo "--- RESTART all containers ---"
docker restart $(docker ps -a -q) 2>&1

In order for the whole thing to work reliably, the three handler scripts must run in the background:

server-config-handler.sh
backup-pre-handler.sh
backup-post-handler.sh

They react to the respective triggers by act_runner from Gitea or by borgmatic. I will now expand the crontab accordingly and then I am done with the task for now. The console output of the entire process looks like this:

--- RUN backup ---
/etc/borgmatic.d/config.yml: Running 4 commands for pre-backup hook
Triggering container shutdown for backup.
--- RUN backup-pre-action.sh ---
Shutting down containers for backup:
gitea... 
fail2ban...
complete.
Assuming container shutdown complete. Creating the backup now.
local: Creating archive
<borg archive stats>
/etc/borgmatic.d/config.yml: Running 4 commands for post-backup hook
Triggering container restart after backup.
--- RUN backup-post-action.sh ---
Restarting containers after backup:
fail2ban...
gitea... 
watchtower...
complete.
Assuming container restart complete. Exiting.
local: Pruning archives
local: Compacting segments
compaction freed about 1.82 MB repository space.
local: Running consistency checks
summary:
/etc/borgmatic.d/config.yml: Successfully ran configuration file
--- STOP all containers ---
<container ids>
--- DEPLOY config ---
sending incremental file list
<files that are copied>
sent 206,558 bytes received 3,463 bytes 420,042.00 bytes/sec
total size is 193,167 speedup is 0.92
--- RESTART all containers ---
<container ids>

Great! Now all I need is for this log to be delivered to me if something goes wrong.

Server configuration with Git

Schallbert — Mon, 15 Jul 2024

What is it about?

I want version control for the configuration of server applications
Discussion of technical solutions for implementation
Tutorial: Exclude Docker/Gitea/other “secrets” from version control
Tutorial: Create a deployment pipeline
introduce automation trigger for subsequent rollout of files on the server
Next article: Integrate automation on the server and install upgrades

Background

I have Watchtower running on my server to automatically keep the installed distributions up to date and freshly patched. Recently I had a case where my Gitea Action Runner would no longer start automatically and even when started manually it gave an error message.

Docker had apparently updated itself without my supervision and was now throwing a volume error when starting up the runner container, which I had never seen before. The error message was clear and could easily be fixed with small changes in a configuration file for Gitea. Nevertheless, I now had the feeling that version control for my configuration would be useful for my future self in order to be able to better understand changes, updates and the reasons behind.

Preliminary considerations

It sounds like a circular reference to me: I record the configuration files for my server in Gitea, which itself runs on my server. This could become interesting with auto-deployment. But more on that later.

Two options spontaneously come to mind for getting version control with automatic synchronization:

1. Hardlink

This solution would store the configuration files scattered across many folders on my server in a folder declared as a repository using a hardlink. Why use a hardlink? Because every service on my server runs encapsulated in itself and has its own configuration files and environment variables stored together with the service. Without hardlinks, I would have to version the entire service folder and make my .gitignore correspondingly complex.

With Git, I would version the files mapped via hardlinks and make their contents available on Gitea in this way.

schallbert@server:/server-config-files –> hardlinks –> repository –> Gitea

So I would get a downstream “version monitoring” with a backup copy in the sense that the files are now lying around multiple times.

2. Repo and auto-deploy to the server

In this scenario, I keep the configuration files locally on my laptop and can version them with Git as usual. On Gitea, I would map the repository, and at the end of the chain, my runner would have to auto-deploy on the server every time the configuration changes and then restart the affected containers.

schallbert@laptop:/server-config-files –> repository –> Gitea –> act-runner –> server:/<service1...ServiceN>/config-files

I would get the administration “main” on my laptop, and the server would follow the mapping on Gitea.

Problems

In both cases, I don’t have the option of testing changed configurations in advance. Everything I do goes straight to “Prod” and would be live. In the worst case, I can easily mess up my setup.

This doesn’t change the status quo ante, where I updated the files directly on the server. So a new problem only in the sense that I didn’t have a Prod operation when I set it up in the first place and so there was no risk of failure.

Solution two seems to be more complex to implement, because I need a deployment pipeline that rolls out the files in the repository on my server. Especially since the runner is in a Docker container, while the configuration files are located directly in the server’s file system.

Direct access to the server file system is technically possible, but according to my limited understanding it would mean a large attack surface for all my services, whose configuration files could now be directly manipulated via the repository in Gitea.

Selecting my solution

Solution two at least places the immediate live problem on my laptop, so that I don’t have to mess around with the production system in the first step. I think it would be easier to set up an integration environment here with which I can check my configuration changes in advance. Since I have all of my services running in Docker, this could perhaps be solved quite easily.

Let’s get to work

Create a configuration repo

Okay, then the first step is to get the configuration files from the server. To do this, I use the file transfer command scp: scp server:/path/to/source path/to/target about a dozen times until I have caught all the files.

I set up the folder structure in this repo exactly as the files are on the server. I hope that this will make my life a little easier later.

Secrets in `docker-compose.yml`

But what do I do with “secrets” in the configuration files? Private keys, registration tokens, hashes? I would rather not have them lying around more or less openly in the repository. When using docker-compose it’s quite simple: I can store secret values in hidden files for environment variables and exclude them from Git tracking. In the simplest case, such files are simply called .env and contain a list of environment variables in the style of

BORG_PASSPHRASE="<redacted>"

In the corresponding docker-compose.yml I pull the variable from the .env file as follows:

- BORG_PASSPHRASE=${BORG_PASSPHRASE}

I make these changes locally on my laptop. I use .gitignore to specify using .* so hidden files and thus .env should not be included in the repository. But how do I know whether the containers are still booting correctly?

Secrets in Gitea’s `app.ini`

With Gitea, I’ve had a much harder time storing secrets in files. The app.ini is also written dynamically by Gitea, so the file looks a little different every time the service is restarted. After a long search, I found this issue, in which a solution for storing secrets separately was sought and found.

The only thing that seems sensible to me is storing the values INTERNAL_TOKEN and SECRET_KEY separately.
The other two properties LFS_JWT_SECRET and JWT_SECRET are automatically generated anyway and regularly overwritten.

In my configuration I only use INTERNAL_TOKEN. So I will copy it in plain text and without quotes into a hidden file (.INTERNAL_TOKEN) and make it available to the container via a Docker volume:

# Gitea's docker-compose.yml
# [...]
 volumes:
 - ./.INTERNAL_TOKEN:/run/secrets/INTERNAL_TOKEN:ro
 # [...]

In the app.ini it is now important to use the path specified in the compose file:

# Gitea's app.ini
# [...]
 server:
 INTERNAL_TOKEN = /run/secrets/INTERNAL_TOKEN
 # [...]

Not INTERNAL_TOKEN_URI=/run/secrets/INTERNAL_TOKEN as stated in the issue linked above, because this creates an error in V1.22.1 I am currently using: Unsupported URI-Scheme.

A very rough test

To check whether the changed configuration files still work, I install docker and docker-compose on my laptop. Then I try to start the containers.

# docker console log
Error: Network 'caddy-proxy' declared as external, but could not be found.

Oh right, Docker is not yet configured here. So create the network: sudo docker network create caddy-proxy and try again. The download of Gitea and its dependencies begins and the container starts - although not as I had imagined: The folder permissions within Docker are incorrect, meaning that neither Gitea nor Act-runner can access all the required files.

Nevertheless, I find the first error:

# gitea container log
WARNING: The GITEA_RUNNER_REGISTRATION_TOKEN variable is not set. Defaulting to a blank string.

I had forgotten to put the token string in quotation marks.

So for a fully functional integration environment, I have to solve at least two more problems: Folder permissions for the Main on my laptop must be set up in such a way that the Docker user also has write permissions. A simple solution for now is to append a :Z to the volumes in question and mark them as private unshared. Now the volume definition in docker-compose.yml looks like this:

# gitea/docker-compose.yml
# [...]
volumes:
 - ./gitea:/data:Z
 # [...]

I need a second file for environment variables to redirect my services to localhost. At least the service starts this way and I can see the log output. I can already spot most of the configuration errors.

I’m not sure, but I might have additional problems with the caddyserver such as certificate management, proxy settings and so on.

Create a deploy pipeline

Now it would be great if the files uploaded to the Gitea repo (and previously tested locally for functionality) would automatically find their way to my server. For this I could create another Docker volume where act-runner would then put the data stored via on:push trigger. They would then be available on my server.

Setup

If we remember my last attempts to provide artifacts on the server using act_runner, we can use a large part of that for this task as well:

# deploy-to-server.yml
# Workflow for saving the server's config repo to the local disk system

name: Upload-server-config
run-name: ${{ gitea.actor }} uploads server configuration files
on:
 push:
 branches:
 - main

jobs:
 # Deploy job
 build:
 runs-on: ubuntu-latest # this is the "label" the runner will use and map to docker target OS
 container: 
 volumes: 
 # left: where the output will end up on disk, right: volume name inside container
 - /opt/server-config:/workspace/schallbert/server-config/tmp

 steps:
 - name: --- CHECKOUT ---
 uses: actions/checkout@v3
 with: 
 path: ./tmp
 - name: --- RUN FILE CHANGE TRIGGER ---
 run: |
 cd ./tmp/automation-hooks-trigger
 touch server-config-update.txt

Strange volume errors

But the road to this point was rocky. For a long time I had only specified /server-config under volumes: on the container side and not the working directory of the runner. Then the action runs through and all commands in the #TEST section also work. But when I look on my server, the server-config folder created by Docker remains empty.

I use the following debug code under RUN FILE CHANGE TRIGGER to help me find the path errors:

# deploy-to-server.yml
# [...]
run: |
echo "hello world"
touch updated.txt
pwd
ls -al

I use echo to check whether my code is even being executed in the runner. The touch command puts the current timestamp in the updated.txt file so that I can later use this as a “hook” for further automation. pwd shows me the active path within the runner so that I can correctly map the Docker volume to the server hard drive. ls -al shows me whether the configuration files compiled in the CHECKOUT step were written correctly.

This tells me that the volume path on the “right side” was wrong. I redirect it to the active directory of the runner:

# deploy-to-server.yml
# [...]
# left: where the output will end up on disk, right: volume name inside container
- /opt/server-config:/workspace/schallbert/server-config/
# [...]

Then I got the following to read:

# gitea / act-runner console log
failed to create container: 'Error response from daemon: Duplicate mount point: /workspace/schallbert/server-config'

It seems that the runner automatically creates the “right side” of the mount point itself and therefore cannot be reassigned. Only by adding another path part, in my case /tmp - see above - I fix the error and the long-awaited folder server-config finally appears on my server 😌 with the following content:

schallbert@schallbert-ubuntu-:/opt/server-config# ls -al
total 52
drwxr-xr-x 9 root root 4096 Jul 12 15:38 .
drwxr-xr-x 9 root root 4096 Jul 11 19:56 ..
-rwxr-xr-x 1 root root 395 Jul 11 20:05 boot-after-backup.sh
drwxr-xr-x 3 root root 4096 Jul 11 20:05 borgmatic
drwxr-xr-x 2 root root 4096 Jul 11 20:05 caddy2
drwxr-xr-x 3 root root 4096 Jul 11 20:05 fail2ban
drwxr-xr-x 8 root root 4096 Jul 12 15:38 .git
drwxr-xr-x 3 root root 4096 Jul 11 20:05 .gitea
drwxr-xr-x 4 root root 4096 Jul 11 20:05 gitea
-rw-r--r-- 1 root root 312 Jul 11 20:05 .gitignore
-rw-r--r-- 1 root root 557 Jul 11 20:05 README.md
-rwxr-xr-x 1 root root 371 Jul 11 20:05 shutdown-for-backup.sh
-rw-r--r-- 1 root root 0 Jul 12 15:38 updated.txt
drwxr-xr-x 2 root root 4096 Jul 11 20:05 watchtower

Distributing the configuration on the server

Okay, that’s the first step. I now have a properly configured Git repository that shows my server configuration and can be maintained and at least rudimentarily tested from my laptop. I can also use an automatic Action to store configuration updates on the server and write an update file with a timestamp.

Now the update has to be received on the server, distributed and the affected programs and services have to be restarted. But we’ll look at this in the article Roll out server configuration.

🗸 Gitea Actions Part2 - Jekyll-Dockerimage

Schallbert — Sat, 09 Dec 2023

What happened so far

I wasn’t able to set up the CI/CD pipeline in Part1 of this sequel. I tried with a minor modified copy of the Github-Action to build Jekyll for Github Pages. Now I wanted to try running Jekyll as a Dockerimage right from the start. There wasn’t too much choice on the Docker hub, still I found something which seemed well-documented and actively maintained at year-end 2023.

“Set up job”

So I add the Docker-jekyllimag to my workflow:

# workflows/jekyll-build-action.yml
jobs:
 # Build job
 build:
 runs-on: ubuntu-latest # this is the "label" the runner will use and map to docker target OS
 container: jvconseil/jekyll-docker
# [...]

Here I’m telling gitea act_runner to run with ubuntu-latest label which in my case points to a minimalistic node16:bullseye machine (Debian11). I’m then telling docker to load the jekyll-docker image, where the dependencies I require for my build are already loaded.

bundle install

Well, initial build fails due to ruby dependencies that are specified in my project’s Gemfile which are not present yet. The error message looks like this:

bundler: failed to load command: jekyll (/usr/gem/bin/jekyll)
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.4.22/lib/bundler/resolver.rb:332:in `raise_not_found!': Could not find gem 'github-pages' in locally installed gems. (Bundler::GemNotFound)
	from /usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.4.22/lib/bundler/resolver.rb:392:in `block in prepare_dependencies'

Bundler is Ruby’s packet manager. Similar to npm for Javascript or pip für Python, bundler is able to load and bind dependencies and libraries for applications running in Ruby. The Gemfile specifies the packages for bundler to install.

So let’s add bundle install to the script like so:

# workflows/jekyll-build-action.yml
# [...]
 steps:
 - name: --- CHECKOUT ---
 uses: actions/checkout@v3
 - name: --- INSTALL GEMS ---
 run: bundle install # will fail with permissions rights to write to Gemfile.lock but anyways installs required dependencies.
 - name: --- BUILD WITH JEKYLL ---
 # Outputs to the './_site' directory by default
 run: bundle exec jekyll build --destination /opt/blog_staging
 env:
 JEKYLL_ENV: production

act_runner: no write permissions to `Gemfile.lock`

This fails again. act_runner:

gitea-runner-1 | [Deploy Jekyll site/build] | `/workspace/schallbert/blog/Gemfile.lock`. It is likely that you need to grant
gitea-runner-1 | [Deploy Jekyll site/build] | write permissions for that path.
gitea-runner-1 | [Deploy Jekyll site/build] ❌ Failure - Main ---INSTALL GEMS ---
gitea-runner-1 | [Deploy Jekyll site/build] exitcode '23': failure

After a lenghty search I saw that Gemfile.lock wasn’t added to version control on my local machine because it is mentioned in .gitignore. So bundler in act_runner now tries to create it out of the given Gemfile without success as it only has read access to the checked-out dataset.

To solve this, I add Gemfile.lock to version control, removing it from .gitignore. As an additional benefit I make sure to have identical build environments locally and remote.

Dependency: `sass-embedded`

Less than 20sec into the build I get the following error:

Resolving dependencies...
Could not find gem 'sass-embedded (= 1.69.5)' with platform 'x86_64-linux' in
rubygems repository https://rubygems.org/ or installed locally.

After some more searching, I found a resolution: Specify a certain version of jekyll-sass-converter in the action or, alternatively, add the gem “github-pages” to the Gemfile. The latter manages to load a working version of CascadingStyleSheets (CSS) Preprocessor Sass.

Finally, bundle install finishes successfully:

✅ Bundle complete! 5 Gemfile dependencies, 43 gems now installed.

jekyll build

As I want act_runner to save the build output on my host machine, I add a Volume to Gitea’s docker-compose.yml:

# gitea/docker-compose.yml
# [...]
volumes:
 - ./runner/blog_staging:/opt/blog_staging
 - ./runner/data:/data
 - /var/run/docker.sock:/var/run/docker.sock

Let’s retry the build:

 Destination: /opt/blog_staging
 Generating... 
 Jekyll Feed: Generating feed for posts
jekyll 3.9.3 | Error: Permission denied @ dir_s_mkdir - /opt/blog_staging
/usr/local/lib/ruby/3.2.0/fileutils.rb:406:in `mkdir': Permission denied @ dir_s_mkdir - /opt/blog_staging (Errno::EACCES)
	from /usr/local/lib/ruby/3.2.0/fileutils.rb:406:in `fu_mkdir'

Also this problem can easily be solved: Either you build to /tmp, where the jekyll user has access. Or you hand over the output directory like so: chown -R jekyll /your/build/output.

With this change, I get successful action runs.

Transfer artifacts

Where do I find the build artifacts now? I don’t see any _site folder: Neither on the host machine’s volume, nor in the Gitea or Runner containers.

Docker volumes for `act_runner` cannot share build artifact

This seems logical as I learn the mechanics of act-runner: It will spawn an own action Dockerimage with own volumes that are associated by Task-ID.

DRIVER VOLUME NAME
--> local GITEA-ACTIONS-TASK-84_WORKFLOW-Deploy-Jekyll-site_JOB-build <--
--> local GITEA-ACTIONS-TASK-84_WORKFLOW-Deploy-Jekyll-site_JOB-build-env <--
local act-toolcache
local blog_staging

If I do not extract the build artifacts from the action run, they will be thrown away together with the action run when it has completed and everything is gone. The runner’s volume set up above doesn’t touch the action as it again executes in an isolated image.

To really understand this has cost me a lot of time and many failing action runs. Don’t repeat my mistakes and read the according section in Gitea’s documentation thoroughly.

Well, this turns out more complicated than I thought. As a single maintainer of my website, this is like taking a sledgehammer to crack a nut. Anyways, I’ll follow through.

But I can use `upload-artifact`, right?

Right? Well, at least I thought so. I modified my Action script accordingly and used the upload function from the Github Actions marketplace:

# workflows/jekyll-build-action.yml
# [...]
# Automatically upload the build folder to Giteas blog repo folder
 - name: --- UPLOAD ARTIFACT ---
 uses: actions/upload-artifact@v3
 with:
 path: /workspace/schallbert/blog/
 name: Blog_Staging
 retention-days: 2

Great, now I have a downloadable Zip file on Gitea’s web surface:

But how do I now get that downloaded to the server’s disk? The action is fully encapsuled and cannot even access its underlying Docker Daemon (for good reasons, I guess).

So I search for the assets in the Gitea-Container that persists the artifacts at/data/gitea/actions_artifacts/BUILD_ID. Unfortunately, they are a heap (hundreds) of .chunk.gz files with cryptic numbers as name where I don’t know how to merge into one single archive.

And there’s another thing I don’t like about upload-artifact: My website is becoming bigger and bigger due to an increasing amount of media that I use. That’s why the uploader requires nearly two minutes for compressing and packing - with a linear ascending outlook.

Dead end.

Also Docker `cp` won’t work here

Then I try using Docker’s copy operation to get the artifacts to my host machine:

# workflows/jekyll-build-action.yml
# [...]
- name: --- COPY ARTIFACT ---
 run: docker cp gitea-runner-1:/workspace/schallbert/blog /tmp/blog_staging

This fails like so:

gitea-runner-1 | docker cp gitea-runner-1:/workspace/schallbert/blog /tmp/blog_staging
[...]
gitea-runner-1 | [Deploy Jekyll site/build] | /var/run/act/workflow/3.sh: line 2: docker: not found
gitea-runner-1 | [Deploy Jekyll site/build] ❌ Failure - Main --- COPY ARTIFACT ---

This was expected as the action container knows no Docker. I cannot move files from “inside” the container to another one. Or can I? By forwarding the host machine’s Daemon docker.sock, according to Gitea documentation, I might be tempted… No, I will not continue here - see above, I’d puncture encapsulation if I did.

Transfer artifacts via SFTP

I’m running low on options. I think I could copy the artifacts via SFTP from within the act_runner’s action image into the Webserver-Container¹.

Luckily, there already is an Action for this: scp-files. Now I have to modify the webserver’s container in a way to be accessible via SCP, inhale an SSH-key, and save incoming artifacts.

On the other hand, people in multiple forums say that this is not a clean solution. “Ac container should only accommodate one application. But there’s a solution again: A docker container with only the SSH-Daemon installed. If I have a shared volume between that one and my server, I might be successful.

Looks like more work still. Another container, piping files out and back into my system - that just doesn’t seem right. So I continue my research.

Handover a volume to the Action-Container

All these failures and dead ends have been with me for over a month now. I still really want to be able to build, test and publish automatically. I’ve learned a lot along the way and now I’m hoping that this approach will finally get me to the solution.

During my search I came across the option to change the configuration of the runner. That way, I might end up including a volume that I can share between the action and my server via the daemon on the host.

If that works, I wouldn’t have an additional security risk like a publicly available file transfer container or the long waiting times caused by upload-artifact. So let’s get to work! There are more reasons why volumes can make sense in the respective action - so there must already be people out there who have managed this.

So I add a volume to the Action script:

# workflows/jekyll-build-action.yml
# [...]
container: 
 image: jvconseil/jekyll-docker:latest
 volumes: 
 - /tmp/blog_staging:/blog_staging
 - /opt/cache:/opt/hostedtoolcache

But the artifacts still don’t show up on my host’s filesystem. I strip down the action as much as I can to just check if the volume creates a folder in the action container:

# workflows/jekyll-build-action.yml
# [...]
steps:
 - name: --- CHECK_VOLUME ---
 run: |
 ls -al /blog_staging

But still:

gitea-runner-1 | [Deploy Jekyll site/build] [DEBUG] Working directory '/workspace/schallbert/blog'
gitea-runner-1 | [Deploy Jekyll site/build] | ls: /blog_staging: No such file or directory
gitea-runner-1 | [Deploy Jekyll site/build] ❌ Failure - Main --- CHECK_VOLUME ---

Some more, fruitless tries (maybe I had misspelled anything?) and extensive research in multiple forums I found out that there is a valid_volumes attribute in act. If I don’t have the volume added here, it won’t bind to the container.

OK, let’s add it to the action script:

# jekyll-build-action.yml
# [...]
container: 
 image: jvconseil/jekyll-docker:latest
 valid_volumes: 
 - '**' # This does not work. Also specialized lists indicating the volumes directly won't work
 volumes: 
 - /tmp/blog_staging:/blog_staging
 - /opt/cache:/opt/hostedtoolcache

Still nothing. At least, now, I get a warning in the logs:

gitea-runner-1 | [Deploy Jekyll site/build] [/tmp/blog_staging] is not a valid volume, will be ignored
gitea-runner-1 | [Deploy Jekyll site/build] [/opt/cache] is not a valid volume, will be ignored

Something is wrong with the transfer of this option. There are reports in which the registration of the volumes works. So I take the instructions and create a configuration file where I enter the following:

# runner/config.yml

# Volumes (including bind mounts) can be mounted to containers. Glob syntax is supported, see https://github.com/gobwas/glob
 # You can specify multiple volumes. If the sequence is empty, no volumes can be mounted.
 # For example, if you only allow containers to mount the `data` volume and all the json files in `/src`, you should change the config to:
 # valid_volumes:
 # - data
 # - /src/*.json
 # If you want to allow any volume, please use the following configuration:
 # valid_volumes:
 # - '**'
 valid_volumes: ["/tmp/blog_staging", "/opt/hostedtoolcache"]

Important: The notation must be a string, comma-separated, and always has to specify the Source of the volume. In other words, the part that comes before the :. However, the error message “not a valid volume” still appears when building.

But then I realize that the config must be made available to the act_runner itself as a volume - otherwise the runner running in the container cannot access it at all!

So my docker-compose.yml for Gitea, section “runner” now looks like this:

# gitea/docker-compose.yml
# [...]
 runner:
 image: gitea/act_runner:latest
 environment:
 - CONFIG_FILE=/config.yml
 - GITEA_INSTANCE_URL=https://git.schallbert.de
 - GITEA_RUNNER_NAME=ichlaufe
 - GITEA_RUNNER_REGISTRATION_TOKEN= <redacted>
 volumes:
 - ./runner/config.yml:/config.yml
 - ./runner/data:/data
 - /opt/hostedtoolcache:/opt/hostedtoolcache
 - /var/run/docker.sock:/var/run/docker.sock

And, finally:

gitea-runner-1 | ls -al /blog_staging
gitea-runner-1 | [Deploy Jekyll site/build] | total 8
gitea-runner-1 | [Deploy Jekyll site/build] | drwxr-xr-x 2 root root 4096 Dec 27 07:32 .
gitea-runner-1 | [Deploy Jekyll site/build] | drwxr-xr-x 1 root root 4096 Dec 27 07:46 ..
gitea-runner-1 | [Deploy Jekyll site/build] ✅ Success - Main --- CHECK_VOLUME ---

What an act. I’m so glad that everything is running smoothly now and that I can actually find the build files on my local host system! 🥳

One last tip

If you work with a reverse proxy like I do and get strange connection refused error messages when starting up the runner like:

docker compose up
[+] Running 2/0
 ✔ Container gitea Created 0.0s 
 ✔ Container gitea-runner-1 Created 0.0s 
Attaching to gitea, gitea-runner-1
gitea-runner-1 | level=info msg="Starting runner daemon"
gitea-runner-1 | level=error msg="fail to invoke Declare" error="unavailable: dial tcp <address>: connect: connection refused"
gitea-runner-1 | Error: unavailable: dial tcp <address> connect: connection refused
gitea | Server listening on :: port 22.
gitea-runner-1 exited with code 1

Then your reverse proxy is either not correctly configured or - like in my case - shut down.

Using SFTP (Secure File Transfer Protocol) which is file sharing per ssh (secure shell), I’d increase attack surface of my system just a bit. This is because the SSH client would be reachable for anyone and not just for my action. ↩︎

🗙 Gitea Actions - Jekyll Workflow

Schallbert — Thu, 30 Nov 2023

After setting up my own little server in this project and adding all necessary applications to serve my blog, I now want to automatically build, integrate, and deploy.

Desired outcome

This is how I want it to be: When my Gitea instance detects a git push to the remote origin of my blog’s repository, I want an action runner to start a Docker container. That container shall load Ruby, Bundler, Jekyll, and all other required dependencies. Then it shall checkout the blog’s source files and then run the bundle exec jekyll build command. When its output is ready in the _site folder, I want the runner to self-terminate and to free all occupied resources.

At this point I’d like to manually check my blog’s integrity, maybe also add some automated checks like detecting broken links or similar.

Finally, I want to be able to have the site deployed and thus made available to the public with another simple command.

Starting point: Github Actions

As usual, I first search other people’s solutions for the same problem. I found the Github Actions starter-workflows Repository, created by Github themselves, and copied most of it into my action as a first shot:

# jekyll-build-pages.yml
# Sample workflow for building and deploying a Jekyll site
name: Deploy Jekyll site
run-name: ${{ gitea.actor }} builds Jekyll site
on: [push]

jobs:
 # Build job
 build:
 runs-on: ubuntu-latest
 steps:
 - name: Checkout
 uses: actions/checkout@v3
 - name: Setup Ruby
 uses: ruby/setup-ruby@55283cc23133118229fd3f97f9336ee23a179fcf # v1.146.0
 with:
 ruby-version: '3.1' # Not needed with a .ruby-version file
 bundler-cache: true # runs 'bundle install' and caches installed gems automatically
 cache-version: 0 # Increment this number if you need to re-download cached gems
#[...]

After uploading this workflow to my blog at workflows/jekyll.yml, the runner indeed started its work:

Problems installing Ruby

That was a quick win. On the other hand, the runner wasn’t even able to successfully install Ruby. Here’s the corresponding log:

::error::The current runner (debian-11-x64) was detected as self-hosted because the platform does not match a GitHub-hosted runner image 
(or that image is deprecated and no longer supported).
In such a case, you should install Ruby in the $RUNNER_TOOL_CACHE yourself, for example using https://github.com/rbenv/ruby-build
You can take inspiration from this workflow for more details: 
https://github.com/ruby/ruby-builder/blob/master/.github/workflows/build.yml

The error message looks reasonable and is well-placed.

Background: My runner operates in Docker which uses a stripped-down Debian image named node:16-bullseye (Referenz). Looks like it is incompatible with the Github Actions Runner which executes atop Ubuntu22.

There are quite some options to resolve:

Troubleshooting options

Run the runner on Ubuntu22, use labels for that.
Don’t use Docker but have the runner operate on the host’s operating system
Use prebuilt Ruby, installed in the runner’s cache
Use an out-of-the-box Jekyll-Dockerimage to not require additional installation steps

Operate the runner with Ubuntu22 instead of Node16

There is a matching Dockerimage, but it will take a lot more disk space and RAM than a minimalistic one. Plus, it takes a little more time until it is live. Let’s see whether there is a more simple solution.

Run the runner on the host’s operating system

This has considerable security drawbacks as unprotected branches might allow third-party injected runners via git push --force. In an extreme case, it would blindly execute malware right on my server system. In addition, I’d kill portability by hardwiring the runner to a machine’s OS.

Use pre-built Ruby in runner toolcache

As proposed by the error message, I followed another manual to map a volume containing prebuilt ruby on my host using Gitea’s docker-compose.yml. In addition, I assigned the targeted folder to the RUNNER_TOOL_CACHE environment variable:

# gitea/docker-compose.yml
# [...]
runner:
 image: gitea/act_runner:nightly
 environment:
 - GITEA_INSTANCE_URL=<redacted>
 - GITEA_RUNNER_NAME=<redacted>
 - RUNNER_TOOL_CACHE=/opt/hostedtoolcache
 - GITEA_RUNNER_REGISTRATION_TOKEN= <redacted>
 volumes:
 - ./runner/data:/data
 - /opt/hostedtoolcache:/opt/hostedtoolcache
 - /var/run/docker.sock:/var/run/docker.sock

Here, I map the host’s filesystem path /opt/hostedtoolcache to Docker’s /opt/hostedtoolcache. Docker creates the folders automatically “on both sides” if they are not yet existing on next docker compose up.

Let’s try to install Ruby on the host machine.

So I download the ruby-build repository via git clone and have the installer run ./ruby-build/install.sh. Then, I execute the build with the requested target path from the error message:ruby-build 3.1.4 /opt/hostedtoolcache/Ruby/3.1.4/x64

==> Downloading openssl-3.1.4.tar.gz...
 % Total % Received % Xferd Average Speed Time Time Time Current
 Dload Upload Total Spent Left Speed
100 14.8M 100 14.8M 0 0 81.3M 0 --:--:-- --:--:-- --:--:-- 81.1M
==> Installing openssl-3.1.4...

BUILD FAILED (Ubuntu 22.04 on x86_64 using ruby-build 20231114)

Problems with prebuilt-Ruby

The logs shows the problem:

[...] No C compiler found, please specify one with the environment variable CC [...]

So I type apt install build-essential and confirm with which gcc that I now have a C compiler installed at /usr/bin/gcc Let’s try again!

crypto/comp/c_zlib.c:36:11: fatal error: zlib.h: No such file or directory

Hmm, let’s load it via apt install libz-dev and re-run the insall. This time it takes a while, then:

*** Following extensions are not compiled:
openssl:
 Could not be configured. It will not be installed.
 /tmp/ruby-build.20231205114312.9174.0gmhNf/ruby-3.1.4/ext/openssl/extconf.rb:100: OpenSSL library could not be found. You might>
 Check ext/openssl/mkmf.log for more details.
readline:
 Could not be configured. It will not be installed.
 /tmp/ruby-build.20231205114312.9174.0gmhNf/ruby-3.1.4/ext/readline/extconf.rb:62: Neither readline nor libedit was found
 Check ext/readline/mkmf.log for more details.
*** Fix the problems, then remove these directories and try again if you want.
[...]

OK, openssl this time. Get it with apt install libssl-dev and apt install libreadline-dev and finally :

Installed ruby-3.1.4 to /opt/hostedtoolcache/Ruby/3.1.4/x64

!🎉!

Still no success with the action

Strange, it is the same error all over again:

💬 ::debug::isExplicit: 3.1.4

gitea-runner-1 | [Deploy Jekyll site/build] 💬 ::debug::checking cache: /opt/hostedtoolcache/Ruby/3.1.4/x64
gitea-runner-1 | [Deploy Jekyll site/build] | ::debug::checking cache: /opt/hostedtoolcache/Ruby/3.1.4/x64
gitea-runner-1 | [Deploy Jekyll site/build] 💬 ::debug::not found
gitea-runner-1 | [Deploy Jekyll site/build] | ::debug::not found
gitea-runner-1 | [Deploy Jekyll site/build] ❗ ::error::The current runner (debian-11-x64) was detected as self-hosted

This results from the following Action code:

// setup-ruby/ruby-builder.js
if (common.shouldUseToolCache(engine, version)) {
 inToolCache = common.toolCacheFind(engine, version)
 if (inToolCache) {
 rubyPrefix = inToolCache
 } else {
 const toolCacheRubyPrefix = common.getToolCacheRubyPrefix(platform, engine, version)
 if (common.isSelfHostedRunner()) {
 const rubyBuildDefinition = engine === 'ruby' ? version : `${engine}-${version}`
 core.error( [...] )

Check docker volumes

Hmpf. Let’s see whether Ruby is really available in that volume. To do this, I use docker ps to get the runner’s container ID and enter it here:

docker exec -it <containerID> bash

Then I navigate to cd /opt/hostedtoolcache, list its contents with ls and there it is: Ruby. It is there but still cannot be found in the Action.

Accept my failure

No way to proceed for me at this point 😖. I opened this problem to the community as act_runner cannot find hostedtoolcache and will try

Another option.

Root cause

Looks like the support of setup-ruby action for self-hosted runners like mine might have been discontinued. Maybe the documentation, creating the above error message, has not been updated accordingly.

Hardware Test

Schallbert — Sun, 23 Apr 2023

A new drive

Recently, I bought a spindle upgrade. I now switched to a 3-phase induction motor with forced air cooling that requires a Variable Frequency Drive so I can set different speeds. The system takes the same analogue 0-10V signal as my previous motor. For the numerical controller it looks the same with one exception: The VFD can now create an emergency stop signal if something goes wrong.

As you can imagine, the new spindle subsystem is by far more complex than the old one. I had to buy, wire, and setup a dedicated electric control box.

So why testing?

With higher complexity comes higher risk of failure. There are more electrical and mechanical parts involved that each can behave incorrectly or even be destroyed if connected improperly. As I am just a human being who makes mistakes, I’d better not jump in at the deep end. So, while wiring everything up, I wondered how such a system should be systematically tested.

Test Strategy

I borrowed the different test scopes from my job in the software industry. I think they fit in quite well:

Unit: Test that verifies behavior of a component of the system in isolation. All external influences are removed or mocked away. Example: Check if the blower turns if supplied with its nominal voltage.
Integration: These tests integrate different components that provide a common functionality. Example: Verify that spindle cooling works with delayed shutdown.
Subsystem: Test that verifies the correct interaction of all components within a subsystem. Example: The spindle speed changes when feeding the analogue input of the VFD with different voltages.
System (End2End): Tests that verify a system from the end-user perspective. Example: When pressing the emergency off switch of the CNC, does the VFD bring the spindle to a halt safely?

Verify early

From my experience, it makes sense to perform tests at the earliest possible point in time. After the spindle was delivered, I was able to check right away whether all bearings were smooth-running, if the fan would turn when voltage was applied, and see whether the motor windings were within expected resistance range, and so on. I could do these unit-level tests even before I started building or wiring the control box.

Your browser does not support the video tag.

Unit test: Spindle cooling fan

This might save a lot of troubleshooting when something doesn’t seem to work correctly during comissioning. And if you find something this early, you can directly get in touch with the manufacturer of the affected component to request an exchange while not being blocked on the other end.

Bottom-up testing

The four levels of testing are my verification strategy here. I try to have many simple and quick early test on unit level, and just a few complex system tests that each involve a multitude of preparation and test steps, being executed just once upon initial startup when the build is complete.

When you look at the concept of the test pyramid that is used a lot in software testing, it may be applicable for hardware tests as well to some extent, maybe with a different number of layers and varying names for the levels of integration.

Test plan

I don’t usually write test plans for hobby projects because the systems I design are rarely as complex as this. Plus, most of them do not require dangerous voltages to operate unlike this one.

To better visualize the system, I drew a layout of all components with the most important inputs and outputs.

From that image, I derived a test plan. I tried to cover each component’s interactions with each other, ending up in more unit than system test cases.

Your browser does not support the video tag.

Unit test: Supply +24V rail OK

Here I checked whether the cooling fan power supply operates correctly.

Your browser does not support the video tag.

Unit test: Time delay setting OK

With this test I verify that the cooling fan relay configuration and activation work as expected. For the purpose of this test, I selected shorter delay times than in the later application.

Your browser does not support the video tag.

Integration test: VFD initial operation

The variable frequency drive is able to turn the spindle in the correct direction of rotation at different speeds.

Test execution

Of course, there were many more hardware-based tests that I didn’t mention in the above plan. Things like is the spindle correctly aligned to the Z-axis? Have I fastened all clamps with the appopriate torque? Are all connectors inserted and locked?

Before trying to run through material on the CNC I performed all the tests and actually found one issue on the system level: I had a pinning error in the spindle speed analogue output of the numerical controller’s signal harness so that the spindle motor system wouldn’t start despite the run signal being present.

This just took minutes to figure out because I had subsystem-tested the VFD before (Test case: “Reacts to speed command”), knowing that it cannot be the culprit.

Documents

In case you plan on adding a VFD-controlled spindle motor to your machine, please find my full test plan below.

VFD-controlled spindle motor test plan

But please take this plan with a grain of salt and note the usual disclaimer:

⚠️ Risk of electric shock. ⚠️

This kind of system should only be built or worked on by professionals. Especially avoid touching live parts or VFD components before the VFD’s bleeder resistors have had enough time to discharge the intermediate circuit’s bulk capacitors.

Github Actions: Troubleshooting

Schallbert — Tue, 14 Feb 2023

Motivation

Why I’m writing a short post about Github Actions? Because I wanted to use it for my latest Software-project, QR-codengrave. When at some distant point in the future, I have a new computer, don’t remember which IDE I used to build and run, test, and create assets with. Or, say, I have a corrupted virtual environment or launch.json, I still want to be able to deploy that application, publish bugfixes or bump a release.

Although that’s not the usual argument for using CI/CD - typically the first reason being that collaboration on a piece of software becomes easier - still the appeal is strong enough to try Github’s automation called Github Actions.

I won’t try to run nightlys or have a continuous deployment pipeline as an end in itself, though, and only want an automated build, lint, test run, and deploy stuff when merging back to the production branch.

Got it, but why a dedicated post?

Because it was such a pain to get it right. I spent hours and hours pushing and hoping that, this time, the goddess of Github Actions would actually have my solution built without errors and at least run some of the tests.

Note: If I had been more diligent, I would have installed yet another tool like this to have my actions run locally, both saving time and the embarassment of tens of failed builds in a row. But I wasn’t.

I had quite some errors that appeared on the way of making QR-codengrave. Some I wasn’t even able to properly resolve so I had to utilize work-arounds. But let’s get to that later.

The script

Github Actions uses YAML to take orders for its pipelines. It is very well documented and provides many readily-working scripts for most scenarios and programming languages.

The so-called “Workflow file” looks like this:

# This workflow will install Python dependencies, run tests and lint with a single version of Python
# For more information see: https://docs.github.com/en/actions/automating-builds-and-tests/building-and-testing-python

name: python_integrate

on:
 push:
 branches: [ "master" ]
 pull_request:
 branches: [ "master" ]

permissions:
 contents: read

jobs:
 build:

 runs-on: ubuntu-latest

 steps:
 - uses: actions/checkout@v3
 - name: Set up Python 3.10
 uses: actions/setup-python@v3
 with:
 python-version: "3.10"
 - name: Install dependencies
 run: |
 python -m pip install --upgrade pip
 pip install flake8 pytest
 pip install qrcodegen # Dependency install of qrcodegen
 if [ -f requirements.txt ]; then pip install -r requirements.txt; fi
 - name: Lint with flake8
 run: |
 # stop the build if there are Python syntax errors or undefined names
 flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics
 # exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide
 flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics
 - name: Test with pytest
 uses: GabrielBB/xvfb-action@v1  # Diverts tkinter GUI to a virtual frame buffer (VFB)
 with:
 run: |
 pytest

And it does not divert too much from the standard template that I was using first. The only things I added are the run pytest entry and the usage of a virtual frame buffer to cirumvent issues with my GUI which makes the tests fail when the GUI tries to fire up and there’s no screen to show it on.

Github Action runner errors

The following list of fails describe the errors and resolutions that I faced until I had a stable CI at around action run #50.

Directory mismatch?

Output:

/opt/hostedtoolcache/Python/3.10.9/x64/lib/python3.10/importlib/__init__.py:126: in import_module
 return _bootstrap._gcd_import(name[level:], package, level)
test/test_machinify_vector.py:4: in <module>
 from bin.platform.machinify_vector import MachinifyVector, Tool, EngraveParams
E ModuleNotFoundError: No module named 'bin'

Reason: I placed my source files under /bin and made it available to a local instance of PyInstaller (the tool I chose to render my python sources into an executable under Windows). Upon upload, Github Actions couldn’t relate to the paths I chose and had that error prepared for me.

Resolution: Add an empty file with name __init__.py into the /bin folder. This will flag anything in that folder as package, thus making it available to GH actions. So what looked like a directory mismatch actually was a package-not-found error that my IDE didn’t have as it knew which files I had created.

YAML syntax errors

This error occurs because I was trying to put two workflows into one file. Somehow Github Actions seems to only accept a single one per file.

Path-not-found errors

I had a lot of those and they were painful to resolve. The folder structure that I was targeting looked like:

- assets  # images, persistence file, etc.
- src  # source files
- test  # pytest files for unit and integration testing
- dist  # build artifacts
- build  # build process files

But somehow, it was super hard to make my local IDE build environment, Pyinstaller, and the Github Actions Worker and its trigger of Pyinstaller to cooperate. One of those four parties would always complain that a path is missing or something else was wrong. That’s why I decided to abandon relative paths and instead use python’s importlib_resources. But in the end, this turned out to be problematic with Pyinstuller on remote Github Actions.

What finally solved the problem was to move assets into src. This way, the ./ command couldn’t go wrong anywhere and although I don’t like the construct very much, I was tired of putting more time into that for a clean fix (If you know how to handle path pointing in Python for both local and remote, let me know in the discussions).

Tkinter headless testing

Later in the development process, I decided to add some integration tests into my solution. This way I wanted to make sure that child windows would actually perform the callbacks to update the main application, and vice versa e.g. that the persisted tool list is cascaded into the tool configuration window.

These tests were running locally but with a disadvantage: When I simulated an error or warning case, the corresponding messageBox would insist to be closed manually by the user before continuing the tests. Which was just a bit annoying for me in the IDE to do, but would just be impossible to do from within Github Action’s CI pipeline.

So I bit the bullet and added a wrapper class to tkinter’s messageBox so I could inject a mock that wouldn’t actually trigger the popup:

class MsgBox:
 """Re-implementation due to testing purposes: With this trick, we are able to mock these windows
 so we do not have to wait for users to manually close the dialog, unblocking the application again."""
 def showinfo(self, title, message):
 showinfo(title=title, message=message)

 def error(self, title, message):
 showerror(title=title, message=message)

Although in theory it should be possible also to invoke the popup’s “OK button” from within the test, I wasn’t able to have that automated reliably.

When this worked, I pushed to Github with high expectations - and got another beautiful error:

_tkinter.TclError: no display name and no $DISPLAY environment variable

At least that was easy to understand: Tkinter just didn’t know where to draw the windows - seems legit when there’s no display connected.

So I searched the internet and found a single line of code that miraculously solved my problem by introducing a virtual frame buffer:

uses: GabrielBB/xvfb-action@v1 # Diverts tkinter GUI to a virtual frame buffer (VFB)

Now, on Push, Github Actions rained green ticks down on me which felt really good for a change.

Jekyll `-incremental` option

Schallbert — Thu, 03 Jun 2021

The images

I worked on my assets/images, they tend to be somewhat “big” so that page load times might be a reason to worry about. I’m using paint.net to shrink them to a size I think I can afford and save them as .jpg with compression max’ed out and some compromises on image quality. This way, most of my pictures take less than 10% or the original size in kB.

Github Pages: Moving out

Schallbert — Mon, 01 Jan 0001

Project stats

Difficulty: medium 3/5
Cost: 5-50€/Month
Time: ~10h

Motivation

I decided to move my Website from Github-Pages to my own server. I have two reasons for that: First, I want the server to be situated in Germany (data protection) and second, I wanted a simple way of providing my page in two languages.

In addition, I might be able to realize additional wishes: Setup a miniature file server and add a Website for my small side-hustle. But those will be handled in a future post.

Overview

In this project, I’ll rent a virtual server, configure access via SSH and install a container manager called Docker.

In one of the containers, I want an instance of gitea to provide version controls for website, and have a CI/CD¹ pipeline setup that automatically integrates and deploys my site.

Not unlike Github, Gitea Actions allows me to deploy the site built with Jekyll via Workflow files on a caddy-Server instance.

caddy is not just a webserver, but also provides a reverse proxy which does Port mapping and address resolution. In addition, caddy manages my https certificates easily per Let’s Encrypt.

As caddy will run in an own docker container, I have to generate and configure virtual networks so the containers can “talk” to each other.

Let’s get going - step by step.

Configuring my server

The upcoming sections describe the individual steps of configuring a cloud-server for deployment of a static site, including version control and CI.

configure SSH for the server

Right after ordering the “Cloud-server”, I enabled SSH access and deactivated its password login. To connect from my machine, I check whether the ssh-agent is already up and running per eval ssh-agent.

Then, I create a new key pair via

ssh-keygen -t ed25519 -C "your_email@example.com" -f "~/.ssh/my-cloud-server"

This line creates a key pair with algorithm (-t) ed25519, taking a mail address as comment (-C) that is then saved with filename (-f) my-cloud-server.

The public key file my-cloud-server.pub is then uploaded on the web portal of my server using the “security” tab. The resulting fingerprint is then verified with:

ssh-keygen -lf <fingerprint_from_server> "~/.ssh/mein-cloud-server"

Finally, I modified the .ssh/config file so that, to connect, I just need to enter ssh server into my console instead of the server’s IP-Address - pretty Lazy 😀

#.ssh/config
Host server
 User root
 Hostname ip_address_of_server
 PreferredAuthentications publickey
 IdentityFile ~/.ssh/my-cloud-server

Now when I enter ssh server the first time, I get a prompt about a potential connection to a yet unknown host with fingerprint xyz and get asked whether I’d want that to happen. I compare this fingerprint to the server’s (use provider’s Website to verify). This way I can make sure that I’m trying to connect to the correct party. Once I give my yes, the fingerprint is added to the known_hosts file and future connections will be made directly and without warning.

Install Docker 🐳

My server runs a standard Ubuntu that supports apt commands out of the box. So installing Docker per Manual is really easy: Connect to the server per SSH, type apt install docker, done. I’ll be using docker-compose which also auto-installs. Compose interprets configuration files that define the target container and make it really easy to get them up and running.

Install caddy 🛒

First, I create a folder in my /opt directory via mkdir caddy2. To have caddy run in an own container, I change into that directory and create the following docker-compose.yml file:

#caddy2/docker-compose.yml

version: '3'
services:
 caddy:
 image: caddy:latest
 container_name: caddy
 ports:
 - "80:80"
 - "443:443"
 environment:
 - ACME_AGREE=true
 restart: unless-stopped
 volumes:
 - ./Caddyfile:/etc/caddy/Caddyfile
 - ./caddy_data:/data
 - ./caddy_config:/config
 - ./www:/www
 networks:
 - caddy-proxy

networks:
 caddy-proxy:
 external: true

This file tells docker to use latest caddy release. The container is named caddy and uses ports 80 (http) and 443 (https). Certificates are requested via ACME_AGREE and the container is able to access required files (Caddyfile, data, config) on my host machine via a volume. It will also be able to access files in /www folder that, in the future, will host my blog’s site data. The container is part of the caddy-proxy network that has been provided externally.

So, to be able to run this, I’ll first have to create that network:docker network create caddy-proxy. Then a quick check per docker network ls shows that the network has been added:

NETWORK ID NAME DRIVER SCOPE
<redacted> bridge bridge local
<redacted> caddy2_default bridge local
<redacted> caddy-proxy bridge local

Now I create a Caddyfile configuration with following content:

# caddy2/Caddyfile
blog.schallbert.de {
 root * /www/blog
 encode gzip
 file_server
}

Caddy is told that the content of /www/blog shall be provided as a (static) file server, supporting gzip-compression, available under blog.schallbert.de.

Create a test website 🧪

Before we start with the page design, we should be able to have the address resolved². I log into my DNS-manager and bind a static IP address to my subdomain:

TYPE NAME VALUE TTL
A blog.schallbert.de <redacted> 86400

My server shall be able to show something. So let’s create a minimalistic test page:

mkdir ~/caddy2/www/blog
cd ~/caddy2/www/blog
nano index.html

Hmm, Just a little something:

<!-- blog/index.html -->
<h1>ALL YOUR BASE ARE BELONG TO US</h1>

Save it, done.

This is how caddy’s root directory looks like now:

*caddy_config* *caddy_data* Caddyfile docker-compose.yml *www*

We’re ready now to create the container:

docker compose up -d

The Option -d means “detach”, so the container is started and detached from the console, unblocking it so we can reuse it for more prompts.

[+] Running 1/1
 ✔ Container caddy Started

This looks good. Now I navigate to the page:

Install Gitea 🍵

I need Gitea for two purposes: I want version control also for private repositories and I want to host my website with Jekyll with full CI/CD automation.

Like caddy, it starts with a folder:

cd /opt
mkdir gitea
cd gitea
nano docker-compose.yml

I compiled Gitea’s docker-compose.yml out of dfferent example files. Here it is:

# gitea/docker-compose.yml
version: "3"

networks:
 gitea:
 external: false
 caddy-proxy:
 external: true
 name: caddy-proxy

services:
 gitea:
 image: gitea/gitea:latest
 container_name: gitea
 restart: always
 environment:
 - USER_UID=1000
 - USER_GID=1000
 - SSH_DOMAIN=git.schallbert.de
 - GITEA_HOSTNAME=git.schallbert.de
 networks:
 - gitea
 - caddy-proxy
 volumes:
 - ./gitea:/data
 - ./log:/app/gitea/log
 - /etc/timezone:/etc/timezone:ro
 - /etc/localtime:/etc/localtime:ro
 ports:
 - "127.0.0.1:3000:3000"
 - "222:22"
 runner:
 image: gitea/act_runner:nightly
 environment:
 - CONFIG_FILE=/config.yml
 - GITEA_INSTANCE_URL=https://git.schallbert.de
 - GITEA_RUNNER_NAME=ichlaufe
 - GITEA_RUNNER_REGISTRATION_TOKEN=
 volumes:
 - ./runner/config.yml:/config.yml
 - ./runner/data:/data
 - /var/run/docker.sock:/var/run/docker.sock

This is what it does: gitea is member of docker networks gitea and caddy-proxy. The latter has been added so I can reach gitea from “the internet”. The container running the service is named gitea and is automatically restarted when down. User-ID is set to 1000 so it won’t interfere with admins or other special users. Per SSH and http, gitea can be reached under git.schallbert.de on ports 22 and 3000 - the latter is localhost only.

Gitea has access to the host’s time zone and local time, it saves its logs under /logand files under /data. Gitea gets an action runner that runs on the gitea instance. Its name is ichlaufe and the config is available under config.yml as a volume. The runner can access Docker daemon’s standard-socket to run actions, e.g. install ubuntu-latest in an own container to run Jekyll builds. I took the basic config for the runner from the official repository.

Gitea will only accept runners that are registered via GITEA_RUNNER_REGISTRATION_TOKEN. We’ll do this at a later point in time.

Now we start Gitea for the first time and look at the console print:

docker compose up
[+] Running 4/2
 ✔ Network gitea_default Created 0.0s 
 ✔ Network gitea_gitea Created 0.1s 
 ✔ Container gitea Created 0.1s 
 ✔ Container gitea-runner-1 Created 0.0s 
Attaching to gitea, gitea-runner-1
gitea | Server listening on :: port 22.
gitea | Server listening on 0.0.0.0 port 22.
gitea | 2023/11/18 17:19:52 cmd/web.go:242:runWeb() [I] Starting Gitea on PID: 15
gitea | 2023/11/18 17:19:52 cmd/web.go:111:showWebStartupMessage() [I] Gitea version: 1.21.0 built with GNU Make 4.4.1, go1.21.4 : bindata, timetzdata, sqlite, sqlite_unlock_notify
gitea | 2023/11/18 17:19:52 cmd/web.go:112:showWebStartupMessage() [I] * RunMode: prod
gitea | 2023/11/18 17:19:52 cmd/web.go:113:showWebStartupMessage() [I] * AppPath: /usr/local/bin/gitea
gitea | 2023/11/18 17:19:52 cmd/web.go:114:showWebStartupMessage() [I] * WorkPath: /data/gitea
gitea | 2023/11/18 17:19:52 cmd/web.go:115:showWebStartupMessage() [I] * CustomPath: /data/gitea
gitea | 2023/11/18 17:19:52 cmd/web.go:116:showWebStartupMessage() [I] * ConfigFile: /data/gitea/conf/app.ini
gitea | 2023/11/18 17:19:52 cmd/web.go:117:showWebStartupMessage() [I] Prepare to run web server

Configure web access

Now I have another subdomain added to my DNS manager:

TYPE NAME VALUE TTL
A git.schallbert.de <redacted> 86400

I also have to modify Caddyfile so that web requests are forwarded accordingly (reverse proxy). To do this, I navigate back to the caddy2 folder, type nano Caddyfile and add the following section:

# caddy2/Caddyfile
git.schallbert.de {
 reverse_proxy * http://gitea:3000
}

Now requests to git.schallbert.de are forwarded to the (local) port 3000 of the Gitea-container that I opened in the docker compose file before.

Configuring Gitea per web page

When I use my browser to navigate to the above page, I get a login screen - identical to this demo.

Now I create a nice password, click Register and then configure Gitea using my browser. I only do a rough config because I’ll have to modify it again later to properly add the runner.

When done, I create a new SSH key pair as described in the section Configure SSH above with filename -f ~/.ssh/gitea. It is uploaded and verified under Settings->SSH/GPG Keys.

Now I can create another entry on my local machine’s .ssh/config file:

# .ssh/config
Host gitea
 User schallbert
 Hostname git.schallbert.de
 PreferredAuthentications publickey
 Identityfile ~/.ssh/schallbert_gitea

Let’s check if this works:

ssh gitea
Host 'git.schallbert.de' is known and matches the ED25519 host key.
debug1: Found key in /home/schallbert/.ssh/known_hosts

Perfect.

Complete Gitea config via `app.ini`

As the configuration is not complete, I shut down the container again:

docker compose down

and then navigate to /gitea/gitea/conf and modify the app.ini just a little:

# conf/app.ini
APP_NAME = Gitea: Git with a cup of tea
RUN_MODE = prod
RUN_USER = git
WORK_PATH = /data/gitea

[repository]
ROOT = /data/git/repositories
DEFAULT_BRANCH = main

[repository.local]
LOCAL_COPY_PATH = /data/gitea/tmp/local-repo

[repository.upload]
TEMP_PATH = /data/gitea/uploads

[server]
APP_DATA_PATH = /data/gitea
DOMAIN = git.schallbert.de
SSH_DOMAIN = git.schallbert.de
HTTP_PORT = 3000
ROOT_URL = https://git.schallbert.de/
DISABLE_SSH = false
SSH_PORT = 222
SSH_LISTEN_PORT = 22
LFS_START_SERVER = true
LFS_JWT_SECRET = <redacted>
OFFLINE_MODE = false

[database]
PATH = /data/gitea/gitea.db
DB_TYPE = sqlite3
HOST = localhost:3306
NAME = gitea
USER = root
PASSWD =
LOG_SQL = false
SCHEMA =
SSL_MODE = disable

[indexer]
ISSUE_INDEXER_PATH = /data/gitea/indexers/issues.bleve

[session]
PROVIDER_CONFIG = /data/gitea/sessions
PROVIDER = file

[picture]
AVATAR_UPLOAD_PATH = /data/gitea/avatars
REPOSITORY_AVATAR_UPLOAD_PATH = /data/gitea/repo-avatars

[attachment]
PATH = /data/gitea/attachments

[log]
MODE = file
LEVEL = info
ROOT_PATH = /data/gitea/log

[security]
INSTALL_LOCK = true
SECRET_KEY = <redacted>
REVERSE_PROXY_LIMIT = 1
REVERSE_PROXY_TRUSTED_PROXIES = *
INTERNAL_TOKEN = <redacted>
PASSWORD_HASH_ALGO = pbkdf2

[service]
DISABLE_REGISTRATION = true
REQUIRE_SIGNIN_VIEW = false
REGISTER_EMAIL_CONFIRM = false
ENABLE_NOTIFY_MAIL = false
ALLOW_ONLY_EXTERNAL_REGISTRATION = false
ENABLE_CAPTCHA = false
DEFAULT_KEEP_EMAIL_PRIVATE = false
DEFAULT_ALLOW_CREATE_ORGANIZATION = false
DEFAULT_ENABLE_TIMETRACKING = true
NO_REPLY_ADDRESS = noreply.localhost

[lfs]
PATH = /data/git/lfs

[mailer]
ENABLED = false

[openid]
ENABLE_OPENID_SIGNIN = false
ENABLE_OPENID_SIGNUP = false

[cron.update_checker]
ENABLED = false

[repository.pull-request]
DEFAULT_MERGE_STYLE = merge

[repository.signing]
DEFAULT_TRUST_MODEL = committer

[oauth2]
JWT_SECRET = <redacted>

[actions]
ENABLED = true

Here it is important to make sure that ROOT_URL and port configuration match the one specified in docker-compose.yml. I configure its database (sqlite3) and make it available to Gitea. Plus, I deactivate Giteas ability to register new users DISABLE_REGISTRATION = true and the option to sign up/in via openID ENABLE_OPENID_SIGNIN = false ENABLE_OPENID_SIGNUP = false. Finally, I enable Gitea Actions ENABLED = true and save the file.

Now I can restart the container:

docker compose up
[+] Running 4/4
 ✔ Network gitea_default Created 0.1s 
 ✔ Network gitea_gitea Created 0.0s 
 ✔ Container gitea Started 0.0s 
 x Container gitea-runner-1 Error 0.1s

Very well, Gitea is up and running. In a last step, I configure the Action Runner.

Setup Action Runner 🏃

I use the official documentation provided by Gitea for inspiration.

I login to Giteas Web surface and see that activating Actions in the app.ini added another menu: Settings->Actions. Here I click Create new Runner und copy its REGISTRATION TOKEN.

Then, I re-open Gitea’s docker-compose.yml and add the following lines as recommended in the doc page:

# gitea/docker-compose.yml
# [...]
runner:
 image: gitea/act_runner:nightly
 environment:
 - CONFIG_FILE=/config.yml
 - GITEA_INSTANCE_URL=https://git.schallbert.de
 - GITEA_RUNNER_NAME=ichlaufe
 - GITEA_RUNNER_REGISTRATION_TOKEN= <redacted>

I restart the container with docker compose down && docker compose up -d and get a “started” message for the runner in the logs:

✔ Network gitea_default Created 0.1s 
✔ Network gitea_gitea Created 0.0s 
✔ Container gitea Started 0.0s 
✔ Container gitea-runner-1 Started 0.1s

Great, let’s have a look at the web page:

Now we have to know if the runner can do something for us.

Runner functional test

Once again, I have a look at the Runner’s documentation. To check, I create a new Repository on the web surface, name it runner-test, and add a file to /.gitea/workflows/01-test.yml containing the recommended runner check.

Commit, push, and - nothing.

Oh, the workflow starts on: [push]. This cannot work when it has just been uploaded itself.

Well, let’s touch another random file in the repository. Commit, push, result:

Fantastic! 🥳

Interim conclusion

In this article, I pretended that the road to a working server setup was straight and without obstacles. Of course, that is not entirely true. It was really simple to run images in Docker, but to have them communicate turned out less straight-forward. I also had quite some problems with Caddyfile as it is hard to debug and I couldn’t see what I was doing wrong.

For instance, I was able to ping the website on my host but couldn’t reach it from the outside. So I knew it is the caddy config to modify.

Root cause was missing the Caddyfile in the Docker volume list and forgetting a leading / in the target volume folder.

On the other hand, configuring Gitea was much easier than I thought. Even complex process of registering the runner and its routing through Docker worked at first try.

Now I have a quick and lightweight server that will be able to host my blog with ease.

Next steps - creating a build process for my site - is tried in this post, but then replaced with native Jekyll over here.

Ultimately, I want to get back to creating content. And not spend any more time than necessary on all the other stuff.

CI/CD “Continuous Integration / Continuous deployment” cares for automatic build, verification, and deployment of my web page. ↩︎
DNS = Domain Name System, a distributed hierarchical system for name resolution of websites. It exists because it’s much less straight forward to enter an IPV6 address like 2a02:ec80:300:ed1a:0:0:0:1 into your web browser than en.wikipedia.org. ↩︎

GoogleTest Demo

Schallbert — Mon, 01 Jan 0001

Project stats

Difficulty: medium 3/5
Cost: 0€
Time: ~2h

Motivation

It is nice to have a powerful unit test framework at hand when developing more complex projects, possibly saving a major amount of time debugging (especially on-Target debugging in the embedded world can be really time-consuming), improving code quality and possibly boosting insights to APIs and architecture for others.

The embedded world

Traditionally, microcontrollers were limited on the memory side. In the past, they neither had much RAM nor PROGMEM so the code I wrote was mostly in lean C, and configuration took place in lists of #define that wouldn’t weigh much because the preprocessor would do the work; not the µC at runtime. My unit tests were alyways bound to the hardware they were running on - take Unity as a well-known example - but now I wanted to try something new: Write Object-oriented style code, and get the business logic tested independently of the hardware. I selected the GoogleTest framework and I wanted a seamless integration into my workflow with VScode and its extension PlatformIO.

Until recently, the only way to do this was to run GoogleTest in parallel using the GNU Compiler Collection and have it controlled by PlatformIO. Recently, however, GoogleTest can be used directly as a PlatformIO Library. Therefore, I’ll cover both options here.

Using GoogleTest directly in PlatformIO

The installation is particularly easy here:

Clone my Github repository PlatformIO-gTestgMock.
Open a console window in the repository folder and switch to the corresponding branch using git checkout googletest-native.
Follow the instructions in the repository’s README file and test the setup.

This is what it looks like when the tests have been successfully run:

Using GoogleTest with GCC

You can install the required software yourself (even on a Windows¹ computer) to get started with GoogleTest.

Download and install MinGW

See this thread in the PlatformIO forum if you have any questions about the following steps.

Unix users: Skip the steps up to this point, as you probably only need the libpthreadgc libraries. Check for the presence of all libraries (but omit the mingw32- prefix) and install any missing ones. Make sure the PATH variable is set correctly.
Windows users:

Download MinGW, e.g., from Sourceforge
Install by running mingw-get-setup.exe. If you like graphical user interfaces, leave the corresponding box checked. I do not recommend changing the installation directory. If you do, you’ll need to remember the new path.

Configure MinGW

For GoogleTest to run, you’ll require additional packages to be installed:
- mingw32-gcc-g++ (of course!)
- mingw32-libmingwex
- mingw32-libmingwex-dll
- mingw32-libmingwex-dev
- mingw32-libpthreadgc-dll
- mingw32-libpthreadgc-dev
To install these,
- If you use the GUI: search libraries under “MinGW Standard Libraries”, add/activate/install them one by one.
- Command Line Interpreter users: type mingw-get install and then add the package name. Easy. If you get an error message saying that some of those do already exist, that’s good for you!
Now you need to add MinGW’s \bin folder to your system path variables so PlatformIO can find them later. If you didn’t hand-modify it, it should be C:\MinGW\bin.
- GUI fans: follow this guide
- CLI: type set PATH=%PATH%;C:\MinGW\bin, check if it worked with echo %PATH:;=&echo.%.

Congratulations! You have installed MinGW, set its environment variables to be found more easily by third party applications, and configured it to work with GoogleTest.

Configure PlatformIO

After setting the PATH variable, you’ll have to restart VScode.
To make PlatformIO correctly link to the compiler you just install, make it use the native environment where your compiler sits. Open a PlatformIO terminal within one of your projects and type pio platform install native
Two choices:

Follow through this tutorial and use my example project as described below. You can safely skip the upcoming section as that project comes pre-configured and is open for your modifications as you have basic hardware abstraction interfaces at the ready right from the start.
Take your own project and modify its Platformio.ini so that GoogleTest can be run.

This what you should add to your Platformio.ini:

[env:desktop]
platform = native
build_flags = 
	-std=gnu++11 # use installed GNU C++11 compiler.
	-pthread # found in gtest documentation
lib_deps = 
	googletest # Will automatically load latest googletest lib
lib_ignore = 
 src # most main.cpp's directly access the hardware. 
 # Can be removed if your project is different.
 #
 # PLUS: Any files that contain bare-metal hardware commands 
 # (e.g. digitalWrite), g++ won't have the headers or specialized compiler knowledge!
lib_compat_mode = off # Must-have for external stuff like gtest!

For the environment desktop (= your computer without target), it uses the native compiler which is GCC/G++ that you installed and set the PATH for. GoogleTest uses C++11 standard and threading which you have to add to the build flags. Of course, the build depends on the googletest library and you want to ignore all files/folders that contain source or header files using hardware-related commands or headers. Library compatibility mode has to be switched off for the Library Dependency Finder to include googletest.

(Optional) In case you like using GUI buttons better than typing a line into your console to start the tests, create a custom configuration.

Running Tests

For a quicker start, you can clone my example repository (on branch main). It contains all the necessary elements to test whether gTest and gMock are working correctly. This allows you to immediately verify whether the installation was successful.

Unzip the zip file or use SSH to clone the repository.
Open the PlatformIO extension home page, click “Open Project,” and select the folder.
To run the unit tests, open a PlatformIO terminal and enter pio test -vvv -e desktop -f test_desktop (-vvv = verbose -e = environment, -f = filter). The parameters are required because the tests can only be run on the desktop environment, not on the microcontroller.
After a while, the message shown below should appear.

Great! Unit tests within PlatformIO have been built and executed!

Tips and Tricks

Note that your PlatformIO_gTestgMock/.pio/build/desktop folder now contains an executable called program.exe. If you run this from the CLI of your choice, you should see the same result as in the above image. You might want to choose this option more frequently once you have more tests that not only fail but crash the program on execution, and to debug tests (and not the user code), as PlatformIO’s CLI tends to hang or miss out the reason for why the test crashed. You might also want to have the stack trace printed in these cases although this would really go far beyond scope of this exercise.

From here,

you might want to continue reading below for a deeper understanding of the repository contents and how GoogleTest works
you could go practice writing tests by adding more cases, e.g. to verify if the LOW state is looped through correctly
You could also enhance the interface of these groundworks, e.g. by adding the analogRead() function to the hardware abstraction layer
You could write your project logic on top of this, cleanly cut out the hardware dependencies, and enjoy the ability for easy porting of your project to different hardware devices and manufacturers
Or do this other brilliant stuff that you were up to before you were diverted by this tutorial.

Repository contents explained

The following sections cover most of the files in the repository and explain what they do in detail. The implemented business logic is minimal and should be seen as example code - to create the least confusion possible when dealing with this complex matter. For a real-life project that uses all of the patterns illustrated on this page, please refer to my Tonuino page.

Hardware abstraction layer interface

Let’s have a look at the bespoke interface and how it’s structured.

// hal_if.h
namespace hardwareAbstraction {

// Hardware Abstraction Layer Interface class
class Hal_IF
{
public:
 virtual ~Hal_IF() = default;

 virtual bool digitalRead(uint8_t pinId) const = 0;
 virtual void digitalWrite(uint8_t pinId, bool value) const = 0;
};

} // hardwareAbstraction

The Hal_IF class has two methods which are both virtual. This means that a client class using this interface can call the method, and is thus “auto-forwarded” to the concrete implementation of the method. The = 0; makes these methods pure virtual, so overriding them with a derived class is not optional but a must. This is what we want here, because if this method wasn’t overridden, it wouldn’t have any behavior. Note that the destructor of an interface class almost always should be virtual, too.

LoopInOut

The logic under test. Note that it does not directly access the input/output hardware of the µC, but the m_hal variable which is an implementation of the Hal_IF interface at runtime. It is passed into the class by reference (marked by the & at the type declaration) upon object construction².

// loopInToOut.h

class LoopInToOut
{
public:
 LoopInToOut(hardwareAbstraction::Hal_IF& hal): m_hal(hal){};
 ~LoopInToOut() = default;

 bool loopThrough(uint8_t inPin, uint8_t outPin);

private:
 hardwareAbstraction::Hal_IF& m_hal;
};

This is enabling the real magic of gMock: as the client code works with an interface, I can hand over the “real” implementation (of the hardware access) at chip runtime, but for the test I can hand over a “mock” implementation which I have control over at the time the test is running.

// loopInToOut.cpp

bool LoopInToOut::loopThrough(uint8_t inPin, uint8_t outPin)
{
 bool input = m_hal.digitalRead(inPin);

 m_hal.digitalWrite(outPin, input);
 return input;
}

loopThrough() again is really simple. It reads the input pin’s state and maps it to the output pin. It returns the input pin state to the caller.

Mock Hal

GoogleTest uses macros to define behavior. The mock class is also deriving from the interface and just provides declarations of the interface method it overrides. There’s no behavior involved just yet.

// hal_mock.h

class Hal_mock : public hardwareAbstraction::Hal_IF
{
public:
 MOCK_METHOD(bool, digitalRead, (uint8_t pinId), (const, override));
 MOCK_METHOD(void, digitalWrite, (uint8_t pinId, bool value), (const, override));
};

The test fixture

The test fixture contains common data and behavior for all test cases within a test suite. Test suite name = class name, and this exact name has to be reused in the tests that should belong to this suite.

// unittest_loopInToOut.cpp
class loopInToOut_Test : public ::testing::Test
{
protected:
 virtual void SetUp()
 {
 m_loopInToOut = new LoopInToOut(hal_mock);
 }

 virtual void TearDown()
 {
 delete m_loopInToOut;
 }

 NiceMock<Hal_mock> hal_mock{};
 LoopInToOut* m_loopInToOut{nullptr};
};

Thus, every test has some data prepared it can use: the class under test is set up as a pointer with a mocked implementation of its dependency.

The actual test

The test is defined by the TEST_F macro meaning “test with fixture” that expects test suite and test names. The ON_CALL macro is a command to gMock providing behavior to the mocked class how to behave when digitalRead(_) is called with any input (underscore = any): It shall return true by default.

// unittest_loopInToOut.cpp

TEST_F(loopInToOut_Test, loop_inputHigh_writesCorrectOutputHigh)
{
 ON_CALL(hal_mock, digitalRead(_)).WillByDefault(Return(true));

 EXPECT_CALL(hal_mock, digitalWrite(2, true));

 m_loopInToOut->loopThrough(1, 2);
}

Then there’s an expectation expressed on the mocked class, stating that digitalWrite is called with input values of (2, true) which are also checked for correctness.

The last statement actually calls the implementation so that the expecations can be verified.

Hardware behavior implementation

As main.cpp accesses the hardware, there has to be some kind of interface implementation for this.

// hal.h
class Hal : public Hal_IF
{
public:
 Hal() = default;
 ~Hal() = default;

 bool digitalRead(uint8_t pinId) const override;
 void digitalWrite(uint8_t pinId, bool value) const override;
};

The header states that Hal derives from Hal_IF and overrides both of its methods.

// hal.cpp
#include "Arduino.h"

#include "hal.h"

namespace hardwareAbstraction{

bool Hal::digitalRead(uint8_t pinId) const
{
 return digitalRead(pinId);
}

void Hal::digitalWrite(uint8_t pinId, bool value) const
{
 digitalWrite(pinId, value);
}

Note that only the source file actually includes the Arduino.h to access the hardware. So the dependency to hardware functions is very limited and buried in interface implementations that can be selected by the program’s structure. While main.cpp will want to create an object of the above implementation, the test will want to instead hand over a mock.

Gtest key concepts

Google’s official github page on GoogleTest will help you a lot, not only for getting started³. The gmock Cookbook is an excellent guide that I have been using over the months as my tests became more complex hard to write, and the gmock Cheat Sheet will help you when you lost track of how to write a test for this and that special problem.

Dependency Injection

Dependency Injection is the basic mechanics you need to understand to make gMock work. Let me draw an image for you to make it more apparent:

The important concept is, that the interface is merely a placeholder for the implementation of the client classes’ dependency. At runtime, e.g. when the host creates the client object, it will handover (i.e. inject) the implementation of the dependency into the client which then can call any of its methods, as their signatures are known at compile time due to the interface that the client is using.

The mock now merely is an implementation of the interface specialized for testing.
The production implementation is used in all other contexts, it features the “real” behavior of that class.

Hardware Abstraction Layer

Yep, well, this might be one of the tricky bits when working with embedded software. At some point, it boils down to outside world interaction, be it with General Purpose Input Output (GPIO) pins, accessing one of the myriad of hardware bus systems like I2C, UART, SPI, CAN to control external hardware, or simply to use system internals like the EEPROM - that’s where the trouble begins.

Gtest runs on GCC/G++ compiler which just doesn’t know anything what the specialized compiler of that particular chip knows. It wouldn’t even understand the header files. Now, there are two options around this issue:

Write fake headers for each µC-specific thing that can be used by GCC/G++
Cut! At! The! Interface! to any µC-specific thing using an (Hardware) Abstraction Layer so your production environment uses the chip hardware as it normally does, but the test environment uses mocks instead.

There are quite some advantages for the second approach:

It’s clean. It will force you to not cross abstraction layers which should not be crossed.
It’s easy. You can test your business logic without even programming your µC! Just mock the inputs and intercept the outputs, write some tests and run them in the terminal.
It’s portable. Exchange the implementation of your abstraction layer, and you’re almost ready to execute on a different controller family or even on another manufacturer’s chips.

Drawbacks

Hmm, there must be some shadow somewhere. Right you are. In my opinion, these are the main disadvantages when getting everything ready to be tested without hardware attached:

Memory. Both ROM and RAM will be cluttered more easily when you have a massive amount of classes and vtables flying around.
Ownership. Injecting dependencies separates ownership from useage of an object, and you need to make up your mind how to organize both.
3rd party libaries. You’ll have to create an interface header, an adapter implementation, and a mock for each library that uses “chip-internals”.
Complexity. As your objects are injected via an interface, you’ll have to use call-by-reference or hand over by pointer 😨 now. The former is additionally limited because it will only work for constructor injection, not for any other kind (method injection, setter injection, etc.)

Some more tips

Keep everything “behind” your abstraction layer as simple as possible: You can only test and debug this stuff on target. Example: each method only executes one command (as in my example code).
Create a loader class or a Dependency Injection Container (Nicolas Croad) that holds all objects to be injected and can spit them out on request.
Make up your mind about the drawbacks mentioned above and how you create your low-level Architecture to accomodate for this.
Don’t hesitate to create more than one level of interfaces. It might be a good idea to use this pattern not only for the direct hardware interaction, but also in between software modules, which then can be tested in a higher granularity so your overall progress might even be quicker versus one big ball of classes you can just control via mocks in the Hardware Abstraction Layer.

Tough Stuff!

You earn a break now. This was much information to deal with. Take your time. Contact me on this topic’s discussions page if you feel something is missing, incorrect, to be improved, or just exactly what you needed right now.

Schallbert

Installation under Linux is actually more easy, as gcc and g++ come already pre-installed. So even if they do not feature the correct packages yet to support GoogleTest, their error message will be of much better help than the ones you get on Windows when some libraries are missing… 🙃 ↩︎
This is also called Constructor Injection, a subtype of Dependency Injection which is a means to achieve Inversion Of Control. I have prepared some links for you to continue reading, it is a rich topic that whole books have been written about.` ↩︎
By The way: the googletest site is also static like mine and it uses Jekyll as well 👋 …do you like it? I’d like to see your opinion on my discussions page ↩︎

This site Demo

Schallbert — Mon, 01 Jan 0001

Project stats

Difficulty: medium 3/5
Cost: 0€
Time: ~15h

The idea

The idea originated through my MobFobAmp project of which I didn’t have a github repository as there was no code or similar documentation to keep. So I decided I wanted a website so I could host instructions for this Open Hardware project for others to build their own copy. I wanted an easy-to-host solution that wouldn’t take too much work to set up. As I don’t have an own server, I asked for help. moritzmar recommended to use github pages as a site host to start with for easy auto-deployment and global scope. So I gave it a try.

Getting started

I figured out that a static site would be the right thing to use, without the need for a database or huge Content Management frameworks as my site layout should be simple and minimalistic; I just wanted to host this one project in the beginning. Github recommended to use Jekyll

Jekyll Logo ^[1]

as a site generator. I had some trouble with the Windows Subsystem for Linux (WSL) as it wouldn’t accept my terminal inputs at some point, so I followed the guide and used the RubyInstaller which worked right out of the box for me.

Site setup

I realized that I might want to document more of my projects, and not only the “hardware” ones. That’s why I came up with the idea of grouping projects together, each with a header, a little teaser text, and a thumbnail image. Clicking on which would lead to the individual project page. Which would involve a lot of programming for page setup and layouting which I didn’t want to spend too much time on - my other projects were waiting…

First try with Themes

So I followed another suggestion by Github pages: The usage of pre-defined “Themes”. I first used the Agency Theme. This one has a great landing page with a huge picture and it supports collections that enable the group-the-project thing that I wanted. Unfortunately, the theme is a “remote theme” so customization was limited, e.g. I didn’t manage to have a colletion point to another page, it would always open a “popup” and I just didn’t want this. (Later, I figured out that remote layouts can be overridden, but that was too late so I already switched to another theme supplier…) The theme I’m currently using the Minimal Mistakes Theme (see site footer) which is very well documented. After downloading and a couple of first steps, I struggled with custom colors (_sass) and the collections - again - as I didn’t understand at first how Jenkill would do the bindings behind the scenes. Some of my struggles can be found in the Blog posts from June 2021.

Site layouts: hero screens

A bit of thinking should go into how the site layout should be like, to make navigation easy and intuitive. The approach will differ with every page type due to differences in complexity and document structure.

Take the landing / home page for example, I want the navigation to be sticky so that the site’s categories are always visible. It shall receive a huge, wide overlay image with the page description. Below, latest posts will be listed. This is the example for the collection’s page hero screen:

The projects and their contents shall contain a page navigation in the left sidebar, may contain a scrollable header image, and has a wide setting to display a high amount of text in the content area.

The “hero screen” scetches for this site’s layouts can be found here.

Collections

Collections are a great way for grouping content. Look at my Open Hardware Projects page. You can find all projects in this category with a little teaser image and description here to get an overview.

This is how the collection’s configuration can work out:

Think of a fitting name for your collection, e.g. electronics.
Collection name is defined in _config.yml as collections: electronics.
Collection name has to be set in #defaults as type: electronics so that Jekyll knows how to interpret the actual content.
Collection page has to be created (e.g. in the _pages folder) with the attribute collection: electronics. This is the page that will actually display the overview of items in the collection.
Content folder with the same name as the collection’s name with leading underscore has to be created: _electronics.
All documents in this content folder will show up in the collection page automatically after a full build.
These documents should get a “front matter” (section before the actual content in the file, marked with ---) so that they can be displayed as described in the corresponding _layouts file, e.g. collection

More than the standard Theme

I have overridden some theme default and I’m using most of the existing Theme infrastructure. But my site has a couple of special needs so I had to customize and add Theme features. These are described in the following sections.

As my site navigation tree is pretty flat and I don’t have lots of pages to navigate to, I wanted the left sidebar to hold the page navigation rather than the site navigation. Thus, I wouldn’t need a right sidebar at all, so that there would be more space for the page content.

Table Of Contents on the left

As written here and there, I never was 100% happy with the way my project pages looked like:

The original idea of the theme is that the content is framed by two sidebars: The left one can either display Author information, custom content, or the site navigation. The content area is configurable between wide and normal, in the latter case leaving space for a second sidebar on the right. The right sidebar is configurable to not be there at all or to show the page’s table of contents.

So the target I set was to eliminate the right sidebar for my design, and instead of having the site navigation in the left sidebar, move the page navigation there.

It’s all about the /_layouts folder that keeps information about how a page layout should look like. The page design in question is called single.html and it at some point calls include sidebar.html which resides in /_includes/sidebar.html. So I thought of a new value I could use to control showing the table of contents within the left sidebar, chose toc_left and added a condition:

<div class="sidebar sticky">
<!-- ...
{.% if page.toc_left %}
 {.% include toc_left nav=page.toc_left %}
{.% endif %}
... -->
</div>

(By the way: I had to comment-out and add .’s to the jinja code, Liquid would otherwise interpret these as commands!)

Implementing the Table Of Contents

I wanted it to show up in the left sidebar container. So I just had to create a toc_left file that would call the included toc generator like this:

<aside class="toc_left">
<nav class="toc" markdown="1">
<header><h4 class="nav__title"><i class="fas fa-{{ include.icon | default: 'file-alt' }}"></i>
<!-- ...
{.{ include.title | .default: site.data.ui-text[site.locale].toc_label }}</h4></header>
{.% include toc.html sanitize=true html=content h_min=1 h_max=6 class="toc__menu" skip_no_ids=true %} 
... -->
</nav>
</aside>

Audio Embeds

Audio embeds are these little player-like things on a website, often forwarding to a streaming platform or similar. I need them for some of my audio projects like this mobfobamp: just a caption and the player below.

This post helped me a lot, so I could just add it to the _include, modify it a bit, and it worked!

Placing the favicon

I followed this guide to place a favicon for the site and it worked flawlessly.

The favicon’s path has to be valid independently of the folder the rendered page resides in. I had to add ../ according to the pages’ depth within the site’s folder structure to make sure that the favicon could be displayed not only on the “home” page, but also from sub-paSges.

Optimizing for search engines

As my site was not found by search engines for a couple of weeks, I decided to perform so-called “SEO” (search engine optimization). That’s why I added jekyll-seo-tag and e.g. reworked all my links following Juliette Sinibardy’s SEO 101. When complete, I put my webpage through Lighthouse’s web-analysis audit.

For a first try, I think it looks pretty well. Biggest issue on the performance side seems to be CSS overhead.

Content Security Policy

The Lighthouse audit revealed that it would be a good idea to restrict external scripting to my site, so I introduced a Content Security Policy (CSP) like this:

webrick:
 headers:
 Content-Security-Policy:
 default-src 'self'
 script-src 'self';
 connect-src 'self';
 img-src 'self';
 style-src 'self';
 font-src 'self';

Initially, unfortunately, my home page’s backup image wouldn’t show, my embed-audio files wouldn’t play and all icons loaded externally were gone as well. Adding media-src: 'self'; and some more reading in Jakob Wilforss’ blog and especially Lighthouse’s recommendation helped to solve this. By using the Browser’s Web developer Tools, I figured out the missing resources the theme is loading in the background, so my updated CSP looks like this:

webrick:
 headers:
 Content-Security-Policy:
 default-src 'self' 'unsafe-inline' https://cdn.jsdelivr.net/; 
 script-src 'self';
 connect-src 'self';
 img-src 'self' 'unsafe-inline' https://upload.wikimedia.org/wikipedia/commons/ https://raw.githubusercontent.com;
 style-src 'self' 'unsafe-inline' https://cdn.jsdelivr.net/;
 font-src 'self' 'unsafe-inline' https://cdn.jsdelivr.net/; 
 media-src 'self'; 
 object-src 'none';
 base-uri 'none';

Errors

I’m collecting some simple errors here that cost me quite some time fixing.

Jekyll serve: `UTF-8` incompatible character encoding

You enter bundle exec jekyll serve and get a strange UTF-8 “incompatible character encoding” error? The cause is simple: One of your files most likely has no header that can be converted into the front matter of the page that is generated from that file. Just add your usual header to your file like so and it will just work:

---
title: "Feeds and Speeds database"
description: "for (light) CNC milling machines"
header:
 image: "blah/testimage.jpg"
[...]
---

Webpage is not being displayed at all: `404`

This error was very, very hidden. I found it by accident browsing my site online only, as localhost serve was not affected. The symptom is a no-show blank page.

Cause: Due to translation, my paths are getting longer and I wanted subpages to not have extremely long links. That’s why I was using the permalink: Attribute of the pages’ fontmatter. Now, only if you have a “trailing slash” here, the page will display on the remote server.

# Wrong:
---
title: "This Page won't be displayed"
permalink: /projects/pageerror
# [...]
---

# Correct:
---
title: "This Page will be displayed!"
permalink: /projects/nopageerror/
# [...]
# Unfolds on the english blog to:
# /en/projects/nopageerror/
---

Site hosting

This site itself is a normal Gitea repository. It is built automatically with Gitea Actions once changes are added to the repository’s main branch.

Summary

To get familiar with how Jekyll creates the site, how the theme would interact with my commands and how to override Theme behavior cost quite a couple of hours, but I think this site has a clear structure and a more or less “professional” look - And I didn’t have to write a single line of JavaScript or HTML to get this done. Nice!

Contents written in markdown *.md
Basic site configuration in YAML _config.yml
Page layouts written in html and jinja *.html, placed in _layouts and _includes
Page variables and appearance in SASS *.scss, automatically generating css files

Contribute?

You have ideas, criticism, or interesting stuff you’d like to let me know? Get started on my Discussions page!

CI/CD on Schallbert's Blog

act_runner rootless: no start

The error

Issue ticket on Gitea

Problem with kernel permissions?

Why containers are not allowed to start other containers

Consequences if something goes wrong despite DinD

Simple solution: Start the DinD container as privileged

Dead end: Only grant the rights that are absolutely necessary

Future solution: Virtualisation or daemonless

Linux Security Modules (seccomp, AppArmor, SELinux)

AppArmor

Simple solution: Disable completely

The correct solution: Tailor settings for each application

Jekyll-dockerimage: Bundler and Gemfile

What is bundler?

bundler in the CI pipeline

No write permissions

Specification file for bundler

Troubleshooting

Optimize Gemfile.lock for different environments

Gitea act_runner: Jump-start issues

Problem statement

Interim corrective action

Root cause analysis

Solution: Reflecting dependencies

depends_on

Healthcheck

Sending error logs

What is this about?

Verification and monitoring

What characterizes good monitoring for me?

Methods of automatic reporting

Mail via console - curl

Mail via console - mail, mailx, mailutils, swaks

Create RSS feed

borgmatic

fail2ban

Gitea

Server

Selecting the reporting program

Communication channel

Logging

When should logs be sent?

What should I include in a report?

Rough version of a report

Implementation

borgmatic

Server

Server configuration

Gitea

Result

Server configuration rollout

What is this about?

Ways and possibilities

Automation services Ansible, cdist, Kubernetes

My approach

Risk of circular reference

Preparation: Create folder system and scripts

Implementation: Shell script to handle the update

First test and hooking into “autostart”

Backup

Freeze state and data

Borgmatic: Trigger handler mechanism #2 and #3

Creating the backup

Second test to create the backup

Troubleshooting for “failed to acquire the lock”

Transferring the configuration

Switching to rsync

Restart all applications

The finished automation

Server configuration with Git

What is it about?

Background

Preliminary considerations

1. Hardlink

2. Repo and auto-deploy to the server

Problems

Selecting my solution

Let’s get to work

Simple solution: Start the DinD container as `privileged`

Optimize `Gemfile.lock` for different environments

Secrets in `docker-compose.yml`

Secrets in Gitea’s `app.ini`

act_runner: no write permissions to `Gemfile.lock`

Dependency: `sass-embedded`

Docker volumes for `act_runner` cannot share build artifact

But I can use `upload-artifact`, right?

Also Docker `cp` won’t work here

Complete Gitea config via `app.ini`