Code on Schallbert's Blog

Auto-Upgrade Docker Images

Schallbert — Sun, 01 Nov 2026

Why automatic updates?

Software ages. Or rather, time passes it by, meaning it requires maintenance and must be kept up to date.

“Code is a liability (not an asset).” - Cory Doctorow, Jan 2026

On my blog, I have frequently reported on issues such as critical vulnerabilities in Gitea or in PHP, resolved system crashes, and recovered unintentionally published pages. All of which could be addressed through updates or reconfiguration.

My server’s operating system and the key programs running on it are already set up for automatic updates, as described in my article on server auto-upgrades.

Nevertheless, I am not quite finished with this topic.

Updating Docker images with Watchtower

Until now, I have used the program Watchtower to manage my Docker images. But there are several reasons to move on:

It is no longer actively maintained and has been archived. However, there are forks that are still being maintained.
To function, it requires access to the Docker daemon via docker.sock. This is like granting the container root access to my system which I want to avoid in the future.

Looking for a direct replacement for Watchtower? Linuxhandbook lists a few solutions here. A common drawback among them, is that they either only notify you about available updates without deploying them, or they also require docker.sock to be mounted as a volume.

A Script-Based Solution

In search of an approach that avoids the security risk posed by docker.sock, I quickly arrived at the idea of running a script daily via cron. Since this script executes with root privileges, it can run Docker commands directly.

Complicated: `docker pull`, comparing against existing images, and deploying if necessary

My first script is 164 lines of code (164loc) and performs the following steps:

Verifies that the required tools curl, docker, docker compose, and the docker-compose.yml file-are installed and functioning correctly.
Identifies the current image digests for the services defined in the docker-compose.yml file.
Updates the registry for each service using docker compose pull.
Compares the “new” image digests with the “old” ones using docker image inspect.
Launches the new images via docker compose up and saves the logs.
Sends a notification to me via ntfy if any updates have taken place.

This approach proved too complicated. The tool I use, docker compose, includes the necessary option out of the box already: docker compose up --pull always.

Simple: Include everything in a single command via an option

The core of my new script (61loc) looks like this:

compose=(docker compose --progress plain -f "$COMPOSE_FILE")

before=$("${compose[@]}" ps --all --quiet | sort)

if output=$("${compose[@]}" up -d --pull always --quiet-pull --wait \
 --wait-timeout "$WAIT_TIMEOUT" 2>&1); then
 status=0
else
 status=$?
fi

printf '[%s] docker compose up (exit %s)\n%s\n\n' \
 "$(date -Is)" "$status" "$output" >> "$CHANGELOG"

if (( status != 0 )); then
 report_failure "WARNING: Docker update failed" "warning" "$output"
 exit "$status"
fi

after=$("${compose[@]}" ps --all --quiet | sort)
if [[ "$before" != "$after" ]]; then
 notify "INFO: Docker containers updated" "white_check_mark" "$output"
fi

In the script, I capture the state prior to the update by listing all services specified in the Compose file and storing them in a variable. I then execute docker compose up -d as usual, but include the flags --pull always, --quiet-pull and --wait --wait-timeout <time[s]>.

The pull option prompts Docker to check the registry for new images, which are then downloaded and deployed if available in non-verbose fashion (no progress for individual download packages, etc). The timeout sets a specific duration for the command to complete before it terminates with an error, ensuring the script cannot hang indefinitely.

The final steps involve documentation and monitoring: the command’s output is recorded in a changelog, and in the event of a failure, I receive a notification implemented via Ntfy, as described in my post on server logging. This is where the initial state (captured before the update) comes into play: I only receive a notification if an error occurs or an update is performed.

Finally, I make the script executable per chmod +x docker-image-update.sh and assign ownership to root with chown root:root docker-image-update.sh.

Cron job entry

After manually testing the script (./docker-image-update.sh) and verifying its output, the changelog, and the new container versions, I configure the cron daemon to execute the script daily:

schallbert@machine: crontab -e
[...]
0 2 * * * /path/to/docker-image-update.sh

Which version was installed?

Instead of using service version labels, Docker relies on image digests. This sha256-hash is unique to each image, but it is not human-readable:

sha256:94a00394bc5a8ef503fb59db0a7d0ae9e1110866e8aee8ba40cd864cea69ea1a # example Image digest

Service version labels such as Gitea v28.2.1 are not supported by default in image metadata. Consequently, I do not currently have a solution for displaying a version number. I chose to omit the digests from the output altogether.

Notification upon update

The notification could be improved: ideally, I only need the names of the containers that were updated and confirmation that all images are “healthy” afterward. At present, I receive the full status of the containers, from which I would first need to extract the relevant information. Given that the added value of filtering this data is minimal, it is not worth the time required to expand the script. After all, the Recreate process already indicates which container has been updated.

I am satisfied. By using just a few lines of code, relying on built-in tools and only cron as an additional dependency, I have successfully solved the problem of updating containers managed via Docker Compose from the host machine.

PHP2shell: Critical vulnerability

Schallbert — Wed, 21 Oct 2026

Step 1: Recognition: We have a problem.

TL;DR: PHP currently has two critical vulnerabilities that allow for Remote Code Execution. Their identifiers are CVE-2026-109618 and CVE-2026-109619. Since the root cause lies within the engine itself, rather than the frontend, the only solution is to update to PHP 8.5.12, 8.4.27, 8.3.36, or 8.2.35.

“I don’t use PHP on my website,” I think to myself upon receiving the following message this morning:

“Hi, quick heads-up: You might want to […] disconnect anything that uses PHP for now; […] a patch is coming later today.”

Or am I affected after all…? I do use a managed Nextcloud instance, and isn’t Nextcloud written in PHP, among other things…? As I write these lines, a fix for Nextcloud is becoming available. Everything has to happen very quickly. There will certainly be a lot of overtime worked today.

Status: October 21, 2026, 10:30 AM CEST

Let’s rewind ten hours. There is no fix yet. Information about the CVEs has leaked and is already a few hours old. According to Wikipedia, 69% of all websites use PHP. Exploiting such a highly critical vulnerability (with a score of 9.2) in such a widely used technology naturally has a huge blast radius. Consequently, “the internet” is quite nervous.

“[…] organisations should consider isolating internet-facing applications from public networks until a fix is applied and verified.” — CERT-EU, accessed October 21, 2026, 10:40 AM

Interesting. The recommendation is to “take it offline for now.”

Step 2: Immediate Measures: Taking Nextcloud Offline

To get ahead of a potential attack, I attempt to take my instance offline. I log in to my hoster’s platform, navigate to the “Settings” tab, and click on the ownCloud Console section. This is Nextcloud’s command-line interface.

Once there, I discover that there is no command available to shut down or “de-publish” the instance. So, I opt for the next best thing:

occ maintenance:mode --on

So, I put my instance into maintenance mode. This prevents new logins and freezes the database. Afterward, I read up a bit on the CVEs to see if this step is sufficient.

Apparently, it is not.

Through various forums and HackerNews, I learn that an unauthenticated request can trigger the attack before any PHP code is even executed on the server side.

In other words, it doesn’t matter what state I set my instance to, as long as it is connected to the network, it remains vulnerable.

Step 3: Interim Containment: Having the Nextcloud Instance taken offline

Even after several online searches, I cannot find a way to shut down my managed instance myself. I briefly consider killing the name resolution and DNS routing, but I quickly realize that by the time those changes propagate, the fix will likely already be available.

Status: October 21, 2026, 11:43 AM CEST

So, I submit a support ticket to my hoster, requesting assistance in taking my instance offline.

Status: October 21, 2026, 12:01 PM CEST

I receive an email from my hoster!

Sure enough, I can no longer access the instance. I’m met with an ERR_CERT_COMMON_NAME_INVALID error. Dismissing that, a message appears: REQUESTED PAGE LOCKED. Well done. The provider’s response time, including notifying the customer, was just 18 minutes.

Now, I shift from participant to observer. I’m seeing initial exploit attempts (or at least claims of them) circulating online, which brings to mind a slide from a presentation I recently watched: Security in the LLM age (at the 11:24 mark).

In it, a Linux kernel expert points out that, as of 2026, exploits are appearing in the wild an average of seven days before a patch for the affected software is even released.

discover --> disclose --> patch --> deploy

This traditional sequence of events only works well against “slow” attackers. We need something that allows us to get back ahead of the situation.

Step 4: Monitoring the Situation - Patch Availability and Initial Deployment

According to The Hacker News, the patch is set to become available around 4:00 PM CEST. I am obtaining the RSS feed from my hosting provider and awaiting further updates.

Status: October 21, 2026, 14:20 CEST

The Federal Office for Information Security (BSI) has also issued an IT security advisory, which classifies the vulnerability as having a “criticality level of 3/Orange” and notes the following:

“Der Patch für die Schwachstelle ist heute 16 Uhr deutscher Zeit (14:00 UTC) angekündigt und sollte unverzüglich zu diesem Zeitpunkt installiert werden.” - BSI, 21.10.2026

Status 21.10.2026 16:34 CEST

My hoster reports that patches are now available and that implementation has begun. Affected systems include not only the Nextcloud instances but also the hoster’s entire administration interface and their webmail service.

Status 21.10.2026 19:52 CEST

My hoster reports that the patches have been successfully applied to their web hosting services and managed servers.

Status 21.10.2026 22:40 CEST

Hetzner’s KonsoleH application, which I use to manage my Nextcloud instance, is down.

Status 21.10.2026 23:00 CEST

KonsoleH is back online (and patched).

Step 5 Insights and Conclusions

What have I learned from the events of this turbulent day? How should I handle security vulnerabilities in the future? And how can I even tell when a problem exists?

It is not just the service as a whole that is vulnerable, but its individual components as well. –> Know all your components and keep an eye out for vulnerabilities and patches!
Adversaries have become incredibly fast (thanks to LLMs). –> Expect exploits to appear even before patches are available!

I am currently compiling a list of reliable sources for information on vulnerabilities, news, and patches. Once the list is finalized, I will share it.

Step 6 Actions

Remove sensitive data from the network whenever possible. Line of thought: What if this were to disappear? Would availability within the LAN/intranet suffice?
Improved monitoring and abuse detection are the logical next steps: detect early, block early, report early, shut down early, and patch immediately.
Identify standard sources for information regarding the software in use. Perhaps build a CVE list scanner? Or a Hacker News filter?

Addendum: Oct 22, 2026, 09:30 CEST

The hoster has patched my Nextcloud instance and brought it back online. All of my systems are back to normal. There is now also a Wikipedia entry on the subject, which is kept very up to date.

Step 7: Initial Reflection

A few days have passed since the two security vulnerabilities were publicly disclosed and patches were made available. Let’s take a look at the current landscape across the web.

Status: October 24, 2026, 16:30 CEST

There are now many, many proofs of concept (PoCs) demonstrating how to exploit these PHP vulnerabilities. Most notably on the platform GitHub.

Personally, I am not an expert in this field, so I have to rely on the assessments of others regarding the quality of these PoCs, the actual feasibility of exploiting them “in the wild,” and the reasons why this particular vulnerability accumulated such significant media attention.

In general, the rule is, of course: “Patch first, ask questions later.” That is exactly what I did. To quote Felix von Leitner once again:

“One of the lessons learned from 20 years of working in the security field is that when one person claims a bug exists, and another claims it isn’t exploitable, it almost always turns out, in hindsight, that it was exploitable after all.” - Feliks von Leitner, alias Fefe, April 12, 2014, accessed October 24, 2026

Regarding exploitability, flare.io which describes itself as a threat intelligence site, notes that most proof-of-concept (PoC) exploits for PHP2shell only functioned within laboratory environments. They are rarely applicable to real-world targets. Either because they require internal operational knowledge of the systems being attacked, because they depend on other vulnerabilities (such as the disabling of memory randomization), or because the PoCs themselves are merely “AI-slop”: poorly validated, hastily written software that promises more than it delivers.

eliastaylor expresses a similar view:

“There isn’t an easy exploit for this PHP vulnerability.” - https://eliastaylor.com/noexploit/, accessed October 24, 2026

Writing less than ten hours after the attack vector and patch were released, he argues that:

none of the proof-of-concept (PoC) exploits actually work. Not even on a standard installation and configuration of a pre-patch version of PHP
these two CVEs are far less critical than other historical security vulnerabilities (citing Heartbleed in OpenSSL as an example)
the exploits were not crafted by actual humans, but rather generated with the help of LLMs.

The core of the argument is that people are delegating their cognitive work to language models, failing to verify whether the system is actually doing what it claims to do. Let alone reading, understanding, or improving the code the system generates. Malicious botnets are still crafted by human hands, a process that takes time, especially when the vulnerabilities are not obvious or trivial to exploit.

As a brief exercise to help form an own assessment of the situation, I recommend scanning the “issue” discussions found on the GitHub repositories for many of these “PHP2Shell” PoCs.

However, the significant media attention has some advantages: it seems the world is patching quickly. I’ve taken a look at the discourse across various German and international forums: everywhere, people are pushing their hosting providers to apply patches, discussing the potential impact, and tracking which vendors have already released updates.

That still leaves the question of who actually benefits from all this. I have a few theories. However, it is still too early to draw conclusions or form a judgment.

Status: October 27, 2026, 17:00 CEST

The vulnerability has been officially confirmed as being exploited.

Step 8: Analyzing observations and adjusting information channels

I am now focusing on reports regarding the exploitation of these vulnerabilities. The sheer volume of Proofs of Concept has become unmanageable for any single individual. Some private individuals and companies like Cloudflare have begun organizing “Capture The Flag” (CTF) competitions: they place intentionally vulnerable PHP instances online and challenge participants to compromise them and provide proof.

This is a clever way to separate the sheep from the goats. It forces individuals to test their PoCs against sparring partners who do not disclose their underlying stack. As an added layer of difficulty, some organizers require that standard defense mechanisms, such as Web Application Firewalls (WAFs), be bypassed as well. If a PoC succeeds under these conditions, it can be considered a valid exploit demonstrated in a realistic environment.

Understanding the PoC

Thanks to eliastaylor, I have located the PoC that successfully solved the “StealMyBitcoin” PHP2Shell CTF. Both the comments within the Python file regarding the exploit mechanism and the discussion on GitHub—covering the specific LLM variant, prompt, and configuration used, are quite interesting.

As I interpret the sources and commentary surrounding this PoC, it offers an elegant solution to the shortcomings of previous attempts:

Effectively forcing PHP into a memory-constrained state during a multipart upload: Instead of sending thousands of requests in rapid succession, hoping to hit the “right” worker thread, this attack employs “deeply nested field names”¹. This keeps the payload small while causing server-side memory consumption to explode, all within a single request.

This targets the vulnerability CVE-2026-109618, where an “out-of-memory” condition can interrupt PHP while internal request data is only partially initialized; the subsequent request cleanup process may then mistakenly treat this invalid data as valid. The purpose of the heap_consumer is to manipulate the residual data left behind following the memory overflow.

Bypassing ASLR (Address Space Layout Randomization): Rather than ignoring ASLR in a real-world scenario (or simply disabling it in a lab environment), the memory overflow, data manipulation, and the subsequent request to trigger the processing of that manipulated data must all occur within the same session and on the same, exhausted worker process². Otherwise, the affected memory region becomes nearly impossible to locate and will eventually be reset by the system once it detects that the worker has stalled. Consequently, the exploit must ensure that the malicious code is already in place by the time PHP invokes the corresponding destructor, which ultimately triggers the RCE (Remote Code Execution).

This relates to the second CVE in the set: CVE-2026-109619 states “request-owned internal state can survive after its backing memory has been freed. A later request handled by the same worker may use that stale state before normal initialization resets it.”

It would be great to hear from a PHP engine expert on this! :)

This effectively places an tool in the hands of malicious actors, making (automated) exploitation feasible against “easy targets” that lack proper hardening.

Hardening

The only way to resolve this issue is by applying the patch. The following hardening measures can at least increase resistance against this specific attack vector:

Either avoid offering the PHP_SESSION_UPLOAD_PROGRESS command which is used to carry out the attack. Or restrict requests so that they cannot occur in rapid succession, within the same session, or from the same machine.
Increase the number of workers to the maximum, even if this impacts the machine’s performance. This makes it more difficult for an attacker to re-capture and exploit a worker that has already been compromised.

Later that same day

The Hacker News confirms the exploitation and lists affected companies, major corporations, and suppliers. The bottom line: patch, patch, patch!

Status as of October 28, 2026, 21:30 CEST

Cloudflare has since released new firewall rules for its WAF designed to intercept attackers before they reach the target servers. The Hacker News reports on specific incidents involving initial damage. Threat intelligence firms have confirmed these findings.

To round out this update, here is a useful collection page containing information about “x-2-shell” vulnerabilities.

Use Password manager with hardware security keys

Schallbert — Sun, 18 Oct 2026

The Password Manager

The web is full of various password management solutions. This is understandable, given that we have an ever-increasing number of accounts, and no one wants to memorize passwords in the hundreds. So, let’s get straight to the point:

From the vast array of options, I evaluated three password managers based on my criteria (secure, user-friendly, open-source, self-managed) to make a final choice:

Firefox password manager (Online/Offline)
Vaultwarden (Online)
KeePassXC/KeePassDX (Offline)

Not the Firefox Password Manager

I ruled out Firefox because I want to use my hardware security key to unlock it rather than memorizing a long master password. On the plus side, however, it offers great ease of use, automatic form filling, and synchronization capabilities, albeit via third-party infrastructure.

Not Vaultwarden

I had been focusing on Vaultwarden for a long time. Running it on a local network requires meeting numerous prerequisites and setting up supporting software. Specifically things like HTTPS on the LAN and ACME challenges. Consequently, you need a dedicated small computer on your LAN, and ideally, a virtual private server to handle HTTPS traffic termination. Vaultwarden even supports incredibly convenient passkey-based login and can synchronize the password database across all devices on the network.

Ultimately, however, I avoided the complex setup, its additional dependencies, and software packages, all of which I would have had to maintain.

KeePassXC

I experimented with various versions of KeePass and eventually settled on KeePassXC. The overall package simply feels more polished and modern; I can use the software across different operating systems, and the extensions needed for hardware security keys (HSKs) are included right out of the box.

Setting up KeePassXC with an HSK

KeePass uses the keyed-hash message authentication code (HMAC) method to unlock the password database using hardware security keys. It relies on a shared secret key that must be present on both sides but is not exchanged between the parties.

Typically, you generate the secret key on the hardware security key and then copy it into the KeePassXC input field. But let’s take it step by step, as the process depends slightly on the specific HSK being used.

YubiKey

Yubico has prepared a guide specifically for use with KeePass. A YubiKey has two configurable slots that support various cryptographic functions. From the function selection, choose Challenge-response: program a challenge-response credential. The YubiKey then generates a 20-byte HEX code, which can subsequently be copied into KeePassXC.

It is important to note that the displayed code cannot be extracted later. Therefore, if the database is to have a secondary key (which is recommended), the code must be written down and then also stored in the secondary key.

For the YubiKey, I recommend leaving the requires touch option unchecked. Otherwise, you have to touch the YubiKey every time you use a password. Even more annoying is that touching the key also becomes necessary when adding or modifying a password entry.

Nitrokey

Nitrokey specifies that the secret be generated as a 20-byte base32 code. It can store exactly one HMAC secret. Nitrokey also provides an online guide for storing the HMAC secret for use with KeePassXC. As with the YubiKey, this guide relies on using the GUI app.

YubiKey Compatibility

Ideally, I would store the key generated on the YubiKey onto the Nitrokey, or vice versa. That way, I would have a backup key and could access the KeePassXC database using either one.

But behold, the Nitrokey does not accept the YubiKey’s secret:

The same thing happens when I try to store a secret generated on the Nitrokey onto the YubiKey:

I needed a workaround for this. So, taking a pragmatic approach, I created a second password database. Both were intended to hold the same content but use different keys to unlock. I later learned that this wasn’t actually necessary. Nevertheless, here are the provisional solutions I came up with initially.

Dead End A: Cloning entries from the database

I populate the first database with passwords, sorting them into groups. Then, I clone a group, open the second database, and drag and drop the clone over.

Drawbacks: I have to repeat this process for every group, as drag-and-drop functionality isn’t supported for the root group. Whenever I make a change in Database_yubikey, I also have to update Database_nitrokey.

This becomes tedious, especially while the database is still under construction.

Dead End B: Copying the entire database and updating the HMAC

To do this, you store the HMAC secrets for both hardware security keys (HSKs) within the password manager. When the database changes, you create a copy of the database file, open it, and copy the secret belonging to the other HSK. Then, you navigate to Database > Database Security > Change Challenge-Response and enter that secret. The database now works with the other key.

Drawback: Requires manual updating of both databases. The challenge-response setup has to be reconfigured every time.

Solution: Converting the HMAC secret

I wondered if I could use the command line instead of the app to store the HMAC secret for my Nitrokey. Command-line tools like nitropy usually offer more options than their GUI counterparts. Perhaps this would allow me to solve the compatibility issue. Here is the link to the nitropy installation guide.

While looking for suitable documentation, I came across this interesting post about challenge-response. It made me realize something that tech-savvy people, or at least those who are used to cryptography, or work close to hardware, have known for a long time:

Data with a size of 20 bytes can be represented as 40 characters in hex AND as 32 characters in base32.

Consequently, an HMAC secret can be converted and stored on both the YubiKey and the Nitrokey. To do this, I use the small utility xxd to generate a base32 phrase for my Nitrokey from the YubiKey’s hex code.

# open a new terminal
# make sure the secrets are not saved to history (command for zsh)
history -p 
# c0ffee = secret hex key
# xxd convert input to binary (-r), output as plain hexdump (-p)
# interpret as base32 string
echo -n 'c0ffee' | xxd -r -p | base32
YD764===
# close terminal

Alternatively, it also works the other way round:

# open a new terminal
# make sure the secrets are not saved to history (command for zsh)
history -p 
# YD764=== = secret base32 key
echo -n 'YD764===' | base32 -d | xxd -p
c0ffee
# close terminal

KeePassXC: Keeping Passwords Neatly Organized

If you avoid using “Log in with Google” options for third-party services, you can end up accumulating a large number of passwords over time. To make finding and using these passwords easier on both mobile and desktop, it makes sense to organize them into logical groups.

Password Families

After some experimentation, the following naming scheme has proven effective for titles when I need to store multiple related passwords:

baseapp.attribute.specialization

For example, here is how the login details for my hardened Gitea setup would look:

gitea.login.schallbert
gitea.login.testuser
gitea.runner.registration
gitea.recoverykey

Password groups

I consider the following groups of passwords to be sensible:

Group	Description
Devices	Computer, phone, router, keys
Finance	Banks, credit cards, insurances, payment
Identity	Email, identity providers, single sign-in
Shopping	Online stores, streaming, subscriptions
Games	Platforms, launchers, servers
Social	Forums, Messengers, Social networks
Backup	Recovery keys, licenses, passphrases

In doing so, I make sure not to create too many subgroups. Otherwise, it is easy to get confused, especially in the mobile view.

Icons 🔑

KeePassXC offers the wonderful feature of downloading the target website’s favicon and using it as an icon. This makes navigating the password manager even easier.

However, the favicons are all bitmaps. If you prefer to use high-quality vector graphics, you can specify the path to an icon pack under “Choose Icon.” I downloaded the Aegis icons for this purpose and am very happy with them.

Critical vulnerabilities in Gitea (and Forgejo)

Schallbert — Sun, 04 Oct 2026

Becoming Aware

As is so often the case, this story begins with a network of human communication. This invaluable exchange has already contributed significantly to the security of the software powering this blog. Now, I am able to close another security vulnerability. But first things first.

Gitea as a Target

I heard that the self-hosted version control solution Gitea used by an acquaintance of mine had become the target of an attack. This attack enabled RCE (Remote Code Execution), which was subsequently exploited.

Why is Gitea an attractive target?

Theft of (corporate) secrets: Access to repositories puts confidential source code at risk of exposure.
Exfiltration of access credentials: If a product with interfaces linked to Gitea is managed there, API keys, accounts, and passwords are at risk if stored unencrypted. Tools like git-secret offer a remedy, but they are not yet widely adopted due to the added complexity they introduce.
Classic privilege escalation: Under certain circumstances, breaking out of the Gitea software can allow an attacker to take over the entire machine.
Exploiting the runner: If an attacker manages to hijack the runner, they can use the host machine’s resources. Even without elevated privileges. And misuse them for their own purposes (e.g., cryptominers in CI/CD).

What is RCE?

Remote Code Execution (RCE) refers to a successful attack in which unauthorized code is injected and executed remotely over a network, without any action required by the owner or administrator of the affected resource.

The Gitea Ecosystem

As a version control system and forge, Gitea’s core functionality includes the ability to develop and build software collaboratively via a web interface. Consequently, the system is designed to allow users to register and, in some cases, execute Actions to automate software builds and/or deployments.

This very mechanism of automatically fetching and executing code can have serious consequences if security vulnerabilities exist within the management software itself.

Vulnerability via Open Registration

If an operator leaves the new account registration setting (“self-registration”) at its default value (DISABLE_REGISTRATION = false) in the Gitea configuration, users can sign up, create their own repositories, open issues, and perform any action that does not require administrative privileges.

If the default setting REQUIRE_SIGNIN_VIEW = false is also used, unauthenticated visitors can view public repositories, browse the Gitea site, and, most importantly, interact with the public Gitea API.

Vulnerability due to default configuration for reverse proxies

In the default configuration of the Gitea Docker image, Gitea versions prior to v1.26.3 trust all proxies.

# gitea/app.ini

[security]
REVERSE_PROXY_LIMIT = 1
REVERSE_PROXY_TRUSTED_PROXIES = * # This is a vulnerability!

The Security Vulnerabilities

What are we dealing with here? Several critical security vulnerabilities with a CVSS score of 9.8/10, for which known instances of exploitation exist.

Side Note: The CVSS Rating

To put the “critical” label into perspective: CVSS criteria are structured similarly to those used in classical hazard and risk analysis, a branch of engineering. Unlike in that field, however, the focus is not on analyzing potential failure modes, severity, probabilities of occurrence or failure, or detectability; instead, it centers on the difficulty of exploitation and the potential impact.

Significance of Individual CVSS Metrics

The higher the CVSS score, the easier it is to exploit the vulnerability and the more severe the potential consequences regarding data security criteria (Confidentiality, Integrity, Availability). The image above illustrates a scenario involving a potential remote (network-based) attack that requires minimal effort (low complexity regarding system hardening) and no specific preconditions (such as targeting a specific vulnerable code path, requiring a specific system state, triggering a race condition, etc.). Furthermore, the attack requires no elevated privileges (it can be performed as an unauthenticated/unauthorized guest) and involves no interaction with the system’s user or administrator.

In isolation, this might not be critical if the system is not required for operations (Score 0.0) and holds no valuable data. However, if sensitive data is present (Confidentiality, 8.7), or if there is a database whose values can be manipulated (Integrity, 8.7), or even if the availability of essential data is compromised (Availability, 8.7), the risk level shifts to “High.”

With this context in mind, we can now apply these concepts to the critical vulnerabilities found in Gitea.

CVE-2026-20896: Authentication bug in Gitea through reverse proxies

If an attacker gains access behind the reverse proxy (or if no reverse proxy is deployed in front of the system), they can send a request containing the X-WEBAUTH-USER header directly to the Gitea instance, effectively impersonating a reverse proxy. If Gitea is configured to trust all reverse proxies (REVERSE_PROXY_TRUSTED_PROXIES = *), it assumes that the user has already been authenticated and a valid session exists. The attacker thus gains user rights with write access to repositories, even though they haven’t even created an account.

The actual vulnerability here is the unquestioning trust placed in the header’s sender without any further validation against a session token or similar mechanism; simply accepting the header from external sources is what creates the vulnerability.

There is evidence of attacks that have already exploited this bug.

CVE-2026-59774: File-read flaw affecting any file readable by the Gitea service account

This critical vulnerability is caused by insufficient input validation in Gitea’s built-in text renderer for org-mode files. A detailed description of the attack vector and potential exploitation methods can be found on Satyam Rastogi’s excellent blog.

In short, an attacker with access to at least one public repository can craft a malicious .org file, e.g. to read Gitea’s app.ini configuration. This allows them to access secrets contained within the file, granting them elevated privileges or enabling lateral movement within the system.

According to The Hacker News and Gitea’s own team, this flaw can also be exploited to extract runner registration tokens and hijack the runners themselves.

CVE-2026-60004: RCE allowing an attacker with standard user privileges to execute arbitrary shell commands as the Gitea OS user.

This security vulnerability is particularly severe because it also enables privilege escalation.

“The vulnerability in question is CVE-2026-60004 (CVSS score: 9.8), a case of remote code execution that allows an attacker with ordinary write access to a repository to execute arbitrary shell commands as the Gitea OS user.” - thehackernews, Aug-2026

Exploiting this vulnerability can therefore allow an attacker to take over the entire Gitea instance, provided they have write access to any repository (even a public one).

The Situation

There are other critical security vulnerabilities; however, by combining the scenarios described above, attackers can (without significant effort)

Operate as normal users on the Gitea system without having registered and without requiring SSH access
Impersonate the Gitea service user
Take over Actions or even inject and execute workflows
With slightly more effort, take over the entire instance, encrypt it, demand a ransom, etc.

My acquaintance’s experience involved points one and three. The attack was most likely automated, carried out after the internet had been scanned for vulnerable instances.

In summary, the case at hand appears to be a mass-exploitation campaign aimed at cryptomining.

The Cryptominer

On my acquaintance’s system, the following file is stored as a statically linked executable (!): gitea/data/gitea/sys_health_s3 In his view, the file’s hash suggests that an XMRig cryptominer was deployed as a malicious payload; it communicates with pool.hashvault.pro and consumes a large portion of the available CPU power.

Potential for Even Greater Damage

I was even able to find a recent (Sep 2026) example on Hacker News regarding a worst-case scenario: source code theft, credential harvesting, and privilege escalation leading to root access on the affected system.

How do I detect an attack?

In the worst case, you don’t. If professionals are at work targeting the system, only the log files from the time immediately following the breach might offer clues. Provided, of course, that you read and interpret them right away. But I digress¹.

Nevertheless, you should ask yourself the following questions:

Am I running a vulnerable version of Gitea (<1.27.2)?
Have I avoided to use a monitoring solution that would have alerted me to anomalies?
Is there unusually high system CPU load? This could reveal a cryptominer infection.
For this specific miner, you can also check if the following file exists on the system:

find / -name '.sys_health_s3' 2>/dev/null

If the answer to even one of these questions is “yes,” immediate action is required. Given the severity of the vulnerabilities, the entire host system could be considered compromised, and should be if the Gitea container or any attached runner was configured as “rootful” (i.e., granted access to the Docker socket).

In Case of Infection: Quarantine

Next steps: Immediately isolate the affected machine and take it offline. Shut down the containers, reboot the host machine, and apply updates. Afterward, update your Docker registry images and check log files (look at timestamps!) to determine if other parts of the system are affected. For further details, see my article about a Gitea attack I experienced firsthand.

Solutions

Update Gitea!

Gitea urgently needs to be updated to a version >1.27.1. To do this, you can modify the docker-compose.yml file and then restart the containers:

# docker-compose.yml
gitea:
image: gitea/gitea:latest
container_name: gitea

Further information on recent fixes can be found on Gitea’s blog. The major release (version 27) addresses a long list of critical security vulnerabilities. I have singled out only a few of them in this article, as they affect my immediate environment and exploits for them are known.

Running Gitea and the runner in “rootless” mode

If the exploitation of a security vulnerability leads to the compromise of Gitea or the Action Runner, the host system itself should not also be compromised. Running them in a container can limit the potential damage. I have therefore described how to set up a “rootless” solution here.

Additional measures (hardening)

CVE-2026-20896: Restricting network access

For the next steps, I am following the recommendation from d00ftech.

Include the address range of your reverse proxy. First, you need to determine the network address at which the reverse proxy is accessible. On my server, for example, both Gitea and the reverse proxy (Caddy) run within Docker. I therefore find the IP address as follows:

docker inspect caddy \
--format='{{range $name, $network := .NetworkSettings.Networks}}{{$name}}: {{$network.IPAddress}}{{"\n"}}{{end}}'
caddy-proxy: 172.20.0.2

This value is now entered into gitea/app.ini:

[security]
REVERSE_PROXY_LIMIT = 1
REVERSE_PROXY_TRUSTED_PROXIES = 172.20.0.2/32

Block direct network access to Gitea that bypasses your own reverse proxy. To do this, the Gitea port (usually 3000) must not appear in the port mappings—for example, in the Docker configuration:

# docker-compose.yml
services:
 gitea:
 ports:
 - "3000:3000" # remove this!

CVE-2026-59774: Update Gitea!

While it is possible to disable the org-mode renderer in app.ini (see the Gitea documentation on this topic), in my view, the far better approach is to update Gitea itself!

CVE-2026-60004: Disable self-registration if possible

The actual fix requires updating the Gitea version. For smaller instances with few users, it may also make sense to disable self-registration for new users. From now on, they will have to request registration from the administrators (unless they exploit one of the other CVEs described above).

# gitea/app.ini
[service]
DISABLE_REGISTRATION = true

Other helpful settings include:

REGISTER_EMAIL_CONFIRM = true # Requires user's email to be confirmed for registration to complete
ENABLE_OPENID_SIGNUP = false # Disallow logons via external API
REQUIRE_SIGNIN_VIEW = true # Force users to login before they can view content or use the public API

Outlook and Conclusion

Version control systems are highly attractive targets for a wide range of attackers: from cryptominers seeking to hijack resources and ransomware groups targeting source code and credentials, to highly professional, targeted attackers and state-sponsored actors, given that the software effectively serves as a form of infrastructure.

Therefore:

“Apply all patches. Always. Immediately. Without exception.” - Felix von Leitner, however, also says BSI

What leaves me feeling a bit perplexed is the sheer number and severity of the security vulnerabilities that have been patched (35 CVEs in a single month, according to Gitea’s own blog!). From my naive security perspective, this solution is full of holes and I’m not sure if I should keep running Gitea.

Immediate: Harden the system

Obviously: Update Gitea to latest and implement the hardening measures described above.

Short-term: Automated updates

Over the coming days and weeks, I’ll be working on automatically scanning the Docker registry for software updates and deploying them to my server automatically as well. Blog post available now.

Medium-term: Time for monitoring

I’ll be looking for a small, lightweight monitoring solution that flags anomalies—such as “CPU load unusually high since HH:MM:SS”—and can both execute and monitor health checks. Crucially, the tool itself shouldn’t become a potential entry point for attackers, meaning it needs to operate with virtually no privileges. I wonder if that’s easily achievable?

Medium-term 2: Get rid of secrets

Unencrypted secrets have no place in repositories. Even a .env file sitting on the host itself can no longer be considered secure. I’ll be giving this some thought and will report back once I’ve rolled out a solution.

Medium-term 3: Set up news feeds

This involves subscribing to newsletters or RSS feeds, ideally from all my software vendors, IT security institutions or journalists, and other sources. To stay informed about security vulnerabilities, patches, and system hardening measures.

Long-term: Keeping the conversation going

It ends just as it begins. Before I can solve problems with my software, before I even know those problems exist, right there! At the very beginning, before I’ve even decided which software to use, there are: other people.

People I can ask questions or turn to for help. People whose time I can claim for my own needs. Like messaging them out of the blue and pulling them away from what they were focused on.

People who know more than I do, who see things differently, or whose opinions I find interesting.

Colleagues, current and former. Acquaintances, friends, and family whom I meet, whether by chance or by appointment, to connect. Over coffee, ice cream, or a cold drink; to make music or spend an afternoon playing board games.

People who help me kindly and patiently, despite the interruptions and my silly questions. People who might roll their eyes for a moment but then explain things to me one more time anyway. People who somehow tolerate my monologues, critical follow-up questions, and occasional pushback.

People with whom I can engage in an exchange that, in the end, hopefully benefits both sides.

Thanks to all of you. Everything is much easier with people like you. Including running this blog, which takes a lot of my time. With you, I can discuss decisions, learn how you did things (and why you did them that way), and shift perspectives. And develop new ideas.

Please stay in my life.

A nod to “Fefe” Felix von Leitner. Quote: “Assumptions […]: We are being attacked. We notice it promptly. Experience shows: No, we don’t. Typical timeframe before detecting stowaways: six months to five years.” - from the Hackback presentation; link leads to Codeberg. ↩︎

Hugo vs. Jekyll: Benchmark

Schallbert — Wed, 30 Sep 2026

Loading Times

Let’s take a look at how my site loads using Jekyll compared to Hugo. To do this, I open the Developer Tools (F12) in my browser, select the Network tab, and click on All. Then, I reload the page.

This results in the following list:

Image below: Load time of my landing page using Hugo

It is worth noting that I am not actually comparing Jekyll with Hugo here, but rather their output. Consequently, the chosen themes are the primary factors determining size and time to load. In effect, I am comparing the Minimal Mistakes theme in Jekyll with the Terminal theme in Hugo. In this instance, Hugo is about 10% faster than Jekyll.

Pictured below: Load time of my landing page using Jekyll

Size

I get a different chart when I click the stopwatch icon at the bottom left to run a performance analysis. For my Jekyll implementation, the pie charts show that images make up the largest share, followed by CSS and JS. With Hugo, JS usage drops to near zero, and the share of CSS has also decreased.

The table below compares a few of my pages directly. However, this overview should be viewed with caution: the values can fluctuate upon reloading (even with the cache disabled). This is particularly true for embedded third-party content, such as my videos, which can yield widely varying figures. Overall, however, I have achieved significant optimizations using Hugo and the ‘Terminal’ theme.

Item	minimal-mistakes	terminal-schallbert		Jekyll	Hugo
	transferred	[kB]	Diff [%]	time	[ms]	Diff [%]
Landing	163.47	98.68	-39	30	30	0
Post	2861	1767	-38	340	270	-20
List page	1507	140	-90	30	30	0

I manually implemented conditional loading for KaTeX in Hugo, ensuring it only loads on pages containing mathematical expressions (this saves 90kB in min.css for users).
With Hugo, my CSS files are smaller. Only what I actually use gets loaded; there is hardly any unused code left.
Hugo automatically derives media preview files from high-resolution originals and uses an image processor to optimize them at build time for the display dimensions of the site. This optimization saves a significant amount of bandwidth when there are many small images.
Unlike the Minimal Mistakes theme, the Terminal theme does not display “related posts” cards. This eliminates the need to transfer four preview images.
I have almost no JavaScript content left in Hugo—one of the goals I had set for myself.

Delivery/Continuous Deployment

Building and deploying with Jekyll in Gitea proved somewhat cumbersome: it wasn’t until I used a Jekyll Docker image that I was able to set up Ruby correctly. Because of this, and the fact that I don’t allow my Actions to use caching, the build process takes quite a long time: first pulling the latest Docker container, updating Ruby and Jekyll, installing gems, checking out source files, and finally building and deploying. Even under optimal conditions, it takes more than 90 seconds for a new post to go live.

For Hugo, the entire process is weigh faster thanks to quicker build times and pre-built containers (see my article on the Continuous Deployment Pipeline with Hugo in Docker).

Even in the worst-case scenario (see build below), for example, after restarting the runner following an update and a change to the Docker repository for my Hugo container, I remain well under 30 seconds.

Complexity

Clearly, Hugo is unbeatable here compared to Jekyll.

Since themes in Hugo are pluggable modules, they don’t require a dependency tree of Gems (as Jekyll does) to be fetched and installed. Consequently, there is much less that can go wrong. Locally, my “full build” time (bilingual) drops by 75%, and automatic builds triggered by changes to my Markdown files are both 5x faster and closer to the real thing: with Jekyll, enabling “incremental build” means paginator pages are no longer updated, meaning list views and overview pages aren’t built.

On the other hand, Jekyll allows for highly flexible, fine-grained manual intervention. I can design my own Gems and quite easily create special features for my site, such as a table of contents that scrolls along with the page. That is something I miss a bit in Hugo.

Conclusion: In terms of day-to-day usability, I definitely prefer Hugo over Jekyll. - schallbert, Sep-2026

Continuous Deployment for a Hugo site

Schallbert — Tue, 15 Sep 2026

I have just migrated my website from Hugo to Jekyll. Now, I want to automate the build and deployment process so that changes are rolled out as soon as I finish an article and push it via git push followed by a merge into main to my Gitea instance.

The Command Chain

I rent a VPS where I run all my applications in Docker. The task now is to create a workflow in Gitea (“Actions”) that checks out the repository update, passes it to Hugo to generate the HTML pages, and then saves the output to the disk in a location where my web server can read and serve the files.

Lessons Learned

I had already managed to set this up for my Jekyll website build and deployment, so I can draw on some previous experience.

Typically, standardized actions are used for this purpose. I’ll simplify things significantly here and list only the key components, without deep-diving in messy details like permission settings (is Hugo allowed to write to this folder? Does the hugo user have sufficient privileges?):

checkout makes a specific version of the source repository available
actions-hugo installs Hugo within the Action’s virtual machine so the website can be built
upload-artifact uploads the site generated by Hugo to the web server

However, since I am operating within Docker on the same machine, I can skip the hassle of logging into the web server “from the outside”. So, I’m taking a different approach.

Docker shared volumes

I can make the built files directly available to my Caddy application by passing the volume (that Caddy reads from) through to the Action.

Heads up: Action volume != Hugo volume

But this doesn’t work with actions-hugo: It is not a Docker container itself, so it cannot mount volumes. While it does successfully build the website, it sends the artifacts into oblivion as soon as the Action finishes running.

So, I need a Hugo runtime that executes inside a Docker container itself, following the DinD approach. I can then pass the volume with write access.

The workflow

But which Docker container should I choose?

Imagine here several hours of research and numerous failed attempts while setting up the workflow on my server.

My investigation revealed the following:

checkout@v4 requires Node.js—specifically, at least version node:18. The Docker container must have it installed.
The action requires git. Ideally, the Docker container should already include this as well.
Hugo needs go. That makes sense.

Adjusting the runner operating system

My runner from 2023 did not age well. It is based on node:16-bullseye, a Debian-based VM. I experiment a bit with Alpine, but that installation is too minimal for my needs and would require too many manual adjustments. After all, the runner has to execute other workflows besides this one. Ultimately, I decide to stick with Debian and just upgrade the Node version. The implementation in the runner configuration looks like this:

# /gitea/runner/config.yml

# The labels of a runner are used to determine which jobs the runner can run, and how to run them. 
# Like: ["macos-arm64:host", "ubuntu-latest:docker://node:16-bullseye", "ubuntu-22.04:docker://node:16-bullseye"]
# If it's empty when registering, it will ask for inputting labels. 
# If it's empty when execute `deamon`, will use labels in `.runner` file. 
labels: [ubuntu-latest:docker://node:20-bullseye]

Afterwards, the runner container must be restarted. Ideally, you should reboot Gitea as well; this ensures that runner registration works reliably.

Writing the workflow script

To handle the dependencies described above, I’m opting for a “batteries-included” approach. This allows me to assemble the components without manual adjustments and rely on the long-term maintenance of the containers, given their widespread use (with pull counts in the millions). This website was very helpful in making my selection.

Finally, I create my workflow based on the container from Hugomods.

# /.gitea/workflows/build-deploy-with-hugo.yml

name: Deploy Hugo site
run-name: ${{ gitea.actor }} builds Hugo site
on:
 push:
 branches:
 - main

jobs:
 # Build job
 build:
 runs-on: ubuntu-latest # this is the "label" the runner will use and map to docker target OS
 container: 
 image: hugomods/hugo:latest
 volumes: 
 - /tmp/blog-artifacts:/tmp/blog-artifacts

 steps:
 - name: --- CHECKOUT ---
 uses: actions/checkout@v4
 with:
 submodules: recursive  # Fetches Hugo themes
 fetch-depth: 0

 - name: --- BUILD WITH HUGO --- # chown -R hugo /tmp/blog-artifacts
 run: |
 hugo \
 --minify \
 --cleanDestinationDir \
 --destination /tmp/blog-artifacts

Cleaning the destination folder

In the example above, I opted for a “direct deployment”. This is a basic approach without a safety net, and without copying artifacts around or restarting the server. The reason: the Hugo build process takes less than 2,000ms for my site. It is only within this window that Caddy could theoretically encounter and serve “corrupt” data. Specifically, the window is even smaller as occurrence is limited to the Hugo build writing artifacts to the disk.

To minimize inconsistencies regardless, I use the --cleanDestinationDir option with Hugo. This ensures the destination directory is always clean; consequently, in the event of an issue, Caddy won’t display a broken site but will instead generate a 404 error.

Submodules

As noted in my project documentation, you can use “submodules” in Hugo to load themes. These are integrated into the project and—provided they are linked correctly—can be utilized within the workflow:

with:
submodules: recursive  # Fetches Hugo themes
fetch-depth: 0

If everything is configured correctly, you should not encounter the following error message:

# workflow action's error message due to incoherent submodules' commit references
# actions/checkout@v4:
Fetching submodules
/usr/bin/git submodule sync
/usr/bin/git -c protocol.version=2 submodule update --init --force
Submodule 'themes/terminal' (https://github.com/panr/hugo-theme-terminal.git) registered for path 'themes/terminal'
Cloning into '/workspace/schallbert/blog-hugo/themes/terminal'...
fatal: remote error: upload-pack: not our ref 719505fc89332baa69bffb90cee708ff124dd143
Fetched in submodule path 'themes/terminal', but it did not contain 719505fc89332baa69bffb90cee708ff124dd143. Direct fetching of that commit failed.
Error: The process '/usr/bin/git' failed with exit code 1
Error: Process completed with exit code 1.

A prerequisite is that the submodules have been correctly loaded locally:

schallbert@machine:~ git submodule add -f https://github.com/<my/hugo-theme>.git themes/my-hugo-theme
schallbert@machine:~ git submodule update --init --recursive

The status check must not return an empty string; instead, it must include a commit hash. Example:

schallbert@machine:~ git submodule status
44d9a1890d228745ffc300b37a7d73e940ef9fa9 themes/terminal (v4.2.5)

Further confirmation is provided by the .gitmodules file, which must contain a reference to the theme:

schallbert@machine:➜/blog git:(main) ✗ nano .gitmodules
[submodule "themes/terminal"]
path = themes/terminal
url = https://github.com/panr/hugo-theme-terminal.git

If this does not work, the submodules must be completely removed and then re-initialized as described above:

git submodule deinit -f themes/<my-theme>
git rm -r --cached themes/<my-theme>
rm -rf .git/modules/themes/<my-theme>

Common Issues: Hugo Versions and Deprecations

The checkout action completed successfully, but the Hugo build fails:

error calling partial: "/workspace/schallbert/blog-hugo/layouts/_partials/head.html:37:40": execute of template failed: template: _partials/head.html:37:40: executing "_partials/head.html" at <$.Page.Language.Locale>: can't evaluate field Locale in type *langs.Language
Error: Process completed with exit code 1.

Locally on my machine, Hugo builds successfully but issues a few warnings:

Changes to Language Identifiers and Evaluation as of v0.158

schallbert@machine:➜/blog git:(main) ✗ hugo build --logLevel info
WARN deprecated: site.Language.Locale was deprecated in Hugo v0.158.0 and will be removed in a future release. Use .Page.Language.Locale instead.
WARN deprecated: .Language.Lang was deprecated in Hugo v0.158.0 and will be removed in a future release.

You might think these errors would be easy to fix. Unfortunately, they can be very good at hiding:

in submodules loaded as third-party content (e.g., in themes). A project-wide search in the IDE usually excludes submodules and yields no results.
as variables in your own code that access the old parameters only indirectly: Up until now, I’ve been using .Page.Site.Home.AllTranslations to determine if I am currently on the “main language” version of a given page. The scoping alone is illogical, and in my opinion, it is rightly being removed. What I want to achieve can now be done much more easily using .Page.Language.IsDefault.

Updating Submodules

I see two possible approaches for handling submodules:

Contact the maintainers (or open an issue) to let them know their code is becoming outdated and ask them to update their themes.
Copy the affected files out of the submodule and integrate them into my own folder structure. In my case, for example, the affected language-menu.html would find a new home at /layouts/partials/language-menu.html. I can then fix the issues there myself.

Implementing Custom Solutions

If the error message doesn’t make the solution obvious, the Hugo community is a great resource. For instance, there is detailed documentation available regarding the deprecations in v0.158.

Comparing Versions

Hmm. As the image shows, my container uses the hugo extended package from hugomods, which provides version v0.154.5; I had selected a stable release for the Action. Let’s compare this with my local version, which now builds perfectly without a single warning:

➜ blog git:(main) ✗ hugo version
hugo v0.162.1+extended linux/amd64

The classic scenario: A change introduced in v0.158 altered the internal API. Older versions don’t recognize the new fields and throw errors. Newer versions, however, flag the use of the old fields as an issue.

I believe these problems can only be reliably avoided if the pipeline is identical for both local and remote environments. In other words: boot up a virtual machine on your local computer running the CI operating system and encapsulate everything within Docker. That certainly makes sense if you work in a larger organization where multiple users face this same issue.

To fix the problem, I switch to the image: hugomods/hugo:latest version within the remote container instead. However, I still end up with version mismatches between my local setup and the CD pipeline. The latest version frankly isn’t available for my operating system yet, and the Docker release also lags slightly behind the original releases.

The build and release times are fantastic, aren’t they? Getting a website change online in just 12 seconds, including spinning up the container and so on, is something I’ve certainly never experienced before. The process takes around 20 seconds when it has to fetch and download container references from scratch, such as when the latest tag in the registry is updated.

The linked article offers a detailed benchmark comparing Hugo and Jekyll.

Migrating from Jekyll to Hugo

Schallbert — Sun, 06 Sep 2026

Project Profile

Difficulty: Medium (3/5)
Cost: €0
Time: ~20h

In this project, I describe my migration from Jekyll using the Minimal Mistakes theme to Hugo, utilizing a heavily modified Terminal theme.

Why switch at all?

For quite some time, I hadn’t been fully satisfied with building my website using Jekyll. The trigger for the switch was a day when I suddenly couldn’t build the site locally due to a compatibility issue in Ruby. Standard commands like bundle update --conservative or bundle install didn’t help. Even manually installing the problematic packages using commands like gem install commonmarker-0.23.12 or gem install posix-spawn -v 0.3.15 -- --with-cflags="-Wno-incompatible-function-pointer-types" failed to resolve the problem.

Updating Ruby and Jekyll themselves didn’t work either. Besides, build times with Jekyll had become quite long. Since moving the site from GitHub Pages to my own server, I am no longer tied to Jekyll, making a switch more attractive.

This no-build-locally problem was just the final nudge that made me begin this laborious task. I hate it when technology doesn’t do what I want - even though I admit that often times, I am the root cause for that myself.

Further difficulties with Jekyll

CI&CD: My deployment pipeline had failed months ago due to similar errors, forcing me to revert to an older version of Jekyll. This increases technical debt and the attack surface of my GitHub Actions. I would have to look for a well-maintained container solution from another provider. Searching for and setting one up would require additional time.
RSS feed: I wanted to provide an RSS feed for my website. However, my experiments with Jekyll showed that only “posts” were included, not “announcements.” I wanted to fix this.
Image formats: Jekyll normally generates cover images (thumbnails) for posts within the RSS feed. Unfortunately, this stopped working after I switched from .jpg to the more space-efficient .avif format. Despite several attempts, I couldn’t get a solution approved through the pull request review process.
Icons: The theme I was using relies on Font Awesome to display icons and is quite large overall for a static website. Since I didn’t want to load assets from external sources, I stored the library locally. My plan was to select only the icons I actually needed and delete the rest. With the switch to Hugo, this is no longer necessary.

Opportunities

I expect Hugo to deliver faster build times, a simplified CI/CD process (without the need to install Bundler, Ruby, etc.), and a resolution to the difficulties described above. Hugo’s approach to dependency and package management differs fundamentally from Jekyll’s: additional content is integrated via Git Submodules, and dependency management relies on Go modules rather than a Gemfile.

The migration tool

The folder structures for content differ significantly between Jekyll and Hugo. I am therefore using an import tool included with Hugo, which correctly imports at least the articles located under /posts. However, specific content such as overview pages (/tags, /pages/error, /pages/legal), and pages hosting projects and announcements (/_announcements, /_projects), will not be migrated automatically and must be copied and modified manually.

Adjusting Front Matter

Many of the customizations I implemented in Jekyll either do not work correctly in Hugo or cause the build process to fail. Consequently, the front matter of every Markdown file must be adjusted and, in some cases, cleaned up.

---
# Alt: So kann die Front matter einer Übersichtsseite in Jekyll aussehen
lang: de
title: Beitragsarchiv
subtitle: Diese Sammlung enthält alle meine Beiträge
layout: collection
collection: posts
permalink: /posts-archive/
entries_layout: grid
classes: wide
author_profile: false
---

In my case, there are several theme-specific parameters in the front matter, such as layout, classes, and author_profile. The lang parameter from my Jekyll internationalization extension isn’t even valid Hugo syntax; Hugo expects the language setting either as a field under params: lang or defined at the folder level. Other custom parameters tailored to personal preference can also be defined and passed via params.

Permalinks are no longer included in the front matter; instead, configuration is handled centrally.

Hugo-compatible front matter might look like this:

---
title: Post Archive
description: This collection contains all my posts
layout: collection
---

Theme Configuration

I can install a basic configuration for my theme using the following command:

➜ blog git:(main) ✗ git submodule add https://github.com/panr/hugo-theme-terminal ./themes/terminal

To activate it, I need to specify it in the main configuration file:

# /hugo.toml

# Add it only if you keep the theme in the `themes` directory.
# Remove it if you use the theme as a remote Hugo Module.
theme = "terminal"

And just like that, Hugo builds the site using the Terminal theme and its default colors.

However, the layout isn’t vertically centered and is too narrow for my taste; the text is quite small, and I want “my” blog colors to be different as well. Therefore, I am creating a .css file that overrides some properties of the base theme. It must be located in the /static folder and named after the theme.

/* /static/terminal.css */

:root{
 --background: #252a34;
 --background-highlight: #475064;
 --foreground: #d2eaef;
 --foreground-dimmed: #8b9ea2;
 --accent: #25a679;
 --accent-highlight: #79ae9c;
 --radius: 8px;
 --font-size: 1.35rem;
 --line-height: 1.8em;
}

/* Use system fonts, reduce transferred kB */
body{
 font-family: system-ui, -apple-system, "Segoe UI", Roboto, Arial, sans-serif !important; 
}

/* make content wider */
body .container{
 max-width: 70rem;
 width: 90%;
}

And just like that, it looks almost like it used to.

Organizing the folder structure

Unlike the version of Jekyll I was using, Hugo features built-in internationalization (i18n). This requires me to restructure my folders and explicitly sort content into /de and /en directories. Instead of sitting at a higher folder level, content is now stored directly within the content folder—under /_pages, /_announcements, and /_posts—which makes things much tidier.

Before

.
├── _announcements
├── assets
│ ├── css
│ │ ├── fontawesome
│ │ ├── katex
│ │ ├── main.scss # <-- theme import, customization
│ │ └── webfonts
│ ├── js
│ │ ├── lunr
│ │ ├── _main.js
│ │ ├── main.min.js
│ │ ├── plugins
│ │ └── vendor
│ └── video
├── banner.js
├── CNAME
├── _config.yml
├── _data
│ ├── de
│ │ ├── l10n.yml
│ └── en
│ ├── l10n.yml
├── en
│ └── index.html
├── Gemfile
├── Gemfile.lock
├── _includes # <-- Layouts (HTML), partials, 3rd-party
├── index.html # <-- Landing page
├── _layouts # <-- Page layouts
├── _pages
│ ├── 404.md # <-- German version
│ ├── [...]
│ ├── en
│ │ ├── 404.md # <-- English version
│ │ ├── [...]
├── _posts/
│ └── en 
├── _projects/
├── _sass
│ ├── minimal-mistakes # <-- Styles (SCSS) for layouts and partials
│ ├── minimal-mistakes.scss
│ └── _video.scss
├── _site/ # <-- Build artifacts
├── staticman.yml

After

Hugo’s documentation provides a good overview of the folder structure. The root folders content, layouts, and static are the most important for day-to-day work with articles and the website’s appearance.

.
├── archetypes
├── assets
├── content
│ ├── de
│ │ ├── about.md
│ │ ├── announcements/
│ │ ├── posts/
│ │ └── projects/
│ └── en
│ ├── about.md
│ ├── [...]
├── data
├── hugo.yml
├── layouts
│ ├── _default
│ │ ├── baseof.html # <-- defines header, add-ons, content, footer structure
│ │ ├── index.html # <-- landing page
│ │ ├── list.html # <-- grid view
│ │ └── single.html # <-- post view
│ ├── _markup
│ │ └── render-link.html # <-- link highlighting and function (referrer, tabs)
│ ├── _partials
│ │ ├── cover.html # <-- cover (thumbnail) image rendering
│ │ └── math.html # <-- math rendering (katex)
│ └── _shortcodes
│ ├── audio.html
│ ├── image.html
│ └── [...]
├── resources
│ └── _gen
│ └── assets/
├── static
│ ├── assets
│ │ ├── audio/
│ │ ├── css/ # <-- contains overrides for custom layouts 
│ │ ├── docs/
│ │ ├── images/
│ │ ├── js/ # <-- math (katex) JS lives here
│ │ └── video/
│ └── terminal.css # <-- Central theme override (colors, formatting, styling)
└── themes
 └── terminal # <-- GIT submodule: Vanilla "Terminal" theme

With both solutions, layouts are defined in HTML and styles/formatting in CSS. However, since I had to use more plugins with Jekyll, the setup is more extensive.

Manual Adjustments: Shortcodes

Hugo’s shortcodes allow you to define non-text content (media, links, formatting, styles) that is then rendered according to specific rules. To do this, you create an HTML file that defines the wrapper and includes the relevant CSS classes. You also add the desired formatting to the theme’s overriding CSS file.

Since I am by no means an expert, I simply experiment until I am happy with the layout and no longer see any artifacts or overlapping elements.

.
├── layouts
│ └── _shortcodes
│ └── my-shortcode.html # <-- Custom shortcode file
├── static
│ └── terminal.css # <-- Central theme override (colors, formatting, styling)

Syntax: Hugo and Jekyll

In Hugo, shortcodes are represented in the Markdown file using the following syntax:

{{< shortcode-html-filename options >}}
{{< image src="/path/to/image.avif" alt="Image: alt text" position="right" >}}

By comparison, shortcode-like decorators in Jekyll using Liquid, the templating language employed by Jekyll look like this:

### Liquid in interpreter code
{% Liquid bracket syntax %}
### Liquid "shortcode" pendant
{% include gallery id="gallery" caption="Eindrücke von **MobFobAmp**. Zum Vergrößern anklicken." %}
### Liquid decorator for layouting
{:.list-inline}

image

Example of image display. A centered image follows on the next line.

The standard Markdown image syntax [Alt Text](/path/to/image) renders as a centered image without text wrapping by default in Hugo. However, for a pleasing layout, I need a few variations that require some additional CSS and HTML code.

Centered image, no text wrapping
Left-aligned image, text wrapping on the right
Right-aligned image, text wrapping on the left

First, Hugo retrieves the file path and alt text. Then, it reads the desired formatting defined via the position variable from the Markdown file. Finally, it applies the image-wrapper class from the CSS file. To ensure better readability, the image alignment applies only to the desktop version of the site (>684px).

The theme customizations required to display images correctly are quite extensive. I also developed a gallery feature for posts containing a large number of images. Consequently, I wrote a separate article covering image formatting and alignment.

audio

I use standard browser-native features to display the audio player.

Your browser does not support the audio element.

Beispielplayer: A lead melody on electric bass

{{- /* /layouts/_shortcodes/audio.html */ -}}

{{ $src := .Get "src" }}
{{ $caption := .Get "caption" | default "" }}

<div class="media-wrapper">
 <div class="media-container">
 <audio controls>
 <source src="{{ $src }}" type="audio/mp3">
 Your browser does not support the audio element.
 </audio>
 {{ with $caption }}
 <div class="media-title">{{ . }}</div>
 {{ end }}
 </div>
</div>

In css, I use a wrapper and a media container to display titles within the player frame and to simplify formatting. Here, too, I have implemented custom behavior for a responsive layout on mobile devices or when the browser window is resized.

video

The CSS is identical to that of the audio element; there are only minor differences in the HTML. The container I use supports both “embed links” from sites like PeerTube and local source files via the src= attribute.

Your browser does not support the video tag.

Video example (local): linear glides with ball bearing.

{{- /* /layouts/_shortcodes/video.html */ -}}

{{- $src := .Get "src" -}}
{{- $title := .Get "title" | default "" -}}
{{- $caption := .Get "caption" | default "" -}}

<div class="media-wrapper">
 <div class="media-container">
 {{- if hasPrefix $src "http" -}}
 <iframe src="{{ $src }}" frameborder="0" allowfullscreen></iframe>
 {{- else -}}
 <video controls>
 <source src="{{ $src }}" type="video/mp4">
 Your browser does not support the video tag.
 </video>
 {{- end -}}

 {{- if or $title $caption -}}
 <div class="media-title">
 {{- if $title -}}{{ $title }}{{- else -}}{{ $caption }}{{- end -}}
 </div>
 {{- end -}}
 </div>
</div>

Updates

My content ages, too. Sometimes I switch technologies or providers, and sometimes third parties move away from solutions I’ve described implementing here on the blog. That’s why I introduced update-shortcodes.

Links

I want the following behavior for links on my blog:

internal links should open in the same tab and redirect the user.
internal links should point to pages in the same language, rather than leading to the default language or back to the homepage.
external links should be marked with an icon: ↗
external links should open in a new tab: target="_blank"
external links should neither access the original content nor be able to see the referrer: rel="noopener noreferrer"

To achieve this, I create the file render-link.html in /layouts/_markup. I present the code and the background details in a separate post: Hugo Multi-language: Internal Permalinks

The CSS override looks like this:

/* /static/terminal.css */

.external-link .external-link-icon{
display: inline-block; 
font-size: var(--font-size); 
vertical-align: middle; 
transform: translateY(-0.1em);
}

Manual Customizations: Partials

You can think of partials as the layout building blocks for a Hugo website. They are stored in /layouts/_partials and override the defaults of the theme being used. Since the original file is no longer loaded at all, the recommended approach is to copy the original file into the folder and then customize it manually.

index

The blog’s homepage is entirely custom-written; it is not in Markdown format but in HTML. It displays its content differently than all other blog pages, so it wasn’t worth the effort for me to program specific templates. In the corresponding /layouts/_partials/index.html file, I load the necessary CSS files and define the page structure. The following table provides an overview of my changes.

File	Purpose	Customization
cover	thumbnail / cover image for overview pages, lists, and grids	Add custom cover image decorator, alt text
head	Defines Favicon, search console, page parameters, feeds etc.	Custom favicon path
header	Page header containing menu, logo, navigation etc.	Add logo and subtitle to menu, lang selector
logo	Custom page logo, title, subtitle	Full
math	Displays math content with Latex/Katex	None, implement math support

Math

My blog is designed to support mathematical notation. I use KaTeX for this.

{{- /* /layouts/_partials/math.html */ -}}

<link rel="stylesheet" href="/assets/css/katex.min.css">
<script defer src="/assets/js/katex.min.js"></script>
<script defer src="/assets/js/contrib/auto-render.min.js" onload="renderMathInElement(document.body);"></script>

<script>
 document.addEventListener("DOMContentLoaded", function() {
 renderMathInElement(document.body, {
 delimiters: [
 {left: '\\[', right: '\\]', display: true}, // block
 {left: '$$', right: '$$', display: true}, // block
 {left: '\\(', right: '\\)', display: false}, // inline
 ],
 throwOnError : false
 });
 });
</script>

cover image

I use the term cover to refer to a blog post’s title image. It is displayed on overview pages, in article and tag lists, etc., and is intended to complement the blog post’s content. My RSS feed should also use these images to provide an overview of various articles on my readers’ devices.

It is highly compressed and requires minimal bandwidth, resulting in a low resolution (440x220 or 640x352). To ensure a consistent look, the images are also always monochrome.

I use the following Markdown syntax to embed the cover image:

---
cover_alt: 'Image: Hugo console build output. It shows a full build time below 2sec. (~90% reduction over build time with Jekyll)'
cover: /assets/images/migrating-hugo-to-jekyll/hugo-jekyll-build-compare-cover.avif
---

Next, I create the file cover.html in the layouts/_partials directory and add the following code:

{{- /* /layouts/_partials/cover.html */ -}}

{{- /* Handles cover-image generation */ -}}

{{- $cover := false -}}
{{- $autoCover := default $.Site.Params.autoCover false }}

{{- if index .Params "cover" -}}
 {{- if .Resources.GetMatch .Params.Cover }}
 {{- $cover = (.Resources.GetMatch .Params.Cover).RelPermalink -}}
 {{- else -}}
 {{- $cover = absURL .Params.Cover -}}
 {{- end -}}
{{- else if $.Site.Params.AutoCover -}}
 {{- if (not .Params.Cover) -}}
 {{- if .Resources.GetMatch "cover.*" -}}
 {{- $cover = (.Resources.GetMatch "cover.*").RelPermalink -}}
 {{- end -}}
 {{- end -}}
{{- end -}}

{{if $cover -}}
 <!-- Cover image found -->
 <img src="{{ $cover }}"
 class="post-cover"
 alt="{{ .Params.cover_alt | plainify | default (.Description| plainify) }}"
 title="{{ .Params.CoverCredit |plainify|default (.Title | plainify) }}" />
{{- end }}

The code searches for and extracts the path to the cover image. If a valid path is found, the image is displayed as post-cover with the corresponding CSS.

I insert the cover image into the XML template as the first element within the <description> tag; the actual article follows only after that. Most feed readers then use this image as the cover for their overview lists.

Manual Adjustments: Layouts

To ensure my Hugo site continues to feel very similar to my previous Jekyll site, I need to make a few changes to the layouts. I have described them below.

baseof.html

This is where the base content is defined. Definitions for the header and footer are imported, and stylesheets are linked. The vast majority of the changes described above are imported here.

{{- /* /layouts/baseof.html */ -}}

<!DOCTYPE html>
<html lang="{{ $.Site.Language }}">
<head>
 {{ if .Param "math" }}
 {{ partialCached "math.html" . }}
 {{ end }}
 {{ block "title" . }}
 <title>{{ if .IsHome }}{{ $.Site.Title }}{{ else }}{{ .Title }} :: {{ $.Site.Title }}{{ end }}</title>
 {{ end }}
 {{ partial "head.html" . }}
 <link rel="stylesheet" href="{{ "/assets/css/gallery.css" | relURL }}"> 
 <link rel="stylesheet" href="{{ "/assets/css/media.css" | relURL }}">
</head>
<body>
{{ $container := cond ($.Site.Params.FullWidthTheme | default false) "container full" (cond ($.Site.Params.CenterTheme | default false) "container center" "container") }}

<div class="{{- $container -}}{{- cond ($.Site.Params.oneHeadingSize | default false) " headings--one-size" "" }}">

 {{ partial "header.html" . }}

 <div class="content">
 {{ block "main" . }}
 {{ end }}
 </div>

 {{ block "footer" . }}
 {{ partial "footer.html" . }}
 {{ end }}
</div>

</body>
</html>

list.html

I need a list view for overview pages such as Articles, Tags, and Projects. I want to display four items per row. Each item gets its own “card” featuring a cover image, title, short description, and, if available, additional details like reading time. Hugo’s paginator crawls the relevant folder and compiles the content, while my list.css stylesheet handles the formatting and display.

{{- /* /layouts/_default/list.html */ -}}

{{ define "main" }}
 <link rel="stylesheet" href="{{ "/assets/css/list.css" | relURL }}">
 {{ with .Content }}
 <div class="index-content">
 {{ . }}
 </div>
 {{ end }}

 <div class="posts posts-grid">
 {{ range .Paginator.Pages }}
 <article class="post on-list">
 <a href="{{ .Permalink }}" class="post-card" aria-label="{{ .Title }}">
 {{ partial "cover.html" . }}

 <div class="post-card-body">
 <h4 class="post-title">
 {{ .Title | markdownify }}
 </h4>

 <div class="post-excerpt">
 {{ if .Description }}
 <p>{{ .Description }}</p>
 {{ end }}
 </div>

 {{- if and (.Param "readingTime") (eq (.Param "readingTime") true) -}}
 <div class="post-reading-time">
 {{ .ReadingTime }} {{ $.Site.Params.minuteReadingTime | default "min read" }}
 </div>
 {{- end -}}
 </div>
 </a>
 </article>
 {{ end }}

 {{ partial "pagination.html" . }}
 </div>
{{ end }}

single.html

My single.html is almost identical to the default implementation found in terminal. I only added the option to include a “banner-image” associated with the title and imported the _partial/cover.html snippet to load a cover image.

<article class="post">
{{ with .Params.banner }}
<img
src="{{ . }}"
class="post-banner"
alt="{{ $.Params.banner_alt | plainify | default ' ' }}"
/>
{{ end }}

Given all these changes, one might consider forking the theme to create a custom version… 🤔

Quality Control

Another new addition to my website is a check for broken links. The site has grown up and now contains so many internal and external links that manual checking is no longer practical. The article on htmltest configuration for Hugo documents the tool I use and how I account for Hugo-specific quirks during the process.

Publishing

I also need an automated pipeline for Hugo. Whenever I push a change to the main branch, the website should be automatically rebuilt and deployed. I describe exactly how this works in the article Building and Deploying a Hugo Website with GitHub Actions and Docker.

Performance

What has all this effort achieved? For users, the appearance and navigation remain very similar to the Jekyll-based site. However, the improvements are evident in faster load times, reduced data transfer, and better overall clarity. I provide a detailed comparison in the article Jekyll vs. Hugo Benchmarking.

Image alignment and formatting in Hugo

Schallbert — Sat, 29 Aug 2026

Why do this?

My theme cannot derive image formatting or positioning from standard Markdown syntax like ![Image Alt Text](/path/to/image/). However, I need this capability to place images aligned left, center, or right, with or without text wrap. So, I have to take matters into my own hands and code a solution myself.

Getting started with shortcodes, layouts, and CSS

I get an overview at cshire and quickly realize that my initial attempt using /layouts/render-image.html isn’t working.

Debugging

This method lets me determine whether render-image.html is actually being read and applied—independent of any CSS:

<div style="border: 3px dashed red; padding: 10px; margin: 10px; font-weight: bold;">
RENDER-IMAGE.HTML IS BEING READ!
</div>

That works. So, Hugo must be missing something during the Markdown processing stage.

Troubleshooting

I can see now that Hugo is rendering the layout but isn’t processing my images correctly. Why? Because a trigger, called shortcode, is missing. In Hugo, shortcodes are created in Markdown using {{ <shortcode-name> parameter1="String" parametern="string" }}, processed via the corresponding template file /layouts/shortcodes/<shortcode-name>.html, and finally displayed based on CSS styling.

Shortcode: image.html

So, I create a shortcode template that allows me to pass a few extra parameters to the image beyond just the source file (src) and alt text (alt): position, caption, attrid, and attrlink. The image position and caption are self-explanatory. I wanted a way to include attribution for images I didn’t create myself. For these, I either access open content governed by terms such as Creative Commons or obtain prior written consent from the copyright holder.

{{- /* layouts/_shortcodes/image.html */ -}}

{{- $src := .Get "src" -}}
{{- $alt := .Get "alt" | default "" -}}
{{- $position := .Get "position" | default "center" -}}
{{- $caption := .Get "caption" -}}
{{- $attrid := .Get "attrid" -}}
{{- $attrlink := .Get "attrlink" -}}

<figure class="media-frame media-frame--{{ $position }}">
 <img src="{{ $src }}" alt="{{ $alt }}">

 {{- if $caption -}}
 <figcaption class="media-caption">
 <span class="caption-text">{{ $caption }}</span>
 {{- if and $attrid $attrlink -}}
 <a
 href="{{ $attrlink }}"
 class="attr-link"
 aria-label="Attribution {{ $attrid }}"
 >
 <sup class="attr-id">[{{ $attrid }}]</sup>
 </a>
 {{- end -}}
 </figcaption>
 {{- end -}}
</figure>

This template reads the parameters provided via the shortcode and wraps the <img> tag in a <figure class="media-frame"> container, which holds the positioning information. If an image caption and/or attribution is provided, it is displayed within a <figcaption class="media-caption"> element.

Using the shortcode

In Markdown, a fully configured image would look like this:

{{< image src="/path/to/image.jpg"
alt="Example placeholder image for shortcode demonstration purposes"
caption="Shortcode demo image"
attrid="1" attrlink="https://example.com/"
position="right" >}}

Migration

But how do I convert the ![alt](/path/to/image) {: .align-center} syntax I previously used in Jekyll into the format shown above?

With a script! To do this, I’m creating a Python program that scans through all my articles, looking for images using the aforementioned formatting. When it finds a match, it extracts the src, alt, and position attributes and replaces the entire expression with the equivalent Hugo shortcode.

import os
import re
from pathlib import Path

# spaces/tabs only, no newlines
SPACES = r'[ \t]*'

IMG_WITH_OPTIONAL_ALIGN = re.compile(
 r'!\[([^\]]*?)\]\(([^)]*?)\)' # alt, src
 + SPACES + # optional spaces/tabs before decorator
 r'(?:\{:' + SPACES + # literal "{:"
 r'[^}\n]*?\.align-(center|left|right)\b[^}\n]*?' # decorator body (no newlines)
 r'\})?' # optional "}"
)

def convert_image_alignments(directory):
 def repl(m):
 alt = m.group(1).replace('"', "'")
 src = m.group(2)
 position = m.group(3) or ''
 # return built Hugo shortcode. Escape Python f-string curly brace { with {{.
 return f'{{{{< image src="{src}" alt="{alt}" position="{position}" >}}}}'

 for root, _, files in os.walk(directory):
 for file in files:
 if not file.endswith('.md'):
 continue

 filepath = Path(root) / file
 content = filepath.read_text(encoding='utf-8')

 new_content = IMG_WITH_OPTIONAL_ALIGN.sub(repl, content)
 if new_content != content:
 filepath.write_text(new_content, encoding='utf-8')
 print(f"Updated: {filepath}")

if __name__ == "__main__":
 project_root = input("Enter the path to your project root: ").strip()
 if os.path.isdir(project_root):
 convert_image_alignments(project_root)
 print("Conversion complete!")
 else:
 print("Invalid directory path.")

I enlisted the help of the large-language model Mistral for the regexes, but otherwise, the script is kept simple.

In Jekyll, images with titles and/or attribution were declared as figure elements. I am writing a very similar script for these as well and running it during the migration process.

CSS

Let’s move on to image display. I am creating a dedicated CSS file, /assets/css/media.css, designed to standardize the size and formatting of images as well as other embedded media such as audio and video files for both mobile and desktop views. To achieve this, it utilizes the class attributes defined in the template.

Here is an excerpt from the file to illustrate: this section defines the behavior of the media-frame object. Each asset (image, video, or audio) is embedded within a frame that provides the necessary breathing room for the display. The position attribute determines the placement and presentation style.

By default, the content is centered and displayed at its native resolution, provided it does not exceed the maximum text width.

.media-frame img,
.media-frame audio,
.media-frame video {
 display: block;
 width: auto;
 max-width: 100%;
 height: auto;

 margin: 0;
 padding: 0;
 border: 0;
 border-radius: 0;
}

.media-frame--center {
 margin-left: auto;
 margin-right: auto;
}

/* Left/Right align only for Non-mobile use */
@media (min-width: 768px) {
 .media-frame--left {
 float: left;
 max-width: 60%;
 margin-top: 0.35em;
 margin-right: 1rem;
 margin-bottom: 0.75rem;
 }

 .media-frame--right {
 float: right;
 max-width: 60%;
 margin-top: 0.35em;
 margin-left: 1rem;
 margin-bottom: 0.75rem;
 }
}

Images aligned to one side have their width restricted to allow text to flow around them. Additionally, minimum margins relative to other objects are defined to prevent frames from touching when media elements appear in succession. In the mobile view, space is so limited that images are always displayed centered.

Gallery

To display multiple images in a single location, I create a gallery shortcode along with the corresponding CSS. The gallery displays images in a grid. Hovering the mouse over an image makes it appear brighter than the others. Clicking an image displays a large version of it at the bottom of the gallery, reusing the media shortcode described earlier.

The images to be displayed are defined as a list—specifying src and alt attributes—within the Markdown document’s front matter. Here is an excerpt from the template used to process the shortcode.

<section class="hugo-gallery">
 <div class="hugo-gallery__frame">
 <div class="hugo-gallery__grid" role="list">
 {{- range $i, $img := $items -}}
 {{- $src := $img.src -}}
 {{- $alt := $img.alt | default (printf "Gallery image %d" (add $i 1)) -}}

 {{- $thumbURL := $src -}}
 {{- $p := strings.TrimPrefix $src "/assets/" -}}

 {{- with resources.Get $p -}}
 {{- $thumb := .Fit "400x220" -}}
 {{- $thumbURL = $thumb.RelPermalink -}}
 {{- end -}}

 <a
 class="hugo-gallery__thumb"
 href="#gallery-full-{{ $i }}"
 aria-label="{{ $alt }}">
 <img
 src="{{ $thumbURL }}"
 alt="{{ $alt }}"
 class="hugo-gallery__thumb-img"
 loading="lazy">
 </a>
 {{- end -}}
 </div>

I iterate through all the images defined in the front matter, limit their size to a manageable 400x220px, and arrange them in a grid using the class hugo-gallery__grid.

Tests

Creating the templates was relatively quick. However, I’m constantly at odds with CSS. Some override from another file keeps getting in the way, or I end up conflicting with the styles in the base CSS. That’s why I need a place to test formatting and layout.

So, I created an image test page for myself. Curious? Check how media is displayed in Hugo

htmltest configuration ideas for Hugo

Schallbert — Mon, 10 Aug 2026

I am now nearly finished migrating my website to Hugo. I had to make several adjustments to the front matter of each article so that they would function in Hugo much as they did in Jekyll. Changes to the folder structure, partly from the migration itself and partly from my experience as an author, meant I had to revise links in practically every article along the way. I now have nearly 340 of them, counting translations.

I cannot manage the sheer volume of links without some assistance. So, I need a tool.

htmltest

Htmltest is a utility that scans HTML files for issues. Among other things, it tracks down broken links. I install it using

# Red Hat-based Linux package manager install command
sudo dnf install htmltest

and run it against the HTML files built by Hugo in the /public directory:

htmltest public/
# [...]
✘✘✘ failed in 2.890119823s
2612 errors in 337 documents

That is quite a lot of errors! Furthermore, the way the information is presented seems rather cluttered to me: see the image.

Redirecting output to a file

Consequently, I am finding it quite difficult to work with this tool at first. I had assumed a few links might have broken, but I certainly didn’t expect thousands of errors. So I fell victim to the typical “not-invented-here” syndrome and programmed my own tool in Python, which, in hindsight, turned out not to be good enough. In short:

Htmltest is a good and incredibly fast tool. If you make its output readable and interpret it correctly, you can be quite happy with it. After a bit of experimentation, I arrived at the following command:

htmltest public/ 2>&1 | ansi2html > htmltest-output.html

Meaning: “Run htmltest in the /public/ folder. Redirect errors from stderr to stdout. Pipe the output to ansi2html¹. Write its output to a file named htmltest-output.html.”

That already looks much better!

Error categories

For my purposes, the errors found by htmltest can be sorted into a few categories:

Hugo-specific errors, such as “target does not exist” messages caused by livereload.js.
Incomplete links that work in practice but trigger “href lacks trailing slash” errors. These types of errors account for the lion’s share of the reported issues.
Lookup errors (GET <src> [...] failure in name resolution) caused by a lack of connection, insufficient permissions, active bot protection, or the target server being offline.
Hard “target does not exist” errors caused by broken links. These are the specific ones I actually wanted to fix.

Cleaning up the log

How do I ensure only category 4 errors are displayed? By making a few changes to the tools and resolving the trailing-slash errors.

Category 1 errors, triggered by Hugo’s useful livereload function, look like this:

target does not exist --- en/tags/tools/index.html --> /livereload.js?mindelay=10&v=2&port=1313&path=livereload

LiveReload is the feature that saves me from constantly hitting F5 in the browser window whenever Hugo builds the site after I save a Markdown file. However, I can temporarily disable it for the test. The following solution reduces the number of generated errors by one per HTML page:

hugo server --disableLiveReload

I avoid category 3 lookup errors by focusing on internal links, over which I have complete control. The -s, --skip-external parameter in htmltest helps with this:

htmltest -s public/ 2>&1 | ansi2html > htmltest-output.html
# [...]
✘✘✘ failed in 494.7678ms
446 errors in 337 documents

I can fix broken external links later. For now, I want to resolve the issues caused by the migration.

“Missing trailing slash”: Fixing errors via markup

I struggled for a long time with the many “href lacks trailing slash” errors, as I would have had to append a / to almost every Markdown link in every article. Fortunately, I found a solution in my render-link.html markup:

{{- /* /layouts/_markup/render-link.html */ -}}

{{- $url := .Destination | safeURL -}}
{{- $is_external := or (strings.HasPrefix $url "http://") (strings.HasPrefix $url "https://") -}}
{{- $is_anchor := strings.HasPrefix $url "#" -}}
{{- $is_asset := strings.HasPrefix $url "/assets/" -}}

[...]

{{- /* Add / to internal paths that have no file extension */ -}}
{{- if and (not $is_external) (not $is_anchor) (not $is_asset) -}}
 {{- $parsed := urls.Parse $url -}}
 {{- if and $parsed (not (strings.HasSuffix $parsed.Path "/")) (eq (path.Ext $parsed.Path) "") -}}
 {{- $url = printf "%s/" $url -}}
 {{- end -}}
{{- end -}}

[...]

Meaning: Internal links that point to neither headings (is_anchor) nor media files (is_asset) have a / appended if, and only if, they have valid syntax (parsed) and do not already end with a slash.

This resolves 95% of the error messages, resulting in cleaner HTML. I handle the remaining 5% by adding a few slashes in the Hugo configuration. Examples:

# hugo.yml
url: "/en/" # /en -> /en/
logoHomeLink: "/en/" # /en -> /en/

Done! All that remain are hard internal errors.

htmltest -s public/ 2>&1 | ansi2html > htmltest-output.html
# [...]
✘✘✘ failed in 359.28ms
102 errors in 84 documents

Manual Search and Fix

What follows is purely a matter of tedious legwork. I open the target document (the one being linked to) and check for inconsistencies, such as heading links that differ between languages. If I find any, I add specific anchor links to the heading.

### Heading {#heading3}

If everything is fine in the target file, I then open all the source files that use the broken link and correct it manually. In most cases, the target file has moved to a different folder, or the language prefix is incorrect.

The Result

After a few hours of exhausting, lab manual work, I am rewarded:

htmltest public/ -s 2>&1
Skipping the checking of external links.
htmltest started at 10:51:23 on public
========================================================================
✔✔✔ passed in 238.970354ms
tested 338 documents

Soon I’ll be able to go online with my new-old site!

ansi2html (or colorized-logs) is a utility that converts console output to HTML, including formatting and text colors. ↩︎

Hugo Multi-Lang: Internal Permalinks

Schallbert — Mon, 27 Jul 2026

Anyone building a multilingual blog with Hugo might be familiar with the following problem:

Internal links in articles across different languages point only to pages in the default language.

Configuration

The issue stems from the following setting:

# hugo.yml

defaultContentLanguage: "de"
defaultContentLanguageInSubdir: false
disableDefaultLanguageRedirect: true

My default language is German, and the corresponding articles are built into the blog’s root directory. No redirect to the /de language directory takes place.

The directory structure is as follows:

├── content
│ ├── de
│ │ ├── about.md
│ │ ├── announcements
│ │ ├── legal.md
│ │ ├── posts
│ │ ├── privacy.md
│ │ └── projects
│ └── en
│ ├── about.md
│ ├── legal.md
│ ├── posts
│ ├── privacy.md
│ └── projects

The Problem with the Default Language

I make articles in my default language accessible via URLs like example.com/my-post and example.com/my-2nd-post/ by using the slug parameter. In the secondary language, however, they appear as example.com/en/my-post and example.com/en/my-2nd-post/.

If I want to link two articles internally, the link always points to the article in my default language.

# English version of my post
# example.com/en/my-post
---
slug: my-post

---
Post's English content
[2nd](/my-2nd-post)

As a result, instead of the desired /en/my-2nd-post, I end up at the German version: /my-2nd-post. I find this annoying, as it would require me to manually adjust every link in every article. Unfortunately, Hugo does not seem to provide a built-in solution for this. So, I am creating a rule for how links should be rendered:

{{- /* /layouts/_markup/render-link.html */ -}}

Solution: Language-Dependent Link Prefixes

Here is what needs to happen: When a relative link within the site is encountered during the build process, a language prefix (such as /en) should be prepended to the link only if the language of the page being rendered is not the default language (since I do not need a language prefix for the default language, given that defaultContentLanguageInSubdir is set to false).

Retrieving the Default Language

But how do I access the default language? The defaultContentLanguage parameter is not accessible within templates. After some research, I found a suitable solution to the problem in the Hugo community: The default language is usually the first language in the weight-based index (which determines the order in which languages appear): index .Page.Site.Home.AllTranslations 0. When I access this, the “first” one is my default language.

Code for render-link.html

{{- /* /layouts/_markup/render-link.html */ -}}

{{- $url := .Destination | safeURL -}}
{{- $currentLang := .Page.Language.Locale -}}

{{- $is_external := or (strings.HasPrefix $url "http://") (strings.HasPrefix $url "https://") -}}
{{- $is_anchor := strings.HasPrefix $url "#" -}}
{{- $is_asset := strings.HasPrefix $url "/assets/" -}}
{{- $is_lang_prefixed := strings.HasPrefix $url (printf "/%s/" $currentLang) -}}

{{- if $is_anchor -}}
 {{- $url = printf "%s%s" .Page.RelPermalink $url -}}
{{- else -}}
 {{- if not $is_external -}}
 {{- if and (not $is_lang_prefixed) (not $is_asset) (not .Page.Language.IsDefault) -}}
 {{- $url = printf "/%s%s" $currentLang $url -}}
 {{- end -}}
 {{- end -}}
{{- end -}}

{{- /* Add / to internal paths that have no file extension */ -}}
{{- if and (not $is_external) (not $is_anchor) (not $is_asset) -}}
 {{- $parsed := urls.Parse $url -}}
 {{- if and $parsed (not (strings.HasSuffix $parsed.Path "/")) (eq (path.Ext $parsed.Path) "") -}}
 {{- $url = printf "%s/" $url -}}
 {{- end -}}
{{- end -}}

<a href="{{ $url | relURL }}"{{ if $is_external }} target="_blank" rel="noopener noreferrer" class="external-link"{{ end }}>
 {{- .Text | safeHTML -}}
 {{- if $is_external -}}
 <span class="external-link-icon" aria-hidden="true">↗</span>
 {{- end -}}
</a>

As long as the default language and the weight index match, meaning the default language appears at the very top, this works perfectly well.

Additional functions

Here is what else the code does:

Checks if links are “external” - that is, leading away from my site. These are assigned a separate window and the ↗ indicator.
Checks if the link points to a heading “anchor.” See below for details.
Checks if the link points to a (media) file or “asset.” Asset links are not modified. If it is a heading link, it is expanded to the full URL if necessary. This follows this pattern: #heading --> /<opt_lang>/<opt_category>/<title>/#heading

Correctly resolving permalinks to headings

I have also found a solution for highly specific permalinks within a page. Let’s assume I have headings in different languages:

---
# German
# content/de/posts/yyyy-mm-dd-hugo-create-language-specific-permalinks.md

description: Mit _markup interne Links richtig routen
title: "Hugo Multi-Lang: Interne Permalinks"
slug: hugo-create-language-specific-permalinks
---

## Richtig auflösende Permalinks auf Überschriften {#permalinks}
Lorem Ipsum...

---
# English
# /content/en/posts/yyyy-mm-dd-hugo-create-language-specific-permalinks.md

description: Routing internal links with _markup
title: "Hugo Multi-Lang: Internal Permalinks"
slug: hugo-create-language-specific-permalinks
---

## Correct Permalinks to headings {#permalinks}
Lorem Ipsum...

Then, clicking this link back to the #permalinks heading should take the user back to the heading, regardless of the language. The key to this solution is introducing a heading definition (Heading-ID) in Markdown: in this case, {#permalinks}.

It doesn’t work quite yet.

This is because #permalinks points to <blog's baseurl>/#permalinks, meaning the link to the current page is missing. But we can fix that, too: I’ll extend the link renderer with a few lines of code to prepend the current page URL.

Code for render-link.html with heading support

{{- $is_anchor := strings.HasPrefix $url "#" -}}

{{- if $is_anchor -}}
{{- $url = printf "%s%s" .Page.RelPermalink $url -}}
{{- end -}}

Now, (#permalinks) expands to /hugo-create-language-specific-permalinks/#permalinks, so the link finally leads to the correct destination.

SSH connection errors

Schallbert — Fri, 12 Jun 2026

This short article deals with errors encountered when establishing an SSH connection. Whether connecting to other computers, servers, or services like version control systems, the error messages and their corresponding solutions are usually the same. However, it is only worth reading further if standard help pages - such as GitHub’s SSH troubleshooting guide - have not provided the desired solution.

Problem: “No Route To Host”

The console output shows (on a local network):

➜ blog git:(article) ✗ ssh 192.168.0.27
ssh: connect to host 192.168.0.27 port 22: No route to host

Or (on a remote network):

➜ blog git:(article) ✗ ssh 1.2.3.4
ssh: connect to host 1.2.3.4 port 22: Network is unreachable

The cause is very likely an incorrectly entered IP address or a failed DNS lookup. On a local network, a common reason is that the target device is powered off, preventing the router from reaching it. The issue can usually be resolved by correcting the destination address, powering on the target device, or updating the routing table.

Problem: Connection Refused

Terminal output:

➜ blog git:(article) ✗ git push gitea
ssh: connect to host git.schallbert.de port 22: Connection refused
fatal: Could not read from remote repository.

In this case, a connection to the destination cannot be established at all; the process fails before authentication even begins. In my experience, such errors arise for three reasons:

The host is not yet ready. For example, it might be booting up (the network card is active), but the SSH service has not started. The usual solution here is simply to wait.
A firewall, rate limiter, or services like fail2ban reject the connection (REJECT). This is often caused by too many failed login attempts or an overambitious bot detection mechanism. In rare cases, your IP address might also appear on a “deny-list” loaded by the target system. In this instance, too, the best course of action is to wait.
The service in question expects a port other than the standard SSH Port 22. Consequently, the user, target address, and port do not match.

The port issue is easily resolved.

Determine the Port on the host (Server Configuration / SSH). The image shows the settings for my Gitea instance. Alternatively, simply checking the service’s configuration file is usually sufficient.
Specify the correct port number in the SSH configuration file at ~/.ssh/config. Example:

# /.ssh/config
 Host gitserver
 Hostname git.schallbert.de
 Port 222
 [...]

Problem: Too Many Authentication Failures

Terminal output:

➜ blog git:(article) ✗ git pull
Received disconnect from <ipAddress> port <portId>: Too many authentication failures

So, we have a connection to the server 😁, but it won’t let us in 😨. At least it tells us why: it claims we’ve tried too many keys at its front door.

This error occurs when the server’s limit for login attempts is exceeded. For the sshd service, the default value is, for example, MaxAuthTries = 6. If you have saved many keys for multiple connections - or, like me, have recently started using Hardware Security Keys and set up backup keys (effectively doubling the key count) - this limit is quickly reached.

The tricky thing about this error is that it only appears with specific services “further down” in the agent’s keychain: after all, five keys are allowed to fail before the limit is hit. Furthermore, having the keys correctly configured in ~/.ssh/config doesn’t protect you; the agent always attempts to use all available keys unless explicitly told otherwise.

We can prevent this by using IdentitiesOnly yes. This explicitly instructs the agent to use only the identities (i.e., key files or username/password) specified for that particular host.

# /.ssh/config
Host gitserver
Hostname git.schallbert.de
Port 222
User <user>
PreferredAuthentications publickey
IdentityFile <path_to_private_key1>
Identityfile <path_to_private_key2>
IdentitiesOnly yes

Checking the Git config

You can also encounter this error if the repository’s Git configuration (located at <reponame>/.git/config) does not match the server and repository details defined in your SSH configuration (~/.ssh/config). If you change the SSH configuration, all repository configurations maintained locally must also be updated.

Example:

# <reponame>/.git/config
# [...]
[remote "origin"]
	url = ssh://git@git.schallbert.de:222/schallbert/<reponame>.git
	fetch = +refs/heads/*:refs/remotes/origin/*
[branch "main"]
	remote = origin
	merge = refs/heads/main
	vscode-merge-base = origin/main

In this case, the gitserver configuration is not used at all; instead, the connection is established directly. Consequently, the agent does not receive the IdentitiesOnly yes flag and cycles through keys until the server rejects the connection. Solution:

[remote "origin"]
url = git@gitserver:schallbert/<reponame>.git
fetch = +refs/heads/*:refs/remotes/origin/*

Problem: Permission Denied (publickey,password)

Console output:

➜ blog git:(article) ✗ ssh -T git@gitserver
git@git.schallbert.de: Permission denied (publickey,password).

Unfortunately, this problem can have many different causes. GitHub’s guide addresses a few of them, including:

Using sudo
Wrong server
Not using the git user
Using the wrong key (client)
Key not registered (server)

The output for an incorrect username looks like this:

➜ blog git:(article) ✗ ssh -T mit@gitserver
mit@git.schallbert.de: Permission denied (publickey).

Another cause can be a mismatch between the ~/.ssh/config configuration file and the remote address stored in the git client. If they do not match, any existing configuration will not be used at all.

You can verify this using git remote -v.

➜ blog git:(article) ✗ git remote -v
origin	ssh://git@git.schallbert.de:222/schallbert/blog.git (fetch)
origin	ssh://git@git.schallbert.de:222/schallbert/blog.git (push)

The correct approach, using the improved configuration file (with ports explicitly set and identity files defined and restricted to the specified ones), looks like this:

# Update link to server's git repository, use host alias from config file
➜ blog git:(article) ✗ git remote update origin git@gitserver:schallbert/blog.git
# Now check if the update was effective
➜ blog git:(article) ✗ git remote -v
origin	git@gitserver:schallbert/blog.git (fetch)
origin	git@gitserver:schallbert/blog.git (push)

Problem: Agent Refused Operation

Console output:

➜ blog git:(local-setup) ✗ git push --set-upstream origin local-setup
sign_and_send_pubkey: signing failed for ED25519-SK <path-to-private-key> from agent: agent refused operation

I have only encountered this error since I started using hardware security keys for SSH. As I see it, there are two possible causes. Fortunately, both are easy to fix:

The agent does not recognize the key. The ssh-add command resolves this. Verify using ssh-add -l | grep "<your-key's-comment>"
The requested key is missing. This can happen with HSKs and HSMs if they are not plugged in, not positioned correctly on the NFC reader, or if the HSM is unavailable on the network. It can also occur if the ~/.ssh/config file was edited manually and the key has since been renamed or deleted.

Hardware Security Key: add SSH

Schallbert — Sun, 03 May 2026

Motivation

In my post about hardware security keys, I wrote that I want to secure as many of my connections as possible without passwords provided by my new HSK. I use SSH connections for various services and purposes: I access my VPS, use services such as GitHub, Gitea and my own Gitea instance and run backups on the local network via scp, which uses the SSH protocol.

Setting up SSH access for hardware security keys

HSKs such as my Yubikey can store private keys for SecureShell access. This means you no longer need to store them locally on every device (although this remains configurable; see below). Furthermore, they are better protected against unauthorised access than if stored on a computer.

So I’m trying to switch my GitHub SSH access to HSK and am following Yubico’s guide to passwordless authentication for GitHub. The procedure for generating FIDO-compliant SSH keys is the same across different key manufacturers. I’m also linking to the Nitrokey guide for reference.

What doesn’t work

However, even with Yubico’s own recommendations, I am unable to achieve the desired result on my operating system and with the standard installation of openssh. The command assembled from the guide is:

schallbert@machine:~# ssh-keygen -t ed25519-sk -O resident -O verify-required -O application=ssh:github -O user=schallbert -C 'schallbert@github.com'

Once generated, we must store the public key (file extension .pub) in the web application of the respective service under the SSH Access menu item.

Meaning of the command and the most important options

This command¹ tells ssh-keygen to generate a key of type -t ed25519-sk for me. The suffix -sk indicates that the key should be stored on a hardware security key - in my case, the Yubikey.

To understand the -O resident flag, we need to know that the FIDO authenticator key, generated as with -sk, consists of two parts: a key handle part, which remains on the computer’s hard drive, and the private key itself, which exists only on the HSM. Only together can they correctly respond to authentication requests. The flag itself, however, instructs the program to write both parts of the key to the Yubikey. This makes it easier to use multiple devices with the same key: the key handle part can be read from the Yubikey and stored on other end devices. On the other hand, however, simply losing the Yubikey is now enough to grant potential attackers SSH access.

-f filename specifies the names for the key files. This is particularly useful when you need to manage multiple SSH connections or HSMs. I’ve created a naming convention to avoid confusion. It’s based on the syntax used to read keys from an HSM (ssh-keygen -K).

-f <keyType>_<protocol>:<service>_<user>_<device>

As a concrete example, my filename here would be as follows:

-f ed25519-sk_ssh:github_schallbert_yubikey-main

⚠️ Note when using Windows: The colon : is not allowed as a character in filenames. It is replaced by an underscore _, which must also be reflected in the config file (Ctrl+H -> replace : with _)

-C schallbert@github.com adds a comment to the key file itself, such as a contact address. This gives the key a label, which makes it easier to manage when dealing with multiple keys.

-O application=ssh:github is simply for neat listing in the Authenticator and ensures that I cannot confuse one key with another. Otherwise, it would simply say ssh:keygen and I would no longer be able to see which services I have stored keys for. Also helpful to discriminate keys when using the -resident option in congunction with key extraction ssh-keygen -K.

-O user=schallbert has no practical use for me, as SSH is not interested in this flag and, contrary to my expectations, the Yubikey Authenticator does not display the username.

-O no-touch-required prevents the hardware security key from having to be touched every time the key is used. Useful for very frequent use, though it compromises security due to the possibility of automation.

-O verify-required specifies that the HSM’s PIN must be entered before the key can be used. Although intended as an additional security measure, this causes problems for me in my shell. Specifically, the PIN is requested once, but the ssh-agent does not generate a prompt in my console every time. As soon as I try to connect via SSH in another session, I receive an error message: agent refused operation:

schallbert@machine:~# ssh -T git@github.com
sign_and_send_pubkey: signing failed for ED25519-SK "schallbert@github.com" from agent: agent refused operation
root@opnsense: Permission denied (publickey).

What was the problem?

After a long, frustrating troubleshooting session (What am I doing wrong? Does it work with the secondary key? Why does it work the first time but not afterwards? What if I regenerate the keys? What happens on other computers? What does ‘AI’ say about this? - The latter certainly took an hour and left me completely stuck), I try a different command. That works straight away.

What does work, however

schallbert@machine:~# ssh-keygen -t ed25519-sk -O application=ssh:github -C 'schallbert@github.com' -f ed25519-sk_ssh:github_schallbert_yubikey-main

If I omit verify-required, I can log in successfully and the SSH agent runs fine.

What `verify-required` requires

This post sheds some light on my issue. The terminal session I use to create the SSH key pair temporarily stores the PIN entry enforced by the verify-required flag. That is why the login works initially. If I log in later or in a different window, as I did above to check git pull in one of my repos, for example, the SSH agent does not ask for my PIN.

This is likely because the SSH agent itself cannot display prompts to the requesting programme. A submodule of OpenSSH provides a solution: openssh-askpass . This can be passed to the agent, which then transmits the PIN prompt to the user via askpass.

Installing ssh-askpass

My operating system did not have ssh-askpass installed by default, so I’ll install it here. It uses the package manager DNF (‘Dandified YUM’) instead of apt or other common managers. In my opinion, the naming convention isn’t 100% consistent: whilst I address the ssh programme in the terminal as ssh-agent, I have to type openssh-agent for the installation.

# verify-required needs openssh subpackage through Fedora's package manager
schallbert@machine:~# sudo dnf install openssh-askpass
Installing:
 openssh-askpass
[...]
Complete!

Now I want to tell the agent that it can use ssh-askpass to request the PIN. To do this, I need to know the path to the submodule.

schallbert@machine:~# which openssh-askpass
/usr/bin/which: no openssh-askpass [...]

Strange, I installed it above! Let’s have a look what the package manager has to say.

schallbert@machine:~# dnf repoquery --list openssh-askpass
[...]
/usr/libexec/openssh/ssh-askpass

Ah, openssh hides in libexec. Well, then I can link ssh-agent to ssh-askpass.

schallbert@machine:~# eval "$(ssh-agent -s; SSH_ASKPASS=/usr/libexec/openssh/ssh-askpass)"
Agent pid 16747

I check the connection to GitHub and, as expected, a dialogue box appears asking for my PIN. I then need to tap the HSK, after which I receive a success message. To ensure that this all works in a new terminal and after restarting the computer, I add a reference to the key in my configuration file.

# ~/.ssh/config
Host github
 User git
 Hostname github.com
 PreferredAuthentications publickey
 IdentityFile ~/.ssh/ed25519-sk_ssh:github_schallbert_yubikey-main

To be on the safe side, restart the agent and create new terminal sessions. If any login details were cached, this will reset them. When done, validate SSH connectivity.

 schallbert@machine:~# eval "$(ssh-agent -k)" && eval "$(ssh-agent -s)"

Annoying: Gnome pop-up ‘Allow inhibiting shortcuts’

The only downside is that I now get an extra pop-up when connecting via SSH. This is triggered by the gnome graphical user interface. Unfortunately, I can’t find an option to save the relevant setting permanently. As a result, it appears every time I connect with verify-required. With Gnome, the problem is already known to affect other applications, but apparently has not yet been resolved.

Transferring to Windows systems

I generated my keys using the -O resident option. It makes sense to test the extraction of the key-handle on a completely different system: Windows.

I open the command line and type

schallbert@windows-machine:~# ssh-keygen -K
Enter PIN for authenticator:
You may need to touch your authenticator to authorize key download.
Provider "internal" returned failure -1
Unable to load resident keys: invalid format

Invalid format? It’s often mentioned online that the keys sometimes need a line feed character at the end to be recognized correctly. However, I cannot modify them without downloading them from HSK. So I manually transfer the public key and key handle to the computer and try out the git pull command as a test. And look: it works!

So if you want to use the keys with different operating system families, the -resident option doesn’t seem to be of any use. The command ssh-keygen -K does not allow granularity: It will load all resident keys from the HSK at once.

VPS: Be careful when replacing keys

Another tip from personal experience: If you’ve already stored SSH keys with a service provider that hosts live-resources itself (in my case, a server as a VPS), new SSH keys are usually not passed on to that resource via the web frontend.

Consequently, the “old” keys are still needed, and the new ones aren’t used, even though they’re stored in the web app!

The new keys must be manually added to the destination.

I follow the instructions from my host Hetzner using the ssh-copy-id command.

Manually store new keys on the VPS

schallbert@machine:~# ssh-copy-id -i ./ed25519-sk_ssh:server_schallbert_yubikey-main.pub server

Here, server is the SSH alias for my VPS. The connection is established using the familiar “old” key, and the “new” one is saved. But something isn’t working as expected:

/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: WARNING: All keys were skipped because they already exist on the remote system.
		(if you think this is a mistake, you may want to use -f option)

Was the key transferred successfully?

So I log back into my server and check whether the new key actually appears in the ~/.ssh/authorized_keys file. It doesn’t. So I run the command above again, this time with the -f (Force) option enabled. And sure enough, the transfer works!

schallbert@machine:~# ssh-copy-id -f -i ./ed25519-sk_ssh:server_schallbert_yubikey-main.pub server 
Number of key(s) added: 1

Now try logging into the machine, with: “ssh -i <identityFile> 'server'”
and check to make sure that only the key(s) you wanted were added.

To test it as recommended, log in with the new key and… It works! Only now can the old keys be safely deleted.

How to avoid ssh-copy-id

If you want to transmit the public key without using additional tools, you can copy the contents of the public key file <key-filename>.pub directly into a new line of ~/.ssh/authorized_keys. You just need an open SSH connection to your target.

Thoughts on convenience and usability

When I look at my SSH connections, they broadly fall into two categories:

Admin access (VPS, NAS, Raspberry Pi) and signatures
Protocol access (Git etc.)

In my view, the former require a higher level of security. Furthermore, I don’t use them as frequently as version control systems. I therefore recommend verify-required for the first category and would omit this for the second. I would take the same approach with the no-touch-required option: if a commit is to be uploaded every few minutes, constantly having to touch the key is rather annoying.

All options are displayed when you open the manual using man ssh-keygen. ↩︎

Hardware Security Keys

Schallbert — Sat, 25 Apr 2026

Definitions: TPM and HSK

Acronym	Term	Brief description
TPM	Trusted Platform Module	Soldered securely. Protects a device
HSK	Hardware Security Key	Mobile. Protects accounts, certificates, private keys

The Trusted Platform Module is a security chip that securely stores cryptographic keys and provides functions such as Secure Boot, BitLocker encryption and device identity. It is permanently soldered onto the motherboard of a computer, smartphone, etc.

A Hardware Security Key HSK, on the other hand, is usually portable and can be connected to various devices. It can not only store keys, but often also perform cryptographic operations (key generation, signing, encryption). HSKs are designed to be highly secure and tamper-proof. They can be externally certified or audited, meaning that even highly sensitive processes such as payment transactions can be carried out.

What happened so far

I listened to a podcast from the Bundesamt für Sicherheit in der Informationstechnik (BSI) on the topic of ‘Passkeys instead of passwords’. Following the recommendations, I bought a Hardware Security Key (HSK) so that I can use passkeys for my services in future. For a long time now, I have been unhappy with my fragmented secret management system consisting of notes, browser plugins, offline password managers such as KeePass, SSH credentials stored on my hard drive, backup keys and emergency codes from various providers.

I’d finally like to bring some order to the situation without tying myself so tightly to a commercial provider that I become dependent on them.

Objectives

No longer having to remember passwords or login details
Setting up a system for managing passwords that is as simple as possible, yet secure
Bringing order to the jumble of accounts
Organising my digital legacy
Enabling others to do the same

Welche Erwartung ich jetzt schon einmal nehmen kann: Ganz ohne Passwörter oder Pins geht es auch mit dem HSK nicht. Ein Yubikey ist kein Passwortmanager. Er kann zwar Zertifikate, Passkeys und Secrets für OTPs halten, gibt diese Geheimnisse dann aber nicht mehr preis. Mit seiner Hilfe kann man signieren (Authentizität beweisen), Verschlüsseln und abgeleitete Schlüssel erstellen.

In fact, password management can be centralised and simplified using the Yubikey. Popular password managers support passkeys meaning a master password is no longer required.

Yubikey

As an HSK, I use two Yubikey key fobs. One to carry with me at all times and one as a spare, which is kept in a safe place. There are other providers of mobile Hardware Security Keys, such as Nitrokey. I wasn’t aware of them before writing this article. That’s a shame, as their product is open-source and therefore an ideal candidate for testing on my blog 😥.

Yubico Authenticator

The Yubikey Authenticator GUI programme is useful for setting up the device, assigning functions to the Slot key (e.g. confirmation or password entry via keyboard simulation) and if you use 2FA-Einmalkennwörter z.B. als TOTP. Furthermore, my Yubikey supports PKI, meaning that SSL certificates and private keys can be stored securely.

Concept of the HSK

Part of the concept is that private keys are either generated directly on the hardware key or - without being permanently stored locally - are created on an end device but stored on the Yubikey. These remain on the key and cannot be exported. It supports common asymmetric methods such as RSA and ECDH. For further details, see this BSI publication on cryptographic methods. Furthermore, the key will be kept offline most of the time, i.e. without a power supply. Thus it is virtually impossible to compromise. An additional PIN as a protective measure in the event of key loss is a further security factor.

Spare key

Nevertheless, you should make a note of which access points you protect using which method, so that the correct accounts can be locked in the event of loss. A ‘spare key’ should be available and set up, which is almost as time-consuming as configuring the main key.

NFC

I’d hoped that my key’s NFC support would make logging in even quicker and easier. However, two things bother me here: I simply cannot get my laptop to recognise the Yubikey via NFC. And on my mobile, NFC recognition works at best flakily. When I hold the key up to it, the app launches and asks me to enter the PIN. If I do so, I have to remove the key and hold it up to the sensor again (‘Do not move the key!’). This doesn’t work every time by a long shot and seems rather prone to glitches overall. A pity.

What I find myself missing when using the Yubikey

Overall, I do miss a few convenience features in my day-to-day use of the Yubikey. When I click on the desired keys or accounts, nothing happens. And it seems I can’t configure anything there either.

When using passkeys, I’d like to see a shortcut. Clicking on the relevant key in the authenticator should open the corresponding app or browser window with the target website. Since I’ve already unlocked the HSK with the PIN at this point, the login process can even take place fully automatically once the key is tapped.

I’d like to see something similar when managing accounts. Why is the module called ‘Accounts’ if I can’t access them directly? Here, too, I have to open the relevant web apps etc. myself, have the password manager ready, and then copy the 2FA codes from the authenticator. There must be an easier way.

Touch, password, PIN, PUK, management key, passphrase

Depending on the module, multi-factor authentication is usually required even when using the Yubikey. The simplest method is presence detection via a touch of the dongle. This ensures that a person is actively operating the HSK and cannot, for example, be logged in automatically. Touch can be used across modules; for detailed configuration, see my article on SSH.

The Accounts module can optionally be secured with a password, whilst the Passkeys module is secured via PIN. Some providers even require a PIN to be set up as a second factor.

The Certificates module must be protected against unauthorised access using PIN, PUK and management key. Private keys or certificates can optionally be protected with a passphrase, which locks the keys themselves (and, in my view, makes no sense for the Yubikey, as the private keys cannot be read anyway).

Feature set and cryptographic landscape

Since the HSK promises such a significant security boost, I naturally want to make use of as many of its capabilities as possible. I’ve been playing around with the various modules for a few weeks now and have found that you really need to get your head around them and learn a lot to be able to use them all effectively. At least, once you’ve done so, you’ll have a rough idea of what the modules do and gain a good overview of internet security, encryption methods and protocols.

“Accounts” and “Passkeys” are easy to set up for input forms (platforms, websites, shops) and apps (email clients, password managers, messengers): Highly recommended! - schallbert

Yubikey module	Description
Accounts	Management of accounts/logins, supports TOTP
Passkeys	Passwordless login procedure via FIDO/WebAuthn (public key credential)
Certificates	Storage and use of certificates; supports OpenPGP (PGP) and X.509 (PIV)
Slots	Two programmable slots for generating static passwords or one-time passwords

In my opinion, things get complicated and time-consuming with public/private keys and certificates. Generation and validation procedures depend on the operating system or the programmes available on it, and instructions are sometimes insufficient or contradictory: only for nerds or professionals. - schallbert

Let’s go through the modules.

Two-factor authentication: “Yubikey: Accounts”

The Yubikey can be used as a generator for time-based one-time passwords. To do this, as with any other authenticator app, you create an account for each service and transfer a security key (secret) to the Yubikey. Once the HSK is connected, one-time codes can now be generated based on this secret.

As there is no way to export these secrets from the Yubikey, the process must be repeated for secondary keys. However, not every provider supports the use of multiple authenticators. So it may be sensible to keep the secret in the clipboard¹ and reuse it when configuring the secondary key(s).

Passkeys ‘Yubikey: Passkeys’

The Passkey module on my Yubikey works quite well. Unfortunately, many services² do not yet support passwordless login or only accept Passkeys as a second factor. With other services³, I cannot disable the password, so I see little security benefit, or I am issued with backup codes that are intended to serve as a second factor in an emergency⁴. At least, I can now bypass the password-TOTP process for some services and log in more quickly.

‘The biggest advantages of the technology for users are, on the one hand, that in future they will no longer have to create and manage passwords. On the other hand, passkeys are immune to the vast majority of known phishing attacks.’ - BSI website, 2025 October

You do not need the Authenticator app to configure passkeys. This is done entirely within the browser or the command line. Both must support the handling of passkeys.

Keys & Certificates ‘Yubikey: Certificates’

To shed some light on this, we first need to understand the difference between a key pair and a certificate in the context of authentication. In both cases, the aim is to verify the client’s identity. This may be done with a server (e.g. via SSH), an institution (e.g. a Certificate Authority) or another client (e.g. via PGP). This allows the destination to confirm that the client attempting to connect is indeed authorised to do so.

Public/Private Key Pair

A key pair alone (i.e. without a certificate) is typically used when the communication destination is controlled by the same party as the client. In such cases, the client can place the public key on the host to establish the connection. This method is used, when setting up SSH connections and for certain APIs, such as Google Search Console.

The host can send a test message to the cient encrypted with the public key, which can only be converted back into plain text using the corresponding private key. If the client sends the correct plain text back to the host, this proves to the host that the connection partner is authentic.

Certificates / PIV

But what if the host is not under the client’s control? How can we then ensure that neither the client nor the server is impersonating someone else and has stored a forged public key? This is precisely where certificates are used.

Put simply, the certificate contains not only the public key but also additional information about the network participant and the desired connection. Now a trusted third party, uninvolved in the communication between client and server, comes into play. In the Private Key Infrastructure PKI, this is known as the Certificate Authority CA.

The network participant now sends the public key, together with metadata about themselves, to the CA and submits a so-called CSR, a Certificate Signing Request. The CA then verifies whether the participant’s channel actually belongs to them.

In the case of an email (keyword PGP/GPG), the CA can, for example, send an individual hash link - derived from the requestor’s public key - to the specified email address. If the link is clicked, this proves to the CA that the email address actually belongs to the client, and the CA can then issue the certificate, i.e. sign the CSR with its private key.

For domains (keyword SSL/TLS), the CA can check with the Domain Name Service provider whether the details match the server’s request.

Clients have the signatures of common Certificate Authorities stored (e.g. in the browser). If the certificate is tampered with on the return journey between server and client, its hash value changes and the client displays a certificate warning: ‘invalid certificate’. However, one must now trust the Certificate Authorities⁵ 🙃

The computer should be offline for this. If you want to be extra secure, carry out the pairing on a live operating system. This usually boots from a USB stick and forgets everything after shutdown. ↩︎
E.g. Not at all: Mailbox; 2FA: Discord and Kraken, as of 2026 April. But perhaps I haven’t found the right settings? ↩︎
E.g. Google, GitHub and Hetzner, as of 2026 April. ↩︎
These backup codes must also be stored securely. Ideally offline again, but definitely independently of the HSK. ↩︎
There are alternatives to institutions such as CAs; see Web of Trust. For example, identity verification can also take place peer-to-peer via uninvolved third parties. ↩︎

act_runner rootless: no start

Schallbert — Sat, 28 Mar 2026

The error

My docker-in-docker (DinD) act_runner crashes shortly after starting with the following error message:

Issue ticket on Gitea

I have created a ticket for this issue in the Gitea community (issue #721). Below, I’ll go into my own findings and summarise the discussion a little.

Problem with kernel permissions?

In a Docker-in-Docker configuration, the act_runner must run its own Docker daemon. Only then can the runner create its own containers for the actions. For security reasons, this is not permitted by default.

Why containers are not allowed to start other containers

In the context of act_runner, an action executes code that is itself part of the repository. If malicious code is introduced into the job container unnoticed, e.g. by a ‘collaborator’, it can, in a Docker installation without DinD, under certain circumstances potentially gain direct access to the host system.

Consequences if something goes wrong despite DinD

When using the DinD concept, an attack can result in access to the dockerd process within the act_runner container - and only if the action is not properly encapsulated. Still not ideal, but acceptable: when the container is restarted, the status quo ante is restored. Access to the host system is indirect, as the container itself is equipped with kernel features.

Simple solution: Start the DinD container as `privileged`

To allow the DinD container to set up its own actions, privileged: true can be set in the configuration. This grants the container all kernel capabilities. According to the (excellent) OWASP Security Cheat Sheet, this should be avoided where possible. Should act_runner itself now come under attack or reveal critical security vulnerabilities, the intruder would already have every opportunity to bypass the encapsulation from the host system.

If you want to take the easy route, the corresponding docker-compose.yml file looks as follows.

# docker-compose.yml
# section ACT_RUNNER
runner:
 image: gitea/act_runner:latest-dind-rootless
 container_name: gitea-runner
 privileged: true
# [...]

“And remember: Do not run containers with the –privileged flag!!!” - OWASP’s Docker Security Cheat Sheet, 2026 License

Dead end: Only grant the rights that are absolutely necessary

After a quick search, I find an article on CodeStudy.net¹, which deals with the topic of DinD. From this, I put together some changes to my docker-compose.yml.

My approach:

Make changes to docker-compose.yml
Restart the service docker compose restart <service>
View logs docker logs --tail 100 <container-name>
If started: Run the programme web-app->repo->jobs->rerun_all-jobs
If it fails: Repeat

I start with few, but very powerful, permissions. After a few iterations, I get:

# docker-compose.yml
# section act_runner DinD-rootless
runner:
 image: gitea/act_runner:latest-dind-rootless
 container_name: gitea-runner
 privileged: false
 cap_add:
 - SYS_ADMIN
 security_opt:
 - no-new-privileges:true
 - apparmor:unconfined 
 - systempaths=unconfined
 - writable-cgroups=true
# [...]

But I’m still missing permissions that lie outside kernel privileges and security zones: the Docker daemon needs to be able to access sysfs and proc, i.e. system file directories and processes.

level=warning msg="[rootlesskit:child ] failed to mount sysfs, falling back to read-only mount: operation not permitted"

I cannot resolve this even with cap_add: ALL. Furthermore, online resources on this technical level are really scarce (and I am not a Docker specialist). A re-mount of the file systems might still help here; but that seems too experimental and error-prone to me.

After several more hours of research and trial and error on my server, I have to agree with @tianon in a discussion on GitHub: The Docker daemon requires so many permissions and capabilities that you might as well stick with privileged: true and avoid having to grapple with an armada of CAP_ADD and system bind mounts.

Future solution: Virtualisation or daemonless

However, there may be better solutions: dedicated container runtime environments such as Sysbox. The box itself has no special privileges on the host system; yet, much like in a virtual machine, it appears to be able to provide applications running within it with full access and capabilities.

Covering everything from installation and setup to fully functional job containers would take this post too far afield. Therefore, for the time being I must refer to other sources.

Switching from Docker to Podman is also a possible solution: Podman is daemonless, rootless and offers a similar range of features to Docker. Migrating to Podman would be a project in its own right for me and does not fit within the scope of this post.

Linux Security Modules (seccomp, AppArmor, SELinux)

Definition: LSM enable various security checks and restrictions at the kernel level. Through ‘Mandatory Access Control’, security extensions such as AppArmor can control kernel capabilities for individual applications and block access where necessary.

AppArmor

I run my server (VPS) on Ubuntu. In its more recent versions, this operating system has integrated ‘user namespace creation restrictions’ into AppArmor, which prevents appimages, WebApps and containers from running with elevated privileges.

“Unprivileged user namespaces are a feature in the Linux kernel […]; it enables unprivileged users to gain administrator (root) permissions within a confined environment […]” - mbelair, Ubuntu Discourse, as of May-2026, Ubuntu Discourse Website

In short, this tool was developed as a security-hardening measure to reduce the attack surface on the host system when running programmes that require elevated privileges.

Simple solution: Disable completely

The sledgehammer approach completely disables the feature for user namespaces. We tell AppArmor that third-party programmes may use kernel features or elevated privileges without restrictions, just as in older operating system versions. I found the relevant command on the AskUbuntu forum.

It disables restrictions for this session by overwriting the kernel parameters at runtime (-w).

schallbertTestsThis@machine:~# sudo sysctl -w kernel.apparmor_restrict_unprivileged_userns=0

I find the command very useful for troubleshooting. If you want to find out whether the desired programme is failing to start because of AppArmor:

Enter the command above
Test the third-party programme, the container, etc.
Reboot. Or enter the command with =1. This will restore the original state.

If you wish to retain this vulnerability permanently, enter:

makeVulnerability@machine:~# echo 'kernel.apparmor_restrict_unprivileged_userns = 0' | sudo tee /etc/sysctl.d/20-apparmor-donotrestrict.conf
makeVulnerability@machine:~# sudo shutdown -r now

This in itself does not constitute a security vulnerability. It has merely become easier, in principle, to exploit any weaknesses in the kernel and escape the container’s “sandbox”.

The correct solution: Tailor settings for each application

There are applications (such as my DinD version of act_runner) that absolutely require elevated privileges to function properly. And only these should be granted the ability to access non-admin user namespaces.

Docker itself has dedicated a section in its documentation to AppAmor. To arrive at the solution for the DinD runner, a bit of transfer (thanks, @thespad) is required. To recap, here is the error message from above:

gitea-runner | [rootlesskit:parent] error: failed to start the child: fork/exec /proc/self/exe: operation not permitted
gitea-runner | s6-svwait: fatal: some services reported permanent failure or their supervisor died

The log file indicates that the parent (Docker daemon in the container) cannot start its child (runner container) because it cannot create processes (/proc/self) for other participants (fork). Who is the user of this daemon? rootlesskit. This is exactly where our solution comes in: we need to enable the AppArmor profile for rootlesskit in the act_runner’s DinD container within the docker-compose.yml file.

# docker-compose.yml
# section act_runner DinD-rootless
runner:
 image: gitea/act_runner:latest-dind-rootless
 container_name: gitea-runner
 privileged: true
 security_opt:
 - apparmor=rootlesskit
 # [...]

The log file now looks fine. Done!

I get the feeling that post was written by an ‘AI’. When it gets specific and talks about ‘critical volume mounts’, there are no actual system paths listed, and the article becomes so vague overall that I can’t actually reach my goal by following the advice. ↩︎

Jekyll-dockerimage: Bundler and Gemfile

Schallbert — Fri, 14 Nov 2025

What is bundler?

Bundler is a tool that can be used to manage and version dependencies between modules and libraries for Ruby language. I use Bundler commands regularly, e.g. to build my blog and put it live on the server.

If I want to build locally, I type

bundle exec jekyll serve --incremental --future

into the console. This command tells Bundler to run the jekyll application in server mode and provides it with parameters that prevent jekyll from completely rebuilding every time a file is changed and still create pages that have not yet been published.

bundler in the CI pipeline

I also use Bundler as part of a Jekyll Docker image to publish my site, as I have often linked to, e.g. when moving to self-hosted. Since switching to a new version of the blog software lately, I have been seeing puzzling errors in my CI pipeline. Locally however, the system builds flawlessly. I have documented how to address and fix such errors here.

“There was an error while trying to write to /path/to/Gemfile.lock” - CI console output

No write permissions

Like the Jekyll user, Bundler itself can only read files on my CI and cannot write them. I can fix this for the Jekyll output with chown, but I don’t want to allow bundler to do this: I require Gemfile.lock to remain identical between my local build environment and the CI so that I can fix errors in advance.

Specification file for bundler

The Gemfile.lock file contains all dependencies of the application used, including version numbers and sources. If only the source code of the application plus this file is provided, bundler can pull a specific version of all dependencies during build time, allowing to work with similar requirements regardless of machine and version.

Troubleshooting

A first attempt to fix the issue by aligning all versions between local and CI failed. The second attempt, to give the Jekyll user write permissions to the Gemfile.lock, also failed. The third attempt led me to the Bundler website, where I took a closer look at the parameters, keyword “frozen”.

The fact is that you can prohibit bundler from rewriting the Gemfile.lock. To do this, use the command

bundle config set frozen true

According to its documentation, processing is aborted as soon as the file is about to be rewritten.

Although I had added this command to my yaml file in the CI, the build failed again. The trigger was the same library nokogiri as before, but the error message was now much more helpful:

Optimize `Gemfile.lock` for different environments

So I follow the suggestion and execute the desired command on my local machine:

bundle lock --add-platform x86_64_musl # my CI runner's environment

Note: musl is a libc-implementation for Linux.

My lockfile now has the following entry:

nokogiri (1.18.10-x86_64-linux-musl)

With this error fixed, I get better portability of my web page creation setup as a side effect.

fail2ban: Configuration error

Schallbert — Thu, 23 Oct 2025

Every few weeks, I usually check my server:

Is everything OK with Docker? docker ps
Are the backups being created correctly? borgmatic repo-info
Are my defenses working? iptables -n -L
Are my update automation daemons running? htop -F /server-config

This time, however, iptables showed a blank page on the DOCKER-USER chain. This is unusual and indicates that something is wrong with fail2ban.

schallbert@machine:~# iptables -n -L

Chain FORWARD (policy DROP)
target prot opt source destination 
DOCKER-USER all -- 0.0.0.0/0 0.0.0.0/0 
DOCKER-FORWARD all -- 0.0.0.0/0 0.0.0.0/0 

Chain DOCKER-USER (1 references)
target port opt source destination

The output reads something like

“The FORWARD chain forwards, as required by Docker, to the container-relevant DOCKER-USER chain. However, no applications are listed under this chain that can reject or prohibit packets from being forwarded (REJECT/DROP).”

Two entries for caddy-server and caddy-ratelimit should appear here.

“fail2ban ERROR Failed during configuration”

So I first check Docker:

docker logs -t fail2ban # Print logs for the container named fail2ban with a timestamp

Instead of being rewarded with <timestamp> Server ready after scrolling through as usual, I see a never-ending flood of log messages. They repeat themselves and show that fail2ban is exiting with Exit with code 255. I get a better overview of the log if I only print the last 100 lines:

docker logs -t --tail 100 fail2ban # return latest 100 lines of log with timestamp

The error message

<timestamp, ID> ERROR Failed during configuration: Bad value substitution:
option 'action' in section 'caddy-ratelimit' contains an interpolation key 'banaction' which is not a valid option name.
Raw value: '%(action_)s'

This message is interesting because I had problems with jail.local at exactly this point when I was creating the article Fail2ban with Caddy. At the time, I had to remove action = iptables-multiport from the configuration because the triggering IP addresses had landed on the wrong CHAIN (INPUT).

Now, the default action on the DOCKER-USER chain seems to no longer work. So, I’m searching a few sources for a possible solution. However, I can’t find any references to updates that would cause major changes in the action.d folder either in the update history of fail2ban or in the linuxserver.io distribution I’m using.

Solution via `banaction = iptables[type=multiport]`

So I tried the web search and found a comment from a fail2ban maintainer:

“[…] With action = iptables[type=multiport, protocol=tcp, chain=DOCKER-USER] you overwrote the default action […] [Solution:] simply set the chain variable only in the jail […]: port = http, https banaction = iptables[type=multiport] chain = DOCKER-USER” - sebres, Dec-2024

The basis for this statement can be found in the Docker Docs. Side note: In all places, it is recommended to use the more modern nftables instead of iptables for future applications. Here is an article on the topic.

This allows me to completely delete the action field that was previously present in my jail.local and use banaction instead. It now looks like this:

# jail.local update Oct-2025

# Server/Client errors: 401 403 404 500
[caddy-status]
enabled = true
chain = DOCKER-USER
port = http,https
filter = caddy-status
logpath = <path/to/logfiles>
findtime = 10min
maxretry = 5
bantime = 1d
banaction = iptables[type=multiport]

With this configuration, fail2ban boots normally and displays Server ready.

Verifying the solution using iptables

As described above, I create entries in the test log files that should trigger the actions of caddy-status and caddy-ratelimit. As expected, I get the following in the “blocklist”:

schallbert@machine:~# iptables -n -L

Chain DOCKER-USER (1 references)
target prot opt source destination 
f2b-caddy-ratelimit tcp -- 0.0.0.0/0 0.0.0.0/0 multiport dports 80,443
f2b-caddy-status tcp -- 0.0.0.0/0 0.0.0.0/0 multiport dports 80,443

passtdas - a helper for your CNC

Schallbert — Mon, 08 Sep 2025

The Problem

A friend of mine wrote to me. He uses a CNC gantry milling machine as a hobby, just like me, and often produces unique pieces. However, the resulting leftovers are sometimes too valuable to throw away. To maximize material utilization, he tries to use these offcuts as blanks for future projects.

“Do you know how I can find out on the machine whether my milled part fits on the blank?” - Anonymous

This happens: According to the dimensions, the workpiece just fits on the blank. If you set the zero point even slightly incorrectly, the milling cutter will move beyond the workpiece boundaries, and scrap it is.

The Solution

It would be great if the machine followed the contour of the workpiece before starting the milling process, wouldn’t it? This way, you could easily find out whether the blank is actually completely within the workpiece boundaries.

Since I can’t find a quick solution online right away, I’m thinking about writing my own software to implement this. So I’m getting to work.

TODO: ADD VIDEO LINK

The Software

For a quick prototype, I choose the high-level language python. Why? I have a fairly good command of python, libraries for the necessary geometric calculations are available, and the written program can be easily compiled into an all-in-one file.

The program is designed to read machine instructions (G-code) and analyze the milling paths. The positions at which the maximum and minimum X and Y values are approached in the milling program are to be saved and output as a motion command in a path file.

If this program is loaded onto the machine, an “extreme value” determined in this way is approached and the Z-axis is lowered from the safety height to the measuring height. The program is then paused so that the operator can make adjustments to the workpiece position. Only by entering another start command will the next extreme value be reached, and so on.

This program will be called passtdas.

Program type

I’m choosing a console program with few parameters:

Shortcut	Parameter name	Description
`-f`	`--file`	The source file to be analyzed. Required.
`-s`	`--zsafety`	Safety height the machine will do flyovers with. Optional.
`-p`	`--zprobe`	Probe height the machine will pinpoint to. Optional.

The two height parameters are given default values, so they don’t always have to be specified.

Input

passtdas reads files and searches them line by line for motion commands like G00, G01, G02, G03. While the analysis is simple for linear movements like G00, G01 - linear movements can be expressed using point-to-point connections whose extreme values cannot lie between the points - it can be quite challenging for circular segments.

The height information on the Z-axis is also read and saved. This allows the machining technician to quickly check whether the maximum milling depth matches the desired one.

If the G-code file contains no instructions or incorrectly formulated instructions, passtdas displays corresponding error messages and aborts the analysis.

Definition of Circular Segments

Background: Circular segments can be defined in two ways in G-code.

[...]
G00 X5.660 Y-0.000 Z10.000
G02 X-5.660 Y0.000 R5.660 F1600
[...]

Command:

“Move a circular arc clockwise from point X5.66 Y0 with radius 5.66mm to point X-5.66 Y0 with feed rate 1600mm/min at height 10mm.”

Drawing these commands creates a hanging semicircle. Circles can be defined using three points. However, only two points are given. The third point must be calculated from the specified arc radius using the start and end points.

Task: Find the extreme points of this arc in the XY plane.

But an arc can also be defined like this:

[...]
G00 X-2.5696 Y5.0431
G03 X-4.6258 Y3.2616 Z-0.5000 I2.5696 J-5.0431
[...]

Command:

“Traverse a circular arc counterclockwise from point X-2.56 Y5.04 with center coordinates I2.56 J-5.04 to point X-4.62 Y3.26.”

Drawing these commands creates a short circular segment in the upper left quadrant. Although three points (start and end points, as well as the center of the circle) are given, it can sometimes be difficult to determine whether an extreme value is being reached somewhere along the arc.

Task: Find the extreme points of this circle in the XY range.

Calculating the extreme points of circle segments

A circle segment can have up to four additional extreme points, depending on its range: X+, Y+, X-, Y-. For a full circle, these are always the coordinate intersection points as seen from the radius.

def get_extremes_from_arc(arc, coordinates):
 # [...]
 x_plus_radius = [center_x + radius, center_y, xyz[2]]
 y_plus_radius = [center_x, center_y + radius, xyz[2]]
 x_minus_radius = [center_x - radius, center_y, xyz[2]]
 y_minus_radius = [center_x, center_y - radius, xyz[2]]
 extremevalue_order = [x_plus_radius, y_plus_radius, x_minus_radius, y_minus_radius]
 if (arc[1] - arc[0]) < 0:
 # crossing the 0° line (x-axis), handle overflow with nested if below
 arc[1] += 360
 result = []
 for crossing_angle in range(0, 721, 90):
 if crossing_angle in range(int(arc[0]), int(arc[1] + 1)):
 i = int(crossing_angle / 90)
 if i > 3:
 i -= 4
 result.append(extremevalue_order[i])
 # [...]

If the circle center and radius are known, the swept angle can be determined from the start and end points. This makes it clear whether the circle’s start and/or end points themselves represent extreme values, or whether additional extreme values arise from sweeping over a coordinate origin. It is precisely what the above code example does.

Data table

All possible extreme values (two for a line, up to four for a circle segment) are now collected in a data table. Once each G-code line has been evaluated and the list is thus complete, the maximum and minimum values for each axis are determined from this list.

This ultimately creates four XYZ coordinates, which describe the maximum extent of the workpiece for each spatial direction.

At the same time, the program searches all commands to the Z-axis and remembers the maximum plunge depth.

Output

The command-line program has no visual output. Therefore, I’ll limit myself to creating a simulation program for the milling machine in which the workpiece extension is approached and paused at each point:

G90

# Go to coordinate zero
MSG "Zmin of this job: -1.0"
G00 Z40
G00 X0 Y0
G01 Z15 F1200

MSG "PathPreview: Hit START to go to Ymin: ['0.0', '-5.66']"
M00
G00 Z40
G00 X0.0 Y-5.66
G01 Z15 F1200

MSG "PathPreview: Hit START to go to Xmin: ['-5.66', '0.0']"
M00
G00 Z40
G00 X-5.66 Y0.0
G01 Z15 F1200

MSG "PathPreview: Hit START to go to Ymax: ['0.0', '5.66']"
M00
G00 Z40
G00 X0.0 Y5.66
G01 Z15 F1200

MSG "PathPreview: Hit START to go to Xmax: ['5.66', '0.0']"
M00
G00 Z40
G00 X5.66 Y0.0
G01 Z15 F1200
# [...]

The above example traverses the extreme values of a circle with r=5.66mm. It always moves to a safety height of Z40 to cover the points and, upon arrival, slowly moves the Z axis at F1200 to the target height of Z15. It is important that the zero point in Z has been correctly measured beforehand - otherwise, a collision with the workpiece may occur.

Testing and Verification

To see if the program works as expected, I created three test files:

Unit tests: Tests individual functions in the program, e.g., whether line and circle segments are correctly recognized
Integration tests: Tests whether functions function correctly in conjunction, e.g., the correct calculation of extreme values based on input values
End-to-end tests: These tests check the entire function chain, e.g., whether incorrectly formatted input files are recognized

Finally, I create several test files with the program and move to the extreme values generated from them with my CNC.

Open-source software

The source code for passtdas is available for free on GitHub, but must be interpreted using Python.

I’ll build a user-friendly and easy-to-use program once enough people have expressed a desire for it.

Rootless act_runner for Gitea

Schallbert — Sun, 10 Aug 2025

In this article, I’ll show how to change the act_runner of my Gitea instance from gitea/act_runner to gitea/act_runner:latest-dind-rootless.

Why all this effort? Because act_runner runs in Docker and requires access to the daemon via /var/run/docker.sock as a so-called volume (meaning a disk) or bind mount to function. The owner of this socket is root, which gives the container virtually full access to the host system.

“RULE #1 - Do not expose the Docker daemon socket (even to the containers)” - OWASP / Docker Security Cheat Sheet

This is highly risky, because it’s inherent in the runner’s design to execute job code generated by third parties. It’s an open gateway. In conjunction with direct access to the host system, this could result in a total failure or hostile takeover of my infrastructure in the event of a successful attack.

Danger from `docker.sock`

In this guide, I follow the OWASP’s recommendation on Docker security and apply its rule(s) in practice. OWASP stands for “Open Web Application Security Project.” It is an organization dedicated to improving the security of web applications and supporting users like me with free articles, documentation, and technology.

The measures described under Rule 1 are:

Leave Docker’s tcp socket disabled

If access to the Docker Daemon is enabled via tcp, it can be connected to via an unsecured connection and without authentication - unless further precautions have been taken. The daemon is then accessible to virtually any internet user and thus vulnerable.

So how do I ensure that the tcp socket is disabled? The instructions for this can be found in the Docker Docs and must be applied in reverse. There, too, there is an explicit warning against opening the tcp socket unprotected.

After completing all the necessary steps, I run netstat once. It lists open sockets, network interfaces, and routing tables:

# check if Docker Daemon's "dockerd" tcp socket is exposed
schallbert@machine:~# netstat -lntp | grep dockerd
# test is a pass if this command does not return anything.

I don’t find any entry with dockerd. My server isn’t vulnerable at this point.

Do not include the Docker socket `/var/run/docker.sock` in other containers

This is where things get a bit more complicated: act_runner needs the socket to create, manage, and ultimately dispose of job containers. Without access to the Docker Daemon via the socket, the build pipeline simply won’t work - unless you want to forgo Docker entirely and run both runner and the build jobs directly on the host machine. This comes with many disadvantages: loss of encapsulation, lack of portability, poor scalability, reduced security…

But there’s also a solution for the Docker option: Setting up act_runner Docker-in-Docker. With this setup, act_runner receives its own Docker Daemon, but with limited permissions and without access to the host system. This then takes over the lifecycle of job containers so that they run completely independently of the host system.

Docker in Docker

The following diagram illustrates the difference:

Starting Act_runner “rootless”

So I follow the instructions and copy together a suitable docker-compose.yml. Important here:

privileged: true must be set. Otherwise, the Docker daemon in the act_runner container cannot start properly because it lacks kernel functions. This causes the entire container to crash repeatedly without generating any helpful error messages.
The environment variable DOCKER_HOST=unix:///var/run/user/1000/docker.sock must be set. Here, the Docker socket is controlled by a non-privileged user and is available to the runner for managing job containers. The daemon runs encapsulated in the container and is not connected to the host machine.

As of Jul-2025: Problem with 0.2.12-dind-rootless

This is where I encountered my first problem. The runner crashes shortly after starting with the following error message:

schallbert@machine:~# docker logs gitea-runner
# [...]
[rootlesskit:parent] error: failed to start the child: fork/exec /proc/self/exe: operation not permitted

Help came from the Gitea community. act_runner runs smoothly with the previous version 0.2.11. However, for reasons unknown to me, it is displayed on Gitea as 0.2.12.

Volume Confusion

Great! Now that the runner is working, I’ll let it run a job right away. Unfortunately, the build fails after just a fraction of a second with this message:

# Runner step: Set up job
failed to start container: Error response from daemon: error while creating mount source path '<volumeSourceFullPath>': mkdir <volumeSourcePath>: permission denied

I had to research this for hours and was under the false impression for a long time that it was due to insufficient permissions on the folders on the host machine. Only later did I truly understand that Docker-in-Docker means exactly what it says: Not only are containers created by containers, but a separate Docker daemon runs within the container!

This means that the classic method of making volumes in job containers available directly from the host system using -v /a/b:/x/y no longer works. Instead, volumes must now be passed through. Example:

host directory /opt/server/www/blog-artifacts -> act_runner volume /tmp/blog-artifacts –> job container volume /tmp/blog-artifacts

The :z in the act_runner volume is now important. It indicates to Docker that this volume is shared between containers. These volumes must not only be specified in act_runner’s docker-compose.yml, but the job scripts in the .gitea/workflows/ folder must also be adjusted accordingly. However, the :z on the “right side” is not needed here.

“is not a valid volume”

Despite all my efforts, my jobs still aren’t running. This time, because of an error message that already seemed familiar:

# Runner step: Set up job
[/tmp/blog-artifacts] is not a valid volume, will be ignored

So, go into the config.yml of act_runner and add the volume names:

# /gitea/runner/config.yml
# [...]
valid_volumes: ["/tmp/blog-artifacts", "/tmp/lectures-artifacts"]
# [...]

In this file, I can leave privileged: false because, unlike act_runner, the job container doesn’t require kernel features.

Set permissions correctly

Now I’m getting Permission Denied error messages again when running my jobs, although not directly in the first step of the actions. Since I’ve now checked the volume paths down to the last detail, it can only be due to the folder permissions on the host machine.

In order for the artifacts created by the job container to be stored via the volumes on my host, I have to pass the directory to be written and all subfolders -R to the previously defined, non-privileged user ID=1000:

schallbert@machine:~# chown -R 1000:1000 /target/path/to/artifact/

Finally, everything is working smoothly, and I’ve put a stop to the (unlikely, but possible) takeover of my host system by malicious job containers.

Add rate limiter to Gitea

Schallbert — Fri, 25 Jul 2025

Here, I’m providing a step-by-step guide to solving my Gitea crashes problem. I’m incorporating the experience I gained from using a rate limiter for my blog with the goal of making the application more robust and protected against undirected denial of service attacks.

Step 1: Specify the log source for the rate limiter

Gitea naturally generates logs itself, theoretically down to the access level by external clients. However, in the past, I haven’t been able to export Gitea’s access logs from Docker. Therefore, we now instruct Caddyserver, which is connected as a reverse proxy, to create logs on behalf of Gitea. Access to the Gitea web interface will then appear in these logs.

In the Caddyfile, the log module looks completely unspectacular.

# /caddy/Caddyfile

#[...]
log {
output file /log/gitea/access.log
}

Exclude internal traffic from the log

Now I look at the logs and see entries that I don’t want. For example, act_runner generates a Fetch Task POST to Gitea every two seconds and a GET request every ten seconds with the same target as a Health check. These don’t even need to appear in the log for me. My first thought here was to simply not log requests from internal IP addresses. However, since Caddy acts as a reverse proxy, all IP addresses run “internally” without exception.

So I have to recognize act_runner-specific logs differently:

# /caddy/Caddyfile

#[...]
git.schallbert.de {

 reverse_proxy * http://gitea:3000

 # Enable logging for fail2ban, don't log for runner
 log_skip /api/actions*
 log {

 output file /log/gitea/access.log
 }
}

The log_skip directive now instructs caddy to no longer create logs for access to /api/actions*.

Reverse Proxy: Displaying Remote IPs

However, I still have a problem: If all requests in the log come from an internal IP address, how am I supposed to block “bad” requests? A web search shows that this is a common problem for reverse proxies in Docker. Unfortunately, many solutions cannot be applied to this situation because they use different server software like nginx or have different services running behind their proxies than Gitea.

But it’s actually quite simple: Just insert a line in the correct place in the Caddyfile, and the remote IP addresses come in unchanged.

# /caddy/Caddyfile

#[...]
git.schallbert.de {
 reverse_proxy * http://gitea:3000 {
 trusted_proxies 172.16.0.0/12 # Docker-internal netwock traffic runs with these IPs
 }
#[...]

The trusted_proxies directive tells Caddy that the network address range provided by Docker can be trusted. Now the actual source IP address is shown instead of the container’s internal interface address. This is exactly what I need to be able to analyze the addresses later with fail2ban.

Step 2: Determine the Use Case

Let’s look at how many HTTP 200 ok requests are coming in the edge case between normal use and “abuse.” To do this, I surf around Gitea and click on a bunch of things that I would never normally do at that speed as a human. Then I analyze the logs.

This gives us some initial benchmarks for the rate limiter.

# rate limiter tests
findtime = 10s
maxretry = 10
bantime = 6h

Step 3: Configure Fail2ban

Next, we configure the filter and jail file of fail2ban to analyze the logs defined above.

Filter

Here, I’m re-using the same filter for my rate limiter from the previous article. We’re only interested in successful requests that we count within a time window.

Jail

I’m using the values from the rate limiter tests 1:1 in the jail file. I’m referring to caddy-ratelimit (link above) as the filter. It is essential to select the DOCKER-USER chain as the requests are routed via Docker’s virtual network.

# /fail2ban/config/fail2ban/jail.local
# [...]

[gitea-ratelimit]
enabled = true
chain = DOCKER-USER
port = http,https
filter = caddy-ratelimit
logpath = /var/log/caddy2/gitea/access.log
findtime = 10s
maxretry = 10
bantime = 6h

I reboot fail2ban to activate the protection.

Step 4: Test the rate limiter

I proceed exactly as in my article fail2ban with caddy and get the same result. It works!

Rate limiter with Caddy and fail2ban

Schallbert — Thu, 10 Jul 2025

Here I describe how to configure fail2ban (commonly used to defend against unauthorized access attempts) to limit the number of successful accesses within a defined time window. This is called “rate limiting” and is intended to thwart denial of service attacks that overload my server.

Enabling Caddy Logs

First, we need to enable the web server’s logger. With Caddy, this can be done with just a few lines of code that I add to my existing configuration:

# caddy2/config/Caddyfile

blog.schallbert.de {
# Define webserver
root * /www/blog
encode gzip
file_server

# Enable logging for fail2ban
log {
output file /log/blog/access.log
}

# [...]

From now on, logs will be created in the specified path, which I manage with a logrotate configuration.

Configuring Fail2ban as a Rate Limiter

To recap: fail2ban accesses the machine’s packet filter rules and, in effect, modifies its firewall to ward off dynamic attacks. For this to work, fail2ban must be provided with log files generated by a host application, such as a web server, that contain the IP addresses of the client computers.

Preliminary consideration: “http 200 OK” as a filter?

Even normal, permitted accesses 200 OK must count for my rate limiter.

To get a feel for how my logger records accesses, I take a look at the logs. I get the number of entries that contain a status of 200 OK returned. To do this, I perform a search with grep and calculate the number of hits line by line using wc -l.

schallbert@machine:/var/log/caddy2/blog# grep -o '"status":200' access.log | wc -l
847

Okay, so there have already been 847 hits today, which received a 200 OK response. Now I’ll call up my blog’s landing page and display the article overview.

schallbert@machine:/var/log/caddy2/blog# grep -o '"status":200' access.log | wc -l
858 // after calling blog.schallbert.de
schallbert@machine:/var/log/caddy2/blog# grep -o '"status":200' access.log | wc -l
882 // page loaded after clicking "Posts" button

Wow, that was 35 entries for two clicks! Looking at the log, it explains the many lines of assets—images and logos—that are being loaded.

Unfortunately, this makes it clear that the filter criterion status:200 for my rate limiter can’t work without further ado. The number of notifications depends largely on the respective article. So, I would have to define a threshold above which no normal person would generate access to my blog.

In order to count only “real” accesses to my pages, I have to somehow exclude the assets from the logs. Fortunately, there’s a simple directive in Caddy for this.

# caddy2/config/Caddyfile

 # Enable logging for fail2ban
 log_skip /assets*
 log {
 output file /log/blog/access.log
 }

log_skip now ensures that all accesses to files in the assets/ folder and below are not logged.

The behavior of Fail2ban is defined using two configuration files:

Directory `filter.d`

filter.d/<filter-name>.conf contains the definition of an event to be monitored. Thus, the failregex “page not found” http 404 status code can be used to trigger the packet filter, just as successful access can be used to set up my rate limiter, which responds to 200 OK. I copied the regex from Rafael Kassner.

# /opt/fail2ban/config/filter.d/caddy-ratelimit.conf

[Definition]
failregex = "client_ip":"<HOST>"(.*)"status":200
datepattern = \d+
ignoreregex =

In addition, further boundary conditions are defined in the file, e.g., the date format datepattern is adapted to the log output of the system to be protected.

`jail.local` file

jail.local determines the conditions under which the packet filter for the client IP becomes active and blocks further access attempts. For use as a rate limiter, I need the following fields:

findtime - the time window in which attacks are counted, maxretry is the number of permissible attempts in the time window, and bantime is the time for which the attack is blocked. ignoreip is usually preconfigured to the relevant internal IP addresses by default. In my case, act_runner, for example, sends a message to gitea internally to query whether any new automation tasks are pending. I definitely don’t want to interfere with these accesses.

The example for fail2ban used below shows the rate limiter on my blog. In the end, the files and log entries for gitea are essentially the same. Only the filter names and parameters differ due to the different requirements of the website and the DevOps platform.

# /opt/fail2ban/jail.local
# ...

[caddy-ratelimit]
enabled = true
chain = DOCKER-USER
action = iptables-multiport
port = http,https
filter = caddy-ratelimit
logpath = /var/log/caddy2/blog/access.log
findtime = 20s
maxretry = 10
bantime = 6h

Once these two files are configured, you can restart fail2ban and view the logs. If there are still errors in the files, fail2ban will output the following:

ERROR Errors in jail 'caddy-ratelimit'. Skipping...

If this occurs, for example, the jail cannot be assigned to the filter (must have the same name!), a field definition such as maxretry is incorrectly typed, or a number format cannot be read. If everything is working correctly, the entry will be:

Reading config files: /etc/fail2ban/filter.d/caddy-ratelimit.conf

Testing the Rate Limiter

Now I want to check if everything is working as expected.

Checking the Filter Regex

The first step is to look at the filter term and check whether it can be reliably found in the logs. Conveniently, fail2ban has a built-in tool for this: fail2ban-regex <logfile> <filter>. So I can simply enter the filter file and a test log file and see if I get any matches.

fail2ban-regex ./access.log /etc/fail2ban/filter.d/caddy-ratelimit.conf 

Running tests
=============

Use filter file : caddy-ratelimit, basedir: /etc/fail2ban
Use datepattern : \d+ : \d+
Use log file : ./access.log.2
Use encoding : UTF-8

Results
=======

Failregex: 2009 total
|- #) [# of hits] regular expression
| 1) [2009] "client_ip":"<HOST>"(.*)"status":200
`-

Ignoreregex: 0 total

Date template hits:
|- [# of hits] date format
| [2685] \d+
`-

Lines: 2685 lines, 0 ignored, 2009 matched, 676 missed
[processed in 0.12 sec]

Quantitatively, I get what I expect: the search procedure described above using grep gives me exactly the same results.

Check Jail

The next step is to test whether the rate limiter is working. To do this, I simulate log entries. I significantly ease jail rules for this test, otherwise I’ll have to generate too many entries in a short period of time.

# /opt/fail2ban/jail.local TEST
# ...

[...]
findtime = 10s
maxretry = 2
bantime = 100s

After restarting fail2ban, I create a second console where I can simulate log entries. This must include at least the time, IP address of the caller, and status; however, I play it safe and use complete entries.

echo '{"ts":1751246139,"remote_ip":"1.1.1.1","status":200,[superLongIrrelevantOtherStuffForFiltering]}' >> /var/log/caddy/blog/access.log

I enter this multiple times to exceed my limit of 2 retries.

In the first console instance, I then check the fail2ban status for the corresponding filter.

fail2ban-client status caddy-ratelimit
Status for the jail: caddy-ratelimit
|- Filter
| |- Currently failed: 0
| |- Total failed: 0
| `- File list: /var/log/caddy2/blog/access.log
`- Actions
|- Currently banned: 0
|- Total banned: 0
`- Banned IP list:

That was nothing. The fail2ban log file reveals why:

# /fail2ban/config/log/fail2ban/fail2ban.log
WARN [caddy-ratelimit] Ignoring all log entries older than 20s; # probably messages generated within a fail2ban restart period

So again with appropriate timestamps. I have to get these from the current system time ($(date +%s.%N)) if I have to constantly adjust it. Therefore, I take a complete log line, modify it to the IP address 1.1.1.1, and insert appropriate timestamps:

 echo "{\"level\":\"info\",\"ts\":$(date +%s.%N),\"logger\":\"http.log.access.log0\",\"msg\":\"handled request\",\"request\":{\"remote_ip\":\"1.1.1.1\",\"remote_port\":\"38229\",\"client_ip\":\"1.1.1.1\",\"proto\":\"HTTP/1.1\",\"method\":\"GET\",\"host\":\"blog.schallbert.de\",\"uri\":\"/vacuum-clamping/\",\"headers\":{\"Accept-Encoding\":[\"gzip, deflate, br\"],\"Connection\":[\"keep-alive\"],\"User-Agent\":[\"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36 Edg/121.0.0.0\"],\"Accept\":[\"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8\"],\"Accept-Language\":[\"en-US,en;q=0.5\"]},\"tls\":{\"resumed\":false,\"version\":772,\"cipher_suite\":4865,\"proto\":\"http/1.1\",\"server_name\":\"blog.schallbert.de\"}},\"bytes_read\":0,\"user_id\":\"\",\"duration\":0.001207354,\"size\":12290,\"status\":200,\"resp_headers\":{\"Last-Modified\":[\"Tue, 10 Jun 2025 19:33:01 GMT\"],\"Content-Encoding\":[\"gzip\"],\"Server\":[\"Caddy\"],\"Alt-Svc\":[\"h3=\\\":443\\\"; ma=2592000\"],\"Vary\":[\"Accept-Encoding\"],\"Etag\":[\"\\\"gzip\\\"\"],\"Content-Type\":[\"text/html; charset=utf-8\"]}}" >> access.log

After calling this command several times, I now get

Status for the jail: caddy-blog-ratelimit
|- Filter
| |- Currently failed:	1
| |- Total failed:	11
| `- File list:	/var/log/caddy2/blog/access.log
`- Actions
 |- Currently banned:	1
 |- Total banned:	4
 `- Banned IP list:	1.1.1.1

That looks great.

Check packet filter

Phew, finally done! This is how fail2ban.log looks like right now. It shows [caddy-ratelimit] entries as expected:

schallbert@machine:~# nano /fail2ban/config/log/fail2ban/fail2ban.log

 <timestamp> <id> INFO [caddy-ratelimit] Found <IP a>
 <timestamp> <id> INFO [caddy-status] Found <IP b>
 <timestamp> <id> INFO [caddy-ratelimit] Ignore 172.18.0.1 by ip
 <timestamp> <id> INFO [caddy-status] Ignore 172.18.0.1 by ip
 <timestamp> <id> INFO [caddy-ratelimit] Found <IP c>
 <timestamp> <id> INFO [caddy-ratelimit] Found <IP c>
 <timestamp> <id> INFO [caddy-status] Found <IP c>
 <timestamp> <id> INFO [caddy-status] Found <IP c>
 <timestamp> <id> INFO [caddy-ratelimit] Found <IP d>
 <timestamp> <id> INFO [caddy-status] Found <IP d>
 <timestamp> <id> INFO [caddy-status] Found <IP e>
 <timestamp> <id> INFO [caddy-ratelimit] Found <IP e>
 <timestamp> <id> INFO [caddy-ratelimit] Found <IP f>

To test the entire chain, we now need to check whether the corresponding IP address is actually blocked in the hardware. To do this, I enter:

schallbert@machine:~# iptables -n -L | grep "1.1.1.1"
REJECT all -- 1.1.1.1 0.0.0.0/0 reject-with icmp-port-unreachable

Everything seems to be fine. For one final test from an external IP address, I call a Link Checker in the browser, which should crawl my website and thus trigger the rate limiter. Although it crawls 44 links in a very short time, the test is displayed as “passed.” Very strange, [caddy-ratelimit] should have triggered!

iptable-chain INPUT instead of FORWARD

Confused, I take a look at the entire iptable. At the same time, I look up the IP address of the link checker. It appears in the iptables. And yet, it’s apparently not blocked. Why is that? A closer look shows:

Chain INPUT (policy ACCEPT)
target prot opt source destination
f2b-caddy-ratelimit tcp -- 0.0.0.0/0 0.0.0.0/0 multiport dports 22

The problem here is that my jail is on the INPUT chain. However, the requests don’t go directly to my server hardware, but are forwarded via Docker to Caddyserver. To work, I have to land on the FORWARD chain, where DOCKER-USER already is. Very strange, since I had specifically specified chain = DOCKER-USER in jail.local. Something must be overriding this definition.

An inconspicuous forum post together with the multiport comment in the iptables printout led me to the solution: The action = iptables-multiport directive overrides my chain = DOCKER-USER statement, because the following is set in the corresponding configuration file iptables.conf: chain = INPUT.

So I simply delete the action entry, so that fail2ban reverts to the default for DOCKER-USER: multiport dports 80,443.

Now the jail name also appears correctly in iptables.

schallbert@machine:~# iptables -n -L
Chain DOCKER-USER (1 references)
target prot opt source destination
f2b-caddy-ratelimit tcp -- 0.0.0.0/0 0.0.0.0/0 multiport dports 80,443

Finally, another link check shows that my rate limiter is working.

Gitea crashes: Too many requests

Schallbert — Mon, 30 Jun 2025

Here I’d like to briefly describe how a service crash repeatedly paralyzed my server for hours. So badly that I could only restart it from the provider’s console. I’ll explain how it happened and how I intend to avoid this and similar problems in the future.

I was attacked. Or was I?

I was working on an article that I wanted to post later. To be on the safe side in terms of backup, I created a commit as usual and wanted to push it to my Gitea instance. But my git push command simply didn’t work.

Confused, I tried to access my website. The response was:

Strange. Then I wanted to log in to my server to check everything was OK: ssh <servername>. Again, the terminal remained unresponsive. Bummer!

Timeout. As a last resort, I logged in to my hoster’s web interface and looked at the server’s graphs:

Oh, what’s going on? I’m trying to shut down the server via the web interface. That doesn’t work either. Only a hard reboot succeeds. I can log in again via SSH and see that all Docker containers are booting up normally.

What happened?

It’s a good thing I keep all the logs for a week. This, along with the machine’s utilization over time, allows me to reconstruct what happened. At least to some extent.

Which logs help?

Localize the crash! Find the triggering application in the kern.log and note the timestamp.
Are there system-wide effects or other services being affected? Check the syslog.
If you suspect the system may have been hacked, auth.log has the details.
If the affected application is running in a container, the relevant logs there may be helpful.
Review the application’s logs. The time just before the crash is particularly interesting.

And here again, in detail, are the logs I reviewed for my behavior and where they can be found.

log name	purpose	relevant content for this issue
/var/log/syslog	System-wide (bare metal) messages	(docker) warning: “health check for container `<ID>` timeout” (containerd) error: “ttrpc: received message on inactive stream”
/var/log/auth.log	Contains authentication messages from external hosts, in my case mostly SSH	None
/var/log/kern.log	Logs all app, service, daemon and system crashes	Out of memory: Killed process `<ID>` (`<serviceName>`)
[.]/gitea/log/gitea.log	Protocol for requests, actions on Gitea’s web interface, repo changes etc.	`GET` requests, crash/restart indications
docker container logs `<containerID>`	Docker’s logs for the container in question	Received signal 15; terminating. (SIGTERM)

kern.log/syslog

Here, next to the stack trace, I can see exactly what happened. The most understandable message is the following:

Out of memory: Killed process `<ID>` (gitea)

So, Gitea was unplugged because it had consumed practically all system resources. If I scroll up in the log, I see a few minutes earlier:

<timestamp> <machineName> dockerd[716]: level=warning msg="Health check for container <ID>"
<timestamp> <machineName> dockerd[716]: level=error msg="stream copy error: reading from a closed fifo"

Already here, I see warnings that the gitea container isn’t working properly. Another advantage of including “health checks” in the docker-compose.yml file. For other reasons, I’ve already written an article about the purpose and implementation of health checks in docker.

gitea.log

The following log entry indicates that Gitea has just been restarted due to a bug in a submodule:

cmd/web.go:205:serveInstalled() [W] Table system_setting Column version db default is , struct default is 1

And further up in the log:

Very interesting. The highlight.css is in my public repo. It’s the scheme for lectures.schallbert.de’s appearance. This file is quite large, almost 1 MiB. And it’s loaded here dozens of times, practically for every commit.

Now I’m looking at other crashes in the past. It’s always bursts of GET commands for large files or requests for compares between two branches of the repository that precede my server crashing.

Who’s behind this?

All crash-triggering requests come from the same IP address range. The crashes started a few weeks ago. But mostly at times of day that I(and apparently many of my readers) didn’t notice. And after a few minutes, the server was always back to normal operation.

Oh, the trail leads to Google’s Cloud Platform (GCP).

But I had blocked robots?

Indeed, I had stopped the search engine indexing for Gitea. Strange. Or am I not dealing with one of the Google spiders or “AI” scrapers, but with a nasty hacker who rented a virtual machine from “Google Cloud”?

Log research: How frequently is the file requested?

A spider would only crawl all my pages once every few weeks, right? And hopefully not ignore my robots.txt. A crawler certainly wouldn’t make the same request multiple times and at short intervals, would it?

To check this, I search the Gitea logs for entries of GET requests to one of the large and therefore resource-intensive files to transfer:

gunzip gitea.log.<date.rotateID>
grep "<filename>" gitea.log.<date.rotateID>

In the list, I see that the same request to the same target file is made multiple times from the same IP address, all within seconds. Shortly before the crash, my server took almost 4 seconds to serve the request.

I also see that the IP address changes every few hours.

Countering a DoS Attack

In summary, I have to conclude that I’m being attacked via a Denial-of-service attack from the GCP address space.

To gather a bit more background information, I visit a few websites on the topic. There, I learn that gitea on my server is crashing under an Application Layer Attack. Now that I know what’s happening and what the problem is, finding solutions is much easier 😅

Of course, I don’t want to give up without a fight by permanently taking my Gitea instance offline. So, what options do I have?

Provide more resources

Admittedly, my machine only has 40GB of memory and 2GB of VRAM, as well as a measly 2-core CPU from 2009. I could book a more powerful server to better handle peak loads. But this wouldn’t prevent the attack, only mitigate its effects.

Rate limiting directly in the web server

Rate limiters limit the number of requests a client can make within a certain time window. This conserves server resources. This is a “soft” defense against DOS attacks, as triggering IP addresses are briefly and gently blocked with an error message. Typically, HTTP status code 429 “Too Many Requests” is returned when the limiter intervenes.

Gitea doesn’t have a rate limiter. In my setup, Gitea runs behind a reverse proxy provided by my Caddyserver. So, that’s where I’d have to start. For Caddy, rate limiters are only available as external modules, which must be manually installed and configured in xcaddy.

Blocking with IP-table rules

Here, you could again use fail2ban and simply block multiple requests for the same resource from an IP address. Gitea has a description of the setup in its documentation. In my case, I would have to continue where I left off, and thus monitor not only SSH but also normal page requests.

This solution sounds the most sensible to me, as it clearly separates concerns into different applications. I only use applications I already have available: Caddy would provide the access logs, and fail2ban would have to read them and set filters in the jail.local configuration so that it acts like a rate limiter.

Let me link the follow-up article Setting up Fail2ban with Caddy here 🙂

My own webshop - Part3

Schallbert — Sun, 15 Jun 2025

Selecting a solution based on key questions

Which solution is straightforward and quick to set up?
Which solution is easier to maintain?
With which system can I, as the operator, most easily understand how everything works (transparency, dependencies)?
Which system has the smallest attack surface?

There’s still quite a bit missing here!

Over the last few weeks, my enthusiasm for setting up my own online shop has waned a bit, as I wanted to focus first on securing my server and finishing off some old, nearly completed projects. I’ll return to this series as soon as that ‘I-want-to-do-something-new’ fever takes hold of me again.

Hang in there! Schallbert

Creating an own Webshop - Part 2"

Schallbert — Thu, 15 May 2025

A few options for webshop software

Let’s assume only software that at least partially meets my requirements get shortlisted. Below, I would like to present a few possible ecommerce solutions.

Shopify

Shopify is an all-inclusive package solution for online retail. Shopify Inc. is headquartered in Canada and handles its European business from Ireland. The Legal sections found on the website, as well as the complexity of the information presented on the topic of Data Protection, lead me to suspect that Shopify is a multinational corporation - and not just a provider of online shops.

But back to the topic.

The domain and hosting are included in the price with Shopify, and you can create your own shop rapidly and without any programming knowledge in a browser window by selecting a theme, and setting a few parameters. Shopify has extensive documentation and offers training and all sorts of other services for its subscribers.

But where’s the fun in that? After all, I run a tech blog and am not a businessman, literally. A bit of programming is definitely something I enjoy. Furthermore, as an open source fan, I don’t particularly like the corporate stance (just look at their modern website, equipped with auto-playing videos and optimized for marketing purposes). The configurability is mediocre at best, and, as with most other solutions, I have to do the optimization for search engine results myself. Regarding data protection and security, they maintain the usual compliance and state-of-the-art communication; so I can’t look behind it and have to assume that they protect their customers well, also in their own interest.

Shopify describes itself as a closed-source SaaS platform.

Therefore, I’m removing this platform from my list of preferred web shop solutions because I don’t want to become unnecessarily dependent on the well-being of third parties. With Shopify, it’s not me who has control over subscription costs or “my” share of the cloud, but Shopify. It’s not me who can determine how many transactions are included in the “plan,” but Shopify. It’s not my customers who remain in control of their data, but Shopify or a third-party company they have commissioned to process it.

Wordpress / WooCommerce

I only know Wordpress as a juggernaut. The jack of all trades, capable of everything you could possibly want thanks to an overwhelming mass of plugins, extensions, and features. And much more you don’t need.

But it’s open source, free, and, according to its own statements, used by many, many shop operators. Let’s take a look at the most well-known e-commerce extension for Wordpress: WooCommerce.

I immediately found countless guides on setting up a WooCommerce store (for example, from GeeksForGeeks or Themeisle), so I’ll refrain from providing my own guide.

After skimming through a few tutorials, I’ll summarize:

A WooCommerce store is easy to set up and configure using a wizard.
There are many themes available, and easy-to-integrate store designs offer a suitable look for every taste.
It helps to already have a WordPress site and some basic knowledge of domains, DNS, and hosting.
Many payment service providers and some shipping providers are preconfigured.
There are numerous YouTube tutorial videos on the topic, and detailed documentation is also available.
Countless plugins allow you to optimize your store for even very specialized areas and purposes.
However, this variety of options can be offset by the fact that choosing the “best solution” can be difficult, and optimizing the site for speed and search engines can become complex.

All in all, WooCommerce strikes me as a balanced mix of “I want to quickly put together something useful” and “I want to be able to configure and modify everything.”

Snipcart

I would consider Snipcart a “shopping plugin” for any website. Even static websites that can essentially only display a product catalog become a fully-fledged eCommerce solution with Snipcart. The core element here is the shopping cart, which is used to process the purchase.

The interface is the website’s HTML markup. Snipcart is integrated as a JavaScript blob and referenced via HTML. The design and appearance are handled via cascading style sheets. In my case, I could use the existing “tech stack” consisting of a static Jekyll site with a CD pipeline in Gitea on a Caddyserver instance for the shop as well, and with minimal adjustments, conjure a seamless look for the blog and shop. Sounds great, right?

Snipcart is developed by a Canadian company in Quebec. It describes itself as a “Shopping Cart Platform”. An account is required for installation, and you must enter your shopping domain in the Snipcart web frontend, similar to Google Search Console or Google Analytics, to receive a so-called API key. Only with this key will the JavaScript blob integrate correctly.

And this brings us to the first major problem: Snipcart restricts my freedom.

<link rel="preconnect" href="<https://app.snipcart.com>" />
<link rel="preconnect" href="<https://cdn.snipcart.com>" />

This shows me that Snipcart would like to keep the data on its server. I have to use their content delivery network to handle the order process and integrate it with mine. So, it’s by no means the case that all connections remain with me and I “buy” the JS code from Snipcart and run it in the customer’s browser or a database solution behind it on my server.

So, if full autonomy and control over your own site isn’t a priority, Snipcart can be a very elegant e-commerce solution. This is especially worthwhile for existing sites. However, for me, it’s out of the question.

OpenCart

OpenCart offers a PHP-based shop module free of charge and open source. It can be booked as “Software as a Service” through third-party providers or downloaded via Github and deployed on your own server.

Installation and administration are handled via web interface and, in my opinion, are straightforward. You can also find a few images on DockerHub that you can get started with right away.

OpenCart is fully customizable to your needs. Templated themes can also be integrated. Out of curiosity, I visited a few websites that use OpenCart: In my opinion, the shop system is well-structured and easy to use. However, the loading speed of the sample pages isn’t always super fast.

Overall, I consider OpenCart, with its customizable, lightweight shop solution in PHP, to be a good solution - also thanks to its still-active community (13+ years!).

Django

Django is a web framework for Python. It offers many features, allows extensions by simply including Python modules, and allows even beginners to quickly create working solutions. Although there are numerous libraries for websites, shops such as django-SHOP, database integrations, integration of payment systems, etc., everything still has to be put together and configured by the developer.

While this gives you full control and a solution that’s perfectly tailored to your needs, it also has a significant drawback: learning time. Database security must be ensured by the developer, and user input validation must be programmed or at least correctly configured. Customer master data is stored in plain text by default. Access and deletion concepts must be created and implemented by the developer, or time should be allowed to identify and integrate suitable modules.

If frameworks are to be used, it is recommended to check for active community maintenance beforehand. For example, it looks to me as if there hasn’t been much activity on the Github repo for django-SHOP since 2021. A quick search, however, reveals a lot of alternatives - now it’s time to consider something suitable or reinvent the wheel yourself 😃

The alternative for beginners like me: no shop system

What is meant by “no shop system”? A form -based approach. Here the user looks at a static website that contains catalog and product pages. In addition, a few elements are shown: E.g. a drop-down menu for variant and a field for the number of pieces. Next to it a simple button. If this is clicked, a form opens that automatically transmits the clicked content and asks about the user’s contact details.

If everything is filled in properly, the form creates a PGP-encrypted email to me when sending. I can then view the order analogously to manually created order emails, send an order confirmation and request payment.

With a few products, a shop equivalent can be implemented supporting a few customers without much effort. Of course not as comfortable as with an eCommerce solution. But maybe that’s enough for the beginning.

In the next article

Selection and Implementation

Creating an own Webshop - Part 1

Schallbert — Tue, 15 Apr 2025

This is part one of a series of articles about building my own webshop. I’m listing my requirements and wishes for my e-commerce solution. This way I narrow it down to a few suitable software solutions that I could later use for my webshop.

What my webshop needs to be able to do

For me as owner, the webshop should operate as lean, transparent, and automated as possible. I want to spend little time on operation, maintenance, and support. Payment processing, order status management, customer registration, sending customer information, and a large number of steps in the shipping process, such as postage and entering shipping confirmations, should be automated or at least prepared for automation.

But the webshop must also be structured in a way that is understandable for customers: simple and with just a few clicks from the homepage to checkout, easy to use, similar to other common webshops, with straightforward payment processing. As little data as possible should be collected from the customer.

My webshop only needs to display a few products. Therefore, navigation can be kept very simple. Due to the small number of product categories, a tree structure may be completely avoidable.

Let’s take a detailed look at what features the web shop ultimately needs to provide.

Functions for Admins

Create a product

Title
Tags
Short description
Description
Images
Videos
Price
Profile / Data sheet / Instructions

Edit a product
Remove a product
Manage orders

Confirm a step
Restart a step
Cancel a process

Manage customers

Create (sandbox)
Delete (data protection)

Functions for Buyers

Browse the webshop
Configure product(s) or select product variants (e.g., color, quality)
Add product(s) to a shopping cart
Manage shopping cart
Register (optional, only for customer data storage and order management)
Select shipping
Make a payment

Automatic functions

Payment registration
Invoice creation and dispatch
Inventory management
Shipping notification

Non-functional Requirements

Here I list everything the web shop should be able to do “on the side.” Things that are not directly required for order processing or visible to customers.

Usability on the Operator Side

The shop should have as few dependencies on third-party software as possible and only integrate those modules that are absolutely necessary. Examples:

Payment processing
Shopping cart
Customer account / address data
Inventory
Order processing (mailer) for buyers
Invoicing.

The shop should only contain as many dynamically generated elements as absolutely necessary.

Usability on the Buyer Side

The web shop should feel like current, common e-commerce solutions: nimble, clear, simple.
No obligation to have a customer account.
The shop should support common payment methods (debit, credit, PayPal, Apple/Google Pay, etc.). Only local (German-based) payment service providers are acceptable.
Inventory levels and production times should be displayed transparently for the user.

Data Protection

The shop should only collect data necessary for business transactions.
If technically and legally possible, the shop should be able to operate without a cookie banner.
Customer accounts should be automatically deleted after two years of non-use (with two weeks’ notice).

Look & Feel

Here are some examples of successful, simple shop designs:

Requirements derived from this:

The web shop’s structure and color scheme should match my Blog page and provide a consistent look.
Only static fonts should be used without reloading.
Product descriptions should be able to be uploaded as Markdown documents (or similar).

Database or file solution?

It doesn’t always have to be a database. For many applications, only a fraction of the functionality of common database solutions is required, but this significantly increases the complexity of the overall system. For example, if concurrent writing rarely or never occurs and there is no strong data concatenation, a file solution can be simpler, faster, and more efficient Linking a related “Engineering Kiosk” podcast episode.

On the other hand, many shop systems come with batteries included, meaning they already have built-in data management that is almost transparent to the programmers.

File solution (static website)

From the operator’s perspective, the web shop should feel like a Git project (everything is a file).
Changes to the offer, item descriptions, prices, etc. should be able to be pushed to the server via git push - just like with a static website.
If a pull request is merged into ‘main’, the web shop should be rebuilt (CD pipeline) and the changes published to the server.
Dynamic elements are required for inventory management, shopping carts, customer interaction, and automated functions. Here, it is important to decide which elements are best handled directly in JavaScript and which are better handled via an automation pipeline.

Database solution (dynamic website)

Products are stored in the database.
The web shop is managed via a web interface.
Database management should be partially automated, with backups and encryption for customer data provided.
The database is transparent for the admin: There is no direct interaction; everything is channeled through e.g. a web interface.

In the next article

Comparing e-commerce solutions for my webshop

PeerTube instance migration

Schallbert — Tue, 01 Apr 2025

Brief definition

Fediverse?

Fediverse calls something Fediverse when several providers use the same protocol to allow users of these providers to interact freely with each other on digital media. This creates highly interconnected but decentralized services, where free choice of provider is part of the concept. The Verge goes into detail here for your reference.

PeerTube?

PeerTube is a decentralized video platform where multiple servers share the load and viewers can also help distribute videos. Roughly speaking, PeerTube is to YouTube what Mastodon is to X.

The Reason

This article begins with an email.

“You’re receiving this non-automated email because you’re one of the handful of people who has a video hosted on this PeerTube instance. I have decided to shut down this service […]” - Tom, scitech.video administrator

He was really sorry, but continuing to operate the instance would cost too much time, maintenance, and money. Furthermore, the platform is currently underused, so the effort seems disproportionate.

So far, so understandable.

Tom’s Story: scitech.video

Since I had uploaded and linked all my videos there, I was very interested in the background. So I asked for more information. My questions were answered comprehensively and promptly.

The Beginnings

Tom set up his PeerTube instance in 2019. Originally to share his own content. Shortly thereafter, he opened the hosting service to others, albeit with some unease.

From an admin perspective, such a video platform seems like the “Wild West”. Ensuring healthy growth for its offerings has proven quite difficult, especially because of the many bots and advertisers involved.

“Growing the service turned out to be challenging. Every time I opened registrations spambots started creating accounts easily and uploading videos with advertisements in the video descriptions […]” - Tom, scitech.video administrator, via E-Mail

Lots of bots and unwanted users around

While he was always able to prevent these unwanted spammers, scammers, and advertisers from contaminating scitech.video, this still meant constant work and manual monitoring of the automatic rejection and moderation tools.

At this point, I’d like to agree with Tom. I have an average of about 340 failed connections from external IP addresses to my server per day, which I then block. On the ssh port alone. And my server is neither particularly large nor well-known, nor can it be considered a worthwhile target. Not to mention the countless attempts to access my Gitea user interface, crawl the website for hidden links, and much more.

For operators of a portal that also grants external users write access (videos, descriptions, tags, comments), the problem is even more serious. Unlike my site, participation is part of the concept here.

Copyright?

Exactly. From the platform operator’s perspective, every video is fundamentally subject to copyright or related rights. But is the user who wants to publish this video also the author? Or has this person acquired usage rights?

Depending on the country and the associated legal situation, administrators must address this issue very carefully. High penalties can be imposed on both administrators and users if a copyright infringement occurs.

The former, at least in Germany, can be resolved according to a Federal Court of Justice ruling of October 23, 2024, as follows: “[…] after a clear indication of a violation of law, the content posted there must be checked for similar violations within the scope of what is technically and economically reasonable, and infringing content must be blocked or deleted.”

However, due to the authorities’ strong focus on specific subject areas, violations are usually quickly noticed - because users can also “report” posts.

Some countries are currently preparing laws to protect children and young people from using social networks and the associated dangers. For reference, here is a link to the Federal Agency for Youth and Media Protection. Here, too, the administrators are required to implement these rules - but how?

At least, this is what Tom is asking himself, given the increasingly diverse legal interpretations of different countries.

Updates and Maintenance

“PeerTube upgrades were a bit of a drag because everything was tightly coupled with their specific nginx configuration, with breaking changes that had to be adjusted every couple of minor versions.” - Tom, scitech.video administrator, via email

The work isn’t particularly difficult to perform, but it does take a lot of time. As with other software, it can happen that a PeerTube server won’t boot after an operating system update, for example, or that incompatibilities arise between different software versions or modules.

For performance reasons, testing proves difficult, so work is often carried out directly in “Prod.” In the event of an error, live operation suffers.

The configuration is relatively complex and has to be adapted to the hardware. Major PeerTube updates also require changes here.

Conclusion

“I’m also disillusioned about the fediverse generally as a communication medium. A constant battle against spam/LLMs, so much anger, so much focus on follows, likes, and boosts.” - Tom, scitech.video administrator, via email

Tom is frustrated that the attention economy is taking hold here as well. Although conceived as ad-free and rolled out as a small grassroots movement, instances created out of idealism are quickly misused for advertising and self-promotion. The costs also amount to several hundred dollars a year, which further worsens the cost-benefit ratio.

One solution here, according to Tom, could be “a return to Web 1.0 or closed communities,” as already exists on Meatspace, Matrix, Discord, etc.

All the other factors also make it difficult to continue operating his instance: The user base never grew particularly large, and there are only a few people who still upload their content there. As a father, he now has much less time available and must prioritize projects. Thus, he is now ending his six-year journey as a provider on the Fediverse.

I wish Tom continued strength, idealism, and enthusiasm for the future.

Jan’s story: makertube.net

Since I now needed to look for a new hosting platform for my videos, I tried searching for “video-maker” on the PeerTube website. Since my content has a clear tech focus and ranges from research and product development to pre-production, it wasn’t difficult to quickly find suitable candidates.

That’s how I came to my channel on MakerTube.

After successfully registering, I wrote an email to the admins with similar questions to the ones I had already asked Tom.

Motivation - why MakerTube?

“I have a small YouTube channel, and the constant advertising and focus on the big channels, clickbait, etc., and how people are forced to adapt to the content machine, was getting on my nerves. […]” - Jan, MakerTube administrator, via email

Jan was already active on the Fediverse and also works professionally with web services that need to scale rapidly. Therefore, it seemed logical to him to set up his own instance - but of course, with the really interesting topics of this world. It’s not intended as a competitor to YouTube and therefore focuses on makers, musicians, artists, etc.

Is it easy to get started?

Jan mentions it takes a lot of effort to set up such an instance and keep it performing well. Framasoft, who develop the PeerTube software, have done a fantastic job and support the hosting community wherever they can.

I can personally confirm that PeerTube feels completely different in its newer versions than it did a few years ago, and with practical features like automatically generated subtitles, it’s getting closer and closer to the functionality of YouTube.

Jan says that the Fediverse remains a niche. Therefore, the growth of his instance is rather moderate and very organic. PeerTube is currently still something for enthusiastic video creators; the content largely goes unnoticed by the general public.

How much time does it take?

Jan focused on low running costs while maintaining scalability in case the project suddenly becomes large. Now that the system is running smoothly, the effort is a few hours per week for user, content, and support moderation.

Exceptions: Major updates, upgrades, and technical issues.

What are people saying?

“There’s positive feedback, and people are grateful that I run the site and put in the work, but I’d say not very much.” - Jan, MakerTube administrator, via email

But of course, there’s also occasional outrage that this isn’t YouTube and not everyone can immediately create an account.

How do you detect copyright infringements?

“Regarding copyright, there are no direct tools. There’s a report function on the videos to point out violations of copyright or instance rules. Users also use this.” - Jan, MakerTube administrator, via email

Responsibility and liability for small providers like MakerTube lie primarily with the user. However, if violations are reported, the admins must take action and delete content. On MakerTube, however, this isn’t a problem in everyday life: New user registrations are carefully reviewed, and in cases of doubt, the content is inquired about. Most violations concern background music in videos, so there seems to be less awareness of copyright than with uploads of video copies.

And how is it all financed?

MakerTube is financed through donations. According to Jan, especially in the beginning, a lot of time was spent explaining this to people and securing sufficient budget for smooth operation. A video platform is resource-intensive: lots of CPU power for transcoding and streaming, a ton of disk space for all the assets, and simultaneously high bandwidth requirements, especially during peak times.

Conclusion

I’m satisfied with MakerTube so far: Transcoding is fast, videos run smoothly. Even Heise-Verlag has now uploaded their first video to the instance. However, it’s much more difficult to get to the landing page without a “fan base,” which was never a problem for me with scitech.video 😅.

Let me close with another quote from the site itself:

“MakerTube is here to stay.” - MakerTube, March 2025, about MakerTube.

Thanks, Jan. Here’s to the future!

Gitea: stop search indexer

Schallbert — Wed, 12 Mar 2025

After ages, I took a trip to the Google Search Console just for fun. Normally, I’m not interested in it because I don’t use Google myself and assumed that the search engines would do their job well out of self-interest.

I was confused when I realized that the majority of my pages didn’t even make it through the indexer.

The problem

How are people supposed to find my website? Even if they knew the correct search terms, they would hardly see anything on Google. After all, the console clearly stated that none of my blog posts appeared in the indexer. On my Gitea instance, on the other hand, several thousand pages were listed. A clear disproportion. And why does Gitea generate such a high volume of files?

Google’s crawler (a program that searches the web for content and makes it indexable for search) apparently finds every commit, no matter how small, in public repositories as well as the files behind them. Automation runs and other metadata are also indexed.

This is of course completely unnecessary and wastes energy in an area that I would much rather have in the presentation of my blog. So a solution is needed.

Possible solutions

A few possibilities immediately occurred to me:

Configure the crawler for each repository so that only the top folder level is searched.
Completely switch off search indexing for git.schallbert.de.
Use Search Console to make corrections until the indexer shows correct assignments on my blog posts.

The most obvious solution for me was to completely switch off the indexer for my Gitea instance. Correction loops only take effect days or weeks later with Google’s crawler and turned out too time-consuming. I deemed a separate robots.txt for each repository too complicated.

My implementation

I read the Gitea documentation. The suggestion of a robots.txt made sense to me immediately, as did its contents:

User-agent: *
Disallow: /

This tells crawlers from any party * that nothing should be indexed from the root directory / onwards.

Unfortunately, I had no idea how Gitea makes this file available in its instance:

To make Gitea serve a custom robots.txt (default: empty 404) for top level installations, create a file with path public/robots.txt in the custom folder or CustomPath.

Gitea’s configuration directory

So I experimented a bit. I created a public folder with the corresponding file, then a custom folder at different levels within the Gitea folder structure. Each time I restarted the container and checked whether the robots file also appeared on the server.

Nothing.

But then I got rid of the folder names and simply put the file in the Gitea configuration directory - the lowest level at which gitea no longer appears as a folder name. And this was the solution.

Test

After restarting, I was able to successfully display the file in the browser.

In the meantime, Google is also starting to remove the unwanted pages from the indexer. My blog articles will hopefully also be available in Google search over the next few weeks.

Repo-Lookout: Fix repository leaks

Schallbert — Fri, 20 Dec 2024

Repo information publicly available

One fine day I got an email from Repo Lookout. It said that one of my repositories was open to access from the internet. This posed a potential security risk as it might contain secret source files, hidden functions or even passwords.

At first I thought this “your repo is not secure” warning was a phishing attempt. But by simply entering the links contained therein, it became clear that Repo Lookout was right and that my Lectures repo was not only public as I had wanted, but that the version control metadata was also openly available on the web server.

Is that bad?

Normally, the internal structure and configuration (actions, discussions, wiki, etc.) behind a repository should not be public. Especially not if the repository is set up as private. But even with public repos, no one should be able to access the structure behind them.

Hence the mission of Repo-Lookout:

“Find source code repositories that have been inadvertently exposed to the public and report them to the domain’s technical contact.” - Repo Lookout /about (Crissy Field GmbH)

In this case, not critical, but unexpected and unpleasant.

Doesn’t do me any harm, because my web server only has the files there for retrieval and even if they were tampered with, they would have had no effect on my repository. I also had all secrets stored in specially created files as described in this article, so that they no longer appear in the configuration files. The repository is also located separately on the Gitea server instance. Nevertheless, only what I consciously want to make public should be connected to the Internet.

The image section from Gitea shows the same commit as the warning from Repo Lookout.

How it came about

In the deploy pipeline for my subdomain lectures.schallbert.de and landing page schallbert.de I have a direct checkout from the Gitea instance to the web server Caddy. This starts automatically as soon as the main branches receive an update. The runner starts a checkout action, which copies the repository to the corresponding directory on the web server.

Checkout action also copies the `.git` folder

The .git folder is simply set up as well. This is where all the data required by the version control software for state management of the repo is stored. I cannot see the copying process itself on Gitea, because the hidden .git folder does not even appear in the directory there. Understandable, because the entire display on Gitea is based on this folder’s content.

So the unwanted behavior remained under my radar - and according to Repo Lookout I am by no means the only one who has this problem.

Option 1: Fix on the web server

The most obvious solution is to block access to the file on the server side. This costs few resources and is easy to set up.

This forum entry shows how to do it: hide-entire-folder-caddyfile. For the files I want to protect, I add the following entries to the Caddyfile:

# /caddy2/Caddyfile

# [...]
respond /.git/* "Access denied" 403
respond /.gitea/* "Access denied" 403

This tells Caddy that when any file /* in the folder /.git is called, it should respond with the error code 403 “Forbidden”. The wildcard (*) is absolutely necessary, otherwise only the folder itself and not the files it contains will be locked.

To check, I check what happens when I request the Git logs:

# Terminal
curl <lectures.schallbert.de>/.git/logs/HEAD
Access denied

Works as designed.

Option 2: Fix in the checkout action

There is, however, a more elegant solution: ensure beforehand in the pipeline that the folder does not appear on the server at all.

Option 1: Using `sparse-checkout`

sparse-checkout allows you to select folders and files that should be part of the checkout. All other files in the repository remain untouched and do not appear in the branch. This saves time and storage space, especially with large repositories. But of course it only makes sense if you already know in advance that not all files need to be touched.

Negative list for `sparse-checkout`

In my case, I don’t want to copy the folders mentioned above to the server using checkout, but I want to copy everything else. How do I do that? Using negation in no-cone mode.

“The user has explicitly said ‘I want these directories and not those directories.’” - Derrick Stolee, Microsoft, on Github

The Checkout Action Guide states that sparse-checkout is also supported for the action automated by the runner.

Now I program with the help of the Github Actions Cheet Sheet:

# /.gitea/workflows/deploy-lectures.yml

# [...]
steps:
 - name: --- CHECKOUT ---
 uses: actions/checkout@v4
 with:
 path: ./tmp
 sparse-checkout: |
 /*
 !.git
 !.gitea
 sparse-checkout-cone-mode: false
 # [...]

To clarify: The script for publishing to my web server uses the action checkout, subfunction sparse-checkout and includes all files in the folder in the root directory tmp and below except .git and .gitea.

What is `no-cone mode`?

By default, sparse-checkout expects a list of folders to include for checkout. In no-cone mode, a list of patterns is expected instead. All operators that can also be used in the .gitignore to specify files, folders, omissions, etc. are possible here. This allows me to exclude certain folders, but has some significant disadvantages. Due to the much higher complexity of the pattern commands, the associated susceptibility to errors and the significantly more computationally intensive evaluation for larger repositories, the use of the no-cone mode is not recommended and is listed as “deprecated” in the documentation. Nevertheless, the proof is in the pudding!

Test with `sparse-checkout`

Now I upload the action to my Gitea instance and let my runner run it once. Then I log in to the web server and check whether the .git folder was created or not:

lectures# ls -la
[...]
drwxr-xr-x 8 root root 4096 Dec 20 10:41 .git
[...]

Damn, the folder is still there. I look in the logs of the action on my Gitea instance:

[...]
hint: 	git branch -m <name>
Initialized empty Git repository in /workspace/schallbert/lectures/tmp/.git/
[...]
::group::Setting up sparse checkout
[command]/usr/bin/git config core.sparseCheckout true

So it’s not because of sparse-checkout. It’s because of the way checkout works: Obviously, the .git folder is absolutely necessary for setting up the repository properly on my web server. So the only option I have is to delete it automatically after checking out.

Option 2: `rm -rf`

And so I try it by force:

# /.gitea/workflows/deploy-lectures.yml

# [...]
steps:
 - name: --- CHECKOUT ---
 uses: actions/checkout@v4
 with:
 path: ./tmp
 - name: --- REMOVE TEMPORARY FILES ---
 run: |
 rm -rfv ./tmp/.git ./tmp/.gitea
# [...]

And finally the .git folder no longer appears on my web server and my “repo leak” is patched. Thanks again to Repo Lookout!

Conclusion

I had the problem that the hidden .git folder, where the configuration and structure of repositories are stored, was published on my web server unintentionally and without my knowledge.

I have presented two working options for solving the problem here:

Set up an access ban on the web server
Modify the pipeline so that it automatically deletes the .git folder after it has been rolled out.

The second option is a bit more complex to implement, but it gets to the root of the problem instead of just fixing the symptoms. It also corresponds to the first principle of data protection: data minimization takes precedence over protective measures.

“What doesn’t exist cannot be lost” - Schallbert

Logrotate-Recursion

Schallbert — Mon, 18 Nov 2024

What isn’t working?

I actually wanted to use logrotate to clean up my logs daily and thus implement data economy: This way, IP addresses that I need for banning can be deleted uniformly and automatically after a short time. Otherwise, I’m not interested in the server logs at all and they just steal storage space.

With my current configuration, logrotate creates one log file per day, but names it xyz.log.1, xyz.log.1.1, xyz.log.1.1.1 instead of the expected xyz.log.1, xyz.log.2, xyz.log.3. This means that the deletion routine no longer works and the number of logs keeps growing.

And why?

I suspect that my logrotate configuration is faulty. At the moment it looks like this:

## LOGROTATE file named 'blog'
blog/* {
# it's ok if the file doesn't already exist
missingok
# Sets the logs to rotate in intervals
daily
# Tells the system to remove old logs and only keep the most recent rotated logs
rotate 7
# Rotated logs will be compressed and kept on disk if they are 10 MB or less.
size 10M
# compress and delaycompress: These two options are used together and indicate that 
# rotated logs should be compressed (gzip) except for the most recent one.
compress

delaycompress
}

As usual, the first thing I do is consult the manual, which I call up in the console using man logrotate. It says that the file to be rotated can be specified directly (and not just selected using a wildcard *). In addition, a log behavior can also be applied to multiple file paths by placing them in front of the curly bracket, separated by spaces.

The asterisk seems to be causing exactly my problem with the cascading logs: This not only rotates the actual target file access.log, but also touches all the files that have already been rotated again.

The solution

According to the manual, I rewrite the processing instructions for Logrotate so that instead of five separate instructions for the respective logs, I only create two files. These are also simpler and shorter than the original version above.

Example:

# rotate webserver log files.
gitea/access.log
blog/access.log
landing/access.log
lectures/access.log
{
 # Sets the logs to rotate in intervals
 daily
 # Tells the system to remove old logs and only keep the most recent rotated logs
 rotate 7
 # compress and delaycompress: These two options are used together and indicate that 
 # rotated logs should be compressed (gzip) except for the most recent one.
 compress
 delaycompress
}

Configuration update

I can roll this out using my existing server configuration pipeline by simply pushing it into the repository on the server. Now I just have to manually copy the files there to /etc/logrotate.d.

Testing Logrotate

Now it would be nice if I could test whether the changes I made actually work. To do this, I use the logrotate user manual again and type:

logrotate -d -v <LOGROTATE-DESCRIPTOR-FILE>

With the debug and verbose flags activated, logrotate will apply the rotation rule specified in the configuration file and provide feedback, but will not actually make any changes or rotations on the file system. For me, the result of the test looks something like this:

lectures/access.log 
 after 1 days (7 rotations)
empty log files are rotated, old logs are removed
considering log gitea/access.log
 Now: 2024-11-28 17:19
 Last rotated at 2024-11-27 00:00
 log needs rotating
considering log blog/access.log
 Now: 2024-11-28 17:19
 Last rotated at 2024-11-16 00:00
 log needs rotating
considering log landing/access.log
 Now: 2024-11-28 17:19
 Last rotated at 2024-11-07 00:00
 log needs rotating
considering log lectures/access.log
 Now: 2024-11-28 17:19
 Last rotated at 2024-09-21 00:00
 log needs rotating
rotating log gitea/access.log, log->rotateCount is 7
[...]
renaming lectures/access.log.7.gz to lectures/access.log.8.gz (rotatecount 7, logstart 1, i 7), 
renaming lectures/access.log.6.gz to lectures/access.log.7.gz (rotatecount 7, logstart 1, i 6), 
renaming lectures/access.log.5.gz to lectures/access.log.6.gz (rotatecount 7, logstart 1, i 5), 
renaming lectures/access.log.4.gz to lectures/access.log.5.gz (rotatecount 7, logstart 1, i 4), 
renaming lectures/access.log.3.gz to lectures/access.log.4.gz (rotatecount 7, logstart 1, i 3), 
renaming lectures/access.log.2.gz to lectures/access.log.3.gz (rotatecount 7, logstart 1, i 2), 
renaming lectures/access.log.1.gz to lectures/access.log.2.gz (rotatecount 7, logstart 1, i 1), 
renaming lectures/access.log.0.gz to lectures/access.log.1.gz (rotatecount 7, logstart 1, i 0)
[...]

The output shows that all the desired log files are taken into account by logrotate and also that the rotation is carried out correctly on a daily basis.

Learned something new again 😃

Broken NVME hard disk?

Schallbert — Tue, 10 Sep 2024

The event

Mid-April 2024:

After a system update, my laptop suddenly wouldn’t boot. The bootloader grub-core reports that it is having problems writing sectors.

error: ../grub-core/efi/efidisk.c:638: failure writing sector 0xXXXXXXXX

When I reboot, the screen remains black after the company logo is displayed with a blinking cursor. If I look behind the mask, loading various services fails:

Your browser does not support the video tag.

Boot errors due to read-only nvme

Initial analysis

I suspected a problem with the computer update and carried out a hard drive test from Lenovo’s UEFI. Result:

The media check is complete, the result is: FAIL

It’s not a problem with the operating system, but rather with the disk. Damn. Do I have to worry about my data? I do have backups lying around, but the last one is three weeks old, so I might be missing several hours of work. A setup from blank will take me a lot of time. So I’m going to try to see if anything can be salvaged.

Self-test

So I run the SMART tests for the disk with smartctl -x /dev/nvm0 and now know that it counts about 7600 hours of operation and has 240GB of usable capacity. The disk used under the name nvme0n1 thinks it is now “read-only”. All partitions including the kernel. Great. The result of my online search was something like:

The disk is irreparably damaged. A new one is needed.

Repair attempt

You don’t have to believe everything you read online. So I use Lenovo’s own repair tool for hard drives. It runs from the UEFI screen and is called Bad Block Recovery Tool. The program scans the disk for defective sectors and attempts to restore affected blocks. The run takes almost half an hour for the last 10%.

Result: Recovery failed. I run the tool again. And a third time in slow pedantic mode. Again and again to no avail. I keep getting critical medium error.

Is all of this not working because the disk has switched to read-only mode?

As I can no longer boot and have no console available, my options with on-board resources are now exhausted.

Comment: The read-only mode for SSDs

According to the Internet (DiskTuna), the read only mode is there to limit damage. If the controller of the storage device detects a serious error, it puts the disk into read-only access. This minimizes the risk of further blocks being corrupted or killed.

In the forum Superuser someone describes an error identical to mine, and there is actually nothing that can be done to keep the medium working.

Data recovery using a boot stick

I continue with a USB stick that I convert to a Linux boot stick at another computer. I plug it into my laptop and can finally boot it up again. I connect an external hard drive as data recovery medium.

Get an overview with fdisk

First I check whether all disks and partitions appear. To do this I type the command sudo fdisk -l into the console.

Luckily, everything is still there! Maybe I can just copy the files from the defective hard drive?

Copying with cp

Let’s try it out.

sudo cp -r -v /mnt/nvme0n1p3/Documents/my/important/files /mnt/externalHdd/save

For a few seconds I think everything is going well. But then cp stops with an error message saying the volume is not available. In fact, the disk is no longer mounted. However, I can find it again and mount it straight away. A few attempts later, I have backed up my really important data, but the copying process always stops after a few seconds. This would take far too long for the entire disk.

The observed behavior - mount, copy for a few seconds, disk disappearing - was confirmed to me in a few related forum posts online, so it seems to be normal behavior.

I need another solution.

Save data with ddrescue

After a quick online search, I installed the program ddrescue.

The documentation states some things very clearly. For example, that you should understand exactly what it does before you use it. Or the following:

“Never try to repair a file system on a drive with I/O errors; you will probably lose even more data. " - ddrescue Manual

Now I’m wondering whether I made the same mistake with the block repair attempt above. Or am I on the safe side with Bad Block Recovery, since this is about data blocks on the disk and not the “overlying” file system?

It also says here that the so-called mapfile is a core function of the program. It offers a great time advantage when aborting, restarting, merging backups and during data recovery. Therefore, it is not a bad idea to create the mapfile when creating an offline backup and to put it away safely. Overall, I found the tool extremely helpful, effective and easy to use and can only recommend it at this point.

Back to action: I find a very simple command to copy the entire data partition of my damaged NVME storage to an external hard drive:

ddrescue --sparse /dev/nvme0n1p3 /run/media/liveuser/myExternalHDD/datapartition

The --sparse option skips areas of the source disk that are filled with zeros and can thus save a lot of space on the target disk.

The program takes a quarter of an hour to complete the entire run. As you can see in the picture above, it can actually save 99.6% of the data and even several months later I haven’t missed the remaining 0.4%. Lucky me.

Verification: View the output of ddrescue

In any case, the program ultimately saves a single file on my external disk: datapartition.img I can view this image with fdisk -l:

datapartition.img: 236.89 GiB, 254356226048 bytes, 496789504 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

Everything is there, that’s good. Is all data included as well?

To check this, I mount the partition and look around using cd and ls -l:

sudo mkdir /mnt/olddisk # Mount point of the rescued data image
sudo mount /run/media/liveuser/myExternalHDD/datapartition.img /mnt/olddisk

Fortunately, everything that should be there was there. Now I could unmount the image, shut down the computer and have a new hard drive installed.

The new hard drive

Here, too, I was lucky: my used laptop was still under warranty and so I got it returned within a week with a new (and twice as big) hard drive. Because I was so afraid of data theft, I removed my old hard drive before sending it. I felt pretty paranoid.

I could have saved myself this work, maybe, because ddrescue has a software solution for paranoid people like me: the --fill-mode option. This allows you to delete the “still good” blocks and sectors after the rescue has been completed. Only the sectors that are already defective remain. From the online documentation:

ddrescue --fill-mode=+ --force /dev/zero bad_drive mapfile

I’m not sure whether this would have worked with a disk like mine, which the on-board controller had set to “read-only”. Whatever.

At least I can now continue writing blog entries with a fully functional laptop ☺️.

Gitea act_runner: Jump-start issues

Schallbert — Fri, 30 Aug 2024

Problem statement

Since I run this page myself I experience problems starting Gitea and act_runner. Sometimes, the runner won’t start. It exits with status -1 and so my workflows wouldn’t run the website build, verification and deploy tasks.

The error message is always the same:

[...]
runner-1 | time="2024-08-26T10:09:46Z" level=info msg="Starting runner daemon"
runner-1 | time="2024-08-26T10:09:46Z" level=error msg="fail to invoke Declare" error="unavailable: 502 Bad Gateway"
runner-1 | Error: unavailable: 502 Bad Gateway
runner-1 exited with code 1
gitea | 2024/08/26 10:09:46 cmd/web.go:242:runWeb() [I] Starting Gitea on PID: 16

Ok, the runner cannot hook onto Gitea. When I restart all services via docker restart, the act_runner container tells me it would be missing a network with ID <long hexcode ID>

Interim corrective action

I didn’t find time for resolving this issue. So I just ran docker compose twice:

# schallbert server-console
docker compose -f /path/to/giteas/composefile up -d

The second time, act_runner would start successfully (because Gitea is ready):

gitea | 2024/08/26 10:38:17 routers/init.go:116:InitWebInstalled() [I] Git version: 2.45.2 (home: /data/gitea/home)
runner-1 | time="2024-08-26T10:38:22Z" level=info msg="Starting runner daemon"
runner-1 | time="2024-08-26T10:38:22Z" level=info msg="runner: action-runner, with version: v0.2.10, with labels: [ubuntu-latest], declare successfully"
runner-1 exited with code 0
runner-1 | time="2024-08-26T10:40:32Z" level=info msg="Started runner daemon"

Not very satisfying to do things twice. Plus, I did this manually. So why is that?

Root cause analysis

First I thought it was due to Caddy not being ready as it runs the reverse proxy, connecting Gitea to the internet. In reality, Gitea is the culprit: At the time, docker starts act_runner, its startup procedure wouldn’t be complete. So I try to find out how to manage dependencies in docker.

Solution: Reflecting dependencies

First, I try linking gitea to the runner in the docker compose file.

depends_on

# gitea/docker-compose.yml
 ## service: runner
 ## [...]
 depends_on:
 gitea:
 condition: service_started

Well, still act_runner seems to be starting early. Docker just makes sure it has started the container, and does not wait for it to return a healthy signal.

Healthcheck

So I have to make sure that gitea’s startup procedure is complete. For this, docker provides the condition service_healthy. I adjust the runner configuration like so:

# gitea/docker-compose.yml
 ## service: runner
 ## [...]
 depends_on:
 gitea:
 condition: service_healthy # required so runner can attach to gitea
 restart: true

The service_healthy qualifier is returned by the healthcheck function. It consists of a so-called test and some environment parameters that define intervall, number of retries, delay and timeout conditions for checking service ready.

For the test I chose a simple command, assuming that Gitea (and, implicitly, Caddy too) would be healthy once it is able to respond to a GET request from curl.

# gitea/docker-compose.yml
## service: gitea
## [...]
 healthcheck:
 test: ["CMD", "curl", "-f", "https://git.schallbert.de/"] # checks if gitea is available
 interval: 10s
 retries: 3
 start_period: 30s
 timeout: 10s

With these lines, docker checks if gitea is “healthy” and only then starts act_runner. This permanently solved the problem and both Gitea and act_runner are stable.

Sending error logs

Schallbert — Tue, 20 Aug 2024

In this article I’ll look at how to set up “monitoring” for my server. Applications and services should be able to send me notifications in the event of an error.

What is this about?

Learn about and select sending mechanisms
Write test messages and verify the automation
Automatically send error report from Borgmatic
Send runner logs through Gitea
Notification when logging into my server via ssh
Create and send logs for server updates / server errors

Verification and monitoring

After running an automation, I want to know whether it was carried out successfully and whether all programs and services started their work as expected. This should apply to any automation - whether it is specifically sending an update, creating automatic backups or an action from Gitea, it doesn’t matter.

Normally I would use reporting mechanisms from act_runner for something like this. In the case of a server update, the runner is not available due to the circular reference already mentioned in the article Rolling out the server configuration, as all applications have to be shut down temporarily.

In addition, applications may have their own procedures for monitoring. So I have to look at mechanisms that allow me to easily access the information.

What characterizes good monitoring for me?

It is unobtrusive, so it only reports in the event of an error or unusual occurrence.
It provides specific information and error messages that are easy to understand.
It uses a message channel that works even if the system being monitored crashes.
It presents reports and error messages in isolation from other topics and does not mix anything.
It is brief.

Methods of automatic reporting

I will break this section down. In the general part, I will discuss the on-board tool for asynchronous monitoring that is available to me on the Ubuntu server. After that, I will look at the solutions that are partly built into my services or the applications that are compatible with them. I do not want to limit myself to the classic tool of email, but also look at “more modern” communication channels such as messenger programs or RSS feeds.

Mail via console - curl

Sending emails as notifications is common practice in many companies. With Linux, this can usually be done without any additional programs: The standard curl can help here.

curl is a program for transferring data from or to a server. If I enter my blog as the target, I get the HTML page output as a text file on the console:

curl https://blog.schallbert.de

This made me notice how much unnecessary data my blog software generates. I’ll have to clean that up later. Back to the topic: You can also use curl to access any web backend - for example a mail server:

# copied from https://stackoverflow.com/questions/8260858/how-to-send-email-from-terminal
curl --url 'smtps://smtp.gmail.com:465' --ssl-reqd \
 --mail-from 'from-email@gmail.com' \
 --mail-rcpt 'to-email@gmail.com' \
 --user 'from-email@gmail.com:YourPassword' \
 -T <(echo -e 'From: from-email@gmail.com\nTo: to-email@gmail.com\nSubject: Curl Test\n\nHello')

This only works for my mail provider if I allow logins from external clients. Google, for example, calls these “less secure apps”. As described in my post Server configuration with Git, I’m not a fan of writing secrets into anything - so I would rather not use the direct route via curl.

Mail via console - mail, mailx, mailutils, swaks

If you don’t want to always provide all the configuration information for the server and secrets, there are various handy tools for the console such as mailutils or swaks. Here, the connection to the mail server is configured once using the tool and can be stored in environment variables, for example. The syntax varies from program to program, but an email always drops out at the end.

The only major disadvantage for me is that information domains are mixed up. I would not want another “report thread” in my emails that I’d have to search for in the mass of messages. Other notification media in contrast allow me to set an automatic expiration date so that they disappear from my list after a set time.

Create RSS feed

Unusual but possible: I could create the monitoring as an RSS feed like my blog (schallberts-blog-feed) has e.g. using a Jekyll instance and put it online as a website. This would be easy to subscribe to, were readable with practically any reader and I could even set it up separately for each application. But there are obvious disadvantages:

High effort: Gitea runner with Jekyll instance, web server and subdomain required.
Publicly available: Suddenly build processes, updates, upgrades and error messages are accessible to everyone.
Error-prone: If Gitea, act_runner, my proxy or the web server crashes, I don’t get any reports.

With the last point at the latest, this option is out of the question for me. I want to get a report when my applications don’t do what they’re supposed to. Let’s take a look at the applications I already run and see how they tackle this problem.

borgmatic

Borgmatic is compatible with a lot of monitoring options. These include apprise, ntfy, healthchecks, cronitor, pagerduty, cronhub and grafana.

Apprise is a library. It is open source and can be integrated into an existing application as a dependency. Like an adapter, it enables asynchronous communication between the application and various communication services such as SMS, mail, messenger (e.g. Signal), various home automation systems or the notification mechanism of various operating systems. The latter, however, only works on the local machine. The trigger for communication must always come from the application.
ntfy is a push notification service. It is open source and can be self-hosted or used as a service via web application. Apprise, for example, supports ntfy as a communication service. The structure is quite simple and works like MQTT via publication subscription / broker client mechanism, but HTTP-based.
Healthchecks is a service. It is open source and can be self-hosted. The service is there to monitor regular activities and can act as a dead man’s switch: If, contrary to expectations, there is no response from the monitored program, it can raise an error message itself. This can in turn be forwarded to various communication services.
Cronitor is a monitoring solution and web application that, in addition to the notifications I need, provides a lot of analysis tools, performance measurements and metrics - mostly for money. There is a free “hacker” account with limited functionality, but this tool is also far too big and complex for me.
PagerDuty sees itself as a commercial “operations” platform that provides “incident management”, automation, “business operations”, “AIOps” etc. It’s out for me straight away. At the latest when I read the word “AIOps” 😅
cronhub looks like a commercial web application to me that, like Healthchecks, can create cron jobs, monitor them and report errors. It is of no interest to me because it does not seem to be open source and I could not host it myself.
Grafana is an open source web application that can either be self-hosted or used as a cloud service. Although many larger companies and projects use the application, it is orders of magnitude too extensive and feature-rich for my purposes.

fail2ban

Fail2ban does not have an automation solution like borgmatic. It simply creates log files that need to be evaluated in order to obtain content for notifications. At the moment I cannot think of anything that I absolutely need to know about Fail2ban. So I am not creating any reports for this for now.

Gitea

Gitea allows the log files to be configured very precisely. Access logs can be written out separately from service logs, repository logs or action logs and then processed further. According to my research, Gitea only offers a mailer as a notification system. In my opinion, this is primarily intended for repository and action logs.

Here, too, it would be most beneficial for me to analyze the logs and create a report myself if necessary.

Server

I already roll out my server configuration files using a script. So, in the event of an error, I could redirect the logs to a file and attach them to a report in any channel.

In addition, successful logins on the server would be worth a message. Then I can immediately determine whether it was me or not.

Selecting the reporting program

I not only have to select a monitoring program, but also choose a communications service through which the reports are sent.

Program	Advantage	Disadvantage
curl	simple	configuration must be provided
	works super easily with ntfy¹	No monitoring if the server crashes
	Onboard tools
	no service required
Apprise	enables integration of the Signal API²	Complex in interaction with external services³
	no service required
Healthchecks	Can act as a watchdog/dead man switch	Registration required
	Can be hosted locally⁴	Local hosting contradicts the watchdog concept

I only have a small server with a few applications, so I keep it as simple as possible and choose Apprise for services like borgmatic, which come with this library anyway, and curl for those whose reporting mechanics I have to program myself.

Communication channel

Program	Advantage	Disadvantage
SMS	High reliability	Registration required
		Limited to a few characters
		No attachments possible
		No topics, domain mixing
Mail	easy to set up	Poor searchability
RSS feed	Good sortability	Complicated to set up, error-prone
	Client very lightweight	Local hosting contradicts watchdog concept
		Content publicly available
ntfy	Simple and lightweight	Registration optional for web use
	Client purpose-oriented	Insecure: No encryption without registration
	Free for small users
	Independent of your own machine
Signal	“Note to self” easy to set up	Setting up and configuring the Signal API complex
	Independent of your own machine	Topics somewhat difficult to implement
		Useless if the server crashes

SMS, mail and RSS feed are already out of the question for me due to some of the disadvantages already explained above. So that leaves ntfy and Signal. ntfy impresses with its simplicity: send an HTTP PUSH request to a self-defined “topic” and subscribe to it on your cell phone - done. It is also easy to expand, because with self-hosting I can later increase security if necessary (encryption) and control the sending behavior. Signal, on the other hand, requires a separate client with relatively complex configuration. In addition, the devices would have to be connected to the server and cell phone, so it is not easy to add new subscribers. On the other hand, the transmission is well secured end-to-end, I do not have to register and it is also free.

For now, I have decided to go for the less complex solution with ntfy.

Logging

Good. Now it’s clear that I’m calling home using curl and apprise via the ntfy service. Let’s see what content needs to be transmitted and how to make it as unobtrusive as possible, but still short and concise.

When should logs be sent?

borgmatic: If a backup fails
If I successfully log in to my server or Gitea
After rolling out an update to the server configuration
Docker: If an application fails or doesn’t start
Gitea: Failed run of act_runner

What should I include in a report?

I want the classic “W questions” answered.

When did it happen (time stamp)?
Which application is reporting?
What happened?
Where (module, line of code) etc. did it happen?
How many were injured (severity, recovery)?

A notification then looks something like this:

TIMESTAMP APPLICATION PRIORITY MESSAGE EFFECT DETAIL

Rough version of a report

An ntfy message could look something like this:

curl \
 -H "Title: Error Borgmatic" \
 -H "Priority: urgent" \
 -H "Tags: warning" \
 -d "YYYY-MM-DD HH:MM:SS Backup creation aborted. Access to repository blocked" \
 ntfy.sh/schallberts-topic

I install the corresponding app on my phone, register on the “Topic” and send the message. It’s nice when things just work! Aha, the app shows the time of receipt. That’s accurate enough for me.

Implementation

Here I’ll take a look at all the services for which I’d like to set up notifications one by one.

borgmatic

Let’s start with a pilot test in small steps. First I configure Borgmatic to send a message via Apprise to ntfy if the backup creation fails:

# /borgmatic.d/config.yml
on_error:
- echo "Error while creating a backup."
- apprise -vv --title "Borgmatic Error"
--body "Could not run {output}. Aborted {error}."
ntfy://schallberts-topic

Then I test the command by calling apprise within the Borgmatic container. And indeed, it works. But getting here took me an hour, as the yml with its syntax rules even interprets within strings and borgmatic constantly refused to read the configuration file due to : and - characters. If you see an error similar to this one:

At 'on_error[1]': {'apprise -vv --title "Borgmatic Error" --body "Could not run {output}': 'Aborted {error}." ntfy://schallberts-topic'} is not of type 'string'

This shows that the interpretation of characters or indentations went wrong and the punctuation needs to be checked. Alternatively, the pipe operator | can be used to combine a command. Reference: yml specification

Server

An application of ntfy to monitor logins on a server can already be found in the documentation of ntfy itself. The description shows that something like this is easy to implement yourself and gives a great example using Pluggable Authentication Modules (PAM library).

Insert the following code at the end of the sshd file in the etc/pam.d directory:

session optional pam_exec.so /usr/bin/ntfy-ssh-login.sh

This line tells PAM that when a session is opened via ssh, it should call the executing module pam_exec, which then runs the script specified below. The value optional means that the configuration file should continue to be run even if the action fails. More details on how to use PAM can be found, for example, on Baeldung.

Then you simply have to store the ntfy call in the ntfy-ssh-login.sh script when the script determines that PAM has detected an open_session event.

That’s exactly how I implemented it and it works straight away. Great! The only disadvantage: I made this modification directly on the server. Without a container and outside of my configuration backup. If I now have to set up the server again for some reason, the change in PAM is lost and I won’t receive any more notifications until I manually enter the change again.

Server configuration

To send a report after running the configuration automation, I actually only have to adapt the server-config-action script that I wrote in the last article.

First, I want to be informed when the script runs without errors. To do this, I add a curl command at the end of the file.

# server-config-action.sh

# action commands...
# [...]
# send success notification
curl \
 -H "Title: server-config-action" \
 -H "Priority: low" \
 -H "Tags: white_check_mark" \
 -d "Rollout successful" \
 ntfy.sh/schallbert-server-config-push-topic

However, if the script does not run without errors, I would like to do the following:

Cancel actions after the error occurs
Create an error log
Send this log

I can achieve the first point by adding a trap for errors:

# server-config-action.sh
trap 'handle_error $LINENO' ERR

# action commands...

This calls the function handle_error. It is given the line number where the error ERR occurred. It also covers errors that can occur when restarting the container. I create the error log by redirecting the output of the individual script commands. I achieve this with the following line:

# server-config-action.sh

exec 3>&1 1>server-config-action.log 2>&1

# action commands ...

With this command, any standard output stdout should be passed on to the file descriptor 3, specified here with the log file server-config-action-log, overwriting it. For “append” there would have to be two redirection operators >>⁵. The log file thus replaces the console output, which I still had at this point in the previous article.

To have the log sent to me, I now define the following function at the beginning of the Bash script:

# server-config-action.sh
# on error, send a notification
handle_error() {
 # stop redirecting to file
 exec 1>&3 1>&2
 curl \
 -H "Title: server-config-action FAILED" \
 -H "Priority: high" \
 -H "Tags: x" \
 -T server-config-action.log \
 -H "Filename: server-config-action.log" \
 ntfy.sh/schallbert-server-config-push-topic
 exit 1
}

# set error trap
# redirect stdout and stderr to file
# action commands...

Gitea

For Github Actions reports, there is already an existing example at ntfy_examples/#github-actions. And the best part: Since the act_runner from Gitea is compatible with Github Actions at in the environment parameters, sending to ntfy works for me straight away. In the runner’s workflow file, all you have to do is enter the curl command specified on the website and you’re done.

Result

Five notifications about the most important processes on my server have now been set up. I have understood the underlying mechanics and can create additional notifications at any time if I need them. If I trigger all notifications as a test, my smartphone will display the as shown.

But I have not yet achieved independence from the system to be monitored. All notifications come from the affected machine and there are sometimes even dependencies between Docker containers. For example, the act_runner has to connect to Gitea via websockets. And that only works if Caddy provides the reverse proxy.

If I find out in the next few months that a service “under my radar” is no longer able to work, I will have to establish independence.

Context for setup with ntfy ↩︎
Context for the Signal API ↩︎
Report by asad-awadia on setting up the Signal API ↩︎
Healthchecks is directly available as a Docker image. ↩︎
Incidentally, the operators << and >> are not difficult for me to remember, because I used to have a lot to do with the programming language C. There - and in many other programming languages too - these are shift operators that can “shift” a value bit by bit or move one field to another. In the RISC architectures that I used at the time, the shift operation was “cheap”, i.e. very fast and memory-saving. ↩︎

Server configuration rollout

Schallbert — Wed, 31 Jul 2024

In the previous article, I brought my configuration files under version control. Now I want to automatically install the updates provided on the server.

What is this about?

Writing a script to automatically detect the update trigger
Server applications should be shut down and a backup copy created
The script should distribute the configuration on the system
All applications should then be restarted

Ways and possibilities

Here, too, I spent several hours researching. For larger projects, infrastructure experts seem to use specialized automation tools. These include Ansible, simpler tools such as cdist or even Kubernetes for highly scalable services.

Automation services Ansible, cdist, Kubernetes

Ansible and cdist seem to follow a similar concept: On the source machine (in this example my laptop) I create the configuration for my server and store this and the instructions for configuring my services in a “playbook” (Ansible) or in “types” (cdist).

To put it simply - as I understand it - the tool then takes care of building the configuration at the push of a button, dialing into the target host via ssh, pushing it over there and then starting it. For Ansible, there are even tutorials like this one for my scenario with docker-compose, which makes getting started even easier.

Kubernetes takes a different approach. It sees itself as more of a container manager, load balancer and scaling agent, but can do similar things for my purposes.

My approach

I, on the other hand, only need a fraction of the capabilities of these programs. I am also put off by the “additional” ssh channel, the configuration effort, the additional programs sometimes required on the target system, and the necessary reading and selection of the best tool for me. Because thanks to my very simple pipeline from the last article, the configuration is already on my server. It “only” needs to be copied to the right places and the affected services restarted.

Therefore, I am trying to solve this problem using on-board tools, my brain in working order and a few searches in relevant forums on the topics “executing a script when a file changes and “copying folder structures in Linux”.

Risk of circular reference

However, I am taking a risk: Since the deployment runs via Gitea, but the configuration affects Gitea itself, if there is an error in this module I can no longer change or reset anything: The Gitea service is then broken. I would have to get the configuration up and running again manually on the server.

I’ll try it out anyway and see if I am actually confronted with such a problem. If so, I’ll just switch to cdist and document it in a separate article! 🤗

Preparation: Create folder system and scripts

I think about it for a moment and create a few folders in the server file system:

# /opt/server-config
mkdir automation-hooks-trigger
mkdir automation-hooks-handler

These folders should represent the two sides of the automation. trigger contains text files that can be manipulated from the (web) service side. For example, act_runner should write the file server-config-update as soon as an update is available.

service –> writes to trigger file ||| server_handler() –> trigger_file.changed ? run_action() : loop()

The handler folder contains script files that make the necessary changes to the server file system.

Implementation: Shell script to handle the update

In the previous article, I executed the command touch server-config-update.txt in the act_runner container to signal the presence of a new server configuration. On the server, I now use the following code to periodically check whether this file has changed.

### set directories
actionfile=/opt/server-config/automation-hooks-handler/server-config-action.sh
triggerfile=/opt/server-config/automation-hooks-trigger/server-config-update.txt

### Set initial time of file
LTIME=`stat -c %Z ${triggerfile}`

while true
do
 ATIME=`stat -c %Z ${triggerfile}`

 if [[ "$ATIME" != "$LTIME" ]]
 then
 ${actionfile} 2>&1
 LTIME=$ATIME
 fi
 sleep 10
done

The operating system is given the interpreter with which the script is to be executed via #!/bin/bash. The command stat -c %Z queries when the file was last changed and returns the time in Epoch format. Finally, the time taken at the beginning of the script is compared with the time taken in the loop and if there is a change, the deploy routine that is still to be written can then run. The reference time is then updated. Finally, the script (sleep 10) pauses for ten seconds before the query starts again. I use absolute paths so I’m able to both start it from console and per service cron oder systemd.

First test and hooking into “autostart”

Now we need to make the file executable for a first test:

# change file mode bits: add "executable" flag to server-config-handler script
chmod +x /opt/server-config/automation-hooks-handler/server-config-handler.sh

Next, I create the server-config-action.sh file and add just an echo command. To test the setup I create my update trigger file locally and start the script. I then modify the triggerfile in another shell using touch server-config-update.txt and --- CONFIG UPDATE TRIGGER detected --- appears in the console. Great!

Later on the server, I need to have the script run automatically after a restart. To do this, I use the cron tool help and create a new entry using crontab -e:

# cron can automatically execute recurring tasks
# in this case, we're running server-config-handler script on reboot
@reboot sh /opt/server-config/automation-hooks-handler/server-config-handler.sh

With the command ps aux I can now check whether the script is actually being executed:

schallbert@server: ps aux
[...]
8:15 0:00 /bin/bash ./server-config-handler.sh
8:15 0:00 sleep 10
[...]

Backup

Making a backup of my applications and files before I roll out the update makes total sense. So I tell Borg that I want to create a backup now. Of course, I have to stop all services first. This means that all data is accessible, coherent and static.

Freeze state and data

To do this, I create a script in automation-hooks-handler that automatically terminates all containers except borg.

#! /bin/bash
# /opt/automation-hooks-handler/backup-pre-action.sh
# this shell script shuts down all docker containers prior to backup

echo "Shutting down containers for backup:"

echo "watchtower..."
cd /opt/watchtower
docker compose down 2>&1
# [...]

The expression 2>&1 means that any error output is redirected to the console. The number 1 represents the file descriptor for stdout, while 2 means stderr. The operator >& functions as a redirect merger. Later, we can go to this point and write the output to a log file - but I’ll leave that out for now for the sake of simplicity.

Borgmatic: Trigger handler mechanism #2 and #3

I can now run the script in two ways:

As a call by the server-config-handler script described above
By the automation solution borgmatic placed in front of borg

I choose the second option and therefore write the following commands in borgmatic.d/config.yml:

# borgmatic.d/config.yml
# List of one or more shell commands or scripts to execute before
# creating a backup, run once per repository.
before_backup:
 - echo "Triggering container shutdown for backup."
 - touch /etc/automation-hooks-trigger/backup-pre.txt
 - sleep 20
 - echo "Assuming container shutdown complete. Creating the backup now."
# [...]
after_backup:
 - echo "Triggering container restart after backup."
 - touch /etc/automation-hooks-trigger/backup-post.txt
 - sleep 10
 - echo "Assuming container restart complete. Exiting."
# [...]

For this to work properly, I have to create a volume in the associated docker-compose.yml and have it point to the path in the server’s file system: ${VOLUME_UPDATE_TRIGGER}:/etc/automation-hooks-trigger Now I set up the other side of these triggers: Handlers monitor trigger files for changes and call action scripts accordingly. These look very identical to server-config-handler.sh except actionfile and triggerfile paths.

Creating the backup

If I were to address borg directly, the backup could be created using create. To do this, I would have to specify in which repository the backup copy should be saved and under which name. In the example below, this is specified using the scope operator: ::config-update. The folders to be backed up are then specified.

borg create /path/to/repo::config-update ~/opt

I use borgmatic, which takes a lot of work off my hands using the configuration file. However, I have to execute the command in the container. To better check whether everything is working, I output statistics “verbose” to the console (--stats -v 1) and display the copied files --files.

docker exec borgmatic sh -c "cd && borgmatic --stats -v 1 --files 2>&1"

I add this line to the automation script.

Second test to create the backup

If everything works now, the complete process looks like this:

After the changed files have been received by Giteas Automation, act_runner (in the Docker container) stores the files on the server and then sets the server-config-update trigger.
Within 10sec the trigger is recognized by server-config-handler.sh, which then calls server-config-action.sh.
borgmatic (in the Docker container) is instructed to create a backup. This in turn writes the pre-backup trigger.
Again within 10sec this pre-backup-handler.sh calls the backup-pre-action.sh script and stops all containers except borgmatic.
Due to the built-in delay, borgmatic waits for this and then creates the backup.
After the backup, borgmatic writes the backup-post-action.sh trigger.
Within another 10 seconds, post-backup-handler.sh recognizes the changed file and restarts all containers via post-backup-actions.sh.
borgmatic tells server-config-update.sh whether any errors have occurred anywhere in the process so far. If not, it continues.
All Docker containers are stopped.
The server configuration is rolled out to the appropriate locations.
All Docker containers are restarted with the new configuration.

A first success: These scripts are already running on my laptop up to point 6:

schallbert@laptop: touch server-config-update.txt
server-config-handler@laptop: --- RUN backup ---
borgmatic@docker: /etc/borgmatic.d/config.yml: Running 4 commands for pre-backup hook
 Triggering container shutdown for backup. 
backup-pre-handler.sh@laptop: --- RUN backup-pre-action.sh ---
 Shutting down containers for backup:
 watchtower...
 [...]
 complete.
borgmatic@docker: Assuming container shutdown complete. Creating the backup now.
 local: Creating archive
 Failed to create/acquire the lock /mnt/repository/lock.exclusive

Troubleshooting for “failed to acquire the lock”

This problem occurs for me when borg reports an error when creating a backup that causes the program to abort. In this case, the repository is apparently not released correctly, so that after restarting the container it remains reserved for the old, now non-existent container. The following command solves this problem:

docker exec borgmatic sh -c "cd && borg break-lock /mnt/repository

Transferring the configuration

Now the configuration files have to be copied to the correct location on the server. Fortunately, I had already cloned the target folder structure when creating the repository, so I should be able to do this with a single copy command without “hardcoding”. After a bit of online research and a look at the user manual for the copy command man cp, I have my command:

# copy recursively contents of folder "server-config" to "/opt"
sudo cp -r -v /opt/server-config/. /opt 2>&1

This tells the operating system to copy the contents (/.) of the folder server-config recursively (-r), including all subfolders into the folder opt, which is located in the root directory /. cp works in an overwrite-supplement manner, so it will create files that do not yet exist and overwrite existing ones, and will not “copy them next to each other” under the same name. With the -v option I can output additional details, and with 2>&1 I redirect the error output to the console.

All folder indicators must be exactly where they are in the command: A slash after /opt/ would copy folders redundantly without overwriting, but would overwrite files. Without . the folder server-config would be created in the target path.

Switching to rsync

Unfortunately the cp command also copies a few files that I don’t want copied: repository-specific folders such as .gitea, or the folders for triggers and handlers. I only need these under server-config, not directly in opt. To fix this, I use the rsync command instead. There I can use a -u option so new files owerwrite older ones only and add --exclude to exclude files and folders that should not be copied. This looks like so:

rsync -r -u -v --exclude '.*' --exclude 'README.md' --exclude '<otherFolders>' /opt/server-config/. /opt 2>&1

But suddenly gitea no longer starts. Error message:

docker@server: [...] failed to load config file "app.ini": open: permission denied

After a long time of pondering and restarting the Docker client several times, I see that rsync has written the permissions of the source file to the target file, which was not the case with cp before: -rw-------. Now I change this by running chmod +r app.ini. Everything starts up again as usual! 🎉

Restart all applications

Since I run everything on my server in Docker, two simple commands are sufficient:

docker stop $(docker ps -a -q) 2>&1
# [...roll out config changes...]
docker restart $(docker ps -a -q) 2>&1

The finished automation

My script is now finished and simply calls the action script.

#! /bin/bash
#/opt/automation-hooks-handler/server-config-handler.sh
#[...]
 if [[ "$ATIME" != "$LTIME" ]]
 then 
 echo "--- CONFIG UPDATE TRIGGER detected ---"
 ./server-config-action.sh 2>&1
 LTIME=$ATIME
 fi
 sleep 10
done

The Script server-config-action then executes the above described actions:

#! /bin/bash
#/opt/automation-hooks-handler/server-config-action.sh

echo "--- RUN backup ---"
docker exec borgmatic sh -c "cd && borgmatic --stats -v 1 --files 2>&1"
echo "--- STOP all containers ---"
docker stop $(docker ps -a -q) 2>&1
echo "--- DEPLOY config ---"
cp -r -v /opt/server-config/. /opt 2>&1
echo "--- RESTART all containers ---"
docker restart $(docker ps -a -q) 2>&1

In order for the whole thing to work reliably, the three handler scripts must run in the background:

server-config-handler.sh
backup-pre-handler.sh
backup-post-handler.sh

They react to the respective triggers by act_runner from Gitea or by borgmatic. I will now expand the crontab accordingly and then I am done with the task for now. The console output of the entire process looks like this:

--- RUN backup ---
/etc/borgmatic.d/config.yml: Running 4 commands for pre-backup hook
Triggering container shutdown for backup.
--- RUN backup-pre-action.sh ---
Shutting down containers for backup:
gitea... 
fail2ban...
complete.
Assuming container shutdown complete. Creating the backup now.
local: Creating archive
<borg archive stats>
/etc/borgmatic.d/config.yml: Running 4 commands for post-backup hook
Triggering container restart after backup.
--- RUN backup-post-action.sh ---
Restarting containers after backup:
fail2ban...
gitea... 
watchtower...
complete.
Assuming container restart complete. Exiting.
local: Pruning archives
local: Compacting segments
compaction freed about 1.82 MB repository space.
local: Running consistency checks
summary:
/etc/borgmatic.d/config.yml: Successfully ran configuration file
--- STOP all containers ---
<container ids>
--- DEPLOY config ---
sending incremental file list
<files that are copied>
sent 206,558 bytes received 3,463 bytes 420,042.00 bytes/sec
total size is 193,167 speedup is 0.92
--- RESTART all containers ---
<container ids>

Great! Now all I need is for this log to be delivered to me if something goes wrong.

Server configuration with Git

Schallbert — Mon, 15 Jul 2024

What is it about?

I want version control for the configuration of server applications
Discussion of technical solutions for implementation
Tutorial: Exclude Docker/Gitea/other “secrets” from version control
Tutorial: Create a deployment pipeline
introduce automation trigger for subsequent rollout of files on the server
Next article: Integrate automation on the server and install upgrades

Background

I have Watchtower running on my server to automatically keep the installed distributions up to date and freshly patched. Recently I had a case where my Gitea Action Runner would no longer start automatically and even when started manually it gave an error message.

Docker had apparently updated itself without my supervision and was now throwing a volume error when starting up the runner container, which I had never seen before. The error message was clear and could easily be fixed with small changes in a configuration file for Gitea. Nevertheless, I now had the feeling that version control for my configuration would be useful for my future self in order to be able to better understand changes, updates and the reasons behind.

Preliminary considerations

It sounds like a circular reference to me: I record the configuration files for my server in Gitea, which itself runs on my server. This could become interesting with auto-deployment. But more on that later.

Two options spontaneously come to mind for getting version control with automatic synchronization:

1. Hardlink

This solution would store the configuration files scattered across many folders on my server in a folder declared as a repository using a hardlink. Why use a hardlink? Because every service on my server runs encapsulated in itself and has its own configuration files and environment variables stored together with the service. Without hardlinks, I would have to version the entire service folder and make my .gitignore correspondingly complex.

With Git, I would version the files mapped via hardlinks and make their contents available on Gitea in this way.

schallbert@server:/server-config-files –> hardlinks –> repository –> Gitea

So I would get a downstream “version monitoring” with a backup copy in the sense that the files are now lying around multiple times.

2. Repo and auto-deploy to the server

In this scenario, I keep the configuration files locally on my laptop and can version them with Git as usual. On Gitea, I would map the repository, and at the end of the chain, my runner would have to auto-deploy on the server every time the configuration changes and then restart the affected containers.

schallbert@laptop:/server-config-files –> repository –> Gitea –> act-runner –> server:/<service1...ServiceN>/config-files

I would get the administration “main” on my laptop, and the server would follow the mapping on Gitea.

Problems

In both cases, I don’t have the option of testing changed configurations in advance. Everything I do goes straight to “Prod” and would be live. In the worst case, I can easily mess up my setup.

This doesn’t change the status quo ante, where I updated the files directly on the server. So a new problem only in the sense that I didn’t have a Prod operation when I set it up in the first place and so there was no risk of failure.

Solution two seems to be more complex to implement, because I need a deployment pipeline that rolls out the files in the repository on my server. Especially since the runner is in a Docker container, while the configuration files are located directly in the server’s file system.

Direct access to the server file system is technically possible, but according to my limited understanding it would mean a large attack surface for all my services, whose configuration files could now be directly manipulated via the repository in Gitea.

Selecting my solution

Solution two at least places the immediate live problem on my laptop, so that I don’t have to mess around with the production system in the first step. I think it would be easier to set up an integration environment here with which I can check my configuration changes in advance. Since I have all of my services running in Docker, this could perhaps be solved quite easily.

Let’s get to work

Create a configuration repo

Okay, then the first step is to get the configuration files from the server. To do this, I use the file transfer command scp: scp server:/path/to/source path/to/target about a dozen times until I have caught all the files.

I set up the folder structure in this repo exactly as the files are on the server. I hope that this will make my life a little easier later.

Secrets in `docker-compose.yml`

But what do I do with “secrets” in the configuration files? Private keys, registration tokens, hashes? I would rather not have them lying around more or less openly in the repository. When using docker-compose it’s quite simple: I can store secret values in hidden files for environment variables and exclude them from Git tracking. In the simplest case, such files are simply called .env and contain a list of environment variables in the style of

BORG_PASSPHRASE="<redacted>"

In the corresponding docker-compose.yml I pull the variable from the .env file as follows:

- BORG_PASSPHRASE=${BORG_PASSPHRASE}

I make these changes locally on my laptop. I use .gitignore to specify using .* so hidden files and thus .env should not be included in the repository. But how do I know whether the containers are still booting correctly?

Secrets in Gitea’s `app.ini`

With Gitea, I’ve had a much harder time storing secrets in files. The app.ini is also written dynamically by Gitea, so the file looks a little different every time the service is restarted. After a long search, I found this issue, in which a solution for storing secrets separately was sought and found.

The only thing that seems sensible to me is storing the values INTERNAL_TOKEN and SECRET_KEY separately.
The other two properties LFS_JWT_SECRET and JWT_SECRET are automatically generated anyway and regularly overwritten.

In my configuration I only use INTERNAL_TOKEN. So I will copy it in plain text and without quotes into a hidden file (.INTERNAL_TOKEN) and make it available to the container via a Docker volume:

# Gitea's docker-compose.yml
# [...]
 volumes:
 - ./.INTERNAL_TOKEN:/run/secrets/INTERNAL_TOKEN:ro
 # [...]

In the app.ini it is now important to use the path specified in the compose file:

# Gitea's app.ini
# [...]
 server:
 INTERNAL_TOKEN = /run/secrets/INTERNAL_TOKEN
 # [...]

Not INTERNAL_TOKEN_URI=/run/secrets/INTERNAL_TOKEN as stated in the issue linked above, because this creates an error in V1.22.1 I am currently using: Unsupported URI-Scheme.

A very rough test

To check whether the changed configuration files still work, I install docker and docker-compose on my laptop. Then I try to start the containers.

# docker console log
Error: Network 'caddy-proxy' declared as external, but could not be found.

Oh right, Docker is not yet configured here. So create the network: sudo docker network create caddy-proxy and try again. The download of Gitea and its dependencies begins and the container starts - although not as I had imagined: The folder permissions within Docker are incorrect, meaning that neither Gitea nor Act-runner can access all the required files.

Nevertheless, I find the first error:

# gitea container log
WARNING: The GITEA_RUNNER_REGISTRATION_TOKEN variable is not set. Defaulting to a blank string.

I had forgotten to put the token string in quotation marks.

So for a fully functional integration environment, I have to solve at least two more problems: Folder permissions for the Main on my laptop must be set up in such a way that the Docker user also has write permissions. A simple solution for now is to append a :Z to the volumes in question and mark them as private unshared. Now the volume definition in docker-compose.yml looks like this:

# gitea/docker-compose.yml
# [...]
volumes:
 - ./gitea:/data:Z
 # [...]

I need a second file for environment variables to redirect my services to localhost. At least the service starts this way and I can see the log output. I can already spot most of the configuration errors.

I’m not sure, but I might have additional problems with the caddyserver such as certificate management, proxy settings and so on.

Create a deploy pipeline

Now it would be great if the files uploaded to the Gitea repo (and previously tested locally for functionality) would automatically find their way to my server. For this I could create another Docker volume where act-runner would then put the data stored via on:push trigger. They would then be available on my server.

Setup

If we remember my last attempts to provide artifacts on the server using act_runner, we can use a large part of that for this task as well:

# deploy-to-server.yml
# Workflow for saving the server's config repo to the local disk system

name: Upload-server-config
run-name: ${{ gitea.actor }} uploads server configuration files
on:
 push:
 branches:
 - main

jobs:
 # Deploy job
 build:
 runs-on: ubuntu-latest # this is the "label" the runner will use and map to docker target OS
 container: 
 volumes: 
 # left: where the output will end up on disk, right: volume name inside container
 - /opt/server-config:/workspace/schallbert/server-config/tmp

 steps:
 - name: --- CHECKOUT ---
 uses: actions/checkout@v3
 with: 
 path: ./tmp
 - name: --- RUN FILE CHANGE TRIGGER ---
 run: |
 cd ./tmp/automation-hooks-trigger
 touch server-config-update.txt

Strange volume errors

But the road to this point was rocky. For a long time I had only specified /server-config under volumes: on the container side and not the working directory of the runner. Then the action runs through and all commands in the #TEST section also work. But when I look on my server, the server-config folder created by Docker remains empty.

I use the following debug code under RUN FILE CHANGE TRIGGER to help me find the path errors:

# deploy-to-server.yml
# [...]
run: |
echo "hello world"
touch updated.txt
pwd
ls -al

I use echo to check whether my code is even being executed in the runner. The touch command puts the current timestamp in the updated.txt file so that I can later use this as a “hook” for further automation. pwd shows me the active path within the runner so that I can correctly map the Docker volume to the server hard drive. ls -al shows me whether the configuration files compiled in the CHECKOUT step were written correctly.

This tells me that the volume path on the “right side” was wrong. I redirect it to the active directory of the runner:

# deploy-to-server.yml
# [...]
# left: where the output will end up on disk, right: volume name inside container
- /opt/server-config:/workspace/schallbert/server-config/
# [...]

Then I got the following to read:

# gitea / act-runner console log
failed to create container: 'Error response from daemon: Duplicate mount point: /workspace/schallbert/server-config'

It seems that the runner automatically creates the “right side” of the mount point itself and therefore cannot be reassigned. Only by adding another path part, in my case /tmp - see above - I fix the error and the long-awaited folder server-config finally appears on my server 😌 with the following content:

schallbert@schallbert-ubuntu-:/opt/server-config# ls -al
total 52
drwxr-xr-x 9 root root 4096 Jul 12 15:38 .
drwxr-xr-x 9 root root 4096 Jul 11 19:56 ..
-rwxr-xr-x 1 root root 395 Jul 11 20:05 boot-after-backup.sh
drwxr-xr-x 3 root root 4096 Jul 11 20:05 borgmatic
drwxr-xr-x 2 root root 4096 Jul 11 20:05 caddy2
drwxr-xr-x 3 root root 4096 Jul 11 20:05 fail2ban
drwxr-xr-x 8 root root 4096 Jul 12 15:38 .git
drwxr-xr-x 3 root root 4096 Jul 11 20:05 .gitea
drwxr-xr-x 4 root root 4096 Jul 11 20:05 gitea
-rw-r--r-- 1 root root 312 Jul 11 20:05 .gitignore
-rw-r--r-- 1 root root 557 Jul 11 20:05 README.md
-rwxr-xr-x 1 root root 371 Jul 11 20:05 shutdown-for-backup.sh
-rw-r--r-- 1 root root 0 Jul 12 15:38 updated.txt
drwxr-xr-x 2 root root 4096 Jul 11 20:05 watchtower

Distributing the configuration on the server

Okay, that’s the first step. I now have a properly configured Git repository that shows my server configuration and can be maintained and at least rudimentarily tested from my laptop. I can also use an automatic Action to store configuration updates on the server and write an update file with a timestamp.

Now the update has to be received on the server, distributed and the affected programs and services have to be restarted. But we’ll look at this in the article Roll out server configuration.

'redir' through Caddyserver

Schallbert — Tue, 30 Jan 2024

When I moved my website from Github Pages to my own server, I had some problems and unanswered questions regarding automatic redirects. My blog migrated from the root domain to a subdomain (blog.schallbert.de).

Requirements for the redirect

Links already referencing to my blog articles under the old name, e.g. to schallbert.de/about should redirect correctly to the subdomain (no 404)
The root page under schallbert.de should be fully usable
Data and files for domain and subdomains should be independent of each other
The web servers should be able to use different technologies for different subpages

Implementation

At first I had thought that I could achieve the redirects in the DNS provider via A and CNAME entries. However, I quickly realized that a) I had no idea how DNS even works and b) the right way is via the configuration of my web server.

First attempt: redir at ‘/’

On my web server (Caddy), redirects are specified in the Caddyfile via redir. This can be done permanently (as so-called http 301) or temporarily via 302. Default is a temporary redirect.

My first thought was to leave schallbert.de unmodified and forward everything after a possible /. This would then look like this in the Caddyfile:

schallbert.de {
 # Define webserver
 root * /www/landing
 encode gzip
 file_server

 # Redir to subdomain if domain has a slash
 @redirect path_regexp /*
 redir @redirect https://blog.schallbert.de{uri}
}

As a result, Caddy simply forwarded everything to the subdomain. The catch was that the subdomain itself also contains slashes or a URI after the slash and is then forwarded yet again. Example:

schallbert.de/post -> blog.schallbert.de/post -> blog.schallbert.de/post -> blog.schallbert.de/post -> [...]

This gave me a recursive endless redirect. I completely paralyzed my server with it 😅

Second attempt: Filtering via regexp

I need better filtering so that only slashes with additional characters behind them are actually redirected and the forwarding cannot be recursive. Fortunately, I had help from a good friend, so that the following Caddyfile was created a short time later:

schallbert.de {
 # Define webserver
 root * /www/landing
 encode gzip
 file_server

 # Redir to subdomain to maintain links for blog
 @redirect path_regexp ^/[^/]+(/.*)?$
 redir @redirect https://blog.schallbert.de{uri}
}

The Regexp checks whether there is a slash after the domain URL and whether it is followed by at least one other character. The redirection now works as desired and my blog is still fully accessible under the old links.

Nevertheless, I was still not completely satisfied. Because when I created a nice “landing page” on my root domain, suddenly only the plain HTML was visible. No favicon, no images, no CSS.

A quick look at the developer options quickly brought the realization that paths of all assets that I had stored on my landing page, marked with / in the folder structure, were forwarded to the subdomain. Where they are not located, of course.

Third attempt: Regexp plus file search

Fortunately, I’m not alone with this problem, so I found the solution in Caddy’s forum:

With the not file marker I can define redirects in Caddyfile if resources on the current page cannot be found.

My solution now looks like this:

schallbert.de {
 @filenotfound {
 path_regexp ^/[^/]+(/.*)?$
 not file
 }

 # show Caddy where to find page resources
 root * /www/landing
 encode gzip

 route {
 # Redirect to subdomain if article cannot found on root
 redir @filenotfound https://blog.schallbert.de{uri}

 # else, define webserver and show page
 file_server
 }
}

Here it is important to define the page with root * /www/<pagelocation> before creating the route. Otherwise the assets will only become available after I have already forwarded them for lack of existing files.

Success!

Now my construction site page is finally displayed correctly.

Automatic Upgrades for everything!

Schallbert — Sat, 20 Jan 2024

As the title says, I would like to have security updates installed as soon as they are released. This applies to the Ubuntu on my cloud server as well as to all applications that I run in Docker. As I’m lazy, I don’t want to carry out these security updates manually.

For function updates and enhancements, however, it is still OK for me to intervene manually from time to time. Let’s start with the system upgrades.

Security updates per `unattended-upgrades`

Ubuntu supplies - as a short query reveals - a configurable tool for automated upgrades called unattended-upgrades. First, I check whether it is installed on my system:

Installation

schallbert:~# which unattended-upgrades
/usr/bin/unattended-upgrades

Great. Otherwise I’d install it with apt install unattended-upgrades.

configure unattended-upgrades

The Debian Wiki to the rescue: Config files can be found at /etc/apt/apt.conf.d.

First I make sure that automatic upgrades are active. To do that I open 20auto-upgrades and check presence of following lines:

APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Unattended-Upgrade "1";

Very well, so automatic package index updates and -upgrades are active. Now, I should set an update time and upgrade types to install. These can be configured within50unattended-upgrades, mostly by removing commented-out lines.

Upgrade sources

The first section of the configuration file is dedicated to the permitted sources for the updates. Of course, only trustworthy providers should be listed here. My configuration includes updates for my Linux distribution and security plus ESM = “Enhanced Security Maintenance” updates. It looks like this:

Unattended-Upgrade::Allowed-Origins {
 "${distro_id}:${distro_codename}";
 "${distro_id}:${distro_codename}-security";
 "${distro_id}ESMApps:${distro_codename}-apps-security";
 "${distro_id}ESM:${distro_codename}-infra-security";
};

Updates for dev releases, too?

I have the value set to auto:

Unattended-Upgrade::DevRelease "auto";

Automatic reboot

Some security updates (e.g. concerning the kernel) require the machine to be restarted. They will therefore only take effect if you have set the corresponding values in the file. WithUsers is a matter of taste, because you are force-logged off if by chance an update has just been installed that requires a restart. To practically rule this out, I have entered a time under Reboot-Time when I am certainly not working with the machine.

Unattended-Upgrade::Automatic-Reboot "true";
Unattended-Upgrade::Automatic-Reboot-WithUsers "true";
Unattended-Upgrade::Automatic-Reboot-Time "03:45";

Of course, automatic reboot has repercussions for all the programs I use. I therefore make sure in my docker-compose.yml files that always or unless-stopped is really noted in restart:.

Notifications

I haven’t switched on automatic mails or other push notifications for the time being, as I’m still in test mode and don’t really want to receive a lot more messages. As soon as I change this, there will definitely be an update here.

Function test

To see if unattended-upgrades does what it is supposed to do, I take a look at the logs under /var/log/unattended-upgrades. The file unattended-upgrades.log contains the following, for example:

# [...]
2024-01-12 06:17:59,781 INFO Starting unattended upgrades script
2024-01-12 06:17:59,781 INFO Allowed origins are: o=Ubuntu,a=jammy, o=Ubuntu,a=jammy-security, o=UbuntuESMApps,a=jammy-apps-security, o>
2024-01-12 06:17:59,781 INFO Initial blacklist:
2024-01-12 06:17:59,782 INFO Initial whitelist (not strict):
2024-01-12 06:14:33,956 INFO Packages that will be upgraded: libc-bin libc-dev-bin libc-devtools libc6 libc6-dev locales python3-twisted
2024-01-12 06:14:33,956 INFO Writing dpkg log to /var/log/unattended-upgrades/unattended-upgrades-dpkg.log
2024-01-12 06:14:54,578 INFO All upgrades installed
# [...]

However, the file unattended-upgrades-shutdown.log is still empty. I will check at a later date whether everything works here too.

Automate container updates with `Watchtower`

Now to the updates of my applications. A quick chat with a couple of admins revealed this:

It is tedious to keep all programs up to date manually
For containerized applications, there are services that solve this task centrally
There have already been bad experiences when the latest release is referenced in docker-compose.yml.
With image: <application>:latest, not all applications are stable or create problems with dependencies

One of these solutions is offered by Watchtower, which again runs in a container.

Installation

As usual I create a docker-compose.yml in a new folder at opt/watchtower:

# copied from https://containrrr.dev/watchtower/notifications/

# watchtower/docker-compose.yml
version: "3"
services:
 watchtower:
 image: containrrr/watchtower
 restart: unless-stopped
 volumes:
 - /var/run/docker.sock:/var/run/docker.sock

Watchtower needs docker.sock to pull the updates and apply them to the existing containers. Apart from that, I have done nothing else for the time being. No notifications (for the reasons mentioned above), no logs, nothing. I hope this setup stays functional and I don’t have to post an update here anytime soon.

Further steps for a validation of Watchtower’s functions, for example, can be found in a tutorial by DigitalOcean.

Procedure

Watchtower checks the installed Docker images for updates on a daily basis according to the documentation. If there is a new image release, Watchtower sends a SIGTERM signal to the containers to be updated, whereupon they shut down (source). The containers are then started up again.

Involuntary functional test

A few days after starting the container, I dialed into my server again and looked through some logs at random. I just wanted to check what was going on.

In the fail2ban log files I suddenly saw pages and pages of entries at a certain point in time, 14:42:56. More than I had seen in sum for days before. When I scrolled up to the beginning of this chain of entries, I saw that fail2ban had probably stopped for a moment.

 2024-01-26 14:42:50,817 <container_id> INFO Shutdown in progress...
 2024-01-26 14:42:50,817 <container_id> INFO Observer stop ... try to end queue 5 seconds
 2024-01-26 14:42:50,837 <daemon_id> INFO Observer stopped, 0 events remaining.
 2024-01-26 14:42:50,878 <container_id> INFO Stopping all jails
 2024-01-26 14:42:50,879 <container_id> INFO Removed logfile: '/var/log/auth.log'
 2024-01-26 14:42:50,879 <container_id> INFO Removed logfile: '/var/log/caddy2/gitea_access.log'
 2024-01-26 14:42:50,879 <container_id> INFO Removed logfile: '/var/log/caddy2/server_access.log'
 2024-01-26 14:42:51,052 <daemon_id> NOTIC [sshd] Flush ticket(s) with iptables
 2024-01-26 14:42:51,063 <container_id> INFO Jail 'sshd' stopped
 2024-01-26 14:42:51,137 <daemon_id> NOTIC [caddy-status] Flush ticket(s) with iptables-multiport
 2024-01-26 14:42:51,137 <container_id> INFO Jail 'caddy-status' stopped
 2024-01-26 14:42:51,138 <container_id> INFO Connection to database closed.
 2024-01-26 14:42:51,139 <container_id> INFO Exiting Fail2ban
 2024-01-26 14:42:56,173 <new_container_id> INFO --------------------------------------------------
 2024-01-26 14:42:56,173 <new_container_id> INFO Starting Fail2ban v1.0.2
 2024-01-26 14:42:56,173 <new_container_id> INFO Observer start...
[...]

All the log entries at the same time start at the end of the section shown above. A whole bunch of IP addresses are loaded there in preparation to be banned. I looked at the Ubuntu syslog, slightly worried:

Jan 26 14:42:55 schallbert systemd[1]: docker-<ID>.scope: Deactivated successful
Jan 26 14:42:55 schallbert systemd[1]: docker-<ID>.scope: Consumed 21:05 CPU time
Jan 26 14:42:55 schallbert dockerd[690]: time="2024-01-26T14:42:55.228607617Z" level=info msg="ignoring event" 
Jan 26 14:42:55 schallbert containerd[639]: time="2024-01-26T14:42:55.229864632Z" level=info msg="shim disconnected" 
Jan 26 14:42:55 schallbert containerd[639]: time="2024-01-26T14:42:55.230125409Z" level=warning msg="cleaning up after shim disconnected" i>
Jan 26 14:42:55 schallbert containerd[639]: time="2024-01-26T14:42:55.230145478Z" level=info msg="cleaning up dead shim"
Jan 26 14:42:55 schallbert containerd[639]: time="2024-01-26T14:42:55.240586557Z" level=warning msg="cleanup warnings time=\"2024-01-26T14:>
Jan 26 14:42:55 schallbert dockerd[690]: time="2024-01-26T14:42:55.241839655Z" level=warning msg="ShouldRestart failed
Jan 26 14:42:55 schallbert containerd[639]: time="2024-01-26T14:42:55.870912387Z" level=info msg="loading plugin \"io.containerd.event.v1.p>
Jan 26 14:42:55 schallbert containerd[639]: time="2024-01-26T14:42:55.870997689Z" level=info msg="loading plugin \"io.containerd.internal.v>
Jan 26 14:42:55 schallbert containerd[639]: time="2024-01-26T14:42:55.871007368Z" level=info msg="loading plugin \"io.containerd.ttrpc.v1.t>
Jan 26 14:42:55 schallbert containerd[639]: time="2024-01-26T14:42:55.871150831Z" level=info msg="starting signal loop" namespace=moby path>

Hm, cleaning up dead shim. That sounds ominous. A quick search shows that the shim forms an intermediate layer between the container manager and the container itself and forwards the container’s input and output. Its runtime is linked to that of the container.

In other words, I saw a normal shutdown followed by a restart. Fortunately, no break-in! All that remains is to find out the reason for the restart of Fail2ban. This reminds me that I don’t even know when Watchtower runs the updates…

So I quickly looked in the Watchtower logs with docker container logs <container_id> and behold:

time="2024-01-26T14:42:47Z" level=info msg="Found new lscr.io/linuxserver/fail2ban:latest image (9bda077d765e)"
time="2024-01-26T14:42:50Z" level=info msg="Stopping /fail2ban (e0317a105d53) with SIGTERM"
time="2024-01-26T14:42:55Z" level=info msg="Creating /fail2ban"
time="2024-01-26T14:42:55Z" level=info msg="Session done" Failed=0 Scanned=6 Updated=1 notify=no

Now I have proof: Watchtower does what it is supposed to do. It upgrades containers to a new version as soon as updates become available. Great!

PS: Since I haven’t configured Watchtower very much, I assume from the timestamps in the logs that the updates are run daily at the time Watchtower is started. I wonder whether I should switch to a certain version of the image instead of using latest, though.

Better security for my server

Schallbert — Fri, 12 Jan 2024

Motivation

OK. I have my server setup, created an automatic build pipeline for my Blog and my website is displaying the Blog as it should. Still, I’m not completely done yet. Because I want to do more for server-side security than deactivating password logins.

And I don’t have backups. That’s never good, so let’s get something done about that.

Locking away unwanted guests

In my SSH-log of my machine’s root system, I get a lot of failed authentications:

[...]
Jan 11 00:19:29 sshd[93842]: Invalid user admin from 41.207.248.204 port 37194
Jan 11 00:19:29 sshd[93842]: pam_unix(sshd:auth): check pass; user unknown
Jan 11 00:19:29 sshd[93842]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ru>
Jan 11 00:19:31 sshd[93842]: Failed password for invalid user admin from 41.207.248.204 port 37194 ssh2
Jan 11 00:19:32 sshd[93842]: Connection closed by invalid user admin 41.207.248.204 port 37194 [preauth]
Jan 11 00:20:14 sshd[93886]: Invalid user svn from 84.108.40.27 port 44968
Jan 11 00:20:14 sshd[93886]: pam_unix(sshd:auth): check pass; user unknown
Jan 11 00:20:14 sshd[93886]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ru>
Jan 11 00:20:16 sshd[93886]: Failed password for invalid user svn from 84.108.40.27 port 44968 ssh2
Jan 11 00:20:16 sshd[93886]: Received disconnect from 84.108.40.27 port 44968:11: Bye Bye [preauth]
Jan 11 00:20:16 sshd[93886]: Disconnected from invalid user svn 84.108.40.27 port 44968 [preauth]
[...]

If I didn’t know that this is typical “internet noise”, I’d be nervous. Isn’t it coming close to thieves trying different keys at our front doors every few seconds? So what can we do about it? Ban them.

fail2ban

That’s exactly what fail2ban can do for me.

 __ _ _ ___ _ 
 / _|__ _(_) |_ ) |__ __ _ _ _ 
 | _/ _` | | |/ /| '_ \/ _` | ' \ 
 |_| \__,_|_|_/___|_.__/\__,_|_||_|
 v1.1.0.dev1 20??/??/??

In short, fail2ban scans access/auth logs¹ for IP-addresses causing multiple failed authentications, and bans them when surpassing a user-defined threshold within a defined period of time.

How does fail2ban work? It modifies iptables, thus accesses packet filter rules (buzzword Firewall) on the network layer. So incoming requests of already blocked IP-addresses will not even reach my applications².

install fail2ban

fail2ban seems to be close to an industry standard for blocking unwanted access requests on Linux. Every hobbyist admin I know is using it.

For installation and configuration of fail2ban there’s a ton of guides out there plus the (well-written) on inside its Github-repository. I won’t be pressing the point here.

I chose an installation as Docker container to get all dependencies auto-delivered as well. I’m using the distribution by linuxserver and write the following docker-compose.yml to have it deployed:

# /fail2ban/docker-compose.yml
version: "2.1"
services:
 fail2ban:
 image: lscr.io/linuxserver/fail2ban:latest
 container_name: fail2ban
 cap_add:
 - NET_ADMIN
 - NET_RAW
 network_mode: host
 environment:
 - PUID=1000
 - PGID=1000
 - TZ=Etc/UTC
 - VERBOSITY=-vv #optional
 volumes:
 - ./config:/config
 - /var/log/auth.log:/var/log/auth.log:ro # host ssh
 - /var/log/caddy2:/var/log/caddy2:ro  # gitea via caddy, caddyserver
 restart: unless-stopped

What I add to add here: fail2ban requires the Access logs as Volume (above added with :ro as read-only volumes). A lot of filter rules come predefined in its config folder, I just had to add a modified jail.local inspiration: linuxserver/fail2ban-confs and add a rule in filter.d for caddy pulled from muetsch.io to complete my door watch.

fail2ban example

This is how a fail2ban-log looks like for my SSH-Daemon (sshd):

# schallbert:/opt/fail2ban/config/log/fail2ban# grep "220.124.89.47" fail2ban.log 
 2024-01-11 17:12:20,600 7FBB3630BB38 INFO [sshd] Found 220.124.89.47 - 2024-01-11 17:12:20
 2024-01-11 17:12:23,003 7FBB3630BB38 INFO [sshd] Found 220.124.89.47 - 2024-01-11 17:12:22
 2024-01-11 17:12:25,205 7FBB3630BB38 INFO [sshd] Found 220.124.89.47 - 2024-01-11 17:12:24
 2024-01-11 17:12:27,206 7FBB3630BB38 INFO [sshd] Found 220.124.89.47 - 2024-01-11 17:12:26
 2024-01-11 17:12:32,610 7FBB3630BB38 INFO [sshd] Found 220.124.89.47 - 2024-01-11 17:12:32
 2024-01-11 17:12:33,045 7FBB36104B38 NOTIC [sshd] Ban 220.124.89.47

kbye!

What does not (yet) work: Gitea & fail2ban

Also on my gitea instance, I get quite some authentication requests that I’d like to block. Unfortunately, I cannot get Gitea to put the registered accesses into a log file. I see them in the console only. On the other hand, the shell access to Gitea is already secured as it is channeled through sshd that also guarantees remote access to my server.

What is missing is a ban for failed web login requests.

I thought I had configured app.ini to get access logs written to file:

# gitea/conf/app.ini
#[...]
[log]
MODE = file
LEVEL = warn
ROOT_PATH = /data/gitea/log
ENABLE_ACCESS_LOGS = true
ENABLE_SSH_LOG = true
logger.access.MODE = access-file

[log.access-file]
MODE = file
ACCESS = file
LEVEL = info
FILE_NAME = access.log
[...]

I have both activated the logs with ENABLE_ACCESS_LOGS and configured log-to-file. access.log is being generated but no authentication tries are being written to it. Maybe caddy’s reverse proxy snitches them away before they reach Gitea? I’m sure I’ll find out at a later point in time.

At some point I asked myself whether fail2ban actually works on the iptables of my server or only operates within the container. Luckily, I was not the first with this question (stackoverflow) and found the proposed solution quite charming:

Block any IP in the Docker container (docker exec -it fail2ban sh): fail2ban-client set sshd banip 111.111.111.111
Check the iptables to see if this IP appears there: iptables -n -L --line-numbers
In my case: enjoy, because it is there: 1 REJECT all -- 111.111.111.111
Unblock the IP again fail2ban-client set sshd unbanip 111.111.111.111

Protection against brute-force attacks via ssh as described above
Protection against overload using a Rate Limiter
Protection against attacks on the APIs (404/403 attacks) using caddy-status configuration for fail2ban, logs in JSON

# /config/fail2ban/filter.d/caddy-status.conf

# this regex works for caddy with json-style logs
failregex = "client_ip":"<HOST>"(.*)"status":(400|401|403|404|500)
datepattern = \d+
ignoreregex =

Regular backups

I want to be able to do disaster recovery. Unforeseen events impacting my server like incompatible updates, machine downtime or security breaches should not keep me from spawning and configuring a new, healthy instance in no time.

I’m not willing to create backups manually, copy and compress folders, finally downloading to a backup location via SCP / SFTP. This should take place automatically, and, ideally, without cost. A quick search reveals two suitable open-source tools: borgmatic and restic.

I randomly chose Borgmatic.

install Borgmatic with Docker

As for all other components so far, I want Borgmatic to run in Docker environment. Luckily there are ready-made solutions that need just a little configuration.

This is how my docker-compose file looks like:

# /borgmatic/docker-compose.yml
version: '3'
services:
 borgmatic:
 image: ghcr.io/borgmatic-collective/borgmatic
 container_name: borgmatic
 volumes:
 - ${VOLUME_SOURCE}:/mnt/source:ro  # backup source
 - ${VOLUME_TARGET}:/mnt/repository  # backup target
 - ${VOLUME_ETC_BORGMATIC}:/etc/borgmatic.d/  # borgmatic config file(s) + crontab.txt
 - ${VOLUME_BORG_CONFIG}:/root/.config/borg  # config and keyfiles
 - ${VOLUME_SSH}:/root/.ssh  # ssh key for remote repositories
 - ${VOLUME_BORG_CACHE}:/root/.cache/borg  # checksums used for deduplication
 # - /var/run/docker.sock:/var/run/docker.sock # add docker sock so borgmatic can start/stop containers to be backupped
 environment:
 - TZ=${TZ}
 - BORG_PASSPHRASE=${BORG_PASSPHRASE}
 restart: always

Inspirations were the related docs on Github. You can see that Borgmatic requires a lot of volumes to work properly, most importantly backup source and target location.

configure Borgmatic

All concrete data behind ${} are summarized in an .env environment variable file. This helps me avoid publishing passphrases by accident. Of course I have noted down the passphrase on paper as “backup” of the backup.

I then modify config.yml in borgmatic.d/ slightly, adding backup source, target, count, and cycle. That’s all, I’m ready for a first test.

testing Borgmatic backup

docker exec borgmatic bash -c \
"cd && borgmatic --stats -v 1 --files 2>&1"

This command has borgmatic create a backup for me through the Docker container.

I got an error message right away, saying repository does not exist. A look into the docs reveals that I have to manually create the backup target repository.

The following command does this, creating an empty, encrypted repo:

docker exec borgmatic bash -c \
"borgmatic init --encryption repokey-blake2"

Now I retry manual backup creation which throws no error .

Ein Backup only is a Backup…

…when it is successfully applied, a friend of mine said.

Right he is. Still, I don’t dare replacing my working server configuration with a backup. As a compromise, I go halfway by creating a docker-compose.restore.yml following the online guide. For additional help, I read modem7’s related blog post.

For my test purposes, I run docker-compose.restore.yml in the container’s shell and enter the following commands:

mkdir backuprestoremount
borg mount /mnt/repository /backuprestoremount

mkdir backuprestore
borgmatic extract --archive latest --destination /backuprestore

Here, I create the folder /backuprestoremount and have it point to my borg backup. Then, I extract the backup into /backuprestore.

Let’s do a sanity check:

# cd /mnt/backuprestore
# /restore/mnt/source ls
borgmatic caddy2 containerd fail2ban gitea hostedtoolcache watchtower

Yeah! The backup contains all applications along with their files. I stop at this point as I’m too lazy and cowardly to overwrite my working setup as mentioned above.

What (yet) doesn’t work: Borgmatic & docker-compose down

Borgmatic offers a simple method to run pre- and post-backup actions. As I want to maintain consistency within my containers, I want them to be shutdown before and restarted after the backup run. So I wrote a script, hooked it into the config and… nothing.

Easy to follow, this can only work directly if Borgmatic ran on bare metal. Docker’s encapsulation doesn’t let borgmatic run shell scripts outside its container. I thought I could circumvent this problem by including var/run/docker.sock in the docker-compose file but without success.

I’m sure this can be solved somehow. For the time being, I have more important problems to solve and thus I’ll live with the assumption “there are no data inconsistencies³.

The SSH-Daemon of my server puts its authentication logs to /var/log/auth.log, but could also be named access.log or similar in other applications. Services like caddy or gitea also create auth log files. ↩︎
However, what I yet do not understand is how fail2ban is able to access the iptables from within the container. I thought a positive side effect of containerization is encapsulation. Or is that only complete with “rootless”-Containers? ↩︎
Background: I’m currently the only one contributing to my gitea instance. My website is static. The webserver caddy only handles changing files once pushed to Gitea. I have scheduled application updates per watchtower and unattended-upgrades (see my follow-up post) at night so they won’t interfere with borgmatic runs. Just the logs for fail2ban are still written independently. I accept the risk of data loss here because packet filtering is subject to change anyways. ↩︎

Jekyll build: broken relative links

Schallbert — Fri, 29 Dec 2023

Problem

Now, after eternal trial and error, I am finally able to provide my Caddy server with the build files via Docker-Volume. Unfortunately, the page looks like this:

Analysis

At first I suspected the web server. But when I call up the website in my browser¹ as a test, I see the same behavior. Strange.

Obviously all stylesheets are missing. But if you look closely, images and other media have not been loaded either. Only their descriptions. Then I click wildly on a few links to articles and subpages. I notice that the paths look different than I would have expected:

# --- Expectation ---
# Image
/assets/images/test.jpg
# Post
/this-is-a-post

Instead I see:

# --- Observation ---
# Image
/pages/<username>/<repositoryname>/assets/images/test.jpg
# Post
/pages/<username>/<repositoryname>/this-is-a-post

This is also shown by the browser’s developer options activated with F12 (see image above): all sources integrated via a relative link cannot be loaded. It is generally a clear recommendation to have the developer options activated in the event of problems with the display of websites.

It is somehow obvious that my Jekyll configuration file _config.yml or my translation header i10.yml contain errors. Fortunately, there are already excellent help pages.

However, a test quickly shows that I cannot leave the attribute repository empty, where the build <username>/<repositoryname> emanates from:

# bundle exec jekyll serve
Generating... 
 Jekyll Feed: Generating feed for posts
 Liquid Exception: No repo name found. 
 Specify using PAGES_REPO_NWO environment variables, 'repository' in your configuration, 
 or set up an 'origin' git remote pointing to your github.com repository. in /_layouts/default.html
 ERROR: YOUR SITE COULD NOT BE BUILT

And, heck, where does the /pages suddenly come from?

Solution

As /pages is added to paths I suspect Jekyll build options and Github-Pages Gem to be the culprits.

Of course I’m not alone with this problem, so I found the solution documented.

I just change JEKYLL_ENV (build environment) in the action script from production to development, like it is normally done for building on a local machine.

# workflows/jekyll-build-action.yml
env:
 JEKYLL_ENV: development # I had "production" here before

And what can I say: as soon as you do it right, it works!

Displaying the website offline works as follows: Drag the built files (the _site folder) to your own computer and open the top-level index.html with the browser. ↩︎

🗸 Gitea Actions Part2 - Jekyll-Dockerimage

Schallbert — Sat, 09 Dec 2023

What happened so far

I wasn’t able to set up the CI/CD pipeline in Part1 of this sequel. I tried with a minor modified copy of the Github-Action to build Jekyll for Github Pages. Now I wanted to try running Jekyll as a Dockerimage right from the start. There wasn’t too much choice on the Docker hub, still I found something which seemed well-documented and actively maintained at year-end 2023.

“Set up job”

So I add the Docker-jekyllimag to my workflow:

# workflows/jekyll-build-action.yml
jobs:
 # Build job
 build:
 runs-on: ubuntu-latest # this is the "label" the runner will use and map to docker target OS
 container: jvconseil/jekyll-docker
# [...]

Here I’m telling gitea act_runner to run with ubuntu-latest label which in my case points to a minimalistic node16:bullseye machine (Debian11). I’m then telling docker to load the jekyll-docker image, where the dependencies I require for my build are already loaded.

bundle install

Well, initial build fails due to ruby dependencies that are specified in my project’s Gemfile which are not present yet. The error message looks like this:

bundler: failed to load command: jekyll (/usr/gem/bin/jekyll)
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.4.22/lib/bundler/resolver.rb:332:in `raise_not_found!': Could not find gem 'github-pages' in locally installed gems. (Bundler::GemNotFound)
	from /usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.4.22/lib/bundler/resolver.rb:392:in `block in prepare_dependencies'

Bundler is Ruby’s packet manager. Similar to npm for Javascript or pip für Python, bundler is able to load and bind dependencies and libraries for applications running in Ruby. The Gemfile specifies the packages for bundler to install.

So let’s add bundle install to the script like so:

# workflows/jekyll-build-action.yml
# [...]
 steps:
 - name: --- CHECKOUT ---
 uses: actions/checkout@v3
 - name: --- INSTALL GEMS ---
 run: bundle install # will fail with permissions rights to write to Gemfile.lock but anyways installs required dependencies.
 - name: --- BUILD WITH JEKYLL ---
 # Outputs to the './_site' directory by default
 run: bundle exec jekyll build --destination /opt/blog_staging
 env:
 JEKYLL_ENV: production

act_runner: no write permissions to `Gemfile.lock`

This fails again. act_runner:

gitea-runner-1 | [Deploy Jekyll site/build] | `/workspace/schallbert/blog/Gemfile.lock`. It is likely that you need to grant
gitea-runner-1 | [Deploy Jekyll site/build] | write permissions for that path.
gitea-runner-1 | [Deploy Jekyll site/build] ❌ Failure - Main ---INSTALL GEMS ---
gitea-runner-1 | [Deploy Jekyll site/build] exitcode '23': failure

After a lenghty search I saw that Gemfile.lock wasn’t added to version control on my local machine because it is mentioned in .gitignore. So bundler in act_runner now tries to create it out of the given Gemfile without success as it only has read access to the checked-out dataset.

To solve this, I add Gemfile.lock to version control, removing it from .gitignore. As an additional benefit I make sure to have identical build environments locally and remote.

Dependency: `sass-embedded`

Less than 20sec into the build I get the following error:

Resolving dependencies...
Could not find gem 'sass-embedded (= 1.69.5)' with platform 'x86_64-linux' in
rubygems repository https://rubygems.org/ or installed locally.

After some more searching, I found a resolution: Specify a certain version of jekyll-sass-converter in the action or, alternatively, add the gem “github-pages” to the Gemfile. The latter manages to load a working version of CascadingStyleSheets (CSS) Preprocessor Sass.

Finally, bundle install finishes successfully:

✅ Bundle complete! 5 Gemfile dependencies, 43 gems now installed.

jekyll build

As I want act_runner to save the build output on my host machine, I add a Volume to Gitea’s docker-compose.yml:

# gitea/docker-compose.yml
# [...]
volumes:
 - ./runner/blog_staging:/opt/blog_staging
 - ./runner/data:/data
 - /var/run/docker.sock:/var/run/docker.sock

Let’s retry the build:

 Destination: /opt/blog_staging
 Generating... 
 Jekyll Feed: Generating feed for posts
jekyll 3.9.3 | Error: Permission denied @ dir_s_mkdir - /opt/blog_staging
/usr/local/lib/ruby/3.2.0/fileutils.rb:406:in `mkdir': Permission denied @ dir_s_mkdir - /opt/blog_staging (Errno::EACCES)
	from /usr/local/lib/ruby/3.2.0/fileutils.rb:406:in `fu_mkdir'

Also this problem can easily be solved: Either you build to /tmp, where the jekyll user has access. Or you hand over the output directory like so: chown -R jekyll /your/build/output.

With this change, I get successful action runs.

Transfer artifacts

Where do I find the build artifacts now? I don’t see any _site folder: Neither on the host machine’s volume, nor in the Gitea or Runner containers.

Docker volumes for `act_runner` cannot share build artifact

This seems logical as I learn the mechanics of act-runner: It will spawn an own action Dockerimage with own volumes that are associated by Task-ID.

DRIVER VOLUME NAME
--> local GITEA-ACTIONS-TASK-84_WORKFLOW-Deploy-Jekyll-site_JOB-build <--
--> local GITEA-ACTIONS-TASK-84_WORKFLOW-Deploy-Jekyll-site_JOB-build-env <--
local act-toolcache
local blog_staging

If I do not extract the build artifacts from the action run, they will be thrown away together with the action run when it has completed and everything is gone. The runner’s volume set up above doesn’t touch the action as it again executes in an isolated image.

To really understand this has cost me a lot of time and many failing action runs. Don’t repeat my mistakes and read the according section in Gitea’s documentation thoroughly.

Well, this turns out more complicated than I thought. As a single maintainer of my website, this is like taking a sledgehammer to crack a nut. Anyways, I’ll follow through.

But I can use `upload-artifact`, right?

Right? Well, at least I thought so. I modified my Action script accordingly and used the upload function from the Github Actions marketplace:

# workflows/jekyll-build-action.yml
# [...]
# Automatically upload the build folder to Giteas blog repo folder
 - name: --- UPLOAD ARTIFACT ---
 uses: actions/upload-artifact@v3
 with:
 path: /workspace/schallbert/blog/
 name: Blog_Staging
 retention-days: 2

Great, now I have a downloadable Zip file on Gitea’s web surface:

But how do I now get that downloaded to the server’s disk? The action is fully encapsuled and cannot even access its underlying Docker Daemon (for good reasons, I guess).

So I search for the assets in the Gitea-Container that persists the artifacts at/data/gitea/actions_artifacts/BUILD_ID. Unfortunately, they are a heap (hundreds) of .chunk.gz files with cryptic numbers as name where I don’t know how to merge into one single archive.

And there’s another thing I don’t like about upload-artifact: My website is becoming bigger and bigger due to an increasing amount of media that I use. That’s why the uploader requires nearly two minutes for compressing and packing - with a linear ascending outlook.

Dead end.

Also Docker `cp` won’t work here

Then I try using Docker’s copy operation to get the artifacts to my host machine:

# workflows/jekyll-build-action.yml
# [...]
- name: --- COPY ARTIFACT ---
 run: docker cp gitea-runner-1:/workspace/schallbert/blog /tmp/blog_staging

This fails like so:

gitea-runner-1 | docker cp gitea-runner-1:/workspace/schallbert/blog /tmp/blog_staging
[...]
gitea-runner-1 | [Deploy Jekyll site/build] | /var/run/act/workflow/3.sh: line 2: docker: not found
gitea-runner-1 | [Deploy Jekyll site/build] ❌ Failure - Main --- COPY ARTIFACT ---

This was expected as the action container knows no Docker. I cannot move files from “inside” the container to another one. Or can I? By forwarding the host machine’s Daemon docker.sock, according to Gitea documentation, I might be tempted… No, I will not continue here - see above, I’d puncture encapsulation if I did.

Transfer artifacts via SFTP

I’m running low on options. I think I could copy the artifacts via SFTP from within the act_runner’s action image into the Webserver-Container¹.

Luckily, there already is an Action for this: scp-files. Now I have to modify the webserver’s container in a way to be accessible via SCP, inhale an SSH-key, and save incoming artifacts.

On the other hand, people in multiple forums say that this is not a clean solution. “Ac container should only accommodate one application. But there’s a solution again: A docker container with only the SSH-Daemon installed. If I have a shared volume between that one and my server, I might be successful.

Looks like more work still. Another container, piping files out and back into my system - that just doesn’t seem right. So I continue my research.

Handover a volume to the Action-Container

All these failures and dead ends have been with me for over a month now. I still really want to be able to build, test and publish automatically. I’ve learned a lot along the way and now I’m hoping that this approach will finally get me to the solution.

During my search I came across the option to change the configuration of the runner. That way, I might end up including a volume that I can share between the action and my server via the daemon on the host.

If that works, I wouldn’t have an additional security risk like a publicly available file transfer container or the long waiting times caused by upload-artifact. So let’s get to work! There are more reasons why volumes can make sense in the respective action - so there must already be people out there who have managed this.

So I add a volume to the Action script:

# workflows/jekyll-build-action.yml
# [...]
container: 
 image: jvconseil/jekyll-docker:latest
 volumes: 
 - /tmp/blog_staging:/blog_staging
 - /opt/cache:/opt/hostedtoolcache

But the artifacts still don’t show up on my host’s filesystem. I strip down the action as much as I can to just check if the volume creates a folder in the action container:

# workflows/jekyll-build-action.yml
# [...]
steps:
 - name: --- CHECK_VOLUME ---
 run: |
 ls -al /blog_staging

But still:

gitea-runner-1 | [Deploy Jekyll site/build] [DEBUG] Working directory '/workspace/schallbert/blog'
gitea-runner-1 | [Deploy Jekyll site/build] | ls: /blog_staging: No such file or directory
gitea-runner-1 | [Deploy Jekyll site/build] ❌ Failure - Main --- CHECK_VOLUME ---

Some more, fruitless tries (maybe I had misspelled anything?) and extensive research in multiple forums I found out that there is a valid_volumes attribute in act. If I don’t have the volume added here, it won’t bind to the container.

OK, let’s add it to the action script:

# jekyll-build-action.yml
# [...]
container: 
 image: jvconseil/jekyll-docker:latest
 valid_volumes: 
 - '**' # This does not work. Also specialized lists indicating the volumes directly won't work
 volumes: 
 - /tmp/blog_staging:/blog_staging
 - /opt/cache:/opt/hostedtoolcache

Still nothing. At least, now, I get a warning in the logs:

gitea-runner-1 | [Deploy Jekyll site/build] [/tmp/blog_staging] is not a valid volume, will be ignored
gitea-runner-1 | [Deploy Jekyll site/build] [/opt/cache] is not a valid volume, will be ignored

Something is wrong with the transfer of this option. There are reports in which the registration of the volumes works. So I take the instructions and create a configuration file where I enter the following:

# runner/config.yml

# Volumes (including bind mounts) can be mounted to containers. Glob syntax is supported, see https://github.com/gobwas/glob
 # You can specify multiple volumes. If the sequence is empty, no volumes can be mounted.
 # For example, if you only allow containers to mount the `data` volume and all the json files in `/src`, you should change the config to:
 # valid_volumes:
 # - data
 # - /src/*.json
 # If you want to allow any volume, please use the following configuration:
 # valid_volumes:
 # - '**'
 valid_volumes: ["/tmp/blog_staging", "/opt/hostedtoolcache"]

Important: The notation must be a string, comma-separated, and always has to specify the Source of the volume. In other words, the part that comes before the :. However, the error message “not a valid volume” still appears when building.

But then I realize that the config must be made available to the act_runner itself as a volume - otherwise the runner running in the container cannot access it at all!

So my docker-compose.yml for Gitea, section “runner” now looks like this:

# gitea/docker-compose.yml
# [...]
 runner:
 image: gitea/act_runner:latest
 environment:
 - CONFIG_FILE=/config.yml
 - GITEA_INSTANCE_URL=https://git.schallbert.de
 - GITEA_RUNNER_NAME=ichlaufe
 - GITEA_RUNNER_REGISTRATION_TOKEN= <redacted>
 volumes:
 - ./runner/config.yml:/config.yml
 - ./runner/data:/data
 - /opt/hostedtoolcache:/opt/hostedtoolcache
 - /var/run/docker.sock:/var/run/docker.sock

And, finally:

gitea-runner-1 | ls -al /blog_staging
gitea-runner-1 | [Deploy Jekyll site/build] | total 8
gitea-runner-1 | [Deploy Jekyll site/build] | drwxr-xr-x 2 root root 4096 Dec 27 07:32 .
gitea-runner-1 | [Deploy Jekyll site/build] | drwxr-xr-x 1 root root 4096 Dec 27 07:46 ..
gitea-runner-1 | [Deploy Jekyll site/build] ✅ Success - Main --- CHECK_VOLUME ---

What an act. I’m so glad that everything is running smoothly now and that I can actually find the build files on my local host system! 🥳

One last tip

If you work with a reverse proxy like I do and get strange connection refused error messages when starting up the runner like:

docker compose up
[+] Running 2/0
 ✔ Container gitea Created 0.0s 
 ✔ Container gitea-runner-1 Created 0.0s 
Attaching to gitea, gitea-runner-1
gitea-runner-1 | level=info msg="Starting runner daemon"
gitea-runner-1 | level=error msg="fail to invoke Declare" error="unavailable: dial tcp <address>: connect: connection refused"
gitea-runner-1 | Error: unavailable: dial tcp <address> connect: connection refused
gitea | Server listening on :: port 22.
gitea-runner-1 exited with code 1

Then your reverse proxy is either not correctly configured or - like in my case - shut down.

Using SFTP (Secure File Transfer Protocol) which is file sharing per ssh (secure shell), I’d increase attack surface of my system just a bit. This is because the SSH client would be reachable for anyone and not just for my action. ↩︎

🗙 Gitea Actions - Jekyll Workflow

Schallbert — Thu, 30 Nov 2023

After setting up my own little server in this project and adding all necessary applications to serve my blog, I now want to automatically build, integrate, and deploy.

Desired outcome

This is how I want it to be: When my Gitea instance detects a git push to the remote origin of my blog’s repository, I want an action runner to start a Docker container. That container shall load Ruby, Bundler, Jekyll, and all other required dependencies. Then it shall checkout the blog’s source files and then run the bundle exec jekyll build command. When its output is ready in the _site folder, I want the runner to self-terminate and to free all occupied resources.

At this point I’d like to manually check my blog’s integrity, maybe also add some automated checks like detecting broken links or similar.

Finally, I want to be able to have the site deployed and thus made available to the public with another simple command.

Starting point: Github Actions

As usual, I first search other people’s solutions for the same problem. I found the Github Actions starter-workflows Repository, created by Github themselves, and copied most of it into my action as a first shot:

# jekyll-build-pages.yml
# Sample workflow for building and deploying a Jekyll site
name: Deploy Jekyll site
run-name: ${{ gitea.actor }} builds Jekyll site
on: [push]

jobs:
 # Build job
 build:
 runs-on: ubuntu-latest
 steps:
 - name: Checkout
 uses: actions/checkout@v3
 - name: Setup Ruby
 uses: ruby/setup-ruby@55283cc23133118229fd3f97f9336ee23a179fcf # v1.146.0
 with:
 ruby-version: '3.1' # Not needed with a .ruby-version file
 bundler-cache: true # runs 'bundle install' and caches installed gems automatically
 cache-version: 0 # Increment this number if you need to re-download cached gems
#[...]

After uploading this workflow to my blog at workflows/jekyll.yml, the runner indeed started its work:

Problems installing Ruby

That was a quick win. On the other hand, the runner wasn’t even able to successfully install Ruby. Here’s the corresponding log:

::error::The current runner (debian-11-x64) was detected as self-hosted because the platform does not match a GitHub-hosted runner image 
(or that image is deprecated and no longer supported).
In such a case, you should install Ruby in the $RUNNER_TOOL_CACHE yourself, for example using https://github.com/rbenv/ruby-build
You can take inspiration from this workflow for more details: 
https://github.com/ruby/ruby-builder/blob/master/.github/workflows/build.yml

The error message looks reasonable and is well-placed.

Background: My runner operates in Docker which uses a stripped-down Debian image named node:16-bullseye (Referenz). Looks like it is incompatible with the Github Actions Runner which executes atop Ubuntu22.

There are quite some options to resolve:

Troubleshooting options

Run the runner on Ubuntu22, use labels for that.
Don’t use Docker but have the runner operate on the host’s operating system
Use prebuilt Ruby, installed in the runner’s cache
Use an out-of-the-box Jekyll-Dockerimage to not require additional installation steps

Operate the runner with Ubuntu22 instead of Node16

There is a matching Dockerimage, but it will take a lot more disk space and RAM than a minimalistic one. Plus, it takes a little more time until it is live. Let’s see whether there is a more simple solution.

Run the runner on the host’s operating system

This has considerable security drawbacks as unprotected branches might allow third-party injected runners via git push --force. In an extreme case, it would blindly execute malware right on my server system. In addition, I’d kill portability by hardwiring the runner to a machine’s OS.

Use pre-built Ruby in runner toolcache

As proposed by the error message, I followed another manual to map a volume containing prebuilt ruby on my host using Gitea’s docker-compose.yml. In addition, I assigned the targeted folder to the RUNNER_TOOL_CACHE environment variable:

# gitea/docker-compose.yml
# [...]
runner:
 image: gitea/act_runner:nightly
 environment:
 - GITEA_INSTANCE_URL=<redacted>
 - GITEA_RUNNER_NAME=<redacted>
 - RUNNER_TOOL_CACHE=/opt/hostedtoolcache
 - GITEA_RUNNER_REGISTRATION_TOKEN= <redacted>
 volumes:
 - ./runner/data:/data
 - /opt/hostedtoolcache:/opt/hostedtoolcache
 - /var/run/docker.sock:/var/run/docker.sock

Here, I map the host’s filesystem path /opt/hostedtoolcache to Docker’s /opt/hostedtoolcache. Docker creates the folders automatically “on both sides” if they are not yet existing on next docker compose up.

Let’s try to install Ruby on the host machine.

So I download the ruby-build repository via git clone and have the installer run ./ruby-build/install.sh. Then, I execute the build with the requested target path from the error message:ruby-build 3.1.4 /opt/hostedtoolcache/Ruby/3.1.4/x64

==> Downloading openssl-3.1.4.tar.gz...
 % Total % Received % Xferd Average Speed Time Time Time Current
 Dload Upload Total Spent Left Speed
100 14.8M 100 14.8M 0 0 81.3M 0 --:--:-- --:--:-- --:--:-- 81.1M
==> Installing openssl-3.1.4...

BUILD FAILED (Ubuntu 22.04 on x86_64 using ruby-build 20231114)

Problems with prebuilt-Ruby

The logs shows the problem:

[...] No C compiler found, please specify one with the environment variable CC [...]

So I type apt install build-essential and confirm with which gcc that I now have a C compiler installed at /usr/bin/gcc Let’s try again!

crypto/comp/c_zlib.c:36:11: fatal error: zlib.h: No such file or directory

Hmm, let’s load it via apt install libz-dev and re-run the insall. This time it takes a while, then:

*** Following extensions are not compiled:
openssl:
 Could not be configured. It will not be installed.
 /tmp/ruby-build.20231205114312.9174.0gmhNf/ruby-3.1.4/ext/openssl/extconf.rb:100: OpenSSL library could not be found. You might>
 Check ext/openssl/mkmf.log for more details.
readline:
 Could not be configured. It will not be installed.
 /tmp/ruby-build.20231205114312.9174.0gmhNf/ruby-3.1.4/ext/readline/extconf.rb:62: Neither readline nor libedit was found
 Check ext/readline/mkmf.log for more details.
*** Fix the problems, then remove these directories and try again if you want.
[...]

OK, openssl this time. Get it with apt install libssl-dev and apt install libreadline-dev and finally :

Installed ruby-3.1.4 to /opt/hostedtoolcache/Ruby/3.1.4/x64

!🎉!

Still no success with the action

Strange, it is the same error all over again:

💬 ::debug::isExplicit: 3.1.4

gitea-runner-1 | [Deploy Jekyll site/build] 💬 ::debug::checking cache: /opt/hostedtoolcache/Ruby/3.1.4/x64
gitea-runner-1 | [Deploy Jekyll site/build] | ::debug::checking cache: /opt/hostedtoolcache/Ruby/3.1.4/x64
gitea-runner-1 | [Deploy Jekyll site/build] 💬 ::debug::not found
gitea-runner-1 | [Deploy Jekyll site/build] | ::debug::not found
gitea-runner-1 | [Deploy Jekyll site/build] ❗ ::error::The current runner (debian-11-x64) was detected as self-hosted

This results from the following Action code:

// setup-ruby/ruby-builder.js
if (common.shouldUseToolCache(engine, version)) {
 inToolCache = common.toolCacheFind(engine, version)
 if (inToolCache) {
 rubyPrefix = inToolCache
 } else {
 const toolCacheRubyPrefix = common.getToolCacheRubyPrefix(platform, engine, version)
 if (common.isSelfHostedRunner()) {
 const rubyBuildDefinition = engine === 'ruby' ? version : `${engine}-${version}`
 core.error( [...] )

Check docker volumes

Hmpf. Let’s see whether Ruby is really available in that volume. To do this, I use docker ps to get the runner’s container ID and enter it here:

docker exec -it <containerID> bash

Then I navigate to cd /opt/hostedtoolcache, list its contents with ls and there it is: Ruby. It is there but still cannot be found in the Action.

Accept my failure

No way to proceed for me at this point 😖. I opened this problem to the community as act_runner cannot find hostedtoolcache and will try

Another option.

Root cause

Looks like the support of setup-ruby action for self-hosted runners like mine might have been discontinued. Maybe the documentation, creating the above error message, has not been updated accordingly.

Multi-language site - l10n

Schallbert — Tue, 14 Nov 2023

Providing a website in different languages is called “internationalization” - in short i16n or “localization” - l10n. If you search these abbreviations in the web, you’ll find a lot of useful material. By the way, the numbers in the brief version stand for the letter count in the word’s center.

Motivation

Right from the start, I was asked to prepare a German version of this website. To date I considered this too much effort and this only changed half a year ago when I was verbally asked by multiple people independently from each other.

So I invested half a year of evenings to get the heavy lifting done, transcribing roughly 80 articles. But I still wasn’t quite done.

Jekyll extension

I looked at a couple of extensions for Jekyll dedicated to multilingual support like demondehellis’s plugin-free solution, untra’s polyglot plugin, kurtsson’s jekyll-multiple-languages-plugin and some other solutions to publish a website in multiple languages.

So I had to decide for one. The plugin-less solution was out quickly because I wanted a concise naming convention where identical posts in different languages have identical names. Plus, I wanted interlinks to be persistent for the selected language. Something would have to keep language pots separated.

After some research, I found leo3418’s wonderful series of blog posts for implementing Polyglot - and so I decided to go with that one.

Polyglot: Installation

Installation of PolyGlot is simple because both the documentation in the repository’sreadme.md and leo3418’s related post do instruct these first steps in detail. They do this so well so I won’t lose another word about it here.

Restructuring the website

Polyglot offers two possibilities to adapt the site’s structure for multilanguage support:

Use the language abbreviation in the post’s filename like this: 2023-11-14-jekyll-polyglot-language-support-en.md, repeat for each article in a different language with the appropriate abbreviation
Create folders named like the language code, e.g. /en and put all english content in there, but keep file names the same between languages.

I decided for the latter option out of laziness - I didn’t want to rename existing articles. Polyglot manages to stick to the same language when navigating a site. If you, for instance, click https://schallbert.de/milling-small-parts/, Polyglot will assume you want the English version to be presented and thus provide it. Where this doesn’t work anymore is section titles as they are translated themselves, invalidating attached links, so the section link #restructuring-the-website will lead to a 404 fault when called in the German version.

Thus, when using section titles, the full post’s path has to be noted.

This is getting a little more complicated. Jekyll’s _config.yml is intended for one version of the website only, so we cannot just create two variants of this file and expect it to work. To still enable full translation, we’ll have to overwrite language sensitive parameters like site.title or site.description externally depending on selected locale.

I have solved this adding l10n.yml files in the _data folder that covers these parameters for each language, and then packing them into a language-code folder like we’re used to do with our posts. To show their content, the relevant liquid parameters in html files (_includes folder) have to be modified to now search there instead of taking the standard config. Example: {{ site.data.ui-text[site.data.l10n.locale].follow_label | remove: ":" | default: "Follow" }}. German version of this site will show FOLGEN: while the English one says FOLLOW:.

Language selector

Of course I want to offer the option to select a locale on every page. To do this, I added the language code and its flag to the footer. On click, it will reload the page in the requested locale. The following code I wrote in Liquid enables this behavior:

{% raw %}
{% for lang in site.languages %}
 {% assign url_parts = page.url | split: "/" %}
 {% assign lang_code = lang %}

 {% assign collection_name = url_parts[1] %}
 {% assign post_url = url_parts | last %}

 {% if lang == site.default_lang %}
 {% if collection_name == post_url %}
 {% assign lang_url = "/" | append: post_url %}
 {% else %}
 {% assign lang_url = "/" | append: collection_name | append: "/" | append: post_url %}
 {% endif %}
 {% else if url_parts.size > 2 %}
 {% assign lang_url = "/" | append: lang_code | append: "/" %}
 {% if collection_name == post_url %}
 {% assign lang_url = lang_url | append: post_url %} 
 {% else %}
 {% assign lang_url = lang_url | append: collection_name | append: lang_url | append: post_url %}
 {% endif %}
 {% endif %}
 <a href=" {{ lang_url }}">{{ site.data[lang].l10n.lang_name }}</a>
{% endfor %}
{% endraw %}

Here I’m splitting the current page url. The last section will be the post_url. The algo assumes that the first part is the post’s collection name, e.g. posts-hardware. The following logic inserts the language code into the url if and where needed so that the correct full url is built depending on the used collection and page navigation depth.

Publishing

I was pretty proud of this elegant way of selecting languages. I thing it all fits in nicely and the multi-language support potentially adds a good number of readers. The big downer is the duplicated work for creating each post in two languages.

Unfortunately, the site now wasn’t compatible anymore with Github Pages - the auto release would show each page twice instead of keeping them separated by language. Two options remained: creating custom Github Actions to publish on Github Pages, or to move the whole site to an own server. I’ll decide for the latter to be on the safe side regarding data protection regulations because in that case I can make sure the hoster is situated in Germany.

Well, this upcoming project is too big to take on in this post.

Github Actions: Troubleshooting

Schallbert — Tue, 14 Feb 2023

Motivation

Why I’m writing a short post about Github Actions? Because I wanted to use it for my latest Software-project, QR-codengrave. When at some distant point in the future, I have a new computer, don’t remember which IDE I used to build and run, test, and create assets with. Or, say, I have a corrupted virtual environment or launch.json, I still want to be able to deploy that application, publish bugfixes or bump a release.

Although that’s not the usual argument for using CI/CD - typically the first reason being that collaboration on a piece of software becomes easier - still the appeal is strong enough to try Github’s automation called Github Actions.

I won’t try to run nightlys or have a continuous deployment pipeline as an end in itself, though, and only want an automated build, lint, test run, and deploy stuff when merging back to the production branch.

Got it, but why a dedicated post?

Because it was such a pain to get it right. I spent hours and hours pushing and hoping that, this time, the goddess of Github Actions would actually have my solution built without errors and at least run some of the tests.

Note: If I had been more diligent, I would have installed yet another tool like this to have my actions run locally, both saving time and the embarassment of tens of failed builds in a row. But I wasn’t.

I had quite some errors that appeared on the way of making QR-codengrave. Some I wasn’t even able to properly resolve so I had to utilize work-arounds. But let’s get to that later.

The script

Github Actions uses YAML to take orders for its pipelines. It is very well documented and provides many readily-working scripts for most scenarios and programming languages.

The so-called “Workflow file” looks like this:

# This workflow will install Python dependencies, run tests and lint with a single version of Python
# For more information see: https://docs.github.com/en/actions/automating-builds-and-tests/building-and-testing-python

name: python_integrate

on:
 push:
 branches: [ "master" ]
 pull_request:
 branches: [ "master" ]

permissions:
 contents: read

jobs:
 build:

 runs-on: ubuntu-latest

 steps:
 - uses: actions/checkout@v3
 - name: Set up Python 3.10
 uses: actions/setup-python@v3
 with:
 python-version: "3.10"
 - name: Install dependencies
 run: |
 python -m pip install --upgrade pip
 pip install flake8 pytest
 pip install qrcodegen # Dependency install of qrcodegen
 if [ -f requirements.txt ]; then pip install -r requirements.txt; fi
 - name: Lint with flake8
 run: |
 # stop the build if there are Python syntax errors or undefined names
 flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics
 # exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide
 flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics
 - name: Test with pytest
 uses: GabrielBB/xvfb-action@v1  # Diverts tkinter GUI to a virtual frame buffer (VFB)
 with:
 run: |
 pytest

And it does not divert too much from the standard template that I was using first. The only things I added are the run pytest entry and the usage of a virtual frame buffer to cirumvent issues with my GUI which makes the tests fail when the GUI tries to fire up and there’s no screen to show it on.

Github Action runner errors

The following list of fails describe the errors and resolutions that I faced until I had a stable CI at around action run #50.

Directory mismatch?

Output:

/opt/hostedtoolcache/Python/3.10.9/x64/lib/python3.10/importlib/__init__.py:126: in import_module
 return _bootstrap._gcd_import(name[level:], package, level)
test/test_machinify_vector.py:4: in <module>
 from bin.platform.machinify_vector import MachinifyVector, Tool, EngraveParams
E ModuleNotFoundError: No module named 'bin'

Reason: I placed my source files under /bin and made it available to a local instance of PyInstaller (the tool I chose to render my python sources into an executable under Windows). Upon upload, Github Actions couldn’t relate to the paths I chose and had that error prepared for me.

Resolution: Add an empty file with name __init__.py into the /bin folder. This will flag anything in that folder as package, thus making it available to GH actions. So what looked like a directory mismatch actually was a package-not-found error that my IDE didn’t have as it knew which files I had created.

YAML syntax errors

This error occurs because I was trying to put two workflows into one file. Somehow Github Actions seems to only accept a single one per file.

Path-not-found errors

I had a lot of those and they were painful to resolve. The folder structure that I was targeting looked like:

- assets  # images, persistence file, etc.
- src  # source files
- test  # pytest files for unit and integration testing
- dist  # build artifacts
- build  # build process files

But somehow, it was super hard to make my local IDE build environment, Pyinstaller, and the Github Actions Worker and its trigger of Pyinstaller to cooperate. One of those four parties would always complain that a path is missing or something else was wrong. That’s why I decided to abandon relative paths and instead use python’s importlib_resources. But in the end, this turned out to be problematic with Pyinstuller on remote Github Actions.

What finally solved the problem was to move assets into src. This way, the ./ command couldn’t go wrong anywhere and although I don’t like the construct very much, I was tired of putting more time into that for a clean fix (If you know how to handle path pointing in Python for both local and remote, let me know in the discussions).

Tkinter headless testing

Later in the development process, I decided to add some integration tests into my solution. This way I wanted to make sure that child windows would actually perform the callbacks to update the main application, and vice versa e.g. that the persisted tool list is cascaded into the tool configuration window.

These tests were running locally but with a disadvantage: When I simulated an error or warning case, the corresponding messageBox would insist to be closed manually by the user before continuing the tests. Which was just a bit annoying for me in the IDE to do, but would just be impossible to do from within Github Action’s CI pipeline.

So I bit the bullet and added a wrapper class to tkinter’s messageBox so I could inject a mock that wouldn’t actually trigger the popup:

class MsgBox:
 """Re-implementation due to testing purposes: With this trick, we are able to mock these windows
 so we do not have to wait for users to manually close the dialog, unblocking the application again."""
 def showinfo(self, title, message):
 showinfo(title=title, message=message)

 def error(self, title, message):
 showerror(title=title, message=message)

Although in theory it should be possible also to invoke the popup’s “OK button” from within the test, I wasn’t able to have that automated reliably.

When this worked, I pushed to Github with high expectations - and got another beautiful error:

_tkinter.TclError: no display name and no $DISPLAY environment variable

At least that was easy to understand: Tkinter just didn’t know where to draw the windows - seems legit when there’s no display connected.

So I searched the internet and found a single line of code that miraculously solved my problem by introducing a virtual frame buffer:

uses: GabrielBB/xvfb-action@v1 # Diverts tkinter GUI to a virtual frame buffer (VFB)

Now, on Push, Github Actions rained green ticks down on me which felt really good for a change.

QR-codengrave V1.1 / V1.2

Schallbert — Sun, 12 Feb 2023

In a nutshell

Faster Engrave algorithm
Several bugfixes
Usability improvements

Demo video

QR-codengrave Quick Demo↗

Version 1.1

Engrave algorithm upgrade

The algorithm now scans the lines horizontally
One line “Left-to-Right”, next line “Right-to-Left” etc. to reduce machine travel
prefers lines over dots, and longer lines over shorter ones for time optimization
draws lines vertically or horizontally
rapid move positioning now is packed into vectors that can be called to new Points(X, Y) simultaneously, not only one dimension at a time
New algorithm just needs 80% of the time for engraving compared to the old one

Bugfixes

splash image (qruwu.jpg) is not shown on startup when Persistence.dat is not available
Crash when pressing Stop Draw although drawing has never been initiated before
Crash when adding a tool on an empty selecion list
Initialization of application with no tool selected disables dropdown menu
Fix a calculation error in QR-code size estimation
Fix an error pixel width calculation for tapered tools
Actually the currently selected tool is not shown “selected” in the dropdown on startup
QR-codes can grow larger than the display area

Usability improvements

Change default tool name from name to default for clarity
Make non-tapered (V-cut) tool the default tool for better usability
Setting a new XY-zero offset became more intuitive (Y down is negative, X right is positive)

Version 1.2

Minor change over 1.1, most importantly a Bugfix that could lead to tool change macro not being called. In addition, tapered tool angles are now taken into account when calculating track width and QR-code engrave size.

Bugfixes

Tool select Tx and tool change command M06 are now in the same line of the G-code to correctly trigger tool change macros
Actual selected tool not shown on startup, instead the first tool of the list was shown
QR-code drawing could grow larger than the screen. This is fixed now.
Crash fix when trying to update status although no engrave parameters available

Usability improvements

Add application version to G-code as comment
Add qr_ prefix to G-code save-to dialog’s default name
Improve job duration estimate

Availability

The latest release is available for download here, free of charge for home and private use. For more info on the project, please find the readme and the background information linked here.

Migration from an earlier version

It is possible to copy the persistence file to the new version’s directory. This way, saved tool lists, engrave parameters, and XY0 offsets can be ported to the new version. How it’s done:

Navigate into src/assets of your QR-codengrave folder.
copy Persistence.dat to a temporary folder or e.g. your Desktop.
Remove the current installation by deleting the folder.
Download the latest release, move it where your previous release was deployed.
Move the Persistence.dat into src/assets of your new installation and start the application.

Convert videos with VLC player

Schallbert — Wed, 27 Oct 2021

Website: media creation

For this website, I have been converting media files a lot. You know, the photos I shoot are really big which leads to long page load times and screen-filling images. While it is quite easy to reduce their size, and increase JPEG compression, it is a lot harder to do so with videos.

The problem

WTF, VLC?! I’m using the popular vlc player and somehow I didn’t seem to get it right. Either my video doesn’t turn out in greyscale as I planned, the output file is empty (Use H.264, instead of H.265 to fix), the aspect ratio seems distorted, or the video suddenly has rotated which I neither wanted nor expected to happen.

Items to cover

This series of steps help me (and maybe you, too) how to make simple video conversions like

How does the filter stuff work?

Preview filters

VLC has a quick-preview feature for the most common filters in the Tools -> Effects and filters menu. This directly affects the video you’re viewing, but won’t affect your saved file. It’s just meant for you previewing and controlling the effect properties these filters will have on your video. View this VideoProc blog entry for in-depth guides - and - more importantly - many images stepping you through. For the impatient fraction: continue reading 😉

Save filtered video to file

To save a filter that you selected for your preview, you must add them to the conversion profile. As long as they remain ticked, any save with this profile will modify your video, eventually even although you unticked all changes you made to the preview filters in the past!

How to change colors in my video?

Open the video file you want to modify.
Click Tools -> Effects and filters (or press Ctrl+E)
A dialog with three tabs opens. Select the Video Effects tab.
Under Essentials, tick Image Adjust.
Make your changes and hit Save, then Close. A preview of the effect is shown in the main window.
Open the playlist view by pressing Toggle Playlist (Ctrl+L)
Right-click your video, select Save... in the context menu.
Press the wrench symbol Edit selected Profile and in the Video codec tab, select the Filters sub-tab.
Tick the Image properties filter. This will make your change permanent once you save your file.
Enter a destination file and hit start.
Don’t forget to untick the Image properties filter in your profile. Also rewind the changes you made in the Video Effects tab (step 4).

How to rotate my video?

Open the video file you want to modify.
Click Tools -> Effects and filters (or press Ctrl+E)
A dialog with three tabs opens. Select the Video Effects tab.
Under Geometry, tick Transform.
Make your changes and hit Save, then Close. A preview of the effect is shown in the main window.
Open the playlist view by pressing Toggle Playlist (Ctrl+L)
Right-click your video, select Save... in the context menu.
Press the wrench symbol Edit selected Profile and in the Video codec tab, select the Filters sub-tab.
Tick the Video transformation filter. This will make your change permanent once you save your file.
Enter a destination file and hit start.
Don’t forget to untick the Image properties filter in your profile. Also rewind the changes you made in the Video Effects tab.

How to crop my video?

Open the video file you want to modify.
Click Tools -> Effects and filters (or press Ctrl+E)
A dialog with three tabs opens. Select the Video Effects tab.
Under Crop, play around with the pixel values and watch the preview window carefully until you’re happy with the crop. Note or memorize the settings you’d like to apply. Close the dialogue.
Click Tools -> Preferences (or press Ctrl+P)
On the Interface (start) Tab, select the radioButton Show Settings - All.
Scroll down to the Video element and open Filters.
Select the Video cropping filter.
find the Cropadd element within the filters list on the left. Enter the values you memorized/ noted down in the preview. Hit Save.
Open the playlist view by pressing Toggle Playlist (Ctrl+L). Then Right-click your video, select Save... in the context menu.
Press the wrench symbol Edit selected Profile and in the Video codec tab, select the Filters sub-tab.
Tick the Video crop filter. This will make your change permanent once you save your file.
Enter a destination file and hit start.
Don’t forget to untick the Video crop filter in your profile. Also rewind the changes you made in the Filters preferences item.

How to trim my video?

Note: A more detailed version of this can be found in this post.

Open the video file you want to modify.
Show Record button in VLC Player (tick View->Advanced Controls)
Jump to where you want the trimmed video to begin.
Press the red record button
Play the video sequence. Stop where you want to end your trim.
VLC’s standard location for video output: C:\Users\Username\Video

How to change video resolution?

Open the video file you want to modify.
Open the playlist view by pressing Toggle Playlist (Ctrl+L). Then Right-click your video, select Save... in the context menu.
Press the wrench symbol Edit selected Profile and in the Video codec tab, select the Resolution sub-tab.
Enter the video size you’d like to convert to and hit Save.

Hope this helps!

Paginator broken, Favorite icon placed

Schallbert — Sun, 13 Jun 2021

Adding a “tab” icon - favicon

I was worrying about a strange error message that Jekyll threw at any page reload since some pushes: [2021-06-13 22:07:43] ERROR '/favicon.ico' not found. Some online research revealed that this is neither a Jekyll, nor a Theme error, but a missing file error. See here how I implemented the icon for this page.

Pagination, again

I found out that somehow, the pagination doesn’t even work on my page while the layouting assumes that it’s preparing the pages. Now, if I try to access the page by pressing the “previous” button, I get a 404 error. Till now, I wasn’t able to fix this because the pagination fails silently and I do not know where. Switching it off completely for now.

Markdown to PDF

Schallbert — Wed, 09 Jun 2021

The tool

Get a tool that can do the job. I selected the Visual Studio Code extension Yzane’s Markdown PDF. Easy to install, easy to use, good documentation. Only the images… well… wouldn’t show.

Getting the images converted, too

The relative links to the images are different between Jekyll’s interpretation and the notion from the PDF generator. I had to move the target markdown file into the root folder and add a . before my /assets to make it work. Pagination is …meh… but I’m not willing to put more work in this right now so that’s as good as it gets.

An issue with Posts not showing up?

Just upon publishing this sixth post of mine, I see that somehow only page 2 of the post overview is shown (looks like 5 posts per page) and I cannot switch to page 1 containing “latest” posts:

. Hm.

Jekyll: images

Schallbert — Mon, 07 Jun 2021

The trick to make images show both locally and on Github pages

When I added some images to my content, all was fine until I pushed to GitHub. Most images were gone and I couldn’t figure out why. Some searching revealed that Github is case sensitive and some of my image paths were still mixed case.

Changing folder names in git

Well, it still didn’t work because git didn’t recognize my renaming. I first had to create the renamed folder in another subdirectory, use git mv folder/tArGet* target to get this to a temporary place, delete tArGet and move target to the place where tArGet was before. Only this fixed the issue and now github and my locals are in sync again, luckily.

This site: Table of Contents

Schallbert — Sun, 06 Jun 2021

There is an easy solution for making the page’s table of contents stick: add toc_sticky: true to the defaults or the specific page’s front matter. How I found the keyword? By searching “toc” in the whole solution and some scrolling through the layout scetches. I modified the original site navigation in the left sidebar and exchanged it with a dummy implementation of a gallery / audio / video contents display.

ToC on the left

To understand how to create a sticky ToC on the left, please continue reading this chapter of the mySite project.

Author’s profile

I decided I only want my author_profile to be shown on pages like about, legal info and not on the typical content pages. So I wrote author_profile: false for the pages I didn’t want it to show up - with no effect. Why? I had set it in the config.yml’s type defaults but it should actually go into the frontmatter of the individual files.

Jekyll `-incremental` option

Schallbert — Thu, 03 Jun 2021

The images

I worked on my assets/images, they tend to be somewhat “big” so that page load times might be a reason to worry about. I’m using paint.net to shrink them to a size I think I can afford and save them as .jpg with compression max’ed out and some compromises on image quality. This way, most of my pictures take less than 10% or the original size in kB.

MinimalMistakes theme - Page design

Schallbert — Wed, 02 Jun 2021

The landing page

Yeah, by changing the layout type to single I was able to get a header image as type overlay in! The buttons are displayed within the image now. The image changes with display size and orientation automatically due to the visual designs defined in the _sass folder.

landing page image

Ha, found it! I was freaking out; my landing page now just wouldn’t show any image, no matter how hard I tried modifying the home.md file… Reason behind was that there was a stock index.html file that came with the template and that seems to have priority. No I just added my changes in there - works perfectly fine! By the way, the overlay image’s height is automatically selected depending on the overlay text’s contents/ height.

Easy solution: type toc: true in the front matter. Unfortunately, eventually you will loose this information as you scroll down because it won’t stick to your scrolled view.
Hard solution: Add page navigation to left sidebar. How? See here!

ClickEncoder

Schallbert — Mon, 01 Jan 0001

Project stats

Difficulty: medium 3/5
Cost: 0€
Time: ~6h

Abstract

Library for the Arduino Framework, written in c++.

It is lightweight because the source code itself is a few hundred lines only, and powerful, because of the improved algorithm uses less calculations while providing a better state interpretation than the original project this is forked from. It is officially released on PlatformIO IDE and hosted on Github. It contains examples both for Arduino and PlatformIO IDEs and provides unittests that can be run upon customization.

Motivation

I used the original library published by 0xPIT at first but that one caused some compiler warnings for me, plus the encoder readings were not flawless and sometimes there was jitter or bouncing I didn’t want to happen.

Solution

As the repo looked unmaintained, I decided to fix these issues by rewriting some code. For my project at that time, I also needed a “repeat” signal when a button is pressed continuously, so I added this, too. And finally, I separated the encoder algorithm from the button handling and provided separate classes so that, if needed, any combination of button and encoder could be used, each with configurable behavior.

Encoder Algorithm explained

API

ClickEncoder’s API is simple and easy to understand: service() runs the business logic and does the state interpretation. It has to be run regularly to check if the encoder has been moved since its last call, into which direction, how many steps, and has these values saved in an accumulator variable. getIncrement() returns the encoder step changes since its last call. getAccumulate() returns the sum of steps taken since startup. getButton() returns the current Button state.

void service();
int16_t getIncrement();
int16_t getAccumulate();
Button::eButtonStates getButton();

Encoder state

The encoder uses a Gray Code to encode the steps. getBitCode() converts this code transmitted via the Encoder’s hardware pins A and B to a 0...3 notch interpretation, numbers increasing when turning right, and decreasing when turning left. This is done efficiently without use of if() or other branching logic.

uint8_t Encoder::getBitCode()
{
 // GrayCode convert
 // !A && !B --> 0
 // !A && B --> 1
 // A && B --> 2
 // A && !B --> 3
 uint8_t currentEncoderRead = digitalRead(pinA);
 currentEncoderRead |= (currentEncoderRead << 1);
 
 // invert result's 0th bit if set
 currentEncoderRead ^= digitalRead(pinB);
 return currentEncoderRead;
}

I used a trick here, so you may have to look at the code snippet above several times. The state of PinA is written to bit1 of currentEncoderRead by me shifting it to the left with the << operation. But at the same time the state of the 0th bit is preserved by |=. If I now bind PinB simply by logical &=, then I would violate the Gray code, after which A&&B=2 should be and not 3.

So if A and B are both not set, the result remains 0, because 00^00=00=0. If only A is set, but B is not, the result is 11^00=11=3. If B is set, but A is not, the result is 00^01=01=1. If both are set, finally 11^01=10=2.

For more information about how logical binary operators work, please refer to my LED fireplace project

State interpretation

After the bit code is read, rawMovement variable is set that tells us how much the encoder has been turned between two runs of the service() routine. Here you can see the time criticality because there will be aliasing errors if the encoder changes notches in a time similarly short to the service interval.

void Encoder::handleEncoder()
{
 uint8_t encoderRead = getBitCode();
 // bit0 set = status changed, bit1 set = "overflow 3" where it goes 0->3 or 3->0
 uint8_t rawMovement = encoderRead - lastEncoderRead;
 lastEncoderRead = encoderRead;
 // This is the uint->int magic, converts raw to: -1 counterclockwise, 0 no turn, 1 clockwise
 int8_t signedMovement = ((rawMovement & 1) - (rawMovement & 2));

 encoderAccumulate += signedMovement;
 encoderAccumulate += handleAcceleration(signedMovement);
}

Here is another trick: signedMovement is calculated by converting the movement into a signed integer representing 0 as “not rotated, -1 as “rotated left” and 1 as “rotated right”. This exploits the integer overflow behavior, in combination with the interpretation of signedness.

Finally, the accumulation is performed and the acceleration, if configured, is added to the accumulator depending on how fast the encoder is rotated.

To program these two methods without branching took me several evenings of grubbling and practically half a pad of squared paper. Frankly, these few lines of code have brought me to the edge of my brain’s capacity. I am just not a “real” programmer 😅

The getAccumulate() method

It’s really simple, it just returns the internal accumulated value accounted to the encoder’s configuration about how many notches make a step.

int16_t Encoder::getAccumulate()
{
 return (encoderAccumulate / stepsPerNotch);
}

The getIncrement() method

Returns how much the encoder’s values have changed since its last call.

int16_t Encoder::getIncrement()
{
 int16_t accu = getAccumulate();
 int16_t encoderIncrements = accu - lastEncoderAccumulate;
 lastEncoderAccumulate = accu;
 return (encoderIncrements);
}

Button state calculation explained

The handleButton() method

All button states derive from the two basic states: pressed or not pressed. This is reflected in the logic as well:

void Button::handleButton()
{
 if (lastGetButtonCount < ENC_BUTTONINTERVAL)
 {
 return;
 }
 lastGetButtonCount = 0;

 if (digitalRead(pinBTN) == pinActiveState)
 {
 handleButtonPressed();
 }
 else
 {
 handleButtonReleased();
 }

 if (doubleClickTicks > 0)
 {
 --doubleClickTicks;
 }
}

The button can be configured to be read less often than the encoder. If this is the case, this method will be left without any action.

Button pressed:

handleButtonPressed() can either return Closed, Held, or LongPressRepeat depending on how the button is configured and how long it has been pressed already.

Button not pressed:

handleButtonReleased() can either return Clicked, Released, or Doubleclicked depending on button configuration and click count within a certain time.

Library configuration

The encoder/button detection behavior can be modified in the encoder.h file using the constants shown below. The values have been tweaked already and feel natural to me, at least. They should be modified if you choose to use a different service interval than 1ms because those values all take this interval as reference.

// ----------------------------------------------------------------------------
// Acceleration configuration (for 1ms calls to ::service())
//
constexpr uint8_t ENC_ACCEL_START = 150; // The smaller this value, the quicker you must turn to activate acceleration.
constexpr uint8_t ENC_ACCEL_SLOPE = 75; // the smaller this value, the stronger the acceleration will manipulate values.

// Button configuration (values for 1ms timer service calls)
//
constexpr uint8_t ENC_BUTTONINTERVAL = 20; // check button every x ms, also debouce time
constexpr uint16_t ENC_DOUBLECLICKTIME = 400; // second click within x ms
constexpr uint16_t ENC_LONGPRESSREPEATINTERVAL = 200; // reports repeating-held every x ms
constexpr uint16_t ENC_HOLDTIME = 1200; // report held button after x ms
// ----------------------------------------------------------------------------

Use it in your project!

The library is open source software, using the MIT license.
PlatformIO user? Easy, just search it in the ‘Libraries’ view, Tags are Arduino, Encoder, Schallbert. It’s then just one more click to add it to your solution
Alternatively, just clone or fork the repository and use the header/cpp file as you deem fit.
Contribute? Maybe you have a more efficient state calculation of the button? I’d be happy to review your pull request :)

Github Pages: Moving out

Schallbert — Mon, 01 Jan 0001

Project stats

Difficulty: medium 3/5
Cost: 5-50€/Month
Time: ~10h

Motivation

I decided to move my Website from Github-Pages to my own server. I have two reasons for that: First, I want the server to be situated in Germany (data protection) and second, I wanted a simple way of providing my page in two languages.

In addition, I might be able to realize additional wishes: Setup a miniature file server and add a Website for my small side-hustle. But those will be handled in a future post.

Overview

In this project, I’ll rent a virtual server, configure access via SSH and install a container manager called Docker.

In one of the containers, I want an instance of gitea to provide version controls for website, and have a CI/CD¹ pipeline setup that automatically integrates and deploys my site.

Not unlike Github, Gitea Actions allows me to deploy the site built with Jekyll via Workflow files on a caddy-Server instance.

caddy is not just a webserver, but also provides a reverse proxy which does Port mapping and address resolution. In addition, caddy manages my https certificates easily per Let’s Encrypt.

As caddy will run in an own docker container, I have to generate and configure virtual networks so the containers can “talk” to each other.

Let’s get going - step by step.

Configuring my server

The upcoming sections describe the individual steps of configuring a cloud-server for deployment of a static site, including version control and CI.

configure SSH for the server

Right after ordering the “Cloud-server”, I enabled SSH access and deactivated its password login. To connect from my machine, I check whether the ssh-agent is already up and running per eval ssh-agent.

Then, I create a new key pair via

ssh-keygen -t ed25519 -C "your_email@example.com" -f "~/.ssh/my-cloud-server"

This line creates a key pair with algorithm (-t) ed25519, taking a mail address as comment (-C) that is then saved with filename (-f) my-cloud-server.

The public key file my-cloud-server.pub is then uploaded on the web portal of my server using the “security” tab. The resulting fingerprint is then verified with:

ssh-keygen -lf <fingerprint_from_server> "~/.ssh/mein-cloud-server"

Finally, I modified the .ssh/config file so that, to connect, I just need to enter ssh server into my console instead of the server’s IP-Address - pretty Lazy 😀

#.ssh/config
Host server
 User root
 Hostname ip_address_of_server
 PreferredAuthentications publickey
 IdentityFile ~/.ssh/my-cloud-server

Now when I enter ssh server the first time, I get a prompt about a potential connection to a yet unknown host with fingerprint xyz and get asked whether I’d want that to happen. I compare this fingerprint to the server’s (use provider’s Website to verify). This way I can make sure that I’m trying to connect to the correct party. Once I give my yes, the fingerprint is added to the known_hosts file and future connections will be made directly and without warning.

Install Docker 🐳

My server runs a standard Ubuntu that supports apt commands out of the box. So installing Docker per Manual is really easy: Connect to the server per SSH, type apt install docker, done. I’ll be using docker-compose which also auto-installs. Compose interprets configuration files that define the target container and make it really easy to get them up and running.

Install caddy 🛒

First, I create a folder in my /opt directory via mkdir caddy2. To have caddy run in an own container, I change into that directory and create the following docker-compose.yml file:

#caddy2/docker-compose.yml

version: '3'
services:
 caddy:
 image: caddy:latest
 container_name: caddy
 ports:
 - "80:80"
 - "443:443"
 environment:
 - ACME_AGREE=true
 restart: unless-stopped
 volumes:
 - ./Caddyfile:/etc/caddy/Caddyfile
 - ./caddy_data:/data
 - ./caddy_config:/config
 - ./www:/www
 networks:
 - caddy-proxy

networks:
 caddy-proxy:
 external: true

This file tells docker to use latest caddy release. The container is named caddy and uses ports 80 (http) and 443 (https). Certificates are requested via ACME_AGREE and the container is able to access required files (Caddyfile, data, config) on my host machine via a volume. It will also be able to access files in /www folder that, in the future, will host my blog’s site data. The container is part of the caddy-proxy network that has been provided externally.

So, to be able to run this, I’ll first have to create that network:docker network create caddy-proxy. Then a quick check per docker network ls shows that the network has been added:

NETWORK ID NAME DRIVER SCOPE
<redacted> bridge bridge local
<redacted> caddy2_default bridge local
<redacted> caddy-proxy bridge local

Now I create a Caddyfile configuration with following content:

# caddy2/Caddyfile
blog.schallbert.de {
 root * /www/blog
 encode gzip
 file_server
}

Caddy is told that the content of /www/blog shall be provided as a (static) file server, supporting gzip-compression, available under blog.schallbert.de.

Create a test website 🧪

Before we start with the page design, we should be able to have the address resolved². I log into my DNS-manager and bind a static IP address to my subdomain:

TYPE NAME VALUE TTL
A blog.schallbert.de <redacted> 86400

My server shall be able to show something. So let’s create a minimalistic test page:

mkdir ~/caddy2/www/blog
cd ~/caddy2/www/blog
nano index.html

Hmm, Just a little something:

<!-- blog/index.html -->
<h1>ALL YOUR BASE ARE BELONG TO US</h1>

Save it, done.

This is how caddy’s root directory looks like now:

*caddy_config* *caddy_data* Caddyfile docker-compose.yml *www*

We’re ready now to create the container:

docker compose up -d

The Option -d means “detach”, so the container is started and detached from the console, unblocking it so we can reuse it for more prompts.

[+] Running 1/1
 ✔ Container caddy Started

This looks good. Now I navigate to the page:

Install Gitea 🍵

I need Gitea for two purposes: I want version control also for private repositories and I want to host my website with Jekyll with full CI/CD automation.

Like caddy, it starts with a folder:

cd /opt
mkdir gitea
cd gitea
nano docker-compose.yml

I compiled Gitea’s docker-compose.yml out of dfferent example files. Here it is:

# gitea/docker-compose.yml
version: "3"

networks:
 gitea:
 external: false
 caddy-proxy:
 external: true
 name: caddy-proxy

services:
 gitea:
 image: gitea/gitea:latest
 container_name: gitea
 restart: always
 environment:
 - USER_UID=1000
 - USER_GID=1000
 - SSH_DOMAIN=git.schallbert.de
 - GITEA_HOSTNAME=git.schallbert.de
 networks:
 - gitea
 - caddy-proxy
 volumes:
 - ./gitea:/data
 - ./log:/app/gitea/log
 - /etc/timezone:/etc/timezone:ro
 - /etc/localtime:/etc/localtime:ro
 ports:
 - "127.0.0.1:3000:3000"
 - "222:22"
 runner:
 image: gitea/act_runner:nightly
 environment:
 - CONFIG_FILE=/config.yml
 - GITEA_INSTANCE_URL=https://git.schallbert.de
 - GITEA_RUNNER_NAME=ichlaufe
 - GITEA_RUNNER_REGISTRATION_TOKEN=
 volumes:
 - ./runner/config.yml:/config.yml
 - ./runner/data:/data
 - /var/run/docker.sock:/var/run/docker.sock

This is what it does: gitea is member of docker networks gitea and caddy-proxy. The latter has been added so I can reach gitea from “the internet”. The container running the service is named gitea and is automatically restarted when down. User-ID is set to 1000 so it won’t interfere with admins or other special users. Per SSH and http, gitea can be reached under git.schallbert.de on ports 22 and 3000 - the latter is localhost only.

Gitea has access to the host’s time zone and local time, it saves its logs under /logand files under /data. Gitea gets an action runner that runs on the gitea instance. Its name is ichlaufe and the config is available under config.yml as a volume. The runner can access Docker daemon’s standard-socket to run actions, e.g. install ubuntu-latest in an own container to run Jekyll builds. I took the basic config for the runner from the official repository.

Gitea will only accept runners that are registered via GITEA_RUNNER_REGISTRATION_TOKEN. We’ll do this at a later point in time.

Now we start Gitea for the first time and look at the console print:

docker compose up
[+] Running 4/2
 ✔ Network gitea_default Created 0.0s 
 ✔ Network gitea_gitea Created 0.1s 
 ✔ Container gitea Created 0.1s 
 ✔ Container gitea-runner-1 Created 0.0s 
Attaching to gitea, gitea-runner-1
gitea | Server listening on :: port 22.
gitea | Server listening on 0.0.0.0 port 22.
gitea | 2023/11/18 17:19:52 cmd/web.go:242:runWeb() [I] Starting Gitea on PID: 15
gitea | 2023/11/18 17:19:52 cmd/web.go:111:showWebStartupMessage() [I] Gitea version: 1.21.0 built with GNU Make 4.4.1, go1.21.4 : bindata, timetzdata, sqlite, sqlite_unlock_notify
gitea | 2023/11/18 17:19:52 cmd/web.go:112:showWebStartupMessage() [I] * RunMode: prod
gitea | 2023/11/18 17:19:52 cmd/web.go:113:showWebStartupMessage() [I] * AppPath: /usr/local/bin/gitea
gitea | 2023/11/18 17:19:52 cmd/web.go:114:showWebStartupMessage() [I] * WorkPath: /data/gitea
gitea | 2023/11/18 17:19:52 cmd/web.go:115:showWebStartupMessage() [I] * CustomPath: /data/gitea
gitea | 2023/11/18 17:19:52 cmd/web.go:116:showWebStartupMessage() [I] * ConfigFile: /data/gitea/conf/app.ini
gitea | 2023/11/18 17:19:52 cmd/web.go:117:showWebStartupMessage() [I] Prepare to run web server

Configure web access

Now I have another subdomain added to my DNS manager:

TYPE NAME VALUE TTL
A git.schallbert.de <redacted> 86400

I also have to modify Caddyfile so that web requests are forwarded accordingly (reverse proxy). To do this, I navigate back to the caddy2 folder, type nano Caddyfile and add the following section:

# caddy2/Caddyfile
git.schallbert.de {
 reverse_proxy * http://gitea:3000
}

Now requests to git.schallbert.de are forwarded to the (local) port 3000 of the Gitea-container that I opened in the docker compose file before.

Configuring Gitea per web page

When I use my browser to navigate to the above page, I get a login screen - identical to this demo.

Now I create a nice password, click Register and then configure Gitea using my browser. I only do a rough config because I’ll have to modify it again later to properly add the runner.

When done, I create a new SSH key pair as described in the section Configure SSH above with filename -f ~/.ssh/gitea. It is uploaded and verified under Settings->SSH/GPG Keys.

Now I can create another entry on my local machine’s .ssh/config file:

# .ssh/config
Host gitea
 User schallbert
 Hostname git.schallbert.de
 PreferredAuthentications publickey
 Identityfile ~/.ssh/schallbert_gitea

Let’s check if this works:

ssh gitea
Host 'git.schallbert.de' is known and matches the ED25519 host key.
debug1: Found key in /home/schallbert/.ssh/known_hosts

Perfect.

Complete Gitea config via `app.ini`

As the configuration is not complete, I shut down the container again:

docker compose down

and then navigate to /gitea/gitea/conf and modify the app.ini just a little:

# conf/app.ini
APP_NAME = Gitea: Git with a cup of tea
RUN_MODE = prod
RUN_USER = git
WORK_PATH = /data/gitea

[repository]
ROOT = /data/git/repositories
DEFAULT_BRANCH = main

[repository.local]
LOCAL_COPY_PATH = /data/gitea/tmp/local-repo

[repository.upload]
TEMP_PATH = /data/gitea/uploads

[server]
APP_DATA_PATH = /data/gitea
DOMAIN = git.schallbert.de
SSH_DOMAIN = git.schallbert.de
HTTP_PORT = 3000
ROOT_URL = https://git.schallbert.de/
DISABLE_SSH = false
SSH_PORT = 222
SSH_LISTEN_PORT = 22
LFS_START_SERVER = true
LFS_JWT_SECRET = <redacted>
OFFLINE_MODE = false

[database]
PATH = /data/gitea/gitea.db
DB_TYPE = sqlite3
HOST = localhost:3306
NAME = gitea
USER = root
PASSWD =
LOG_SQL = false
SCHEMA =
SSL_MODE = disable

[indexer]
ISSUE_INDEXER_PATH = /data/gitea/indexers/issues.bleve

[session]
PROVIDER_CONFIG = /data/gitea/sessions
PROVIDER = file

[picture]
AVATAR_UPLOAD_PATH = /data/gitea/avatars
REPOSITORY_AVATAR_UPLOAD_PATH = /data/gitea/repo-avatars

[attachment]
PATH = /data/gitea/attachments

[log]
MODE = file
LEVEL = info
ROOT_PATH = /data/gitea/log

[security]
INSTALL_LOCK = true
SECRET_KEY = <redacted>
REVERSE_PROXY_LIMIT = 1
REVERSE_PROXY_TRUSTED_PROXIES = *
INTERNAL_TOKEN = <redacted>
PASSWORD_HASH_ALGO = pbkdf2

[service]
DISABLE_REGISTRATION = true
REQUIRE_SIGNIN_VIEW = false
REGISTER_EMAIL_CONFIRM = false
ENABLE_NOTIFY_MAIL = false
ALLOW_ONLY_EXTERNAL_REGISTRATION = false
ENABLE_CAPTCHA = false
DEFAULT_KEEP_EMAIL_PRIVATE = false
DEFAULT_ALLOW_CREATE_ORGANIZATION = false
DEFAULT_ENABLE_TIMETRACKING = true
NO_REPLY_ADDRESS = noreply.localhost

[lfs]
PATH = /data/git/lfs

[mailer]
ENABLED = false

[openid]
ENABLE_OPENID_SIGNIN = false
ENABLE_OPENID_SIGNUP = false

[cron.update_checker]
ENABLED = false

[repository.pull-request]
DEFAULT_MERGE_STYLE = merge

[repository.signing]
DEFAULT_TRUST_MODEL = committer

[oauth2]
JWT_SECRET = <redacted>

[actions]
ENABLED = true

Here it is important to make sure that ROOT_URL and port configuration match the one specified in docker-compose.yml. I configure its database (sqlite3) and make it available to Gitea. Plus, I deactivate Giteas ability to register new users DISABLE_REGISTRATION = true and the option to sign up/in via openID ENABLE_OPENID_SIGNIN = false ENABLE_OPENID_SIGNUP = false. Finally, I enable Gitea Actions ENABLED = true and save the file.

Now I can restart the container:

docker compose up
[+] Running 4/4
 ✔ Network gitea_default Created 0.1s 
 ✔ Network gitea_gitea Created 0.0s 
 ✔ Container gitea Started 0.0s 
 x Container gitea-runner-1 Error 0.1s

Very well, Gitea is up and running. In a last step, I configure the Action Runner.

Setup Action Runner 🏃

I use the official documentation provided by Gitea for inspiration.

I login to Giteas Web surface and see that activating Actions in the app.ini added another menu: Settings->Actions. Here I click Create new Runner und copy its REGISTRATION TOKEN.

Then, I re-open Gitea’s docker-compose.yml and add the following lines as recommended in the doc page:

# gitea/docker-compose.yml
# [...]
runner:
 image: gitea/act_runner:nightly
 environment:
 - CONFIG_FILE=/config.yml
 - GITEA_INSTANCE_URL=https://git.schallbert.de
 - GITEA_RUNNER_NAME=ichlaufe
 - GITEA_RUNNER_REGISTRATION_TOKEN= <redacted>

I restart the container with docker compose down && docker compose up -d and get a “started” message for the runner in the logs:

✔ Network gitea_default Created 0.1s 
✔ Network gitea_gitea Created 0.0s 
✔ Container gitea Started 0.0s 
✔ Container gitea-runner-1 Started 0.1s

Great, let’s have a look at the web page:

Now we have to know if the runner can do something for us.

Runner functional test

Once again, I have a look at the Runner’s documentation. To check, I create a new Repository on the web surface, name it runner-test, and add a file to /.gitea/workflows/01-test.yml containing the recommended runner check.

Commit, push, and - nothing.

Oh, the workflow starts on: [push]. This cannot work when it has just been uploaded itself.

Well, let’s touch another random file in the repository. Commit, push, result:

Fantastic! 🥳

Interim conclusion

In this article, I pretended that the road to a working server setup was straight and without obstacles. Of course, that is not entirely true. It was really simple to run images in Docker, but to have them communicate turned out less straight-forward. I also had quite some problems with Caddyfile as it is hard to debug and I couldn’t see what I was doing wrong.

For instance, I was able to ping the website on my host but couldn’t reach it from the outside. So I knew it is the caddy config to modify.

Root cause was missing the Caddyfile in the Docker volume list and forgetting a leading / in the target volume folder.

On the other hand, configuring Gitea was much easier than I thought. Even complex process of registering the runner and its routing through Docker worked at first try.

Now I have a quick and lightweight server that will be able to host my blog with ease.

Next steps - creating a build process for my site - is tried in this post, but then replaced with native Jekyll over here.

Ultimately, I want to get back to creating content. And not spend any more time than necessary on all the other stuff.

CI/CD “Continuous Integration / Continuous deployment” cares for automatic build, verification, and deployment of my web page. ↩︎
DNS = Domain Name System, a distributed hierarchical system for name resolution of websites. It exists because it’s much less straight forward to enter an IPV6 address like 2a02:ec80:300:ed1a:0:0:0:1 into your web browser than en.wikipedia.org. ↩︎

GoogleTest Demo

Schallbert — Mon, 01 Jan 0001

Project stats

Difficulty: medium 3/5
Cost: 0€
Time: ~2h

Motivation

It is nice to have a powerful unit test framework at hand when developing more complex projects, possibly saving a major amount of time debugging (especially on-Target debugging in the embedded world can be really time-consuming), improving code quality and possibly boosting insights to APIs and architecture for others.

The embedded world

Traditionally, microcontrollers were limited on the memory side. In the past, they neither had much RAM nor PROGMEM so the code I wrote was mostly in lean C, and configuration took place in lists of #define that wouldn’t weigh much because the preprocessor would do the work; not the µC at runtime. My unit tests were alyways bound to the hardware they were running on - take Unity as a well-known example - but now I wanted to try something new: Write Object-oriented style code, and get the business logic tested independently of the hardware. I selected the GoogleTest framework and I wanted a seamless integration into my workflow with VScode and its extension PlatformIO.

Until recently, the only way to do this was to run GoogleTest in parallel using the GNU Compiler Collection and have it controlled by PlatformIO. Recently, however, GoogleTest can be used directly as a PlatformIO Library. Therefore, I’ll cover both options here.

Using GoogleTest directly in PlatformIO

The installation is particularly easy here:

Clone my Github repository PlatformIO-gTestgMock.
Open a console window in the repository folder and switch to the corresponding branch using git checkout googletest-native.
Follow the instructions in the repository’s README file and test the setup.

This is what it looks like when the tests have been successfully run:

Using GoogleTest with GCC

You can install the required software yourself (even on a Windows¹ computer) to get started with GoogleTest.

Download and install MinGW

See this thread in the PlatformIO forum if you have any questions about the following steps.

Unix users: Skip the steps up to this point, as you probably only need the libpthreadgc libraries. Check for the presence of all libraries (but omit the mingw32- prefix) and install any missing ones. Make sure the PATH variable is set correctly.
Windows users:

Download MinGW, e.g., from Sourceforge
Install by running mingw-get-setup.exe. If you like graphical user interfaces, leave the corresponding box checked. I do not recommend changing the installation directory. If you do, you’ll need to remember the new path.

Configure MinGW

For GoogleTest to run, you’ll require additional packages to be installed:
- mingw32-gcc-g++ (of course!)
- mingw32-libmingwex
- mingw32-libmingwex-dll
- mingw32-libmingwex-dev
- mingw32-libpthreadgc-dll
- mingw32-libpthreadgc-dev
To install these,
- If you use the GUI: search libraries under “MinGW Standard Libraries”, add/activate/install them one by one.
- Command Line Interpreter users: type mingw-get install and then add the package name. Easy. If you get an error message saying that some of those do already exist, that’s good for you!
Now you need to add MinGW’s \bin folder to your system path variables so PlatformIO can find them later. If you didn’t hand-modify it, it should be C:\MinGW\bin.
- GUI fans: follow this guide
- CLI: type set PATH=%PATH%;C:\MinGW\bin, check if it worked with echo %PATH:;=&echo.%.

Congratulations! You have installed MinGW, set its environment variables to be found more easily by third party applications, and configured it to work with GoogleTest.

Configure PlatformIO

After setting the PATH variable, you’ll have to restart VScode.
To make PlatformIO correctly link to the compiler you just install, make it use the native environment where your compiler sits. Open a PlatformIO terminal within one of your projects and type pio platform install native
Two choices:

Follow through this tutorial and use my example project as described below. You can safely skip the upcoming section as that project comes pre-configured and is open for your modifications as you have basic hardware abstraction interfaces at the ready right from the start.
Take your own project and modify its Platformio.ini so that GoogleTest can be run.

This what you should add to your Platformio.ini:

[env:desktop]
platform = native
build_flags = 
	-std=gnu++11 # use installed GNU C++11 compiler.
	-pthread # found in gtest documentation
lib_deps = 
	googletest # Will automatically load latest googletest lib
lib_ignore = 
 src # most main.cpp's directly access the hardware. 
 # Can be removed if your project is different.
 #
 # PLUS: Any files that contain bare-metal hardware commands 
 # (e.g. digitalWrite), g++ won't have the headers or specialized compiler knowledge!
lib_compat_mode = off # Must-have for external stuff like gtest!

For the environment desktop (= your computer without target), it uses the native compiler which is GCC/G++ that you installed and set the PATH for. GoogleTest uses C++11 standard and threading which you have to add to the build flags. Of course, the build depends on the googletest library and you want to ignore all files/folders that contain source or header files using hardware-related commands or headers. Library compatibility mode has to be switched off for the Library Dependency Finder to include googletest.

(Optional) In case you like using GUI buttons better than typing a line into your console to start the tests, create a custom configuration.

Running Tests

For a quicker start, you can clone my example repository (on branch main). It contains all the necessary elements to test whether gTest and gMock are working correctly. This allows you to immediately verify whether the installation was successful.

Unzip the zip file or use SSH to clone the repository.
Open the PlatformIO extension home page, click “Open Project,” and select the folder.
To run the unit tests, open a PlatformIO terminal and enter pio test -vvv -e desktop -f test_desktop (-vvv = verbose -e = environment, -f = filter). The parameters are required because the tests can only be run on the desktop environment, not on the microcontroller.
After a while, the message shown below should appear.

Great! Unit tests within PlatformIO have been built and executed!

Tips and Tricks

Note that your PlatformIO_gTestgMock/.pio/build/desktop folder now contains an executable called program.exe. If you run this from the CLI of your choice, you should see the same result as in the above image. You might want to choose this option more frequently once you have more tests that not only fail but crash the program on execution, and to debug tests (and not the user code), as PlatformIO’s CLI tends to hang or miss out the reason for why the test crashed. You might also want to have the stack trace printed in these cases although this would really go far beyond scope of this exercise.

From here,

you might want to continue reading below for a deeper understanding of the repository contents and how GoogleTest works
you could go practice writing tests by adding more cases, e.g. to verify if the LOW state is looped through correctly
You could also enhance the interface of these groundworks, e.g. by adding the analogRead() function to the hardware abstraction layer
You could write your project logic on top of this, cleanly cut out the hardware dependencies, and enjoy the ability for easy porting of your project to different hardware devices and manufacturers
Or do this other brilliant stuff that you were up to before you were diverted by this tutorial.

Repository contents explained

The following sections cover most of the files in the repository and explain what they do in detail. The implemented business logic is minimal and should be seen as example code - to create the least confusion possible when dealing with this complex matter. For a real-life project that uses all of the patterns illustrated on this page, please refer to my Tonuino page.

Hardware abstraction layer interface

Let’s have a look at the bespoke interface and how it’s structured.

// hal_if.h
namespace hardwareAbstraction {

// Hardware Abstraction Layer Interface class
class Hal_IF
{
public:
 virtual ~Hal_IF() = default;

 virtual bool digitalRead(uint8_t pinId) const = 0;
 virtual void digitalWrite(uint8_t pinId, bool value) const = 0;
};

} // hardwareAbstraction

The Hal_IF class has two methods which are both virtual. This means that a client class using this interface can call the method, and is thus “auto-forwarded” to the concrete implementation of the method. The = 0; makes these methods pure virtual, so overriding them with a derived class is not optional but a must. This is what we want here, because if this method wasn’t overridden, it wouldn’t have any behavior. Note that the destructor of an interface class almost always should be virtual, too.

LoopInOut

The logic under test. Note that it does not directly access the input/output hardware of the µC, but the m_hal variable which is an implementation of the Hal_IF interface at runtime. It is passed into the class by reference (marked by the & at the type declaration) upon object construction².

// loopInToOut.h

class LoopInToOut
{
public:
 LoopInToOut(hardwareAbstraction::Hal_IF& hal): m_hal(hal){};
 ~LoopInToOut() = default;

 bool loopThrough(uint8_t inPin, uint8_t outPin);

private:
 hardwareAbstraction::Hal_IF& m_hal;
};

This is enabling the real magic of gMock: as the client code works with an interface, I can hand over the “real” implementation (of the hardware access) at chip runtime, but for the test I can hand over a “mock” implementation which I have control over at the time the test is running.

// loopInToOut.cpp

bool LoopInToOut::loopThrough(uint8_t inPin, uint8_t outPin)
{
 bool input = m_hal.digitalRead(inPin);

 m_hal.digitalWrite(outPin, input);
 return input;
}

loopThrough() again is really simple. It reads the input pin’s state and maps it to the output pin. It returns the input pin state to the caller.

Mock Hal

GoogleTest uses macros to define behavior. The mock class is also deriving from the interface and just provides declarations of the interface method it overrides. There’s no behavior involved just yet.

// hal_mock.h

class Hal_mock : public hardwareAbstraction::Hal_IF
{
public:
 MOCK_METHOD(bool, digitalRead, (uint8_t pinId), (const, override));
 MOCK_METHOD(void, digitalWrite, (uint8_t pinId, bool value), (const, override));
};

The test fixture

The test fixture contains common data and behavior for all test cases within a test suite. Test suite name = class name, and this exact name has to be reused in the tests that should belong to this suite.

// unittest_loopInToOut.cpp
class loopInToOut_Test : public ::testing::Test
{
protected:
 virtual void SetUp()
 {
 m_loopInToOut = new LoopInToOut(hal_mock);
 }

 virtual void TearDown()
 {
 delete m_loopInToOut;
 }

 NiceMock<Hal_mock> hal_mock{};
 LoopInToOut* m_loopInToOut{nullptr};
};

Thus, every test has some data prepared it can use: the class under test is set up as a pointer with a mocked implementation of its dependency.

The actual test

The test is defined by the TEST_F macro meaning “test with fixture” that expects test suite and test names. The ON_CALL macro is a command to gMock providing behavior to the mocked class how to behave when digitalRead(_) is called with any input (underscore = any): It shall return true by default.

// unittest_loopInToOut.cpp

TEST_F(loopInToOut_Test, loop_inputHigh_writesCorrectOutputHigh)
{
 ON_CALL(hal_mock, digitalRead(_)).WillByDefault(Return(true));

 EXPECT_CALL(hal_mock, digitalWrite(2, true));

 m_loopInToOut->loopThrough(1, 2);
}

Then there’s an expectation expressed on the mocked class, stating that digitalWrite is called with input values of (2, true) which are also checked for correctness.

The last statement actually calls the implementation so that the expecations can be verified.

Hardware behavior implementation

As main.cpp accesses the hardware, there has to be some kind of interface implementation for this.

// hal.h
class Hal : public Hal_IF
{
public:
 Hal() = default;
 ~Hal() = default;

 bool digitalRead(uint8_t pinId) const override;
 void digitalWrite(uint8_t pinId, bool value) const override;
};

The header states that Hal derives from Hal_IF and overrides both of its methods.

// hal.cpp
#include "Arduino.h"

#include "hal.h"

namespace hardwareAbstraction{

bool Hal::digitalRead(uint8_t pinId) const
{
 return digitalRead(pinId);
}

void Hal::digitalWrite(uint8_t pinId, bool value) const
{
 digitalWrite(pinId, value);
}

Note that only the source file actually includes the Arduino.h to access the hardware. So the dependency to hardware functions is very limited and buried in interface implementations that can be selected by the program’s structure. While main.cpp will want to create an object of the above implementation, the test will want to instead hand over a mock.

Gtest key concepts

Google’s official github page on GoogleTest will help you a lot, not only for getting started³. The gmock Cookbook is an excellent guide that I have been using over the months as my tests became more complex hard to write, and the gmock Cheat Sheet will help you when you lost track of how to write a test for this and that special problem.

Dependency Injection

Dependency Injection is the basic mechanics you need to understand to make gMock work. Let me draw an image for you to make it more apparent:

The important concept is, that the interface is merely a placeholder for the implementation of the client classes’ dependency. At runtime, e.g. when the host creates the client object, it will handover (i.e. inject) the implementation of the dependency into the client which then can call any of its methods, as their signatures are known at compile time due to the interface that the client is using.

The mock now merely is an implementation of the interface specialized for testing.
The production implementation is used in all other contexts, it features the “real” behavior of that class.

Hardware Abstraction Layer

Yep, well, this might be one of the tricky bits when working with embedded software. At some point, it boils down to outside world interaction, be it with General Purpose Input Output (GPIO) pins, accessing one of the myriad of hardware bus systems like I2C, UART, SPI, CAN to control external hardware, or simply to use system internals like the EEPROM - that’s where the trouble begins.

Gtest runs on GCC/G++ compiler which just doesn’t know anything what the specialized compiler of that particular chip knows. It wouldn’t even understand the header files. Now, there are two options around this issue:

Write fake headers for each µC-specific thing that can be used by GCC/G++
Cut! At! The! Interface! to any µC-specific thing using an (Hardware) Abstraction Layer so your production environment uses the chip hardware as it normally does, but the test environment uses mocks instead.

There are quite some advantages for the second approach:

It’s clean. It will force you to not cross abstraction layers which should not be crossed.
It’s easy. You can test your business logic without even programming your µC! Just mock the inputs and intercept the outputs, write some tests and run them in the terminal.
It’s portable. Exchange the implementation of your abstraction layer, and you’re almost ready to execute on a different controller family or even on another manufacturer’s chips.

Drawbacks

Hmm, there must be some shadow somewhere. Right you are. In my opinion, these are the main disadvantages when getting everything ready to be tested without hardware attached:

Memory. Both ROM and RAM will be cluttered more easily when you have a massive amount of classes and vtables flying around.
Ownership. Injecting dependencies separates ownership from useage of an object, and you need to make up your mind how to organize both.
3rd party libaries. You’ll have to create an interface header, an adapter implementation, and a mock for each library that uses “chip-internals”.
Complexity. As your objects are injected via an interface, you’ll have to use call-by-reference or hand over by pointer 😨 now. The former is additionally limited because it will only work for constructor injection, not for any other kind (method injection, setter injection, etc.)

Some more tips

Keep everything “behind” your abstraction layer as simple as possible: You can only test and debug this stuff on target. Example: each method only executes one command (as in my example code).
Create a loader class or a Dependency Injection Container (Nicolas Croad) that holds all objects to be injected and can spit them out on request.
Make up your mind about the drawbacks mentioned above and how you create your low-level Architecture to accomodate for this.
Don’t hesitate to create more than one level of interfaces. It might be a good idea to use this pattern not only for the direct hardware interaction, but also in between software modules, which then can be tested in a higher granularity so your overall progress might even be quicker versus one big ball of classes you can just control via mocks in the Hardware Abstraction Layer.

Tough Stuff!

You earn a break now. This was much information to deal with. Take your time. Contact me on this topic’s discussions page if you feel something is missing, incorrect, to be improved, or just exactly what you needed right now.

Schallbert

Installation under Linux is actually more easy, as gcc and g++ come already pre-installed. So even if they do not feature the correct packages yet to support GoogleTest, their error message will be of much better help than the ones you get on Windows when some libraries are missing… 🙃 ↩︎
This is also called Constructor Injection, a subtype of Dependency Injection which is a means to achieve Inversion Of Control. I have prepared some links for you to continue reading, it is a rich topic that whole books have been written about.` ↩︎
By The way: the googletest site is also static like mine and it uses Jekyll as well 👋 …do you like it? I’d like to see your opinion on my discussions page ↩︎

QR-Codengrave

Schallbert — Mon, 01 Jan 0001

Project stats

Difficulty: hard 4/5
Cost: 0€
Time: ~50h

The idea

A friend asked me to fabricate a coaster for him. A custom logo, some letters. Nothing too interesting. On the backside, he wanted to have a QR-code engraved that keeps a link to his website.

So I had a QR-code created with a standard web-tool and exported as .svg. Unfortunately, in my CAM I’d have to klick every single pixel and create a carve action for that pixel, which would take quite some minutes of my time for click-select-and-enter-value procedures. Plus, I’d have to use a tapered tool and V-carve to get a square pixel which takes much more machining time than a round pixel representation.

That’s where I decided to throw many, many more hours of my time at this problem and come up with a generic solution for QR-code engravings - a piece of software named QR-codengrave.

Getting started

Specification

This is what I wanted the tool to do:

Must-haves:

Take a text input and convert it to a QR-code. No additional config to be done by the user.
Provide an option to manage different CNC tools.
Provide inputs for engrave parameters: engrave depth, in-job rapid movement hover overs, flyover height
Provide inputs for custom workpiece XY0-offsets
Vectorize the QR-code’s pixel output
Have paths created from the vectors, for simplicity start with an inwards spiral
Handover the paths to a module that uses the selected tool’s parameters to create machine paths
Have the machine paths converted to G-code that can later be interpreted by my machine
Thoroughly verify the algorithms with automated unit tests (no crashes in the physical world please)
Provide a graphical user interface to make parameter input as simple as possible

Optionals

Have a Continuous Integration pipeline created both for my IDE and Github Actions
Have a Continuous Deploy pipeline created for my IDE (PyInstaller) and Github Actions (Artifacts, releases)
Add integration tests for the GUI
Optimize the path generation algorithm as “inwards spiral” has a high manufacturing quality and is nice to look at but is not very efficient.

Uncertainities

At the beginning of this project, I had some open questions and risks that could have limited usability of the program or even make the whole project unfeasible. They all turned out to be wrong (thankfully), but still these were the risks that could have rendered all my work useless:

Cameras won’t read round dot-as-pixels QR-codes
The contrast my engraving materials provide is too low so read results could be erratic or not obtainable
Machining time could be very high, making the project uneconomic
Limitations of programming tools or my abilities to code could increase development time to where it’s not worth putting more effort into

That’s where I decided to write unit tests along with the code to get quick feedback about whether my code could do what it should. And I limited the program’s functionality to one engraving strategy and the GUI to only show a preview of the QR-code, not XY-offsets or other fancy stuff for a start.

Programming language

I took some sheets of paper and noted down the essentials.

One module shall take care of creating a QR-code from text input, another one should create vectors and paths to have that QR-code engraved, and a GUI in the middle shall host the text input, a machining preview, and input options for milling tools, engrave parameters, and workpiece offsets.

I had a couple of programming languages that I “speak” well enough to choose from:

C++ with QT framework
Java
Python with Tkinter framework

I decided on the - in my view - most simple solution to have a running prototype as quickly as possible: Python with Tkinter. I know tkinter from an earlier project of mine and although it has its caveats and limitations, it’s still powerful and quick to setup.

Proof of concept

I did some research and found a pretty good open source QR-code library on the web that’s available also in Python. Based on its output, I created a vectorizing algorithm that could interrogate the QR-code library to return values of the QR-code’s pixels in a scan line:

 def _qr_bitstream_from_line(self, line):
 """Creates a bitstream from an input line of a QR-code data representation
 :param line: a QR-code data representation (line of bits within the QR-code)
 :returns bitstream: returns an array of bits reflecting the QR code's state at the respective
 point of the line."""
 bitstream = []
 if line.get_direction() == Direction.RIGHT:
 for x in range(line.get_p_start().x, line.get_p_end().x + 1):
 bitstream.append(self._qr.get_module(x, abs(line.get_p_start().y)))
 if line.get_direction() == Direction.LEFT:
 for x in range(line.get_p_start().x, line.get_p_end().x - 1, -1):
 bitstream.append(self._qr.get_module(x, abs(line.get_p_start().y)))
 if line.get_direction() == Direction.UP:
 for y in range(line.get_p_start().y, line.get_p_end().y + 1):
 bitstream.append(self._qr.get_module(line.get_p_start().x, abs(y)))
 if line.get_direction() == Direction.DOWN:
 for y in range(line.get_p_start().y, line.get_p_end().y - 1, -1):
 bitstream.append(self._qr.get_module(line.get_p_start().x, abs(y)))
 return bitstream

Then, I had to create vectors from these lines, so that consequent pixels with the same values could be merged to a line that’s much quicker to manufacture than multiple dots:

 def _vectorize_bitstream(self, bitstream):
 """Creates a QrLineData object from an input bistream
 :param bitstream: an array of bits = a line of the QR-code
 :returns line_vector: A list of QrLineData objects"""
 line_vector = []
 data = QrLineData(bitstream[0], True)
 for bit in range(1, len(bitstream)):
 if bitstream[bit] == data.get_state():
 data.add_length()
 else:
 line_vector.append(data)
 data = QrLineData(bitstream[bit])
 if not data.get_state():
 data.finalize()
 line_vector.append(data)

 return line_vector

These two methods would now be called by a supervising method that knows a spiral path through the QR-code. My first GUI just had three elements: An entry field for the text to be converted to a QR-code, a button to run the algorithm, and an output canvas where the algorithm’s engrave path would be drawn.

By the way, I used the Turtle module to draw the vectorized QR-code path, a tool that’s been written for college kids who take a dip into program lamguages like Python. It is slow but was easy to setup so that was enough for me to get started with. I hard-coded it in to save time, and my loops are more C-ish than pythonic but nevermind.

 for vect in self._spiral_path[i].get_z_vector():
 if self._stop_draw:
 break
 length = vect.get_length()
 if vect.get_state():
 self.turtle.down()
 self.turtle.forward(self.pen_size * length)
 self.turtle.up()
 else:
 self.turtle.forward(self.pen_size * length)
 self.turtle.right(90)

This very simple concept looked promising to me but I had to tweak it a couple of hours because my initial output looked like this:

Your browser does not support the video tag.

QR-codengrave GUI test

Still, after a while I was able to reliably draw a (21x21 Pixel) QR-code on screen, and use my smartphone’s camera to yield the webpage link. 🥳 Yeah!

Your browser does not support the video tag.

QR-codengrave vectorizer working

Unit tests

The “platform” parts - vectorize_qr and machinify_vector modules both contain a lot of verifiable calculations and business logic. That’s why my tests with Python’s standard unittest module are quick and easy to write. This is an example where I verify that the line class I created correctly calculates a length (n+1 style):

class TestLine(unittest.TestCase):

 def test_line_x_calculates_valid_length(self):
 line = Line(Point(0, 0), Point(5, 0))
 self.assertEqual(6, line.get_abs_length())

But it quickly became more complex when I wanted to test whether the output of the QR-Code generator library is handled correctly in my program. For that I had to inject the library into my class under test so I could sneak a mock implementation in that’s then interrogated from within the test.

This pattern of dependency injection is very similar to how it’s done in C++ where I’m much more proficient (see a project here or another one there, for example), only that in Python far less boilerplate code is needed to get there. Some internet research helped me write the following test case:

 def test_bitstream_qr_input_loop_xpos_correct(self):
 mock_qr = QrCode.encode_text("schallbert.de", QrCode.Ecc.MEDIUM)
 mock_qr.get_module = MagicMock()
 cam = VectorizeQr(mock_qr, 1)
 cam._qr_bitstream_from_line(self.test_line_x)
 mock_qr.get_module.assert_called_with(20, 0)

I inject the “real” QrCode library but with one mock method which is of interest to me, get_module(). In the test, I’m asserting that it is called with the expected parameters from within my class under test.

With these tests in place, I can be pretty sure that the building blocks of my algorithm work as intended, and that regressions are quick to be found.

The long way to Minimum Viable Product

Boosted by this quick win, I went all-in. I had to design a tool selector before implementing the G-code generator because I’d have to have the tool properties available before setting dimensions and tool speed / feed for the QR-code engraving action.

After some unsuccessful tries in the main window, I decided the tool config window should be a dialog instead (that Tkinter calls Toplevel) so tool parameters can be entered and sent back to the main GUI once complete. I made the dialog modal with

 self._dialog.grab_set()

So it would always have focus to avoid trigger several instances of this window. I keep the configured tools in a numbered list so once fed, the application would provide the list so the user could select the required tool for the job from a dropdown menu.

With this step, I had to implement two things:

Checks for value type and range
A Persistence module so that entered tools would survice an application restart.

Value type and range checks

This could be accomplished using an event hook within the Entry Box widget like so:

 reg = config_tool_frame.register(validate_number)
 tool_nr_entry.config(validate="key", validatecommand=(reg, '%P'))

And then handing over the entered value to a simple validator that just tries to cast the input value into a float and returns False if that generates an error,

def validate_number(entry):
 """Helper function that is used in validators to check
 that the entered keystroke is a number"""
 if entry == '':
 return True
 try:
 float(entry)
 return True
 except ValueError:
 return False

Range checks would be made not when something is typed into the fields, but only when the OK button is pressed to relay the entered data back into the application. Once out of range, a warning Popup (tkinter.messagebox module) would appear pointing to the possible issue.

Persistence

The Persistence module was a bit less easy to implement. I decided on creating an item what C++ guys would call a static class, i.e. a class that doesn’t have to be instantiated to call one of its methods. This class just has two methods called load and save that do what their name suggests using Python’s pickle for serialization of objects that I wanted to be persisted. In Python, I used the @classmethod decorator to indicate the static nature of that class.

Have a look at how simple data serialization can become with the right tools:

 with open(app_persistence_path, 'wb') as file:
 pickle.dump([cls._tool_list,
 cls._z_params,
 cls._xy0],
 file, protocol=2)

When this reliably worked, I added an engrave parameter configuration widget.

Engrave parameters

To generate G-code from my QR-code data, I not only needed tool and speed / feed data, but also I’d have to know how deep to engrave, how far above the workpiece’s surface the tool should hover for rapid moves, and what safety height to use for the CNC to return to origin or home.

It is created in a similar manner to the tool_configuration dialog and uses the same mechanisms for type and range checking. The only difference is that it’s launched not with a button click, but via click on a label that shows the current engrave parameter value in the main window.

Workpiece XY Zero offset

The workpiece offset config dialog features some radiobuttons to select most common workpiece origins, and provides a custom option where the user can enter values for workpiece’s X and Y axis zero. A function in the background calculates the required offset relative to the engrave spiral’s starting point.

Normally, I wouldn’t want the GUI to do calculations or a lot of branching logic because it’s harder to verify as test fixtures become larger and more difficult to maintain. Still, in this case I decided to do the offset calculation with help of a lookup table within the GUI module to avoid having to communicate through all GUI modules to platform and back (4 classes total comms length).

 def _get_xy_offset(self, offset):
 """calculates an XY coordinate offset from a preset point, taking the selected tool diameter into account.
 :param offset the selected offset from the above enum class
 :returns a XY Point where XY0 is assumed for the engraving."""
 qr = self._qr_dimension
 d = self._tool_diameter
 offsets = {Offset.CENTER: Point((d - qr) / 2, (qr - d) / 2),
 Offset.TOPLEFT: Point(d / 2, -d / 2),
 Offset.TOPRIGHT: Point(d / 2 - qr, -d / 2),
 Offset.BOTTOMLEFT: Point(d / 2, qr - d / 2),
 Offset.BOTTOMRIGHT: Point(d / 2 - qr, qr - d / 2)
 }

 if offset in offsets:
 return offsets[offset]

Initial release V1.0

The initial release is a minimum viable product. It does its job but still has some flaws (like throwing an error when there’s no tool selected in the drop-down) and imperfections (like drawing a QR-code is really slow, re-drawing tilts the QR-code a bit etc.) but it should just work, helping the user avoid the worst blunders on their way.

I have tested the execution on my CNC machine and at least my phone is able to navigate to a website just by holding the camera at the workpiece 📱

Automated integration tests for the GUI: frustrating work

When my initial release was out, I wanted to add some tests related to the GUI. As I didn’t manage to have absolutely no business logic there, I at least wanted to verify the range checks as well as the communication with the platform to ensure that it all works as intended.

Stuck in Messagebox

For that, I wrote some tests that ran on my local machine with the disadvantage that when the test hit a warning or error dialog messagebox.showinfo / messagebox.showerror, a manual user interaction was absolutely required to have the window closed and the test to continue.

I didn’t find an easy way to circumvent that issue as the messagebox acts as an independent widget that cannot easily be tampered with. Even my tries to invoke a keystroke as a side effect from within the test did not prove to be fruitful (and, in the end, wouldn’t have been a clean sollution, I suppose).

My solution after a couple of hours cursing under my breath is that the messagebox widgets are injected into the GUI classes that use it:

@patch('src.gui.gui_tool_manage.GuiToolManager')
 def setUp(self, mock_guitoolmanager):
 tk.Tk() # required to have tk variables properly instantiated
 self.mock_guitoolmanager = mock_guitoolmanager
 self.mock_msg = MsgBox()
 self.mock_msg.showinfo = MagicMock()
 Persistence.set_mock_msgbox(self.mock_msg)
 self.config_tool = GuiConfigureTool(self.mock_guitoolmanager, self.mock_msg, {'padx': 5, 'pady': 5})

This way, I can throw mock_msg into the gui class so not the real showinfo was taken but instead the mock would know whether it has been called as intended. I could add it to the test fixture so my test code itself could stay DRY.

It adds more arguments to the constructor and delegates the instantiation to a higher level class, but still the usage inside the class under test stays trivial.

Static class issues

As the persistence class I mentioned above is static in a way that it doesn’t require instantiation, I had the next issue built in my code right from the start: An integration test would now try to write to an actual persistence.dat file, a dependency that I didn’t want to have, especially not in the continuous integration environment.

So I moved Persistence up the stream into the main GUI file that wouldn’t be tested following Steven Anderson’s splendid blog post. The main GUI should only contain declarative code anyways, and the loaded data is then cascaded into the classes that use it instead of the persistence class that actually does the load/save operation.

Future releases

This is what I’m planning for future releases:

Drawing speed of the QR-code must be increased
Graphic bugs (scaling, tilting) and blurry edges of drawing screen to be fixed
CNC paths to be optimized for lower manufacturing time
Bugfixes

At least if there are any; let’s see how much I’ll be using this tool. The things that annoy me most will likely be fixed first.

Ah, one last thing: be rewarded for following through on this article. You can download your copy of QR-codengrave for free to play around with, and for home use. If you’re planning for commercial use, please let me know upfront.

This site Demo

Schallbert — Mon, 01 Jan 0001

Project stats

Difficulty: medium 3/5
Cost: 0€
Time: ~15h

The idea

The idea originated through my MobFobAmp project of which I didn’t have a github repository as there was no code or similar documentation to keep. So I decided I wanted a website so I could host instructions for this Open Hardware project for others to build their own copy. I wanted an easy-to-host solution that wouldn’t take too much work to set up. As I don’t have an own server, I asked for help. moritzmar recommended to use github pages as a site host to start with for easy auto-deployment and global scope. So I gave it a try.

Getting started

I figured out that a static site would be the right thing to use, without the need for a database or huge Content Management frameworks as my site layout should be simple and minimalistic; I just wanted to host this one project in the beginning. Github recommended to use Jekyll

Jekyll Logo ^[1]

as a site generator. I had some trouble with the Windows Subsystem for Linux (WSL) as it wouldn’t accept my terminal inputs at some point, so I followed the guide and used the RubyInstaller which worked right out of the box for me.

Site setup

I realized that I might want to document more of my projects, and not only the “hardware” ones. That’s why I came up with the idea of grouping projects together, each with a header, a little teaser text, and a thumbnail image. Clicking on which would lead to the individual project page. Which would involve a lot of programming for page setup and layouting which I didn’t want to spend too much time on - my other projects were waiting…

First try with Themes

So I followed another suggestion by Github pages: The usage of pre-defined “Themes”. I first used the Agency Theme. This one has a great landing page with a huge picture and it supports collections that enable the group-the-project thing that I wanted. Unfortunately, the theme is a “remote theme” so customization was limited, e.g. I didn’t manage to have a colletion point to another page, it would always open a “popup” and I just didn’t want this. (Later, I figured out that remote layouts can be overridden, but that was too late so I already switched to another theme supplier…) The theme I’m currently using the Minimal Mistakes Theme (see site footer) which is very well documented. After downloading and a couple of first steps, I struggled with custom colors (_sass) and the collections - again - as I didn’t understand at first how Jenkill would do the bindings behind the scenes. Some of my struggles can be found in the Blog posts from June 2021.

Site layouts: hero screens

A bit of thinking should go into how the site layout should be like, to make navigation easy and intuitive. The approach will differ with every page type due to differences in complexity and document structure.

Take the landing / home page for example, I want the navigation to be sticky so that the site’s categories are always visible. It shall receive a huge, wide overlay image with the page description. Below, latest posts will be listed. This is the example for the collection’s page hero screen:

The projects and their contents shall contain a page navigation in the left sidebar, may contain a scrollable header image, and has a wide setting to display a high amount of text in the content area.

The “hero screen” scetches for this site’s layouts can be found here.

Collections

Collections are a great way for grouping content. Look at my Open Hardware Projects page. You can find all projects in this category with a little teaser image and description here to get an overview.

This is how the collection’s configuration can work out:

Think of a fitting name for your collection, e.g. electronics.
Collection name is defined in _config.yml as collections: electronics.
Collection name has to be set in #defaults as type: electronics so that Jekyll knows how to interpret the actual content.
Collection page has to be created (e.g. in the _pages folder) with the attribute collection: electronics. This is the page that will actually display the overview of items in the collection.
Content folder with the same name as the collection’s name with leading underscore has to be created: _electronics.
All documents in this content folder will show up in the collection page automatically after a full build.
These documents should get a “front matter” (section before the actual content in the file, marked with ---) so that they can be displayed as described in the corresponding _layouts file, e.g. collection

More than the standard Theme

I have overridden some theme default and I’m using most of the existing Theme infrastructure. But my site has a couple of special needs so I had to customize and add Theme features. These are described in the following sections.

As my site navigation tree is pretty flat and I don’t have lots of pages to navigate to, I wanted the left sidebar to hold the page navigation rather than the site navigation. Thus, I wouldn’t need a right sidebar at all, so that there would be more space for the page content.

Table Of Contents on the left

As written here and there, I never was 100% happy with the way my project pages looked like:

The original idea of the theme is that the content is framed by two sidebars: The left one can either display Author information, custom content, or the site navigation. The content area is configurable between wide and normal, in the latter case leaving space for a second sidebar on the right. The right sidebar is configurable to not be there at all or to show the page’s table of contents.

So the target I set was to eliminate the right sidebar for my design, and instead of having the site navigation in the left sidebar, move the page navigation there.

It’s all about the /_layouts folder that keeps information about how a page layout should look like. The page design in question is called single.html and it at some point calls include sidebar.html which resides in /_includes/sidebar.html. So I thought of a new value I could use to control showing the table of contents within the left sidebar, chose toc_left and added a condition:

<div class="sidebar sticky">
<!-- ...
{.% if page.toc_left %}
 {.% include toc_left nav=page.toc_left %}
{.% endif %}
... -->
</div>

(By the way: I had to comment-out and add .’s to the jinja code, Liquid would otherwise interpret these as commands!)

Implementing the Table Of Contents

I wanted it to show up in the left sidebar container. So I just had to create a toc_left file that would call the included toc generator like this:

<aside class="toc_left">
<nav class="toc" markdown="1">
<header><h4 class="nav__title"><i class="fas fa-{{ include.icon | default: 'file-alt' }}"></i>
<!-- ...
{.{ include.title | .default: site.data.ui-text[site.locale].toc_label }}</h4></header>
{.% include toc.html sanitize=true html=content h_min=1 h_max=6 class="toc__menu" skip_no_ids=true %} 
... -->
</nav>
</aside>

Audio Embeds

Audio embeds are these little player-like things on a website, often forwarding to a streaming platform or similar. I need them for some of my audio projects like this mobfobamp: just a caption and the player below.

This post helped me a lot, so I could just add it to the _include, modify it a bit, and it worked!

Placing the favicon

I followed this guide to place a favicon for the site and it worked flawlessly.

The favicon’s path has to be valid independently of the folder the rendered page resides in. I had to add ../ according to the pages’ depth within the site’s folder structure to make sure that the favicon could be displayed not only on the “home” page, but also from sub-paSges.

Optimizing for search engines

As my site was not found by search engines for a couple of weeks, I decided to perform so-called “SEO” (search engine optimization). That’s why I added jekyll-seo-tag and e.g. reworked all my links following Juliette Sinibardy’s SEO 101. When complete, I put my webpage through Lighthouse’s web-analysis audit.

For a first try, I think it looks pretty well. Biggest issue on the performance side seems to be CSS overhead.

Content Security Policy

The Lighthouse audit revealed that it would be a good idea to restrict external scripting to my site, so I introduced a Content Security Policy (CSP) like this:

webrick:
 headers:
 Content-Security-Policy:
 default-src 'self'
 script-src 'self';
 connect-src 'self';
 img-src 'self';
 style-src 'self';
 font-src 'self';

Initially, unfortunately, my home page’s backup image wouldn’t show, my embed-audio files wouldn’t play and all icons loaded externally were gone as well. Adding media-src: 'self'; and some more reading in Jakob Wilforss’ blog and especially Lighthouse’s recommendation helped to solve this. By using the Browser’s Web developer Tools, I figured out the missing resources the theme is loading in the background, so my updated CSP looks like this:

webrick:
 headers:
 Content-Security-Policy:
 default-src 'self' 'unsafe-inline' https://cdn.jsdelivr.net/; 
 script-src 'self';
 connect-src 'self';
 img-src 'self' 'unsafe-inline' https://upload.wikimedia.org/wikipedia/commons/ https://raw.githubusercontent.com;
 style-src 'self' 'unsafe-inline' https://cdn.jsdelivr.net/;
 font-src 'self' 'unsafe-inline' https://cdn.jsdelivr.net/; 
 media-src 'self'; 
 object-src 'none';
 base-uri 'none';

Errors

I’m collecting some simple errors here that cost me quite some time fixing.

Jekyll serve: `UTF-8` incompatible character encoding

You enter bundle exec jekyll serve and get a strange UTF-8 “incompatible character encoding” error? The cause is simple: One of your files most likely has no header that can be converted into the front matter of the page that is generated from that file. Just add your usual header to your file like so and it will just work:

---
title: "Feeds and Speeds database"
description: "for (light) CNC milling machines"
header:
 image: "blah/testimage.jpg"
[...]
---

Webpage is not being displayed at all: `404`

This error was very, very hidden. I found it by accident browsing my site online only, as localhost serve was not affected. The symptom is a no-show blank page.

Cause: Due to translation, my paths are getting longer and I wanted subpages to not have extremely long links. That’s why I was using the permalink: Attribute of the pages’ fontmatter. Now, only if you have a “trailing slash” here, the page will display on the remote server.

# Wrong:
---
title: "This Page won't be displayed"
permalink: /projects/pageerror
# [...]
---

# Correct:
---
title: "This Page will be displayed!"
permalink: /projects/nopageerror/
# [...]
# Unfolds on the english blog to:
# /en/projects/nopageerror/
---

Site hosting

This site itself is a normal Gitea repository. It is built automatically with Gitea Actions once changes are added to the repository’s main branch.

Summary

To get familiar with how Jekyll creates the site, how the theme would interact with my commands and how to override Theme behavior cost quite a couple of hours, but I think this site has a clear structure and a more or less “professional” look - And I didn’t have to write a single line of JavaScript or HTML to get this done. Nice!

Contents written in markdown *.md
Basic site configuration in YAML _config.yml
Page layouts written in html and jinja *.html, placed in _layouts and _includes
Page variables and appearance in SASS *.scss, automatically generating css files

Contribute?

You have ideas, criticism, or interesting stuff you’d like to let me know? Get started on my Discussions page!

Tonuino - alternative Firmware

Schallbert — Mon, 01 Jan 0001

Project stats

Difficulty: hard 5/5
Cost: 0€
Time: ~400h

Impressions

My alternative tonuino-Firmware in action.

NFC-Tags are birds that can be placed in the bird's house to play an album.

A microcontroller implements my encoder library to run this buttonless design of Tonuino.

Tonuino in an old battery-powered guitar amp.

The player is controlled via three buttons that I took from an old computer keyboard.

A software written for the Arduino Framework that controls an Mp3 Player (with SD card), an NFC Tag Reader, and provides user controls that together work like a jukebox that can be controlled via NFC Tag. For example, an NFC tag can be linked to a folder on the SD card containing audio files. The tag will also store information about the requested play mode for this album, e.g. “Random”. Once a linked tag is placed on this jukebox, the configured folder will play with the selected playmode. User controls allow to select the track, change volume and even to navigate a voice menu that allows deleting a tag, linking/ configuring a tag, and to lock/ unlock the input keys if required.

Motivation

Idea and fantastic execution found on Thorsten Voß’s blog. It began with a feature (encoder support) that I wanted to add. Unfortunately, the original code was a single-file application with several thousand lines of code and poor readabilty, so I had a hard time getting my changes in. So I re-wrote the code from scratch for better structure, readability, maintainability, and extensibility. By doing this, in parallel I read some books about Object Oriented design in C++, Clean Code and Software Design Patterns and applied some where I tought they’d fit in. This way, I was able to both improve modularity and readability of the code, plus I taught myself how to write code in a “bigger” project.

Hurdles to overcome

This was first real OO C++ project, and, maybe, it was a little big to start with. All the new-to-me concepts, writing to an interface, first-time usage of Platformio as IDE, utilizing the googletest unit test framework, using coding patterns like factory or Dependency Injection took nearly a year of evenings before this project could be completed. And there’s always something left to refactor and to improve.

Quick facts

Unit test suite of 250+ test cases
Serial debug output configurable
Loosely coupled C++ OO architecture
Individual modules with clear tasks, scaleable and easily accessible for future feature add
custom Hardware abstraction layer; most of the code should be portable to other mcus without changes
custom Dependency Injection framework (Loader class)

Features

Configurable user input (buttons or encoder)
Auto-Poweroff when no button pressed for a configurable time
Power on via “play” button press
Autoplay feature
Status Led feature
User Input lock/unlock feature
Multiple playmodes [Album, Random, SaveProgress, OneTrack] available per Nfc Tag
Voice menus for linking or deletion of Nfc Tags
Optimized for battery applications (e.g. powerbank) using sleep states
Low power consumption @5V: ~40mA in idle, ~75mA playing medium volume
Config file for init volume, lullabye timer etc.
Auto-recovery from stuck prompts
Voice prompts for most common state errors

Features not incliuded

No powerbank state of charge detection
No config settings menu (init volume, lullabye timer duration, standby duration etc.)
…

Documentation

The Project is build for the Arduino Framework - tested on an Arduino nano board - using the PlatformIO IDE. The following sections show the design.

Project Module overview

Folder name in `/lib`	Purpose
Arduino	minimalistic close-to-Hardware implementations, not unit-testable
Arduino_HardwareAbstraction	Hardware abstraction to enable portability and testing
Config	System configuration parameters
Folder	Playlist and playmode business logic
Loader	Dependency Injection Framework
MessageHandler	System messages and Debug Framework
Mp3	Mp3 control (Status, Folder, Prompts, Advertisements)
Nfc	Tag control (Status, Read, Write, Delete)
PowerManager	Control Status Led and Keep Alive based on system state
Tonuino	Main task scheduler
UserInput	Button or Encoder input handling
Utilities	Timers, Led Control, Pin control
VoiceMenu	Link / Delete / Config menu business logic

External Libraries

Auto-install through Platformio on initial build through Library Dependency Finder.

Class Diagrams

Class diagrams would have been too much non-automated work. Take this instead, please. Software module overview It should give an accurate understanding of how the software module are interacting and what APIs the modules are offering.

Get started!

It’s DIY time! Clone my repository to get started with. Read the README.md document. Build it with PlatformIO. Below you can find what you need to get started with your unique speaker design.

Testing

Unit tests

Unit tests are written with the gtest C++ unit test framework. The reside in the test/desktop folder. Note that googletest will require gcc with some libraries to be installed, how to can be found in one of my other projects. Once these prerequisites are complete and a simple ASSERT_TRUE(false); test fails as expected using the gtest environment, the project’s Unit Tests can be built and run using pio test -e desktop -f desktop, detailed out here, in the PlatformIO CLI (terminal).

Acceptance tests

Although these tests could be automated, it is much easier to just perform those tests by hand after you programmed your µC and put all electronic components together. Each line in the tables below is a test case on its own. The test suites (Heading name) do have a System Pre: property, that needs to be re-established prior to executing the individual test cases. If the expectation clause is met, the test is a PASS.

Switching ON

System Pre: System is OFF

Action	Expectation
press Play/Pause button	LED flashes slowly?
press Play/pause button	Welcome prompt plays?

Switching OFF

System Pre: System is ON

Action	Expectation
Playback on pause, no button input	System switches off after a time?
System shutdown	Farewell prompt plays?
System shutdown	LED switched off?

Play Help Prompt

System Pre: System is ON

Action	Expectation
Long press Play/Pause button	Help prompt plays?
Help prompty playing	Can be interrupted with any button press?

Behavior without Tag

System Pre: System is ON, no Tag present

Action	Expectation
press Play/Pause button	prompts “couldn’t find track” error?
press Next button	prompts “couldn’t find track” error?
press Prev button	prompts “couldn’t find track” error?
doubleclick Play/Pause button	Delete Menu prompt played?

Behavior with Tag

System Pre: System is ON, linked Tag available

Precondition	Action	Expectation
no Tag placed	place known Tag	starts Playback of correct Folder?
active playback	press Next button	plays next track?
active playback	press Next button	next track in accordance with selected playMode of Folder?
active playback	long press Next button	increases volume?
active playback	press Prev button	plays previous track?
active playback	press Prev button	previous track in accordance with selected playMode of Folder?
active playback	long press Prev button	decreases volume?
active playback	press Play/Pause button	pauses track?
paused playback	press Play/Pause button	resumes track?
active playback	doubleclick Play/Pause button	Lock prompt played?
active playback	doubleclick Play/Pause button	Playback resumes after Lock prompt played?
active playback	doubleclick Play/Pause button	locks button input?
locked button input	doubleclick Play/Pause button	Unlock prompt played?
locked button input	doubleclick Play/Pause button	Playback resumes after Unlock prompt played?
locked button input	doubleclick Play/Pause button	unlocks button input?
paused playback	doubleclick Play/Pause button	Playback resumes after Lock prompt played?

Behavior with unlinked Tag

System Pre: System is ON

Action	Expectation
place unlinked Tag	plays LinkMenu Prompt?
navigate through LinkMenu	plays Configuration Success Prompt?
place Tag again	plays linked folder?

Hardware

Material list

Minimal discrete parts approach

Amount	Item	Purpose
1	Arduino (e.g. Nano)	Brains
1	Dfplayer Mini Mp3 Player	Mouth
1	Micro SD Card	Memory
1	MFRC522 Nfc Reader	Psi-Sense
1	Bistable Relay 5V, e.g. HFE20	Coffee - no sleep
2	Diodes e.g. 1n4007	vene valve
1	PNP Transistor e.g. BC327	Coffee maker switch
2	Resistors 1k	blood-stream conditioner
1	Resistor 220	dim blink strength
1	LED, color of choice	blink
1	Powerbank/ 5V supply	Food
3	push buttons, e.g. cherry MX keys	pressure sensors
1	speaker, e.g.5W@4Ohms	Lungs
1	USB-A plug	Straw
1	housing	Body

Additional coponents and tips

Jumper wires, PCB, Sockets, and expendable materials as you deem fit Same as for the “original” project; Alternatively buy one rotary encoder instead of push buttons and configure the project to use it. As the system is optimized for battery usage, take a bi-stable relay or a JFET transistor (with low gate voltage and low Source-Drain voltage drop) for keepAlive functionality mitigating additional current add through always-on relay coils. A cheap powerbank will last for many hours. Alternatively, e.g. 3xAA batteries can be used - but the Dfmini will really make some noise if not enough current can be supplied due to running resets (and may even get bricked through current surges, so beware).

Code on Schallbert's Blog

Auto-Upgrade Docker Images

Why automatic updates?

Updating Docker images with Watchtower

A Script-Based Solution

Complicated: docker pull, comparing against existing images, and deploying if necessary

Simple: Include everything in a single command via an option

Cron job entry

Which version was installed?

Notification upon update

PHP2shell: Critical vulnerability

Step 1: Recognition: We have a problem.

Status: October 21, 2026, 10:30 AM CEST

Step 2: Immediate Measures: Taking Nextcloud Offline

Step 3: Interim Containment: ​​Having the Nextcloud Instance taken offline

Status: October 21, 2026, 11:43 AM CEST

Status: October 21, 2026, 12:01 PM CEST

Step 4: Monitoring the Situation - Patch Availability and Initial Deployment

Status: October 21, 2026, 14:20 CEST

Status 21.10.2026 16:34 CEST

Status 21.10.2026 19:52 CEST

Status 21.10.2026 22:40 CEST

Status 21.10.2026 23:00 CEST

Step 5 Insights and Conclusions

Step 6 Actions

Addendum: Oct 22, 2026, 09:30 CEST

Step 7: Initial Reflection

Status: October 24, 2026, 16:30 CEST

Status: October 27, 2026, 17:00 CEST

Step 8: Analyzing observations and adjusting information channels

Understanding the PoC

Hardening

Later that same day

Status as of October 28, 2026, 21:30 CEST

Use Password manager with hardware security keys

The Password Manager

Not the Firefox Password Manager

Not Vaultwarden

KeePassXC

Setting up KeePassXC with an HSK

YubiKey

Nitrokey

YubiKey Compatibility

Dead End A: Cloning entries from the database

Dead End B: Copying the entire database and updating the HMAC

Solution: Converting the HMAC secret

KeePassXC: Keeping Passwords Neatly Organized

Password Families

Password groups

Icons 🔑

Critical vulnerabilities in Gitea (and Forgejo)

Becoming Aware

Gitea as a Target

Why is Gitea an attractive target?

What is RCE?

The Gitea Ecosystem

Vulnerability via Open Registration

Vulnerability due to default configuration for reverse proxies

The Security Vulnerabilities

Side Note: The CVSS Rating

Significance of Individual CVSS Metrics

CVE-2026-20896: Authentication bug in Gitea through reverse proxies

CVE-2026-59774: File-read flaw affecting any file readable by the Gitea service account

CVE-2026-60004: RCE allowing an attacker with standard user privileges to execute arbitrary shell commands as the Gitea OS user.

The Situation

The Cryptominer

Potential for Even Greater Damage

How do I detect an attack?

In Case of Infection: Quarantine

Solutions

Update Gitea!

Running Gitea and the runner in “rootless” mode

Additional measures (hardening)

CVE-2026-20896: Restricting network access

CVE-2026-59774: Update Gitea!

CVE-2026-60004: Disable self-registration if possible

Outlook and Conclusion

Immediate: Harden the system

Short-term: Automated updates

Medium-term: Time for monitoring

Complicated: `docker pull`, comparing against existing images, and deploying if necessary

Step 3: Interim Containment: Having the Nextcloud Instance taken offline