This Site on Schallbert's Blog

Continuous Deployment for a Hugo site

Schallbert — Tue, 15 Sep 2026

I have just migrated my website from Hugo to Jekyll. Now, I want to automate the build and deployment process so that changes are rolled out as soon as I finish an article and push it via git push followed by a merge into main to my Gitea instance.

The Command Chain

I rent a VPS where I run all my applications in Docker. The task now is to create a workflow in Gitea (“Actions”) that checks out the repository update, passes it to Hugo to generate the HTML pages, and then saves the output to the disk in a location where my web server can read and serve the files.

Lessons Learned

I had already managed to set this up for my Jekyll website build and deployment, so I can draw on some previous experience.

Typically, standardized actions are used for this purpose. I’ll simplify things significantly here and list only the key components, without deep-diving in messy details like permission settings (is Hugo allowed to write to this folder? Does the hugo user have sufficient privileges?):

checkout makes a specific version of the source repository available
actions-hugo installs Hugo within the Action’s virtual machine so the website can be built
upload-artifact uploads the site generated by Hugo to the web server

However, since I am operating within Docker on the same machine, I can skip the hassle of logging into the web server “from the outside”. So, I’m taking a different approach.

Docker shared volumes

I can make the built files directly available to my Caddy application by passing the volume (that Caddy reads from) through to the Action.

Heads up: Action volume != Hugo volume

But this doesn’t work with actions-hugo: It is not a Docker container itself, so it cannot mount volumes. While it does successfully build the website, it sends the artifacts into oblivion as soon as the Action finishes running.

So, I need a Hugo runtime that executes inside a Docker container itself, following the DinD approach. I can then pass the volume with write access.

The workflow

But which Docker container should I choose?

Imagine here several hours of research and numerous failed attempts while setting up the workflow on my server.

My investigation revealed the following:

checkout@v4 requires Node.js—specifically, at least version node:18. The Docker container must have it installed.
The action requires git. Ideally, the Docker container should already include this as well.
Hugo needs go. That makes sense.

Adjusting the runner operating system

My runner from 2023 did not age well. It is based on node:16-bullseye, a Debian-based VM. I experiment a bit with Alpine, but that installation is too minimal for my needs and would require too many manual adjustments. After all, the runner has to execute other workflows besides this one. Ultimately, I decide to stick with Debian and just upgrade the Node version. The implementation in the runner configuration looks like this:

# /gitea/runner/config.yml

# The labels of a runner are used to determine which jobs the runner can run, and how to run them. 
# Like: ["macos-arm64:host", "ubuntu-latest:docker://node:16-bullseye", "ubuntu-22.04:docker://node:16-bullseye"]
# If it's empty when registering, it will ask for inputting labels. 
# If it's empty when execute `deamon`, will use labels in `.runner` file. 
labels: [ubuntu-latest:docker://node:20-bullseye]

Afterwards, the runner container must be restarted. Ideally, you should reboot Gitea as well; this ensures that runner registration works reliably.

Writing the workflow script

To handle the dependencies described above, I’m opting for a “batteries-included” approach. This allows me to assemble the components without manual adjustments and rely on the long-term maintenance of the containers, given their widespread use (with pull counts in the millions). This website was very helpful in making my selection.

Finally, I create my workflow based on the container from Hugomods.

# /.gitea/workflows/build-deploy-with-hugo.yml

name: Deploy Hugo site
run-name: ${{ gitea.actor }} builds Hugo site
on:
 push:
 branches:
 - main

jobs:
 # Build job
 build:
 runs-on: ubuntu-latest # this is the "label" the runner will use and map to docker target OS
 container: 
 image: hugomods/hugo:latest
 volumes: 
 - /tmp/blog-artifacts:/tmp/blog-artifacts

 steps:
 - name: --- CHECKOUT ---
 uses: actions/checkout@v4
 with:
 submodules: recursive  # Fetches Hugo themes
 fetch-depth: 0

 - name: --- BUILD WITH HUGO --- # chown -R hugo /tmp/blog-artifacts
 run: |
 hugo \
 --minify \
 --cleanDestinationDir \
 --destination /tmp/blog-artifacts

Cleaning the destination folder

In the example above, I opted for a “direct deployment”. This is a basic approach without a safety net, and without copying artifacts around or restarting the server. The reason: the Hugo build process takes less than 2,000ms for my site. It is only within this window that Caddy could theoretically encounter and serve “corrupt” data. Specifically, the window is even smaller as occurrence is limited to the Hugo build writing artifacts to the disk.

To minimize inconsistencies regardless, I use the --cleanDestinationDir option with Hugo. This ensures the destination directory is always clean; consequently, in the event of an issue, Caddy won’t display a broken site but will instead generate a 404 error.

Submodules

As noted in my project documentation, you can use “submodules” in Hugo to load themes. These are integrated into the project and—provided they are linked correctly—can be utilized within the workflow:

with:
submodules: recursive  # Fetches Hugo themes
fetch-depth: 0

If everything is configured correctly, you should not encounter the following error message:

# workflow action's error message due to incoherent submodules' commit references
# actions/checkout@v4:
Fetching submodules
/usr/bin/git submodule sync
/usr/bin/git -c protocol.version=2 submodule update --init --force
Submodule 'themes/terminal' (https://github.com/panr/hugo-theme-terminal.git) registered for path 'themes/terminal'
Cloning into '/workspace/schallbert/blog-hugo/themes/terminal'...
fatal: remote error: upload-pack: not our ref 719505fc89332baa69bffb90cee708ff124dd143
Fetched in submodule path 'themes/terminal', but it did not contain 719505fc89332baa69bffb90cee708ff124dd143. Direct fetching of that commit failed.
Error: The process '/usr/bin/git' failed with exit code 1
Error: Process completed with exit code 1.

A prerequisite is that the submodules have been correctly loaded locally:

schallbert@machine:~ git submodule add -f https://github.com/<my/hugo-theme>.git themes/my-hugo-theme
schallbert@machine:~ git submodule update --init --recursive

The status check must not return an empty string; instead, it must include a commit hash. Example:

schallbert@machine:~ git submodule status
44d9a1890d228745ffc300b37a7d73e940ef9fa9 themes/terminal (v4.2.5)

Further confirmation is provided by the .gitmodules file, which must contain a reference to the theme:

schallbert@machine:➜/blog git:(main) ✗ nano .gitmodules
[submodule "themes/terminal"]
path = themes/terminal
url = https://github.com/panr/hugo-theme-terminal.git

If this does not work, the submodules must be completely removed and then re-initialized as described above:

git submodule deinit -f themes/<my-theme>
git rm -r --cached themes/<my-theme>
rm -rf .git/modules/themes/<my-theme>

Common Issues: Hugo Versions and Deprecations

The checkout action completed successfully, but the Hugo build fails:

error calling partial: "/workspace/schallbert/blog-hugo/layouts/_partials/head.html:37:40": execute of template failed: template: _partials/head.html:37:40: executing "_partials/head.html" at <$.Page.Language.Locale>: can't evaluate field Locale in type *langs.Language
Error: Process completed with exit code 1.

Locally on my machine, Hugo builds successfully but issues a few warnings:

Changes to Language Identifiers and Evaluation as of v0.158

schallbert@machine:➜/blog git:(main) ✗ hugo build --logLevel info
WARN deprecated: site.Language.Locale was deprecated in Hugo v0.158.0 and will be removed in a future release. Use .Page.Language.Locale instead.
WARN deprecated: .Language.Lang was deprecated in Hugo v0.158.0 and will be removed in a future release.

You might think these errors would be easy to fix. Unfortunately, they can be very good at hiding:

in submodules loaded as third-party content (e.g., in themes). A project-wide search in the IDE usually excludes submodules and yields no results.
as variables in your own code that access the old parameters only indirectly: Up until now, I’ve been using .Page.Site.Home.AllTranslations to determine if I am currently on the “main language” version of a given page. The scoping alone is illogical, and in my opinion, it is rightly being removed. What I want to achieve can now be done much more easily using .Page.Language.IsDefault.

Updating Submodules

I see two possible approaches for handling submodules:

Contact the maintainers (or open an issue) to let them know their code is becoming outdated and ask them to update their themes.
Copy the affected files out of the submodule and integrate them into my own folder structure. In my case, for example, the affected language-menu.html would find a new home at /layouts/partials/language-menu.html. I can then fix the issues there myself.

Implementing Custom Solutions

If the error message doesn’t make the solution obvious, the Hugo community is a great resource. For instance, there is detailed documentation available regarding the deprecations in v0.158.

Comparing Versions

Hmm. As the image shows, my container uses the hugo extended package from hugomods, which provides version v0.154.5; I had selected a stable release for the Action. Let’s compare this with my local version, which now builds perfectly without a single warning:

➜ blog git:(main) ✗ hugo version
hugo v0.162.1+extended linux/amd64

The classic scenario: A change introduced in v0.158 altered the internal API. Older versions don’t recognize the new fields and throw errors. Newer versions, however, flag the use of the old fields as an issue.

I believe these problems can only be reliably avoided if the pipeline is identical for both local and remote environments. In other words: boot up a virtual machine on your local computer running the CI operating system and encapsulate everything within Docker. That certainly makes sense if you work in a larger organization where multiple users face this same issue.

To fix the problem, I switch to the image: hugomods/hugo:latest version within the remote container instead. However, I still end up with version mismatches between my local setup and the CD pipeline. The latest version frankly isn’t available for my operating system yet, and the Docker release also lags slightly behind the original releases.

The build and release times are fantastic, aren’t they? Getting a website change online in just 12 seconds, including spinning up the container and so on, is something I’ve certainly never experienced before. The process takes around 20 seconds when it has to fetch and download container references from scratch, such as when the latest tag in the registry is updated.

The linked article offers a detailed benchmark comparing Hugo and Jekyll.

Migrating from Jekyll to Hugo

Schallbert — Sun, 06 Sep 2026

Project Profile

Difficulty: Medium (3/5)
Cost: €0
Time: ~20h

In this project, I describe my migration from Jekyll using the Minimal Mistakes theme to Hugo, utilizing a heavily modified Terminal theme.

Why switch at all?

For quite some time, I hadn’t been fully satisfied with building my website using Jekyll. The trigger for the switch was a day when I suddenly couldn’t build the site locally due to a compatibility issue in Ruby. Standard commands like bundle update --conservative or bundle install didn’t help. Even manually installing the problematic packages using commands like gem install commonmarker-0.23.12 or gem install posix-spawn -v 0.3.15 -- --with-cflags="-Wno-incompatible-function-pointer-types" failed to resolve the problem.

Updating Ruby and Jekyll themselves didn’t work either. Besides, build times with Jekyll had become quite long. Since moving the site from GitHub Pages to my own server, I am no longer tied to Jekyll, making a switch more attractive.

This no-build-locally problem was just the final nudge that made me begin this laborious task. I hate it when technology doesn’t do what I want - even though I admit that often times, I am the root cause for that myself.

Further difficulties with Jekyll

CI/CD: My deployment pipeline had failed months ago due to similar errors, forcing me to revert to an older version of Jekyll. This increases technical debt and the attack surface of my GitHub Actions. I would have to look for a well-maintained container solution from another provider. Searching for and setting one up would require additional time.
RSS feed: I wanted to provide an RSS feed for my website. However, my experiments with Jekyll showed that only “posts” were included, not “announcements.” I wanted to fix this.
Image formats: Jekyll normally generates cover images (thumbnails) for posts within the RSS feed. Unfortunately, this stopped working after I switched from .jpg to the more space-efficient .avif format. Despite several attempts, I couldn’t get a solution approved through the pull request review process.
Icons: The theme I was using relies on Font Awesome to display icons and is quite large overall for a static website. Since I didn’t want to load assets from external sources, I stored the library locally. My plan was to select only the icons I actually needed and delete the rest. With the switch to Hugo, this is no longer necessary.

Opportunities

I expect Hugo to deliver faster build times, a simplified CI/CD process (without the need to install Bundler, Ruby, etc.), and a resolution to the difficulties described above. Hugo’s approach to dependency and package management differs fundamentally from Jekyll’s: additional content is integrated via Git Submodules, and dependency management relies on Go modules rather than a Gemfile.

The migration tool

The folder structures for content differ significantly between Jekyll and Hugo. I am therefore using an import tool included with Hugo, which correctly imports at least the articles located under /posts. However, specific content such as overview pages (/tags, /pages/error, /pages/legal), and pages hosting projects and announcements (/_announcements, /_projects), will not be migrated automatically and must be copied and modified manually.

Adjusting Front Matter

Many of the customizations I implemented in Jekyll either do not work correctly in Hugo or cause the build process to fail. Consequently, the front matter of every Markdown file must be adjusted and, in some cases, cleaned up.

---
# Alt: So kann die Front matter einer Übersichtsseite in Jekyll aussehen
lang: de
title: Beitragsarchiv
subtitle: Diese Sammlung enthält alle meine Beiträge
layout: collection
collection: posts
permalink: /posts-archive/
entries_layout: grid
classes: wide
author_profile: false
---

In my case, there are several theme-specific parameters in the front matter, such as layout, classes, and author_profile. The lang parameter from my Jekyll internationalization extension isn’t even valid Hugo syntax; Hugo expects the language setting either as a field under params: lang or defined at the folder level. Other custom parameters tailored to personal preference can also be defined and passed via params.

Permalinks are no longer included in the front matter; instead, configuration is handled centrally.

Hugo-compatible front matter might look like this:

---
title: Post Archive
description: This collection contains all my posts
layout: collection
---

Theme Configuration

I can install a basic configuration for my theme using the following command:

➜ blog git:(main) ✗ git submodule add https://github.com/panr/hugo-theme-terminal ./themes/terminal

To activate it, I need to specify it in the main configuration file:

# /hugo.toml

# Add it only if you keep the theme in the `themes` directory.
# Remove it if you use the theme as a remote Hugo Module.
theme = "terminal"

And just like that, Hugo builds the site using the Terminal theme and its default colors.

However, the layout isn’t vertically centered and is too narrow for my taste; the text is quite small, and I want “my” blog colors to be different as well. Therefore, I am creating a .css file that overrides some properties of the base theme. It must be located in the /static folder and named after the theme.

/* /static/terminal.css */

:root{
 --background: #252a34;
 --background-highlight: #475064;
 --foreground: #d2eaef;
 --foreground-dimmed: #8b9ea2;
 --accent: #25a679;
 --accent-highlight: #79ae9c;
 --radius: 8px;
 --font-size: 1.35rem;
 --line-height: 1.8em;
}

/* Use system fonts, reduce transferred kB */
body{
 font-family: system-ui, -apple-system, "Segoe UI", Roboto, Arial, sans-serif !important; 
}

/* make content wider */
body .container{
 max-width: 70rem;
 width: 90%;
}

And just like that, it looks almost like it used to.

Organizing the folder structure

Unlike the version of Jekyll I was using, Hugo features built-in internationalization (i18n). This requires me to restructure my folders and explicitly sort content into /de and /en directories. Instead of sitting at a higher folder level, content is now stored directly within the content folder—under /_pages, /_announcements, and /_posts—which makes things much tidier.

Before

.
├── _announcements
├── assets
│ ├── css
│ │ ├── fontawesome
│ │ ├── katex
│ │ ├── main.scss # <-- theme import, customization
│ │ └── webfonts
│ ├── js
│ │ ├── lunr
│ │ ├── _main.js
│ │ ├── main.min.js
│ │ ├── plugins
│ │ └── vendor
│ └── video
├── banner.js
├── CNAME
├── _config.yml
├── _data
│ ├── de
│ │ ├── l10n.yml
│ └── en
│ ├── l10n.yml
├── en
│ └── index.html
├── Gemfile
├── Gemfile.lock
├── _includes # <-- Layouts (HTML), partials, 3rd-party
├── index.html # <-- Landing page
├── _layouts # <-- Page layouts
├── _pages
│ ├── 404.md # <-- German version
│ ├── [...]
│ ├── en
│ │ ├── 404.md # <-- English version
│ │ ├── [...]
├── _posts/
│ └── en 
├── _projects/
├── _sass
│ ├── minimal-mistakes # <-- Styles (SCSS) for layouts and partials
│ ├── minimal-mistakes.scss
│ └── _video.scss
├── _site/ # <-- Build artifacts
├── staticman.yml

After

Hugo’s documentation provides a good overview of the folder structure. The root folders content, layouts, and static are the most important for day-to-day work with articles and the website’s appearance.

.
├── archetypes
├── assets
├── content
│ ├── de
│ │ ├── about.md
│ │ ├── announcements/
│ │ ├── posts/
│ │ └── projects/
│ └── en
│ ├── about.md
│ ├── [...]
├── data
├── hugo.yml
├── layouts
│ ├── _default
│ │ ├── baseof.html # <-- defines header, add-ons, content, footer structure
│ │ ├── index.html # <-- landing page
│ │ ├── list.html # <-- grid view
│ │ └── single.html # <-- post view
│ ├── _markup
│ │ └── render-link.html # <-- link highlighting and function (referrer, tabs)
│ ├── _partials
│ │ ├── cover.html # <-- cover (thumbnail) image rendering
│ │ └── math.html # <-- math rendering (katex)
│ └── _shortcodes
│ ├── audio.html
│ ├── image.html
│ └── [...]
├── resources
│ └── _gen
│ └── assets/
├── static
│ ├── assets
│ │ ├── audio/
│ │ ├── css/ # <-- contains overrides for custom layouts 
│ │ ├── docs/
│ │ ├── images/
│ │ ├── js/ # <-- math (katex) JS lives here
│ │ └── video/
│ └── terminal.css # <-- Central theme override (colors, formatting, styling)
└── themes
 └── terminal # <-- GIT submodule: Vanilla "Terminal" theme

With both solutions, layouts are defined in HTML and styles/formatting in CSS. However, since I had to use more plugins with Jekyll, the setup is more extensive.

Manual Adjustments: Shortcodes

Hugo’s shortcodes allow you to define non-text content (media, links, formatting, styles) that is then rendered according to specific rules. To do this, you create an HTML file that defines the wrapper and includes the relevant CSS classes. You also add the desired formatting to the theme’s overriding CSS file.

Since I am by no means an expert, I simply experiment until I am happy with the layout and no longer see any artifacts or overlapping elements.

.
├── layouts
│ └── _shortcodes
│ └── my-shortcode.html # <-- Custom shortcode file
├── static
│ └── terminal.css # <-- Central theme override (colors, formatting, styling)

Syntax: Hugo and Jekyll

In Hugo, shortcodes are represented in the Markdown file using the following syntax:

{{< shortcode-html-filename options >}}
{{< image src="/path/to/image.avif" alt="Image: alt text" position="right" >}}

By comparison, shortcode-like decorators in Jekyll using Liquid, the templating language employed by Jekyll look like this:

### Liquid in interpreter code
{% Liquid bracket syntax %}
### Liquid "shortcode" pendant
{% include gallery id="gallery" caption="Eindrücke von **MobFobAmp**. Zum Vergrößern anklicken." %}
### Liquid decorator for layouting
{:.list-inline}

image

Example of image display. A centered image follows on the next line.

The standard Markdown image syntax [Alt Text](/path/to/image) renders as a centered image without text wrapping by default in Hugo. However, for a pleasing layout, I need a few variations that require some additional CSS and HTML code.

Centered image, no text wrapping
Left-aligned image, text wrapping on the right
Right-aligned image, text wrapping on the left

First, Hugo retrieves the file path and alt text. Then, it reads the desired formatting defined via the position variable from the Markdown file. Finally, it applies the image-wrapper class from the CSS file. To ensure better readability, the image alignment applies only to the desktop version of the site (>684px).

The theme customizations required to display images correctly are quite extensive. I also developed a gallery feature for posts containing a large number of images. Consequently, I wrote a separate article covering image formatting and alignment.

audio

I use standard browser-native features to display the audio player.

Your browser does not support the audio element.

Beispielplayer: A lead melody on electric bass

{{- /* /layouts/_shortcodes/audio.html */ -}}

{{ $src := .Get "src" }}
{{ $caption := .Get "caption" | default "" }}

<div class="media-wrapper">
 <div class="media-container">
 <audio controls>
 <source src="{{ $src }}" type="audio/mp3">
 Your browser does not support the audio element.
 </audio>
 {{ with $caption }}
 <div class="media-title">{{ . }}</div>
 {{ end }}
 </div>
</div>

In css, I use a wrapper and a media container to display titles within the player frame and to simplify formatting. Here, too, I have implemented custom behavior for a responsive layout on mobile devices or when the browser window is resized.

video

The CSS is identical to that of the audio element; there are only minor differences in the HTML. The container I use supports both “embed links” from sites like PeerTube and local source files via the src= attribute.

Your browser does not support the video tag.

Video example (local): linear glides with ball bearing.

{{- /* /layouts/_shortcodes/video.html */ -}}

{{- $src := .Get "src" -}}
{{- $title := .Get "title" | default "" -}}
{{- $caption := .Get "caption" | default "" -}}

<div class="media-wrapper">
 <div class="media-container">
 {{- if hasPrefix $src "http" -}}
 <iframe src="{{ $src }}" frameborder="0" allowfullscreen></iframe>
 {{- else -}}
 <video controls>
 <source src="{{ $src }}" type="video/mp4">
 Your browser does not support the video tag.
 </video>
 {{- end -}}

 {{- if or $title $caption -}}
 <div class="media-title">
 {{- if $title -}}{{ $title }}{{- else -}}{{ $caption }}{{- end -}}
 </div>
 {{- end -}}
 </div>
</div>

Updates

My content ages, too. Sometimes I switch technologies or providers, and sometimes third parties move away from solutions I’ve described implementing here on the blog. That’s why I introduced update-shortcodes.

Links

I want the following behavior for links on my blog:

internal links should open in the same tab and redirect the user.
internal links should point to pages in the same language, rather than leading to the default language or back to the homepage.
external links should be marked with an icon: ↗
external links should open in a new tab: target="_blank"
external links should neither access the original content nor be able to see the referrer: rel="noopener noreferrer"

To achieve this, I create the file render-link.html in /layouts/_markup. I present the code and the background details in a separate post: Hugo Multi-language: Internal Permalinks

The CSS override looks like this:

/* /static/terminal.css */

.external-link .external-link-icon{
display: inline-block; 
font-size: var(--font-size); 
vertical-align: middle; 
transform: translateY(-0.1em);
}

Manual Customizations: Partials

You can think of partials as the layout building blocks for a Hugo website. They are stored in /layouts/_partials and override the defaults of the theme being used. Since the original file is no longer loaded at all, the recommended approach is to copy the original file into the folder and then customize it manually.

index

The blog’s homepage is entirely custom-written; it is not in Markdown format but in HTML. It displays its content differently than all other blog pages, so it wasn’t worth the effort for me to program specific templates. In the corresponding /layouts/_partials/index.html file, I load the necessary CSS files and define the page structure. The following table provides an overview of my changes.

File	Purpose	Customization
cover	thumbnail / cover image for overview pages, lists, and grids	Add custom cover image decorator, alt text
head	Defines Favicon, search console, page parameters, feeds etc.	Custom favicon path
header	Page header containing menu, logo, navigation etc.	Add logo and subtitle to menu, lang selector
logo	Custom page logo, title, subtitle	Full
math	Displays math content with Latex/Katex	None, implement math support

Math

My blog is designed to support mathematical notation. I use KaTeX for this.

{{- /* /layouts/_partials/math.html */ -}}

<link rel="stylesheet" href="/assets/css/katex.min.css">
<script defer src="/assets/js/katex.min.js"></script>
<script defer src="/assets/js/contrib/auto-render.min.js" onload="renderMathInElement(document.body);"></script>

<script>
 document.addEventListener("DOMContentLoaded", function() {
 renderMathInElement(document.body, {
 delimiters: [
 {left: '\\[', right: '\\]', display: true}, // block
 {left: '$$', right: '$$', display: true}, // block
 {left: '\\(', right: '\\)', display: false}, // inline
 ],
 throwOnError : false
 });
 });
</script>

cover image

I use the term cover to refer to a blog post’s title image. It is displayed on overview pages, in article and tag lists, etc., and is intended to complement the blog post’s content. My RSS feed should also use these images to provide an overview of various articles on my readers’ devices.

It is highly compressed and requires minimal bandwidth, resulting in a low resolution (440x220 or 640x352). To ensure a consistent look, the images are also always monochrome.

I use the following Markdown syntax to embed the cover image:

---
cover_alt: 'Image: Hugo console build output. It shows a full build time below 2sec. (~90% reduction over build time with Jekyll)'
cover: /assets/images/migrating-hugo-to-jekyll/hugo-jekyll-build-compare-cover.avif
---

Next, I create the file cover.html in the layouts/_partials directory and add the following code:

{{- /* /layouts/_partials/cover.html */ -}}

{{- /* Handles cover-image generation */ -}}

{{- $cover := false -}}
{{- $autoCover := default $.Site.Params.autoCover false }}

{{- if index .Params "cover" -}}
 {{- if .Resources.GetMatch .Params.Cover }}
 {{- $cover = (.Resources.GetMatch .Params.Cover).RelPermalink -}}
 {{- else -}}
 {{- $cover = absURL .Params.Cover -}}
 {{- end -}}
{{- else if $.Site.Params.AutoCover -}}
 {{- if (not .Params.Cover) -}}
 {{- if .Resources.GetMatch "cover.*" -}}
 {{- $cover = (.Resources.GetMatch "cover.*").RelPermalink -}}
 {{- end -}}
 {{- end -}}
{{- end -}}

{{if $cover -}}
 <!-- Cover image found -->
 <img src="{{ $cover }}"
 class="post-cover"
 alt="{{ .Params.cover_alt | plainify | default (.Description| plainify) }}"
 title="{{ .Params.CoverCredit |plainify|default (.Title | plainify) }}" />
{{- end }}

The code searches for and extracts the path to the cover image. If a valid path is found, the image is displayed as post-cover with the corresponding CSS.

I insert the cover image into the XML template as the first element within the <description> tag; the actual article follows only after that. Most feed readers then use this image as the cover for their overview lists.

Manual Adjustments: Layouts

To ensure my Hugo site continues to feel very similar to my previous Jekyll site, I need to make a few changes to the layouts. I have described them below.

baseof.html

This is where the base content is defined. Definitions for the header and footer are imported, and stylesheets are linked. The vast majority of the changes described above are imported here.

{{- /* /layouts/baseof.html */ -}}

<!DOCTYPE html>
<html lang="{{ $.Site.Language }}">
<head>
 {{ if .Param "math" }}
 {{ partialCached "math.html" . }}
 {{ end }}
 {{ block "title" . }}
 <title>{{ if .IsHome }}{{ $.Site.Title }}{{ else }}{{ .Title }} :: {{ $.Site.Title }}{{ end }}</title>
 {{ end }}
 {{ partial "head.html" . }}
 <link rel="stylesheet" href="{{ "/assets/css/gallery.css" | relURL }}"> 
 <link rel="stylesheet" href="{{ "/assets/css/media.css" | relURL }}">
</head>
<body>
{{ $container := cond ($.Site.Params.FullWidthTheme | default false) "container full" (cond ($.Site.Params.CenterTheme | default false) "container center" "container") }}

<div class="{{- $container -}}{{- cond ($.Site.Params.oneHeadingSize | default false) " headings--one-size" "" }}">

 {{ partial "header.html" . }}

 <div class="content">
 {{ block "main" . }}
 {{ end }}
 </div>

 {{ block "footer" . }}
 {{ partial "footer.html" . }}
 {{ end }}
</div>

</body>
</html>

list.html

I need a list view for overview pages such as Articles, Tags, and Projects. I want to display four items per row. Each item gets its own “card” featuring a cover image, title, short description, and, if available, additional details like reading time. Hugo’s paginator crawls the relevant folder and compiles the content, while my list.css stylesheet handles the formatting and display.

{{- /* /layouts/_default/list.html */ -}}

{{ define "main" }}
 <link rel="stylesheet" href="{{ "/assets/css/list.css" | relURL }}">
 {{ with .Content }}
 <div class="index-content">
 {{ . }}
 </div>
 {{ end }}

 <div class="posts posts-grid">
 {{ range .Paginator.Pages }}
 <article class="post on-list">
 <a href="{{ .Permalink }}" class="post-card" aria-label="{{ .Title }}">
 {{ partial "cover.html" . }}

 <div class="post-card-body">
 <h4 class="post-title">
 {{ .Title | markdownify }}
 </h4>

 <div class="post-excerpt">
 {{ if .Description }}
 <p>{{ .Description }}</p>
 {{ end }}
 </div>

 {{- if and (.Param "readingTime") (eq (.Param "readingTime") true) -}}
 <div class="post-reading-time">
 {{ .ReadingTime }} {{ $.Site.Params.minuteReadingTime | default "min read" }}
 </div>
 {{- end -}}
 </div>
 </a>
 </article>
 {{ end }}

 {{ partial "pagination.html" . }}
 </div>
{{ end }}

single.html

My single.html is almost identical to the default implementation found in terminal. I only added the option to include a “banner-image” associated with the title and imported the _partial/cover.html snippet to load a cover image.

<article class="post">
{{ with .Params.banner }}
<img
src="{{ . }}"
class="post-banner"
alt="{{ $.Params.banner_alt | plainify | default ' ' }}"
/>
{{ end }}

Given all these changes, one might consider forking the theme to create a custom version… 🤔

Quality Control

Another new addition to my website is a check for broken links. The site has grown up and now contains so many internal and external links that manual checking is no longer practical. The article on htmltest configuration for Hugo documents the tool I use and how I account for Hugo-specific quirks during the process.

Publishing

I also need an automated pipeline for Hugo. Whenever I push a change to the main branch, the website should be automatically rebuilt and deployed. I describe exactly how this works in the article Building and Deploying a Hugo Website with GitHub Actions and Docker.

Performance

What has all this effort achieved? For users, the appearance and navigation remain very similar to the Jekyll-based site. However, the improvements are evident in faster load times, reduced data transfer, and better overall clarity. I provide a detailed comparison in the article Jekyll vs. Hugo Benchmarking.

Schallberts Blog with Hugo

Schallbert — Sun, 06 Sep 2026

What’s happened?

To mark its fifth anniversary 🥳, I’m moving schallbert’s Blog onto a new technical foundation!

I have migrated this blog from Jekyll to the site generator Hugo. It was a major undertaking: I dedicated a large portion of my free time to it over the course of about four months. The lion’s share of that effort went into creating and refining the styling, adapting my content to the new environment, fixing broken links, and documenting the migration process.

Why did I do it?

I was increasingly at odds with Jekyll: the main reasons were the numerous dependencies prone to frequent breakage, the ever-increasing build times, and the complex environment required for deployment to my server.

My site now builds about five times faster than before.

It loads less styling (CSS, JS), and the images are even more space-efficient.

Plus, I no longer have to keep Ruby and countless gems up to date.

When choosing a Hugo theme, I prioritized functionality over mere visual appeal. As a result, the site is now much simpler and more performant.

What’s in it for readers?

RSS logo ^[1]

The blog now offers an RSS feed. Plus, my projects are now included in the tags view. Finally, the migration process spawned numerous articles; these are intended to make such a move easier for others than it was for me and to clarify Hugo’s structural concepts.

What’s left to do?

I haven’t quite finished the overhaul yet:

Optimize the website for mobile viewing ✓
Enable centered display of images at native resolution ✓
Ensure the feed correctly displays article cover images ✓
Achieve a more consistent look via CSS (colors, image borders, etc.)
Make overview pages span the full screen width and display larger images
Add missing English translations ✓
Implement tags more specifically and with clearer distinctions
Add a sticky table of contents on the left-hand side

These pages will see further detailed improvements over the coming weeks. I’m looking forward to it!

schallbert

htmltest configuration ideas for Hugo

Schallbert — Mon, 10 Aug 2026

I am now nearly finished migrating my website to Hugo. I had to make several adjustments to the front matter of each article so that they would function in Hugo much as they did in Jekyll. Changes to the folder structure, partly from the migration itself and partly from my experience as an author, meant I had to revise links in practically every article along the way. I now have nearly 340 of them, counting translations.

I cannot manage the sheer volume of links without some assistance. So, I need a tool.

htmltest

Htmltest is a utility that scans HTML files for issues. Among other things, it tracks down broken links. I install it using

# Red Hat-based Linux package manager install command
sudo dnf install htmltest

and run it against the HTML files built by Hugo in the /public directory:

htmltest public/
# [...]
✘✘✘ failed in 2.890119823s
2612 errors in 337 documents

That is quite a lot of errors! Furthermore, the way the information is presented seems rather cluttered to me: see the image.

Redirecting output to a file

Consequently, I am finding it quite difficult to work with this tool at first. I had assumed a few links might have broken, but I certainly didn’t expect thousands of errors. So I fell victim to the typical “not-invented-here” syndrome and programmed my own tool in Python, which, in hindsight, turned out not to be good enough. In short:

Htmltest is a good and incredibly fast tool. If you make its output readable and interpret it correctly, you can be quite happy with it. After a bit of experimentation, I arrived at the following command:

htmltest public/ 2>&1 | ansi2html > htmltest-output.html

Meaning: “Run htmltest in the /public/ folder. Redirect errors from stderr to stdout. Pipe the output to ansi2html¹. Write its output to a file named htmltest-output.html.”

That already looks much better!

Error categories

For my purposes, the errors found by htmltest can be sorted into a few categories:

Hugo-specific errors, such as “target does not exist” messages caused by livereload.js.
Incomplete links that work in practice but trigger “href lacks trailing slash” errors. These types of errors account for the lion’s share of the reported issues.
Lookup errors (GET <src> [...] failure in name resolution) caused by a lack of connection, insufficient permissions, active bot protection, or the target server being offline.
Hard “target does not exist” errors caused by broken links. These are the specific ones I actually wanted to fix.

Cleaning up the log

How do I ensure only category 4 errors are displayed? By making a few changes to the tools and resolving the trailing-slash errors.

Category 1 errors, triggered by Hugo’s useful livereload function, look like this:

target does not exist --- en/tags/tools/index.html --> /livereload.js?mindelay=10&v=2&port=1313&path=livereload

LiveReload is the feature that saves me from constantly hitting F5 in the browser window whenever Hugo builds the site after I save a Markdown file. However, I can temporarily disable it for the test. The following solution reduces the number of generated errors by one per HTML page:

hugo server --disableLiveReload

I avoid category 3 lookup errors by focusing on internal links, over which I have complete control. The -s, --skip-external parameter in htmltest helps with this:

htmltest -s public/ 2>&1 | ansi2html > htmltest-output.html
# [...]
✘✘✘ failed in 494.7678ms
446 errors in 337 documents

I can fix broken external links later. For now, I want to resolve the issues caused by the migration.

“Missing trailing slash”: Fixing errors via markup

I struggled for a long time with the many “href lacks trailing slash” errors, as I would have had to append a / to almost every Markdown link in every article. Fortunately, I found a solution in my render-link.html markup:

{{- /* /layouts/_markup/render-link.html */ -}}

{{- $url := .Destination | safeURL -}}
{{- $is_external := or (strings.HasPrefix $url "http://") (strings.HasPrefix $url "https://") -}}
{{- $is_anchor := strings.HasPrefix $url "#" -}}
{{- $is_asset := strings.HasPrefix $url "/assets/" -}}

[...]

{{- /* Add / to internal paths that have no file extension */ -}}
{{- if and (not $is_external) (not $is_anchor) (not $is_asset) -}}
 {{- $parsed := urls.Parse $url -}}
 {{- if and $parsed (not (strings.HasSuffix $parsed.Path "/")) (eq (path.Ext $parsed.Path) "") -}}
 {{- $url = printf "%s/" $url -}}
 {{- end -}}
{{- end -}}

[...]

Meaning: Internal links that point to neither headings (is_anchor) nor media files (is_asset) have a / appended if, and only if, they have valid syntax (parsed) and do not already end with a slash.

This resolves 95% of the error messages, resulting in cleaner HTML. I handle the remaining 5% by adding a few slashes in the Hugo configuration. Examples:

# hugo.yml
url: "/en/" # /en -> /en/
logoHomeLink: "/en/" # /en -> /en/

Done! All that remain are hard internal errors.

htmltest -s public/ 2>&1 | ansi2html > htmltest-output.html
# [...]
✘✘✘ failed in 359.28ms
102 errors in 84 documents

Manual Search and Fix

What follows is purely a matter of tedious legwork. I open the target document (the one being linked to) and check for inconsistencies, such as heading links that differ between languages. If I find any, I add specific anchor links to the heading.

### Heading {#heading3}

If everything is fine in the target file, I then open all the source files that use the broken link and correct it manually. In most cases, the target file has moved to a different folder, or the language prefix is incorrect.

The Result

After a few hours of exhausting, lab manual work, I am rewarded:

htmltest public/ -s 2>&1
Skipping the checking of external links.
htmltest started at 10:51:23 on public
========================================================================
✔✔✔ passed in 238.970354ms
tested 338 documents

Soon I’ll be able to go online with my new-old site!

ansi2html (or colorized-logs) is a utility that converts console output to HTML, including formatting and text colors. ↩︎

Hugo Multi-Lang: Internal Permalinks

Schallbert — Mon, 27 Jul 2026

Anyone building a multilingual blog with Hugo might be familiar with the following problem:

Internal links in articles across different languages point only to pages in the default language.

Configuration

The issue stems from the following setting:

# hugo.yml

defaultContentLanguage: "de"
defaultContentLanguageInSubdir: false
disableDefaultLanguageRedirect: true

My default language is German, and the corresponding articles are built into the blog’s root directory. No redirect to the /de language directory takes place.

The directory structure is as follows:

├── content
│ ├── de
│ │ ├── about.md
│ │ ├── announcements
│ │ ├── legal.md
│ │ ├── posts
│ │ ├── privacy.md
│ │ └── projects
│ └── en
│ ├── about.md
│ ├── legal.md
│ ├── posts
│ ├── privacy.md
│ └── projects

The Problem with the Default Language

I make articles in my default language accessible via URLs like example.com/my-post and example.com/my-2nd-post/ by using the slug parameter. In the secondary language, however, they appear as example.com/en/my-post and example.com/en/my-2nd-post/.

If I want to link two articles internally, the link always points to the article in my default language.

# English version of my post
# example.com/en/my-post
---
slug: my-post

---
Post's English content
[2nd](/my-2nd-post)

As a result, instead of the desired /en/my-2nd-post, I end up at the German version: /my-2nd-post. I find this annoying, as it would require me to manually adjust every link in every article. Unfortunately, Hugo does not seem to provide a built-in solution for this. So, I am creating a rule for how links should be rendered:

{{- /* /layouts/_markup/render-link.html */ -}}

Solution: Language-Dependent Link Prefixes

Here is what needs to happen: When a relative link within the site is encountered during the build process, a language prefix (such as /en) should be prepended to the link only if the language of the page being rendered is not the default language (since I do not need a language prefix for the default language, given that defaultContentLanguageInSubdir is set to false).

Retrieving the Default Language

But how do I access the default language? The defaultContentLanguage parameter is not accessible within templates. After some research, I found a suitable solution to the problem in the Hugo community: The default language is usually the first language in the weight-based index (which determines the order in which languages appear): index .Page.Site.Home.AllTranslations 0. When I access this, the “first” one is my default language.

Code for render-link.html

{{- /* /layouts/_markup/render-link.html */ -}}

{{- $url := .Destination | safeURL -}}
{{- $currentLang := .Page.Language.Locale -}}

{{- $is_external := or (strings.HasPrefix $url "http://") (strings.HasPrefix $url "https://") -}}
{{- $is_anchor := strings.HasPrefix $url "#" -}}
{{- $is_asset := strings.HasPrefix $url "/assets/" -}}
{{- $is_lang_prefixed := strings.HasPrefix $url (printf "/%s/" $currentLang) -}}

{{- if $is_anchor -}}
 {{- $url = printf "%s%s" .Page.RelPermalink $url -}}
{{- else -}}
 {{- if not $is_external -}}
 {{- if and (not $is_lang_prefixed) (not $is_asset) (not .Page.Language.IsDefault) -}}
 {{- $url = printf "/%s%s" $currentLang $url -}}
 {{- end -}}
 {{- end -}}
{{- end -}}

{{- /* Add / to internal paths that have no file extension */ -}}
{{- if and (not $is_external) (not $is_anchor) (not $is_asset) -}}
 {{- $parsed := urls.Parse $url -}}
 {{- if and $parsed (not (strings.HasSuffix $parsed.Path "/")) (eq (path.Ext $parsed.Path) "") -}}
 {{- $url = printf "%s/" $url -}}
 {{- end -}}
{{- end -}}

<a href="{{ $url | relURL }}"{{ if $is_external }} target="_blank" rel="noopener noreferrer" class="external-link"{{ end }}>
 {{- .Text | safeHTML -}}
 {{- if $is_external -}}
 <span class="external-link-icon" aria-hidden="true">↗</span>
 {{- end -}}
</a>

As long as the default language and the weight index match, meaning the default language appears at the very top, this works perfectly well.

Additional functions

Here is what else the code does:

Checks if links are “external” - that is, leading away from my site. These are assigned a separate window and the ↗ indicator.
Checks if the link points to a heading “anchor.” See below for details.
Checks if the link points to a (media) file or “asset.” Asset links are not modified. If it is a heading link, it is expanded to the full URL if necessary. This follows this pattern: #heading --> /<opt_lang>/<opt_category>/<title>/#heading

Correctly resolving permalinks to headings

I have also found a solution for highly specific permalinks within a page. Let’s assume I have headings in different languages:

---
# German
# content/de/posts/yyyy-mm-dd-hugo-create-language-specific-permalinks.md

description: Mit _markup interne Links richtig routen
title: "Hugo Multi-Lang: Interne Permalinks"
slug: hugo-create-language-specific-permalinks
---

## Richtig auflösende Permalinks auf Überschriften {#permalinks}
Lorem Ipsum...

---
# English
# /content/en/posts/yyyy-mm-dd-hugo-create-language-specific-permalinks.md

description: Routing internal links with _markup
title: "Hugo Multi-Lang: Internal Permalinks"
slug: hugo-create-language-specific-permalinks
---

## Correct Permalinks to headings {#permalinks}
Lorem Ipsum...

Then, clicking this link back to the #permalinks heading should take the user back to the heading, regardless of the language. The key to this solution is introducing a heading definition (Heading-ID) in Markdown: in this case, {#permalinks}.

It doesn’t work quite yet.

This is because #permalinks points to <blog's baseurl>/#permalinks, meaning the link to the current page is missing. But we can fix that, too: I’ll extend the link renderer with a few lines of code to prepend the current page URL.

Code for render-link.html with heading support

{{- $is_anchor := strings.HasPrefix $url "#" -}}

{{- if $is_anchor -}}
{{- $url = printf "%s%s" .Page.RelPermalink $url -}}
{{- end -}}

Now, (#permalinks) expands to /hugo-create-language-specific-permalinks/#permalinks, so the link finally leads to the correct destination.

act_runner rootless: no start

Schallbert — Sat, 28 Mar 2026

The error

My docker-in-docker (DinD) act_runner crashes shortly after starting with the following error message:

Issue ticket on Gitea

I have created a ticket for this issue in the Gitea community (issue #721). Below, I’ll go into my own findings and summarise the discussion a little.

Problem with kernel permissions?

In a Docker-in-Docker configuration, the act_runner must run its own Docker daemon. Only then can the runner create its own containers for the actions. For security reasons, this is not permitted by default.

Why containers are not allowed to start other containers

In the context of act_runner, an action executes code that is itself part of the repository. If malicious code is introduced into the job container unnoticed, e.g. by a ‘collaborator’, it can, in a Docker installation without DinD, under certain circumstances potentially gain direct access to the host system.

Consequences if something goes wrong despite DinD

When using the DinD concept, an attack can result in access to the dockerd process within the act_runner container - and only if the action is not properly encapsulated. Still not ideal, but acceptable: when the container is restarted, the status quo ante is restored. Access to the host system is indirect, as the container itself is equipped with kernel features.

Simple solution: Start the DinD container as `privileged`

To allow the DinD container to set up its own actions, privileged: true can be set in the configuration. This grants the container all kernel capabilities. According to the (excellent) OWASP Security Cheat Sheet, this should be avoided where possible. Should act_runner itself now come under attack or reveal critical security vulnerabilities, the intruder would already have every opportunity to bypass the encapsulation from the host system.

If you want to take the easy route, the corresponding docker-compose.yml file looks as follows.

# docker-compose.yml
# section ACT_RUNNER
runner:
 image: gitea/act_runner:latest-dind-rootless
 container_name: gitea-runner
 privileged: true
# [...]

“And remember: Do not run containers with the –privileged flag!!!” - OWASP’s Docker Security Cheat Sheet, 2026 License

Dead end: Only grant the rights that are absolutely necessary

After a quick search, I find an article on CodeStudy.net¹, which deals with the topic of DinD. From this, I put together some changes to my docker-compose.yml.

My approach:

Make changes to docker-compose.yml
Restart the service docker compose restart <service>
View logs docker logs --tail 100 <container-name>
If started: Run the programme web-app->repo->jobs->rerun_all-jobs
If it fails: Repeat

I start with few, but very powerful, permissions. After a few iterations, I get:

# docker-compose.yml
# section act_runner DinD-rootless
runner:
 image: gitea/act_runner:latest-dind-rootless
 container_name: gitea-runner
 privileged: false
 cap_add:
 - SYS_ADMIN
 security_opt:
 - no-new-privileges:true
 - apparmor:unconfined 
 - systempaths=unconfined
 - writable-cgroups=true
# [...]

But I’m still missing permissions that lie outside kernel privileges and security zones: the Docker daemon needs to be able to access sysfs and proc, i.e. system file directories and processes.

level=warning msg="[rootlesskit:child ] failed to mount sysfs, falling back to read-only mount: operation not permitted"

I cannot resolve this even with cap_add: ALL. Furthermore, online resources on this technical level are really scarce (and I am not a Docker specialist). A re-mount of the file systems might still help here; but that seems too experimental and error-prone to me.

After several more hours of research and trial and error on my server, I have to agree with @tianon in a discussion on GitHub: The Docker daemon requires so many permissions and capabilities that you might as well stick with privileged: true and avoid having to grapple with an armada of CAP_ADD and system bind mounts.

Future solution: Virtualisation or daemonless

However, there may be better solutions: dedicated container runtime environments such as Sysbox. The box itself has no special privileges on the host system; yet, much like in a virtual machine, it appears to be able to provide applications running within it with full access and capabilities.

Covering everything from installation and setup to fully functional job containers would take this post too far afield. Therefore, for the time being I must refer to other sources.

Switching from Docker to Podman is also a possible solution: Podman is daemonless, rootless and offers a similar range of features to Docker. Migrating to Podman would be a project in its own right for me and does not fit within the scope of this post.

Linux Security Modules (seccomp, AppArmor, SELinux)

Definition: LSM enable various security checks and restrictions at the kernel level. Through ‘Mandatory Access Control’, security extensions such as AppArmor can control kernel capabilities for individual applications and block access where necessary.

AppArmor

I run my server (VPS) on Ubuntu. In its more recent versions, this operating system has integrated ‘user namespace creation restrictions’ into AppArmor, which prevents appimages, WebApps and containers from running with elevated privileges.

“Unprivileged user namespaces are a feature in the Linux kernel […]; it enables unprivileged users to gain administrator (root) permissions within a confined environment […]” - mbelair, Ubuntu Discourse, as of May-2026, Ubuntu Discourse Website

In short, this tool was developed as a security-hardening measure to reduce the attack surface on the host system when running programmes that require elevated privileges.

Simple solution: Disable completely

The sledgehammer approach completely disables the feature for user namespaces. We tell AppArmor that third-party programmes may use kernel features or elevated privileges without restrictions, just as in older operating system versions. I found the relevant command on the AskUbuntu forum.

It disables restrictions for this session by overwriting the kernel parameters at runtime (-w).

schallbertTestsThis@machine:~# sudo sysctl -w kernel.apparmor_restrict_unprivileged_userns=0

I find the command very useful for troubleshooting. If you want to find out whether the desired programme is failing to start because of AppArmor:

Enter the command above
Test the third-party programme, the container, etc.
Reboot. Or enter the command with =1. This will restore the original state.

If you wish to retain this vulnerability permanently, enter:

makeVulnerability@machine:~# echo 'kernel.apparmor_restrict_unprivileged_userns = 0' | sudo tee /etc/sysctl.d/20-apparmor-donotrestrict.conf
makeVulnerability@machine:~# sudo shutdown -r now

This in itself does not constitute a security vulnerability. It has merely become easier, in principle, to exploit any weaknesses in the kernel and escape the container’s “sandbox”.

The correct solution: Tailor settings for each application

There are applications (such as my DinD version of act_runner) that absolutely require elevated privileges to function properly. And only these should be granted the ability to access non-admin user namespaces.

Docker itself has dedicated a section in its documentation to AppAmor. To arrive at the solution for the DinD runner, a bit of transfer (thanks, @thespad) is required. To recap, here is the error message from above:

gitea-runner | [rootlesskit:parent] error: failed to start the child: fork/exec /proc/self/exe: operation not permitted
gitea-runner | s6-svwait: fatal: some services reported permanent failure or their supervisor died

The log file indicates that the parent (Docker daemon in the container) cannot start its child (runner container) because it cannot create processes (/proc/self) for other participants (fork). Who is the user of this daemon? rootlesskit. This is exactly where our solution comes in: we need to enable the AppArmor profile for rootlesskit in the act_runner’s DinD container within the docker-compose.yml file.

# docker-compose.yml
# section act_runner DinD-rootless
runner:
 image: gitea/act_runner:latest-dind-rootless
 container_name: gitea-runner
 privileged: true
 security_opt:
 - apparmor=rootlesskit
 # [...]

The log file now looks fine. Done!

I get the feeling that post was written by an ‘AI’. When it gets specific and talks about ‘critical volume mounts’, there are no actual system paths listed, and the article becomes so vague overall that I can’t actually reach my goal by following the advice. ↩︎

Jekyll-dockerimage: Bundler and Gemfile

Schallbert — Fri, 14 Nov 2025

What is bundler?

Bundler is a tool that can be used to manage and version dependencies between modules and libraries for Ruby language. I use Bundler commands regularly, e.g. to build my blog and put it live on the server.

If I want to build locally, I type

bundle exec jekyll serve --incremental --future

into the console. This command tells Bundler to run the jekyll application in server mode and provides it with parameters that prevent jekyll from completely rebuilding every time a file is changed and still create pages that have not yet been published.

bundler in the CI pipeline

I also use Bundler as part of a Jekyll Docker image to publish my site, as I have often linked to, e.g. when moving to self-hosted. Since switching to a new version of the blog software lately, I have been seeing puzzling errors in my CI pipeline. Locally however, the system builds flawlessly. I have documented how to address and fix such errors here.

“There was an error while trying to write to /path/to/Gemfile.lock” - CI console output

No write permissions

Like the Jekyll user, Bundler itself can only read files on my CI and cannot write them. I can fix this for the Jekyll output with chown, but I don’t want to allow bundler to do this: I require Gemfile.lock to remain identical between my local build environment and the CI so that I can fix errors in advance.

Specification file for bundler

The Gemfile.lock file contains all dependencies of the application used, including version numbers and sources. If only the source code of the application plus this file is provided, bundler can pull a specific version of all dependencies during build time, allowing to work with similar requirements regardless of machine and version.

Troubleshooting

A first attempt to fix the issue by aligning all versions between local and CI failed. The second attempt, to give the Jekyll user write permissions to the Gemfile.lock, also failed. The third attempt led me to the Bundler website, where I took a closer look at the parameters, keyword “frozen”.

The fact is that you can prohibit bundler from rewriting the Gemfile.lock. To do this, use the command

bundle config set frozen true

According to its documentation, processing is aborted as soon as the file is about to be rewritten.

Although I had added this command to my yaml file in the CI, the build failed again. The trigger was the same library nokogiri as before, but the error message was now much more helpful:

Optimize `Gemfile.lock` for different environments

So I follow the suggestion and execute the desired command on my local machine:

bundle lock --add-platform x86_64_musl # my CI runner's environment

Note: musl is a libc-implementation for Linux.

My lockfile now has the following entry:

nokogiri (1.18.10-x86_64-linux-musl)

With this error fixed, I get better portability of my web page creation setup as a side effect.

Rootless act_runner for Gitea

Schallbert — Sun, 10 Aug 2025

In this article, I’ll show how to change the act_runner of my Gitea instance from gitea/act_runner to gitea/act_runner:latest-dind-rootless.

Why all this effort? Because act_runner runs in Docker and requires access to the daemon via /var/run/docker.sock as a so-called volume (meaning a disk) or bind mount to function. The owner of this socket is root, which gives the container virtually full access to the host system.

“RULE #1 - Do not expose the Docker daemon socket (even to the containers)” - OWASP / Docker Security Cheat Sheet

This is highly risky, because it’s inherent in the runner’s design to execute job code generated by third parties. It’s an open gateway. In conjunction with direct access to the host system, this could result in a total failure or hostile takeover of my infrastructure in the event of a successful attack.

Danger from `docker.sock`

In this guide, I follow the OWASP’s recommendation on Docker security and apply its rule(s) in practice. OWASP stands for “Open Web Application Security Project.” It is an organization dedicated to improving the security of web applications and supporting users like me with free articles, documentation, and technology.

The measures described under Rule 1 are:

Leave Docker’s tcp socket disabled

If access to the Docker Daemon is enabled via tcp, it can be connected to via an unsecured connection and without authentication - unless further precautions have been taken. The daemon is then accessible to virtually any internet user and thus vulnerable.

So how do I ensure that the tcp socket is disabled? The instructions for this can be found in the Docker Docs and must be applied in reverse. There, too, there is an explicit warning against opening the tcp socket unprotected.

After completing all the necessary steps, I run netstat once. It lists open sockets, network interfaces, and routing tables:

# check if Docker Daemon's "dockerd" tcp socket is exposed
schallbert@machine:~# netstat -lntp | grep dockerd
# test is a pass if this command does not return anything.

I don’t find any entry with dockerd. My server isn’t vulnerable at this point.

Do not include the Docker socket `/var/run/docker.sock` in other containers

This is where things get a bit more complicated: act_runner needs the socket to create, manage, and ultimately dispose of job containers. Without access to the Docker Daemon via the socket, the build pipeline simply won’t work - unless you want to forgo Docker entirely and run both runner and the build jobs directly on the host machine. This comes with many disadvantages: loss of encapsulation, lack of portability, poor scalability, reduced security…

But there’s also a solution for the Docker option: Setting up act_runner Docker-in-Docker. With this setup, act_runner receives its own Docker Daemon, but with limited permissions and without access to the host system. This then takes over the lifecycle of job containers so that they run completely independently of the host system.

Docker in Docker

The following diagram illustrates the difference:

Starting Act_runner “rootless”

So I follow the instructions and copy together a suitable docker-compose.yml. Important here:

privileged: true must be set. Otherwise, the Docker daemon in the act_runner container cannot start properly because it lacks kernel functions. This causes the entire container to crash repeatedly without generating any helpful error messages.
The environment variable DOCKER_HOST=unix:///var/run/user/1000/docker.sock must be set. Here, the Docker socket is controlled by a non-privileged user and is available to the runner for managing job containers. The daemon runs encapsulated in the container and is not connected to the host machine.

As of Jul-2025: Problem with 0.2.12-dind-rootless

This is where I encountered my first problem. The runner crashes shortly after starting with the following error message:

schallbert@machine:~# docker logs gitea-runner
# [...]
[rootlesskit:parent] error: failed to start the child: fork/exec /proc/self/exe: operation not permitted

Help came from the Gitea community. act_runner runs smoothly with the previous version 0.2.11. However, for reasons unknown to me, it is displayed on Gitea as 0.2.12.

Volume Confusion

Great! Now that the runner is working, I’ll let it run a job right away. Unfortunately, the build fails after just a fraction of a second with this message:

# Runner step: Set up job
failed to start container: Error response from daemon: error while creating mount source path '<volumeSourceFullPath>': mkdir <volumeSourcePath>: permission denied

I had to research this for hours and was under the false impression for a long time that it was due to insufficient permissions on the folders on the host machine. Only later did I truly understand that Docker-in-Docker means exactly what it says: Not only are containers created by containers, but a separate Docker daemon runs within the container!

This means that the classic method of making volumes in job containers available directly from the host system using -v /a/b:/x/y no longer works. Instead, volumes must now be passed through. Example:

host directory /opt/server/www/blog-artifacts -> act_runner volume /tmp/blog-artifacts –> job container volume /tmp/blog-artifacts

The :z in the act_runner volume is now important. It indicates to Docker that this volume is shared between containers. These volumes must not only be specified in act_runner’s docker-compose.yml, but the job scripts in the .gitea/workflows/ folder must also be adjusted accordingly. However, the :z on the “right side” is not needed here.

“is not a valid volume”

Despite all my efforts, my jobs still aren’t running. This time, because of an error message that already seemed familiar:

# Runner step: Set up job
[/tmp/blog-artifacts] is not a valid volume, will be ignored

So, go into the config.yml of act_runner and add the volume names:

# /gitea/runner/config.yml
# [...]
valid_volumes: ["/tmp/blog-artifacts", "/tmp/lectures-artifacts"]
# [...]

In this file, I can leave privileged: false because, unlike act_runner, the job container doesn’t require kernel features.

Set permissions correctly

Now I’m getting Permission Denied error messages again when running my jobs, although not directly in the first step of the actions. Since I’ve now checked the volume paths down to the last detail, it can only be due to the folder permissions on the host machine.

In order for the artifacts created by the job container to be stored via the volumes on my host, I have to pass the directory to be written and all subfolders -R to the previously defined, non-privileged user ID=1000:

schallbert@machine:~# chown -R 1000:1000 /target/path/to/artifact/

Finally, everything is working smoothly, and I’ve put a stop to the (unlikely, but possible) takeover of my host system by malicious job containers.

Add rate limiter to Gitea

Schallbert — Fri, 25 Jul 2025

Here, I’m providing a step-by-step guide to solving my Gitea crashes problem. I’m incorporating the experience I gained from using a rate limiter for my blog with the goal of making the application more robust and protected against undirected denial of service attacks.

Step 1: Specify the log source for the rate limiter

Gitea naturally generates logs itself, theoretically down to the access level by external clients. However, in the past, I haven’t been able to export Gitea’s access logs from Docker. Therefore, we now instruct Caddyserver, which is connected as a reverse proxy, to create logs on behalf of Gitea. Access to the Gitea web interface will then appear in these logs.

In the Caddyfile, the log module looks completely unspectacular.

# /caddy/Caddyfile

#[...]
log {
output file /log/gitea/access.log
}

Exclude internal traffic from the log

Now I look at the logs and see entries that I don’t want. For example, act_runner generates a Fetch Task POST to Gitea every two seconds and a GET request every ten seconds with the same target as a Health check. These don’t even need to appear in the log for me. My first thought here was to simply not log requests from internal IP addresses. However, since Caddy acts as a reverse proxy, all IP addresses run “internally” without exception.

So I have to recognize act_runner-specific logs differently:

# /caddy/Caddyfile

#[...]
git.schallbert.de {

 reverse_proxy * http://gitea:3000

 # Enable logging for fail2ban, don't log for runner
 log_skip /api/actions*
 log {

 output file /log/gitea/access.log
 }
}

The log_skip directive now instructs caddy to no longer create logs for access to /api/actions*.

Reverse Proxy: Displaying Remote IPs

However, I still have a problem: If all requests in the log come from an internal IP address, how am I supposed to block “bad” requests? A web search shows that this is a common problem for reverse proxies in Docker. Unfortunately, many solutions cannot be applied to this situation because they use different server software like nginx or have different services running behind their proxies than Gitea.

But it’s actually quite simple: Just insert a line in the correct place in the Caddyfile, and the remote IP addresses come in unchanged.

# /caddy/Caddyfile

#[...]
git.schallbert.de {
 reverse_proxy * http://gitea:3000 {
 trusted_proxies 172.16.0.0/12 # Docker-internal netwock traffic runs with these IPs
 }
#[...]

The trusted_proxies directive tells Caddy that the network address range provided by Docker can be trusted. Now the actual source IP address is shown instead of the container’s internal interface address. This is exactly what I need to be able to analyze the addresses later with fail2ban.

Step 2: Determine the Use Case

Let’s look at how many HTTP 200 ok requests are coming in the edge case between normal use and “abuse.” To do this, I surf around Gitea and click on a bunch of things that I would never normally do at that speed as a human. Then I analyze the logs.

This gives us some initial benchmarks for the rate limiter.

# rate limiter tests
findtime = 10s
maxretry = 10
bantime = 6h

Step 3: Configure Fail2ban

Next, we configure the filter and jail file of fail2ban to analyze the logs defined above.

Filter

Here, I’m re-using the same filter for my rate limiter from the previous article. We’re only interested in successful requests that we count within a time window.

Jail

I’m using the values from the rate limiter tests 1:1 in the jail file. I’m referring to caddy-ratelimit (link above) as the filter. It is essential to select the DOCKER-USER chain as the requests are routed via Docker’s virtual network.

# /fail2ban/config/fail2ban/jail.local
# [...]

[gitea-ratelimit]
enabled = true
chain = DOCKER-USER
port = http,https
filter = caddy-ratelimit
logpath = /var/log/caddy2/gitea/access.log
findtime = 10s
maxretry = 10
bantime = 6h

I reboot fail2ban to activate the protection.

Step 4: Test the rate limiter

I proceed exactly as in my article fail2ban with caddy and get the same result. It works!

Rate limiter with Caddy and fail2ban

Schallbert — Thu, 10 Jul 2025

Here I describe how to configure fail2ban (commonly used to defend against unauthorized access attempts) to limit the number of successful accesses within a defined time window. This is called “rate limiting” and is intended to thwart denial of service attacks that overload my server.

Enabling Caddy Logs

First, we need to enable the web server’s logger. With Caddy, this can be done with just a few lines of code that I add to my existing configuration:

# caddy2/config/Caddyfile

blog.schallbert.de {
# Define webserver
root * /www/blog
encode gzip
file_server

# Enable logging for fail2ban
log {
output file /log/blog/access.log
}

# [...]

From now on, logs will be created in the specified path, which I manage with a logrotate configuration.

Configuring Fail2ban as a Rate Limiter

To recap: fail2ban accesses the machine’s packet filter rules and, in effect, modifies its firewall to ward off dynamic attacks. For this to work, fail2ban must be provided with log files generated by a host application, such as a web server, that contain the IP addresses of the client computers.

Preliminary consideration: “http 200 OK” as a filter?

Even normal, permitted accesses 200 OK must count for my rate limiter.

To get a feel for how my logger records accesses, I take a look at the logs. I get the number of entries that contain a status of 200 OK returned. To do this, I perform a search with grep and calculate the number of hits line by line using wc -l.

schallbert@machine:/var/log/caddy2/blog# grep -o '"status":200' access.log | wc -l
847

Okay, so there have already been 847 hits today, which received a 200 OK response. Now I’ll call up my blog’s landing page and display the article overview.

schallbert@machine:/var/log/caddy2/blog# grep -o '"status":200' access.log | wc -l
858 // after calling blog.schallbert.de
schallbert@machine:/var/log/caddy2/blog# grep -o '"status":200' access.log | wc -l
882 // page loaded after clicking "Posts" button

Wow, that was 35 entries for two clicks! Looking at the log, it explains the many lines of assets—images and logos—that are being loaded.

Unfortunately, this makes it clear that the filter criterion status:200 for my rate limiter can’t work without further ado. The number of notifications depends largely on the respective article. So, I would have to define a threshold above which no normal person would generate access to my blog.

In order to count only “real” accesses to my pages, I have to somehow exclude the assets from the logs. Fortunately, there’s a simple directive in Caddy for this.

# caddy2/config/Caddyfile

 # Enable logging for fail2ban
 log_skip /assets*
 log {
 output file /log/blog/access.log
 }

log_skip now ensures that all accesses to files in the assets/ folder and below are not logged.

The behavior of Fail2ban is defined using two configuration files:

Directory `filter.d`

filter.d/<filter-name>.conf contains the definition of an event to be monitored. Thus, the failregex “page not found” http 404 status code can be used to trigger the packet filter, just as successful access can be used to set up my rate limiter, which responds to 200 OK. I copied the regex from Rafael Kassner.

# /opt/fail2ban/config/filter.d/caddy-ratelimit.conf

[Definition]
failregex = "client_ip":"<HOST>"(.*)"status":200
datepattern = \d+
ignoreregex =

In addition, further boundary conditions are defined in the file, e.g., the date format datepattern is adapted to the log output of the system to be protected.

`jail.local` file

jail.local determines the conditions under which the packet filter for the client IP becomes active and blocks further access attempts. For use as a rate limiter, I need the following fields:

findtime - the time window in which attacks are counted, maxretry is the number of permissible attempts in the time window, and bantime is the time for which the attack is blocked. ignoreip is usually preconfigured to the relevant internal IP addresses by default. In my case, act_runner, for example, sends a message to gitea internally to query whether any new automation tasks are pending. I definitely don’t want to interfere with these accesses.

The example for fail2ban used below shows the rate limiter on my blog. In the end, the files and log entries for gitea are essentially the same. Only the filter names and parameters differ due to the different requirements of the website and the DevOps platform.

# /opt/fail2ban/jail.local
# ...

[caddy-ratelimit]
enabled = true
chain = DOCKER-USER
action = iptables-multiport
port = http,https
filter = caddy-ratelimit
logpath = /var/log/caddy2/blog/access.log
findtime = 20s
maxretry = 10
bantime = 6h

Once these two files are configured, you can restart fail2ban and view the logs. If there are still errors in the files, fail2ban will output the following:

ERROR Errors in jail 'caddy-ratelimit'. Skipping...

If this occurs, for example, the jail cannot be assigned to the filter (must have the same name!), a field definition such as maxretry is incorrectly typed, or a number format cannot be read. If everything is working correctly, the entry will be:

Reading config files: /etc/fail2ban/filter.d/caddy-ratelimit.conf

Testing the Rate Limiter

Now I want to check if everything is working as expected.

Checking the Filter Regex

The first step is to look at the filter term and check whether it can be reliably found in the logs. Conveniently, fail2ban has a built-in tool for this: fail2ban-regex <logfile> <filter>. So I can simply enter the filter file and a test log file and see if I get any matches.

fail2ban-regex ./access.log /etc/fail2ban/filter.d/caddy-ratelimit.conf 

Running tests
=============

Use filter file : caddy-ratelimit, basedir: /etc/fail2ban
Use datepattern : \d+ : \d+
Use log file : ./access.log.2
Use encoding : UTF-8

Results
=======

Failregex: 2009 total
|- #) [# of hits] regular expression
| 1) [2009] "client_ip":"<HOST>"(.*)"status":200
`-

Ignoreregex: 0 total

Date template hits:
|- [# of hits] date format
| [2685] \d+
`-

Lines: 2685 lines, 0 ignored, 2009 matched, 676 missed
[processed in 0.12 sec]

Quantitatively, I get what I expect: the search procedure described above using grep gives me exactly the same results.

Check Jail

The next step is to test whether the rate limiter is working. To do this, I simulate log entries. I significantly ease jail rules for this test, otherwise I’ll have to generate too many entries in a short period of time.

# /opt/fail2ban/jail.local TEST
# ...

[...]
findtime = 10s
maxretry = 2
bantime = 100s

After restarting fail2ban, I create a second console where I can simulate log entries. This must include at least the time, IP address of the caller, and status; however, I play it safe and use complete entries.

echo '{"ts":1751246139,"remote_ip":"1.1.1.1","status":200,[superLongIrrelevantOtherStuffForFiltering]}' >> /var/log/caddy/blog/access.log

I enter this multiple times to exceed my limit of 2 retries.

In the first console instance, I then check the fail2ban status for the corresponding filter.

fail2ban-client status caddy-ratelimit
Status for the jail: caddy-ratelimit
|- Filter
| |- Currently failed: 0
| |- Total failed: 0
| `- File list: /var/log/caddy2/blog/access.log
`- Actions
|- Currently banned: 0
|- Total banned: 0
`- Banned IP list:

That was nothing. The fail2ban log file reveals why:

# /fail2ban/config/log/fail2ban/fail2ban.log
WARN [caddy-ratelimit] Ignoring all log entries older than 20s; # probably messages generated within a fail2ban restart period

So again with appropriate timestamps. I have to get these from the current system time ($(date +%s.%N)) if I have to constantly adjust it. Therefore, I take a complete log line, modify it to the IP address 1.1.1.1, and insert appropriate timestamps:

 echo "{\"level\":\"info\",\"ts\":$(date +%s.%N),\"logger\":\"http.log.access.log0\",\"msg\":\"handled request\",\"request\":{\"remote_ip\":\"1.1.1.1\",\"remote_port\":\"38229\",\"client_ip\":\"1.1.1.1\",\"proto\":\"HTTP/1.1\",\"method\":\"GET\",\"host\":\"blog.schallbert.de\",\"uri\":\"/vacuum-clamping/\",\"headers\":{\"Accept-Encoding\":[\"gzip, deflate, br\"],\"Connection\":[\"keep-alive\"],\"User-Agent\":[\"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36 Edg/121.0.0.0\"],\"Accept\":[\"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8\"],\"Accept-Language\":[\"en-US,en;q=0.5\"]},\"tls\":{\"resumed\":false,\"version\":772,\"cipher_suite\":4865,\"proto\":\"http/1.1\",\"server_name\":\"blog.schallbert.de\"}},\"bytes_read\":0,\"user_id\":\"\",\"duration\":0.001207354,\"size\":12290,\"status\":200,\"resp_headers\":{\"Last-Modified\":[\"Tue, 10 Jun 2025 19:33:01 GMT\"],\"Content-Encoding\":[\"gzip\"],\"Server\":[\"Caddy\"],\"Alt-Svc\":[\"h3=\\\":443\\\"; ma=2592000\"],\"Vary\":[\"Accept-Encoding\"],\"Etag\":[\"\\\"gzip\\\"\"],\"Content-Type\":[\"text/html; charset=utf-8\"]}}" >> access.log

After calling this command several times, I now get

Status for the jail: caddy-blog-ratelimit
|- Filter
| |- Currently failed:	1
| |- Total failed:	11
| `- File list:	/var/log/caddy2/blog/access.log
`- Actions
 |- Currently banned:	1
 |- Total banned:	4
 `- Banned IP list:	1.1.1.1

That looks great.

Check packet filter

Phew, finally done! This is how fail2ban.log looks like right now. It shows [caddy-ratelimit] entries as expected:

schallbert@machine:~# nano /fail2ban/config/log/fail2ban/fail2ban.log

 <timestamp> <id> INFO [caddy-ratelimit] Found <IP a>
 <timestamp> <id> INFO [caddy-status] Found <IP b>
 <timestamp> <id> INFO [caddy-ratelimit] Ignore 172.18.0.1 by ip
 <timestamp> <id> INFO [caddy-status] Ignore 172.18.0.1 by ip
 <timestamp> <id> INFO [caddy-ratelimit] Found <IP c>
 <timestamp> <id> INFO [caddy-ratelimit] Found <IP c>
 <timestamp> <id> INFO [caddy-status] Found <IP c>
 <timestamp> <id> INFO [caddy-status] Found <IP c>
 <timestamp> <id> INFO [caddy-ratelimit] Found <IP d>
 <timestamp> <id> INFO [caddy-status] Found <IP d>
 <timestamp> <id> INFO [caddy-status] Found <IP e>
 <timestamp> <id> INFO [caddy-ratelimit] Found <IP e>
 <timestamp> <id> INFO [caddy-ratelimit] Found <IP f>

To test the entire chain, we now need to check whether the corresponding IP address is actually blocked in the hardware. To do this, I enter:

schallbert@machine:~# iptables -n -L | grep "1.1.1.1"
REJECT all -- 1.1.1.1 0.0.0.0/0 reject-with icmp-port-unreachable

Everything seems to be fine. For one final test from an external IP address, I call a Link Checker in the browser, which should crawl my website and thus trigger the rate limiter. Although it crawls 44 links in a very short time, the test is displayed as “passed.” Very strange, [caddy-ratelimit] should have triggered!

iptable-chain INPUT instead of FORWARD

Confused, I take a look at the entire iptable. At the same time, I look up the IP address of the link checker. It appears in the iptables. And yet, it’s apparently not blocked. Why is that? A closer look shows:

Chain INPUT (policy ACCEPT)
target prot opt source destination
f2b-caddy-ratelimit tcp -- 0.0.0.0/0 0.0.0.0/0 multiport dports 22

The problem here is that my jail is on the INPUT chain. However, the requests don’t go directly to my server hardware, but are forwarded via Docker to Caddyserver. To work, I have to land on the FORWARD chain, where DOCKER-USER already is. Very strange, since I had specifically specified chain = DOCKER-USER in jail.local. Something must be overriding this definition.

An inconspicuous forum post together with the multiport comment in the iptables printout led me to the solution: The action = iptables-multiport directive overrides my chain = DOCKER-USER statement, because the following is set in the corresponding configuration file iptables.conf: chain = INPUT.

So I simply delete the action entry, so that fail2ban reverts to the default for DOCKER-USER: multiport dports 80,443.

Now the jail name also appears correctly in iptables.

schallbert@machine:~# iptables -n -L
Chain DOCKER-USER (1 references)
target prot opt source destination
f2b-caddy-ratelimit tcp -- 0.0.0.0/0 0.0.0.0/0 multiport dports 80,443

Finally, another link check shows that my rate limiter is working.

Gitea crashes: Too many requests

Schallbert — Mon, 30 Jun 2025

Here I’d like to briefly describe how a service crash repeatedly paralyzed my server for hours. So badly that I could only restart it from the provider’s console. I’ll explain how it happened and how I intend to avoid this and similar problems in the future.

I was attacked. Or was I?

I was working on an article that I wanted to post later. To be on the safe side in terms of backup, I created a commit as usual and wanted to push it to my Gitea instance. But my git push command simply didn’t work.

Confused, I tried to access my website. The response was:

Strange. Then I wanted to log in to my server to check everything was OK: ssh <servername>. Again, the terminal remained unresponsive. Bummer!

Timeout. As a last resort, I logged in to my hoster’s web interface and looked at the server’s graphs:

Oh, what’s going on? I’m trying to shut down the server via the web interface. That doesn’t work either. Only a hard reboot succeeds. I can log in again via SSH and see that all Docker containers are booting up normally.

What happened?

It’s a good thing I keep all the logs for a week. This, along with the machine’s utilization over time, allows me to reconstruct what happened. At least to some extent.

Which logs help?

Localize the crash! Find the triggering application in the kern.log and note the timestamp.
Are there system-wide effects or other services being affected? Check the syslog.
If you suspect the system may have been hacked, auth.log has the details.
If the affected application is running in a container, the relevant logs there may be helpful.
Review the application’s logs. The time just before the crash is particularly interesting.

And here again, in detail, are the logs I reviewed for my behavior and where they can be found.

log name	purpose	relevant content for this issue
/var/log/syslog	System-wide (bare metal) messages	(docker) warning: “health check for container `<ID>` timeout” (containerd) error: “ttrpc: received message on inactive stream”
/var/log/auth.log	Contains authentication messages from external hosts, in my case mostly SSH	None
/var/log/kern.log	Logs all app, service, daemon and system crashes	Out of memory: Killed process `<ID>` (`<serviceName>`)
[.]/gitea/log/gitea.log	Protocol for requests, actions on Gitea’s web interface, repo changes etc.	`GET` requests, crash/restart indications
docker container logs `<containerID>`	Docker’s logs for the container in question	Received signal 15; terminating. (SIGTERM)

kern.log/syslog

Here, next to the stack trace, I can see exactly what happened. The most understandable message is the following:

Out of memory: Killed process `<ID>` (gitea)

So, Gitea was unplugged because it had consumed practically all system resources. If I scroll up in the log, I see a few minutes earlier:

<timestamp> <machineName> dockerd[716]: level=warning msg="Health check for container <ID>"
<timestamp> <machineName> dockerd[716]: level=error msg="stream copy error: reading from a closed fifo"

Already here, I see warnings that the gitea container isn’t working properly. Another advantage of including “health checks” in the docker-compose.yml file. For other reasons, I’ve already written an article about the purpose and implementation of health checks in docker.

gitea.log

The following log entry indicates that Gitea has just been restarted due to a bug in a submodule:

cmd/web.go:205:serveInstalled() [W] Table system_setting Column version db default is , struct default is 1

And further up in the log:

Very interesting. The highlight.css is in my public repo. It’s the scheme for lectures.schallbert.de’s appearance. This file is quite large, almost 1 MiB. And it’s loaded here dozens of times, practically for every commit.

Now I’m looking at other crashes in the past. It’s always bursts of GET commands for large files or requests for compares between two branches of the repository that precede my server crashing.

Who’s behind this?

All crash-triggering requests come from the same IP address range. The crashes started a few weeks ago. But mostly at times of day that I(and apparently many of my readers) didn’t notice. And after a few minutes, the server was always back to normal operation.

Oh, the trail leads to Google’s Cloud Platform (GCP).

But I had blocked robots?

Indeed, I had stopped the search engine indexing for Gitea. Strange. Or am I not dealing with one of the Google spiders or “AI” scrapers, but with a nasty hacker who rented a virtual machine from “Google Cloud”?

Log research: How frequently is the file requested?

A spider would only crawl all my pages once every few weeks, right? And hopefully not ignore my robots.txt. A crawler certainly wouldn’t make the same request multiple times and at short intervals, would it?

To check this, I search the Gitea logs for entries of GET requests to one of the large and therefore resource-intensive files to transfer:

gunzip gitea.log.<date.rotateID>
grep "<filename>" gitea.log.<date.rotateID>

In the list, I see that the same request to the same target file is made multiple times from the same IP address, all within seconds. Shortly before the crash, my server took almost 4 seconds to serve the request.

I also see that the IP address changes every few hours.

Countering a DoS Attack

In summary, I have to conclude that I’m being attacked via a Denial-of-service attack from the GCP address space.

To gather a bit more background information, I visit a few websites on the topic. There, I learn that gitea on my server is crashing under an Application Layer Attack. Now that I know what’s happening and what the problem is, finding solutions is much easier 😅

Of course, I don’t want to give up without a fight by permanently taking my Gitea instance offline. So, what options do I have?

Provide more resources

Admittedly, my machine only has 40GB of memory and 2GB of VRAM, as well as a measly 2-core CPU from 2009. I could book a more powerful server to better handle peak loads. But this wouldn’t prevent the attack, only mitigate its effects.

Rate limiting directly in the web server

Rate limiters limit the number of requests a client can make within a certain time window. This conserves server resources. This is a “soft” defense against DOS attacks, as triggering IP addresses are briefly and gently blocked with an error message. Typically, HTTP status code 429 “Too Many Requests” is returned when the limiter intervenes.

Gitea doesn’t have a rate limiter. In my setup, Gitea runs behind a reverse proxy provided by my Caddyserver. So, that’s where I’d have to start. For Caddy, rate limiters are only available as external modules, which must be manually installed and configured in xcaddy.

Blocking with IP-table rules

Here, you could again use fail2ban and simply block multiple requests for the same resource from an IP address. Gitea has a description of the setup in its documentation. In my case, I would have to continue where I left off, and thus monitor not only SSH but also normal page requests.

This solution sounds the most sensible to me, as it clearly separates concerns into different applications. I only use applications I already have available: Caddy would provide the access logs, and fail2ban would have to read them and set filters in the jail.local configuration so that it acts like a rate limiter.

Let me link the follow-up article Setting up Fail2ban with Caddy here 🙂

My own webshop - Part3

Schallbert — Sun, 15 Jun 2025

Selecting a solution based on key questions

Which solution is straightforward and quick to set up?
Which solution is easier to maintain?
With which system can I, as the operator, most easily understand how everything works (transparency, dependencies)?
Which system has the smallest attack surface?

There’s still quite a bit missing here!

Over the last few weeks, my enthusiasm for setting up my own online shop has waned a bit, as I wanted to focus first on securing my server and finishing off some old, nearly completed projects. I’ll return to this series as soon as that ‘I-want-to-do-something-new’ fever takes hold of me again.

Hang in there! Schallbert

Creating an own Webshop - Part 2"

Schallbert — Thu, 15 May 2025

A few options for webshop software

Let’s assume only software that at least partially meets my requirements get shortlisted. Below, I would like to present a few possible ecommerce solutions.

Shopify

Shopify is an all-inclusive package solution for online retail. Shopify Inc. is headquartered in Canada and handles its European business from Ireland. The Legal sections found on the website, as well as the complexity of the information presented on the topic of Data Protection, lead me to suspect that Shopify is a multinational corporation - and not just a provider of online shops.

But back to the topic.

The domain and hosting are included in the price with Shopify, and you can create your own shop rapidly and without any programming knowledge in a browser window by selecting a theme, and setting a few parameters. Shopify has extensive documentation and offers training and all sorts of other services for its subscribers.

But where’s the fun in that? After all, I run a tech blog and am not a businessman, literally. A bit of programming is definitely something I enjoy. Furthermore, as an open source fan, I don’t particularly like the corporate stance (just look at their modern website, equipped with auto-playing videos and optimized for marketing purposes). The configurability is mediocre at best, and, as with most other solutions, I have to do the optimization for search engine results myself. Regarding data protection and security, they maintain the usual compliance and state-of-the-art communication; so I can’t look behind it and have to assume that they protect their customers well, also in their own interest.

Shopify describes itself as a closed-source SaaS platform.

Therefore, I’m removing this platform from my list of preferred web shop solutions because I don’t want to become unnecessarily dependent on the well-being of third parties. With Shopify, it’s not me who has control over subscription costs or “my” share of the cloud, but Shopify. It’s not me who can determine how many transactions are included in the “plan,” but Shopify. It’s not my customers who remain in control of their data, but Shopify or a third-party company they have commissioned to process it.

Wordpress / WooCommerce

I only know Wordpress as a juggernaut. The jack of all trades, capable of everything you could possibly want thanks to an overwhelming mass of plugins, extensions, and features. And much more you don’t need.

But it’s open source, free, and, according to its own statements, used by many, many shop operators. Let’s take a look at the most well-known e-commerce extension for Wordpress: WooCommerce.

I immediately found countless guides on setting up a WooCommerce store (for example, from GeeksForGeeks or Themeisle), so I’ll refrain from providing my own guide.

After skimming through a few tutorials, I’ll summarize:

A WooCommerce store is easy to set up and configure using a wizard.
There are many themes available, and easy-to-integrate store designs offer a suitable look for every taste.
It helps to already have a WordPress site and some basic knowledge of domains, DNS, and hosting.
Many payment service providers and some shipping providers are preconfigured.
There are numerous YouTube tutorial videos on the topic, and detailed documentation is also available.
Countless plugins allow you to optimize your store for even very specialized areas and purposes.
However, this variety of options can be offset by the fact that choosing the “best solution” can be difficult, and optimizing the site for speed and search engines can become complex.

All in all, WooCommerce strikes me as a balanced mix of “I want to quickly put together something useful” and “I want to be able to configure and modify everything.”

Snipcart

I would consider Snipcart a “shopping plugin” for any website. Even static websites that can essentially only display a product catalog become a fully-fledged eCommerce solution with Snipcart. The core element here is the shopping cart, which is used to process the purchase.

The interface is the website’s HTML markup. Snipcart is integrated as a JavaScript blob and referenced via HTML. The design and appearance are handled via cascading style sheets. In my case, I could use the existing “tech stack” consisting of a static Jekyll site with a CD pipeline in Gitea on a Caddyserver instance for the shop as well, and with minimal adjustments, conjure a seamless look for the blog and shop. Sounds great, right?

Snipcart is developed by a Canadian company in Quebec. It describes itself as a “Shopping Cart Platform”. An account is required for installation, and you must enter your shopping domain in the Snipcart web frontend, similar to Google Search Console or Google Analytics, to receive a so-called API key. Only with this key will the JavaScript blob integrate correctly.

And this brings us to the first major problem: Snipcart restricts my freedom.

<link rel="preconnect" href="<https://app.snipcart.com>" />
<link rel="preconnect" href="<https://cdn.snipcart.com>" />

This shows me that Snipcart would like to keep the data on its server. I have to use their content delivery network to handle the order process and integrate it with mine. So, it’s by no means the case that all connections remain with me and I “buy” the JS code from Snipcart and run it in the customer’s browser or a database solution behind it on my server.

So, if full autonomy and control over your own site isn’t a priority, Snipcart can be a very elegant e-commerce solution. This is especially worthwhile for existing sites. However, for me, it’s out of the question.

OpenCart

OpenCart offers a PHP-based shop module free of charge and open source. It can be booked as “Software as a Service” through third-party providers or downloaded via Github and deployed on your own server.

Installation and administration are handled via web interface and, in my opinion, are straightforward. You can also find a few images on DockerHub that you can get started with right away.

OpenCart is fully customizable to your needs. Templated themes can also be integrated. Out of curiosity, I visited a few websites that use OpenCart: In my opinion, the shop system is well-structured and easy to use. However, the loading speed of the sample pages isn’t always super fast.

Overall, I consider OpenCart, with its customizable, lightweight shop solution in PHP, to be a good solution - also thanks to its still-active community (13+ years!).

Django

Django is a web framework for Python. It offers many features, allows extensions by simply including Python modules, and allows even beginners to quickly create working solutions. Although there are numerous libraries for websites, shops such as django-SHOP, database integrations, integration of payment systems, etc., everything still has to be put together and configured by the developer.

While this gives you full control and a solution that’s perfectly tailored to your needs, it also has a significant drawback: learning time. Database security must be ensured by the developer, and user input validation must be programmed or at least correctly configured. Customer master data is stored in plain text by default. Access and deletion concepts must be created and implemented by the developer, or time should be allowed to identify and integrate suitable modules.

If frameworks are to be used, it is recommended to check for active community maintenance beforehand. For example, it looks to me as if there hasn’t been much activity on the Github repo for django-SHOP since 2021. A quick search, however, reveals a lot of alternatives - now it’s time to consider something suitable or reinvent the wheel yourself 😃

The alternative for beginners like me: no shop system

What is meant by “no shop system”? A form -based approach. Here the user looks at a static website that contains catalog and product pages. In addition, a few elements are shown: E.g. a drop-down menu for variant and a field for the number of pieces. Next to it a simple button. If this is clicked, a form opens that automatically transmits the clicked content and asks about the user’s contact details.

If everything is filled in properly, the form creates a PGP-encrypted email to me when sending. I can then view the order analogously to manually created order emails, send an order confirmation and request payment.

With a few products, a shop equivalent can be implemented supporting a few customers without much effort. Of course not as comfortable as with an eCommerce solution. But maybe that’s enough for the beginning.

In the next article

Selection and Implementation

Creating an own Webshop - Part 1

Schallbert — Tue, 15 Apr 2025

This is part one of a series of articles about building my own webshop. I’m listing my requirements and wishes for my e-commerce solution. This way I narrow it down to a few suitable software solutions that I could later use for my webshop.

What my webshop needs to be able to do

For me as owner, the webshop should operate as lean, transparent, and automated as possible. I want to spend little time on operation, maintenance, and support. Payment processing, order status management, customer registration, sending customer information, and a large number of steps in the shipping process, such as postage and entering shipping confirmations, should be automated or at least prepared for automation.

But the webshop must also be structured in a way that is understandable for customers: simple and with just a few clicks from the homepage to checkout, easy to use, similar to other common webshops, with straightforward payment processing. As little data as possible should be collected from the customer.

My webshop only needs to display a few products. Therefore, navigation can be kept very simple. Due to the small number of product categories, a tree structure may be completely avoidable.

Let’s take a detailed look at what features the web shop ultimately needs to provide.

Functions for Admins

Create a product

Title
Tags
Short description
Description
Images
Videos
Price
Profile / Data sheet / Instructions

Edit a product
Remove a product
Manage orders

Confirm a step
Restart a step
Cancel a process

Manage customers

Create (sandbox)
Delete (data protection)

Functions for Buyers

Browse the webshop
Configure product(s) or select product variants (e.g., color, quality)
Add product(s) to a shopping cart
Manage shopping cart
Register (optional, only for customer data storage and order management)
Select shipping
Make a payment

Automatic functions

Payment registration
Invoice creation and dispatch
Inventory management
Shipping notification

Non-functional Requirements

Here I list everything the web shop should be able to do “on the side.” Things that are not directly required for order processing or visible to customers.

Usability on the Operator Side

The shop should have as few dependencies on third-party software as possible and only integrate those modules that are absolutely necessary. Examples:

Payment processing
Shopping cart
Customer account / address data
Inventory
Order processing (mailer) for buyers
Invoicing.

The shop should only contain as many dynamically generated elements as absolutely necessary.

Usability on the Buyer Side

The web shop should feel like current, common e-commerce solutions: nimble, clear, simple.
No obligation to have a customer account.
The shop should support common payment methods (debit, credit, PayPal, Apple/Google Pay, etc.). Only local (German-based) payment service providers are acceptable.
Inventory levels and production times should be displayed transparently for the user.

Data Protection

The shop should only collect data necessary for business transactions.
If technically and legally possible, the shop should be able to operate without a cookie banner.
Customer accounts should be automatically deleted after two years of non-use (with two weeks’ notice).

Look & Feel

Here are some examples of successful, simple shop designs:

Requirements derived from this:

The web shop’s structure and color scheme should match my Blog page and provide a consistent look.
Only static fonts should be used without reloading.
Product descriptions should be able to be uploaded as Markdown documents (or similar).

Database or file solution?

It doesn’t always have to be a database. For many applications, only a fraction of the functionality of common database solutions is required, but this significantly increases the complexity of the overall system. For example, if concurrent writing rarely or never occurs and there is no strong data concatenation, a file solution can be simpler, faster, and more efficient Linking a related “Engineering Kiosk” podcast episode.

On the other hand, many shop systems come with batteries included, meaning they already have built-in data management that is almost transparent to the programmers.

File solution (static website)

From the operator’s perspective, the web shop should feel like a Git project (everything is a file).
Changes to the offer, item descriptions, prices, etc. should be able to be pushed to the server via git push - just like with a static website.
If a pull request is merged into ‘main’, the web shop should be rebuilt (CD pipeline) and the changes published to the server.
Dynamic elements are required for inventory management, shopping carts, customer interaction, and automated functions. Here, it is important to decide which elements are best handled directly in JavaScript and which are better handled via an automation pipeline.

Database solution (dynamic website)

Products are stored in the database.
The web shop is managed via a web interface.
Database management should be partially automated, with backups and encryption for customer data provided.
The database is transparent for the admin: There is no direct interaction; everything is channeled through e.g. a web interface.

In the next article

Comparing e-commerce solutions for my webshop

Gitea: stop search indexer

Schallbert — Wed, 12 Mar 2025

After ages, I took a trip to the Google Search Console just for fun. Normally, I’m not interested in it because I don’t use Google myself and assumed that the search engines would do their job well out of self-interest.

I was confused when I realized that the majority of my pages didn’t even make it through the indexer.

The problem

How are people supposed to find my website? Even if they knew the correct search terms, they would hardly see anything on Google. After all, the console clearly stated that none of my blog posts appeared in the indexer. On my Gitea instance, on the other hand, several thousand pages were listed. A clear disproportion. And why does Gitea generate such a high volume of files?

Google’s crawler (a program that searches the web for content and makes it indexable for search) apparently finds every commit, no matter how small, in public repositories as well as the files behind them. Automation runs and other metadata are also indexed.

This is of course completely unnecessary and wastes energy in an area that I would much rather have in the presentation of my blog. So a solution is needed.

Possible solutions

A few possibilities immediately occurred to me:

Configure the crawler for each repository so that only the top folder level is searched.
Completely switch off search indexing for git.schallbert.de.
Use Search Console to make corrections until the indexer shows correct assignments on my blog posts.

The most obvious solution for me was to completely switch off the indexer for my Gitea instance. Correction loops only take effect days or weeks later with Google’s crawler and turned out too time-consuming. I deemed a separate robots.txt for each repository too complicated.

My implementation

I read the Gitea documentation. The suggestion of a robots.txt made sense to me immediately, as did its contents:

User-agent: *
Disallow: /

This tells crawlers from any party * that nothing should be indexed from the root directory / onwards.

Unfortunately, I had no idea how Gitea makes this file available in its instance:

To make Gitea serve a custom robots.txt (default: empty 404) for top level installations, create a file with path public/robots.txt in the custom folder or CustomPath.

Gitea’s configuration directory

So I experimented a bit. I created a public folder with the corresponding file, then a custom folder at different levels within the Gitea folder structure. Each time I restarted the container and checked whether the robots file also appeared on the server.

Nothing.

But then I got rid of the folder names and simply put the file in the Gitea configuration directory - the lowest level at which gitea no longer appears as a folder name. And this was the solution.

Test

After restarting, I was able to successfully display the file in the browser.

In the meantime, Google is also starting to remove the unwanted pages from the indexer. My blog articles will hopefully also be available in Google search over the next few weeks.

Repo-Lookout: Fix repository leaks

Schallbert — Fri, 20 Dec 2024

Repo information publicly available

One fine day I got an email from Repo Lookout. It said that one of my repositories was open to access from the internet. This posed a potential security risk as it might contain secret source files, hidden functions or even passwords.

At first I thought this “your repo is not secure” warning was a phishing attempt. But by simply entering the links contained therein, it became clear that Repo Lookout was right and that my Lectures repo was not only public as I had wanted, but that the version control metadata was also openly available on the web server.

Is that bad?

Normally, the internal structure and configuration (actions, discussions, wiki, etc.) behind a repository should not be public. Especially not if the repository is set up as private. But even with public repos, no one should be able to access the structure behind them.

Hence the mission of Repo-Lookout:

“Find source code repositories that have been inadvertently exposed to the public and report them to the domain’s technical contact.” - Repo Lookout /about (Crissy Field GmbH)

In this case, not critical, but unexpected and unpleasant.

Doesn’t do me any harm, because my web server only has the files there for retrieval and even if they were tampered with, they would have had no effect on my repository. I also had all secrets stored in specially created files as described in this article, so that they no longer appear in the configuration files. The repository is also located separately on the Gitea server instance. Nevertheless, only what I consciously want to make public should be connected to the Internet.

The image section from Gitea shows the same commit as the warning from Repo Lookout.

How it came about

In the deploy pipeline for my subdomain lectures.schallbert.de and landing page schallbert.de I have a direct checkout from the Gitea instance to the web server Caddy. This starts automatically as soon as the main branches receive an update. The runner starts a checkout action, which copies the repository to the corresponding directory on the web server.

Checkout action also copies the `.git` folder

The .git folder is simply set up as well. This is where all the data required by the version control software for state management of the repo is stored. I cannot see the copying process itself on Gitea, because the hidden .git folder does not even appear in the directory there. Understandable, because the entire display on Gitea is based on this folder’s content.

So the unwanted behavior remained under my radar - and according to Repo Lookout I am by no means the only one who has this problem.

Option 1: Fix on the web server

The most obvious solution is to block access to the file on the server side. This costs few resources and is easy to set up.

This forum entry shows how to do it: hide-entire-folder-caddyfile. For the files I want to protect, I add the following entries to the Caddyfile:

# /caddy2/Caddyfile

# [...]
respond /.git/* "Access denied" 403
respond /.gitea/* "Access denied" 403

This tells Caddy that when any file /* in the folder /.git is called, it should respond with the error code 403 “Forbidden”. The wildcard (*) is absolutely necessary, otherwise only the folder itself and not the files it contains will be locked.

To check, I check what happens when I request the Git logs:

# Terminal
curl <lectures.schallbert.de>/.git/logs/HEAD
Access denied

Works as designed.

Option 2: Fix in the checkout action

There is, however, a more elegant solution: ensure beforehand in the pipeline that the folder does not appear on the server at all.

Option 1: Using `sparse-checkout`

sparse-checkout allows you to select folders and files that should be part of the checkout. All other files in the repository remain untouched and do not appear in the branch. This saves time and storage space, especially with large repositories. But of course it only makes sense if you already know in advance that not all files need to be touched.

Negative list for `sparse-checkout`

In my case, I don’t want to copy the folders mentioned above to the server using checkout, but I want to copy everything else. How do I do that? Using negation in no-cone mode.

“The user has explicitly said ‘I want these directories and not those directories.’” - Derrick Stolee, Microsoft, on Github

The Checkout Action Guide states that sparse-checkout is also supported for the action automated by the runner.

Now I program with the help of the Github Actions Cheet Sheet:

# /.gitea/workflows/deploy-lectures.yml

# [...]
steps:
 - name: --- CHECKOUT ---
 uses: actions/checkout@v4
 with:
 path: ./tmp
 sparse-checkout: |
 /*
 !.git
 !.gitea
 sparse-checkout-cone-mode: false
 # [...]

To clarify: The script for publishing to my web server uses the action checkout, subfunction sparse-checkout and includes all files in the folder in the root directory tmp and below except .git and .gitea.

What is `no-cone mode`?

By default, sparse-checkout expects a list of folders to include for checkout. In no-cone mode, a list of patterns is expected instead. All operators that can also be used in the .gitignore to specify files, folders, omissions, etc. are possible here. This allows me to exclude certain folders, but has some significant disadvantages. Due to the much higher complexity of the pattern commands, the associated susceptibility to errors and the significantly more computationally intensive evaluation for larger repositories, the use of the no-cone mode is not recommended and is listed as “deprecated” in the documentation. Nevertheless, the proof is in the pudding!

Test with `sparse-checkout`

Now I upload the action to my Gitea instance and let my runner run it once. Then I log in to the web server and check whether the .git folder was created or not:

lectures# ls -la
[...]
drwxr-xr-x 8 root root 4096 Dec 20 10:41 .git
[...]

Damn, the folder is still there. I look in the logs of the action on my Gitea instance:

[...]
hint: 	git branch -m <name>
Initialized empty Git repository in /workspace/schallbert/lectures/tmp/.git/
[...]
::group::Setting up sparse checkout
[command]/usr/bin/git config core.sparseCheckout true

So it’s not because of sparse-checkout. It’s because of the way checkout works: Obviously, the .git folder is absolutely necessary for setting up the repository properly on my web server. So the only option I have is to delete it automatically after checking out.

Option 2: `rm -rf`

And so I try it by force:

# /.gitea/workflows/deploy-lectures.yml

# [...]
steps:
 - name: --- CHECKOUT ---
 uses: actions/checkout@v4
 with:
 path: ./tmp
 - name: --- REMOVE TEMPORARY FILES ---
 run: |
 rm -rfv ./tmp/.git ./tmp/.gitea
# [...]

And finally the .git folder no longer appears on my web server and my “repo leak” is patched. Thanks again to Repo Lookout!

Conclusion

I had the problem that the hidden .git folder, where the configuration and structure of repositories are stored, was published on my web server unintentionally and without my knowledge.

I have presented two working options for solving the problem here:

Set up an access ban on the web server
Modify the pipeline so that it automatically deletes the .git folder after it has been rolled out.

The second option is a bit more complex to implement, but it gets to the root of the problem instead of just fixing the symptoms. It also corresponds to the first principle of data protection: data minimization takes precedence over protective measures.

“What doesn’t exist cannot be lost” - Schallbert

Logrotate-Recursion

Schallbert — Mon, 18 Nov 2024

What isn’t working?

I actually wanted to use logrotate to clean up my logs daily and thus implement data economy: This way, IP addresses that I need for banning can be deleted uniformly and automatically after a short time. Otherwise, I’m not interested in the server logs at all and they just steal storage space.

With my current configuration, logrotate creates one log file per day, but names it xyz.log.1, xyz.log.1.1, xyz.log.1.1.1 instead of the expected xyz.log.1, xyz.log.2, xyz.log.3. This means that the deletion routine no longer works and the number of logs keeps growing.

And why?

I suspect that my logrotate configuration is faulty. At the moment it looks like this:

## LOGROTATE file named 'blog'
blog/* {
# it's ok if the file doesn't already exist
missingok
# Sets the logs to rotate in intervals
daily
# Tells the system to remove old logs and only keep the most recent rotated logs
rotate 7
# Rotated logs will be compressed and kept on disk if they are 10 MB or less.
size 10M
# compress and delaycompress: These two options are used together and indicate that 
# rotated logs should be compressed (gzip) except for the most recent one.
compress

delaycompress
}

As usual, the first thing I do is consult the manual, which I call up in the console using man logrotate. It says that the file to be rotated can be specified directly (and not just selected using a wildcard *). In addition, a log behavior can also be applied to multiple file paths by placing them in front of the curly bracket, separated by spaces.

The asterisk seems to be causing exactly my problem with the cascading logs: This not only rotates the actual target file access.log, but also touches all the files that have already been rotated again.

The solution

According to the manual, I rewrite the processing instructions for Logrotate so that instead of five separate instructions for the respective logs, I only create two files. These are also simpler and shorter than the original version above.

Example:

# rotate webserver log files.
gitea/access.log
blog/access.log
landing/access.log
lectures/access.log
{
 # Sets the logs to rotate in intervals
 daily
 # Tells the system to remove old logs and only keep the most recent rotated logs
 rotate 7
 # compress and delaycompress: These two options are used together and indicate that 
 # rotated logs should be compressed (gzip) except for the most recent one.
 compress
 delaycompress
}

Configuration update

I can roll this out using my existing server configuration pipeline by simply pushing it into the repository on the server. Now I just have to manually copy the files there to /etc/logrotate.d.

Testing Logrotate

Now it would be nice if I could test whether the changes I made actually work. To do this, I use the logrotate user manual again and type:

logrotate -d -v <LOGROTATE-DESCRIPTOR-FILE>

With the debug and verbose flags activated, logrotate will apply the rotation rule specified in the configuration file and provide feedback, but will not actually make any changes or rotations on the file system. For me, the result of the test looks something like this:

lectures/access.log 
 after 1 days (7 rotations)
empty log files are rotated, old logs are removed
considering log gitea/access.log
 Now: 2024-11-28 17:19
 Last rotated at 2024-11-27 00:00
 log needs rotating
considering log blog/access.log
 Now: 2024-11-28 17:19
 Last rotated at 2024-11-16 00:00
 log needs rotating
considering log landing/access.log
 Now: 2024-11-28 17:19
 Last rotated at 2024-11-07 00:00
 log needs rotating
considering log lectures/access.log
 Now: 2024-11-28 17:19
 Last rotated at 2024-09-21 00:00
 log needs rotating
rotating log gitea/access.log, log->rotateCount is 7
[...]
renaming lectures/access.log.7.gz to lectures/access.log.8.gz (rotatecount 7, logstart 1, i 7), 
renaming lectures/access.log.6.gz to lectures/access.log.7.gz (rotatecount 7, logstart 1, i 6), 
renaming lectures/access.log.5.gz to lectures/access.log.6.gz (rotatecount 7, logstart 1, i 5), 
renaming lectures/access.log.4.gz to lectures/access.log.5.gz (rotatecount 7, logstart 1, i 4), 
renaming lectures/access.log.3.gz to lectures/access.log.4.gz (rotatecount 7, logstart 1, i 3), 
renaming lectures/access.log.2.gz to lectures/access.log.3.gz (rotatecount 7, logstart 1, i 2), 
renaming lectures/access.log.1.gz to lectures/access.log.2.gz (rotatecount 7, logstart 1, i 1), 
renaming lectures/access.log.0.gz to lectures/access.log.1.gz (rotatecount 7, logstart 1, i 0)
[...]

The output shows that all the desired log files are taken into account by logrotate and also that the rotation is carried out correctly on a daily basis.

Learned something new again 😃

Gitea act_runner: Jump-start issues

Schallbert — Fri, 30 Aug 2024

Problem statement

Since I run this page myself I experience problems starting Gitea and act_runner. Sometimes, the runner won’t start. It exits with status -1 and so my workflows wouldn’t run the website build, verification and deploy tasks.

The error message is always the same:

[...]
runner-1 | time="2024-08-26T10:09:46Z" level=info msg="Starting runner daemon"
runner-1 | time="2024-08-26T10:09:46Z" level=error msg="fail to invoke Declare" error="unavailable: 502 Bad Gateway"
runner-1 | Error: unavailable: 502 Bad Gateway
runner-1 exited with code 1
gitea | 2024/08/26 10:09:46 cmd/web.go:242:runWeb() [I] Starting Gitea on PID: 16

Ok, the runner cannot hook onto Gitea. When I restart all services via docker restart, the act_runner container tells me it would be missing a network with ID <long hexcode ID>

Interim corrective action

I didn’t find time for resolving this issue. So I just ran docker compose twice:

# schallbert server-console
docker compose -f /path/to/giteas/composefile up -d

The second time, act_runner would start successfully (because Gitea is ready):

gitea | 2024/08/26 10:38:17 routers/init.go:116:InitWebInstalled() [I] Git version: 2.45.2 (home: /data/gitea/home)
runner-1 | time="2024-08-26T10:38:22Z" level=info msg="Starting runner daemon"
runner-1 | time="2024-08-26T10:38:22Z" level=info msg="runner: action-runner, with version: v0.2.10, with labels: [ubuntu-latest], declare successfully"
runner-1 exited with code 0
runner-1 | time="2024-08-26T10:40:32Z" level=info msg="Started runner daemon"

Not very satisfying to do things twice. Plus, I did this manually. So why is that?

Root cause analysis

First I thought it was due to Caddy not being ready as it runs the reverse proxy, connecting Gitea to the internet. In reality, Gitea is the culprit: At the time, docker starts act_runner, its startup procedure wouldn’t be complete. So I try to find out how to manage dependencies in docker.

Solution: Reflecting dependencies

First, I try linking gitea to the runner in the docker compose file.

depends_on

# gitea/docker-compose.yml
 ## service: runner
 ## [...]
 depends_on:
 gitea:
 condition: service_started

Well, still act_runner seems to be starting early. Docker just makes sure it has started the container, and does not wait for it to return a healthy signal.

Healthcheck

So I have to make sure that gitea’s startup procedure is complete. For this, docker provides the condition service_healthy. I adjust the runner configuration like so:

# gitea/docker-compose.yml
 ## service: runner
 ## [...]
 depends_on:
 gitea:
 condition: service_healthy # required so runner can attach to gitea
 restart: true

The service_healthy qualifier is returned by the healthcheck function. It consists of a so-called test and some environment parameters that define intervall, number of retries, delay and timeout conditions for checking service ready.

For the test I chose a simple command, assuming that Gitea (and, implicitly, Caddy too) would be healthy once it is able to respond to a GET request from curl.

# gitea/docker-compose.yml
## service: gitea
## [...]
 healthcheck:
 test: ["CMD", "curl", "-f", "https://git.schallbert.de/"] # checks if gitea is available
 interval: 10s
 retries: 3
 start_period: 30s
 timeout: 10s

With these lines, docker checks if gitea is “healthy” and only then starts act_runner. This permanently solved the problem and both Gitea and act_runner are stable.

Sending error logs

Schallbert — Tue, 20 Aug 2024

In this article I’ll look at how to set up “monitoring” for my server. Applications and services should be able to send me notifications in the event of an error.

What is this about?

Learn about and select sending mechanisms
Write test messages and verify the automation
Automatically send error report from Borgmatic
Send runner logs through Gitea
Notification when logging into my server via ssh
Create and send logs for server updates / server errors

Verification and monitoring

After running an automation, I want to know whether it was carried out successfully and whether all programs and services started their work as expected. This should apply to any automation - whether it is specifically sending an update, creating automatic backups or an action from Gitea, it doesn’t matter.

Normally I would use reporting mechanisms from act_runner for something like this. In the case of a server update, the runner is not available due to the circular reference already mentioned in the article Rolling out the server configuration, as all applications have to be shut down temporarily.

In addition, applications may have their own procedures for monitoring. So I have to look at mechanisms that allow me to easily access the information.

What characterizes good monitoring for me?

It is unobtrusive, so it only reports in the event of an error or unusual occurrence.
It provides specific information and error messages that are easy to understand.
It uses a message channel that works even if the system being monitored crashes.
It presents reports and error messages in isolation from other topics and does not mix anything.
It is brief.

Methods of automatic reporting

I will break this section down. In the general part, I will discuss the on-board tool for asynchronous monitoring that is available to me on the Ubuntu server. After that, I will look at the solutions that are partly built into my services or the applications that are compatible with them. I do not want to limit myself to the classic tool of email, but also look at “more modern” communication channels such as messenger programs or RSS feeds.

Mail via console - curl

Sending emails as notifications is common practice in many companies. With Linux, this can usually be done without any additional programs: The standard curl can help here.

curl is a program for transferring data from or to a server. If I enter my blog as the target, I get the HTML page output as a text file on the console:

curl https://blog.schallbert.de

This made me notice how much unnecessary data my blog software generates. I’ll have to clean that up later. Back to the topic: You can also use curl to access any web backend - for example a mail server:

# copied from https://stackoverflow.com/questions/8260858/how-to-send-email-from-terminal
curl --url 'smtps://smtp.gmail.com:465' --ssl-reqd \
 --mail-from 'from-email@gmail.com' \
 --mail-rcpt 'to-email@gmail.com' \
 --user 'from-email@gmail.com:YourPassword' \
 -T <(echo -e 'From: from-email@gmail.com\nTo: to-email@gmail.com\nSubject: Curl Test\n\nHello')

This only works for my mail provider if I allow logins from external clients. Google, for example, calls these “less secure apps”. As described in my post Server configuration with Git, I’m not a fan of writing secrets into anything - so I would rather not use the direct route via curl.

Mail via console - mail, mailx, mailutils, swaks

If you don’t want to always provide all the configuration information for the server and secrets, there are various handy tools for the console such as mailutils or swaks. Here, the connection to the mail server is configured once using the tool and can be stored in environment variables, for example. The syntax varies from program to program, but an email always drops out at the end.

The only major disadvantage for me is that information domains are mixed up. I would not want another “report thread” in my emails that I’d have to search for in the mass of messages. Other notification media in contrast allow me to set an automatic expiration date so that they disappear from my list after a set time.

Create RSS feed

Unusual but possible: I could create the monitoring as an RSS feed like my blog (schallberts-blog-feed) has e.g. using a Jekyll instance and put it online as a website. This would be easy to subscribe to, were readable with practically any reader and I could even set it up separately for each application. But there are obvious disadvantages:

High effort: Gitea runner with Jekyll instance, web server and subdomain required.
Publicly available: Suddenly build processes, updates, upgrades and error messages are accessible to everyone.
Error-prone: If Gitea, act_runner, my proxy or the web server crashes, I don’t get any reports.

With the last point at the latest, this option is out of the question for me. I want to get a report when my applications don’t do what they’re supposed to. Let’s take a look at the applications I already run and see how they tackle this problem.

borgmatic

Borgmatic is compatible with a lot of monitoring options. These include apprise, ntfy, healthchecks, cronitor, pagerduty, cronhub and grafana.

Apprise is a library. It is open source and can be integrated into an existing application as a dependency. Like an adapter, it enables asynchronous communication between the application and various communication services such as SMS, mail, messenger (e.g. Signal), various home automation systems or the notification mechanism of various operating systems. The latter, however, only works on the local machine. The trigger for communication must always come from the application.
ntfy is a push notification service. It is open source and can be self-hosted or used as a service via web application. Apprise, for example, supports ntfy as a communication service. The structure is quite simple and works like MQTT via publication subscription / broker client mechanism, but HTTP-based.
Healthchecks is a service. It is open source and can be self-hosted. The service is there to monitor regular activities and can act as a dead man’s switch: If, contrary to expectations, there is no response from the monitored program, it can raise an error message itself. This can in turn be forwarded to various communication services.
Cronitor is a monitoring solution and web application that, in addition to the notifications I need, provides a lot of analysis tools, performance measurements and metrics - mostly for money. There is a free “hacker” account with limited functionality, but this tool is also far too big and complex for me.
PagerDuty sees itself as a commercial “operations” platform that provides “incident management”, automation, “business operations”, “AIOps” etc. It’s out for me straight away. At the latest when I read the word “AIOps” 😅
cronhub looks like a commercial web application to me that, like Healthchecks, can create cron jobs, monitor them and report errors. It is of no interest to me because it does not seem to be open source and I could not host it myself.
Grafana is an open source web application that can either be self-hosted or used as a cloud service. Although many larger companies and projects use the application, it is orders of magnitude too extensive and feature-rich for my purposes.

fail2ban

Fail2ban does not have an automation solution like borgmatic. It simply creates log files that need to be evaluated in order to obtain content for notifications. At the moment I cannot think of anything that I absolutely need to know about Fail2ban. So I am not creating any reports for this for now.

Gitea

Gitea allows the log files to be configured very precisely. Access logs can be written out separately from service logs, repository logs or action logs and then processed further. According to my research, Gitea only offers a mailer as a notification system. In my opinion, this is primarily intended for repository and action logs.

Here, too, it would be most beneficial for me to analyze the logs and create a report myself if necessary.

Server

I already roll out my server configuration files using a script. So, in the event of an error, I could redirect the logs to a file and attach them to a report in any channel.

In addition, successful logins on the server would be worth a message. Then I can immediately determine whether it was me or not.

Selecting the reporting program

I not only have to select a monitoring program, but also choose a communications service through which the reports are sent.

Program	Advantage	Disadvantage
curl	simple	configuration must be provided
	works super easily with ntfy¹	No monitoring if the server crashes
	Onboard tools
	no service required
Apprise	enables integration of the Signal API²	Complex in interaction with external services³
	no service required
Healthchecks	Can act as a watchdog/dead man switch	Registration required
	Can be hosted locally⁴	Local hosting contradicts the watchdog concept

I only have a small server with a few applications, so I keep it as simple as possible and choose Apprise for services like borgmatic, which come with this library anyway, and curl for those whose reporting mechanics I have to program myself.

Communication channel

Program	Advantage	Disadvantage
SMS	High reliability	Registration required
		Limited to a few characters
		No attachments possible
		No topics, domain mixing
Mail	easy to set up	Poor searchability
RSS feed	Good sortability	Complicated to set up, error-prone
	Client very lightweight	Local hosting contradicts watchdog concept
		Content publicly available
ntfy	Simple and lightweight	Registration optional for web use
	Client purpose-oriented	Insecure: No encryption without registration
	Free for small users
	Independent of your own machine
Signal	“Note to self” easy to set up	Setting up and configuring the Signal API complex
	Independent of your own machine	Topics somewhat difficult to implement
		Useless if the server crashes

SMS, mail and RSS feed are already out of the question for me due to some of the disadvantages already explained above. So that leaves ntfy and Signal. ntfy impresses with its simplicity: send an HTTP PUSH request to a self-defined “topic” and subscribe to it on your cell phone - done. It is also easy to expand, because with self-hosting I can later increase security if necessary (encryption) and control the sending behavior. Signal, on the other hand, requires a separate client with relatively complex configuration. In addition, the devices would have to be connected to the server and cell phone, so it is not easy to add new subscribers. On the other hand, the transmission is well secured end-to-end, I do not have to register and it is also free.

For now, I have decided to go for the less complex solution with ntfy.

Logging

Good. Now it’s clear that I’m calling home using curl and apprise via the ntfy service. Let’s see what content needs to be transmitted and how to make it as unobtrusive as possible, but still short and concise.

When should logs be sent?

borgmatic: If a backup fails
If I successfully log in to my server or Gitea
After rolling out an update to the server configuration
Docker: If an application fails or doesn’t start
Gitea: Failed run of act_runner

What should I include in a report?

I want the classic “W questions” answered.

When did it happen (time stamp)?
Which application is reporting?
What happened?
Where (module, line of code) etc. did it happen?
How many were injured (severity, recovery)?

A notification then looks something like this:

TIMESTAMP APPLICATION PRIORITY MESSAGE EFFECT DETAIL

Rough version of a report

An ntfy message could look something like this:

curl \
 -H "Title: Error Borgmatic" \
 -H "Priority: urgent" \
 -H "Tags: warning" \
 -d "YYYY-MM-DD HH:MM:SS Backup creation aborted. Access to repository blocked" \
 ntfy.sh/schallberts-topic

I install the corresponding app on my phone, register on the “Topic” and send the message. It’s nice when things just work! Aha, the app shows the time of receipt. That’s accurate enough for me.

Implementation

Here I’ll take a look at all the services for which I’d like to set up notifications one by one.

borgmatic

Let’s start with a pilot test in small steps. First I configure Borgmatic to send a message via Apprise to ntfy if the backup creation fails:

# /borgmatic.d/config.yml
on_error:
- echo "Error while creating a backup."
- apprise -vv --title "Borgmatic Error"
--body "Could not run {output}. Aborted {error}."
ntfy://schallberts-topic

Then I test the command by calling apprise within the Borgmatic container. And indeed, it works. But getting here took me an hour, as the yml with its syntax rules even interprets within strings and borgmatic constantly refused to read the configuration file due to : and - characters. If you see an error similar to this one:

At 'on_error[1]': {'apprise -vv --title "Borgmatic Error" --body "Could not run {output}': 'Aborted {error}." ntfy://schallberts-topic'} is not of type 'string'

This shows that the interpretation of characters or indentations went wrong and the punctuation needs to be checked. Alternatively, the pipe operator | can be used to combine a command. Reference: yml specification

Server

An application of ntfy to monitor logins on a server can already be found in the documentation of ntfy itself. The description shows that something like this is easy to implement yourself and gives a great example using Pluggable Authentication Modules (PAM library).

Insert the following code at the end of the sshd file in the etc/pam.d directory:

session optional pam_exec.so /usr/bin/ntfy-ssh-login.sh

This line tells PAM that when a session is opened via ssh, it should call the executing module pam_exec, which then runs the script specified below. The value optional means that the configuration file should continue to be run even if the action fails. More details on how to use PAM can be found, for example, on Baeldung.

Then you simply have to store the ntfy call in the ntfy-ssh-login.sh script when the script determines that PAM has detected an open_session event.

That’s exactly how I implemented it and it works straight away. Great! The only disadvantage: I made this modification directly on the server. Without a container and outside of my configuration backup. If I now have to set up the server again for some reason, the change in PAM is lost and I won’t receive any more notifications until I manually enter the change again.

Server configuration

To send a report after running the configuration automation, I actually only have to adapt the server-config-action script that I wrote in the last article.

First, I want to be informed when the script runs without errors. To do this, I add a curl command at the end of the file.

# server-config-action.sh

# action commands...
# [...]
# send success notification
curl \
 -H "Title: server-config-action" \
 -H "Priority: low" \
 -H "Tags: white_check_mark" \
 -d "Rollout successful" \
 ntfy.sh/schallbert-server-config-push-topic

However, if the script does not run without errors, I would like to do the following:

Cancel actions after the error occurs
Create an error log
Send this log

I can achieve the first point by adding a trap for errors:

# server-config-action.sh
trap 'handle_error $LINENO' ERR

# action commands...

This calls the function handle_error. It is given the line number where the error ERR occurred. It also covers errors that can occur when restarting the container. I create the error log by redirecting the output of the individual script commands. I achieve this with the following line:

# server-config-action.sh

exec 3>&1 1>server-config-action.log 2>&1

# action commands ...

With this command, any standard output stdout should be passed on to the file descriptor 3, specified here with the log file server-config-action-log, overwriting it. For “append” there would have to be two redirection operators >>⁵. The log file thus replaces the console output, which I still had at this point in the previous article.

To have the log sent to me, I now define the following function at the beginning of the Bash script:

# server-config-action.sh
# on error, send a notification
handle_error() {
 # stop redirecting to file
 exec 1>&3 1>&2
 curl \
 -H "Title: server-config-action FAILED" \
 -H "Priority: high" \
 -H "Tags: x" \
 -T server-config-action.log \
 -H "Filename: server-config-action.log" \
 ntfy.sh/schallbert-server-config-push-topic
 exit 1
}

# set error trap
# redirect stdout and stderr to file
# action commands...

Gitea

For Github Actions reports, there is already an existing example at ntfy_examples/#github-actions. And the best part: Since the act_runner from Gitea is compatible with Github Actions at in the environment parameters, sending to ntfy works for me straight away. In the runner’s workflow file, all you have to do is enter the curl command specified on the website and you’re done.

Result

Five notifications about the most important processes on my server have now been set up. I have understood the underlying mechanics and can create additional notifications at any time if I need them. If I trigger all notifications as a test, my smartphone will display the as shown.

But I have not yet achieved independence from the system to be monitored. All notifications come from the affected machine and there are sometimes even dependencies between Docker containers. For example, the act_runner has to connect to Gitea via websockets. And that only works if Caddy provides the reverse proxy.

If I find out in the next few months that a service “under my radar” is no longer able to work, I will have to establish independence.

Context for setup with ntfy ↩︎
Context for the Signal API ↩︎
Report by asad-awadia on setting up the Signal API ↩︎
Healthchecks is directly available as a Docker image. ↩︎
Incidentally, the operators << and >> are not difficult for me to remember, because I used to have a lot to do with the programming language C. There - and in many other programming languages too - these are shift operators that can “shift” a value bit by bit or move one field to another. In the RISC architectures that I used at the time, the shift operation was “cheap”, i.e. very fast and memory-saving. ↩︎

Server configuration rollout

Schallbert — Wed, 31 Jul 2024

In the previous article, I brought my configuration files under version control. Now I want to automatically install the updates provided on the server.

What is this about?

Writing a script to automatically detect the update trigger
Server applications should be shut down and a backup copy created
The script should distribute the configuration on the system
All applications should then be restarted

Ways and possibilities

Here, too, I spent several hours researching. For larger projects, infrastructure experts seem to use specialized automation tools. These include Ansible, simpler tools such as cdist or even Kubernetes for highly scalable services.

Automation services Ansible, cdist, Kubernetes

Ansible and cdist seem to follow a similar concept: On the source machine (in this example my laptop) I create the configuration for my server and store this and the instructions for configuring my services in a “playbook” (Ansible) or in “types” (cdist).

To put it simply - as I understand it - the tool then takes care of building the configuration at the push of a button, dialing into the target host via ssh, pushing it over there and then starting it. For Ansible, there are even tutorials like this one for my scenario with docker-compose, which makes getting started even easier.

Kubernetes takes a different approach. It sees itself as more of a container manager, load balancer and scaling agent, but can do similar things for my purposes.

My approach

I, on the other hand, only need a fraction of the capabilities of these programs. I am also put off by the “additional” ssh channel, the configuration effort, the additional programs sometimes required on the target system, and the necessary reading and selection of the best tool for me. Because thanks to my very simple pipeline from the last article, the configuration is already on my server. It “only” needs to be copied to the right places and the affected services restarted.

Therefore, I am trying to solve this problem using on-board tools, my brain in working order and a few searches in relevant forums on the topics “executing a script when a file changes and “copying folder structures in Linux”.

Risk of circular reference

However, I am taking a risk: Since the deployment runs via Gitea, but the configuration affects Gitea itself, if there is an error in this module I can no longer change or reset anything: The Gitea service is then broken. I would have to get the configuration up and running again manually on the server.

I’ll try it out anyway and see if I am actually confronted with such a problem. If so, I’ll just switch to cdist and document it in a separate article! 🤗

Preparation: Create folder system and scripts

I think about it for a moment and create a few folders in the server file system:

# /opt/server-config
mkdir automation-hooks-trigger
mkdir automation-hooks-handler

These folders should represent the two sides of the automation. trigger contains text files that can be manipulated from the (web) service side. For example, act_runner should write the file server-config-update as soon as an update is available.

service –> writes to trigger file ||| server_handler() –> trigger_file.changed ? run_action() : loop()

The handler folder contains script files that make the necessary changes to the server file system.

Implementation: Shell script to handle the update

In the previous article, I executed the command touch server-config-update.txt in the act_runner container to signal the presence of a new server configuration. On the server, I now use the following code to periodically check whether this file has changed.

### set directories
actionfile=/opt/server-config/automation-hooks-handler/server-config-action.sh
triggerfile=/opt/server-config/automation-hooks-trigger/server-config-update.txt

### Set initial time of file
LTIME=`stat -c %Z ${triggerfile}`

while true
do
 ATIME=`stat -c %Z ${triggerfile}`

 if [[ "$ATIME" != "$LTIME" ]]
 then
 ${actionfile} 2>&1
 LTIME=$ATIME
 fi
 sleep 10
done

The operating system is given the interpreter with which the script is to be executed via #!/bin/bash. The command stat -c %Z queries when the file was last changed and returns the time in Epoch format. Finally, the time taken at the beginning of the script is compared with the time taken in the loop and if there is a change, the deploy routine that is still to be written can then run. The reference time is then updated. Finally, the script (sleep 10) pauses for ten seconds before the query starts again. I use absolute paths so I’m able to both start it from console and per service cron oder systemd.

First test and hooking into “autostart”

Now we need to make the file executable for a first test:

# change file mode bits: add "executable" flag to server-config-handler script
chmod +x /opt/server-config/automation-hooks-handler/server-config-handler.sh

Next, I create the server-config-action.sh file and add just an echo command. To test the setup I create my update trigger file locally and start the script. I then modify the triggerfile in another shell using touch server-config-update.txt and --- CONFIG UPDATE TRIGGER detected --- appears in the console. Great!

Later on the server, I need to have the script run automatically after a restart. To do this, I use the cron tool help and create a new entry using crontab -e:

# cron can automatically execute recurring tasks
# in this case, we're running server-config-handler script on reboot
@reboot sh /opt/server-config/automation-hooks-handler/server-config-handler.sh

With the command ps aux I can now check whether the script is actually being executed:

schallbert@server: ps aux
[...]
8:15 0:00 /bin/bash ./server-config-handler.sh
8:15 0:00 sleep 10
[...]

Backup

Making a backup of my applications and files before I roll out the update makes total sense. So I tell Borg that I want to create a backup now. Of course, I have to stop all services first. This means that all data is accessible, coherent and static.

Freeze state and data

To do this, I create a script in automation-hooks-handler that automatically terminates all containers except borg.

#! /bin/bash
# /opt/automation-hooks-handler/backup-pre-action.sh
# this shell script shuts down all docker containers prior to backup

echo "Shutting down containers for backup:"

echo "watchtower..."
cd /opt/watchtower
docker compose down 2>&1
# [...]

The expression 2>&1 means that any error output is redirected to the console. The number 1 represents the file descriptor for stdout, while 2 means stderr. The operator >& functions as a redirect merger. Later, we can go to this point and write the output to a log file - but I’ll leave that out for now for the sake of simplicity.

Borgmatic: Trigger handler mechanism #2 and #3

I can now run the script in two ways:

As a call by the server-config-handler script described above
By the automation solution borgmatic placed in front of borg

I choose the second option and therefore write the following commands in borgmatic.d/config.yml:

# borgmatic.d/config.yml
# List of one or more shell commands or scripts to execute before
# creating a backup, run once per repository.
before_backup:
 - echo "Triggering container shutdown for backup."
 - touch /etc/automation-hooks-trigger/backup-pre.txt
 - sleep 20
 - echo "Assuming container shutdown complete. Creating the backup now."
# [...]
after_backup:
 - echo "Triggering container restart after backup."
 - touch /etc/automation-hooks-trigger/backup-post.txt
 - sleep 10
 - echo "Assuming container restart complete. Exiting."
# [...]

For this to work properly, I have to create a volume in the associated docker-compose.yml and have it point to the path in the server’s file system: ${VOLUME_UPDATE_TRIGGER}:/etc/automation-hooks-trigger Now I set up the other side of these triggers: Handlers monitor trigger files for changes and call action scripts accordingly. These look very identical to server-config-handler.sh except actionfile and triggerfile paths.

Creating the backup

If I were to address borg directly, the backup could be created using create. To do this, I would have to specify in which repository the backup copy should be saved and under which name. In the example below, this is specified using the scope operator: ::config-update. The folders to be backed up are then specified.

borg create /path/to/repo::config-update ~/opt

I use borgmatic, which takes a lot of work off my hands using the configuration file. However, I have to execute the command in the container. To better check whether everything is working, I output statistics “verbose” to the console (--stats -v 1) and display the copied files --files.

docker exec borgmatic sh -c "cd && borgmatic --stats -v 1 --files 2>&1"

I add this line to the automation script.

Second test to create the backup

If everything works now, the complete process looks like this:

After the changed files have been received by Giteas Automation, act_runner (in the Docker container) stores the files on the server and then sets the server-config-update trigger.
Within 10sec the trigger is recognized by server-config-handler.sh, which then calls server-config-action.sh.
borgmatic (in the Docker container) is instructed to create a backup. This in turn writes the pre-backup trigger.
Again within 10sec this pre-backup-handler.sh calls the backup-pre-action.sh script and stops all containers except borgmatic.
Due to the built-in delay, borgmatic waits for this and then creates the backup.
After the backup, borgmatic writes the backup-post-action.sh trigger.
Within another 10 seconds, post-backup-handler.sh recognizes the changed file and restarts all containers via post-backup-actions.sh.
borgmatic tells server-config-update.sh whether any errors have occurred anywhere in the process so far. If not, it continues.
All Docker containers are stopped.
The server configuration is rolled out to the appropriate locations.
All Docker containers are restarted with the new configuration.

A first success: These scripts are already running on my laptop up to point 6:

schallbert@laptop: touch server-config-update.txt
server-config-handler@laptop: --- RUN backup ---
borgmatic@docker: /etc/borgmatic.d/config.yml: Running 4 commands for pre-backup hook
 Triggering container shutdown for backup. 
backup-pre-handler.sh@laptop: --- RUN backup-pre-action.sh ---
 Shutting down containers for backup:
 watchtower...
 [...]
 complete.
borgmatic@docker: Assuming container shutdown complete. Creating the backup now.
 local: Creating archive
 Failed to create/acquire the lock /mnt/repository/lock.exclusive

Troubleshooting for “failed to acquire the lock”

This problem occurs for me when borg reports an error when creating a backup that causes the program to abort. In this case, the repository is apparently not released correctly, so that after restarting the container it remains reserved for the old, now non-existent container. The following command solves this problem:

docker exec borgmatic sh -c "cd && borg break-lock /mnt/repository

Transferring the configuration

Now the configuration files have to be copied to the correct location on the server. Fortunately, I had already cloned the target folder structure when creating the repository, so I should be able to do this with a single copy command without “hardcoding”. After a bit of online research and a look at the user manual for the copy command man cp, I have my command:

# copy recursively contents of folder "server-config" to "/opt"
sudo cp -r -v /opt/server-config/. /opt 2>&1

This tells the operating system to copy the contents (/.) of the folder server-config recursively (-r), including all subfolders into the folder opt, which is located in the root directory /. cp works in an overwrite-supplement manner, so it will create files that do not yet exist and overwrite existing ones, and will not “copy them next to each other” under the same name. With the -v option I can output additional details, and with 2>&1 I redirect the error output to the console.

All folder indicators must be exactly where they are in the command: A slash after /opt/ would copy folders redundantly without overwriting, but would overwrite files. Without . the folder server-config would be created in the target path.

Switching to rsync

Unfortunately the cp command also copies a few files that I don’t want copied: repository-specific folders such as .gitea, or the folders for triggers and handlers. I only need these under server-config, not directly in opt. To fix this, I use the rsync command instead. There I can use a -u option so new files owerwrite older ones only and add --exclude to exclude files and folders that should not be copied. This looks like so:

rsync -r -u -v --exclude '.*' --exclude 'README.md' --exclude '<otherFolders>' /opt/server-config/. /opt 2>&1

But suddenly gitea no longer starts. Error message:

docker@server: [...] failed to load config file "app.ini": open: permission denied

After a long time of pondering and restarting the Docker client several times, I see that rsync has written the permissions of the source file to the target file, which was not the case with cp before: -rw-------. Now I change this by running chmod +r app.ini. Everything starts up again as usual! 🎉

Restart all applications

Since I run everything on my server in Docker, two simple commands are sufficient:

docker stop $(docker ps -a -q) 2>&1
# [...roll out config changes...]
docker restart $(docker ps -a -q) 2>&1

The finished automation

My script is now finished and simply calls the action script.

#! /bin/bash
#/opt/automation-hooks-handler/server-config-handler.sh
#[...]
 if [[ "$ATIME" != "$LTIME" ]]
 then 
 echo "--- CONFIG UPDATE TRIGGER detected ---"
 ./server-config-action.sh 2>&1
 LTIME=$ATIME
 fi
 sleep 10
done

The Script server-config-action then executes the above described actions:

#! /bin/bash
#/opt/automation-hooks-handler/server-config-action.sh

echo "--- RUN backup ---"
docker exec borgmatic sh -c "cd && borgmatic --stats -v 1 --files 2>&1"
echo "--- STOP all containers ---"
docker stop $(docker ps -a -q) 2>&1
echo "--- DEPLOY config ---"
cp -r -v /opt/server-config/. /opt 2>&1
echo "--- RESTART all containers ---"
docker restart $(docker ps -a -q) 2>&1

In order for the whole thing to work reliably, the three handler scripts must run in the background:

server-config-handler.sh
backup-pre-handler.sh
backup-post-handler.sh

They react to the respective triggers by act_runner from Gitea or by borgmatic. I will now expand the crontab accordingly and then I am done with the task for now. The console output of the entire process looks like this:

--- RUN backup ---
/etc/borgmatic.d/config.yml: Running 4 commands for pre-backup hook
Triggering container shutdown for backup.
--- RUN backup-pre-action.sh ---
Shutting down containers for backup:
gitea... 
fail2ban...
complete.
Assuming container shutdown complete. Creating the backup now.
local: Creating archive
<borg archive stats>
/etc/borgmatic.d/config.yml: Running 4 commands for post-backup hook
Triggering container restart after backup.
--- RUN backup-post-action.sh ---
Restarting containers after backup:
fail2ban...
gitea... 
watchtower...
complete.
Assuming container restart complete. Exiting.
local: Pruning archives
local: Compacting segments
compaction freed about 1.82 MB repository space.
local: Running consistency checks
summary:
/etc/borgmatic.d/config.yml: Successfully ran configuration file
--- STOP all containers ---
<container ids>
--- DEPLOY config ---
sending incremental file list
<files that are copied>
sent 206,558 bytes received 3,463 bytes 420,042.00 bytes/sec
total size is 193,167 speedup is 0.92
--- RESTART all containers ---
<container ids>

Great! Now all I need is for this log to be delivered to me if something goes wrong.

Server configuration with Git

Schallbert — Mon, 15 Jul 2024

What is it about?

I want version control for the configuration of server applications
Discussion of technical solutions for implementation
Tutorial: Exclude Docker/Gitea/other “secrets” from version control
Tutorial: Create a deployment pipeline
introduce automation trigger for subsequent rollout of files on the server
Next article: Integrate automation on the server and install upgrades

Background

I have Watchtower running on my server to automatically keep the installed distributions up to date and freshly patched. Recently I had a case where my Gitea Action Runner would no longer start automatically and even when started manually it gave an error message.

Docker had apparently updated itself without my supervision and was now throwing a volume error when starting up the runner container, which I had never seen before. The error message was clear and could easily be fixed with small changes in a configuration file for Gitea. Nevertheless, I now had the feeling that version control for my configuration would be useful for my future self in order to be able to better understand changes, updates and the reasons behind.

Preliminary considerations

It sounds like a circular reference to me: I record the configuration files for my server in Gitea, which itself runs on my server. This could become interesting with auto-deployment. But more on that later.

Two options spontaneously come to mind for getting version control with automatic synchronization:

1. Hardlink

This solution would store the configuration files scattered across many folders on my server in a folder declared as a repository using a hardlink. Why use a hardlink? Because every service on my server runs encapsulated in itself and has its own configuration files and environment variables stored together with the service. Without hardlinks, I would have to version the entire service folder and make my .gitignore correspondingly complex.

With Git, I would version the files mapped via hardlinks and make their contents available on Gitea in this way.

schallbert@server:/server-config-files –> hardlinks –> repository –> Gitea

So I would get a downstream “version monitoring” with a backup copy in the sense that the files are now lying around multiple times.

2. Repo and auto-deploy to the server

In this scenario, I keep the configuration files locally on my laptop and can version them with Git as usual. On Gitea, I would map the repository, and at the end of the chain, my runner would have to auto-deploy on the server every time the configuration changes and then restart the affected containers.

schallbert@laptop:/server-config-files –> repository –> Gitea –> act-runner –> server:/<service1...ServiceN>/config-files

I would get the administration “main” on my laptop, and the server would follow the mapping on Gitea.

Problems

In both cases, I don’t have the option of testing changed configurations in advance. Everything I do goes straight to “Prod” and would be live. In the worst case, I can easily mess up my setup.

This doesn’t change the status quo ante, where I updated the files directly on the server. So a new problem only in the sense that I didn’t have a Prod operation when I set it up in the first place and so there was no risk of failure.

Solution two seems to be more complex to implement, because I need a deployment pipeline that rolls out the files in the repository on my server. Especially since the runner is in a Docker container, while the configuration files are located directly in the server’s file system.

Direct access to the server file system is technically possible, but according to my limited understanding it would mean a large attack surface for all my services, whose configuration files could now be directly manipulated via the repository in Gitea.

Selecting my solution

Solution two at least places the immediate live problem on my laptop, so that I don’t have to mess around with the production system in the first step. I think it would be easier to set up an integration environment here with which I can check my configuration changes in advance. Since I have all of my services running in Docker, this could perhaps be solved quite easily.

Let’s get to work

Create a configuration repo

Okay, then the first step is to get the configuration files from the server. To do this, I use the file transfer command scp: scp server:/path/to/source path/to/target about a dozen times until I have caught all the files.

I set up the folder structure in this repo exactly as the files are on the server. I hope that this will make my life a little easier later.

Secrets in `docker-compose.yml`

But what do I do with “secrets” in the configuration files? Private keys, registration tokens, hashes? I would rather not have them lying around more or less openly in the repository. When using docker-compose it’s quite simple: I can store secret values in hidden files for environment variables and exclude them from Git tracking. In the simplest case, such files are simply called .env and contain a list of environment variables in the style of

BORG_PASSPHRASE="<redacted>"

In the corresponding docker-compose.yml I pull the variable from the .env file as follows:

- BORG_PASSPHRASE=${BORG_PASSPHRASE}

I make these changes locally on my laptop. I use .gitignore to specify using .* so hidden files and thus .env should not be included in the repository. But how do I know whether the containers are still booting correctly?

Secrets in Gitea’s `app.ini`

With Gitea, I’ve had a much harder time storing secrets in files. The app.ini is also written dynamically by Gitea, so the file looks a little different every time the service is restarted. After a long search, I found this issue, in which a solution for storing secrets separately was sought and found.

The only thing that seems sensible to me is storing the values INTERNAL_TOKEN and SECRET_KEY separately.
The other two properties LFS_JWT_SECRET and JWT_SECRET are automatically generated anyway and regularly overwritten.

In my configuration I only use INTERNAL_TOKEN. So I will copy it in plain text and without quotes into a hidden file (.INTERNAL_TOKEN) and make it available to the container via a Docker volume:

# Gitea's docker-compose.yml
# [...]
 volumes:
 - ./.INTERNAL_TOKEN:/run/secrets/INTERNAL_TOKEN:ro
 # [...]

In the app.ini it is now important to use the path specified in the compose file:

# Gitea's app.ini
# [...]
 server:
 INTERNAL_TOKEN = /run/secrets/INTERNAL_TOKEN
 # [...]

Not INTERNAL_TOKEN_URI=/run/secrets/INTERNAL_TOKEN as stated in the issue linked above, because this creates an error in V1.22.1 I am currently using: Unsupported URI-Scheme.

A very rough test

To check whether the changed configuration files still work, I install docker and docker-compose on my laptop. Then I try to start the containers.

# docker console log
Error: Network 'caddy-proxy' declared as external, but could not be found.

Oh right, Docker is not yet configured here. So create the network: sudo docker network create caddy-proxy and try again. The download of Gitea and its dependencies begins and the container starts - although not as I had imagined: The folder permissions within Docker are incorrect, meaning that neither Gitea nor Act-runner can access all the required files.

Nevertheless, I find the first error:

# gitea container log
WARNING: The GITEA_RUNNER_REGISTRATION_TOKEN variable is not set. Defaulting to a blank string.

I had forgotten to put the token string in quotation marks.

So for a fully functional integration environment, I have to solve at least two more problems: Folder permissions for the Main on my laptop must be set up in such a way that the Docker user also has write permissions. A simple solution for now is to append a :Z to the volumes in question and mark them as private unshared. Now the volume definition in docker-compose.yml looks like this:

# gitea/docker-compose.yml
# [...]
volumes:
 - ./gitea:/data:Z
 # [...]

I need a second file for environment variables to redirect my services to localhost. At least the service starts this way and I can see the log output. I can already spot most of the configuration errors.

I’m not sure, but I might have additional problems with the caddyserver such as certificate management, proxy settings and so on.

Create a deploy pipeline

Now it would be great if the files uploaded to the Gitea repo (and previously tested locally for functionality) would automatically find their way to my server. For this I could create another Docker volume where act-runner would then put the data stored via on:push trigger. They would then be available on my server.

Setup

If we remember my last attempts to provide artifacts on the server using act_runner, we can use a large part of that for this task as well:

# deploy-to-server.yml
# Workflow for saving the server's config repo to the local disk system

name: Upload-server-config
run-name: ${{ gitea.actor }} uploads server configuration files
on:
 push:
 branches:
 - main

jobs:
 # Deploy job
 build:
 runs-on: ubuntu-latest # this is the "label" the runner will use and map to docker target OS
 container: 
 volumes: 
 # left: where the output will end up on disk, right: volume name inside container
 - /opt/server-config:/workspace/schallbert/server-config/tmp

 steps:
 - name: --- CHECKOUT ---
 uses: actions/checkout@v3
 with: 
 path: ./tmp
 - name: --- RUN FILE CHANGE TRIGGER ---
 run: |
 cd ./tmp/automation-hooks-trigger
 touch server-config-update.txt

Strange volume errors

But the road to this point was rocky. For a long time I had only specified /server-config under volumes: on the container side and not the working directory of the runner. Then the action runs through and all commands in the #TEST section also work. But when I look on my server, the server-config folder created by Docker remains empty.

I use the following debug code under RUN FILE CHANGE TRIGGER to help me find the path errors:

# deploy-to-server.yml
# [...]
run: |
echo "hello world"
touch updated.txt
pwd
ls -al

I use echo to check whether my code is even being executed in the runner. The touch command puts the current timestamp in the updated.txt file so that I can later use this as a “hook” for further automation. pwd shows me the active path within the runner so that I can correctly map the Docker volume to the server hard drive. ls -al shows me whether the configuration files compiled in the CHECKOUT step were written correctly.

This tells me that the volume path on the “right side” was wrong. I redirect it to the active directory of the runner:

# deploy-to-server.yml
# [...]
# left: where the output will end up on disk, right: volume name inside container
- /opt/server-config:/workspace/schallbert/server-config/
# [...]

Then I got the following to read:

# gitea / act-runner console log
failed to create container: 'Error response from daemon: Duplicate mount point: /workspace/schallbert/server-config'

It seems that the runner automatically creates the “right side” of the mount point itself and therefore cannot be reassigned. Only by adding another path part, in my case /tmp - see above - I fix the error and the long-awaited folder server-config finally appears on my server 😌 with the following content:

schallbert@schallbert-ubuntu-:/opt/server-config# ls -al
total 52
drwxr-xr-x 9 root root 4096 Jul 12 15:38 .
drwxr-xr-x 9 root root 4096 Jul 11 19:56 ..
-rwxr-xr-x 1 root root 395 Jul 11 20:05 boot-after-backup.sh
drwxr-xr-x 3 root root 4096 Jul 11 20:05 borgmatic
drwxr-xr-x 2 root root 4096 Jul 11 20:05 caddy2
drwxr-xr-x 3 root root 4096 Jul 11 20:05 fail2ban
drwxr-xr-x 8 root root 4096 Jul 12 15:38 .git
drwxr-xr-x 3 root root 4096 Jul 11 20:05 .gitea
drwxr-xr-x 4 root root 4096 Jul 11 20:05 gitea
-rw-r--r-- 1 root root 312 Jul 11 20:05 .gitignore
-rw-r--r-- 1 root root 557 Jul 11 20:05 README.md
-rwxr-xr-x 1 root root 371 Jul 11 20:05 shutdown-for-backup.sh
-rw-r--r-- 1 root root 0 Jul 12 15:38 updated.txt
drwxr-xr-x 2 root root 4096 Jul 11 20:05 watchtower

Distributing the configuration on the server

Okay, that’s the first step. I now have a properly configured Git repository that shows my server configuration and can be maintained and at least rudimentarily tested from my laptop. I can also use an automatic Action to store configuration updates on the server and write an update file with a timestamp.

Now the update has to be received on the server, distributed and the affected programs and services have to be restarted. But we’ll look at this in the article Roll out server configuration.

'redir' through Caddyserver

Schallbert — Tue, 30 Jan 2024

When I moved my website from Github Pages to my own server, I had some problems and unanswered questions regarding automatic redirects. My blog migrated from the root domain to a subdomain (blog.schallbert.de).

Requirements for the redirect

Links already referencing to my blog articles under the old name, e.g. to schallbert.de/about should redirect correctly to the subdomain (no 404)
The root page under schallbert.de should be fully usable
Data and files for domain and subdomains should be independent of each other
The web servers should be able to use different technologies for different subpages

Implementation

At first I had thought that I could achieve the redirects in the DNS provider via A and CNAME entries. However, I quickly realized that a) I had no idea how DNS even works and b) the right way is via the configuration of my web server.

First attempt: redir at ‘/’

On my web server (Caddy), redirects are specified in the Caddyfile via redir. This can be done permanently (as so-called http 301) or temporarily via 302. Default is a temporary redirect.

My first thought was to leave schallbert.de unmodified and forward everything after a possible /. This would then look like this in the Caddyfile:

schallbert.de {
 # Define webserver
 root * /www/landing
 encode gzip
 file_server

 # Redir to subdomain if domain has a slash
 @redirect path_regexp /*
 redir @redirect https://blog.schallbert.de{uri}
}

As a result, Caddy simply forwarded everything to the subdomain. The catch was that the subdomain itself also contains slashes or a URI after the slash and is then forwarded yet again. Example:

schallbert.de/post -> blog.schallbert.de/post -> blog.schallbert.de/post -> blog.schallbert.de/post -> [...]

This gave me a recursive endless redirect. I completely paralyzed my server with it 😅

Second attempt: Filtering via regexp

I need better filtering so that only slashes with additional characters behind them are actually redirected and the forwarding cannot be recursive. Fortunately, I had help from a good friend, so that the following Caddyfile was created a short time later:

schallbert.de {
 # Define webserver
 root * /www/landing
 encode gzip
 file_server

 # Redir to subdomain to maintain links for blog
 @redirect path_regexp ^/[^/]+(/.*)?$
 redir @redirect https://blog.schallbert.de{uri}
}

The Regexp checks whether there is a slash after the domain URL and whether it is followed by at least one other character. The redirection now works as desired and my blog is still fully accessible under the old links.

Nevertheless, I was still not completely satisfied. Because when I created a nice “landing page” on my root domain, suddenly only the plain HTML was visible. No favicon, no images, no CSS.

A quick look at the developer options quickly brought the realization that paths of all assets that I had stored on my landing page, marked with / in the folder structure, were forwarded to the subdomain. Where they are not located, of course.

Third attempt: Regexp plus file search

Fortunately, I’m not alone with this problem, so I found the solution in Caddy’s forum:

With the not file marker I can define redirects in Caddyfile if resources on the current page cannot be found.

My solution now looks like this:

schallbert.de {
 @filenotfound {
 path_regexp ^/[^/]+(/.*)?$
 not file
 }

 # show Caddy where to find page resources
 root * /www/landing
 encode gzip

 route {
 # Redirect to subdomain if article cannot found on root
 redir @filenotfound https://blog.schallbert.de{uri}

 # else, define webserver and show page
 file_server
 }
}

Here it is important to define the page with root * /www/<pagelocation> before creating the route. Otherwise the assets will only become available after I have already forwarded them for lack of existing files.

Success!

Now my construction site page is finally displayed correctly.

Automatic Upgrades for everything!

Schallbert — Sat, 20 Jan 2024

As the title says, I would like to have security updates installed as soon as they are released. This applies to the Ubuntu on my cloud server as well as to all applications that I run in Docker. As I’m lazy, I don’t want to carry out these security updates manually.

For function updates and enhancements, however, it is still OK for me to intervene manually from time to time. Let’s start with the system upgrades.

Security updates per `unattended-upgrades`

Ubuntu supplies - as a short query reveals - a configurable tool for automated upgrades called unattended-upgrades. First, I check whether it is installed on my system:

Installation

schallbert:~# which unattended-upgrades
/usr/bin/unattended-upgrades

Great. Otherwise I’d install it with apt install unattended-upgrades.

configure unattended-upgrades

The Debian Wiki to the rescue: Config files can be found at /etc/apt/apt.conf.d.

First I make sure that automatic upgrades are active. To do that I open 20auto-upgrades and check presence of following lines:

APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Unattended-Upgrade "1";

Very well, so automatic package index updates and -upgrades are active. Now, I should set an update time and upgrade types to install. These can be configured within50unattended-upgrades, mostly by removing commented-out lines.

Upgrade sources

The first section of the configuration file is dedicated to the permitted sources for the updates. Of course, only trustworthy providers should be listed here. My configuration includes updates for my Linux distribution and security plus ESM = “Enhanced Security Maintenance” updates. It looks like this:

Unattended-Upgrade::Allowed-Origins {
 "${distro_id}:${distro_codename}";
 "${distro_id}:${distro_codename}-security";
 "${distro_id}ESMApps:${distro_codename}-apps-security";
 "${distro_id}ESM:${distro_codename}-infra-security";
};

Updates for dev releases, too?

I have the value set to auto:

Unattended-Upgrade::DevRelease "auto";

Automatic reboot

Some security updates (e.g. concerning the kernel) require the machine to be restarted. They will therefore only take effect if you have set the corresponding values in the file. WithUsers is a matter of taste, because you are force-logged off if by chance an update has just been installed that requires a restart. To practically rule this out, I have entered a time under Reboot-Time when I am certainly not working with the machine.

Unattended-Upgrade::Automatic-Reboot "true";
Unattended-Upgrade::Automatic-Reboot-WithUsers "true";
Unattended-Upgrade::Automatic-Reboot-Time "03:45";

Of course, automatic reboot has repercussions for all the programs I use. I therefore make sure in my docker-compose.yml files that always or unless-stopped is really noted in restart:.

Notifications

I haven’t switched on automatic mails or other push notifications for the time being, as I’m still in test mode and don’t really want to receive a lot more messages. As soon as I change this, there will definitely be an update here.

Function test

To see if unattended-upgrades does what it is supposed to do, I take a look at the logs under /var/log/unattended-upgrades. The file unattended-upgrades.log contains the following, for example:

# [...]
2024-01-12 06:17:59,781 INFO Starting unattended upgrades script
2024-01-12 06:17:59,781 INFO Allowed origins are: o=Ubuntu,a=jammy, o=Ubuntu,a=jammy-security, o=UbuntuESMApps,a=jammy-apps-security, o>
2024-01-12 06:17:59,781 INFO Initial blacklist:
2024-01-12 06:17:59,782 INFO Initial whitelist (not strict):
2024-01-12 06:14:33,956 INFO Packages that will be upgraded: libc-bin libc-dev-bin libc-devtools libc6 libc6-dev locales python3-twisted
2024-01-12 06:14:33,956 INFO Writing dpkg log to /var/log/unattended-upgrades/unattended-upgrades-dpkg.log
2024-01-12 06:14:54,578 INFO All upgrades installed
# [...]

However, the file unattended-upgrades-shutdown.log is still empty. I will check at a later date whether everything works here too.

Automate container updates with `Watchtower`

Now to the updates of my applications. A quick chat with a couple of admins revealed this:

It is tedious to keep all programs up to date manually
For containerized applications, there are services that solve this task centrally
There have already been bad experiences when the latest release is referenced in docker-compose.yml.
With image: <application>:latest, not all applications are stable or create problems with dependencies

One of these solutions is offered by Watchtower, which again runs in a container.

Installation

As usual I create a docker-compose.yml in a new folder at opt/watchtower:

# copied from https://containrrr.dev/watchtower/notifications/

# watchtower/docker-compose.yml
version: "3"
services:
 watchtower:
 image: containrrr/watchtower
 restart: unless-stopped
 volumes:
 - /var/run/docker.sock:/var/run/docker.sock

Watchtower needs docker.sock to pull the updates and apply them to the existing containers. Apart from that, I have done nothing else for the time being. No notifications (for the reasons mentioned above), no logs, nothing. I hope this setup stays functional and I don’t have to post an update here anytime soon.

Further steps for a validation of Watchtower’s functions, for example, can be found in a tutorial by DigitalOcean.

Procedure

Watchtower checks the installed Docker images for updates on a daily basis according to the documentation. If there is a new image release, Watchtower sends a SIGTERM signal to the containers to be updated, whereupon they shut down (source). The containers are then started up again.

Involuntary functional test

A few days after starting the container, I dialed into my server again and looked through some logs at random. I just wanted to check what was going on.

In the fail2ban log files I suddenly saw pages and pages of entries at a certain point in time, 14:42:56. More than I had seen in sum for days before. When I scrolled up to the beginning of this chain of entries, I saw that fail2ban had probably stopped for a moment.

 2024-01-26 14:42:50,817 <container_id> INFO Shutdown in progress...
 2024-01-26 14:42:50,817 <container_id> INFO Observer stop ... try to end queue 5 seconds
 2024-01-26 14:42:50,837 <daemon_id> INFO Observer stopped, 0 events remaining.
 2024-01-26 14:42:50,878 <container_id> INFO Stopping all jails
 2024-01-26 14:42:50,879 <container_id> INFO Removed logfile: '/var/log/auth.log'
 2024-01-26 14:42:50,879 <container_id> INFO Removed logfile: '/var/log/caddy2/gitea_access.log'
 2024-01-26 14:42:50,879 <container_id> INFO Removed logfile: '/var/log/caddy2/server_access.log'
 2024-01-26 14:42:51,052 <daemon_id> NOTIC [sshd] Flush ticket(s) with iptables
 2024-01-26 14:42:51,063 <container_id> INFO Jail 'sshd' stopped
 2024-01-26 14:42:51,137 <daemon_id> NOTIC [caddy-status] Flush ticket(s) with iptables-multiport
 2024-01-26 14:42:51,137 <container_id> INFO Jail 'caddy-status' stopped
 2024-01-26 14:42:51,138 <container_id> INFO Connection to database closed.
 2024-01-26 14:42:51,139 <container_id> INFO Exiting Fail2ban
 2024-01-26 14:42:56,173 <new_container_id> INFO --------------------------------------------------
 2024-01-26 14:42:56,173 <new_container_id> INFO Starting Fail2ban v1.0.2
 2024-01-26 14:42:56,173 <new_container_id> INFO Observer start...
[...]

All the log entries at the same time start at the end of the section shown above. A whole bunch of IP addresses are loaded there in preparation to be banned. I looked at the Ubuntu syslog, slightly worried:

Jan 26 14:42:55 schallbert systemd[1]: docker-<ID>.scope: Deactivated successful
Jan 26 14:42:55 schallbert systemd[1]: docker-<ID>.scope: Consumed 21:05 CPU time
Jan 26 14:42:55 schallbert dockerd[690]: time="2024-01-26T14:42:55.228607617Z" level=info msg="ignoring event" 
Jan 26 14:42:55 schallbert containerd[639]: time="2024-01-26T14:42:55.229864632Z" level=info msg="shim disconnected" 
Jan 26 14:42:55 schallbert containerd[639]: time="2024-01-26T14:42:55.230125409Z" level=warning msg="cleaning up after shim disconnected" i>
Jan 26 14:42:55 schallbert containerd[639]: time="2024-01-26T14:42:55.230145478Z" level=info msg="cleaning up dead shim"
Jan 26 14:42:55 schallbert containerd[639]: time="2024-01-26T14:42:55.240586557Z" level=warning msg="cleanup warnings time=\"2024-01-26T14:>
Jan 26 14:42:55 schallbert dockerd[690]: time="2024-01-26T14:42:55.241839655Z" level=warning msg="ShouldRestart failed
Jan 26 14:42:55 schallbert containerd[639]: time="2024-01-26T14:42:55.870912387Z" level=info msg="loading plugin \"io.containerd.event.v1.p>
Jan 26 14:42:55 schallbert containerd[639]: time="2024-01-26T14:42:55.870997689Z" level=info msg="loading plugin \"io.containerd.internal.v>
Jan 26 14:42:55 schallbert containerd[639]: time="2024-01-26T14:42:55.871007368Z" level=info msg="loading plugin \"io.containerd.ttrpc.v1.t>
Jan 26 14:42:55 schallbert containerd[639]: time="2024-01-26T14:42:55.871150831Z" level=info msg="starting signal loop" namespace=moby path>

Hm, cleaning up dead shim. That sounds ominous. A quick search shows that the shim forms an intermediate layer between the container manager and the container itself and forwards the container’s input and output. Its runtime is linked to that of the container.

In other words, I saw a normal shutdown followed by a restart. Fortunately, no break-in! All that remains is to find out the reason for the restart of Fail2ban. This reminds me that I don’t even know when Watchtower runs the updates…

So I quickly looked in the Watchtower logs with docker container logs <container_id> and behold:

time="2024-01-26T14:42:47Z" level=info msg="Found new lscr.io/linuxserver/fail2ban:latest image (9bda077d765e)"
time="2024-01-26T14:42:50Z" level=info msg="Stopping /fail2ban (e0317a105d53) with SIGTERM"
time="2024-01-26T14:42:55Z" level=info msg="Creating /fail2ban"
time="2024-01-26T14:42:55Z" level=info msg="Session done" Failed=0 Scanned=6 Updated=1 notify=no

Now I have proof: Watchtower does what it is supposed to do. It upgrades containers to a new version as soon as updates become available. Great!

PS: Since I haven’t configured Watchtower very much, I assume from the timestamps in the logs that the updates are run daily at the time Watchtower is started. I wonder whether I should switch to a certain version of the image instead of using latest, though.

Better security for my server

Schallbert — Fri, 12 Jan 2024

Motivation

OK. I have my server setup, created an automatic build pipeline for my Blog and my website is displaying the Blog as it should. Still, I’m not completely done yet. Because I want to do more for server-side security than deactivating password logins.

And I don’t have backups. That’s never good, so let’s get something done about that.

Locking away unwanted guests

In my SSH-log of my machine’s root system, I get a lot of failed authentications:

[...]
Jan 11 00:19:29 sshd[93842]: Invalid user admin from 41.207.248.204 port 37194
Jan 11 00:19:29 sshd[93842]: pam_unix(sshd:auth): check pass; user unknown
Jan 11 00:19:29 sshd[93842]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ru>
Jan 11 00:19:31 sshd[93842]: Failed password for invalid user admin from 41.207.248.204 port 37194 ssh2
Jan 11 00:19:32 sshd[93842]: Connection closed by invalid user admin 41.207.248.204 port 37194 [preauth]
Jan 11 00:20:14 sshd[93886]: Invalid user svn from 84.108.40.27 port 44968
Jan 11 00:20:14 sshd[93886]: pam_unix(sshd:auth): check pass; user unknown
Jan 11 00:20:14 sshd[93886]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ru>
Jan 11 00:20:16 sshd[93886]: Failed password for invalid user svn from 84.108.40.27 port 44968 ssh2
Jan 11 00:20:16 sshd[93886]: Received disconnect from 84.108.40.27 port 44968:11: Bye Bye [preauth]
Jan 11 00:20:16 sshd[93886]: Disconnected from invalid user svn 84.108.40.27 port 44968 [preauth]
[...]

If I didn’t know that this is typical “internet noise”, I’d be nervous. Isn’t it coming close to thieves trying different keys at our front doors every few seconds? So what can we do about it? Ban them.

fail2ban

That’s exactly what fail2ban can do for me.

 __ _ _ ___ _ 
 / _|__ _(_) |_ ) |__ __ _ _ _ 
 | _/ _` | | |/ /| '_ \/ _` | ' \ 
 |_| \__,_|_|_/___|_.__/\__,_|_||_|
 v1.1.0.dev1 20??/??/??

In short, fail2ban scans access/auth logs¹ for IP-addresses causing multiple failed authentications, and bans them when surpassing a user-defined threshold within a defined period of time.

How does fail2ban work? It modifies iptables, thus accesses packet filter rules (buzzword Firewall) on the network layer. So incoming requests of already blocked IP-addresses will not even reach my applications².

install fail2ban

fail2ban seems to be close to an industry standard for blocking unwanted access requests on Linux. Every hobbyist admin I know is using it.

For installation and configuration of fail2ban there’s a ton of guides out there plus the (well-written) on inside its Github-repository. I won’t be pressing the point here.

I chose an installation as Docker container to get all dependencies auto-delivered as well. I’m using the distribution by linuxserver and write the following docker-compose.yml to have it deployed:

# /fail2ban/docker-compose.yml
version: "2.1"
services:
 fail2ban:
 image: lscr.io/linuxserver/fail2ban:latest
 container_name: fail2ban
 cap_add:
 - NET_ADMIN
 - NET_RAW
 network_mode: host
 environment:
 - PUID=1000
 - PGID=1000
 - TZ=Etc/UTC
 - VERBOSITY=-vv #optional
 volumes:
 - ./config:/config
 - /var/log/auth.log:/var/log/auth.log:ro # host ssh
 - /var/log/caddy2:/var/log/caddy2:ro  # gitea via caddy, caddyserver
 restart: unless-stopped

What I add to add here: fail2ban requires the Access logs as Volume (above added with :ro as read-only volumes). A lot of filter rules come predefined in its config folder, I just had to add a modified jail.local inspiration: linuxserver/fail2ban-confs and add a rule in filter.d for caddy pulled from muetsch.io to complete my door watch.

fail2ban example

This is how a fail2ban-log looks like for my SSH-Daemon (sshd):

# schallbert:/opt/fail2ban/config/log/fail2ban# grep "220.124.89.47" fail2ban.log 
 2024-01-11 17:12:20,600 7FBB3630BB38 INFO [sshd] Found 220.124.89.47 - 2024-01-11 17:12:20
 2024-01-11 17:12:23,003 7FBB3630BB38 INFO [sshd] Found 220.124.89.47 - 2024-01-11 17:12:22
 2024-01-11 17:12:25,205 7FBB3630BB38 INFO [sshd] Found 220.124.89.47 - 2024-01-11 17:12:24
 2024-01-11 17:12:27,206 7FBB3630BB38 INFO [sshd] Found 220.124.89.47 - 2024-01-11 17:12:26
 2024-01-11 17:12:32,610 7FBB3630BB38 INFO [sshd] Found 220.124.89.47 - 2024-01-11 17:12:32
 2024-01-11 17:12:33,045 7FBB36104B38 NOTIC [sshd] Ban 220.124.89.47

kbye!

What does not (yet) work: Gitea & fail2ban

Also on my gitea instance, I get quite some authentication requests that I’d like to block. Unfortunately, I cannot get Gitea to put the registered accesses into a log file. I see them in the console only. On the other hand, the shell access to Gitea is already secured as it is channeled through sshd that also guarantees remote access to my server.

What is missing is a ban for failed web login requests.

I thought I had configured app.ini to get access logs written to file:

# gitea/conf/app.ini
#[...]
[log]
MODE = file
LEVEL = warn
ROOT_PATH = /data/gitea/log
ENABLE_ACCESS_LOGS = true
ENABLE_SSH_LOG = true
logger.access.MODE = access-file

[log.access-file]
MODE = file
ACCESS = file
LEVEL = info
FILE_NAME = access.log
[...]

I have both activated the logs with ENABLE_ACCESS_LOGS and configured log-to-file. access.log is being generated but no authentication tries are being written to it. Maybe caddy’s reverse proxy snitches them away before they reach Gitea? I’m sure I’ll find out at a later point in time.

At some point I asked myself whether fail2ban actually works on the iptables of my server or only operates within the container. Luckily, I was not the first with this question (stackoverflow) and found the proposed solution quite charming:

Block any IP in the Docker container (docker exec -it fail2ban sh): fail2ban-client set sshd banip 111.111.111.111
Check the iptables to see if this IP appears there: iptables -n -L --line-numbers
In my case: enjoy, because it is there: 1 REJECT all -- 111.111.111.111
Unblock the IP again fail2ban-client set sshd unbanip 111.111.111.111

Protection against brute-force attacks via ssh as described above
Protection against overload using a Rate Limiter
Protection against attacks on the APIs (404/403 attacks) using caddy-status configuration for fail2ban, logs in JSON

# /config/fail2ban/filter.d/caddy-status.conf

# this regex works for caddy with json-style logs
failregex = "client_ip":"<HOST>"(.*)"status":(400|401|403|404|500)
datepattern = \d+
ignoreregex =

Regular backups

I want to be able to do disaster recovery. Unforeseen events impacting my server like incompatible updates, machine downtime or security breaches should not keep me from spawning and configuring a new, healthy instance in no time.

I’m not willing to create backups manually, copy and compress folders, finally downloading to a backup location via SCP / SFTP. This should take place automatically, and, ideally, without cost. A quick search reveals two suitable open-source tools: borgmatic and restic.

I randomly chose Borgmatic.

install Borgmatic with Docker

As for all other components so far, I want Borgmatic to run in Docker environment. Luckily there are ready-made solutions that need just a little configuration.

This is how my docker-compose file looks like:

# /borgmatic/docker-compose.yml
version: '3'
services:
 borgmatic:
 image: ghcr.io/borgmatic-collective/borgmatic
 container_name: borgmatic
 volumes:
 - ${VOLUME_SOURCE}:/mnt/source:ro  # backup source
 - ${VOLUME_TARGET}:/mnt/repository  # backup target
 - ${VOLUME_ETC_BORGMATIC}:/etc/borgmatic.d/  # borgmatic config file(s) + crontab.txt
 - ${VOLUME_BORG_CONFIG}:/root/.config/borg  # config and keyfiles
 - ${VOLUME_SSH}:/root/.ssh  # ssh key for remote repositories
 - ${VOLUME_BORG_CACHE}:/root/.cache/borg  # checksums used for deduplication
 # - /var/run/docker.sock:/var/run/docker.sock # add docker sock so borgmatic can start/stop containers to be backupped
 environment:
 - TZ=${TZ}
 - BORG_PASSPHRASE=${BORG_PASSPHRASE}
 restart: always

Inspirations were the related docs on Github. You can see that Borgmatic requires a lot of volumes to work properly, most importantly backup source and target location.

configure Borgmatic

All concrete data behind ${} are summarized in an .env environment variable file. This helps me avoid publishing passphrases by accident. Of course I have noted down the passphrase on paper as “backup” of the backup.

I then modify config.yml in borgmatic.d/ slightly, adding backup source, target, count, and cycle. That’s all, I’m ready for a first test.

testing Borgmatic backup

docker exec borgmatic bash -c \
"cd && borgmatic --stats -v 1 --files 2>&1"

This command has borgmatic create a backup for me through the Docker container.

I got an error message right away, saying repository does not exist. A look into the docs reveals that I have to manually create the backup target repository.

The following command does this, creating an empty, encrypted repo:

docker exec borgmatic bash -c \
"borgmatic init --encryption repokey-blake2"

Now I retry manual backup creation which throws no error .

Ein Backup only is a Backup…

…when it is successfully applied, a friend of mine said.

Right he is. Still, I don’t dare replacing my working server configuration with a backup. As a compromise, I go halfway by creating a docker-compose.restore.yml following the online guide. For additional help, I read modem7’s related blog post.

For my test purposes, I run docker-compose.restore.yml in the container’s shell and enter the following commands:

mkdir backuprestoremount
borg mount /mnt/repository /backuprestoremount

mkdir backuprestore
borgmatic extract --archive latest --destination /backuprestore

Here, I create the folder /backuprestoremount and have it point to my borg backup. Then, I extract the backup into /backuprestore.

Let’s do a sanity check:

# cd /mnt/backuprestore
# /restore/mnt/source ls
borgmatic caddy2 containerd fail2ban gitea hostedtoolcache watchtower

Yeah! The backup contains all applications along with their files. I stop at this point as I’m too lazy and cowardly to overwrite my working setup as mentioned above.

What (yet) doesn’t work: Borgmatic & docker-compose down

Borgmatic offers a simple method to run pre- and post-backup actions. As I want to maintain consistency within my containers, I want them to be shutdown before and restarted after the backup run. So I wrote a script, hooked it into the config and… nothing.

Easy to follow, this can only work directly if Borgmatic ran on bare metal. Docker’s encapsulation doesn’t let borgmatic run shell scripts outside its container. I thought I could circumvent this problem by including var/run/docker.sock in the docker-compose file but without success.

I’m sure this can be solved somehow. For the time being, I have more important problems to solve and thus I’ll live with the assumption “there are no data inconsistencies³.

The SSH-Daemon of my server puts its authentication logs to /var/log/auth.log, but could also be named access.log or similar in other applications. Services like caddy or gitea also create auth log files. ↩︎
However, what I yet do not understand is how fail2ban is able to access the iptables from within the container. I thought a positive side effect of containerization is encapsulation. Or is that only complete with “rootless”-Containers? ↩︎
Background: I’m currently the only one contributing to my gitea instance. My website is static. The webserver caddy only handles changing files once pushed to Gitea. I have scheduled application updates per watchtower and unattended-upgrades (see my follow-up post) at night so they won’t interfere with borgmatic runs. Just the logs for fail2ban are still written independently. I accept the risk of data loss here because packet filtering is subject to change anyways. ↩︎

Jekyll build: broken relative links

Schallbert — Fri, 29 Dec 2023

Problem

Now, after eternal trial and error, I am finally able to provide my Caddy server with the build files via Docker-Volume. Unfortunately, the page looks like this:

Analysis

At first I suspected the web server. But when I call up the website in my browser¹ as a test, I see the same behavior. Strange.

Obviously all stylesheets are missing. But if you look closely, images and other media have not been loaded either. Only their descriptions. Then I click wildly on a few links to articles and subpages. I notice that the paths look different than I would have expected:

# --- Expectation ---
# Image
/assets/images/test.jpg
# Post
/this-is-a-post

Instead I see:

# --- Observation ---
# Image
/pages/<username>/<repositoryname>/assets/images/test.jpg
# Post
/pages/<username>/<repositoryname>/this-is-a-post

This is also shown by the browser’s developer options activated with F12 (see image above): all sources integrated via a relative link cannot be loaded. It is generally a clear recommendation to have the developer options activated in the event of problems with the display of websites.

It is somehow obvious that my Jekyll configuration file _config.yml or my translation header i10.yml contain errors. Fortunately, there are already excellent help pages.

However, a test quickly shows that I cannot leave the attribute repository empty, where the build <username>/<repositoryname> emanates from:

# bundle exec jekyll serve
Generating... 
 Jekyll Feed: Generating feed for posts
 Liquid Exception: No repo name found. 
 Specify using PAGES_REPO_NWO environment variables, 'repository' in your configuration, 
 or set up an 'origin' git remote pointing to your github.com repository. in /_layouts/default.html
 ERROR: YOUR SITE COULD NOT BE BUILT

And, heck, where does the /pages suddenly come from?

Solution

As /pages is added to paths I suspect Jekyll build options and Github-Pages Gem to be the culprits.

Of course I’m not alone with this problem, so I found the solution documented.

I just change JEKYLL_ENV (build environment) in the action script from production to development, like it is normally done for building on a local machine.

# workflows/jekyll-build-action.yml
env:
 JEKYLL_ENV: development # I had "production" here before

And what can I say: as soon as you do it right, it works!

Displaying the website offline works as follows: Drag the built files (the _site folder) to your own computer and open the top-level index.html with the browser. ↩︎

Paginator broken, Favorite icon placed

Schallbert — Sun, 13 Jun 2021

Adding a “tab” icon - favicon

I was worrying about a strange error message that Jekyll threw at any page reload since some pushes: [2021-06-13 22:07:43] ERROR '/favicon.ico' not found. Some online research revealed that this is neither a Jekyll, nor a Theme error, but a missing file error. See here how I implemented the icon for this page.

Pagination, again

I found out that somehow, the pagination doesn’t even work on my page while the layouting assumes that it’s preparing the pages. Now, if I try to access the page by pressing the “previous” button, I get a 404 error. Till now, I wasn’t able to fix this because the pagination fails silently and I do not know where. Switching it off completely for now.

Jekyll: images

Schallbert — Mon, 07 Jun 2021

The trick to make images show both locally and on Github pages

When I added some images to my content, all was fine until I pushed to GitHub. Most images were gone and I couldn’t figure out why. Some searching revealed that Github is case sensitive and some of my image paths were still mixed case.

Changing folder names in git

Well, it still didn’t work because git didn’t recognize my renaming. I first had to create the renamed folder in another subdirectory, use git mv folder/tArGet* target to get this to a temporary place, delete tArGet and move target to the place where tArGet was before. Only this fixed the issue and now github and my locals are in sync again, luckily.

This site: Table of Contents

Schallbert — Sun, 06 Jun 2021

There is an easy solution for making the page’s table of contents stick: add toc_sticky: true to the defaults or the specific page’s front matter. How I found the keyword? By searching “toc” in the whole solution and some scrolling through the layout scetches. I modified the original site navigation in the left sidebar and exchanged it with a dummy implementation of a gallery / audio / video contents display.

ToC on the left

To understand how to create a sticky ToC on the left, please continue reading this chapter of the mySite project.

Author’s profile

I decided I only want my author_profile to be shown on pages like about, legal info and not on the typical content pages. So I wrote author_profile: false for the pages I didn’t want it to show up - with no effect. Why? I had set it in the config.yml’s type defaults but it should actually go into the frontmatter of the individual files.

Jekyll `-incremental` option

Schallbert — Thu, 03 Jun 2021

The images

I worked on my assets/images, they tend to be somewhat “big” so that page load times might be a reason to worry about. I’m using paint.net to shrink them to a size I think I can afford and save them as .jpg with compression max’ed out and some compromises on image quality. This way, most of my pictures take less than 10% or the original size in kB.

MinimalMistakes theme - Page design

Schallbert — Wed, 02 Jun 2021

The landing page

Yeah, by changing the layout type to single I was able to get a header image as type overlay in! The buttons are displayed within the image now. The image changes with display size and orientation automatically due to the visual designs defined in the _sass folder.

landing page image

Ha, found it! I was freaking out; my landing page now just wouldn’t show any image, no matter how hard I tried modifying the home.md file… Reason behind was that there was a stock index.html file that came with the template and that seems to have priority. No I just added my changes in there - works perfectly fine! By the way, the overlay image’s height is automatically selected depending on the overlay text’s contents/ height.

Easy solution: type toc: true in the front matter. Unfortunately, eventually you will loose this information as you scroll down because it won’t stick to your scrolled view.
Hard solution: Add page navigation to left sidebar. How? See here!