Gitea act_runner: Jump-start issues

2 minute read

Problem statement

Since I run this page myself I experience problems starting Gitea and act_runner. Sometimes, the runner won’t start. It exits with status -1 and so my workflows wouldn’t run the website build, verification and deploy tasks.

The error message is always the same:

[...]
runner-1  | time="2024-08-26T10:09:46Z" level=info msg="Starting runner daemon"
runner-1  | time="2024-08-26T10:09:46Z" level=error msg="fail to invoke Declare" error="unavailable: 502 Bad Gateway"
runner-1  | Error: unavailable: 502 Bad Gateway
runner-1 exited with code 1
gitea     | 2024/08/26 10:09:46 cmd/web.go:242:runWeb() [I] Starting Gitea on PID: 16

Ok, the runner cannot hook onto Gitea. When I restart all services via docker restart, the act_runner container tells me it would be missing a network with ID <long hexcode ID>

Interim corrective action

I didn’t find time for resolving this issue. So I just ran docker compose twice:

# schallbert server-console
docker compose -f /path/to/giteas/composefile up -d

The second time, act_runner would start successfully (because Gitea is ready):

gitea     | 2024/08/26 10:38:17 routers/init.go:116:InitWebInstalled() [I] Git version: 2.45.2 (home: /data/gitea/home)
runner-1  | time="2024-08-26T10:38:22Z" level=info msg="Starting runner daemon"
runner-1  | time="2024-08-26T10:38:22Z" level=info msg="runner: action-runner, with version: v0.2.10, with labels: [ubuntu-latest], declare successfully"
runner-1 exited with code 0
runner-1  | time="2024-08-26T10:40:32Z" level=info msg="Started runner daemon"

Not very satisfying to do things twice. Plus, I did this manually. So why is that?

Root cause analysis

First I thought it was due to Caddy not being ready as it runs the reverse proxy, connecting Gitea to the internet. In reality, Gitea is the culprit: At the time, docker starts act_runner, its startup procedure wouldn’t be complete. So I try to find out how to manage dependencies in docker.

Solution: Reflecting dependencies

First, I try linking gitea to the runner in the docker compose file.

depends_on

# gitea/docker-compose.yml
  ## service: runner
  ## [...]
    depends_on:
      gitea:
        condition: service_started

Well, still act_runner seems to be starting early. Docker just makes sure it has started the container, and does not wait for it to return a healthy signal.

Healthcheck

So I have to make sure that gitea’s startup procedure is complete. For this, docker provides the condition service_healthy. I adjust the runner configuration like so:

# gitea/docker-compose.yml
  ## service: runner
  ## [...]
    depends_on:
        gitea:
            condition: service_healthy # required so runner can attach to gitea
            restart: true

The service_healthy qualifier is returned by the healthcheck function. It consists of a so-called test and some environment parameters that define intervall, number of retries, delay and timeout conditions for checking service ready.

For the test I chose a simple command, assuming that Gitea (and, implicitly, Caddy too) would be healthy once it is able to respond to a GET request from curl.

# gitea/docker-compose.yml
## service: gitea
## [...]
    healthcheck:
        test: ["CMD", "curl", "-f", "https://git.schallbert.de/"] # checks if gitea is available
        interval: 10s
        retries: 3
        start_period: 30s
        timeout: 10s

With these lines, docker checks if gitea is “healthy” and only then starts act_runner. This permanently solved the problem and both Gitea and act_runner are stable.

You may also enjoy