Add rate limiter to Gitea

3 minute read

Here, I’m providing a step-by-step guide to solving my Gitea crashes problem. I’m incorporating the experience I gained from using a rate limiter for my blog with the goal of making the application more robust and protected against undirected denial of service attacks.

Step 1: Specify the log source for the rate limiter

Gitea naturally generates logs itself, theoretically down to the access level by external clients. However, in the past, I haven’t been able to export Gitea’s access logs from Docker. Therefore, we now instruct Caddyserver, which is connected as a reverse proxy, to create logs on behalf of Gitea. Access to the Gitea web interface will then appear in these logs.

In the Caddyfile, the log module looks completely unspectacular.

# /caddy/Caddyfile

#[...]
log {
output file /log/gitea/access.log
}

Exclude internal traffic from the log

Now I look at the logs and see entries that I don’t want. For example, act_runner generates a Fetch Task POST to Gitea every two seconds and a GET request every ten seconds with the same target as a Health check. These don’t even need to appear in the log for me. My first thought here was to simply not log requests from internal IP addresses. However, since Caddy acts as a reverse proxy, all IP addresses run “internally” without exception.

So I have to recognize act_runner-specific logs differently:

# /caddy/Caddyfile

#[...]
git.schallbert.de {

  reverse_proxy * http://gitea:3000

  # Enable logging for fail2ban, don't log for runner
  log_skip /api/actions*
  log {

  output file /log/gitea/access.log
  }
}

The log_skip directive now instructs caddy to no longer create logs for access to /api/actions*.

Reverse Proxy: Displaying Remote IPs

However, I still have a problem: If all requests in the log come from an internal IP address, how am I supposed to block “bad” requests? A web search shows that this is a common problem for reverse proxies in Docker. Unfortunately, many solutions cannot be applied to this situation because they use different server software like nginx or have different services running behind their proxies than Gitea.

But it’s actually quite simple: Just insert a line in the correct place in the Caddyfile, and the remote IP addresses come in unchanged.

# /caddy/Caddyfile

#[...]
git.schallbert.de {
  reverse_proxy * http://gitea:3000 {
    trusted_proxies 172.16.0.0/12 # Docker-internal netwock traffic runs with these IPs
  }
#[...]

The trusted_proxies directive tells Caddy that the network address range provided by Docker can be trusted. Now the actual source IP address is shown instead of the container’s internal interface address. This is exactly what I need to be able to analyze the addresses later with fail2ban.

Step 2: Determine the Use Case

Let’s look at how many HTTP 200 ok requests are coming in the edge case between normal use and “abuse.” To do this, I surf around Gitea and click on a bunch of things that I would never normally do at that speed as a human. Then I analyze the logs.

This gives us some initial benchmarks for the rate limiter.

# rate limiter tests
findtime = 10s
maxretry = 10
bantime = 6h

Step 3: Configure Fail2ban

Next, we configure the filter and jail file of fail2ban to analyze the logs defined above.

Filter

Here, I’m re-using the same filter for my rate limiter from the previous article. We’re only interested in successful requests that we count within a time window.

Jail

I’m using the values from the rate limiter tests 1:1 in the jail file. I’m referring to caddy-ratelimit (link above) as the filter. It is essential to select the DOCKER-USER chain as the requests are routed via Docker’s virtual network.

# /fail2ban/config/fail2ban/jail.local
# [...]

[gitea-ratelimit]
enabled     = true
chain       = DOCKER-USER
port        = http,https
filter      = caddy-ratelimit
logpath     = /var/log/caddy2/gitea/access.log
findtime    = 10s
maxretry    = 10
bantime     = 6h

I reboot fail2ban to activate the protection.

Step 4: Test the rate limiter

I proceed exactly as in my article fail2ban with caddy and get the same result. It works!

Continue reading