Crowdsec with SWAG on a Top-Level Domain
Gotchas appear when using swag-crowdsec with a TLD
Background
I use Linuxserver.io’s (LSIO) dockerized nginx reverse-proxy solution, SWAG, as the point of ingress for public-facing services in my homelab. In addition to being easy to configure LSIO containers can be sideloaded with Docker Mods that can provide additional functionality to the main service. Some of these are generic but most are specific to the container they are running on.
One of thes docker mods specific to SWAG installs and configures an nginx bouncer for Crowdsec, a community-driven quasi WAF. This mod, along with LSIO’s guide for setting up a full Crowdsec solution, is a large facet of my homelab’s defensive design.
The Setup
Crowdsec reads access.log from nginx and then uses leaky buckets along with Crowdsec scenarios (patterns) to detect bad behavior from IPs accessing my web server. When a scenario “overflows” a bucket (detected X number of times within Y minutes, simplified) the IP is then banned (added to a list in Crowdsec) for a period of time. The bouncer becomes relevant here: on each request nginx is supposed to check the banned list of IPs and return a “banned” page with 403 if the $remote_addr of the request matches it, regardless of what the IP is requesting.
The banning behavior was working as expected when a banned IP attempted to access any of my services behind an existing subdomain (proxy-conf in SWAG terms) but was not occurring when the request was to a top-level domain or non-existing subdomain.
1
2
3
4
5
realSubdomainA.myTLD.com <-- CS intercepts, banned response
realSubdomainB.myTLD.com/something <-- CS intercepts, banned response
notASubdomainC.myTLD.com <-- not intercepted! Returns 307
myTLD.com <-- not intercepted! Returns 307
myTLD.com/something <-- not intercepted! Returns 307
This was perplexing…why was it only working sometimes? And basically only not working when I needed it most! IE when an attacker is probing for .env files and the such.
Current Nginx Configuration
Let’s take a look at my abridged configuration…this is foreshadowing. The problem is present below but, trickily, is not immediately obvious as it is a valid config.
nginx.confis setup to includehttp.d, where the crowdsec bouncer is configuredsite-confs/default.confis largely the same as SWAG sets it up, but with these changes
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
# ...
server {
listen 443 ssl default_server;
listen [::]:443 ssl default_server;
server_name _;
# ...
# modified so that
# any requests to TLDs or non-existent subdomains
# redirect to my portfolio static site
location / {
return 307 https://myStaticSite.com;
}
# ...
}
include /config/nginx/proxy-confs/*.subdomain.conf;
# ...
The non-intercepting behavior seems suspiciously consistent with what the redirect is supposed to do…
But let me tell you it was a rabbit hole to get to this point. I debugged the crowdsec side of things thoroughly thinking maybe it was an issue with scenario buckets not overflowing…or the bouncer not getting IP ban decisions in time. It wasn’t until I turned on debug-level logging in nginx with error_log ... debug;, and was able to see the bouncer not logging anything, that it clued me in that it might be my nginx directive instead.
Well There’s Your Problem
Once I got some feedback from the helpful devs on the LSIO discord server, who suspected it might be an order-of-operations issue, I really started digging into what the nginx bouncer did and how nginx directives are processed.
Nginx procceses requests in a sequences of phases. If an earlier phase ends execution or causes a response to be sent then all subsequent phases are not run.
The return directive in the location block is part of the rewrite phase, but the nginx bouncer runs in an access_by_lua_block handler that is part of the access phase. Critically, the rewrite phase runs before the access phase.
So, since the location / block was catching all these non-existing routes it was returning a response using a phase that occurred before the nginx bouncer ever ran!
The Solution
So we need to get rid of the return directive and use something that lets the access phase run first.
The Easy Way
Well, we already know that proxy_pass works since it’s what we’ve been using with LSIO. Yep, random helpful netizen on stackoverflow confirms proxy_pass is part of the content phase which does run after access phase.
So let’s create a simple, inline server and park our return directive behind a proxy_pass to it.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
server {
# prevent double logging to access.log
access_log off;
listen 127.0.0.1:11111;
return 307 https://myStaticSite.com;
}
server {
listen 443 ssl default_server;
listen [::]:443 ssl default_server;
server_name _;
# ...
location / {
proxy_pass http://127.0.0.1:11111;
}
Boom! Problem solved. Crowdsec is intercepting all requests now. Hooray!
The Hard Way
What’s that? You want to be clever? Don’t want to create new server blocks? Do it all in the same location block? Fine. But you asked for it.
Let’s revisit that lua handler provided by the crowdsec bouncer. If we do some digging we can determine that re-defining the same handler within a nested directive (http -> server -> location) will cause the most specific handler to be used. Essentially, we can override crowdsec’s handler with our own.
So let’s copy-paste their handle into our location block and add some code to do the redirect after cs runs.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
server {
# ...
location / {
access_by_lua_block {
local cs = require "crowdsec"
if ngx.var.unix == "1" then
ngx.log(ngx.ERR, "[Crowdsec] Unix socket request ignoring...")
else
cs.Allow(ngx.var.remote_addr)
# custom redirect after CS
ngx.req.set_method(ngx.HTTP_GET)
return ngx.redirect("https://myStaticSite.com", 307)
end
}
}
}
But wait..this isn’t working? If we add debug logging after cs.Allow we can see ngx.redirect never runs! What gives?
Crowdsec’s bouncer code is running ngx.exit(ngx.DECLINED) when the IP is not blocked. This causes the handler to exit early and not run our code after cs.Allow.
So lets replace that exit with a return…
1
2
3
4
end
- ngx.exit(ngx.DECLINED)
+ return
end
Now our redirect is running! Hooray!
But now you need to deal with patching this file any time the docker-mod is updated or the container is recreated. Manging this dependency would be a PITA but it is possible. I’ve commented on an issue on the bouncer to make this easier but it’s still not as end-user friendly as the proxy_pass solution. Good luck with that.