[READ-ONLY] Mirror of https://github.com/FoxxMD/blog.
blog.foxxmd.dev
fix: More cleanup
+18
-15
+18
-15
_posts/2025-02-12-swag-crowdsec-tld.md
+18
-15
_posts/2025-02-12-swag-crowdsec-tld.md
···
11
11
12
12
## Background
13
13
14
-
I use [Linuxserver.io's (LSIO)](https://www.linuxserver.io/) dockerized nginx reverse-proxy solution, [SWAG](https://docs.linuxserver.io/general/swag/), as the point of ingress for public-facing services in my homelab. In addition to being easy to configure LSIO containers can be sideloaded with ["Docker Mods"](https://www.linuxserver.io/blog/2019-09-14-customizing-our-containers#docker-mods) that can provide additional functionality to the main service. Some of these are [generic](https://mods.linuxserver.io/?mod=universal) but most are specific to the container they are running on.
14
+
I use [Linuxserver.io's (LSIO)](https://www.linuxserver.io/) dockerized nginx reverse-proxy solution, [SWAG](https://docs.linuxserver.io/general/swag/), as the point of ingress for public-facing services in my homelab. In addition to being easy to configure LSIO containers can be sideloaded with [Docker Mods](https://www.linuxserver.io/blog/2019-09-14-customizing-our-containers#docker-mods) that can provide additional functionality to the main service. Some of these are [generic](https://mods.linuxserver.io/?mod=universal) but most are specific to the container they are running on.
15
15
16
16
One of thes docker mods specific to SWAG installs and configures an [nginx bouncer](https://github.com/linuxserver/docker-mods/tree/swag-crowdsec) for [Crowdsec](https://crowdsec.net/), a community-driven quasi [WAF](https://www.cloudflare.com/learning/ddos/glossary/web-application-firewall-waf/). This mod, along with [LSIO's guide for setting up a full Crowdsec solution](https://www.linuxserver.io/blog/blocking-malicious-connections-with-crowdsec-and-swag), is a large facet of my homelab's defensive design.
17
17
···
33
33
34
34
### Current Nginx Configuration
35
35
36
-
Let's take a look at my abridged configuration...this is foreshadowing. The problem is present below but trickily not immediately obvious as it is a valid config.
36
+
Let's take a look at my abridged configuration...this is foreshadowing. The problem is present below but, trickily, is not immediately obvious as it is a valid config.
37
37
38
-
* `nginx.conf` is setup to include `http.d` where the crowdsec bouncer is configured
39
-
* `site-confs/default.conf` is largely the same as SWAG sets it up with these changes
38
+
* `nginx.conf` is setup to include `http.d`, where the crowdsec bouncer is configured
39
+
* `site-confs/default.conf` is largely the same as SWAG sets it up, but with these changes
40
40
41
41
```nginx
42
42
# ...
···
66
66
67
67
The non-intercepting behavior seems suspiciously consistent with what the redirect is supposed to do...
68
68
69
-
But let me tell you it was a rabbit hole to get to this point. I debugged the crowdsec side of things thoroughly thinking maybe it was an issue with scenario buckets not overflowing...or the bouncer not getting IP ban decisions in time. It wasn't until I turned on debug-level logging in nginx with `error_log ... debug;` that I was able to see the bouncer not logging anything that it clued me in that it might be my nginx directive instead.
69
+
But let me tell you it was a rabbit hole to get to this point. I debugged the crowdsec side of things thoroughly thinking maybe it was an issue with scenario buckets not overflowing...or the bouncer not getting IP ban decisions in time. It wasn't until I turned on debug-level logging in nginx with `error_log ... debug;`, and was able to see the bouncer not logging anything, that it clued me in that it might be my nginx directive instead.
70
70
71
71
## Well There's Your Problem
72
72
73
-
It wasn't until I got some feedback from the helpful devs on the LSIO discord server, who suspected it might be an order-of-operations issue, that I really started digging into what the nginx bouncer did and how nginx directives are processed.
73
+
Once I got some feedback from the helpful devs on the LSIO discord server, who suspected it might be an order-of-operations issue, I really started digging into what the nginx bouncer did and how nginx directives are processed.
74
74
75
-
Nginx procceses request in a [sequences of **phases**](https://nginx.org/en/docs/dev/development_guide.html#http_phases). If an earlier phase ends execution or causes a response to sent then all subsequent phases are not run.
75
+
Nginx procceses requests in a [sequences of **phases**](https://nginx.org/en/docs/dev/development_guide.html#http_phases). If an earlier phase ends execution or causes a response to be sent then all subsequent phases are not run.
76
76
77
-
The `return` directive in the `location` block is part of the [**rewrite phase**](https://nginx.org/en/docs/http/ngx_http_rewrite_module.html) but the nginx bouncer runs in an [`access_by_lua_block`](https://github.com/openresty/lua-nginx-module?tab=readme-ov-file#access_by_lua_block) [handler](https://github.com/crowdsecurity/cs-nginx-bouncer/blob/main/nginx/crowdsec_nginx.conf#L19) that is part of the **access phase**. Critically, the rewrite phase runs before the access phase.
77
+
The `return` directive in the `location` block is part of the [**rewrite phase**](https://nginx.org/en/docs/http/ngx_http_rewrite_module.html), but the nginx bouncer runs in an [`access_by_lua_block`](https://github.com/openresty/lua-nginx-module?tab=readme-ov-file#access_by_lua_block) [handler](https://github.com/crowdsecurity/cs-nginx-bouncer/blob/main/nginx/crowdsec_nginx.conf#L19) that is part of the **access phase**. Critically, **the rewrite phase runs before the access phase.**
78
78
79
-
So, since the `location /` block was catching all these non-existing routes it was returning a response using a phase that occurred before the nginx bouncer ever run!
79
+
So, since the `location /` block was catching all these non-existing routes it was returning a response using a phase that occurred before the nginx bouncer ever ran!
80
80
81
81
## The Solution
82
82
···
84
84
85
85
### The Easy Way
86
86
87
-
Well, we already know that `proxy_pass` works since it's what we've been using with LSIO. Yep, random helpful internetizen on stackoverflow confirms[`proxy_pass` is part of the **content phase**](https://stackoverflow.com/a/78595091/1469797) which does run after access phase.
87
+
Well, we already know that `proxy_pass` works since it's what we've been using with LSIO. Yep, random helpful netizen on stackoverflow confirms [`proxy_pass` is part of the **content phase**](https://stackoverflow.com/a/78595091/1469797) which does run after access phase.
88
88
89
-
So let's create a dummy server inline and park our `return directive` behind a proxy.
89
+
So let's create a simple, inline server and park our `return` directive behind a `proxy_pass` to it.
90
90
91
91
```nginx
92
92
server {
···
116
116
117
117
What's that? You want to be clever? Don't want to create new `server` blocks? Do it all in the same `location` block? Fine. But you asked for it.
118
118
119
-
Let's revisit that [lua handler](https://github.com/crowdsecurity/cs-nginx-bouncer/blob/main/nginx/crowdsec_nginx.conf#L19) provided by the crowdsec bouncer. If we do some digging we can determine that re-defining the same handler within a nested directive (http -> server -> location) will cause the ["most specific" handler to be used.](https://groups.google.com/g/openresty-en/c/0RmRy6Q2DOA). Essentially, we can override crowdsec's handler with our own.
119
+
Let's revisit that [lua handler](https://github.com/crowdsecurity/cs-nginx-bouncer/blob/main/nginx/crowdsec_nginx.conf#L19) provided by the crowdsec bouncer. If we do some digging we can determine that re-defining the same handler within a nested directive (http -> server -> location) will cause the [_most specific_ handler to be used.](https://groups.google.com/g/openresty-en/c/0RmRy6Q2DOA) Essentially, we can override crowdsec's handler with our own.
120
120
121
121
So let's copy-paste their handle into our location block and add some code to do the redirect after cs runs.
122
122
···
139
139
end
140
140
}
141
141
}
142
+
}
142
143
```
143
144
{: file='site-confs/default.conf'}
144
145
145
-
But wait..this isn't working? And if we add debug logging we can see `ngx.redirect` never runs! What gives?
146
+
But wait..this isn't working? If we add debug logging after `cs.Allow` we can see `ngx.redirect` never runs! What gives?
146
147
147
-
Crowdsec's bouncer code is running [`ngx.exit(ngx.DECLINED)`](https://github.com/crowdsecurity/lua-cs-bouncer/blob/main/lib/crowdsec.lua#L662) when the IP is not blocked. This causes the handler to exit early and not run out code after `cs.Allow`.
148
+
Crowdsec's bouncer code is running [`ngx.exit(ngx.DECLINED)`](https://github.com/crowdsecurity/lua-cs-bouncer/blob/main/lib/crowdsec.lua#L662) when the IP is not blocked. This causes the handler to exit early and not run our code after `cs.Allow`.
148
149
149
150
So lets replace that exit with a return...
150
151
···
154
155
+ return
155
156
end
156
157
```
158
+
{: file='crowdsec.lua#L662'}
159
+
157
160
158
161
Now our redirect is running! Hooray!
159
162
160
-
But now you need to deal with patching this file any time the docker-mod is updated. Manging this dependency would be a PITA but it is possible. [I've commented on an issue on the bouncer to make this easier](https://github.com/crowdsecurity/lua-cs-bouncer/issues/95) but it's still not as end-user friendly as the proxy_pass solution. Good luck with that.
163
+
But now you need to deal with patching this file any time the docker-mod is updated or the container is recreated. Manging this dependency would be a PITA but it is possible. [I've commented on an issue on the bouncer to make this easier](https://github.com/crowdsecurity/lua-cs-bouncer/issues/95) but it's still not as end-user friendly as the proxy_pass solution. Good luck with that.