The Caddy Proxy Mystery: From a Deprecated Directive to a Version "Bug"

This post documents a time when using a forward proxy within a Caddy reverse proxy led me down a rabbit hole involving directive evolution, behavioral differences between versions, and even made me question the accuracy of the official documentation.

The forward_proxy_url Fog and the Version Mystery

#

Our core requirement was to connect to a final backend service through an internal forward proxy, which itself was behind a Caddy reverse proxy. This created a classic “proxy chaining” scenario.

Target Architecture: User -> Caddy (Reverse Proxy) -> Internal Proxy -> Backend App

First Attempt: forward_proxy_url

#

Based on experience, we wrote the following configuration in our Caddyfile:

1app.example.com {
2    # ...
3    reverse_proxy backend-service:8188 {
4        transport http {
5            # Use forward_proxy_url to specify the next-hop proxy
6            forward_proxy_url http://internal-proxy:1025
7        }
8    }
9}

However, when reloading the Caddy configuration, I received the first confusing message:

{"level":"warn", ... "msg":"The 'forward_proxy_url' field is deprecated. Use 'network_proxy <url>' instead."}

This was a clear deprecation warning. It told us that forward_proxy_url was obsolete and should be replaced with the new network_proxy directive.

Second Attempt: The network_proxy Frustration

#

Following the warning’s advice, we modified the configuration:

 1# Plan A: Place network_proxy inside transport http
 2reverse_proxy backend-service:8188 {
 3    transport http {
 4        network_proxy http://internal-proxy:1055
 5    }
 6}
 7
 8# Plan B: Place network_proxy as a direct subdirective of reverse_proxy
 9reverse_proxy backend-service:8188 {
10    network_proxy http://internal-proxy:1055
11}

However, regardless of which approach we tried, Caddy returned an error: unrecognized subdirective network_proxy.

This left me confused: Caddy was warning me that the old directive was deprecated, yet it didn’t recognize the new one it recommended. Furthermore, the official Caddy documentation mentioned forward_proxy_url with no mention of network_proxy.

Turns Out, It Was a Version “Bug”

#

After repeatedly confirming that the Caddy version I was using (v2.10.0) was relatively new and didn’t require any custom-built plugins, the focus finally shifted to Caddy itself.

Eventually, I painstakingly found the cause: https://github.com/caddyserver/caddy/pull/6978

It was a bug in version 2.10.0, which was fixed in 2.10.1… what a pain.

The Solution:

I ended up rebuilding my Caddy image using version 2.10.2. I simply ignored the deprecation warning, as the network_proxy parameter did not work correctly, and continued using forward_proxy_url.

1# Final working configuration: Ignore the warning and carry on
2reverse_proxy backend-service:8188 {
3    transport http {
4        forward_proxy_url http://internal-proxy:1055
5    }
6}

This experience taught me that even stable software releases can have minor disconnects between documentation, warnings, and actual behavior. When faced with such a situation, trust the actual runtime results.

Host Header Causing a Browser Redirect Loop

#

After solving the proxy directive issue, I ran into a second problem. Testing the proxy chain from within the Caddy container using curl was completely successful:

curl -x http://internal-proxy:1055 http://backend-service:8188

curl could fetch the backend application’s page perfectly. However, accessing https://app.example.com in a browser resulted in an infinite redirect loop.

Symptoms and Root Cause

#

After analyzing Caddy logs and researching the differences between curl and Caddy’s behavior, I found the culprit:

curl was successful because the Host header it sent to the backend was the backend’s IP address.

In contrast, when a browser accessed the site via Caddy, the Host header being passed along was the public domain name (app.example.com). Many backend applications, upon receiving a Host header that doesn’t match their own listening address, will initiate a redirect for security or normalization reasons. This redirect often conflicts with Caddy’s automatic HTTPS feature, leading to a loop.

Solution: Spoofing the Host Header

#

To make Caddy’s behavior consistent with the successful curl command, we used header_up to forcibly modify the Host header being sent to the backend.

1reverse_proxy backend-service:8188 {
2    # Forcibly change the Host header to the backend's own address
3    header_up Host {http.reverse_proxy.upstream.hostport}
4
5    transport http {
6        forward_proxy_url http://internal-proxy:1055
7    }
8}

With this simple line of configuration, we made the backend application receive the Host header it expected, which stopped the unnecessary redirects, and browser access returned to normal.

Summary

#

1. Treat Warnings Rationally: While deprecation warnings are important, when the new solution doesn’t work, trust the old solution that is still effective in your current version.
2. The Host Header is the Devil in the Details: In a reverse proxy environment, the Host header is the prime suspect for issues where “curl works, but the browser doesn’t.” Control it with header_up.