URL-encoded request urls break header rewriting

When we first encountered strange cookie headers with broken URL encoding, we just ignored the issue, assuming some client-side problem. Up until the first time that we had the problem ourselves.

Our is basically a nginx as reverse proxy, which delegates work to the microservices behind it. For some reasons, we need to delete certain cookie values before passing the request on. We did that by using the suggestion found here: https://forum.nginx.org/read.php?2,252944,252947

set $new_cookie $http_cookie;
if ($http_cookie ~ "(.*)(?:^|;)\s*secret=[^;]+(.*)") {
  set $new_cookie $1$2;
}
proxy_pset_header Cookie $new_cookie;

That worked like a charm (except when it didn't). We found out that just sometimes, nginx will do one additional url encoding for $new_cookie, effectively resulting in a broken Cookie-header.

Finally, we found out what caused the behavior. Adding an url encoded character to the request url will lead to this additional url encoding of the capture groups.

The solution we found in https://trac.nginx.org/nginx/ticket/348. They suggest to use named capture groups to work around the problem, an that helped in our case, too.

A colleague posted a minimal reproducible example here: https://github.com/morbidick/nginx-regex-capture-repro

http {
  include       mime.types;
  charset       utf-8;
  server {
    server_name   _;
    listen        80;

    # matches any cookie header and copies its
    # value to $captured
    if ($http_cookie ~* "(.*)") {
      set $captured $1;
    }

    if ($http_cookie ~* "(?<name>.*)") {
      set $named $name;
    }

    set $direct $http_cookie;

    location / {
        return 200 "Request uri: $request_uri \n
                    Cookie: $http_cookie \n
                    captured: $captured \n
                    direct: $direct \n
                    named: $named \n\n";
    }
  }
}

What actually concerns me here is that just by changing the request url, you can break stuff at some completely unrelated place. The different behavior of named and unnamed capture groups saved us, but also seems questionable.