Allow not setting Content-Length when Transfer-Encoding header is set

[katbow]

Basic Info

• Faraday Version: 1.10.3
• Ruby Version: 3.2.2

Issue description

When I set Transfer-Encoding to "chunked", I consistently get a 400 error when attempting to upload an image (to BigCommerce products API, if it matters). I found that the issue was Content-Length and Transfer-Encoding were both being set, with the former being set automatically by faraday.

After some digging, I found that Content-Length should not be sent alongside Transfer-Encoding. I believe this to be a non-standard use of these headers, and instead only one should be sent at a time.

I would like to be able to set either one or the other as I need it to be configured. For example, if I set Transfer-Encoding to "chunked", I would expect Content-Length to not be set.

Steps to reproduce

Perform any request with Transfer-Encoding set to "chunked" and Content-Length will also be set automatically

I am using a fairly old version of faraday, but I couldn't find anything searching through newer releases that indicated this is fixed.

[iMacTia]

Thank you for opening this issue @katbow, the resource you shared is pretty clear on this:

The Content-Length header field must not be sent if these two lengths are different
(i.e., if a Transfer-Encoding header field is present).

It also goes on saying that "If a message is received with both a Transfer-Encoding header field and a Content-Length header field, the latter must be ignored." so we might argue the BigCommerce API could handle this slightly better, but point taking we should not be sending both headers to begin with 😄

First things first, I tried reproducing this on Faraday's latest version (2.9.0):

conn = Faraday.new('https://httpbingo.org') do |f|
  f.response :logger, nil, { headers: true }
end
puts conn.post('/headers', 'body=abc', { 'Transfer-Encoding' => 'chunked' }).body

The request above logs the following request:

I, [2024-05-30T16:41:02.149120 #74437]  INFO -- request: POST https://httpbingo.org/headers
I, [2024-05-30T16:41:02.149204 #74437]  INFO -- request: User-Agent: "Faraday v2.9.0"
Transfer-Encoding: "chunked"

and returns the following JSON

{
  "headers": {
    "Accept": [
      "*/*"
    ],
    "Accept-Encoding": [
      "gzip;q=1.0,deflate;q=0.6,identity;q=0.3"
    ],
    "Content-Length": [
      "8"
    ],
    "Content-Type": [
      "application/x-www-form-urlencoded"
    ],
    "Host": [
      "httpbingo.org"
    ],
    "User-Agent": [
      "Faraday v2.9.0"
    ],
    ...
  }
}

As you can see, Faraday only sets the Transfer-Encoding header, but the request then is sent without, while the Content-Length gets set.
At this point, my guess is that this is the underlying HTTP library (in my example, the default adapter, a.k.a. Net::HTTP) messing with the headers, so I'd be interested if you're also using the default adapter and if you could try making the same request with Net::HTTP natively, without Faraday?

My hunch is that you're trying to stream your request, but that's not something we support out of the box (as opposed to streaming responses, which is supported). Net::HTTP just gracefully makes the request non-streaming, while other adapters I tried (httpx, async-http) just fail to send the request altogether when I include the Transfer-Encoding header.

In conclusion, I'd really need some more information about what you're trying to achieve and your exact setup, because the issue is not easily reproducible without those