h11

Security fix

Reject certain malformed Transfer-Encoding: chunked bodies that were previously accepted. These could have enabled request-smuggling attacks when an h11-based HTTP server was placed behind a load balancer with a matching bug in its chunked handling.

Advisory with more details: https://github.com/python-hyper/h11/security/advisories/GHSA-vqfr-h8mv-ghfj

Reported by: Jeppe Bonde Weikop

Bugfixes

• Reject Content-Lengths >= 1 zettabyte (1 billion terabytes) early, without attempting to parse the integer (#181)

Miscellaneous internal changes

• Remove the tests folder from wheel files. This reduces the zipped file size by 20KB (about 30%). (#158)

Features

• Allow additional trailing whitespace in chunk headers for additional compatibility with existing servers. (#133)
• Improve the type hints for Sentinel types, which should make it easier to type hint h11 usage. (#151 & #144))

Deprecations and Removals

• Python 3.6 support is removed. h11 now requires Python>=3.7 including PyPy 3. Users running pip install h11 on Python 2 will automatically get the last Python 2-compatible version. (#138)

Features

• Clarify that the Headers class is a Sequence and inherit from the collections Sequence abstract base class to also indicate this (and gain the mixin methods). See also #104. (#112)
• Switch event classes to dataclasses for easier typing and slightly improved performance. (#124)
• Shorten traceback of protocol errors for easier readability (#132).
• Add typing including a PEP 561 marker for usage by type checkers (#135).
• Expand the allowed status codes to [0, 999] from [0, 600] (`#134 https://github.com/python-hyper/h11/issues/134`__).

Backwards incompatible changes

• Ensure request method is a valid token (`#141 https://github.com/python-hyper/h11/pull/141>`__).

Features

• Added support for servers with broken line endings.
  After this change h11 accepts both \r\n and \n as a headers delimiter. (#7)
• Add early detection of invalid http data when request line starts with binary (#122)

Deprecations and Removals

• Python 2.7 and PyPy 2 support is removed. h11 now requires Python>=3.6 including PyPy 3. Users running pip install h11 on Python 2 will automatically get the last Python 2-compatible version. (#114)

New features:

• h11 now stores and makes available the raw header name as received. In addition h11 will write out header names with the same casing as passed to it. This allows compatibility with systems that expect titlecased header names. See #31.
• Multiple content length headers are now merged into a single header if all the values are equal, if any are unequal a LocalProtocol error is raised (as before). See #92.

Backwards incompatible changes:

• Headers added by h11, rather than passed to it, now have titlecased names. Whilst this should help compatibility it replaces the previous lowercased header names.

Other changes:

• Drop support for Python 3.4.
• Support Python 3.8.
• Make error messages returned by match failures less ambiguous (#98).

Bug fixes:

• Allow a broader range of characters in header values. This violates the RFC, but is apparently required for compatibility with real-world code, like Google Analytics cookies (#57, #58).
• Validate incoming and outgoing request paths for invalid characters. This prevents a variety of potential security issues that have affected other HTTP clients. (#69).
• Force status codes to be integers, thereby allowing stdlib HTTPStatus IntEnums to be used when constructing responses (#72).

Other changes:

• Make all sentinel values inspectable by IDEs, and split SEND_BODY_DONE into SEND_BODY, and DONE (#75).
• Drop support for Python 3.3.
• LocalProtocolError raised in start_next_cycle now shows states for more informative errors (#80).

Bug fixes:

• Always return headers as bytes objects (#60)

Other changes:

• Added proper license notices to the Javascript used in our documentation (#61)

Backwards incompatible changes:

• h11 now performs stricter validation on outgoing header names and header values: illegal characters are now rejected (example: you can’t put a newline into an HTTP header), and header values with leading/trailing whitespace are also rejected (previously h11 would silently discard the whitespace). All these checks were already performed on incoming headers; this just extends that to outgoing headers.

New features:

• New method Connection.send_failed(), to notify a Connection object when data returned from Connection.send() was not sent.

Bug fixes:

• Make sure that when computing the framing headers for HEAD responses, we produce the same results as we would for the corresponding GET.
• Error out if a request has multiple Host: headers.
• Send the Host: header first, as recommended by RFC 7230.
• The Expect: header is case-insensitive, so use case-insensitive matching when looking for 100-continue.

Other changes:

• Better error messages in several cases.
• Provide correct error_status_hint in exception raised when encountering an invalid Transfer-Encoding header.
• For better compatibility with broken servers, h11 now tolerates responses where the reason phrase is missing (not just empty).
• Various optimizations and documentation improvements.

New features (backwards compatible):

• Made it so that sentinels are instances of themselves, to enable certain dispatch tricks on the return value of Connection.next_event() (see issue #8 for discussion).
• Added Data.chunk_start and Data.chunk_end properties to the Data event. These provide the user information about where chunk delimiters are in the data stream from the remote peer when chunked transfer encoding is in use. You probably shouldn’t use these, but sometimes there’s no alternative (see issue #19 for discussion).
• Expose Response.reason attribute, making it possible to read or set the textual “reason phrase” on responses (issue #13).

Bug fixes:

• Fix the error message given when a call to an event constructor is missing a required keyword argument (issue #14).
• Fixed encoding of empty Data events (Data(data=b"")) when using chunked encoding (issue #21).

This is the first release since we started using h11 to write non-trivial server code, and this experience triggered a number of substantial API changes.

Backwards incompatible changes:

• Split the old receive_data() into the new receive_data() and next_event(), and replaced the old Pausedpseudo-event with the new NEED_DATA and PAUSEDsentinels.
• Simplified the API by replacing the old Connection.state_of(), Connection.client_state, Connection.server_state with the new Connection.states.
• Renamed the old prepare_to_reuse() to the new start_next_cycle().
• Removed the Paused pseudo-event.

Backwards compatible changes:

• State machine: added a DONE -> MUST_CLOSE transition triggered by our peer being in the ERROR state.
• Split ProtocolError into LocalProtocolError and RemoteProtocolError (see Error handling). Use case: HTTP servers want to be able to distinguish between an error that originates locally (which produce a 500 status code) versus errors caused by remote misbehavior (which produce a 4xx status code).
• Changed the PRODUCT_ID from h11/<verson> to python-h11/<version>. (This is similar to what requests uses, and much more searchable than plain h11.)

Other changes:

• Added a minimal benchmark suite, and used it to make a few small optimizations (maybe ~20% speedup?).