[http] Work around stateful authentication schemes

As pointedly documented in RFC7230 section 2.3, HTTP is a stateless
protocol: each request message can be understood in isolation from any
other requests or responses.  Various authentication schemes such as
NTLM break this fundamental property of HTTP and rely on the same TCP
connection being reused.

Work around these broken authentication schemes by ensuring that the
most recently pooled connection is reused for the subsequent
authentication retry.

Reported-by: Andreas Hammarskjöld <junior@2PintSoftware.com>
Tested-by: Andreas Hammarskjöld <junior@2PintSoftware.com>
Signed-off-by: Michael Brown <mcb30@ipxe.org>

src/net/tcp/httpconn.c

@@ -252,8 +252,13 @@ int http_connect ( struct interface *xfer, struct uri *uri ) {
 	/* Identify port */
 	port = uri_port ( uri, scheme->port );
 
-	/* Look for a reusable connection in the pool */
+	/* Look for a reusable connection in the pool.  Reuse the most
-	list_for_each_entry ( conn, &http_connection_pool, pool.list ) {
+	 * recent connection in order to accommodate authentication
+	 * schemes that break the stateless nature of HTTP and rely on
+	 * the same connection being reused for authentication
+	 * responses.
+	 */
+	list_for_each_entry_reverse ( conn, &http_connection_pool, pool.list ) {
 
 		/* Sanity checks */
 		assert ( conn->uri != NULL );

src/net/tcp/httpcore.c

@@ -778,6 +778,18 @@ static int http_transfer_complete ( struct http_transaction *http ) {
 	http->len = 0;
 	assert ( http->remaining == 0 );
 
+	/* Retry immediately if applicable.  We cannot rely on an
+	 * immediate timer expiry, since certain Microsoft-designed
+	 * HTTP extensions such as NTLM break the fundamentally
+	 * stateless nature of HTTP and rely on the same connection
+	 * being reused for authentication.  See RFC7230 section 2.3
+	 * for further details.
+	 */
+	if ( ! http->response.retry_after ) {
+		http_reopen ( http );
+		return 0;
+	}
+
 	/* Start timer to initiate retry */
 	DBGC2 ( http, "HTTP %p retrying after %d seconds\n",
 		http, http->response.retry_after );