[ipxe-devel] iPXE TLS handshake failure for messages in multiple TLS records?

[Peter Wagemans]

Dear iPXE developers,

Apologies if you feel this is not an appropriate mail for this
address. In that case please tell me where I should submit this issue
instead.

In some experiments, I tried to get iPXE to talk https to a RedHat
Satellite server. That resulted in http://ipxe.org/1c0de802, "Error:
Invalid Handshake record".

Using openssl s_client I noticed that this Satellite server does a
client certificate request with a rather large list of acceptable CA's
(about 20k) that is transferred in two TLS records. My suspicion is
that the iPXE TLS code has problems with that. If I read the code
correctly, in tls.c, for each received record, tls_new_record, for
records of type handshake, calls tls_new_handshake. That routine does

     payload_len = tls_uint24 ( &handshake->length );
     if ( payload_len > ( remaining - sizeof ( *handshake ) ) ) {
     ...
         return -EINVAL_HANDSHAKE;

which is one of the two lines with the error code in the test.

It doesn't seem to take into account that the payload_len of the
handshake message can be bigger than the received TLS record and will
continue in a subsequent TLS record. Unfortunately, that is the case
in the current setup of the tested Satellite server.

Can you confirm this theory about a handshake failure mechanism?

If the code is indeed the culprit, then a workaround is probably to
reconfigure the Satellite server to either not ask for client
certificates on the URLs that iPXE retrieves, or to configure a
smaller list of acceptable client certificate CAs.

Regards and thanks,

Peter Wagemans