English version of our German article on the same topic.
We run a certificate-based IKEv2 VPN for our workstations. The gateway is iked from the OpenBSD base system, the certificates come from our own PKI, all of them ECDSA on the P-384 curve. Windows (native IKEv2 client with a machine certificate) and Linux (strongSwan) connect without any fuss. The first Mac did not. This article describes how we found the cause, why it is not a configuration issue, and what needs to change in the iked source. If you want to connect Apple clients to an OpenBSD gateway, this will probably save you an evening.
The symptom
The macOS profile (a .mobileconfig with root CA, intermediate CA, a PKCS#12 identity and an IKEv2 connection with CertificateType = ECDSA384) installed cleanly. On connecting, macOS only said “authentication failed”. The gateway logged:
ikev2_auth_verify: ikev2_msg_authverify failed
ikev2_send_auth_failed: authentication failed for UFQDN/user@example.com
ikev2_send_auth_failed: issuer: /C=DE/O=Example/CN=Example VPN CAThe remarkable part: iked had already validated the client certificate against the chain and the CRL. The policy matched, the pool address was assigned, the traffic selectors were negotiated. Only the last step failed, verifying the signature in the AUTH payload. And it failed without any further reason, even with ikectl log verbose.
The RSA detour
The obvious suspicion was an ECDSA incompatibility. So we built a second profile with an RSA-4096 certificate and CertificateType = RSA. Result: the gateway now accepted the Mac’s signature, but the Mac aborted instead. Apple’s client wrote to its log:
Initiator packet authentication method DigitalSignatureECDSA256
is not compatible with configuration RSASignatureThat is lesson one: Apple’s IKEv2 client requires the gateway to use the same signature family as the profile’s CertificateType. An RSA profile only works against a gateway with an RSA key. If your gateway runs on a P-384 key, as ours does, you have to solve the ECDSA problem; you cannot route around it.
The proof from Apple’s log
You cannot get further without a test device. We rented a Mac mini in the cloud for a day and installed the profile there. Apple’s network extension logs every IKE exchange in clear text, retrievable with:
log show --last 15m --info --predicate 'process == "NEIKEv2Provider"'It contains the AUTH payload the Mac sends:
Authentication Protocol = ECDSA384
Authentication Data = {length = 208, bytes = 0x00000000 00000000 00000000 00000000 ... }That is the cause. RFC 4754 specifies that for ECDSA-384 the two signature values r and s are exactly 48 bytes each, 96 bytes in total, zero-padded at the front where necessary. Apple sends 208 bytes that start with a long run of zeros. Those are r and s at 104 bytes each, and 104 is exactly the maximum length of a DER-encoded P-384 signature. Apple apparently uses the DER maximum as the width of both values. Mathematically this is harmless; the numbers are merely extended with zeros. Conformant it is not.
Why does nobody else notice? strongSwan, the gateway behind most Apple client guides, simply splits the received data in half and hands the halves to BN_bin2bn(), which ignores leading zeros. iked checks the length first.
The cause in the source
In sbin/iked/crypto.c, _dsa_verify_prepare() converts the raw r|s into the DER format libcrypto needs for verification. A pre-check sits in front of it:
case IKEV2_AUTH_ECDSA_256:
case IKEV2_AUTH_ECDSA_384:
case IKEV2_AUTH_ECDSA_521:
/* ... */
if (*lenp < 64 || *lenp > 132 || *lenp % 2)
goto done;
bnlen = (*lenp)/2;
/* sigp points to concatenation: r|s */
if ((obj = ECDSA_SIG_new()) == NULL ||
(r = BN_bin2bn(*sigp, bnlen, NULL)) == NULL ||
(s = BN_bin2bn(*sigp+bnlen, bnlen, NULL)) == NULL ||
ECDSA_SIG_set0(obj, r, s) == 0 ||
(len = i2d_ECDSA_SIG(obj, &ptr)) <= 0)
goto done;The upper bound of 132 is the length of r|s for P-521 (2 × 66 bytes). Apple’s 208 bytes exceed it, goto done returns −1, and the caller only reports “authentication failed”. There is no log line for this case and no knob in iked.conf. ikeauth ecdsa384 does not change anything either, because the verification path stays the same.
The fix
The length check is a sanity filter, not a security control. After the split, r and s are DER-encoded and checked against the curve order by EVP_DigestVerifyFinal(); an oversized or wrong r or s fails there. So it is enough to raise the upper bound:
--- sbin/iked/crypto.c
+++ sbin/iked/crypto.c
@@ -1149,7 +1149,7 @@
* buffer (we cannot override the given buffer) and the caller
* has to free this buffer ('freeme').
*/
- if (*lenp < 64 || *lenp > 132 || *lenp % 2)
+ if (*lenp < 64 || *lenp > 512 || *lenp % 2)
goto done;
bnlen = (*lenp)/2;
/* sigp points to concatenation: r|s */The value 512 still bounds the allocation and lets all three curves through with Apple’s padding. A stricter variant would derive the expected width from the key’s curve and accept longer data only if the surplus bytes are zero. For production use the simple form is sufficient, because the real verification happens in libcrypto anyway.
This is how we built and installed the binary on OpenBSD 7.9:
# release sources, verify the signature, extract only sbin/iked
ftp https://cdn.openbsd.org/pub/OpenBSD/7.9/{SHA256.sig,src.tar.gz}
signify -C -p /etc/signify/openbsd-79-base.pub -x SHA256.sig src.tar.gz
mkdir -p /tmp/src && tar xzf src.tar.gz -C /tmp/src sbin/iked
# errata source patches for iked (009, 027), also signed
cd /tmp/src && patch -p0 < 009_iked.patch && patch -p0 < 027_iked.patch
# patch, build, check, back up, install
cd sbin/iked && patch -p0 < crypto.c-apple-ecdsa.diff && make
doas ./iked -n # configuration OK
doas cp -p /sbin/iked /sbin/iked.orig-7.9
doas install -o root -g bin -m 555 iked /sbin/iked
doas rcctl restart ikedRun syspatch to the current level first, otherwise a later erratum overwrites your binary with the old check again. The rollback is the saved file. And because any system update may replace the binary, our monitoring now checks the checksum of /sbin/iked.
The result: the Mac connected on its very next attempt, with P-384 like every other client. We reported the issue together with the patch to the OpenBSD project; once a fix lands in the base system, the custom build becomes unnecessary.
Three side findings for Apple profiles
Along the way we stumbled over three more quirks that affect every profile for Apple clients:
- PKCS#12 only in the legacy format. Apple’s importer rejects containers as produced by OpenSSL 3 by default (AES-256-CBC, PBKDF2, SHA-256 MAC) and then reports “wrong password”. It has to be
pbeWithSHA1And3-KeyTripleDES-CBCwith a SHA-1 MAC, OpenSSL’s-legacyswitch. Security then rests on the password, not on the wrapping. - No split tunnel in the profile. Apple’s client has no client-side routes. Which networks go through the tunnel is decided solely by the traffic selectors the gateway returns. With iked that means several
from <network> to dynamiclines in the policy instead offrom any. The Mac installs exactly those routes and leaves the default route on the local interface. - No RFC 7427. Apple announces hash algorithms for signature authentication but then uses the classic methods: RSA with SHA-1, or ECDSA according to RFC 4754. If you wonder why a gateway signs with RFC 7427 and the Mac still reports ECDSA-256 in its log, this is the explanation.
Conclusion
The configuration was right from the start. That it still did not work came down to two implementations that are each consistent in their own way: Apple pads signatures wider than the standard requires, and iked rejects whatever does not fit the standard’s frame. Without the client log on a real Mac the cause would have been impossible to find; the gateway only sees “authentication failed”. Anyone who wants to run Apple clients against OpenBSD needs the patch shown above until the base system is corrected, or a gateway with an RSA key, which forces a change on every other client. The one line is the smaller intervention.