SSL Client Authentication with smart card works in Java 6 but fails in Java 7

authentication, java-7, pkcs#11, smartcard, ssl

Solution

Solved by adding the line of code:

Security.removeProvider("IAIK");

before the line:

Security.addProvider(sunpkcs11);

This works correctly with all versions of java6 and java7 (hoping they won't screw again something in java8... :)

Looks like the IAIK provider inserts itself as a PKCS11 provider but then makes calls to the private key using it as a software key... because it's actually a software provider.

Problem

the following code creates a client authenticated SSL context using PKCS#11 device (smart card). It all works great with Java 6: ``` // Configure the SunPkcs11 provider String pkcs11config; pkcs11config = "name = Cryptoki"; pkcs11config += "\nlibrary = /SCDriver/libbit4ipki.dylib"; InputStream confStream = new ByteArrayInputStream(pkcs11config.getBytes()); SunPKCS11 sunpkcs11 = new SunPKCS11(confStream); Security.addProvider(sunpkcs11); // Specify keystore builder parameters for PKCS#11 keystores Builder scBuilder = Builder.newInstance("PKCS11", sunpkcs11, new KeyStore.CallbackHandlerProtection(new PasswordRetriever())); // Create and init KeyManagerFactory KeyManagerFactory factory = KeyManagerFactory.getInstance("NewSunX509"); factory.init(new KeyStoreBuilderParameters(scBuilder)); // create and init ssl context m_ssl_context = SSLContext.getInstance("TLS"); m_ssl_context.init(factory.getKeyManagers(), new TrustManager[] {new PkTrustManager()}, null); SSLContext.setDefault(m_ssl_context); ``` The PkTrustManager is simply and 'empty' class, taking any server/client certificate for good, and PasswordRetriever just asks for password through a dialog box (By request I post source code for these). On Java 7 instead I get the following exception during SSL handshake of the ssl context: ``` java.security.InvalidKeyException: Class does not represent an RSA key: sun.security.pkcs11.P11Key$P11PrivateKey at iaik.pkcs.pkcs1.RSACipher.engineInit(Unknown Source) at iaik.pkcs.pkcs1.RSACipher.engineInit(Unknown Source) at iaik.security.rsa.RSA.init(Unknown Source) at iaik.security.rsa.RawRSASignature.engineInitSign(Unknown Source) at java.security.SignatureSpi.engineInitSign(SignatureSpi.java:103) at java.security.Signature.initSign(Signature.java:529) at sun.security.ssl.RSASignature.engineInitSign(RSASignature.java:125) at java.security.Signature$Delegate.engineInitSign(Signature.java:1136) at java.security.Signature.initSign(Signature.java:529) at sun.security.ssl.HandshakeMessage$CertificateVerify.<init>(HandshakeMessage.java:1556) at sun.security.ssl.ClientHandshaker.serverHelloDone(ClientHandshaker.java:982) ... 14 more ``` In the best case, I'd say something has changed in Java internals, but checking the Oracle documentation, I didn't find any supposed changes on the NewSunX509 key manager, or other components. I double checked the code and it seems compliant to the specifications (through there's for sure something I missed!). I tried to add the configuration flags: ``` System.setProperty("javax.net.ssl.keyStoreType", "pkcs11"); System.setProperty("javax.net.ssl.keyStore", "NONE"); System.setProperty("javax.net.ssl.trustStoreType", "pkcs11"); System.setProperty("javax.net.ssl.trustStore", "NONE"); System.setProperty("javax.net.ssl.keyStoreProvider", sunpkcs11.getName() ); JCEMapper.setProviderId(sunpkcs11.getName()); ``` But no change, same error... and they are not required in Java 6 where all works. Thanks in advance anyone can help or has any ideas! PS: By request from @owlstead added -Djava.security.debug=sunpkcs11 and got the following output: ``` SunPKCS11 loading ---DummyConfig-1--- sunpkcs11: Initializing PKCS#11 library /SCDriver/libbit4ipki.dylib Information for provider SunPKCS11-Cryptoki Library info: cryptokiVersion: 2.20 manufacturerID: bit4id srl flags: 0 libraryDescription: bit4id PKCS#11 libraryVersion: 1.02 All slots: 0 Slots with tokens: 0 Slot info for slot 0: slotDescription: bit4id miniLector-U38 00 00 manufacturerID: unknown flags: CKF_TOKEN_PRESENT | CKF_REMOVABLE_DEVICE | CKF_HW_SLOT hardwareVersion: 0.00 firmwareVersion: 0.00 Token info for token in slot 0: label: CNS manufacturerID: ST Incard model: CNS (LB) serialNumber: 7420057800291590 flags: CKF_RNG | CKF_LOGIN_REQUIRED | CKF_USER_PIN_INITIALIZED | CKF_TOKEN_INITIALIZED ulMaxSessionCount: CK_EFFECTIVELY_INFINITE ulSessionCount: 0 ulMaxRwSessionCount: CK_EFFECTIVELY_INFINITE ulRwSessionCount: CK_UNAVAILABLE_INFORMATION ulMaxPinLen: 8 ulMinPinLen: 5 ulTotalPublicMemory: 31988 ulFreePublicMemory: CK_UNAVAILABLE_INFORMATION ulTotalPrivateMemory: 780 ulFreePrivateMemory: CK_UNAVAILABLE_INFORMATION hardwareVersion: 0.00 firmwareVersion: 0.00 utcTime: 0000000000000000 Mechanism CKM_RSA_PKCS: ulMinKeySize: 1024 ulMaxKeySize: 1024 flags: 2561 = CKF_HW | CKF_DECRYPT | CKF_SIGN Mechanism CKM_RSA_PKCS_KEY_PAIR_GEN: ulMinKeySize: 1024 ulMaxKeySize: 1024 flags: 65537 = CKF_HW | CKF_GENERATE_KEY_PAIR Mechanism CKM_SHA1_RSA_PKCS: ulMinKeySize: 1024 ulMaxKeySize: 1024 flags: 2049 = CKF_HW | CKF_SIGN Mechanism CKM_SHA_1: ulMinKeySize: 0 ulMaxKeySize: 0 flags: 1024 = CKF_DIGEST Mechanism CKM_SHA256: ulMinKeySize: 0 ulMaxKeySize: 0 flags: 1024 = CKF_DIGEST Mechanism CKM_SHA256_RSA_PKCS: ulMinKeySize: 1024 ulMaxKeySize: 1024 flags: 2049 = CKF_HW | CKF_SIGN Mechanism CKM_SHA384: ulMinKeySize: 0 ulMaxKeySize: 0 flags: 1024 = CKF_DIGEST Mechanism CKM_SHA384_RSA_PKCS: ulMinKeySize: 1024 ulMaxKeySize: 1024 flags: 2049 = CKF_HW | CKF_SIGN Mechanism CKM_SHA512: ulMinKeySize: 0 ulMaxKeySize: 0 flags: 1024 = CKF_DIGEST Mechanism CKM_SHA512_RSA_PKCS: ulMinKeySize: 1024 ulMaxKeySize: 1024 flags: 2049 = CKF_HW | CKF_SIGN Mechanism CKM_RSA_X_509: ulMinKeySize: 1024 ulMaxKeySize: 1024 flags: 2561 = CKF_HW | CKF_DECRYPT | CKF_SIGN Password per token PKCS11 [SunPKCS11-Cryptoki]: sunpkcs11: login succeeded sunpkcs11: user already logged in javax.net.ssl.SSLHandshakeException: Error signing certificate verify at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1886) *....... (continues with the exception as described above)* ``` Note: With the last update of Java 6, my code stops working on Java 6 as well :(

Original source