CSP Out of Memory with CryptVerifySignature
I have a problem with CryptVerifySignature returning NTE_NO_MEMORY (CSP ran out of time during operation.)
I am trying to verify a signature generated with OpenSSL, I have already decrypted the PEM public key, changed the byte sequence and imported it.
I tried the exploit by specifying MS Enhanced Provider and my key length is 4096 bits (<16384 bits).
I've seen some comments on the forums suggesting memory alignment issues, but I'm on an Intel Pentium dual core processor and my buffers are 32 bit aligned.
Has anyone seen this problem before?
source to share
It's good to know that the problem is solved by chanting RSA_private_encrypt. My case is the opposite of you. I signed using CryptSignHash and verified it with RSA_verify.
I noticed that the byte order of the signature generated by CryptSignHash and RSA_sign for the same data is exactly the opposite. Check this link.
Try changing the bytes and see if that works for you.
I have signed successfully with CryptSignHash and verified with RSA_verify and it works fine.
source to share