FULL PRODUCT VERSION :
java version "1.7.0_03"
Java(TM) SE Runtime Environment (build 1.7.0_03-b05)
Java HotSpot(TM) Client VM (build 22.1-b02, mixed mode, sharing)

ADDITIONAL OS VERSION INFORMATION :
Microsoft Windows [Version 6.1.7601]

A DESCRIPTION OF THE PROBLEM :
If trying to use MGF1 and specify SHA-256 as message digest both for OAEP and MGF1 the algorithm fails with (part of stack trace, see full in other field):
Exception in thread "main" javax.crypto.BadPaddingException: java.security.DigestException: Length must be at least 32 for SHA-256digests
at sun.security.rsa.RSAPadding.mgf1(RSAPadding.java:469)
at sun.security.rsa.RSAPadding.padOAEP(RSAPadding.java:394)

I found the code for RSAPadding and the problem seems to be that it was made for SHA-1 only (around the line number above):

  byte[] digest = new byte[20]; // 20 bytes is length of SHA-1 digest

This causes the message digest made to fail due to the array being too small. A fix could be to use MessageDigest.getDigestLength() or to use the digest() that returns the result.


STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
The following code should reproduce the problem (full program in other field):

    Cipher cip = Cipher.getInstance("RSA/ECB/OAEPWITHSHA-256ANDMGF1PADDING");
    cip.init(Cipher.ENCRYPT_MODE, pubKey, new OAEPParameterSpec("SHA-256",
                                                                "MGF1",
                                                                MGF1ParameterSpec.SHA256,
                                                                PSource.PSpecified.DEFAULT));
    final byte[] enc = cip.doFinal(new byte[50]);


EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
enc containing an OAEP encryption with the relevant algorithms used.
ACTUAL -
BadPaddingException caused by a DigestException.

ERROR MESSAGES/STACK TRACES THAT OCCUR :
Exception in thread "main" javax.crypto.BadPaddingException: java.security.DigestException: Length must be at least 32 for SHA-256digests
at sun.security.rsa.RSAPadding.mgf1(RSAPadding.java:469)
at sun.security.rsa.RSAPadding.padOAEP(RSAPadding.java:394)
at sun.security.rsa.RSAPadding.pad(RSAPadding.java:246)
at sun.security.rsa.RSAPadding.pad(RSAPadding.java:228)
at com.sun.crypto.provider.RSACipher.doFinal(RSACipher.java:351)
at com.sun.crypto.provider.RSACipher.engineDoFinal(RSACipher.java:382)
at javax.crypto.Cipher.doFinal(Cipher.java:2086)
at Test.main(Test.java:33)


REPRODUCIBILITY :
This bug can be reproduced always.

---------- BEGIN SOURCE ----------
import java.security.KeyPair;
import java.security.KeyPairGenerator;
import java.security.PublicKey;
import java.security.spec.MGF1ParameterSpec;

import javax.crypto.Cipher;
import javax.crypto.spec.OAEPParameterSpec;
import javax.crypto.spec.PSource;

public class Test {
  public static void main(String[] args) throws Exception {

    KeyPairGenerator gen = KeyPairGenerator.getInstance("RSA");
    KeyPair keyPair = gen.generateKeyPair();
    PublicKey pubKey = keyPair.getPublic();

    Cipher cip = Cipher.getInstance("RSA/ECB/OAEPWITHSHA-256ANDMGF1PADDING");
    cip.init(Cipher.ENCRYPT_MODE, pubKey, new OAEPParameterSpec("SHA-256",
                                                                "MGF1",
                                                                MGF1ParameterSpec.SHA256,
                                                                PSource.PSpecified.DEFAULT));
    final byte[] enc = cip.doFinal(new byte[50]);
  }
}

---------- END SOURCE ----------

CUSTOMER SUBMITTED WORKAROUND :
Make complete encoding and performing a raw (NoPadding) RSA encrypt.