We Keep Coding

RSA decryption according to the private key reports an error

My code using RSA on the front end：
const rsa = new JSEncrypt();
rsa.setPublicKey(k);
const resultText = rsa.encrypt("violet");
console.log(resultText);
My code using RSA in the backend：
byte[] inputByte = org.apache.commons.codec.binary.Base64.decodeBase64(str.getBytes("UTF-8"));
byte[] decoded = org.apache.commons.codec.binary.Base64.decodeBase64(privateKey);
PrivateKey priKey = KeyFactory.getInstance("RSA").generatePrivate(new
PKCS8EncodedKeySpec(decoded));
Cipher cipher = Cipher.getInstance("RSA");
cipher.init(Cipher.DECRYPT_MODE,priKey);
String outStr=new String(cipher.doFinal(inputByte));
return outStr;
PublicKey like this:
-----BEGIN PUBLIC KEY-----
MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA13gYdCmOjR9yqQD7ldzG
ZXabSon6SiLceCK6vRXf4NMbF+EQke0vRpqU3IZ/S1pFdvoQswQabsA4zf0WACVT
iaGIhWDlPu3mecri8rYtmOSfd8GCE0vEgFNvSD6IXRLPeLCB+i7WENBa4fCEtW8W
Hzdas96CLiESbjSAruRasQXP2OLqEA2GU83/069vh8uRKzui+yw0aAXZFyFyFRFa
lxYltFadVpz3+kBplvpzuj82t4fc3yCRbrpeRyTyX1sz0ULSxx/k3/p1OuJtIq9Y
9uN0G4gxhcDFJ4L41uXOln5CPapk7tlsYobhhvxYHw1rrweY+06hrQ7r0Hblv2nH
GQIDAQAB
-----END PUBLIC KEY-----
PrivateKey like this:
-----BEGIN RSA PRIVATE KEY-----
MIIEogIBAAKCAQEA13gYdCmOjR9yqQD7ldzGZXabSon6SiLceCK6vRXf4NMbF+EQ
ke0vRpqU3IZ/S1pFdvoQswQabsA4zf0WACVTiaGIhWDlPu3mecri8rYtmOSfd8GC
E0vEgFNvSD6IXRLPeLCB+i7WENBa4fCEtW8WHzdas96CLiESbjSAruRasQXP2OLq
EA2GU83/069vh8uRKzui+yw0aAXZFyFyFRFalxYltFadVpz3+kBplvpzuj82t4fc
3yCRbrpeRyTyX1sz0ULSxx/k3/p1OuJtIq9Y9uN0G4gxhcDFJ4L41uXOln5CPapk
7tlsYobhhvxYHw1rrweY+06hrQ7r0Hblv2nHGQIDAQABAoIBAAyqFmXde294BblB
QYhRbafRDNeYvIlW+zZkdC1g98OzJMiGhf7NvhWcSFud3CWFrMeNcyXSe+s+iRgy
Y/SmPP6969RLGa5VNVK7RhOV+aAe7/COAyM3NNmGDehlJIaz8FXbqggWcKaUWIMn
K+WuHdK/4ijoTyZ+8yJfG6Er8tisryLQ9io9+op9g/ZWzaUKgu934/cDxUt70bfm
x+ZEPi6YfkJ1uOpXnnadDyw2RUDcvCM3fK3KF5fqM7SJAXY9b1pmLr+Ccn1qkT9G
I+QHidEsGfJciX5AoHnlIMLPMVIPKBbq4GwC/Ngb41LprNJWlPR38N2ySjky/Jyt
159XWHECgYEA9lx2KfFmyLyVjnkIF3JI50mSZAw4YPBBqB27UInacvxXbjfVURht
xK60GB9OkDbFdeNh89x86Gfwvm5bTq4W8YSH4Obd5Fg8XjTuaicTi03CSfF5SdJn
JLLOUmlqP75gkbEPNUoOfqhqq6IbyJVB3egyL90cd2/wCdJOVLEUly8CgYEA3+Y4
lNdl0xedkDNkWsCyyA4iPSUzVxsuch9hW/VGBwzga8rtiNllpKifVvctQOEu/KUe
vVQRF78ojZaMGT33l6TivnUL54z9Lo9uWghoG8TqMfnG34pFPe3R+zvGP87Hrozw
1EUhiMT198SlB/YHrgGGGlJbG+rlm5GIx3lEdDcCgYA4RSw0LlA0v2ZFkX14pYDj
WxmVwnjKI3ZLqObU4XfE1cA+i4AssrC3wNOfwt7V77ywTYxc/9qD1uHVDS3LzdWt
uoCyrOi3tDOtrNdb5asAIXWkIAR9CRLH/hNEHZHIF3rFLDT2DgE7iso6g59m9DiE
L/nulsleunGQPLnpfDzgvwKBgDRV5Q3tl3CTSZJGYQPRnTikDR7LzkdjJCUq7qAH
IhpNyTuJEKL3ZgnqHGzAlERhHpGRqzDIMMKjPUBzW0YfNPuuYA3y4Bh83UV/42SK
KIOtMK0D3JeuA2dparbWDw4lMIm9iiGkEyWcHH6Q6Z6FxN/InWcTrxZEfu0xRI6T
6wtbAoGAfl5dW9LNoaNfQbgIq+PJKZm9f1frza55mFTJgo3ravdb3GmzWVHN8xRf
nLKyKyNLqbkT35IGc39NkALJLxT5RibkAZLiUiwqdMF63sgODbA9AGTmhe+JHS+V
hBmFnCyp6UiN9E4ZAWcZQILa0rRMftMFngAJ3El0ZP+HziRnNzs=
-----END RSA PRIVATE KEY-----
Bu-t, when i do the java code decryption, it reported such an error：
java.security.InvalidKeyException: IOException : DerInputStream.getLength(): lengthTag=111, too big.
How can i solve this problem ?

1. You are decoding wrong. PEM format has a dash-BEGIN line identifying the type of data, a block of base64 encoding the data, and a dash-END line. The BEGIN and END lines are part of the format, but they do not contain base64-encoded data; only the lines in between contain the base64-encoded data. You are apparently passing the whole thing, including the BEGIN and END lines, to commons.codec.Base64, which results in decoding a bunch of garbage before and after the actual data. That garbage isn't valid ASN.1 DER, so when Java tries to parse it as DER it fails.
2. Plus your data is not a PKCS8-clear privatekey. The PEM type 'RSA PRIVATE KEY' is an OpenSSL-defined format that contains a 'traditional' or 'legacy' format, namely the PKCS1 representation of the private key. This is not PKCS8, which is the only key format Java supports natively; that's why the spec class is named PKCS8EncodedKeySpec, because it is a key spec encoded as PKCS8 and more specifically PKCS8-clear. If you fix the above problem by removing the BEGIN and END lines before base64-decoding, Java can parse the result as DER, but not as a PKCS8-clear key; you get a different exception about 'algid parse error, not a sequence'. To fix this there are 5 approaches:
change whatever process you use to initially generate the keypair so it generates PKCS8, not OpenSSL-legacy PKCS1. Especially since you need anyway to replace the keypair you compromised by publishing it, as 207421 said. You give no clue what that process is or was, so I can't give any details.
convert your generated privatekey, or a copy, to PKCS8-clear. This is not programming or development and offtopic, but if you have or get OpenSSL (on the same or any accessible and secure system), you can do
openssl pkey -in oldfile -out newfile # 1.0.0 up only, but older is now rare
# or
openssl pkcs8 -topk8 -nocrypt -in oldfile -out newfile # even ancient versions
Once you have a PKCS8-clear file, just remove the BEGIN and END lines and base64-decode what is left, and pass that to KeyFactory as PKCS8EncodedKeySpec as you already do.
use https://www.bouncycastle.org . The 'bcpkix' jar has (Java) code to read a large range of OpenSSL-supported PEM formats, including the RSA-PKCS1 private key format you have. There are lots of existing Qs about this; just search for PEMParser and JcaPEMKeyConverter.
convert it yourself. Decode the body of the file you have, after removing the BEGIN and END lines, to get the PKCS1 key, then build the PKCS8 format for that key, and then pass it to KeyFactory as PKCS8EncodedKeySpec. See answers by Noa Resare and Jean-Alexis Aufauvre on Getting RSA private key from PEM BASE64 Encoded private key file or mine in Java: Convert DKIM private key from RSA to DER for JavaMail .
do it entirely yourself. Decode the file you have without BEGIN/END to get PCKS1, parse that as DER following e.g. RFC8447, and build RSAPrivateCrtKeySpec. Some other As on the Q I linked just above do this. However, this requires either: using undocumented internal sun.* classes, which used to work in Java (hence the existing answers) but which 'modular' Java versions (9 up) since 2017 have steadily made more difficult or impossible; using BouncyCastle which has documented (and good) support for ASN.1 -- but then it's easier to use bcpkix for the whole job as above; or writing your own ASN.1 parsing, which is a good deal of work.
PS: encrypting text with RSA is usually a bad design; it's not suited for that. But that's not really a programming issue and doesn't belong here.

Generating public key instance for JWT validation

today we changed on from the HMAC to the RSA on our login server so I have to change the code in our Andoid app to work with that. Problem is when I try to generate a RSA public key instance from a string, all I get is java.security.spec.InvalidKeySpecException: com.google.android.gms.org.conscrypt.OpenSSLX509CertificateFactory$ParsingException: Error parsing public key exception and I have no idea what am I doing wrong.
Code I use is pretty simple:
val base64KeyString ="LS0tLS.......BASE 64 PUBLIC KEY STRING..........LRVktLS0tLQ=="
val keySpec = X509EncodedKeySpec(base64KeyString.toByteArray())
val keyFactory = KeyFactory.getInstance("RSA")
val pubKey = keyFactory.generatePublic(keySpec) //<= this is where the exception is thrown
Thanks for any help

Java crypto uses actual data on its APIs not base64, so as MichaelFehr commented you need to base64-decode your string; in standard Java (8 up) this can be done with java.util.Base64 but I don't know for Android. But less obviously, your base64 string is wrong -- it is NOT the base64 encoding of a publickey, but rather the encoding of something that begins with 4 bytes with value 2D (which is the character code for hyphen) and ends with 7 bytes which are the character codes for "EY-----" -- in other words, it appears to be the base64 encoding of a slightly broken PEM file (that might possibly encode a publickey) not an actual publickey structure.
As a working example, given the following keypair represented by the (OpenSSL-compatible and widely used) PEM encodings
-----BEGIN RSA PRIVATE KEY-----
MIIEogIBAAKCAQEAsYU/MoIxYeVuCFcrX+fEH5XX/he/nJTSQ6P4RF38DKLNnNtQ
9XfnaMEPk2EXgSIHk+VXhE9vZNxsV7Emjg6C+eRxUB1NDBtWGX/Ts2cOYD8ZhSum
hOplKKEmdEkzGU/Rem3myfB2BQeN6RSRFigZfsR2I4oaQ5YDWJ3uk6Ix0XN4zQwm
tw1t9W9tgDZATDLW0+8uU02Wc2HrIcyOvDqJDXz/XVY+F+pkrws/ygOhxtVdNY7r
3meSKjSUVkOiXMdaqK+20l/YrvoZRVe6HEITpu9t/x6YDaoTThsbptjkAKBYpSG2
UeXJnnQMyKD2GT3Ouyiupv7lPovvLZZiqfVIWQIDAQABAoIBAGilPKjgmI21+pQO
FrKVZEaeRVIy27BJBl37Rbm/kXo9ammh22qoQRC+Zhkh7gIgdkkXs+x6Cxdw3Kkg
87JgGZOHt8hbTXTqlGm5sakbAh14dOnM3n+R0QoPXSTFlcrBA6JhMOAv0NKMLnqM
ti5Sex0AYQf4PB1FTDmr99ytiyKFK8bvAPiirKB4cEdbMTrxn6rtY324dN4PKdHR
7Sq9JnpYqOoY9adNlhPuN5vq91LCIjoDDkrZ8E8T8o9hUsJXFwsW0GDf4M7s8krj
Dt37zUPh6HSEg/Q1wM+ncEcE3PABeYvIibngfmcwVaMo/0T/9RjQL0EPUOuqxCg+
UlGyFFECgYEA63/vwM+uEil+4Zwqx7dyQ2gbNypEqNnbKelSKj0RpjJvkO7oQ39A
fcr198T1giwNo0cj3oUaduMtPBJUdpUqEYs/p1eNwcGbYSI5csLKy1awHIX5YIDY
oouIq5qipZPYXGFe5bsIZV2kRvzH/e1pW4/eDVgojVD3vKib9nPsi10CgYEAwPlB
KYu3uH5Di2UqNRAryAMCogJzCKHbC0xEMeEhOy0sWvblT1BaqrS941RuAbiaVBn/
nzltKF1WmYHfX2OK6GummYmYkT8QB1TC6ir9etlT54h/w7Hjc3eysGoCLGUr2V/q
kEUou4ut2KiVfUUxyG05Ec3BtyZ+4ZV0fdnAXS0CgYB9j2LHDHOk31b09wygGyDi
65jHGtQsqqr9d5cFSjYkxHNTdO2FP02lRBdMmUjEtLQ4v+9R7umjRZCSnLtH1lPt
sq6njDjae11atqKmm1EAhSG0s4G4gDoAwCCIThQ179PFvWyZU9UwJnM2HgSLUI+B
7/zWZJCKeAb+IW+2QSx3SQKBgEo3UqVc09LD1Mxmov33cy8gYUHXuVAnl6vXsB6S
3An7TKTLcdO4LraafrFQhmJpEgmoWhRtrJqqkyTEuxjfmsvaox9HuQKbg1mecrxG
vmgh71ALwj+MSdLdxT5t9toGKwAWEItd3dWLWdgKtjuQgh34S8uaG5eJ2aB+r2e7
eG0dAoGAX/kgK4WBBH/Paa0g4l5cHD99Lgq02IxCMODG73Vbujxu72spJElQzZGd
eyD3XjHwpa7uL6hzIcXXSpfjLxBCQRu37s7KXbYRWC7YBhV6RlUZlPkrMKr5d+68
vx5dwrelvikoKvhf3sCRbHgD0xdJtUkx0eOD/5kJAbWuh/VWJxE=
-----END RSA PRIVATE KEY-----
-----BEGIN PUBLIC KEY-----
MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAsYU/MoIxYeVuCFcrX+fE
H5XX/he/nJTSQ6P4RF38DKLNnNtQ9XfnaMEPk2EXgSIHk+VXhE9vZNxsV7Emjg6C
+eRxUB1NDBtWGX/Ts2cOYD8ZhSumhOplKKEmdEkzGU/Rem3myfB2BQeN6RSRFigZ
fsR2I4oaQ5YDWJ3uk6Ix0XN4zQwmtw1t9W9tgDZATDLW0+8uU02Wc2HrIcyOvDqJ
DXz/XVY+F+pkrws/ygOhxtVdNY7r3meSKjSUVkOiXMdaqK+20l/YrvoZRVe6HEIT
pu9t/x6YDaoTThsbptjkAKBYpSG2UeXJnnQMyKD2GT3Ouyiupv7lPovvLZZiqfVI
WQIDAQAB
-----END PUBLIC KEY-----
and a token created on jwt.io, the following (Java) code using jjwt verifies and parses the token:
String pubkeyb64 = // note BEGIN/END lines and linebreaks from PEM removed
"MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAsYU/MoIxYeVuCFcrX+fE"+
"H5XX/he/nJTSQ6P4RF38DKLNnNtQ9XfnaMEPk2EXgSIHk+VXhE9vZNxsV7Emjg6C"+
"+eRxUB1NDBtWGX/Ts2cOYD8ZhSumhOplKKEmdEkzGU/Rem3myfB2BQeN6RSRFigZ"+
"fsR2I4oaQ5YDWJ3uk6Ix0XN4zQwmtw1t9W9tgDZATDLW0+8uU02Wc2HrIcyOvDqJ"+
"DXz/XVY+F+pkrws/ygOhxtVdNY7r3meSKjSUVkOiXMdaqK+20l/YrvoZRVe6HEIT"+
"pu9t/x6YDaoTThsbptjkAKBYpSG2UeXJnnQMyKD2GT3Ouyiupv7lPovvLZZiqfVI"+
"WQIDAQAB";
String jws = "eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiYWRtaW4iOnRydWUsImlhdCI6MTUxNjIzOTAyMn0.ry5trgLoSgYZt17NOttSkyFhXIKBHjw3OZy8SK_uwukUCMCSh6O5njlRhTA306lXTFrUkb4VCKYk43y_3vTBDzNA7aAZEuwl0pL2O2gfvvcytXnHsnkjaPiigUvNF-pqpvJkOXhTXobKq-76m7ozHTabxbIO0OM4PTkte_2c2922U22qAFuVmfuKRqr3rsMwp24QImppmWsa_rElSqGhc6wIDGwrtdTIFe311Fi8o7L7wWIswRvH4WlsLG6fLGv6yNt-e2Ld5X1kAfozaZqy-hdR9vpq9LqWVWId6VhJ4kayPVnImevctyMY14TlVlwJdRBBgTAdVVl9QHFbMgwozg";
byte[] pubkeyder = Base64.getDecoder().decode(pubkeyb64);
PublicKey pubkey = KeyFactory.getInstance("RSA").generatePublic(new X509EncodedKeySpec(pubkeyder));
Claims parsed = Jwts.parserBuilder().setSigningKey(pubkey).build().parseClaimsJws(jws).getBody();
System.out.println (parsed);
Modification to Kotlin, if you need that, should be trivial.

What is/are the equivelant openssl command(s) to do the following in Java code

Hi I have the following java code.
What is the Openssl commands equivalent. I want to be able to run from OS command line.
This is what i think the openssl command equivalent is... but I am not entirely sure.
openssl dgst -sha256 -sign my_private.key -out /tmp/sign.sha256 codeTosign.txt
When i try both and then compare them
sign.sha256
compare to output from
System.out.println("Signature="+base64Signature);
they are not equal / do not match.
import java.security.KeyFactory;
import java.security.Signature;
import java.security.spec.PKCS8EncodedKeySpec;
import java.util.Base64;
public class SHA256RSA {
public static void main(String[] args) throws Exception {
String input = "sample input";
// Not a real private key! Replace with your private key!
String strPk = "-----BEGIN PRIVATE KEY-----\nMIIEvwIBADANBgkqhkiG9"
+ "w0BAQEFAASCBKkwggSlAgEAAoIBAQDJUGqaRB11KjxQ\nKHDeG"
+ "........................................................"
+ "Ldt0hAPNl4QKYWCfJm\nNf7Afqaa/RZq0+y/36v83NGENQ==\n"
+ "-----END PRIVATE KEY-----\n";
String base64Signature = signSHA256RSA(input,strPk);
System.out.println("Signature="+base64Signature);
}
// Create base64 encoded signature using SHA256/RSA.
private static String signSHA256RSA(String input, String strPk) throws Exception {
// Remove markers and new line characters in private key
String realPK = strPk.replaceAll("-----END PRIVATE KEY-----", "")
.replaceAll("-----BEGIN PRIVATE KEY-----", "")
.replaceAll("\n", "");
byte[] b1 = Base64.getDecoder().decode(realPK);
PKCS8EncodedKeySpec spec = new PKCS8EncodedKeySpec(b1);
KeyFactory kf = KeyFactory.getInstance("RSA");
Signature privateSignature = Signature.getInstance("SHA256withRSA");
privateSignature.initSign(kf.generatePrivate(spec));
privateSignature.update(input.getBytes("UTF-8"));
byte[] s = privateSignature.sign();
return Base64.getEncoder().encodeToString(s);
}
}
references : https://www.quickprogrammingtips.com/java/how-to-create-sha256-rsa-signature-using-java.html

Aside from the difference in input (an in-memory variable versus file contents, which I assume you have made exactly the same), your Java code base64 encodes the result (and sticks Signature= on the front) but openssl dgst -sign does not.
The simplest way is to have Java write the binary (without encoding it), but then you have to compare in binary, for example cmp on Unix or fc /b on Windows; a visual or text-mode comparison will not give reliable results.
You can instead base64-encode the OpenSSL output, but be careful: there are many base64 encoding schemes, which often produce similar but not identical output, thus the output may compare unequal even when the signatures are the same (see below). In particular if that Base64 is Java8+ java.util.Base64 then the default 'basic' encoder uses the classic PEM/MIME alphabet and no line terminators. If you pipe the output of openssl dgst -sign into openssl base64 it defaults to the same alphabet but line terminators every 64 characters. If you use openssl base64 -A it should suppress the line terminators. If you have other base64-encoding programs on your system, they may (each) behave differently. If you really want the Signature= on the front, use any appropriate text-manipulation program like sed awk perl or even most shells on Unix.
Aside: you should not assume you can verify a signature by comparing it to a newly-generated one. The scheme used here, RSASSA-PKCS1-v1_5 aka 'classic type 01', happens to be deterministic, and comparing happens to work, but many other digital signature schemes are not deterministic. The correct way to verify a signature is to apply the verification process defined by the signature scheme. For examples, Java Signature.initVerify(pubkeyobj), update(databuf), verify(signaturebuf) or OpenSSL commandline dgst -$hash <datafile -verify pubkeyfile -signature file .

Thanks every one for help.
The ultimate problem i had in the end was when i copied an pasted string
"sample input" it was appending a "\n".
When this was removed. The output of the openssl command matched the output of the java snippet.
Thanks all.

What is the openssl equivalent of using java's KeyPairGenerator?

Firstly I tried to ask this on security -- I got some upvotes but it seems it has had no answer for a week now. I understand that this is openssl related, however it stems from using a java KeyPairGenerator object so I feel it may be valid for stack overflow. Please see the code below:
I have been using java's KeyPairGenerator in order to generate public/private keys within a program so that I can encrypt and decrypt files(also using java encrypt/decrypt methods). I want to be able to move over to using openssl in order to generate these public private keypairs however I keep getting padding exceptions when I decrypt my files if i used command line generated openssl keys. For example instead of using java's KeyPairGenerator, I try and use openssl to generate keys:
openssl rsa -in keypair.pem -outform DEF -pubout -out public.der
openssl pkcs8 -topk8 -nocrypt -in keypair.pem -outform DER -out private.der
Where I try to use the DER files to encrypt/decrypt my files. Ultimately every key format I have tried seems to give me problems.
I'm assuming that this means that the format of the keys in my openssl commands do not match up how java's KeyPairGenerator works. Here is a snippet of my key generating code:
public void createPublicPrivateKeys() throws IOException, NoSuchAlgorithmException {
BufferedReader in = new BufferedReader(new InputStreamReader(System.in));
logger.info("Password to encrypt the private key: ");
String password = in.readLine();
logger.info("Generating an RSA keypair.");
// Create an RSA key key pair
KeyPairGenerator keyPairGenerator = KeyPairGenerator.getInstance("RSA");
keyPairGenerator.initialize(4096);
KeyPair keyPair = keyPairGenerator.genKeyPair();
logger.info("Done generating the keypair.\n");
// Write public key out to a file
String publicKeyFilename = "publicKey.der";
logger.info("Public key filename: " + publicKeyFilename);
// Get encoded form of the public key for future usage -- this is X.509 by default.
byte[] publicKeyBytes = keyPair.getPublic().getEncoded();
// Write the encoded public key
FileOutputStream fos = new FileOutputStream(publicKeyFilename);
fos.write(publicKeyBytes);
fos.close();
String privateKeyFilename = "privateKey.der";
logger.info("Private key filename: " + privateKeyFilename);
// Get the encoded form -- PKCS#8 by default.
byte[] privateKeyBytes = keyPair.getPrivate().getEncoded();
// Encrypt the password
byte[] encryptedPrivateKeyBytes = new byte[0];
try {
encryptedPrivateKeyBytes = passwordEncrypt(password.toCharArray(), privateKeyBytes);
} catch (Exception e) {
e.printStackTrace();
}
fos = new FileOutputStream(privateKeyFilename);
fos.write(encryptedPrivateKeyBytes);
fos.close();
}
What is the openssl equivalent command line statement of using java's standard KeyPairGenerator? Also note that using outside packages such as bouncy castle is not an option.

OpenSSL commandline rsa or pkcs8 do not generate a keypair; they only convert from one form to another and/or display. For RSA (only), genrsa generates a keypair, in OpenSSL's 'legacy' privatekey format which is not (easily) compatible with Java. Your rsa and pkcs8 commands do convert the legacy format to the "X.509" (SPKI) and PKCS#8 DER formats Java prefers. OpenSSL since 1.0.0 also has genpkey which generates keys, or where applicable parameters, for all supported asymmetric algorithms including RSA, and defaults to PKCS#8 output.
Your Java code has a passwordEncrypt step that is not standard Java and not explained. OpenSSL library supports password-based encryption of PKCS#8 according to that standard if that is what your passwordEncrypt does, but most OpenSSL commandline functions don't.
If you are trying to use the private.der of your example in Java which is trying to do any kind of password-based (or other) decryption on it, that won't work because it isn't encrypted; it isn't even in the PKCS#8 structure used for an encrypted key.
However that error would occur before you even try to decrypt any data, or more likely a working key.

Convert a PKCS#8 private key to PEM in java

Hello everyone I'm trying to convert a PKCS#8 private key that I generate in my java program to a PEM encoded file.
Security.addProvider(new BouncyCastleProvider());
SecureRandom rand = new SecureRandom();
JDKKeyPairGenerator.RSA keyPairGen = new JDKKeyPairGenerator.RSA();
keyPairGen.initialize(2048, rand);
KeyPair keyPair = keyPairGen.generateKeyPair();
PEMWriter privatepemWriter = new PEMWriter(new FileWriter(new File(dir + "private.key")));
privatepemWriter.writeObject(keyPair.getPrivate());
After running the program I have the private key in both formats and a public key(the code isn't shown as it works). I then use this openssl command to conver the private.key back to a pem formated file.
openssl pkcs8 -nocrypt -inform DER -in private.key -out private2.pem
When I compare private.pem and private2.pem they are different and obviously when I try to use private.pem it says it's not a valid file.
What step am I missing in order to properly convert this private key into the PEM format that I need? I can't use OpenSSL from within my program, otherwise I would simply add that function call. I have access to BouncyCastle libs in this program, so maybe it has a solution I'm overlooking.

You can use the PEMWriter class in Bouncycastle.

The fact that OpenSSL uses it's own format is really the only thing that makes this challenging. Thankfully the bouncy castle PEMWriter makes this easy, but the interface isn't very well documented. I found some code by searching through the mailing list. I've adapted it below:
KeyPairGenerator keyGen = KeyPairGenerator.getInstance("RSA");
keyGen.initialize(2048);
KeyPair keyPair = keyGen.generateKeyPair();
StringWriter stringWriter = new StringWriter();
PEMWriter pemWriter = new PEMWriter(stringWriter);
pemWriter.writeObject( keyPair.getPrivate());
pemWriter.close();
privateKeyString = stringWriter.toString();

Use the header:
-----BEGIN PRIVATE KEY-----
… and the footer:
-----END PRIVATE KEY-----
Note that the "RSA" is left out—The Java code is using PKCS #8 encoding for the private key, and that encoding includes the algorithm.
The openssl command that you show is converting a standard PKCS #8 key in DER form to a proprietary OpenSSL key in PEM form. To keep the PKCS #8 format, but convert from DER to PEM, add the -topk8 option. Then the OpenSSL output should match what your Java code is producing.
If you need to produce the OpenSSL key, instead of PKCS #8, it's possible, but you'll have to create your own OpenSSL structure with the BouncyCastle ASN.1 library and encode that. Please clarify if that's what you need.