We Keep Coding

decompressing with zlib in Java (incorrect header check)

I try to program (and understand) compression/decompression.
I have a file which is compressed with zlib and I thought that I found the solution to decompress my file:
import java.util.Scanner;
import java.util.zip.*;
import java.io.*;
public class ZLibCompression
{
public static void main(String args[])throws IOException, DataFormatException {
File compressed = new File("./MyFile.hlb");
decompress(compressed, new File("./MyFile.txt"));
}
public static void decompress(File compressed, File raw)
throws IOException
{
try (InputStream inputStream = new InflaterInputStream(new FileInputStream(compressed));
OutputStream outputStream = new FileOutputStream(raw))
{
copy(inputStream, outputStream);
}
}
private static void copy(InputStream inputStream, OutputStream outputStream)
throws IOException
{
byte[] buffer = new byte[1000];
int length;
while ((length = inputStream.read(buffer)) > 0)
{
outputStream.write(buffer, 0, length);
}
}
}
But I get the following error stack trace:
Exception in thread "main" java.util.zip.ZipException: incorrect header check
at java.base/java.util.zip.InflaterInputStream.read(InflaterInputStream.java:164)
at java.base/java.io.FilterInputStream.read(FilterInputStream.java:106)
at ZLibCompression.copy(ZLibCompression.java:46)
at ZLibCompression.decompress(ZLibCompression.java:20)
at ZLibCompression.main(ZLibCompression.java:11)
Then I checked the header of my file and it says:
{
"compression" : {
"crc32" : 2575274738,
"decompressed_size" : 9020404,
"type" : "zlib"
},
"encoded_data" : "eNrsvV2Xm0i
What is my error? I found a Python script that works fine with the same file:
#!/usr/bin/env python
import sys
import os
import json
import base64
import zlib
SETLIST_OR_BUNDLE = "MyFile.hlb"
infile = open(SETLIST_OR_BUNDLE)
data = json.load(infile)
infile.close()
keys = list(data.keys())
if 'encoded_data' in keys:
unz = zlib.decompress(base64.b64decode(data['encoded_data']))
setlist_or_bundle = json.loads(unz)
keys = list(setlist_or_bundle.keys())
if 'setlists' in keys:
setlists = setlist_or_bundle['setlists']
elif 'presets' in keys:
setlists = [setlist_or_bundle]
for setlist in setlists:
keys = list(setlist.keys())
if 'meta' in keys:
print()
print("SETLIST: %s" % (setlist['meta']['name']))
presets = setlist['presets']
#print json.dumps(presets, indent=4)
for preset in presets:
if 'meta' in list(preset.keys()):
meta = preset['meta']
preset_name = meta['name']
print(" ", preset_name)
I think it has something to do with the base64 part and I found a similar question where someone mentioned "you have to decode the Base64 string into a byte array first" - OK fine - Can anyone explain or give me a link to a tutorial?
All I need is the same functionality in Java like the Python script above has - And of course I want to learn something...

First of all, it looks like your file is not compressed as a whole. Instead, it is a JSON-String containing the actual compressed data as encoded_data. You also need to unwrap the JSON-String then. The easiest way to deal with JSON encrypted data is by using a library. Check this post for some comparisons of different libraries.
Next, as you can see in your python code, the encoded data gets decoded from Base64 before passed through the ZLIB-Decompressor (zlib.decompress(base64.b64decode(data)))
The java equivalent to un-Base64 a String would be:
Base64.getDecoder().decode(string);

How to refactor Java based base64 encoding to Python3?

We have a Java code that creates passwords for MariaDB. Now we have a Python3 REST API which in some sort of way creates credentials for MariaDB too. The problem is that patterns after creation don't match. We should follow the Java code algorithm.
JAVA code:
private AccountInfo retriveAccountInfo(InetAddress addr, String login, String password, boolean autoCreateIfEnabled) {
try {
final var md = MessageDigest.getInstance("SHA");
final var raw = password.getBytes(UTF_8);
final var hashBase64 = Base64.getEncoder().encodeToString(md.digest(raw));
...
exampleoutput : 'ALSenzd4DnVgfBW9Vdx+fvjxr+c='
How to achieve the same results using Python3 code?
Edit: the answer
import hashlib
import base64
password = 'test'
hashBase64 = base64.b64encode(hashlib.sha1(password.encode('utf8')).digest())
print(hashBase64)
Thanks for the help.

How to generate Signature in AWS from Java

When I invoke API endpoints from REST client, I got error by concerning with Signature.
Request:
Host: https://xxx.execute-api.ap-southeast-1.amazonaws.com/latest/api/name
Authorization: AWS4-HMAC-SHA256 Credential={AWSKEY}/20160314/ap-southeast-1/execute-api/aws4_request,SignedHeaders=host;range;x-amz-date,Signature={signature}
X-Amz-Date: 20160314T102915Z
Response:
{
"message": "The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details. The Canonical String for this request should have been 'xxx' "
}
From Java code, I followed AWS reference of how to generate Signature.
String secretKey = "{mysecretkey}";
String dateStamp = "20160314";
String regionName = "ap-southeast-1";
String serviceName = "execute-api";
byte[] signature = getSignatureKey(secretKey, dateStamp, regionName, serviceName);
System.out.println("Signature : " + Hex.encodeHexString(signature));
static byte[] HmacSHA256(String data, byte[] key) throws Exception {
String algorithm="HmacSHA256";
Mac mac = Mac.getInstance(algorithm);
mac.init(new SecretKeySpec(key, algorithm));
return mac.doFinal(data.getBytes("UTF8"));
}
static byte[] getSignatureKey(String key, String dateStamp, String regionName, String serviceName) throws Exception {
byte[] kSecret = ("AWS4" + key).getBytes("UTF8");
byte[] kDate = HmacSHA256(dateStamp, kSecret);
byte[] kRegion = HmacSHA256(regionName, kDate);
byte[] kService = HmacSHA256(serviceName, kRegion);
byte[] kSigning = HmacSHA256("aws4_request", kService);
return kSigning;
}
May I know what I was wrong while generating Signature?
Reference how to generate Signature : http://docs.aws.amazon.com/general/latest/gr/signature-v4-examples.html#signature-v4-examples-java

You can use classes from aws-java-sdk-core: https://github.com/aws/aws-sdk-java/tree/master/aws-java-sdk-core
More specifically, Request, Aws4Signer and a few other ones:
//Instantiate the request
Request<Void> request = new DefaultRequest<Void>("es"); //Request to ElasticSearch
request.setHttpMethod(HttpMethodName.GET);
request.setEndpoint(URI.create("http://..."));
//Sign it...
AWS4Signer signer = new AWS4Signer();
signer.setRegionName("...");
signer.setServiceName(request.getServiceName());
signer.sign(request, new AwsCredentialsFromSystem());
//Execute it and get the response...
Response<String> rsp = new AmazonHttpClient(new ClientConfiguration())
.requestExecutionBuilder()
.executionContext(new ExecutionContext(true))
.request(request)
.errorResponseHandler(new SimpleAwsErrorHandler())
.execute(new SimpleResponseHandler<String>());
If you want a cleaner design, you can use the Decorator pattern to compose some elegant classes and hide the above mess. An example for that here: http://www.amihaiemil.com/2017/02/18/decorators-with-tunnels.html

From the code example above it looks like you are not creating a canonical request and including it in the string that gets signed as per http://docs.aws.amazon.com/general/latest/gr/sigv4-create-canonical-request.html
Instead of implementing this yourself have you looked at using a third-party library.
aws-v4-signer-java is a lightweight, zero-dependency library that makes it easy to generate AWS V4 signatures.
String contentSha256 = "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855";
HttpRequest request = new HttpRequest("GET", new URI("https://examplebucket.s3.amazonaws.com?max-keys=2&prefix=J"));
String signature = Signer.builder()
.awsCredentials(new AwsCredentials(ACCESS_KEY, SECRET_KEY))
.header("Host", "examplebucket.s3.amazonaws.com")
.header("x-amz-date", "20130524T000000Z")
.header("x-amz-content-sha256", contentSha256)
.buildS3(request, contentSha256)
.getSignature();
Disclaimer: I'm the libraries author.

This is possible using 100% java libraries without additional dependencies, just use the query parameters generated here:
import java.security.InvalidKeyException;
import java.security.NoSuchAlgorithmException;
import java.security.SignatureException;
import java.util.Formatter;
import javax.crypto.Mac;
import javax.crypto.spec.SecretKeySpec;
import java.util.Base64;
...
private static final String ACCESS_KEY = "...";
private static final String SECRET_KEY = "...";
private static final int expiresTime = 1 * 24 * 60 * 60;
private static final String HMAC_SHA1_ALGORITHM = "HmacSHA1";
public void sign(String protocol, String bucketName, String contentPath) throws Exception {
Calendar cal = Calendar.getInstance();
cal.add(Calendar.HOUR_OF_DAY, 24);
String host = bucketName + ".s3-us-west-2.amazonaws.com";
long expireTime = cal.getTimeInMillis() / 1000;
String signString = "GET\n" +
"\n" +
"\n" +
expireTime + "\n" +
"/" + bucketName + contentPath;
SecretKeySpec signingKey = new SecretKeySpec(SECRET_KEY.getBytes(), HMAC_SHA1_ALGORITHM);
Mac mac = Mac.getInstance(HMAC_SHA1_ALGORITHM);
mac.init(signingKey);
String signature = URLEncoder.encode(new String(Base64.getEncoder().encode(mac.doFinal(signString.getBytes()))));
System.out.println(signature);
String fullPayload = "?AWSAccessKeyId=" + ACCESS_KEY +
"&Expires=" + expireTime +
"&Signature=" + signature;
System.out.println(protocol + "://" + host + contentPath + fullPayload);
}
...

The signing process is lengthy and error-prone, here are some tips
Make sure your access key and secret is correct, try to use Postman to test the request at first, it's easy and fast, see Use Postman to Call a REST API
Make sure you use UTC time
The signing process uses both timestamp(YYYYMMDD'T'HHMMSS'Z') and datetime(YYYYMMDD), so double check your implementation for that
Use any online hash tool to verify your hash algorithm behaves as expected
Read the python implementation carefully, see Examples of the Complete Version 4 Signing Process (Python)
See my fully working java implementation on Github - A Java(SpringBoot) template for Java and AWS SageMaker DeepAR model endpoint invocation integration

You may investigate code samples that is shared by AWS web site. I used some of the util classes and a few java class I need. So you don't have to use all classes and other stuff. I left the link below.
AWS Java Samples in doc of Amazon

For me, in Java, the following code worked to generate a signed request to sent to web socket client via api gateway -
Request<Void> request = new DefaultRequest<Void>("execute-api"); //Request to API gateway
request.setHttpMethod(HttpMethodName.POST);
request.setEndpoint(URI.create(url));
String bodyContnt= "test data";
InputStream targetStream = new ByteArrayInputStream(bodyContnt.getBytes());
request.setContent(targetStream);
//Sign it...
AWS4Signer signer = new AWS4Signer();
signer.setRegionName("ap-south-1");
signer.setServiceName(request.getServiceName());
signer.sign(request, new Creds());
signer.setOverrideDate(new Date()); // needed as current ts is required
//Execute it and get the response...
Response<String> rsp = new AmazonHttpClient(new ClientConfiguration())
.requestExecutionBuilder()
.executionContext(new ExecutionContext(true))
.request(request)
.errorResponseHandler(new SimpleAwsErrorHandler(true))
.execute(new SimpleResponseHandler());

computing checksum for an input stream

I need to compute checksum for an inputstream(or a file) to check if the file contents are changed. I have this below code that generates a different value for each execution though I'm using the same stream. Can someone help me to do this right?
public class CreateChecksum {
public static void main(String args[]) {
String test = "Hello world";
ByteArrayInputStream bis = new ByteArrayInputStream(test.getBytes());
System.out.println("MD5 checksum for file using Java : " + checkSum(bis));
System.out.println("MD5 checksum for file using Java : " + checkSum(bis));
}
public static String checkSum(InputStream fis){
String checksum = null;
try {
MessageDigest md = MessageDigest.getInstance("MD5");
//Using MessageDigest update() method to provide input
byte[] buffer = new byte[8192];
int numOfBytesRead;
while( (numOfBytesRead = fis.read(buffer)) > 0){
md.update(buffer, 0, numOfBytesRead);
}
byte[] hash = md.digest();
checksum = new BigInteger(1, hash).toString(16); //don't use this, truncates leading zero
} catch (Exception ex) {
}
return checksum;
}
}

You're using the same stream object for both calls - after you've called checkSum once, the stream will not have any more data to read, so the second call will be creating a hash of an empty stream. The simplest approach would be to create a new stream each time:
String test = "Hello world";
byte[] bytes = test.getBytes(StandardCharsets.UTF_8);
System.out.println("MD5 checksum for file using Java : "
+ checkSum(new ByteArrayInputStream(bytes)));
System.out.println("MD5 checksum for file using Java : "
+ checkSum(new ByteArrayInputStream(bytes)));
Note that your exception handling in checkSum really needs fixing, along with your hex conversion...

Check out the code in org/apache/commons/codec/digest/DigestUtils.html

Changes on a file are relatively easy to monitor, File.lastModified() changes each time a file is changed (and closed). There is even a build-in API to get notified of selected changes to the file system: http://docs.oracle.com/javase/tutorial/essential/io/notification.html
The hashCode of an InputStream is not suitable to detect changes (there is no definition how an InputStream should calculate its hashCode - quite likely its using Object.hashCode, meaning the hashCode doesn't depend on anything but object identity).
Building an MD5 like you try works, but requires reading the entire file every time. Quite a performance killer if the file is large and/or watching for multiple files.

You are confusing two related, but different responsibilities.
First you have a Stream which provides stuff to be read. Then you have a checksum on that stream; however, your implementation is a static method call, effectively divorcing it from a class, meaning that nobody has the responsibility for maintaining the checksum.
Try reworking your solution like so
public ChecksumInputStream implements InputStream {
private InputStream in;
public ChecksumInputStream(InputStream source) {
this.in = source;
}
public int read() {
int value = in.read();
updateChecksum(value);
return value;
}
// and repeat for all the other read methods.
}
Note that now you only do one read, with the checksum calculator decorating the original input stream.

The issue is after you first read the inputstream. The pos has reach the end. The quick way to resolve your issue is
ByteArrayInputStream bis = new ByteArrayInputStream(test.getBytes());
System.out.println("MD5 checksum for file using Java : " + checkSum(bis));
bis = new ByteArrayInputStream(test.getBytes());
System.out.println("MD5 checksum for file using Java : " + checkSum(bis));

How can I save MessageDigest internal state into database?

Is it possible, and if than how, to save the internal state of MessageDigest object? I want to save it in a database, so have to use only primitive data like String, int, byte[].
What I'm trying to achieve is to be able to receive a fragmented file (during a long period of time), save all the fragments in database, and after receiving last fragment verify the SHA512 digest of the file without getting back all the data previously saved in database.
So basically I want something like this:
MessageDigest md = MessageDigest.getInstance("SHA-512");
// restore previous internal state of md
md.update(dataSegment);
// save internal md state

you could serialize the object to String (XML format) and return it back.
check:
http://x-stream.github.io/tutorial.html
public class DigestTest {
private static final byte[] TEST_DATA = "Some test data for digest computations".getBytes();
#Test
public void shouldStoreAndRestoreDigest() throws Exception {
final MessageDigest referenceDigest = MessageDigest.getInstance("SHA-512");
MessageDigest testDigest = MessageDigest.getInstance("SHA-512");
referenceDigest.update(TEST_DATA);
testDigest.update(TEST_DATA);
// store state
final XStream xs = new XStream(new StaxDriver());
xs.alias("md", MessageDigest.class);
final String serializedMd = xs.toXML(testDigest);
System.out.println(serializedMd);
// restore state
testDigest = (MessageDigest)xs.fromXML(serializedMd);
// ---
referenceDigest.update(TEST_DATA);
testDigest.update(TEST_DATA);
Assert.assertArrayEquals(referenceDigest.digest(), testDigest.digest());
}
}