We Keep Coding

Writing StringBuilder to AWS S3 File

I have a StringBuilder (~1GB size) which converts to an almost 1GB String. Then it is sent to AWSS3Client to be put as a file on s3. Now to write the String to s3, I need to further convert the string to a ByteArrayInputStream, which takes another 1GB.
InputStream dataInputStream = new ByteArrayInputStream(data.getBytes(StandardCharsets.UTF_8));
amazonS3Client.putObject(bucket, key, dataInputStream, new ObjectMetadata());
Now I end up with a JVM heap of 3GB. Is there a way to directly send the 1GB StringBuilder directly to S3 with little over 1GB heap?
The only way I could think of is to convert StringBuilder to a File instance (using FileWriter) --> S3 API which takes File as input. But this approach requires a 2-IOPS to the SSD -- penalty that I want to avoid.
Any better way to handle the problem is welcome.
[edit 1]
(as per comments) The builder size changed from 5GB to 1GB. I wanted to convey that the builderis a huge one (and hence used 5GB previously, which was my mistake)

The string-oriented classes are frustratingly locked down with final in many cases, which means that oftentimes buffers that already exist in memory have to be duplicated. This is probably to do with security but is nevertheless wasteful and frustrating. You should be able to do something with the following if you have a StringBuilder. Obviously you'll need a much bigger buffer than in the illustrative main method.
import java.io.ByteArrayInputStream;
import java.io.FilterInputStream;
import java.io.FileInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.io.FileOutputStream;
import java.nio.ByteBuffer;
import java.nio.CharBuffer;
import java.nio.charset.CharsetEncoder;
import java.nio.charset.Charset;
import java.util.Objects;
/**
*
* #author CEHJ
* #version 1.0
*/
public class CharSequenceInputStream extends InputStream {
private CharSequence charSequence;
private int position;
private CharsetEncoder enc;
private Charset outputEncoding;
private CharBuffer cb;
private ByteBuffer bb;
public CharSequenceInputStream(CharSequence charSequence) {
this(charSequence, Charset.forName(System.getProperty("native.encoding", Charset.defaultCharset().name())));
}
public CharSequenceInputStream(CharSequence charSequence, Charset outputEncoding) {
this.charSequence = charSequence;
this.outputEncoding = outputEncoding;
position = 0;
enc = outputEncoding.newEncoder();
cb = CharBuffer.allocate(1);
bb = ByteBuffer.allocate(8);
}
// For testing only
public static void main(String[] args) throws IOException {
if (args.length < 2) {
System.err.printf("Usage: java CharSequenceInputStream <String to be stored> <Output file> [Char encoding for output]");
System.exit(1);
}
String content = args[0];
String outputPath = args[1];
StringBuilder sb = new StringBuilder(content);
CharSequenceInputStream in = null;
if (args.length > 2) {
in = new CharSequenceInputStream(sb, Charset.forName(args[2]));
} else {
in = new CharSequenceInputStream(sb);
}
try (OutputStream out = new FileOutputStream(outputPath)) {
int bytesRead = -1;
final int BUF_SIZE = 8;
byte[] buf = new byte[BUF_SIZE];
while ((bytesRead = in.read(buf, 0, BUF_SIZE)) > -1) {
out.write(buf, 0, bytesRead);
}
}
}
#Override
public int read() throws IOException {
int result = -1;
if (position < charSequence.length()) {
if (bb.remaining() == bb.capacity() || bb.position() == bb.limit()) {
// At start - nothing yet has been decoded
// OR - we've read all bytes in the buffer and need to refill it
cb.clear();
bb.clear();
char currentChar = charSequence.charAt(position++);
cb.append(currentChar);
cb.flip();
enc.reset();
enc.encode(cb, bb, false);
enc.encode(cb, bb, true);
enc.flush(bb);
bb.flip();
}
}
if (bb.position() < bb.limit()) {
result = bb.get() & 0xFF;
}
return result;
}
#Override
public int read(byte[] b) throws IOException {
return read(b, 0, b.length);
}
#Override
public int read(byte b[], int off, int len) throws IOException {
Objects.checkFromIndexSize(off, len, b.length);
if (len == 0) {
return 0;
}
int c = read();
if (c == -1) {
return -1;
}
b[off] = (byte) c;
int i = 1;
try {
for (; i < len; i++) {
c = read();
if (c == -1) {
break;
}
b[off + i] = (byte) c;
}
} catch (IOException ee) {
}
return i;
}
#Override
public void close() {
// NO OP
}
}

Incorrect decompression of Java Gzip in .NET

I'm trying to repeat in .NET the algorithm that was originally written in Java and I'm having troubles with the GZIP decompression.
At the bottom of the post I inserted the hex string that is converted to byte array in both .NET and Java. The resulting byte array is then decompressed in Java with the following method:
public static Object readObjectFromByte(byte[] bytes)
{
ObjectInputStream oos = null;
try {
ByteArrayInputStream baos = new ByteArrayInputStream(bytes);
zis = new GZIPInputStream(baos);
oos = new ObjectInputStream(zis);
return oos.readObject();
} catch (Throwable t) { GZIPInputStream zis;
return null;
} finally {
try {
if (oos != null) {
oos.close();
}
} catch (IOException e) {
e.printStackTrace();
}
}
}
After decompression the resulting byte array has a length of 3952 which is probably correct. At the same time I tried different .NET classes/libs to decompress, but it always gives a byte array of 3979 bytes which is probably incorrect.
I tried:
GZipStream
DotNetZip
SevenZipLib
SharpZipLib
I read a lot of articles about GZIP issues in .NET trying to fix this. I use .NET 4.5, and for example my last decompression version is this:
Ionic.Zlib.GZipStream.UncompressBuffer(compressedBytes)
It's weird but even if I try:
Ionic.Zlib.GZipStream.CompressBuffer(Ionic.Zlib.GZipStream.UncompressBuffer(compressedBytes)).SequenceEquals(compressedBytes)
It gives me FALSE.
The hex string:
EDIT:
Java Code:
import java.io.ByteArrayInputStream;
import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.ObjectInputStream;
import java.io.ObjectOutputStream;
import java.util.zip.GZIPInputStream;
import java.util.zip.GZIPOutputStream;
public class JavaFiddle
{
public static void main(String[] args)
{
String hex = "PLEASE_UPDATE"; //update this from the hex constant at the end of the post
byte[] compressedBytes = hexStringToByteArray(hex);
byte[] decompressedBytes = (byte[])readObjectFromByte(compressedBytes);
System.out.println(decompressedBytes.length); //THIS GIVES 3952
}
public static byte[] hexStringToByteArray(String s) {
int len = s.length();
byte[] data = new byte[len / 2];
for (int i = 0; i < len; i += 2) {
data[i / 2] = (byte) ((Character.digit(s.charAt(i), 16) << 4)
+ Character.digit(s.charAt(i+1), 16));
}
return data;
}
public static Object readObjectFromByte(byte[] bytes)
{
ObjectInputStream oos = null;
try {
ByteArrayInputStream baos = new ByteArrayInputStream(bytes);
GZIPInputStream zis = new GZIPInputStream(baos);
oos = new ObjectInputStream(zis);
return oos.readObject();
} catch (Throwable t) { GZIPInputStream zis;
return null;
} finally {
try {
if (oos != null) {
oos.close();
}
} catch (IOException e) {
e.printStackTrace();
}
}
}
}
.NET Code
private byte[] StringToByteArray(string hex)
{
int NumberChars = hex.Length;
byte[] bytes = new byte[NumberChars / 2];
for (int i = 0; i < NumberChars; i += 2)
bytes[i / 2] = Convert.ToByte(hex.Substring(i, 2), 16);
return bytes;
}
...
var hex = "PLEASE_UPDATE"; //update this from the hex constant at the end of the post
var compressedBytes = StringToByteArray(hex);
var decompressedBytes = Ionic.Zlib.GZipStream.UncompressBuffer(compressedBytes);
//decompressedBytes.Length is 3979, Note that this is using one of the external libraries, the same result is for built-in GZipStream in .NET
1F8B0800000000000000ADCEF923130A0000E0596C8A1CB90AA1D0E428916C4AE6889C61071E1D2A77B42C67D1731BF2DE6C662C5744C6DE7BC8DC4791DCB79E35315EC2C62437E1FDBE9FF7FD055FC53240282418007437AB5893DB060923594000201C030088639882D19C31C860FB7838A60DBBF039C3B41821FF606044C89D24EB4C1D6A627FBF8E978D3616796C2ACDD165ABB32B81A0C493A35F163889E74665A9341C7C36F06FA0F991201BEAAB6A47842C43A50DA6B8C000AD794840CF03E29A527D0C921FAE6D607C71AEF7C01DB34037A7F8CDD8CB1F61DB9C15A87DBBDD1F933FDF58312B054D7B5AF72FE75A0AA101714EEA6980BF405A15A1F13A0979FB5BBE1636A367CB4C943FB7F87A3245573BB6857DA6AA0B412CCB553B39E06973EB2A3864E98915E46C16E4EE06E9DA63DB642915AF6E86811FD2C8F3A1F325C82677EA701D1E3574885696DEB3B8A0C29AE1FDD41DA26C55C6381506A3CF77DD15BD37783F545F76E8DCF9FB6A8B0776E2427FBFF431B73748CE96786EA569E307B5DC8990A68A52838D9DC1BA19291F283BF2C7EE43D3E4516D77A07FDE04934BB7C254057BDFC3AD153A0A711E7EFDDDED31AED367FAAEA6F494A55E029BF6AECA9433CA7D45B52EC24BCF9074315BFE5F5494F4F0DDDE8D73F56397A0667992950E62AD4020A4102DBE7C4D6B85E457F184DC5AA51D2D536313E1E16636928BD6500207571B83C8830DB9E7CF2C92AE4DDB12B8DF3C6AAF83B09CAACB78AA78CDB0DB393D71A599F185B413BFDB907FC99598D2E56ADE701EE72DB71C9650E820DB452F7DD4904E4679FFD0CB6FEEDF7527AF21BE5A5404B8BAE4E9951AE5C88BF9B5D5FFC6447989445C7A2DB80B3E4A2DBCB51978BA1729F97BDCAD174D345417B96941BF2CC5C10539C952B2526770BA4637A5820DCF150E16AED3159AD9724A8DFCFEA8BADA2A08BD1A0C9B05B019CE61E01EDECF7F028B466E4B53A5F1FAE56CB137EF269E1946896D1A3DDCDB756364A96D8E47F3FB33E9B70BBED37FA8719F2D2EA89BBFB0CAFB71CC56947F7907439E9FE97C61D2A326F830592509674860BA7CC7E713EFD69D2724DED740A2959846CFE801B313BB1DAB45594D70E888F2A3063C67085BFDB345D64D116372F5BE42BB08ABBC6D65D90E1E50C849EDD9A6F977751C4DB5A454C66B997D6A0CA148827E03AEE30F1697392124A8C44C6A914C3C69C21447F9C5660665C5D0D3F758D57086D0938608E9BECE9BA95D359C41E255B0524C55EDC97816A76E32B29BF55BAE0EC1F71DBF3FF548DC2C181755D85ABCAFEDF23C248EF7D3C1580AB88D28CDCF9935F1DA797145A7F0B18603C1DA39EA8E969A2279F20F077E7F62BD5C1F7C095BF1B21D36561A254836F37EBC0AEB8C8EAE8EC725044EE47F107DE413DE804EED41473E4D3BEC93E9234255F9FDD1FD91F00B32BF13B2629C4373888813E5FD08F66AB7295D281E180ACE99A77952DE6DCD675B20E021A71E91F78B899A1FC2F8FD913C0E4D980B1D6CF13F7A76E823BDF75FDE8FC513C3C491F51305F1A483B81A11884254CB2DB396F864B08DDEDD005A90EF457E7F8EDFB64FF7C7443C6E2135183619ED69F27E1464BF862FB14A66509C379818517F6B2423D0D1BE794E4D9C7130D9F54AB19DDF9FC825CF8B3B5EBD8090811D372C0315CEFBB926701AC2B84CB4820D25DD2D71F2F6630FB3FA88A3EAAF9E44C709EFACC05FF2FBD342E844AB8A22F433B7C2161FEDE70BF07EF29A6B6355DAF2D33E7A2679C8A38511939EEE73DABDE98B021A9F0E21A5CFFEE6F7A7C44CE2358158F5B15F837CE96B75622FEFE702653D87222B9319D46D321EEDA628713D1B355A30E0D07A655F12A55B6397CFEFCFA3DB3B76DB80347FDCA613FD1F3F6022EF276A1666DDA3BE518CB634F67B185BD2016DB2DC20E56C1C9B58CD411710E802FCFED0E3E1E182C37A9684A007EF0DEBBBCB783F37A0D39E2B6FAB147A3BCD75B1C24FA5939BDD9501D5E0365A60E58BE429C821BF3F774DC4603D43CE355412CC3DBF47639DF7B33CFEF4138133255770F2659E44F2FC9EE285FCDCEC414B473CCB31BA39BA798CDF1FF28A3EBD7549341C8E809E8AABF836C7FB59DB4025ED1DC5BF1874EDF4511ECCC397050C8C3F404FB04BF52DF4F35411EBFCFEFCB70C2AF6EE8AFC32A539F624F0E30D47DE8F9138B0C9B24F03F5CD53E05CA77D63CAA8800CA567E4D60D8D4ECBA9F9FE3C75FADED32267AB07ED635AD14CB6A87B1F8D5B5BD43F7C04E14DB30935A6B6CA780F2FBD10CD2EAE8DD13D6C9FA89334A8ADEAC40D63C54B6E881F9F382D3056BF7DACFFC6914A6DF60F64E7C5FE8EB222FD71632318CDC5E3CF8EC48602D238E95FC06C9BDBBD3D85535DED236D1FFA4EB1F3939B334A222BBF7FE034EB1599A7E20DFF89F92E9B3EE0D45FA7103A9B8BE6F727EC6D338C464AAA1B6B8AF426BA2153793FF7C43EEE7A43E6038296B83E130BAA9D467DB7A9B8E29E18B8F4E4493DB3E1BE4CDC1ABA3074A8D69058420FB2F56BCCD3CAB0806FA9C1E825D6F18644CD9137AB9195DD63A8BD3BA9F5C9BF0AEC9F5B6E605B2C02CAA93FB10A5F03ADD7370A981F3A99F0F3CC2DA93C298292E9859C01BBB088FE6CD979FC5A02127305D63C72B319EDD48A0EE6BE431C17B3F8E4FFDAC1F5D9A9F4EB7443EE2D3CAD6A3508A918C48DBB7F89081B7087C94A60F9FD297839305DA999BE163F234E3ED7D8D8CDFBA16AE5327F3E9D5925DF533EF5FEA2F26D3784CFB578877E15799DB73DAEC8D767F9FD81597235251D2DC413C2E1F58887AF75783F8248EC923F57F93D81A88D9E3E1E045997FD3C40B1BF83CEFC9CC2CCC079197928232333623BE67DA6A70B4818095DEEC23ABEC73E4DC9E1A74E04AEC1D420EB533CF3C06C242E1612F85A560A2861A0E54C218AC1834FCCDBD2F61FBC3B32983F25B29F5967A7219C74A10C769002C1869E9DC926A99C9D1AC92CC8BC3A5D151D405693BA0DBE292D94F40B918AFCEE6851DFC215B3738975F1EFAA32923037D96656C2D2B94B015CD1DA5F43D0A8FFA063099BA0BFB246724C8E61150FB2126EDE245963F4454EBCFF5105BDA267D598E2D1E9985136AF4348D8FCF96F355B27AE6E6F5378415D8A8D1CAD3EC3426969BF5790DBA0C01EF9B7E86095D9313F74309E01C7BA00B9B0E85DA89900D6F89DEA5BCA9273C017B56139F9CF13E7DB74724240262CBDFCEC05FB5DDAA64CB98257C9AAF7AA48AAD36577B3D6F11CD0A074D5B15860B2791E8E8B87AB2A5B2419F61561E6ECF56746E2185EBFFF0F96F821B38B0F0000
Thanks,

Now we've got more of the Java code, we can see the problem: you've got an extra layer of serialization around your real data. That has nothing to do with compression really.
Here's an example to show what I mean:
import java.io.*;
public class Test {
public static void main(String[] args) throws Exception {
try (ByteArrayOutputStream output = new ByteArrayOutputStream()) {
try (ObjectOutputStream oos = new ObjectOutputStream(output)) {
oos.writeObject(new byte[5]);
}
byte[] data = output.toByteArray();
System.out.println(data.length);
}
}
}
That's writing a byte array that's 10 bytes long - but the result is 32 bytes long, because of the extra "wrapper" information. Note that the extra 27 bytes is the same as the discrepancy you've seen.
Fundamentally, it's odd to wrap a byte array in this way, and if you can possibly change the original code, that would be for the best. If you absolutely can't do that, it may be safe to just ignore the first 27 bytes of the resulting data.

need to convert my json string to hash value by using SHA1 with base64Encoding Format in java

Am passing JSON Input String which needs to convert as hashvalue by using SHA1 with Base64 Encoder Format..
Like below i have tried,
import java.security.MessageDigest;
import java.security.NoSuchAlgorithmException;
import org.apache.commons.codec.binary.Base64;
import org.json.JSONObject;
import java.util.Base64.Encoder;;
public class HashTextTest {
public static void main(String[] args) throws NoSuchAlgorithmException {
System.out.println(sha1(""));
}
static String sha1(String input) throws NoSuchAlgorithmException {
MessageDigest mDigest = MessageDigest.getInstance("SHA1");
byte[] result = mDigest.digest(input.getBytes());
StringBuffer sb = new StringBuffer();
for (int i = 0; i < result.length; i++) {
sb.append(Integer.toString((result[i] & 0xff) + 0x100, 16).substring(1));
}
return sb.toString();
}
}
But Above code Takes only string input, here am not able to pass JSON String, and its return Hashvalue perfectly but i need that hashvalue in Base64Encoder Format..
help me

Java NIO ByteBuffer: put and get strings

I've the following test code. I'd like to know how I can put, and get a String using a Java NIO ByteBuffer.
I've added two comments where I need help.
package testPipe;
import java.io.UnsupportedEncodingException;
import java.nio.ByteBuffer;
import java.nio.charset.Charset;
import java.nio.charset.CharsetDecoder;
import java.nio.charset.CharsetEncoder;
public class TestMemBuff {
static final String dataFile = "invoicedata";
static final double[] prices = { 19.99, 9.99, 15.99, 3.99, 4.99 };
static final int[] units = { 12, 8, 13, 29, 50 };
static final String[] descs = { "Java T-shirt", "Java Mug",
"Duke Juggling Dolls", "Java Pin", "Java Key Chain" };
public static Charset charset = Charset.forName("UTF-8");
public static CharsetEncoder encoder = charset.newEncoder();
public static CharsetDecoder decoder = charset.newDecoder();
public static void main(String[] args) throws UnsupportedEncodingException {
double price;
int unit;
String desc;
double total = 0.0;
ByteBuffer buf = ByteBuffer.allocate(1024);
for (int i = 0; i < prices.length; i++) {
buf.putDouble(prices[i]);
buf.putInt(units[i]);
buf.asCharBuffer().put(descs[i]); // Is it correct?
}
buf.flip();
// INPUT
while (buf.hasRemaining()) {
price = buf.getDouble();
unit = buf.getInt();
desc = buf.asCharBuffer().toString(); //This must be wrong!
System.out.format("You ordered %d" + " units of %s at $%.2f%n",
unit, desc, price);
total += unit * price;
}
}
}
This is the output when I execute it:
You ordered 12 units of ????
[...]
and so on.
Thanks for your attention

Writing part:
for (int i = 0; i < prices.length; i++) {
buf.putDouble(prices[i]);
buf.putInt(units[i]);
byte[] descsBytes = descs[i].getBytes();
buf.putInt(descsBytes.length);
buf.put(descsBytes);
}
Reading part:
while (buf.hasRemaining()) {
price = buf.getDouble();
unit = buf.getInt();
int len = buf.getInt();
byte[] bytes = new byte[len];
buf.get(bytes);
desc = new String(bytes);
System.out.format("You ordered %d" + " units of %s at $%.2f%n",
unit, desc, price);
total += unit * price;
}

ByteBuffer.asCharBuffer() has a somewhat own behaviour. It creates a new buffer that shares the content with the ByteBuffer but has an own, independent position & limit.
So put() and get() on the CharBuffer do not change the position on the ByteBuffer!
So I would refrain from using ByteBuffer.asCharBuffer() at all as long as this is not exactly what you need.
So the solution would be to write and read the bytes into the original ByteBuffer:
buf.putInt(descs[i].length);
buf.put(descs[i].getBytes());`
To read the string again read the length and allocate the byte[] yourself:
int stringLength = buf.getInt();
byte[] bytes = new byte[stringLength];
buf.get(bytes);
desc = new String(bytes);

Deserialize ArrayList in Java

I have my own little serialiser class
package mypackage.shared;
import java.io.ByteArrayInputStream;
import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.io.ObjectInputStream;
import java.io.ObjectOutputStream;
public class Serializer {
static final String HEXES = "0123456789ABCDEF";
public static String serialize(Object o) throws IOException {
ByteArrayOutputStream stream = new ByteArrayOutputStream();
ObjectOutputStream os = new ObjectOutputStream(stream);
os.writeObject(o);
os.close();
return toHex(stream.toByteArray());
}
public static Object deserialize(String hexString) throws IOException, ClassNotFoundException {
byte[] serializedBytes = toByteArray(hexString);
ByteArrayInputStream bis = new ByteArrayInputStream(serializedBytes);
ObjectInputStream ois = new ObjectInputStream(bis);
Object o = null;
o = ois.readObject();
ois.close();
return o;
}
public static String toHex( byte [] raw ) {
if ( raw == null ) {
return null;
}
final StringBuilder hex = new StringBuilder( 2 * raw.length );
for ( final byte b : raw ) {
hex.append(HEXES.charAt((b & 0xF0) >> 4))
.append(HEXES.charAt((b & 0x0F)));
}
return hex.toString();
}
private static byte[] toByteArray(String s) {
int len = s.length();
byte[] data = new byte[len / 2];
for (int i = 0; i < len; i += 2) {
data[i / 2] = (byte) ((Character.digit(s.charAt(i), 16) << 4)
+ Character.digit(s.charAt(i+1), 16));
}
return data;
}
}
It works great for some home made objects I have. However, as soon as I have an ArrayList of these objects it fails to deserialize them. ANy idea why that would be? I find it quite hard to debug since readObject() just fails

If serialization doesn't work it means that some object is not serializable.
Try to deserialize an arraylist containing only one home object and do it for all home objects