I have a multi-threaded scenario in which a Sender class which uses a ConcurrentLinkedQueue of type Packet is used. Packet is a POJO defined as:
class Packet {
int seq;
byte[] data;
}
Sender class is defined as:
class Sender implements Runnable {
private static ConcurrentLinkedQueue<Packet> queue;
public Sender() {
if (ccl == null)
ccl = new ConcurrentLinkedQueue<Packet>();
}
public static boolean enqueue(Packet p) {
if (queue == null) {
queue = new ConcurrentLinkedQueue<Packet>();
}
//System.out.println(p.toString());
return queue.add(p);
}
#Override
public void run() {
TcpSend tcp = new TcpSend();
while (true) {
Packet p = queue.remove();
// some code here
//System.out.println(p.toString());
tcp.send(p);
}
}
}
From another java class I am reading a file into byte array and adding it to ConcurrentLinkedQueue. When I print the details of packet in enqueue(), I get the correct details. But when I print the details in run() i.e. after removing the packet from queue, I get the correct seq but I get the data of the last packed that was added to the queue. This happens for all the packets which are removed from the queue.
The data is added by the following method
public void addPacket() {
int bytesRead = 0;
int seq = 1;
byte[] fileInBytes = new byte[1500];
BufferedInputStream in = new BufferedInputStream(new
FileInputStream(fileName));
while((bytesRead = in.read(fileInBytes)) != -1) {
Sender.enqueue(new Packet(seq, fileInBytes);
seq++;
}
}
Please suggest. TIA
You are using the same byte array (fileInBytes) to read from the stream:
byte[] fileInBytes = new byte[1500];
BufferedInputStream in = new BufferedInputStream(new
FileInputStream(fileName));
while((bytesRead = in.read(fileInBytes)) != -1) {
Sender.enqueue(new Packet(seq, fileInBytes);
seq++;
}
I guess that the constructor of Packet does not copy the byte array:
class Packet {
int seq;
byte[] data;
public Packet(int seq, byte[] data) {
this.seq = seq;
this.data = data;
}
}
That means that all of your Packet.data fields point to the same byte array which is overwritten each time your read from the stream.
You need to use a copy:
byte[] fileInBytes = new byte[1500];
BufferedInputStream in = new BufferedInputStream(new
FileInputStream(fileName));
while((bytesRead = in.read(fileInBytes)) != -1) {
byte[] packetBytes = Arrays.copyOf(fileInBytes, bytesRead);
Sender.enqueue(new Packet(seq, packetBytes );
seq++;
}
Related
Im trying to "pack" several files (previously inside a jar archive) in another single non-jar file by using DataInputStream / DataOutputStream.
The idea was:
First int = number of entries
First UTF is the first entry name
Second Int is entry byte array length (entry size)
Then repeat for every entry.
The code:
public static void main(String[] args) throws Throwable {
test();
System.out.println("========================================================================================");
final DataInputStream dataInputStream = new DataInputStream(new FileInputStream(new File("C:\\Users\\Admin\\Desktop\\randomJarOut")));
for (int int1 = dataInputStream.readInt(), i = 0; i < int1; ++i) {
final String utf = dataInputStream.readUTF();
System.out.println("Entry name: " + utf);
final byte[] array = new byte[dataInputStream.readInt()];
for (int j = 0; j < array.length; ++j) {
array[j] = dataInputStream.readByte();
}
System.out.println("Entry bytes length: " + array.length);
}
}
Unpacking original & packing to new one:
private static void test() throws Throwable {
JarInputStream stream = new JarInputStream(new FileInputStream(new File("C:\\Users\\Admin\\Desktop\\randomJar.jar")));
JarInputStream stream1 = new JarInputStream(new FileInputStream(new File("C:\\Users\\Admin\\Desktop\\randomJar.jar")));
final byte[] buffer = new byte[2048];
final DataOutputStream outputStream = new DataOutputStream(new FileOutputStream(new File("C:\\Users\\Admin\\Desktop\\randomJarOut")));
int entryCount = 0;
for (ZipEntry entry; (entry = stream.getNextJarEntry()) != null; ) {
entryCount++;
}
outputStream.writeInt(entryCount);
for (JarEntry entry; (entry = stream1.getNextJarEntry()) != null; ) {
int entryRealSize = stream1.read(buffer);
if (!(entryRealSize == -1)) {
System.out.println("Writing: " + entry.getName() + " Length: " + entryRealSize);
outputStream.writeUTF(entry.getName());
outputStream.writeInt(entryRealSize);
for (int len = stream1.read(buffer); len != -1; len = stream1.read(buffer)) {
outputStream.write(buffer, 0, len);
}
}
}
outputStream.flush();
outputStream.close();
}
Apparently im able to unpack the first entry without any problems, the second one and others:
Entry name: META-INF/services/org.jd.gui.spi.ContainerFactory
Entry bytes length: 434
Exception in thread "main" java.io.UTFDataFormatException: malformed input around byte 279
at java.io.DataInputStream.readUTF(DataInputStream.java:656)
at java.io.DataInputStream.readUTF(DataInputStream.java:564)
at it.princekin.esercizio.Bootstrap.main(Bootstrap.java:29)
Disconnected from the target VM, address: '127.0.0.1:54384', transport: 'socket'
Process finished with exit code 1
Does anyone knows how to fix this? Why is this working for the first entry but not the others?
My take on this is that the jar file (which in fact is a zip file) has a Central Directory which is only read with the ZipFile (or JarFile) class.
The Central Directory contains some data about the entries such as the size.
I think the ZipInputStream will not read the Central Directory and thus the ZipEntry will not contain the size (returning -1 as it is unknown) whereas reading ZipEntry from ZipFile class will.
So if you first read the size of each entry using a ZipFile and store that in a map, you can easily get it when reading the data with the ZipInputStream.
This page includes some good examples as well.
So my version of your code would be:
import java.io.*;
import java.util.HashMap;
import java.util.Map;
import java.util.zip.ZipEntry;
import java.util.zip.ZipFile;
import java.util.zip.ZipInputStream;
public class JarRepacker {
public static void main(String[] args) throws Throwable {
JarRepacker repacker = new JarRepacker();
repacker.repackJarToMyFileFormat("commons-cli-1.3.1.jar", "randomJarOut.bin");
repacker.readMyFileFormat("randomJarOut.bin");
}
private void repackJarToMyFileFormat(String inputJar, String outputFile) throws Throwable {
int entryCount;
Map<String, Integer> sizeMap = new HashMap<>();
try (ZipFile zipFile = new ZipFile(inputJar)) {
entryCount = zipFile.size();
zipFile.entries().asIterator().forEachRemaining(e -> sizeMap.put(e.getName(), (int) e.getSize()));
}
try (final DataOutputStream outputStream = new DataOutputStream(new FileOutputStream(outputFile))) {
outputStream.writeInt(entryCount);
try (ZipInputStream stream = new ZipInputStream(new BufferedInputStream(new FileInputStream(inputJar)))) {
ZipEntry entry;
final byte[] buffer = new byte[2048];
while ((entry = stream.getNextEntry()) != null) {
final String name = entry.getName();
outputStream.writeUTF(name);
final Integer size = sizeMap.get(name);
outputStream.writeInt(size);
//System.out.println("Writing: " + name + " Size: " + size);
int len;
while ((len = stream.read(buffer)) > 0) {
outputStream.write(buffer, 0, len);
}
}
}
outputStream.flush();
}
}
private void readMyFileFormat(String fileToRead) throws IOException {
try (DataInputStream dataInputStream
= new DataInputStream(new BufferedInputStream(new FileInputStream(fileToRead)))) {
int entries = dataInputStream.readInt();
System.out.println("Entries in file: " + entries);
for (int i = 1; i <= entries; i++) {
final String name = dataInputStream.readUTF();
final int size = dataInputStream.readInt();
System.out.printf("[%3d] Reading: %s of size: %d%n", i, name, size);
final byte[] array = new byte[size];
for (int j = 0; j < array.length; ++j) {
array[j] = dataInputStream.readByte();
}
// Still need to do something with this array...
}
}
}
}
The problem, probably, lies in that you are mixing not reciprocal read/write methods:
The writer method writes with outputStream.writeInt(entryCount) and the main method reads with dataInputStream.readInt(). That is OK.
The writer method writes with outputStream.writeUTF(entry.getName()) and the main method reads with dataInputStream.readUTF(). That is OK.
The writer method writes with outputStream.writeInt(entryRealSize) and the main method reads with dataInputStream.readInt(). That is OK.
The writer method writes with outputStream.write(buffer, 0, len) and the main method reads with dataInputStream.readByte() several times. WRONG.
If you write an array of bytes with write(buffer, offset, len), you must read it with read(buffer, offset, len), because write(buffer, offset, len) writes exactly len physical bytes onto the output stream, while writeByte (the counterpart of readByte) writes a lot of metadata overhead about the object type, and then its state variables.
Bugs in the writer method
There is also a mayor bug in the writer method: It invokes up to three times stream1.read(buffer), but it just uses once the buffer contents. The result is that the real size of file is actually written onto the output stream metadata, but it is followed by just a small part of the data.
If you need to know the input file size before writing it in the output stream, you have two choices:
Either chose a large enough buffer size (like 204800) which will allow you to read the whole file in just one read and write it in just one write.
Or either separate read from write algorithms: First a method to read the whole file and store it in memory (a byte[], for example), and then another method to write the byte[] onto the output stream.
Full fixed solution
I've fixed your program, with specific, decoupled methods for each task. The process consists in parsing the input file to a memory model, write it to an intermediate file according to your custom definition, and then read it back.
public static void main(String[] args)
throws Throwable
{
File inputJarFile=new File(args[0]);
File intermediateFile=new File(args[1]);
List<FileData> fileDataEntries=parse(inputJarFile);
write(fileDataEntries, intermediateFile);
read(intermediateFile);
}
public static List<FileData> parse(File inputJarFile)
throws IOException
{
List<FileData> list=new ArrayList<>();
try (JarInputStream stream=new JarInputStream(new FileInputStream(inputJarFile)))
{
for (ZipEntry entry; (entry=stream.getNextJarEntry()) != null;)
{
byte[] data=readAllBytes(stream);
if (data.length > 0)
{
list.add(new FileData(entry.getName(), data));
}
stream.closeEntry();
}
}
return list;
}
public static void write(List<FileData> fileDataEntries, File output)
throws Throwable
{
try (DataOutputStream outputStream=new DataOutputStream(new FileOutputStream(output)))
{
int entryCount=fileDataEntries.size();
outputStream.writeInt(entryCount);
for (FileData fileData : fileDataEntries)
{
int entryRealSize=fileData.getData().length;
{
System.out.println("Writing: " + fileData.getName() + " Length: " + entryRealSize);
outputStream.writeUTF(fileData.getName());
outputStream.writeInt(entryRealSize);
outputStream.write(fileData.getData());
}
}
outputStream.flush();
}
}
public static void read(File intermediateFile)
throws IOException
{
try (DataInputStream dataInputStream=new DataInputStream(new FileInputStream(intermediateFile)))
{
for (int entryCount=dataInputStream.readInt(), i=0; i < entryCount; i++)
{
String utf=dataInputStream.readUTF();
int entrySize=dataInputStream.readInt();
System.out.println("Entry name: " + utf + " size: " + entrySize);
byte[] data=readFixedLengthBuffer(dataInputStream, entrySize);
System.out.println("Entry bytes length: " + data.length);
}
}
}
private static byte[] readAllBytes(InputStream input)
throws IOException
{
byte[] buffer=new byte[4096];
byte[] total=new byte[0];
int len;
do
{
len=input.read(buffer);
if (len > 0)
{
byte[] total0=total;
total=new byte[total0.length + len];
System.arraycopy(total0, 0, total, 0, total0.length);
System.arraycopy(buffer, 0, total, total0.length, len);
}
}
while (len >= 0);
return total;
}
private static byte[] readFixedLengthBuffer(InputStream input, int size)
throws IOException
{
byte[] buffer=new byte[size];
int pos=0;
int len;
do
{
len=input.read(buffer, pos, size - pos);
if (len > 0)
{
pos+=len;
}
}
while (pos < size);
return buffer;
}
private static class FileData
{
private final String name;
private final byte[] data;
public FileData(String name, byte[] data)
{
super();
this.name=name;
this.data=data;
}
public String getName()
{
return this.name;
}
public byte[] getData()
{
return this.data;
}
}
I need to convert a Reader object into InputStream. My solution right now is below. But my concern is since this will handle big chunks of data, it will increase the memory usage drastically.
private static InputStream getInputStream(final Reader reader) {
char[] buffer = new char[10240];
StringBuilder builder = new StringBuilder();
int charCount;
try {
while ((charCount = reader.read(buffer, 0, buffer.length)) != -1) {
builder.append(buffer, 0, charCount);
}
reader.close();
} catch (final IOException e) {
e.printStackTrace();
}
return new ByteArrayInputStream(builder.toString().getBytes(StandardCharsets.UTF_8));
}
Since I use StringBuilder this will keep the full content of the reader object in memory. I want to avoid this. Is there a way I can pipe Reader object? Any help regarding this highly appreciated.
Using the Apache Commons IO library, you can do this conversion in one line:
//import org.apache.commons.io.input.ReaderInputStream;
InputStream inputStream = new ReaderInputStream(reader, StandardCharsets.UTF_8);
You can read the documentaton for this Class at https://commons.apache.org/proper/commons-io/apidocs/org/apache/commons/io/input/ReaderInputStream.html
It might be worth trying this to see if it solves the memory issue too.
First: a rare requirement, often it is the other way around, or there is a FileChannel, so one can use a ByteBuffer.
A PipedInputStream would be possible, starting a PipedOutputStream in a second thread. However that is unneeded.
A Reader gives chars. Unicode code points are derived from either one or two chars (the latter a surrogate pair).
/**
* Reader for an InputSteam of UTF-8 text bytes.
*/
public class ReaderInputStream extends InputStream {
private final Reader reader;
private boolean eof;
private int byteCount;
private byte[] bytes = new byte[6];
public ReaderInputStream(Reader reader) {
this.reader = reader;
}
#Override
public int read() throws IOException {
if (byteCount > 0) {
int c = bytes[0];
--byteCount;
for (int i = 0; i < byteCount; ++i) {
bytes[i] = bytes[i + 1];
}
return c;
}
if (eof) {
return -1;
}
int c = reader.read();
if (c == -1) {
eof = true;
return -1;
}
char ch = (char) c;
String s;
if (Character.isHighSurrogate(ch)) {
c = reader.read();
if (c == -1) {
// Error, low surrogate expected.
eof = true;
//return -1;
throw new IOException("Expected a low surrogate char i.o. EOF");
}
char ch2 = (char) c;
if (!Character.isLowSurrogate(ch2)) {
throw new IOException("Expected a low surrogate char");
}
s = new String(new char [] {ch, ch2});
} else {
s = Character.toString(ch);
}
byte[] bs = s.getBytes(StandardCharsets.UTF_8);
byteCount = bs.length;
System.arraycopy(bs, 0, bytes, 0, byteCount);
return read();
}
}
Path source = Paths.get("...");
Path target = Paths.get("...");
try (Reader reader = Files.newBufferedReader(source, StandardCharsets.UTF_8);
InputStream in = new ReaderInputStream(reader)) {
Files.copy(in, target);
}
As per my requirement I want to create checksum value using SHA-256, from InputStream,
As below:
private InputStream createZipInput(List<ResponsePack> aList, byte[] manifestData)
{
final int bufferSize = 2048;
byte buffer[] = new byte[bufferSize];
ByteArrayOutputStream byteStream = new ByteArrayOutputStream();
ZipOutputStream zipFileToSend = new ZipOutputStream(byteStream);
LOG.trace("Compressing the file {}");
try
{
for (ResponsePack info : aList)
{
ByteArrayOutputStream byteStreamCheckSum = new ByteArrayOutputStream();
ZipOutputStream zipFileToSendCheckSum = new ZipOutputStream(byteStreamCheckSum);
zipFileToSend.putNextEntry(new ZipEntry(info.getFileName()));
zipFileToSendCheckSum.putNextEntry(new ZipEntry(info.getFileName()));
InputStream in = info.getFileContentStream();
int length;
while ((length = in.read(buffer)) >= 0)
{
zipFileToSend.write(buffer, 0, length);
zipFileToSendCheckSum.write(buffer, 0, length);
}
zipFileToSend.closeEntry();
zipFileToSendCheckSum.closeEntry();
String checksum = validChecksum(byteStreamCheckSum.toByteArray());
LOG.error("Checksum {}", checksum);
zipFileToSendCheckSum.flush();
zipFileToSendCheckSum.close();
}
zipFileToSend.close();
}
catch (IOException e)
{
return e;
}
return new ByteArrayInputStream(byteStream.toByteArray());
}
private static String validChecksum(byte[] dataCopy)
{
printLOG("Byte Array Size {}", dataCopy.length);
try (ZipInputStream zipInputStream = new ZipInputStream(new ByteArrayInputStream(dataCopy)))
{
ZipEntry zipEntry;
MessageDigest digest = DigestUtils.getSha256Digest();
DWriter writer = new DWriter(digest);
while ((zipEntry = zipInputStream.getNextEntry()) != null)
{
org.apache.commons.io.output.ByteArrayOutputStream dest = StreamUtils.extractFileAsByteArrayStream(zipInputStream);
LOG.error("CheckSum Entity creating");
if(zipEntry != null)
{
printLOG("CheckSum Entity file Name {}", zipEntry.getName());
}
LOG.error("Byte array size {}", dest.toByteArray().length);
writer.write(dest.toByteArray());
dest.flush();
dest.close();
}
if (writer.getChecksum() != null)
{
return writer.getChecksum();
}
else
{
return "";
}
}
catch (Exception e)
{
printLOG("Exception encountered while creating checksum: {}", e.getMessage());
return "";
}
}
static class DWriter
{
private final MessageDigest myDigest;
DWriter(MessageDigest digest)
{
myDigest = digest;
}
public void write(byte[] data)
{
myDigest.update(data);
}
public String getChecksum()
{
return new String(Hex.encodeHex(myDigest.digest()));
}
}
But the problem is when I checked the log, found byte array contains value but still checksum always creating for empty string, as below
Byte Array Size 3948
CheckSum Entity creating
CheckSum Entity file Name 20200911104812526.json
Byte array size 20854
Checksum e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
Help me where I am doing wrong, due to which I am getting checksum for an empty string
I'm not sure what's wrong with the code but it seems overly complicated: you're writing the input into a zipped stream and the dezip it in memory to read it again.
You don't need to do that: storing the input in a (non-zipped) byte array should be enough.
I think you need to make sure that in.read() works as intended (and that there's actually some data to read).
You get the checksum for a null input and your zip entry is also empty, so it looks like the input was empty. Add some logs or use a debugger to investigate what's happening.
private InputStream createZipInput(List<ResponsePack> aList, byte[] manifestData) {
final int bufferSize = 2048;
byte buffer[] = new byte[bufferSize];
ByteArrayOutputStream byteStream = new ByteArrayOutputStream();
ZipOutputStream zipFileToSend = new ZipOutputStream(byteStream);
LOG.trace("Compressing the file {}");
try {
for (ResponsePack info : aList) {
ByteArrayOutputStream byteStreamCheckSum = new ByteArrayOutputStream();
zipFileToSend.putNextEntry(new ZipEntry(info.getFileName()));
InputStream in = info.getFileContentStream();
int length;
while ((length = in.read(buffer)) != -1) {
zipFileToSend.write(buffer, 0, length);
byteStreamCheckSum.write(buffer, 0, length);
}
zipFileToSend.closeEntry();
MessageDigest digest = DigestUtils.getSha256Digest();
digest.update(byteStreamCheckSum.toByteArray());
String checksum = new String(Hex.encodeHex(digest.digest()));
LOG.error("Checksum {}", checksum);
}
zipFileToSend.close();
} catch (IOException e) {
throw e;
}
return new ByteArrayInputStream(byteStream.toByteArray());
This question already has answers here:
Java multiple file transfer over socket
(3 answers)
Closed 7 years ago.
I'm making a communicator with ability to send files.
So far I managed to make text sending working using additional thread (listener).
I'm trying to make the same thing with files, but I don't know, how can I make a file listener - a thread, that detects incoming file, downloads it and listens for another file. Also, I don't know if I'm making my file sender properly. Could you help?
Current sender code:
try {
InputStream in = new FileInputStream(fileToSend);
OutputStream out = fileConn.getOutputStream();
Controller.copyData(in, out);
out.close();
in.close();
} catch (IOException e) {
System.out.println("Problem!");
}
And receiver code:
while (true)
{
try {
InputStream in = socket.getInputStream();
OutputStream out = new FileOutputStream("hi.txt"); //temporary
Controller.copyData(in, out);
out.close();
in.close();
} catch (IOException e) {
System.out.println("Problem!");
}
}
EDIT: I forgot to add my copyData. There it is:
public static void copyData(InputStream in, OutputStream out) throws IOException{
byte[] buf = new byte[8192];
int len = 0;
while ((len = in.read(buf)) != -1) {
out.write(buf, 0, len);
}
}
You can achive tha by just adding to your listening thread option to wait for diffrent messages/options and react accordingly. For example:
private class WaitingThread extends Thread {
volatile boolean awaitsServer = false;
DataInputStream dataInput = new DataInputStream(inputStream);
public void run() {
while (connected) {
int message = 0;
if (awaitsServer == true) {
if (dIn.available() ==0) {
view.setLog("waiting");
} else {
message = dIn.readInt();
switch (tempMessage) {
// TO DO ALL KIND OF COMMUNICATION
case 1:
int filesize = dataInput.readInt();
int bytesRead;
int currentTot = 0;
byte[] bytearray = new byte[filesize];
int len = dataInput.readInt();
FileOutputStream fos = new FileOutputStream(currentlySelectedFile);
BufferedOutputStream bos = new BufferedOutputStream(fos);
bytesRead = dataInput.read(bytearray, 0, bytearray.length);
currentTot = bytesRead;
do {
bytesRead = dataInput.read(bytearray, currentTot,
(len - currentTot));
if (bytesRead >= 0)
currentTot += bytesRead;
} while (currentTot < len);
bos.write(bytearray, 0, currentTot);
bos.close();
}
case 2: //GET TEXT
case 3: //DO SOMETHING ELSE
}}}
Btw you have example how to send files.
Based on the below question,
http://stackoverflow.com/questions/23753342/java-memory-allocation-for-local-variables
I use SerialPortEvent to read data from serial port,
public String logtext = "";
public void serialEvent(SerialPortEvent evt) {
if (evt.getEventType() == SerialPortEvent.DATA_AVAILABLE) {
try {
StringBuilder sBuilder = new StringBuilder();
int length = input.available();
byte[] array = new byte[length];
int numBytes = input.read(array);
for (byte b : array) {
logText = new String(new byte[] {b});
sBuilder.append(logText);
}
//Finally i append the StringBuilder to JtextPane
.......
......
}
Creating new String() every time the serialEvent is called it will create new instance every time which will increase the memory usage.. In certain scenario this serialEvent will be called every second.
Is there any other efficient way to do this with out using new operator??
Please help
You don't need to do this one byte at a time. I would just do all the bytes at once.
int length = input.available();
byte[] array = new byte[length];
String logText = new String(array, 0); // assume ISO-8859-1 encoding