We Keep Coding

Java, how can you chunk pieces of a large inputstream efficiently?

I have an input stream that is potentially 20-30mb. I'm trying to upload chunks as a multi-part file upload to S3.
I have the content-length available and I have the input-stream available. How can I efficiently do this with memory in mind.
I saw someone had done something like this, but not sure I fully understand it:
int contentLength = inputStreamMetadata.getContentLength();
int partSize = 512 * 1024; // Set part size to 2 MB
int filePosition = 0;
ByteArrayInputStream bais = inputStreamMetadata.getInputStream();
List<PartETag> partETags = new ArrayList<>();
byte[] chunkedFileBytes = new byte[partSize];
for (int i = 1; filePosition < contentLength; i++) {
// Because the last part could be less than 5 MB, adjust the part size as needed.
partSize = Math.min(partSize, (contentLength - filePosition));
filePosition += bais.read(chunkedFileBytes, filePosition, partSize);
// Create the request to upload a part.
UploadPartRequest uploadRequest = new UploadPartRequest()
.withBucketName(bucketName)
.withUploadId(uploadId)
.withKey(fileName)
.withPartNumber(i)
.withInputStream(new ByteArrayInputStream(chunkedFileBytes, 0, partSize))
.withPartSize(partSize);
UploadPartResult uploadResult = client.uploadPart(uploadRequest);
partETags.add(uploadResult.getPartETag());
}
}
Specifically this piece: .withInputStream(new ByteArrayInputStream(bytes, 0, bytesRead))

Sorry, i cannot (easily) test it, but I think you are really close, ... just have to "fix" and "arrange" your loop!
Combining https://stackoverflow.com/a/22128215/592355 with your latest code:
int partSize = 5 * 1024 * 1024; // Set part size to 5 MB
ByteArrayInputStream bais = inputStreamMetadata.getInputStream();
List<PartETag> partETags = new ArrayList<>();
byte[] buff = new byte[partSize];
int partNumber = 1;
while (true) {//!
int readBytes = bais.read(buff);// readBytes in [-1 .. partSize]!
if (readBytes == -1) { //EOF
break;
}
// Create the request to upload a part.
UploadPartRequest uploadRequest = new UploadPartRequest()
.withBucketName(bucketName)
.withUploadId(uploadId)
.withKey(fileName)
.withPartNumber(partNumber++)
.withInputStream(new ByteArrayInputStream(buff, 0, readBytes))
.withPartSize(readBytes);
UploadPartResult uploadResult = client.uploadPart(uploadRequest);
partETags.add(uploadResult.getPartETag());
}
// Complete the multipart upload....
// https://docs.aws.amazon.com/AmazonS3/latest/dev/llJavaUploadFile.html

S3 Multipart Upload Error Code: MalformedXML

I am using the following code to for multipart upload to s3.
public boolean uploadFileToS3(List<InputStream> filePartitions ) throws FileException, IOException{
AWSCredentials credentials = new BasicAWSCredentials("access_key","secret_key");
String existingBucketName = "bucketName";
String keyName = "file.log";
AmazonS3 s3Client = new AmazonS3Client(credentials);
// Create a list of UploadPartResponse objects. You get one of these for
// each part upload.
List<PartETag> partETags = new ArrayList<PartETag>();
// Step 1: Initialize.
InitiateMultipartUploadRequest initRequest = new InitiateMultipartUploadRequest(existingBucketName, keyName);
InitiateMultipartUploadResult initResponse = s3Client.initiateMultipartUpload(initRequest);
try {
// Step 2: Upload parts.
Iterator its = filePartitions.iterator();
int i=0;
while(its.hasNext()){
UploadPartRequest uploadRequest = new UploadPartRequest()
.withBucketName(existingBucketName)
.withKey(keyName)
.withUploadId(initResponse.getUploadId()).withPartNumber(i)
.withInputStream((InputStream)its.next());
i++;
System.out.println("Part " + i + "is uploaded");
}
// Step 3: Complete.
CompleteMultipartUploadRequest compRequest = new CompleteMultipartUploadRequest(existingBucketName,
keyName,
initResponse.getUploadId(),
partETags);
s3Client.completeMultipartUpload(compRequest);
} catch (Exception e) {
System.out.println("******************************");
System.out.println(e.getMessage());
System.out.println(e.getCause());
System.out.println("************************");
s3Client.abortMultipartUpload(new AbortMultipartUploadRequest(
existingBucketName, keyName, initResponse.getUploadId()));
}
return true;
}
I am facing the following exception when I run this code.
The XML you provided was not well-formed or did not validate against our published schema (Service: Amazon S3; Status Code: 400; Error Code: MalformedXML; Request ID: C0538A21C25A2DD4)
In the above code List filePartitions is created from a large file.
each chunk is of 20000 bytes.
I have used the following code to split the file into partitions as I have InputStream data directly not a file. It is a REST API.
List<InputStream> filePartitions = new ArrayList<InputStream>();
InputStream inStream = new BufferedInputStream(a_fileInputStream);
int totalBytesRead = 0;
int FILE_SIZE = a_fileInputStream.available();
int chunkSize = 20000;
while (totalBytesRead < FILE_SIZE) {
int bytesRemaining = FILE_SIZE - totalBytesRead;
if (bytesRemaining < chunkSize) {
chunkSize = bytesRemaining;
}
byte[] buffer=new byte[chunkSize];
//Temporary Byte Array
int read = inStream.read(buffer,0,chunkSize);
int bytesRead = inStream.read(buffer, 0, chunkSize);
if (bytesRead > 0) // If bytes read is not empty
{
totalBytesRead += bytesRead;
//create InputStream from temporary byte array
InputStream partition = new ByteArrayInputStream(buffer);
filePartitions.add(partition);
}
}
return filePartitions;

java.util.zip.ZipOutputStream - Zipping large files faster?

I am wondering how could I speed up the zipping process of 40+ image files, in my android app.
Clients are sending images, which needs to be compressed or placed into a folder before uploading on the server. Now I use the bellow method, but this way files are zipped in about 20-30 seconds, while the phone appears to be frozen and users tend to exit the app :(
The method I use for zipping:
private static final int BUFFER_SIZE = 2048;
public void zip(String[] files, String zipFile) throws IOException {
File zipDirectory = new File(Environment.getExternalStorageDirectory()
+ "/laborator/");
if (!zipDirectory.exists()) {
zipDirectory.mkdirs();
} else {
System.out.println("folder already exists!");
}
BufferedInputStream origin = null;
ZipOutputStream out = new ZipOutputStream(new BufferedOutputStream(
new FileOutputStream(Environment.getExternalStorageDirectory()
+ "/laborator/" + zipFile)));
try {
byte data[] = new byte[BUFFER_SIZE];
for (int i = 0; i < files.length; i++) {
FileInputStream fi = new FileInputStream(files[i]);
origin = new BufferedInputStream(fi, BUFFER_SIZE);
try {
ZipEntry entry = new ZipEntry(files[i].substring(files[i]
.lastIndexOf("/") + 1));
out.putNextEntry(entry);
int count;
while ((count = origin.read(data, 0, BUFFER_SIZE)) != -1) {
out.write(data, 0, count);
}
} finally {
origin.close();
}
}
} finally {
out.close();
System.out.println("ziping done");
sendZip();
}
}

Since your images are jpgs chances are high that you don't get any decent compression within the ZIP file. So you could try to just put the images uncompressed into the ZIP file which should be considerable faster without increasing the size of the ZIP file:
ZipEntry entry = new ZipEntry(files[i].subs...
entry.setMethod(ZipEntry.STORED);
out.putNextEntry(entry);

You could use
out.setLevel(Deflater.NO_COMPRESSION);
This way no need to change ZipEntry.

How to correctly measure download speed with Java / Android

I'm trying to perform an AsyncTask class in my Android application that analyzes the network connection speed in for downloading and uploading. I'm working on the download portion now, but I'm not getting results I expect. I'm testing on a Wifi network that gets 15Mbps down/up speeds consistently, however, the results I'm getting from my application are more around barely 1Mbps. When I run the speed test apk on the device I'm testing on that gets around 3.5Mbps. The function works, just seems to be half the speed it should be. Should the following code produce accurate results?
try {
String DownloadUrl = "http://ipv4.download.thinkbroadband.com:8080/5MB.zip";
String fileName = "testfile.bin";
File dir = new File (context.getFilesDir() + "/temp/");
if(dir.exists()==false) {
dir.mkdirs();
}
URL url = new URL(DownloadUrl); //you can write here any link
File file = new File(context.getFilesDir() + "/temp/" + fileName);
long startTime = System.currentTimeMillis();
Log.d("DownloadManager", "download begining: " + startTime);
Log.d("DownloadManager", "download url:" + url);
Log.d("DownloadManager", "downloaded file name:" + fileName);
/* Open a connection to that URL. */
URLConnection ucon = url.openConnection();
//Define InputStreams to read from the URLConnection.
InputStream is = ucon.getInputStream();
BufferedInputStream bis = new BufferedInputStream(is);
//Read bytes to the Buffer until there is nothing more to read(-1).
ByteArrayBuffer baf = new ByteArrayBuffer(1024);
int current = 0;
while ((current = bis.read()) != -1) {
baf.append((byte) current);
}
long endTime = System.currentTimeMillis(); //maybe
/* Convert the Bytes read to a String. */
FileOutputStream fos = new FileOutputStream(file);
fos.write(baf.toByteArray());
fos.flush();
fos.close();
File done = new File(context.getFilesDir() + "/temp/" + fileName);
Log.d("DownloadManager", "Location being searched: "+ context.getFilesDir() + "/temp/" + fileName);
double size = done.length();
if(done.exists()) {
done.delete();
}
Log.d("DownloadManager", "download ended: " + ((endTime - startTime) / 1000) + " secs");
double rate = (((size / 1024) / ((endTime - startTime) / 1000)) * 8);
rate = Math.round( rate * 100.0 ) / 100.0;
String ratevalue;
if(rate > 1000)
ratevalue = String.valueOf(rate / 1024).concat(" Mbps");
else
ratevalue = String.valueOf(rate).concat(" Kbps");
Log.d("DownloadManager", "download speed: "+ratevalue);
} catch (IOException e) {
Log.d("DownloadManager", "Error: " + e);
}
Example output
10-08 15:09:52.658: D/DownloadManager(13714): download ended: 70 secs
10-08 15:09:52.662: D/DownloadManager(13714): download speed: 585.14 Kbps
Thanks in advance for the help. If there is a better method, please let me know.

Following on my comments, here is an example of how to read several bytes from the stream
//Define InputStreams to read from the URLConnection.
InputStream is = ucon.getInputStream();
BufferedInputStream bis = new BufferedInputStream(is);
//I usually use a ByteArrayOutputStream, as it is more common.
ByteArrayOutputStream baos = new ByteArrayOutputStream();
int red = 0;
// This size can be changed
byte[] buf = new byte[1024];
while ((red = bis.read(buf)) != -1) {
baos.write(buf, 0, red);
}
What this does is it reads into a byte[] buffer, and return the amount of read bytes. This is in turn written to the OutputStream, specifying the amount of bytes to write.
ByteArrayOutputStream also have a toByteArray that behaves similarly.
Alternatively, you can also write directly to the file, if you consider that the write to file operation is significantly faster than the read function :
// Simply start by defining the fileoutputstream
FileOutputStream fos = new FileOutputStream(file);
int red = 0;
// This size can be changed
byte[] buf = new byte[1024];
while ((red = bis.read(buf)) != -1) {
// And directly write to it.
fos.write(buf, 0, red);
}
long endTime = System.currentTimeMillis(); //maybe
// Flush after, as this may trigger a commit to disk.
fos.flush();
fos.close();
Moreover, if you really only care about the download speed, it is not mandatory to write to the file, or to anywhere, this would be sufficient :
long size = 0;
byte[] buf = new byte[1024];
while ((red = bis.read(buf)) != -1) {
size += red;
}

DatagramSocket: File sending, client does not receive image file

I am trying to send bunch of files with using DatagramSocket. But there is a problem which I can not figure out. Transferring files other than image files goes well but while sending image files, Client gets stuck at receive although Server send the file. Here is my client and server side code blocks:
Server:
while (true) {
System.out.print("DatagramSocket: Waiting for file request...\n");
buf = new byte[Integer.SIZE];
packet = new DatagramPacket(buf, buf.length);
datagramSocket.receive(packet);
bb = ByteBuffer.wrap(buf);
// Receive request
buf = new byte[bb.getInt()];
packet = new DatagramPacket(buf, buf.length);
datagramSocket.receive(packet);
System.out.print("DatagramSocket: File request received.\n");
System.out.print("DatagramSocket: Requested file: "+new String(buf)+"\n");
// Check file if it is exist.
File file = new File("kaynak/"+new String(buf));
if (!file.exists()) {
System.out.print("DatagramSocket: File not found!\n");
return;
}
// Send file length.
System.out.printf("DatagramSocket: Sending file length: %d\n", file.length());
bb = ByteBuffer.allocate(Integer.SIZE).putInt((int) file.length());
buf = bb.array();
packet.setData(buf);
packet.setLength(buf.length);
datagramSocket.send(packet);
// Send file's relative path.
String relativePath = file.getAbsolutePath().substring(System.getProperty("user.dir").length() + 1);
System.out.printf("DatagramSocket: Sending file relative path: %s\n", relativePath);
bb.putInt(relativePath.getBytes().length);
datagramSocket.send(packet);
packet.setData(relativePath.getBytes());
packet.setLength(relativePath.getBytes().length);
datagramSocket.send(packet);
// Save file to byte array.
bis = new BufferedInputStream(new FileInputStream(file));
fileByteArray = new byte[(int) file.length()];
bis.read(fileByteArray);
System.out.printf("DatagramSocket: Sending file.\n");
int r = (int) file.length();
int c = 0;
// Send file.
for (int i = 0; i < file.length(); i++) {
c = r < 512 ? r : 512;
packet.setData(ByteBuffer.wrap(fileByteArray, i, c).array());
packet.setLength(c);
datagramSocket.send(packet);
r -= 512;
i += 511;
}
System.out.printf("DatagramSocket: File send.\n\n");
}
Client:
// Send file request.
bb = ByteBuffer.allocate(Integer.SIZE).putInt(files.get(i).getBytes().length);
message = bb.array();
packet = new DatagramPacket(message, message.length, InetAddress.getByName(host), dport);
datagramSocket.send(packet);
message = files.get(i).getBytes();
System.out.print("Requesting: "+new String(message)+"\n");
packet.setData(message);
packet.setLength(message.length);
datagramSocket.send(packet);
// Receive file size.
System.out.print("Requesting file length.\n");
message = ByteBuffer.allocate(Integer.SIZE).array();
packet.setData(message);
packet.setLength(Integer.SIZE);
datagramSocket.receive(packet);
bb = ByteBuffer.wrap(message);
int arraySize = bb.getInt();
System.out.printf("File size = %d baytes.\n", arraySize);
fileByteArray = new byte[arraySize];
// Receive file's relative path.
System.out.print("Requesting file's relative path.\n");
datagramSocket.receive(packet);
message = ByteBuffer.allocate(message.length).array();
packet.setLength(message.length);
datagramSocket.receive(packet);
String htmlPath = new String(packet.getData());
System.out.printf("File's relative path = %s.\n", htmlPath);
File file = new File("hedef/"+htmlPath.substring("kaynak".length()));
file.getParentFile().mkdirs();
file.createNewFile();
// Receive file content.
System.out.print("Requesting file content.\n");
int r = arraySize;
int c = 0;
for (int j = 0; j < arraySize; j++) {
c = r < 512 ? r : 512;
packet.setData(ByteBuffer.wrap(fileByteArray, j, c).array());
packet.setLength(c);
datagramSocket.receive(packet);
r -= 512;
j += 511;
}
// Save file.
System.out.print("Saving file.\n");
bos = new BufferedOutputStream(new FileOutputStream(file));
bos.write(fileByteArray);
bos.flush();
System.out.print("File saved.\n\n");
I also appreciate if can give me hints about my code, performance or wrong usage hints.
Thanks in advance.
Update:
After adding a little sleep time in for loop which sends file itself, now I am able to complete sending files but all images files I got is corrupted.
for (int j = 0; j < arraySize; j++) {
c = r < 512 ? r : 512;
packet.setData(ByteBuffer.wrap(fileByteArray, j, c).array());
packet.setLength(c);
datagramSocket.receive(packet);
r -= 512;
j += 511;
Thread.sleep(50);
}

From DatagramSocket documentation: This method blocks until a datagram is received.
Obviously there are packages getting lost.
Also: It would be better to not allocate an array for the hole data to be received. Think of some 4GB video file... Instead allocate a buffer array the size of your read (512) and write it to your BufferedOutputStream immediately after DatagramSocket.receive().