I need to speed up transfers across my gigabit ethernet connection. Right now, I'm doing something almost exactly like this, but I'm only seeing about 40% of that when I run this code below.
I also ran this script on all of my (Mac Pro) machines before testing
#!/bin/bash
sudo sysctl -w net.inet.tcp.win_scale_factor=8
sudo sysctl -w kern.ipc.maxsockbuf=16777216
sudo sysctl -w net.inet.tcp.sendspace=8388608
sudo sysctl -w net.inet.tcp.recvspace=8388608
The actual code follows:
import java.io.*;
import java.nio.*;
import java.net.*;
public class BandwidthTester {
private static final int OUT_BUF = (1 << 17),
IN_BUF = (1 << 17), SEND_BUF = (1 << 22), RECV_BUF = (1 << 22);
public static void main(String[] args) {
try {
// server
if (args.length == 0) {
ServerSocket sock = new ServerSocket();
sock.bind(new InetSocketAddress(41887));
// wait for connection
Socket s = sock.accept();
s.setSendBufferSize(SEND_BUF);
System.out.println("Buffers: " + s.getSendBufferSize() + " and " + s.getReceiveBufferSize());
sock.close();
BufferedOutputStream bOut = new BufferedOutputStream(s.getOutputStream(), OUT_BUF);
// send lots of data
sendLotsOfData(bOut);
} else if (args.length == 2) {
String host = args[0];
int port = Integer.parseInt(args[1]);
System.out.println("Connecting to " + args[0] + ":" + args[1]);
Socket sock = new Socket();
sock.setReceiveBufferSize(RECV_BUF);
sock.connect(new InetSocketAddress(host, port));
System.out.println("Buffers: " + sock.getSendBufferSize() + " and " + sock.getReceiveBufferSize());
BufferedInputStream bIn = new BufferedInputStream(sock.getInputStream(), IN_BUF);
getLotsOfData(bIn);
}
} catch (Exception e) {
e.printStackTrace();
}
}
public static void getLotsOfData(InputStream in) {
System.out.println("Getting data...");
try {
long start = System.currentTimeMillis();
ByteBuffer intConv = ByteBuffer.allocate(4);
in.read(intConv.array());
int len = intConv.getInt(0);
for (int i=0; i < len; i++) {
in.read(intConv.array());
int val = intConv.getInt(0);
}
long end = System.currentTimeMillis();
double elapsed = ((double)(end - start)) / (1000.0);
System.out.println("Read in " + elapsed + " seconds: " + ( (4.0*8.0*len/elapsed) + " bits per second"));
} catch (Exception e) {
e.printStackTrace();
}
}
public static void sendLotsOfData(OutputStream out) {
System.out.println("Sending data...");
try {
long start = System.currentTimeMillis();
int len = (1 << 29);
ByteBuffer intConv = ByteBuffer.allocate(4);
intConv.putInt(0, len);
out.write(intConv.array());
for (int i=0; i < len; i++) {
intConv.putInt(0, i);
out.write(intConv.array());
}
out.flush();
long end = System.currentTimeMillis();
double elapsed = ((double)(end - start)) / (1000.0);
System.out.println("Sent in " + elapsed + " seconds: " + ( (4.0*8.0*len/elapsed) + " bits per second"));
}
catch (Exception e) {
e.printStackTrace();
}
}
}
Any suggestions? It's taking about 42 seconds to send all of that data, but even a 10% improvement here would have a dramatic impact on my program.
One thing you might try is using a larger buffer for the ByteBuffer. Going from 4 bytes to 16, I went from a 12 second transfer time to a 9 second transfer time. (tested using 2^26 rather then 2^29 for length)
That said, it was being run locally; so no actual network issues should have been encountered.
Somewhat dirty modified code for sending:
ByteBuffer intConv = ByteBuffer.allocate(16);
intConv.putInt(0, len);
out.write(intConv.array(),0,4);
for (int i=0; i < len; i+=4) {
for(int j=0; j<4; j++)
intConv.putInt(4*j, i);
out.write(intConv.array());
}
And Receiving:
ByteBuffer intConv = ByteBuffer.allocate(16);
in.read(intConv.array(),0,4);
int len = intConv.getInt(0);
for (int i=0; i < len; i+=4) {
in.read(intConv.array());
for(int j=0; j<4; j++)
{
int val=intConv.getInt(j*4);
}
}
Clearly the receiving end would need some modification to handle strange and odd cases like 'what if there were only 3 ints remaining/read from the stream', but I think this would be enough to see if it improves performance.
Related
I'm trying to write software that sends a set of data (a portion of a video game) in different formats (chunked, compressed, raw), and measures the speed between each. However, I'm running into an issue while sorting out the CHUNKED method. I've found that, when reading byte array sizes of more than 140000 bytes, the client starts to only read up to around 131072, no matter how much bigger the array actually is. Is there a reason for this, or potentially a better way to do this? My code is shown below. I'm using the read() method of DataInputStream (and its return value).
SERVER
/**
*
* #return Time taken to complete transfer.
*/
public int start(String mode, int length) throws IOException, InterruptedException {
if(mode.equals("RAW")){
byte[] all = new ByteCollector(ServerMain.FILES, length).collect();
output.writeUTF("SENDING " + mode + " " + all.length);
expect("RECEIVING " + mode);
long start = System.currentTimeMillis();
echoSend(all);
return (int) (System.currentTimeMillis() - start);
}else if(mode.equals("CHUNKED")){ /*the important part*/
//split into chunks
byte[] all = new ByteCollector(ServerMain.FILES, length).collect();
int chunks = maxChunks(all);
output.writeUTF("SENDING " + mode + " " + chunks);
System.out.println("Expecting RECEIVING " + chunks + "...");
expect("RECEIVING " + chunks);
int ms = 0;
for(int i = 0; i<chunks; i++){
byte[] currentChunk = getChunk(i, all);
System.out.println("My chunk length is " + currentChunk.length);
long start = System.currentTimeMillis();
System.out.println("Sending...");
echoSend(currentChunk);
ms += System.currentTimeMillis() - start;
}
if(chunks == 0) expect("0"); //still need to confirm, even though no data was sent
return ms;
}else if(mode.equals("COMPRESSED")){
byte[] compressed = new ByteCollector(ServerMain.FILES, length).collect();
compressed = ExperimentUtils.compress(compressed);
output.writeUTF("SENDING " + mode + " " + compressed.length);
expect("RECEIVING " + mode);
long start = System.currentTimeMillis();
echoSend(compressed, length);
return (int) (System.currentTimeMillis() - start);
}
return -1;
}
public static void main(String[] args) throws IOException,InterruptedException{
FILES = Files.walk(Paths.get(DIRECTORY)).filter(Files::isRegularFile).toArray(Path[]::new);
SyncServer server = new SyncServer(new ServerSocket(12222).accept());
System.out.println("--------[CH UNK ED]--------");
short[] chunkedSpeeds = new short[FOLDER_SIZE_MB + 1/*for "zero" or origin*/];
for(int i = 0; i<=FOLDER_SIZE_MB; i++){
chunkedSpeeds[i] = (short) server.start("CHUNKED", i * MB);
System.out.println(i + "MB, Chunked: " + chunkedSpeeds[i]);
}
short[] compressedSpeeds = new short[FOLDER_SIZE_MB + 1];
for(int i = 0; i<=FOLDER_SIZE_MB; i++){
compressedSpeeds[i] = (short) server.start("COMPRESSED", i * MB);
}
short[] rawSpeeds = new short[FOLDER_SIZE_MB + 1];
for(int i = 0; i<=FOLDER_SIZE_MB; i++){
rawSpeeds[i] = (short) server.start("RAW", i * MB);
}
System.out.println("Raw speeds: " + Arrays.toString(rawSpeeds));
System.out.println("\n\nCompressed speeds: " + Arrays.toString(compressedSpeeds));
System.out.println("\n\nChunked speeds: " + Arrays.toString(chunkedSpeeds));
}
CLIENT
public static void main(String[] args) throws IOException, InterruptedException {
Socket socket = new Socket("localhost", 12222);
DataInputStream input = new DataInputStream(socket.getInputStream());
DataOutputStream output = new DataOutputStream(socket.getOutputStream());
while(socket.isConnected()){
String response = input.readUTF();
String[] content = response.split(" ");
if(response.startsWith("SENDING CHUNKED")){
int chunks = Integer.parseInt(content[2]);
System.out.println("Read chunk amount of " + chunks);
output.writeUTF("RECEIVING " + chunks);
for(int i = 0; i<chunks; i++){
byte[] chunk = new byte[32 * MB];
System.out.println("Ready to receive...");
int read = input.read(chunk);
System.out.println("Echoing read length of " + read);
output.writeUTF(String.valueOf(read));
}
if(chunks == 0) output.writeUTF("0");
}else if(response.startsWith("SENDING COMPRESSED")){
byte[] compressed = new byte[Integer.parseInt(content[2])];
output.writeUTF("RECEIVING " + compressed.length);
input.read(compressed);
decompress(compressed);
output.writeInt(decompress(compressed).length);
}else if(response.startsWith("SENDING RAW")){
int length = Integer.parseInt(content[2]);
output.writeUTF("RECEIVING " + length);
byte[] received = new byte[length];
input.read(received);
output.writeInt(received.length);
}
}
}
public static byte[] decompress(byte[] in) throws IOException {
try {
ByteArrayOutputStream out = new ByteArrayOutputStream();
InflaterOutputStream infl = new InflaterOutputStream(out);
infl.write(in);
infl.flush();
infl.close();
return out.toByteArray();
} catch (Exception e) {
System.out.println("Error decompressing byte array with length " + in.length);
throw e;
}
}
Using SDK 17
I tried switching around the byte amount, and found the cutoff was right where I stated above. I even replicated this in a test client/server project with no frills (find that here, and found that the cutoff was even lower! I really hope this isn't an actual issue with Java...
The read() method of DataInputStream doesn't directly correspond to the write() method of DataOutputStream. If you want to know how many bytes were sent in a single method call, the server has to inform the client manually.
This is because the read() method, since it doesn't depend on a set length, treats its process as completed when some bytes are read, as it has no way of knowing how many you want.
I have a code in which I read images on a network drive. i read thousands of images, but only sometimes i get following exception occasionally.
java.io.IOException: An unexpected network error occurred
at java.base/sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at java.base/sun.nio.ch.FileDispatcherImpl.read(FileDispatcherImpl.java:54)
at java.base/sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:276)
at java.base/sun.nio.ch.IOUtil.read(IOUtil.java:245)
at java.base/sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:223)
at java.base/sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:65)
at java.base/sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:109)
at java.base/sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)
at java.base/java.io.InputStream.read(InputStream.java:205)
below is the code for which i get it
`
public static int getEPSSectionOffset(File file) throws Exception {
int result = 0;
try (InputStream inputStream =
Files.newInputStream(Paths.get(file.getAbsolutePath()),StandardOpenOption.READ);) {
byte[] fourBytes = new byte[4];
int totalBytesRead = inputStream.read(fourBytes);
if (log.isDebugEnabled())
log.debug("Total bytes read is " + totalBytesRead + " for file " + file.getPath());
if (fourBytes[0] == (byte) 0xC5 && fourBytes[1] == (byte) 0xD0 && fourBytes[2] == (byte) 0xD3
&& fourBytes[3] == (byte) 0xC6) {
totalBytesRead = inputStream.read(fourBytes);
if (log.isDebugEnabled())
log.debug("Total bytes read is " + totalBytesRead + " for file " + file.getPath());
result = makeInt(fourBytes);
}
return (result);
} catch (Exception e) {
log.error("Get EPS Section Offset - " + e.getMessage(), e);
}
return 0;
}`
I get the exception at this line- int totalBytesRead = inputStream.read(fourBytes);
You probably have a issue with the underlying network connection. This is not a type of problem you can fix, there will always be intermittent problems with networks. This means that you will have to live with it, and mitigate the impact.
Maybe something like this:
public static int getEPSSectionOffsetWithRetry(File file) {
int retryCount = 3;
for(int i=0; i < retryCount i++) {
try {
int offset = getEPSSectionOffset()
return offset;
} catch (IOException ex) {
//Maybe wait i little
}
}
throw new IOException("Retry count exceeded");
}
In this programmer i found Prime numbers in first 100.Numbers are in INT format and totally number of them is in DOUBLE format.I want to read that file and i did it for only INT numbers but i dont know hot to do it for DOUBLE number.
Here is the code:
package int1;
import java.io.File;
import java.io.FileOutputStream;
import java.io.IOException;
import java.nio.ByteBuffer;
import java.nio.DoubleBuffer;
import java.nio.IntBuffer;
import java.nio.LongBuffer;
import java.nio.channels.FileChannel;
public class int_upis {
public static void main(String[] args) {
File a = new File("C:\\Users\\Jovan\\Desktop\\Pisem.txt");
FileOutputStream fos = null;
try {
fos = new FileOutputStream(a);
} catch (Exception e) {
}
FileChannel ch = fos.getChannel();
ByteBuffer bff = ByteBuffer.allocate(100);
IntBuffer ibf = bff.asIntBuffer(); // Int type
DoubleBuffer db = bff.asDoubleBuffer(); // Double type
double p = 0;
for (int i = 1; i <= 100; i++) {
int t = 0;
for (int j = 1; j <= i; j++) {
if (i % j == 0) {
t = t + 1;
}
}
if (t < 3) {
p = p + 1; // number of Prime numbers
System.out.println(i);
ibf.put(i);
bff.position(4 * ibf.position());
bff.flip();
try {
ch.write(bff);
bff.clear();
ibf.clear();
} catch (IOException e) {
}
}
}
try {
db.put(p); //At the end of the txt-file i put double format of number (Number of Prime numbers)
bff.position(8*db.position());
bff.flip();
ch.write(bff);
System.out.println("File is writen with: " + ch.size());
ch.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
Now I tried to read this file:
public class int_ispis {
public static void main(String[] args) throws IOException {
File a = new File("C:\\Users\\Jovan\\Desktop\\Pisem.txt");
FileInputStream fis = null;
try {
fis = new FileInputStream(a);
} catch (Exception e) {
}
FileChannel ch = fis.getChannel();
ByteBuffer bff = ByteBuffer.allocate(6 * 4);
This is one line of Prime Number put in a 6-row array (this line below):
int[] niz = new int[6];
System.out.println("Pre flipa: " + bff.position() + " " + bff.limit());
System.out.println("++++++++++++++++++++++++++++++++++++");
while (ch.read(bff) != -1) {
bff.flip();
System.out.println();
System.out.println("Posle flipa: " + bff.position() + " " + bff.limit());
IntBuffer ib = bff.asIntBuffer();
System.out.println("IB: " + ib.position() + " " + ib.limit());
int read = ib.remaining();
System.out.println(read);
When it come to the end of file it puts Double Number as Integer and writes wrong number(How to separate Integer form Double number?)
ib.get(niz, 0, ib.remaining());
for (int i = 0; i < read; i++) {
System.out.print(niz[i] + " ");
}
System.out.println();
System.out.println("=================================");
ib.clear();
bff.clear();
}
}
}
A binary file does not have any "separators".
You need to know the structure of the file content and use this knowledge.
In this programmer i found Prime numbers in first 100.Numbers are in INT format and totally number of them is in DOUBLE format.
This means that there is only one long value in the file and this is in the last 8 bytes. So you simply have to check if the current position is fileLenght - 8 and then read these last 8 bytes as a long value.
Hello My Respected Seniors :)
My Goal: Download a URL Resource, given a URL, by using Multi-Threading in Java, i.e. download a single file into multiple pieces (much like how IDM does) & at the end of download, combine all of them to 1 final file.
Technology Using: Java, RandomAccessFile, MultiThreading, InputStreams
Problem:
The file is downloaded fine with exact KB size, I've checked many times, but the final file is corrupted. For example, If I download an Image, it will be somewhat blurry, If I download an .exe, it downloads fine but when I run the .exe file, it says "media is damaged, retry download".
This is my Main code from which I call to thread class with parameters such as fileName, starting Range and ending Range for a connection as well as a JProgressBar for every thread which will update its own respectively.
public void InitiateDownload()
{
HttpURLConnection uc = (HttpURLConnection) url.openConnection();
uc.connect();
long fileSize = uc.getContentLengthLong();
System.out.println("File Size = "+ fileSize );
uc.disconnect();
chunkSize = (long) Math.ceil(fileSize/6);
startFrom = 0;
endRange = (startFrom + chunkSize) - 1;
Thread t1 = new MyThread(url, fileName, startFrom, endRange, progressBar_1);
t1.start();
//-----------------------------------------
startFrom += chunkSize;
endRange = endRange + chunkSize;
System.out.println("Part 2 :: Start = " + startFrom + "\tEnd To = " + endRange );
Thread t2 = new MyThread(url, fileName, startFrom, endRange, progressBar_2);
t2.start();
//-----------------------------------------
//..
//..
//..
//-----------------------------------------
startFrom += chunkSize;
long temp = endRange + chunkSize;
endRange = temp + (fileSize - temp); //add any remaining bits, that were rounded off in division
Thread t6 = new MyThread(url, fileName, startFrom, endRange, progressBar_6);
t6.start();
//-----------------------------------------
}
Here is run() function of MyThread class:
public void run() {
Thread.currentThread().setPriority(MAX_PRIORITY);
System.setProperty("http.proxyHost", "192.168.10.50");
System.setProperty("http.proxyPort", "8080");
HttpURLConnection uc = null;
try {
uc = (HttpURLConnection) url.openConnection();
uc.setRequestProperty("Range", "bytes="+startFrom+"-"+range);
uc.connect();
fileSize = uc.getContentLengthLong();
inStream = uc.getInputStream();
int[] buffer = new int[ (int) totalDownloadSize ];
file.seek(startFrom); //adjusted start of file
THIS IS WHERE I THINK THE PROBLEM IS,
run() continued...
for(int i = 0 ; i < totalDownloadSize; i++)
{
buffer[i] = inStream.read();
file.write(buffer[i]);
//Updating Progress bars
totalDownloaded = totalDownloaded + 1;
int downloaded = (int) (100 * ( totalDownloaded/ (float) totalDownloadSize)) ;
progressbar.setValue( downloaded );
}
System.err.println( Thread.currentThread().getName() + "'s download is Finished!");
uc.disconnect();
}
catch(IOException e) {
System.err.println("Exception in " + Thread.currentThread().getName() + "\t Exception = " + e );
}
finally {
try {
file.close();
if(inStream!=null)
inStream.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
}
Now the file is downloaded with complete size, but as I said, a little part of it is corrupt.
Now,
If I replace the for loop with following while loop, the problem is completely solved.
int bytesRead = 0;
byte[] buffer = new byte[ (int) totalDownloadSize ];
file.seek(startFrom); //adjusted start of file
while( (bytesRead = inStream.read(buffer) ) != -1 ) {
file.write(buffer, 0, bytesRead);
}
BUT I NEED for LOOP TO MEASURE HOW MUCH FILE EACH THREAD HAS DOWNLOADED & I WANT TO UPGRADE RESPECTIVE JPROGRESSBARs of THREADS.
Kindly help me out with the for loop logic.
OR
If you can advise on how can I upgrade Jprogressbars within while loop. I can't seem to find a way to quantify how much file a thread has downloaded...
I've spent alot of hours & I'm extremely tired now...
You can use the while loop that works, and then keep track of the total amount of bytes read like this:
int totalRead = 0;
while ((bytesRead = inStream.read(buffer)) != -1) {
totalRead += bytesRead;
file.write(buffer, 0, bytesRead);
progressBar.setValue((int)(totalRead / (double) totalDownloadSize));
}
just remember that for (a; b; c) { ... } is equal to a; while (b) { c; ... }.
I have to make an abstaction in my software - replace direct unblockable NIO sockets ( client/server ) to software abstraction.
For example, instead of connecting via tcp client would exec openssl s_client -connect xxx.xxx.xxx.xxx . I have written a little demo, and it even works. Sometimes :(
The first trouble is that Process's streams can't be used with Selector, so I can't replace socketchannel with any other type of channel, so I have to read/write without any chance to avoid blocking.
The second one is that a protocol is a duplex binary file-transfer protocol ( binkp ), so process's buffered streams are unusabe. I've tried to avoid that converting in/out data to base64 and it works, but also sometimes.
I can't understant why it works or not sometimes. I put a piece of test code below. The first word is frame's length, but first bit is ignored. Please, tell me your guesses. Thanks.
public class BufferedSocketBase64 {
static class InToOut implements Runnable {
InputStream is;
OutputStream os;
boolean direction; //
public InToOut(InputStream is, OutputStream os, boolean direction) {
super();
this.is = is;
this.os = os;
this.direction = direction;
}
#Override
public void run() {
System.out.println(Thread.currentThread().getId() + " start "
+ ((direction) ? "encode from to" : "decode from to"));
boolean eof = false;
while (true) {
if (direction) {
// encode to base64 data
try {
int[] head = new int[2];
for (int i = 0; i < 2; i++) {
head[i] = is.read();
}
int len = (head[0] & 0xff << 8 | head[1] & 0xff) & 0x7FFF;
byte[] buf = new byte[len + 2];
buf[0] = (byte) (head[0] & 0xff);
buf[1] = (byte) (head[1] & 0xff);
for (int i = 2; i < len; i++) {
buf[i] = (byte) (is.read() & 0xff);
}
System.out.println(Thread.currentThread()
.getId() + " << " + new String(buf));
if (len > 0) {
String send = Base64Util.encode(buf, len);
send += "\n";
os.write(send.getBytes());
os.flush();
}
} catch (IOException e) {
eof = true;
}
} else { // decode from base64
try {
StringBuilder sb = new StringBuilder(1024);
byte c = 0x0a;
do {
c = (byte) is.read();
if (c >= 0 && c != 0x0a) {
sb.append(new String(new byte[] { c }));
}
} while (c != 0x0a && c >= 0);
if (sb.length() != 0) {
try {
byte[] buf = Base64Util.decode(sb.toString());
System.out.println(Thread.currentThread()
.getId() + " >> " + buf.length);
os.write(buf);
os.flush();
} catch (StringIndexOutOfBoundsException e) {
System.out
.println(Thread.currentThread().getId()
+ " error on " + sb.toString());
}
}
} catch (IOException e) {
eof = true;
}
}
if (eof) {
System.out.println(Thread.currentThread().getId() + " EOF");
break;
}
}
try {
is.close();
os.close();
} catch (IOException e) {
}
}
}
public static void main(String[] args) throws Exception {
Process proc2 = Runtime.getRuntime().exec("nc -l -p 2020");
Process proc1 = Runtime.getRuntime().exec("nc 127.0.0.1 2020");
Socket sock1 = new Socket();
sock1.connect(new InetSocketAddress("127.0.0.1", 24554), 30);
Socket sock2 = new Socket();
sock2.connect(new InetSocketAddress("127.0.0.1", 24557), 30);
new Thread(new InToOut(sock1.getInputStream(), proc1.getOutputStream(),
true)).start();
new Thread(new InToOut(proc1.getInputStream(), sock1.getOutputStream(),
false)).start();
new Thread(new InToOut(sock2.getInputStream(), proc2.getOutputStream(),
true)).start();
new Thread(new InToOut(proc2.getInputStream(), sock2.getOutputStream(),
false)).start();
}
UPDATED:
I've found right way. I uses syncchronized queries for each stream and synchronized threads to fill or erase that queries. All threads mutually blocks themselves. And it works! :)
Sorry for bother.
I've found right way. I uses syncchronized queries for each stream and synchronized threads to fill or erase that queries. All threads mutually blocks themselves. And it works! :) Sorry for bother.