We Keep Coding

How to programmatically limit the download speed?

I use the following code to limit the download speed of a file in java:
package org;
import java.io.IOException;
import java.io.InputStream;
import java.net.HttpURLConnection;
import java.net.URL;
class MainClass {
public static void main(String[] args) {
download("https://speed.hetzner.de/100MB.bin");
}
public static void download(String link) {
try {
URL url = new URL(link);
HttpURLConnection con = (HttpURLConnection) url.openConnection();
con.setConnectTimeout(5000);
con.setReadTimeout(5000);
InputStream is = con.getInputStream();
CustomInputStream inputStream = new CustomInputStream(is);
byte[] buffer = new byte[2024];
int len;
while ((len = inputStream.read(buffer)) != -1) {
System.out.println("downloaded : " + len);
//save file
}
} catch (IOException e) {
e.printStackTrace();
}
}
public static class CustomInputStream extends InputStream {
private static final int MAX_SPEED = 8 * 1024;
private final long ONE_SECOND = 1000;
private long downloadedWhithinOneSecond = 0L;
private long lastTime = System.currentTimeMillis();
private InputStream inputStream;
public CustomInputStream(InputStream inputStream) {
this.inputStream = inputStream;
lastTime = System.currentTimeMillis();
}
#Override
public int read() throws IOException {
long currentTime;
if (downloadedWhithinOneSecond >= MAX_SPEED
&& (((currentTime = System.currentTimeMillis()) - lastTime) < ONE_SECOND)) {
try {
Thread.sleep(ONE_SECOND - (currentTime - lastTime));
} catch (InterruptedException e) {
e.printStackTrace();
}
downloadedWhithinOneSecond = 0;
lastTime = System.currentTimeMillis();
}
int res = inputStream.read();
if (res >= 0) {
downloadedWhithinOneSecond++;
}
return res;
}
#Override
public int available() throws IOException {
return inputStream.available();
}
#Override
public void close() throws IOException {
inputStream.close();
}
}
}
The download speed is successfully limited, but a new problem arises. When the download is in progress, and I disconnect from the internet, the download does not end and continues for a while. When i disconnect the internet connection, it takes more than 10 seconds to throw a java.net.SocketTimeoutException exception. I do not really understand what happens in the background.
Why does this problem arise?

Your rate limit doesn't actually work like you think it does, because the data is not actually sent byte-per-byte, but in packets. These packets are buffered, and what you observe (download continues without connection) is just your stream reading the buffer. Once it reaches the end of your buffer, it waits 5 seconds before the timeout is thrown (because that is what you configured).
You set the rate to 8 kB/s, and the normal packet size is normally around 1 kB and can go up to 64 kB, so there would be 8 seconds where you are still reading the same packet. Additionally it is possible that multiple packets were already sent and buffered. There exists also a receive buffer, this buffer can be as small as 8 - 32 kB up to several MB. So really you are just reading from the buffer.
[EDIT]
Just to clarify, you are doing the right thing. On average, the rate will be limited to what you specify. The server will send a bunch of data, then wait until the client has emptied its buffer enough to receive more data.

You apparently want to limit download speed on the client side, and you also want the client to respond immediately to the connection being closed.
AFAIK, this is not possible ... without some compromises.
The problem is that the only way that the client application can detect that the connection is closed is by performing a read operation. That read is going to deliver data. But if you have already reached your limit for the current period, then that read will push you over the limit.
Here are a couple of ideas:
If you "integrate" the download rate over a short period (e.g. 1kbytes every second versus 10kbytes every 10 seconds) then you can reduce the length of time for the sleep calls.
When you are close to your target download rate, you could fall back to doing tiny (e.g. 1 byte) reads and small sleeps.
Unfortunately, both of these will be inefficient on the client side (more syscalls), but this is the cost you must pay if you want your application to detect connection closure quickly.
In a comment you said:
I'd expect the connection to be reset as soon as the internet connection is disabled.
I don't think so. Normally, the client-side protocol stack will deliver any outstanding data received from the network before telling the application code that the connection it is reading has been closed.

CPU-Wise, How can I optimize UDP packet sending?

I currently have a game, for which I have implemented a client and a server.
I then have the server sending data to the client about it's position, the client sending movement inputs into the server, etc.
The problem is that the CPU skyrockets to 100%. I have directly connected the high usage to the following code, which is in an update() method that is called ten times per second:
try{
sendToClientUDP(("ID:" + String.valueOf(uid)));
sendToClientUDP(("Scale:" + GameServer.scale));
for (Clients cl : GameServer.players){
//sendToClient(("newShip;ID:" + cl.uid).getBytes(), packet.getAddress(), packet.getPort());
sendToClientUDP((("UID:" + cl.uid +";x:" + cl.x)));
sendToClientUDP((("UID:" + cl.uid +";y:" + cl.y)));
sendToClientUDP((("UID:" + cl.uid +";z:" + cl.z)));
sendToClientUDP((("UID:" + cl.uid +";Rotation:" + (cl.rotation))));
cl.sendToClientUDP(new String("newShip;ID:" + uid));
sendToClientUDP(new String("newShip;ID:" + cl.uid));
}
}catch (Exception e){
e.printStackTrace();
}
Removing the code, and the high CPU usage disappears.
Here is my sendToClientUDP() method.
public void sendToClientUDP(String str){
if (!NPC){ //NPC is checking if it is a computer-controlled player.
UDP.sendData(str.getBytes(), ip, port);
}
}
And here is my UDP.sendData() method:
public static void sendData(String data, InetAddress ip, int port) {
sendData(data.getBytes(), ip, port);
}
public static void sendData(byte[] data, InetAddress ip, int port) {
DatagramPacket packet = new DatagramPacket(data, data.length, ip, port);
try {
socket.send(packet);
} catch (IOException e) {
e.printStackTrace();
}
}
Why is so much CPU being used simply by sending UDP packets? And what, if anything, can I do to reduce it?

I suggest you take out or optimise the code which is producing so much CPU, A CPU profiler is the best place to start but these are likely to be causes of CPU consumption.
creating Strings and byte[] are expensive, I would avoid doing those.
creating multiple packets instead of batching them is also expensive.
Creating a new DatagramPacket can be avoided.
I would remove duplication between messages as this adds redundant work you can avoid.
you might consider using a binary format to avoid the translation overhead of convert to/from text.
There is almost never a good time to use new String() it is almost certainly redundant.
EDIT: This is what I had in mind. Instead of sending 5 packets per client, you send just one packet, total. For ten clients you send 1/50 of the packets, reducing the overhead.
import java.io.IOException;
import java.net.*;
import java.nio.ByteBuffer;
import java.nio.ByteOrder;
import java.util.ArrayList;
import java.util.List;
/**
* Created by peter on 31/07/15.
*/
public class PacketSender {
public static void main(String[] args) throws IOException {
PacketSender ps = new PacketSender(InetAddress.getByName("localhost"), 12345);
List<Client> clients = new ArrayList<>();
for(int i=0;i<10;i++)
clients.add(new Client());
for(int t = 0; t< 3;t++) {
long start = System.nanoTime();
int tests = 100000;
for (int i = 0; i < tests; i++) {
ps.sendData(1234, 1, clients);
}
long time = System.nanoTime() - start;
System.out.printf("Sent %,d messages per second%n", (long) (tests * 1e9 / time));
}
}
final ThreadLocal<ByteBuffer> bufferTL = ThreadLocal.withInitial(() -> ByteBuffer.allocate(8192).order(ByteOrder.nativeOrder()));
final ThreadLocal<DatagramSocket> socketTL;
final ThreadLocal<DatagramPacket> packetTL;
public PacketSender(InetAddress address, int port) {
socketTL = ThreadLocal.withInitial(() -> {
try {
return new DatagramSocket(port, address);
} catch (SocketException e) {
throw new AssertionError(e);
}
});
packetTL = ThreadLocal.withInitial(() -> new DatagramPacket(bufferTL.get().array(), 0, address, port));
}
public void sendData(int uid, int scale, List<Client> clients) throws IOException {
ByteBuffer b = bufferTL.get();
b.clear();
b.putInt(uid);
b.putInt(scale);
b.putInt(clients.size());
for (Client cl : clients) {
b.putInt(cl.x);
b.putInt(cl.y);
b.putInt(cl.z);
b.putInt(cl.rotation);
b.putInt(cl.uid);
}
DatagramPacket dp = packetTL.get();
dp.setData(b.array(), 0, b.position());
socketTL.get().send(dp);
}
static class Client {
int x,y,z,rotation,uid;
}
}
When this performance test runs it prints
Sent 410,118 messages per second
Sent 458,126 messages per second
Sent 459,499 messages per second
Edit: to write/read text you can do the following.
import java.nio.ByteBuffer;
/**
* Created by peter on 09/08/2015.
*/
public enum ByteBuffers {
;
/**
* Writes in ISO-8859-1 encoding. This assumes string up to 127 bytes long.
*
* #param bb to write to
* #param cs to write from
*/
public static void writeText(ByteBuffer bb, CharSequence cs) {
// change to stop bit encoding to have lengths > 127
assert cs.length() < 128;
bb.put((byte) cs.length());
for (int i = 0, len = cs.length(); i < len; i++)
bb.put((byte) cs.charAt(i));
}
public static StringBuilder readText(ByteBuffer bb, StringBuilder sb) {
int len = bb.get();
assert len >= 0;
sb.setLength(0);
for (int i = 0; i < len; i++)
sb.append((char) (bb.get() & 0xFF));
return sb;
}
private static final ThreadLocal<StringBuilder> SB = new ThreadLocal<>() {
#Override
protected Object initialValue() {
return new StringBuilder();
}
};
public static String readText(ByteBuffer bb) {
// TODO use a string pool to reduce String garbage.
return readText(bb, SB.get()).toString();
}
}
If you need something more complicated you should consider using Chronicle-Bytes which I wrote. It has
support for 64-bit memory sizes, including memory mapping 64-bit.
thread safe operation off heap.
UTF-8 encoding of strings.
compressed types such as stop bit encoding.
automatic string pooling to reduce garbage.
deterministic clean up of off heap resources via reference counting.

Calculating the bandwidth by sending several packets through linear regression

I implemented a TCP client-server model to test my bandwidth with the server through sending number of packets with different sizes and see the RTT then calculate the bandwidth through linear regression,
Here is the server code:
import java.io.*;
import java.net.*;
public class Server implements Runnable {
ServerSocket welcomeSocket;
String clientSentence;
Thread thread;
Socket connectionSocket;
BufferedReader inFromClient;
DataOutputStream outToClient;
public Server() throws IOException {
welcomeSocket = new ServerSocket(6588);
connectionSocket = welcomeSocket.accept();
inFromClient = new BufferedReader(new InputStreamReader(connectionSocket.getInputStream()));
outToClient = new DataOutputStream(connectionSocket.getOutputStream());
thread = new Thread(this);
thread.start();
}
#Override
public void run() {
// TODO Auto-generated method stub
while(true)
{
try {
clientSentence = inFromClient.readLine();
if (clientSentence != null) {
System.out.println("Received: " + clientSentence);
outToClient.writeBytes(clientSentence + '\n');
}
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
public static void main(String[] args) throws IOException {
new Server();
}
}
And this is the method in the Client class that return an array of the RTT by each packet
public int [] getResponseTime() throws UnknownHostException, IOException {
timeArray = new int[sizes.length];
for (int i = 0; i < sizes.length; i++) {
sentence = StringUtils.leftPad("", sizes[i], '*');
long start = System.nanoTime();
outToServer.writeBytes(sentence + '\n');
modifiedSentence = inFromServer.readLine();
long end = System.nanoTime();
System.out.println("FROM SERVER: " + modifiedSentence);
timeArray[i] = (int) (end - start);
simpleReg.addData(timeArray[i]* Math.pow(10, -9), sizes[i] * 2); // each char is 2 bytes
}
return timeArray;
}
when i get the slope it returns me a BW with kilo bytes however they are in the same network and the bandwidth should be much more . What i am doing wrong ?

Are you obliged to use linear regression or could it be a different estimator? I am actually not sure if linear regression is the best approach here. I am curious, do you happen to know any sources that suggest to use it in this kind of situation?
Note, that especially the initial BW measurements are much smaller than the real maximal goodput (due to TCP slow-start), so it is important to use a metric estimation that takes large wrong outliers into account.
In previous work I have used the harmonic mean to monitor the bandwidth over a longer period of time and it worked pretty good (also on links with a large bandwidth). The advantage of the harmonic mean over other means, is that while it is still very easy to compute, it mitigates the impact of large outliers, meaning the estimate is not as easily falsified.
Given a series of bandwidth measurements R_i, where i=0,1,2,..., n-1, the harmonic mean is calculated as:
R_total = (n+1)/((n/R_total) + (1/R_n))
It is also good practice to skip the first few measurement values (depending on how often you measure...), e.g., R_(0..5), since you might have initial bursts due to initial preparations in the different layers and are in the slow-start phase anyways.
Here an example implementation in Java. Even though in this case the measurement is done through a file download, it can be easily applied to your environment too - simply use your echo server instead of the file download:
public class Estimator
{
private static double R; // harmonic mean of all bandwidth measurements
private static int n = 0; // number of measurements
private static int skips = 5; // skip measurements for first 5 socket.read() operations
// size in bytes
// start/end in ns
public static double harmonicMean(long start, long end, double size){
// check if we need to skip this initial value, since it might falsify our estimate
if(skips-- > 0) return 0;
// get current value of R
double curR = (size/(1024*1024))/(double)((end - start)*Math.pow(10, -9));
System.out.println(curR);
if(n == 0) {
// initial value
R = curR;
} else {
// use harmonic mean
R = (n+1)/((n/R)+(1/curR));
}
n++;
return R;
}
public static void main(String[] args)
{
// temporary buffer to hold bytes
byte[] buffer = new byte[1024*1024*10]; // 10MB buffer - just in case ...
Socket socket = null;
try {
// measurement done through file download from server
// prepare request
socket = new Socket("yourserver.com",80);
PrintWriter pw = new PrintWriter(socket.getOutputStream());
InputStream is = socket.getInputStream();
pw.println("GET /test_blob HTTP/1.1"); // a test file, e.g., 1MB big
pw.println("Host: yourserver.com");
pw.println("");
pw.flush();
// prepare measurement
long start,end;
double bytes = 0;
double totalBytes = 0;
start = System.nanoTime();
while((bytes = is.read(buffer)) != -1) {
// socket.read() occurred -> calculate harmonic mean
end = System.nanoTime();
totalBytes += bytes;
harmonicMean(start, end, totalBytes);
}
// clean up
is.close();
pw.close();
}
catch(Exception e){
e.printStackTrace();
}
finally {
if(socket != null) {
try{
socket.close();
}
catch(Exception e){
e.printStackTrace();
}
}
}
System.out.println(R+" MB/s");
}
}
Additionally, for the sake of completeness, as I already mentioned in the comments it is important that the test messages/files are big enough, so TCP reaches the full goodput potential of the link.
Please also note, that this is a simplified way to estimate the bandwidth. In this example we start measuring (taking the first timestamp) from when the request was sent, meaning we include the link propagation and server processing delay, which in return will reduce the overall estimated value. Anyways, since you seem to use a local network, I expect the sum of these delays to be rather small, which means they will not falsify the final estimate too much.
I wrote a small blog post concerning measuring TCP connection metrics inside an application layer. Everything is described in more detail there (though the code examples are in C).

How to read (all available) data from serial connection when using JSSC?

I'm trying to work with JSSC.
I built my app according to this link:
https://code.google.com/p/java-simple-serial-connector/wiki/jSSC_examples
My event handler looks like:
static class SerialPortReader implements SerialPortEventListener {
public void serialEvent(SerialPortEvent event) {
if(event.isRXCHAR()){//If data is available
try {
byte buffer[] = serialPort.readBytes();
}
catch (SerialPortException ex) {
System.out.println(ex);
}
}
}
}
}
The problem is that I'm always not getting the incoming data in one piece. (I the message has a length of 100 bytes, Im getting 48 and 52 bytes in 2 separates calls)
- The other side send me messages in different lengths.
- In the ICD Im working with, there is a field which tell us the length of the message. (from byte #10 to byte #13)
- I cant read 14 bytes:
(serialPort.readBytes(14);,
parse the message length and read the rest of the message:
(serialPort.readBytes(messageLength-14);
But if I will do it, I will not have the message in once piece (I will have 2 separates byte[] and I need it in one piece (byte[]) without the work of copy function.
Is it possible ?
When working with Ethernet (SocketChannel) we can read data using ByteBuffer. But with JSSC we cant.
Is there a good alternative to JSSC ?
Thanks

You can't rely on any library to give you all the content you need at once because :
the library dont know how many data you need
the library will give you data as it comes and also depending on buffers, hardware, etc
You must develop your own business logic to handle your packets reception. It will of course depend on how your packets are defined : are they always the same length, are they separated with same ending character, etc.
Here is an example that should work with your system (note you should take this as a start, not a full solution, it doesn't include timeout for example) :
static class SerialPortReader implements SerialPortEventListener
{
private int m_nReceptionPosition = 0;
private boolean m_bReceptionActive = false;
private byte[] m_aReceptionBuffer = new byte[2048];
#Override
public void serialEvent(SerialPortEvent p_oEvent)
{
byte[] aReceiveBuffer = new byte[2048];
int nLength = 0;
int nByte = 0;
switch(p_oEvent.getEventType())
{
case SerialPortEvent.RXCHAR:
try
{
aReceiveBuffer = serialPort.readBytes();
for(nByte = 0;nByte < aReceiveBuffer.length;nByte++)
{
//System.out.print(String.format("%02X ",aReceiveBuffer[nByte]));
m_aReceptionBuffer[m_nReceptionPosition] = aReceiveBuffer[nByte];
// Buffer overflow protection
if(m_nReceptionPosition >= 2047)
{
// Reset for next packet
m_bReceptionActive = false;
m_nReceptionPosition = 0;
}
else if(m_bReceptionActive)
{
m_nReceptionPosition++;
// Receive at least the start of the packet including the length
if(m_nReceptionPosition >= 14)
{
nLength = (short)((short)m_aReceptionBuffer[10] & 0x000000FF);
nLength |= ((short)m_aReceptionBuffer[11] << 8) & 0x0000FF00;
nLength |= ((short)m_aReceptionBuffer[12] << 16) & 0x00FF0000;
nLength |= ((short)m_aReceptionBuffer[13] << 24) & 0xFF000000;
//nLength += ..; // Depending if the length in the packet include ALL bytes from the packet or only the content part
if(m_nReceptionPosition >= nLength)
{
// You received at least all the content
// Reset for next packet
m_bReceptionActive = false;
m_nReceptionPosition = 0;
}
}
}
// Start receiving only if this is a Start Of Header
else if(m_aReceptionBuffer[0] == '\0')
{
m_bReceptionActive = true;
m_nReceptionPosition = 1;
}
}
}
catch(Exception e)
{
e.printStackTrace();
}
break;
default:
break;
}
}
}

After writing data to serial port it need to be flushed. Check the timing and pay attention to the fact that read should occur only after other end has written. read size is just an indication to read system call and is not guaranteed. The data may have arrived and is buffered in serial port hardware buffer but may not have been transferred to operating system buffer hence not to application. Consider using scm library, it flushes data after each write http://www.embeddedunveiled.com/

Try this:
Write your data to the serial port (using serialPort.writeBytes()) and if you are expecting a response, use this:
byte[] getData() throws SerialPortException, IOException {
ByteArrayOutputStream baos = new ByteArrayOutputStream();
byte[] b;
try {
while ((b = serialPort.readBytes(1, 100)) != null) {
baos.write(b);
// System.out.println ("Wrote: " + b.length + " bytes");
}
// System.out.println("Returning: " + Arrays.toString(baos.toByteArray()));
} catch (SerialPortTimeoutException ex) {
; //don't want to catch it, it just means there is no more data to read
}
return baos.toByteArray();
}
Do what you want with the returned byte array; in my case I just display it for testing.
I found it works just fine if you read one byte at a time, using a 100ms timeout, and when it does time out, you've read all data in the buffer.
Source: trying to talk to an Epson serial printer using jssc and ESC/POS.

Java sending byte[] through sockets...wrong length read

For my homework assignment, I have a network of Nodes that are passing messages to each other. Each Node is connected to a set amount of other Nodes (I'm using 4 for testing). Each Link has a weight, and all the Nodes have computed the shortest path for how they want their messages sent. Every Packet that is sent is composed of the message protocol (a hard-coded int), an integer that tells how many messages have passed through the sending Node, and the routing path for the Packet.
Every Node has a Thread for each of its Links. There is an active Socket in each Link. The Packets are sent by adding a 4-byte int to the beginning of the message telling the message's length.
Everything works fine until I stress the network. For my test, there are 10 Nodes, and I get 5 of them to send 10000 packets in a simple while() loop with no Thread.sleep(). Without exception, there is always an error at some point during execution at the if(a!=len) statement.
Please let me know if I can clarify anything. Thanks in advance! Here is the code (from the Link Thread; send() and forward() are called from the Node itself):
protected void listen(){
byte[] b;
int len;
try{
DataInputStream in = new DataInputStream(sock.getInputStream());
while(true){
len = in.readInt();
b = new byte[len];
int a = in.read(b,0,len);
if(a!=len){
System.out.println("ERROR: " + a + "!=" + len);
throw new SocketException(); //may have to fix...this will happen when message is corrupt/incomplete
}
Message m = new Message(b);
int p = m.getProtocol();
switch (p){
case CDNP.PACKET:
owner.incrementTracker();
System.out.print("\n# INCOMMING TRACKER: " + m.getTracker() + "\n>>> ");
owner.forward(m);
}
}
}catch (IOException e){
e.printStackTrace();
}
}
public void send(int tracker){
String[] message = { Conv.is(CDNP.PACKET), Conv.is(tracker), owner.getMST().toString() };
Message m = new Message(message);
forward(m);
}
public synchronized void forward(Message m){
try{
OutputStream out = sock.getOutputStream();
//convert length to byte array of length 4
ByteBuffer bb = ByteBuffer.allocate(4+m.getLength());
bb.putInt(m.getLength());
bb.put(m.getBytes());
out.write(bb.array());
out.flush();
}catch (UnknownHostException e){
System.out.println("ERROR: Could not send to Router at " + sock.getRemoteSocketAddress().toString());
return;
}catch (IOException e1){
}
}

int a = in.read(b,0,len);
if(a!=len){
That won't work. The InputStream may not read all the bytes you want, it may read only what is available right now, and return that much without blocking.
To quote the Javadocs (emphasis mine):
Reads up to len bytes of data from the input stream into an array of bytes. An attempt is made to read as many as len bytes, but a smaller number may be read, possibly zero. The number of bytes actually read is returned as an integer.
You need to continue reading in a loop until you have all the data you want (or the stream is finished).
Or, since you are using a DataInputStream, you can also use
in.readFully(b, 0, len);
which always reads exactly len bytes (blocking until those have arrived, throwing an exception when there is not enough data).