We Keep Coding

How to send compressed data using UDP or TCP (lua / java)

I am making a server with lua clients and Java server.
I need some data to be compressed in order to reduce the data flow.
In order to do this I use LibDeflate for compressing the data on the client side
local config = {level = 1}
local compressed = LibDeflate:CompressDeflate(data, config)
UDP.send("21107"..compressed..serverVehicleID) -- Send data
On the server I use this to receive the packet (TCP)
out = new PrintWriter(clientSocket.getOutputStream(), true);
in = new BufferedReader(new
InputStreamReader(clientSocket.getInputStream(), "UTF-8"));
String inputLine;
while ((inputLine = in.readLine()) != null) { // Wait for data
Log.debug(inputLine); // It is what get printed in the exemple
String[] processedInput = processInput(inputLine);
onDataReceived(processedInput);
}
I already tried sending it using UDP and TCP, the problem is the same.
I tried using LibDeflate:CompressDeflate and LibDeflate:CompressZlib
I tried tweaking the config
Nothing works :/
I expect to receive one packet with the whole string
But I received few packets each of them contains compressed characters. exemple (each line is the server think that he receive a new packet):
(source: noelshack.com)

After a lot of research I finnaly managed to fix my problem !
I used this :
DataInputStream in = new DataInputStream(new BufferedInputStream(clientSocket.getInputStream()));
int count;
byte[] buffer = new byte[8192]; // or 4096, or more
while ((count = in.read(buffer)) > 0) {
String data = new String(buffer, 0, count);
Do something...
}
I still haven't tested to see if the received compressed string works, I'll update my post when I try out.
EDIT: It seems to work
The only problem now is that I don't know what to do when the packet is bigger than the buffer size.
I want to have something that work in every situation and since some packet are bigger than 8192 they are just cut in half.

Assuming that the client side sends a single compressed "document", your server-side code should look something like this (TCP version):
is = new DeflaterInputStream(clientSocket.getInputStream());
in = new BufferedReader(new InputStreamReader(is, "UTF-8"));
String inputLine;
while ((inputLine = in.readLine()) != null) {
...
}
The above is untested, and also needs exception handling and code to ensure that the streams always get closed.
The trick is that your input pipeline needs to decompress the data stream before you attempt to read / process it as lines of text.

Wrting a HTTP proxy in Java using only the Socket class

I'm trying to write an HTTP proxy in Java using only the Socket class. I had attempted to construct one earlier, and I was successfully sending a request by writing to the socket's output stream But I am having a hard time reading the response. the research I have conducted suggests that I should use the input stream and read it line by line, but I have not been able to read any web-pages successfully using this method. Would anyone have any suggestions as to where I could go from here?
My code actually uses a byte buffer to read from the input stream in order to read the page in bytes:
InputStream input = clientSocket.getInputStream()
byte[] buffer = new byte[48*1024];
byte[] redData;
StringBuilder clientData = new StringBuilder();
String redDataText;
int red;
while((red = input.read(buffer)) > -1) {
redData = new byte[red];
System.arraycopy(buffer, 0, redData, 0, red);
redDataText = new String(redData, "UTF-8");
System.out.println("Got message!! " + redDataText);
clientData.append(redDataText);
}

If you are asking for a way to read an InputStream by lines, this one may serve you:
BufferedReader bufferedReader=new BufferedReader(new InputStreamReader(input, "UTF-8"));
String line;
StringBuilder clientData=new StringBuilder();
while ((line=bufferedReader.readLine()) != null)
{
clientData.append(line);
}
You have to be careful not to read an InputStream in this fashion unless you are a priori sure that it contains just plain text (and not binary data).
BTW: For shake of efficiency, I recommend you to pre-size the clientData with an initial size according to the final size (if not, it will start from a default size of 10, and will need to be re-sized more times).

how to send both binary file and text using the same socket

i have to send a short string as text from client to server and then after that send a binary file.
how would I send both binary file and the string using the same socket connection?
the server is a java desktop application and the client is an Android tablet. i have already set it up to send text messages between the client and server in both directions. i have not yet done the binary file sending part.
one idea is to set up two separate servers running at the same time. I think this is possible if i use two different port numbers and set up the servers on two different threads in the application. and i would have to set up two concurrent clients running on two services in the Android app.
the other idea is to somehow use an if else statement to determine which of the two types of files is being sent, either text of binary, and use the appropriate method to receive the file for the file type being sent.
example code for sending text
PrintWriter out;
BufferedReader in;
out = new PrintWriter(new BufferedWriter
(new OutputStreamWriter(Socket.getOutputStream())) true,);
in = new BufferedReader(new InputStreamReader(socket.getInputStream()));
out.println("test out");
String message = in.readLine();
example code for sending binary file
BufferedOutputStream out;
BufferedInputStream in;
byte[] buffer = new byte[];
int length = 0;
out = new BufferedOutputStream(new FileOutputStream("test.pdf));
in = new BufferedInputStream(new FileOutputStream("replacement.pdf"));
while((length = in.read(buffer)) > 0 ){
out.write(buffer, 0, length);
}

I don't think using two threads would be necessary in your case. Simply use the socket's InputStream and OutputStream in order to send binary data after you have sent your text messages.
Server Code
OutputStream stream = socket.getOutputStream();
PrintWriter out = new PrintWriter(
new BufferedWriter(
new OutputStreamWriter(stream)
)
);
out.println("test output");
out.flush(); // ensure that the string is not buffered by the BufferedWriter
byte[] data = getBinaryDataSomehow();
stream.write(data);
Client Code
InputStream stream = socket.getInputStream();
String message = readLineFrom(stream);
int dataSize = getSizeOfBinaryDataSomehow();
int totalBytesRead = 0;
byte[] data = new byte[dataSize];
while (totalBytesRead < dataSize) {
int bytesRemaining = dataSize - totalBytesRead;
int bytesRead = stream.read(data, totalBytesRead, bytesRemaining);
if (bytesRead == -1) {
return; // socket has been closed
}
totalBytesRead += bytesRead;
}
In order to determine the correct dataSize on the client side you have to transmit the size of the binary block somehow. You could send it as a String right before out.flush() in the Server Code or make it part of your binary data. In the latter case the first four or eight bytes could hold the actual length of the binary data in bytes.
Hope this helps.
Edit
As #EJP correctly pointed out, using a BufferedReader on the client side will probably result in corrupted or missing binary data because the BufferedReader "steals" some bytes from the binary data to fill its buffer. Instead you should read the string data yourself and either look for a delimiter or have the length of the string data transmitted by some other means.
/* Reads all bytes from the specified stream until it finds a line feed character (\n).
* For simplicity's sake I'm reading one character at a time.
* It might be better to use a PushbackInputStream, read more bytes at
* once, and push the surplus bytes back into the stream...
*/
private static String readLineFrom(InputStream stream) throws IOException {
InputStreamReader reader = new InputStreamReader(stream);
StringBuffer buffer = new StringBuffer();
for (int character = reader.read(); character != -1; character = reader.read()) {
if (character == '\n')
break;
buffer.append((char)character);
}
return buffer.toString();
}

You can read about how HTTP protocol works which essentially sends 'ascii and human readable' headers (so to speak) and after that any content can be added with appropriate encoding like base64 for example. You may create sth similar yourself.

You need to first send the String, then the size of the byte array then the byte array, use String.startsWith() method to check what is being send.

Read data from a Java Socket

I have a Socket listening on some x port.
I can send the data to the socket from my client app but unable to get any response from the server socket.
BufferedReader bis = new BufferedReader(new
InputStreamReader(clientSocket.getInputStream()));
String inputLine;
while ((inputLine = bis.readLine()) != null)
{
instr.append(inputLine);
}
This code part reads data from server.
But I can't read anything from server until unless the Socket on the server is closed.
Server code is not under my control to edit something on it.
How can I overcome this from client code.
Thanks

Looks like the server may not be sending newline characters (which is what the readLine() is looking for). Try something that does not rely on that. Here's an example that uses the buffer approach:
Socket clientSocket = new Socket("www.google.com", 80);
InputStream is = clientSocket.getInputStream();
PrintWriter pw = new PrintWriter(clientSocket.getOutputStream());
pw.println("GET / HTTP/1.0");
pw.println();
pw.flush();
byte[] buffer = new byte[1024];
int read;
while((read = is.read(buffer)) != -1) {
String output = new String(buffer, 0, read);
System.out.print(output);
System.out.flush();
};
clientSocket.close();

To communicate between a client and a server, a protocol needs to be well defined.
The client code blocks until a line is received from the server, or the socket is closed. You said that you only receive something once the socket is closed. So it probably means that the server doesn't send lines of text ended by an EOL character. The readLine() method thus blocks until such a character is found in the stream, or the socket is closed. Don't use readLine() if the server doesn't send lines. Use the method appropriate for the defined protocol (which we don't know).

For me this code is strange:
bis.readLine()
As I remember, this will try to read into a buffer until he founds a '\n'. But what if is never sent?
My ugly version breaks any design pattern and other recommendations, but always works:
int bytesExpected = clientSocket.available(); //it is waiting here
int[] buffer = new int[bytesExpected];
int readCount = clientSocket.read(buffer);
You should add the verifications for error and interruptions handling too.
With webservices results this is what worked for me ( 2-10MB was the max result, what I have sent)

Here is my implementation
clientSocket = new Socket(config.serverAddress, config.portNumber);
BufferedReader in = new BufferedReader(new InputStreamReader(clientSocket.getInputStream()));
while (clientSocket.isConnected()) {
data = in.readLine();
if (data != null) {
logger.debug("data: {}", data);
}
}

Android socket inputstream read (followed by an EPIPE)

I am creating some client-side socket software to read events from a server. (For example, streaming stock quotes).
PROBLEM: .read(b); is returning immediately with a value of -1. This causes an infinte loop and the phone becomes very hot. Additionally, all checks to s.isConnected(), isOpen(), isBound() return true. Essentially the socket looks connected. (This is an error scenario, so any value of .setSoTimeout(x) has no effect. 12 minutes, or leave empty. The .read(b) always returns -1 immediately).
When I write to it later, via the getOutputStream(), I receive an exception EPIPE (broken pipe).
Here's the core code (log statements/value checks omitted for brevity).
s.connect(new InetSocketAddress(host, port), CONNECT_TIMEOUT_MILLIS);
byte[] b = new byte[1024];
while (s.isConnected()) {
int bytesToRead = s.getInputStream().read(b);
if (bytesToRead <= 0) {
LOGGER.debug("no bytes read? trying again.");
continue;
}
processFrame(b);
}
If I cycle the 3g on the phone, it works fine. Sometimes it gets into this weird state.
Question
Am I doing something wrong? Is this the expected behavior? Is there existing code I could look at to show the right way to do socket programming on Android?

I use the following code without problem:
BufferedReader in = new BufferedReader(new InputStreamReader(socket.getInputStream()));
String line = null;
while ((line = in.readLine()) != null) {
myLineProcess(line); //here you process you line result
}
good luck.

If read(byte[], ...) returns < 0, the peer has closed the connection, so you must close the socket and exit the loop. There will never be any more data.
Input streams are blocking, so the only way read(byte[], ...) can return zero is if you specify a zero length buffer or a zero length, depending on which overload you call. As you aren't doing that, it will never return zero.
Instead it will do exactly what it says in the Javadoc: either return -1 meaning EOS, or block until at least one byte of data is available.
Am I doing something wrong?
Almost everything. Your loop testing for <= 0 is completely pointless and completely incorrect. And so is testing isConnected(). That only tells you whether you ever connected this Socket. It doesn't change with the state of the connection. The return code of -1 tells you that. Your loop should read:
while ((bytesToRead = s.getInputStream().read(b)) > 0)
{
// do something with b[0..bytesToRead -1].
}
s.close();

Well i don't know whether this would be the appropriate answer for this, but still i will like to give you the code, which does works well every time.
Please keep it simple,
Try using InputStream, InputStreamReader, BufferedReader, OutputStream, PrintWriter.
Client Side:
Socket s = new Socket();
s.connect(new InetSocketAddress("Server_IP",Port_no),TimeOut);
// Let Timeout be 5000
Server Side:
ServerSocket ss = new ServerSocket(Port_no);
Socket incoming = ss.accept();
For Reading from the Socket:
InputStream is = s.getInputStream();
InputStreamReader isr = new InputStreamReader(is);
BufferedReader br = new BufferedReader(isr);
boolean isDone = false;
String s = new String();
while(!isDone && ((s=br.readLine())!=null)){
System.out.println(s); // Printing on Console
}
For Writing to the Socket:
OutputStream os = s.getOuptStream();
PrintWriter pw = new PrintWriter(os)
pw.println("Hello");