I have a TCP socket client receiving messages (data) from a server.
messages are of the type length (2 bytes) + data (length bytes), delimited by STX & ETX characters.
I'm using a bufferedReader to retrieve the two first bytes, decode the length, then read again from the same bufferedReader the appropriate length and put the result in a char array.
most of the time, I have no problem, but SOMETIMES (1 out of thousands of messages received), when attempting to read (length) bytes from the reader, I get only part of it, the rest of my array being filled with "NUL" characters. I imagine it's because the buffer has not yet been filled.
char[] bufLen = new char[2];
_bufferedReader.read(bufLen);
int len = decodeLength(bufLen);
char[] _rawMsg = new char[len];
_bufferedReader.read(_rawMsg);
return _rawMsg;
I solved the problem in several iterative ways:
first I tested the last char of my array: if it wasn't ETX I would read chars from the bufferedReader one by one until I would reach ETX, then start over my regular routine. the
consequence is that I would basically DROP one message.
then, in order to still retrieve that message, I would find the first occurence of the NUL char in my "truncated" message, read & store additional characters one at a time until I reached ETX, and append them to my "truncated" messages, confirming length is ok.
it works also, but I'm really thinking there's something I could do better, like checking if the total number of characters I need are available in the buffer before reading it, but can't find the right way to do it...
any idea / pointer ?
thanks !
The InputStream read method may return short reads; you must check the return value to determine how many characters were read, and continue reading in a loop until you get the number you wanted. The method may block, but it only blocks until some data is available, not necessarily all the data you requested.
Most people end up writing a "readFully" method, like DataInputStream, which reads the amount of data expected, or throws an IOException:
static public int readFully(InputStream inp, byte[] arr, int ofs, int len) throws IOException {
int rmn,cnt;
for(rmn=len; rmn>0; ofs+=cnt, rmn-=cnt) {
if((cnt=inp.read(arr,ofs,rmn))==-1) {
throw new IOException("End of stream encountered before reading at least "+len+" bytes from input stream");
}
}
return len;
}
Here is a sample server that I have used for testing
The main rcv is structured like
while((chars_read = from_server.read(buffer)) != -1)
{
to_user.write(buffer,0,chars_read);
to_user.flush();
}
The actual whole server is below ...
public static void main(String[] args) throws IOException
{
try
{
if (args.length != 2)
throw new IllegalArgumentException("Wrong number of Args");
String host = args[0];
int port = Integer.parseInt(args[1]);
Socket s = new Socket(host,port);
final Reader from_server = new InputStreamReader(s.getInputStream());
PrintWriter to_server = new PrintWriter(new OutputStreamWriter(s.getOutputStream()));
BufferedReader from_user = new BufferedReader(new InputStreamReader(System.in));
final PrintWriter to_user = new PrintWriter(new OutputStreamWriter(System.out));
to_user.println("Connected to " + s.getInetAddress() + ":" + s.getPort());
Thread t = new Thread()
{
public void run()
{
char [] buffer = new char[1024];
int chars_read;
try
{
while((chars_read = from_server.read(buffer)) != -1)
{
to_user.write(buffer,0,chars_read);
to_user.flush();
}
}
catch(IOException e)
{
to_user.println(e);
}
to_user.println("Connection closed by server");
to_user.flush();
System.exit(0);
}
};
t.setPriority(Thread.currentThread().getPriority() + 1);
t.start();
String line;
while ((line = from_user.readLine()) != null)
{
to_server.println(line);
to_server.flush();
}
//t.stop();
s.close();
to_user.println("Connection closed by client");
to_user.flush();
}
catch(Throwable e)
{
e.printStackTrace();
System.err.println("Usage : java TCPClient <hostname> <port>");
}
}
Related
I've been trying to do communication from another language to Java, but when I try read data from DataInputStream in a while loop...
static String getData(DataInputStream stream){
int charbyte;
StringBuilder strbuilder = new StringBuilder();
try {
while ((charbyte = stream.read()) != -1){
strbuilder.append(Character.toChars(charbyte));
}
stream.close();
return new String(strbuilder);
} catch (Exception e) {
return null;
}
}
The problem is stream.read() is not returning -1 because it just keeps waiting for new data to be sent. How can I just get the data that was just sent?
The method never returns because the while loop never ends, and this is caused by the connection or the DataInputStream remaining open.
To send a variable number of bytes over a network connection where the reader reads a stream of characters you have three options:
Send the number of bytes to follow, as an int in network order, followed by as many bytes.
If the bytes are printable characters, send a null byte to indicate the end.
Close the stream after sending the bytes.
For #1, change the loop to
try {
int count = stream.readInt();
for( int i = 0; i < count; ++i ){
strbuilder.append(Character.toChars(stream.read()));
}
return strbuilder.toString();
}
For #2, use
try {
while ((charbyte = stream.read()) != 0){
strbuilder.append(Character.toChars(charbyte));
}
return strbuilder.toString();
}
The code you have now is for #3.
I have a client/server application that sends/receives data using BufferedOutputStream / BufferedInputStream . The protocol of communication is the following:
Send part :
first byte is the action to perform
next 4 bytes are the length of the message
next x bytes (x=length of message) are the message itself
Receive part :
read first byte to get the action
read the next 4 bytes to get the message length
read the x (obtained on prev step) bytes to get the message
Now the problem is that sometimes when i sent the length of the message (ex : 23045) on server part when i receive it i get a huge int (ex: 123106847).
A important clue is that this happens only when message exceeds a number of characters (in my case > 10K ) , if i sent a smaller message (ex 4-5k) everything works as expected.
Client send part (outputStream/inputStream are the type BufferedXXXStream):
private String getResponseFromServer( NormalizerActionEnum action, String message) throws IOException{
writeByte( action.id());
writeString( message);
flush(;
return read();
}
private String read() throws IOException{
byte[] msgLen = new byte[4];
inputStream.read(msgLen);
int len = ByteBuffer.wrap(msgLen).getInt();
byte[] bytes = new byte[len];
inputStream.read(bytes);
return new String(bytes);
}
private void writeByte( byte msg) throws IOException{
outputStream.write(msg);
}
private void writeString( String msg) throws IOException{
byte[] msgLen = ByteBuffer.allocate(4).putInt(msg.length()).array();
outputStream.write(msgLen);
outputStream.write(msg.getBytes());
}
private void flush() throws IOException{
outputStream.flush();
}
Server part (_input/_output are the type BufferedXXXStream)
private byte readByte() throws IOException, InterruptedException {
int b = _input.read();
while(b==-1){
Thread.sleep(1);
b = _input.read();
}
return (byte) b;
}
private String readString() throws IOException, InterruptedException {
byte[] msgLen = new byte[4];
int s = _input.read(msgLen);
while(s==-1){
Thread.sleep(1);
s = _input.read(msgLen);
}
int len = ByteBuffer.wrap(msgLen).getInt();
byte[] bytes = new byte[len];
s = _input.read(bytes);
while(s==-1){
Thread.sleep(1);
s = _input.read(bytes);
}
return new String(bytes);
}
private void writeString(String message) throws IOException {
byte[] msgLen = ByteBuffer.allocate(4).putInt(message.length()).array();
_output.write(msgLen);
_output.write(message.getBytes());
_output.flush();
}
....
byte cmd = readByte();
String message = readString();
Any help will be greatly appreciated. If you need additional details let me know.
UPDATE: Due to comments from Jon Skeet and EJP i realized that the read part on the server was having some pointless operations but letting this aside i finally got what the problem was: the key thing is that i keep the streams opened for the full length of the app and the first several times i sent the message length i'm able to read it on the server side BUT as Jon Skeet pointed out the data doesn't arrive all at once so when i try to read the message length again i'm actually reading from the message itself that is why i have bogus message lengths .
~ instead of sending the data length and then reading it all at once i sent it without the length and i read one byte at a time till the end of the string which works perfectly
private String readString() throws IOException, InterruptedException {
StringBuilder sb = new StringBuilder();
byte[] bytes = new byte[100];
int s = 0;
int index=0;
while(true){
s = _input.read();
if(s == 10){
break;
}
bytes[index++] = (byte) (s);
if(index == bytes.length){
sb.append(new String(bytes));
bytes = new byte[100];
index=0;
}
}
if(index > 0){
sb.append(new String(Arrays.copyOfRange(bytes, 0, index)));
}
return sb.toString();
}
Look at this:
byte[] bytes = new byte[len];
s = _input.read(bytes);
while(s==-1){
Thread.sleep(1);
s = _input.read(bytes);
}
return new String(bytes);
Firstly, the loop is pointless: the only time read will return -1 is if it's closed, in which case looping isn't going to help you.
Secondly, you're ignoring the possibility that the data will come in more than one chunk. You're assuming that if you've managed to get any data, you've got all the data. Instead, you should loop something like this:
int bytesRead = 0;
while (bytesRead < bytes.length) {
int chunk = _input.read(bytes, bytesRead, bytes.length - bytesRead);
if (chunk == -1) {
throw new IOException("Didn't get as much data as we should have");
}
bytesRead += chunk;
}
Note that all your other InputStream.read calls also assume that you've managed to read data, and indeed that you've read all the data you need.
Oh, and you're using the platform-default encoding to convert between binary data and text data - not a good idea.
Is there any reason you're not using DataInputStream and DataOutputStream for this? Currently you're reinventing the wheel, and doing so with bugs.
You sending code is bugged:
byte[] msgLen = ByteBuffer.allocate(4).putInt(message.length()).array();
_output.write(msgLen);
_output.write(message.getBytes());
You send the number of characters as the message length, but after that you convert the message to bytes. Depending on the platform encoding String.getBytes() can give you much more bytes than there are characters.
You should never assume that String.length() has any relationship with String.getBytes().length! Those are different concepts and should never be mixed.
We have some Java code that processes a user-provided file by looping through the file using BufferedReader.readline() to read in each line.
The problem is that when the user uploads a file that has extremely long lines, like an arbitrary binary JPG or such, this can cause out-of-memory issues. Even the first readline() may not return. We want to reject the files with long lines before it OOMs.
Is there a standard Java idiom to handle this, or do we just change to read() and write our own safe version of readLine()?
You will need to read the file character by character (or chunk by chunk) yourself (via some form of read()), and then form the lines into Strings when you encounter a newline character. This way you can throw an Exception (avoiding the OOM error) if some maximum number of characters is hit before a newline is encountered.
If you use a Reader instance it should not be too difficult to implement this code, just read from the Reader into a buffer (which you allocate to your maximum possible line length), and then convert the buffer to String when you encounter a newline (or throw an exception if you don't).
There doesn't appear to be any way to set a line length limit for BufferedReader.readLine(), so it will accumulate the entire line before feeding it to your code, however long that line may be.
Therefore, you'll have to do the line-splitting part yourself, and give up once a line is too long.
You might use the following as a starting point:
class LineTooLongException extends Exception {}
class ShortLineReader implements AutoCloseable {
final Reader reader;
final char[] buf = new char[8192];
int nextIndex = 0;
int maxIndex = 0;
boolean eof;
public ShortLineReader(Reader reader) {
this.reader = reader;
}
public String readLine() throws IOException, LineTooLongException {
if (eof) {
return null;
}
for (;;) {
for (int i = nextIndex; i < maxIndex; i++) {
if (buf[i] == '\n') {
String result = new String(buf, nextIndex, i - nextIndex);
nextIndex = i + 1;
return result;
}
}
if (maxIndex - nextIndex > 6000) {
throw new LineTooLongException();
}
System.arraycopy(buf, nextIndex, buf, 0, maxIndex - nextIndex);
maxIndex -= nextIndex;
nextIndex = 0;
int c = reader.read(buf, maxIndex, buf.length - maxIndex);
if (c == -1) {
eof = true;
return new String(buf, nextIndex, maxIndex - nextIndex);
} else {
maxIndex += c;
}
}
}
#Override
public void close() throws Exception {
reader.close();
}
}
public class Test {
public static void main(String[] args) throws Exception {
File file = new File("D:\\t\\output.log");
// try (OutputStream fos = new BufferedOutputStream(new FileOutputStream(file))) {
// for (int i = 0; i < 10000000; i++) {
// fos.write(65);
// }
// }
try (ShortLineReader r = new ShortLineReader(new FileReader(file))) {
String s;
while ((s = r.readLine()) != null) {
System.out.println(s);
}
}
}
}
Note: This assumes unix-style line termination.
Use BufferedInputStream to read binary data rather than BufferedReader...
for example if it is an image file, using ImageIO and InputStream you can do it like this..
File file = new File("image.gif");
image = ImageIO.read(file);
InputStream is = new BufferedInputStream(new FileInputStream("image.gif"));
image = ImageIO.read(is);
hope it helps...
There doesn't appear to be a definite way but a few things you can do:
Check file headers. jMimeMagic seems to be a pretty good library for this purpose.
Check the type of characters the file contains. Essentially do statistical analysis on the first 'x' bytes of the file and use that to estimate the rest of the content.
Check for newlines '\n' or '\r' in the files, binary files usually wont contain newlines.
Hope that helps.
I use the following code (from Bluetooth Chat sample app) to read the incoming data and construct a string out of the bytes read. I want to read until this string has arrived <!MSG>. How to insert this condition with read() function?
The whole string looks like this <MSG><N>xxx<!N><V>yyy<!V><!MSG>. But the read() function does not read entire string at once. When I display the characters, I cannot see all the characters in the same line. It looks like:
Sender: <MS
Sender: G><N>xx
Sender: x<V
.
.
.
I display the characters on my phone (HTC Desire) and I send the data using windows hyperterminal.
How to make sure all the characters are displayed in a single line? I have tried using StringBuilder and StringBuffer instead of new String() but the problem is read() function does not read all the characters sent. The length of the input stream (bytes) is not equal to actual length of the string sent. The construction of string from the read bytes is happening alright.
Thank you for any suggestions and time spent on this. Also please feel free to suggest other mistakes or better way of doing below things, if any.
Cheers,
Madhu
public void run() {
Log.i(TAG, "BEGIN mConnectedThread");
//Writer writer = new StringWriter();
byte[] buffer = new byte[1024];
int bytes;
//String end = "<!MSG>";
//byte compare = new Byte(Byte.parseByte(end));
// Keep listening to the InputStream while connected
while (true) {
try {
//boolean result = buffer.equals(compare);
//while(true) {
// Read from the InputStream
bytes = mmInStream.read(buffer);
//Reader reader = new BufferedReader(new InputStreamReader(mmInStream, "UTF-8"));
//int n;
//while ((bytes = reader.read(buffer)) != -1) {
//writer.write(buffer, 0, bytes);
//StringBuffer sb = new StringBuffer();
//sb = sb.append(buffer);
//String readMsg = writer.toString();
String readMsg = new String(buffer, 0, bytes);
//if (readMsg.endsWith(end))
// Send the obtained bytes to the UI Activity
mHandler.obtainMessage(BluetoothChat.MESSAGE_READ, bytes, -1, readMsg)
.sendToTarget();
//}
} catch (IOException e) {
Log.e(TAG, "disconnected", e);
connectionLost();
break;
}
}
}
The read function does not make any guarantee about the number of bytes it returns (it generally tries to return as many bytes from the stream as it can, without blocking). Therefore, you have to buffer the results, and keep them aside until you have your full message. Notice that you could receive something after the "<!MSG>" message, so you have to take care not to throw it away.
You can try something along these lines:
byte[] buffer = new byte[1024];
int bytes;
String end = "<!MSG>";
StringBuilder curMsg = new StringBuilder();
while (-1 != (bytes = mmInStream.read(buffer))) {
curMsg.append(new String(buffer, 0, bytes, Charset.forName("UTF-8")));
int endIdx = curMsg.indexOf(end);
if (endIdx != -1) {
String fullMessage = curMsg.substring(0, endIdx + end.length());
curMsg.delete(0, endIdx + end.length());
// Now send fullMessage
}
}
I have a log file which gets updated every second. I need to read the log file periodically, and once I do a read, I need to store the file pointer position at the end of the last line I read and in the next periodic read I should start from that point.
Currently, I am using a random access file in Java and using the getFilePointer() method to get he offset value and the seek() method to go to the offset position.
However, I have read in most articles and even the Java doc recommendations to use BufferredReader for efficient reading of a file. How can I achieve this (getting the filepointer and moving to the last line) using a BufferedReader, or is there any other efficient way to achieve this task?
A couple of ways that should work:
open the file using a FileInputStream, skip() the relevant number of bytes, then wrap the BufferedReader around the stream (via an InputStreamReader);
open the file (with either FileInputStream or RandomAccessFile), call getChannel() on the stream/RandomAccessFile to get an underlying FileChannel, call position() on the channel, then call Channels.newInputStream() to get an input stream from the channel, which you can pass to InputStreamReader -> BufferedReader.
I haven't honestly profiled these to see which is better performance-wise, but you should see which works better in your situation.
The problem with RandomAccessFile is essentially that its readLine() method is very inefficient. If it's convenient for you to read from the RAF and do your own buffering to split the lines, then there's nothing wrong with RAF per se-- just that its readLine() is poorly implemented
Neil Coffey's solution is good if you are reading fixed length files. However for files that have variable length (data keep coming in) there are some problems with using BufferedReader directly on FileInputStream or FileChannel inputstream via an InputStreamReader. For ex consider the cases
1)
You want to read data from some offset to current file length. So you use BR on FileInputStream/FileChannel(via an InputStreamReader) and use its readLine method. But while you are busy reading the data let say some data got added which causes BF's readLine to read more data than what you expected(the previous file length)
2)
You finished readLine stuff but when you try to read the current file length/channel position some data got added suddenly which causes the current file length/channel position to increase but you have already read less data than this.
In both of the above cases it is difficult to know the actual data you have read (you cannot just use the length of data read using readLine because it skips some chars like carriage return)
So it is better to read the data in buffered bytes and use a BufferedReader wrapper around this. I wrote some methods like this
/** Read data from offset to length bytes in RandomAccessFile using BufferedReader
* #param offset
* #param length
* #param accessFile
* #throws IOException
*/
public static void readBufferedLines(long offset, long length, RandomAccessFile accessFile) throws IOException{
if(accessFile == null) return;
int bufferSize = BYTE_BUFFER_SIZE;// constant say 4096
if(offset < length && offset >= 0){
int index = 1;
long curPosition = offset;
/*
* iterate (length-from)/BYTE_BUFFER_SIZE times to read into buffer no matter where new line occurs
*/
while((curPosition + (index * BYTE_BUFFER_SIZE)) < length){
accessFile.seek(offset); // seek to last parsed data rather than last data read in to buffer
byte[] buf = new byte[bufferSize];
int read = accessFile.read(buf, 0, bufferSize);
index++;// Increment whether or not read successful
if(read > 0){
int lastnewLine = getLastLine(read,buf);
if(lastnewLine <= 0){ // no new line found in the buffer reset buffer size and continue
bufferSize = bufferSize+read;
continue;
}
else{
bufferSize = BYTE_BUFFER_SIZE;
}
readLine(buf, 0, lastnewLine); // read the lines from buffer and parse the line
offset = offset+lastnewLine; // update the last data read
}
}
// Read last chunk. The last chunk size in worst case is the total file when no newline occurs
if(offset < length){
accessFile.seek(offset);
byte[] buf = new byte[(int) (length-offset)];
int read = accessFile.read(buf, 0, buf.length);
if(read > 0){
readLine(buf, 0, read);
offset = offset+read; // update the last data read
}
}
}
}
private static void readLine(byte[] buf, int from , int lastnewLine) throws IOException{
String readLine = "";
BufferedReader reader = new BufferedReader(new InputStreamReader(new ByteArrayInputStream(buf,from,lastnewLine) ));
while( (readLine = reader.readLine()) != null){
//do something with readLine
System.out.println(readLine);
}
reader.close();
}
private static int getLastLine(int read, byte[] buf) {
if(buf == null ) return -1;
if(read > buf.length) read = buf.length;
while( read > 0 && !(buf[read-1] == '\n' || buf[read-1] == '\r')) read--;
return read;
}
public static void main(String[] args) throws IOException {
RandomAccessFile accessFile = new RandomAccessFile("C:/sri/test.log", "r");
readBufferedLines(0, accessFile.length(), accessFile);
accessFile.close();
}
I had a similar problem, and I created this class to take lines from BufferedStream, and count how many bytes you have read so far by using getBytes(). We assume the line separator has a single byte by default, and we re-instance the BufferedReader for seek() to work.
public class FileCounterIterator {
public Long position() {
return _position;
}
public Long fileSize() {
return _fileSize;
}
public FileCounterIterator newlineLength(Long newNewlineLength) {
this._newlineLength = newNewlineLength;
return this;
}
private Long _fileSize = 0L;
private Long _position = 0L;
private Long _newlineLength = 1L;
private RandomAccessFile fp;
private BufferedReader itr;
public FileCounterIterator(String filename) throws IOException {
fp = new RandomAccessFile(filename, "r");
_fileSize = fp.length();
this.seek(0L);
}
public FileCounterIterator seek(Long newPosition) throws IOException {
this.fp.seek(newPosition);
this._position = newPosition;
itr = new BufferedReader(new InputStreamReader(new FileInputStream(fp.getFD())));
return this;
}
public Boolean hasNext() throws IOException {
return this._position < this._fileSize;
}
public String readLine() throws IOException {
String nextLine = itr.readLine();
this._position += nextLine.getBytes().length + _newlineLength;
return nextLine;
}
}