We Keep Coding

goto file line number in Java

I want to know how to directly reach a particular line no of a text file in java.
one Method is this.
int line=0;
BufferedReader read=new BufferedReader(new FileReader(Filename));
while(read.readLine()!=null){
line++;
if(line==LIMIT) break;
}
But this will create a lot of String objects which wont be freed unless gc runs.
Please provide a solution that will be fast and doesn't consume a lot of memory.
PS:I am reading from a file that has millions of lines.

Lets assume that the text file has variable length lines, and that you haven't preprocessed it to create an index. (Otherwise, it should be possible to predetermine the position of the Nth line, and then "seek" to it.)
First observation is that (with the above assumptions), it is not possible to find the Nth line without examining every character before the start of the Nth line.
But you can still do this in a way that doesn't generate lots of garbage. Here's a simple version:
BufferedReader br = new BufferedReader(new FileReader(filename));
for (int i = 1; i < LIMIT; i++) {
while ((ch = br.read()) != '\n') {
if (ch == -1) {
// reached the end of file too soon ...
throw new IOException("The file has < " + LIMIT + " lines");
}
}
}
line = br.readLine();
The trick is to skip over the lines without forming them into String objects.
Now there is a small flaw in the above. It is assuming that the lines of the text file are terminated by a newline character ('\n'), whereas the readLine can cope with 3 kinds of line separator. But that could be addressed ... without generating extra garbage. I'll leave it as "an exercise for the reader", along with investigating tweaks like using read(char[]) instead of read().
You could probably get better performance if you opened the file using a FileInputStream, obtained the FileChannel, read the bytes into a ByteBuffer and then searched it for (byte) '\n'. But the code is significantly more complicated.
However, I'd like to reinforce a point made in the comments. You are probably wasting your time with this. The chances are that your original version runs fast enough for your purposes, despite generating lots of garbage. In reality, GC is fast when the ratio of garbage to non-garbage is high. And for a program that reads an discards lines, you are pretty much guaranteed that will be the case.
Rather than spending time figuring out how to make your program fast based on a false premise, you would be better of writing a simple version and measuring its performance on typical input files. Only optimize if the program is actually too slow.

Instead of reading strings, you can read data in blocks (may be 1024 bytes block) and search line characters. To read block of data, you can use byte array, so it will be reused and so no memory issues. You have to take care of:
Handling of both \r and \n characters
Encoding of the file (like Unicode or other)
Reading data in blocks instead of byte by byte will be more efficient.

I think this should help :
FileReader fr = new FileReader("file1.txt");
BufferedReader br = new BufferedReader(fr);
LineIterator it = IOUtils.lineIterator(br);
for (int l = 0; it.hasNext(); l++) {
String line = (String) it.next();
if (l == LIMIT) {
return line;
}
}

InputStream's available() tends to halt

This is a bit of an obscure problem that only seems to happen when I'm on certain computers.
I was having this issue today on our school's XP computers and I can't seem to replicate this on my home computer (W7).
Anyway, reading/writing to sockets in Java tends to be problematic whenever I use this code (where: int avail, InputStream socket, byte[] buffer, String output):
while( (avail = input.available()) > 0 )
{
read = input.read( buffer );
output += new String( buffer, 0, read );
}
It seems to make sense (reading all the data until no data is available to a temporary buffer, then to a string), but on our school computers (testing it using IE7), the whole thing somehow pauses. I'm thinking input.available() is causing it to somehow block because the thread just keeps running without ever reaching an endpoint... effectively just pausing somewhere.
OH, I forgot to mention: whenever I run this in debug mode and perform each line step-by-step, it works completely like it should... which just confuses me even more.
When I got home to replicate this issue, it works just fine (just using Firefox and IE8). I have no idea what would be a better alternative to this.
PS:
If the buffer is large enough and I just use:
read = input.read( buffer );
output += new String( buffer, 0, read );
It works just fine, but there's always a worry that the data sent will exceed the buffer size.

You're thinking about available() the wrong way. That method tells you approximately how many bytes can be read right now, without blocking. The commonly accepted idiom for what you're trying to do is
int length;
while ((length = in.read(buffer)) != -1) {
output += new String(buffer, 0, length);
}
or something along those lines (not compiled/tested).
Update: I think you misunderstand the concept of "end of the stream". "End of the stream" doesn't mean that all the data you want to read has been read. It means that there isn't, and won't ever be, anything else to read. For instance, it might mean that you were reading a file and have come to the end of it, or it might mean you were reading from an in-memory byte array and came to the end of that. Those are "end of streams".
In your question, you indicated, or at least implied, that you're reading from a Socket. Are you aware that you'll never get to the end of that stream until the associated Socket or the remote end of the connection is closed? Just because you received a bit of data from it doesn't make it the end of the stream.

Why not use a buffered reader? Something like:
BufferedReader reader = new BufferedReader(new InputStreamReader(input));
String output = "";
try {
String readLine = null;
while ((readLine = reader.readLine()) != null) {
output += readLine + "\n";
}
} catch (IOException e) {
System.err.println("Error: " + e);
}
System.out.println("Read from Socket:" + output);

Your code is invalid. This is a misuse of available(). All it does is tell you how many bytes may be available for reading without blocking. It cannot be used to indicate how many bytes will ever be sent by the peer, and it has no necessary relationship with peer messages. There are no messages in TCP, only a byte stream. If you want to read to EOS, just remove the available() test and read until it returns -1. If you want to read a message, the peer will have to delimit it for you somehow, e.g. by an out-of-band terminator, a length word prefix, or a self-describing protocol such as Object Serialization or XML.
It 'works' in debug mode because you are radically changing the timing with breakpoints. This is further proof that what you are doing is incorrect.

Why does Java read random amounts from a socket but not the whole message?

I am working on a project and have a question about Java sockets. The source file which can be found here.
After successfully transmitting the file size in plain text I need to transfer binary data. (DVD .Vob files)
I have a loop such as
// Read this files size
long fileSize = Integer.parseInt(in.readLine());
// Read the block size they are going to use
int blockSize = Integer.parseInt(in.readLine());
byte[] buffer = new byte[blockSize];
// Bytes "red"
long bytesRead = 0;
int read = 0;
while(bytesRead < fileSize){
System.out.println("received " + bytesRead + " bytes" + " of " + fileSize + " bytes in file " + fileName);
read = socket.getInputStream().read(buffer);
if(read < 0){
// Should never get here since we know how many bytes there are
System.out.println("DANGER WILL ROBINSON");
break;
}
binWriter.write(buffer,0,read);
bytesRead += read;
}
I read a random number of bytes close to 99%. I am using Socket, which is TCP based,
so I shouldn't have to worry about lower layer transmission errors.
The received number changes but is always very near the end
received 7258144 bytes of 7266304 bytes in file GLADIATOR/VIDEO_TS/VTS_07_1.VOB
The app then hangs there in a blocking read. I am confounded. The server is sending the correct
file size and has a successful implementation in Ruby but I can't get the Java version to work.
Why would I read less bytes than are sent over a TCP socket?
The above is because of a bug many of you pointed out below.
BufferedReader ate 8Kb of my socket's input. The correct implementation can be found
Here

If your in is a BufferedReader then you've run into the common problem with buffering more than needed. The default buffer size of BufferedReader is 8192 characters which is approximately the difference between what you expected and what you got. So the data you are missing is inside BufferedReader's internal buffer, converted to characters (I wonder why it didn't break with some kind of conversion error).
The only workaround is to read the first lines byte-by-byte without using any buffered classes readers. Java doesn't provide an unbuffered InputStreamReader with readLine() capability as far as I know (with the exception of the deprecated DataInputStream.readLine(), as indicated in the comments below), so you have to do it yourself. I would do it by reading single bytes, putting them into a ByteArrayOutputStream until I encounter an EOL, then converting the resulting byte array into a String using the String constructor with the appropriate encoding.
Note that while you can't use a BufferedInputReader, nothing stops you from using a BufferedInputStream from the very beginning, which will make byte-by-byte reads more efficient.
Update
In fact, I am doing something like this right now, only a bit more complicated. It is an application protocol that involves exchanging some data structures that are nicely represented in XML, but they sometimes have binary data attached to them. We implemented this by having two attributes in the root XML: fragmentLength and isLastFragment. The first one indicates how much bytes of binary data follow the XML part and isLastFragment is a boolean attribute indicating the last fragment so the reading side knows that there will be no more binary data. XML is null-terminated so we don't have to deal with readLine(). The code for reading looks like this:
InputStream ins = new BufferedInputStream(socket.getInputStream());
while (!finished) {
ByteArrayOutputStream buf = new ByteArrayOutputStream();
int b;
while ((b = ins.read()) > 0) {
buf.write(b);
}
if (b == -1)
throw new EOFException("EOF while reading from socket");
// b == 0
Document xml = readXML(new ByteArrayInputStream(buf.toByteArray()));
processAnswers(xml);
Element root = xml.getDocumentElement();
if (root.hasAttribute("fragmentLength")) {
int length = DatatypeConverter.parseInt(
root.getAttribute("fragmentLength"));
boolean last = DatatypeConverter.parseBoolean(
root.getAttribute("isLastFragment"));
int read = 0;
while (read < length) {
// split incoming fragment into 4Kb blocks so we don't run
// out of memory if the client sent a really large fragment
int l = Math.min(length - read, 4096);
byte[] fragment = new byte[l];
int pos = 0;
while (pos < l) {
int c = ins.read(fragment, pos, l - pos);
if (c == -1)
throw new EOFException(
"Preliminary EOF while reading fragment");
pos += c;
read += c;
}
// process fragment
}
Using null-terminated XML for this turned out to be a really great thing as we can add additional attributes and elements without changing the transport protocol. At the transport level we also don't have to worry about handling UTF-8 because XML parser will do it for us. In your case you're probably fine with those two lines, but if you need to add more metadata later you may wish to consider null-terminated XML too.

Here is your problem. The first few lines of the program your using in.readLine() which is probably some sort of BufferedReader. BufferedReaders will read data off the socket in 8K chunks. So when you did the first readLine() it read the first 8K into the buffer. The first 8K contains your two numbers followed by newlines, then some portion of the head of the VOB file (that's the missing chunk). Now when you switched to using the getInputStream() off the socket you are 8K into the transmission assuming your starting at zero.
socket.getInputStream().read(buffer); // you can't do this without losing data.
While the BufferedReader is nice for reading character data, switching between binary and character data in a stream is not possible with it. You'll have to switch to using InputStream instead of Reader and convert the first few portions by hand to character data. If you read the file using a buffered byte array you can read the first chunk, look for your newlines and convert everything to the left of that to character data. Then write everything to the right to your file, then start reading the rest of the file.
This used to be easier with DataInputStream, but it doesn't do a good job handling character conversion for you (readLine is deprecated with BufferedReader being the only replacement - doh). Probably should write a DataInputStream replacement that under the covers uses Charset to properly handle string conversion. Then switching between characters and binary would be easier.

Your basic problem is that BufferedReader will read as much data is available and place in its buffer. It will give you the data as you ask for it. This is the whole point of buffereing i.e. to reduce the number of calls to the OS. The only safe way to use an buffered input is to use the same buffer over the life of the connection.
In your case, you only use the buffer to read two lines, however it is highly likely that 8192 bytes has been read into the buffer. (The default size of the buffer) Say the first two lines consist of 32 bytes, this leaves 8160 waiting for you to read, however you by-pass the buffer to perform the read() on the socket directly leading to 8160 bytes left in the buffer you end up discarding. (the amount you are missing)
BTW: You should be able to see this in a debugger if you inspect the contents of your buffered reader.

Sergei may have been right about data being lost inside the buffer, but I'm not sure about his explanation. (BufferedReaders don't usually hold onto data inside their buffers. He may be thinking of a problem with BufferedWriters, which can lose data if the underlying stream is shut down prematurely.) [Never mind; I had misread Sergei's answer. The rest of this is valid AFAIK.]
I think you have a problem that's specific to your application. In your client code, you start reading as follows:
public static void recv(Socket socket){
try {
BufferedReader in = new BufferedReader(new InputStreamReader(socket.getInputStream()));
//...
int numFiles = Integer.parseInt(in.readLine());
... and you proceed to use in for the start of the exchange. But then you switch to using the raw socket stream:
while(bytesRead > fileSize){
read = socket.getInputStream().read(buffer);
Because in is a BufferedReader, it's already going to have filled its buffer with up to 8192 bytes from the socket input stream. Any bytes that are in that buffer, and which you don't read from in, will be lost. Your app is hanging because it believes that the server is holding onto some bytes, but the server doesn't have them.
The solution is not to do byte-by-byte reads from the socket (ouch! your poor CPU!), but to use the BufferedReader consistently. Or, to use buffering with binary data, change the BufferedReader to a BufferedInputStream that wraps the socket's InputStream.
By the way, TCP is not as reliable as many people assume it to be. For example, when the server socket closes, it's possible for it to have written data into the socket which then gets lost as the socket connection is shutdown. Calling Socket.setSoLinger can help to prevent this problem.
EDIT: Also BTW, you're playing with fire by treating byte and character data as if they're interchangeable, as you do below. If the data really is binary, then the conversion to String risks corrupting the data. Perhaps you want to be writing into a BufferedOutputStream?
// Java is retarded and reading and writing operate with
// fundamentally different types. So we write a String of
// binary data.
fileWriter.write(new String(buffer));
bytesRead += read;
EDIT 2: Clarified (or attempted to clarify :-} the handling of binary vs. String data.

Trying to packetize TCP with non-blocking IO is hard! Am I doing something wrong?

Oh how I wish TCP was packet-based like UDP is! [see comments] But alas, that's not the case, so I'm trying to implement my own packet layer. Here's the chain of events so far (ignoring writing packets)
Oh, and my Packets are very simply structured: two unsigned bytes for length, and then byte[length] data. (I can't imagine if they were any more complex, I'd be up to my ears in if statements!)
Server is in an infinite loop, accepting connections and adding them to a list of Connections.
PacketGatherer (another thread) uses a Selector to figure out which Connection.SocketChannels are ready for reading.
It loops over the results and tells each Connection to read().
Each Connection has a partial IncomingPacket and a list of Packets which have been fully read and are waiting to be processed.
On read():
Tell the partial IncomingPacket to read more data. (IncomingPacket.readData below)
If it's done reading (IncomingPacket.complete()), make a Packet from it and stick the Packet into the list waiting to be processed and then replace it with a new IncomingPacket.
There are a couple problems with this. First, only one packet is being read at a time. If the IncomingPacket needs only one more byte, then only one byte is read this pass. This can of course be fixed with a loop but it starts to get sorta complicated and I wonder if there is a better overall way.
Second, the logic in IncomingPacket is a little bit crazy, to be able to read the two bytes for the length and then read the actual data. Here is the code, boiled down for quick & easy reading:
int readBytes; // number of total bytes read so far
byte length1, length2; // each byte in an unsigned short int (see getLength())
public int getLength() { // will be inaccurate if readBytes < 2
return (int)(length1 << 8 | length2);
}
public void readData(SocketChannel c) {
if (readBytes < 2) { // we don't yet know the length of the actual data
ByteBuffer lengthBuffer = ByteBuffer.allocate(2 - readBytes);
numBytesRead = c.read(lengthBuffer);
if(readBytes == 0) {
if(numBytesRead >= 1)
length1 = lengthBuffer.get();
if(numBytesRead == 2)
length2 = lengthBuffer.get();
} else if(readBytes == 1) {
if(numBytesRead == 1)
length2 = lengthBuffer.get();
}
readBytes += numBytesRead;
}
if(readBytes >= 2) { // then we know we have the entire length variable
// lazily-instantiate data buffers based on getLength()
// read into data buffers, increment readBytes
// (does not read more than the amount of this packet, so it does not
// need to handle overflow into the next packet's data)
}
}
public boolean complete() {
return (readBytes > 2 && readBytes == getLength()+2);
}
Basically I need feedback on my code and overall process. Please suggest any improvements. Even overhauling my entire system would be okay, if you have suggestions for how better to implement the whole thing. Book recommendations are welcome too; I love books. I just get the feeling that something isn't quite right.
Here's the general solution I came up with thanks to Juliano's answer: (feel free to comment if you have any questions)
public void fillWriteBuffer() {
while(!writePackets.isEmpty() && writeBuf.remaining() >= writePackets.peek().size()) {
Packet p = writePackets.poll();
assert p != null;
p.writeTo(writeBuf);
}
}
public void fillReadPackets() {
do {
if(readBuf.position() < 1+2) {
// haven't yet received the length
break;
}
short packetLength = readBuf.getShort(1);
if(readBuf.limit() >= 1+2 + packetLength) {
// we have a complete packet!
readBuf.flip();
byte packetType = readBuf.get();
packetLength = readBuf.getShort();
byte[] packetData = new byte[packetLength];
readBuf.get(packetData);
Packet p = new Packet(packetType, packetData);
readPackets.add(p);
readBuf.compact();
} else {
// not a complete packet
break;
}
} while(true);
}

Probably this is not the answer you are looking for, but someone would have to say it: You are probably overengineering the solution for a very simple problem.
You do not have packets before they arrive completely, not even IncomingPackets. You have just a stream of bytes without defined meaning. The usual, the simple solution is to keep the incoming data in a buffer (it can be a simple byte[] array, but a proper elastic and circular buffer is recommended if performance is an issue). After each read, you check the contents of the buffer to see if you can extract an entire packet from there. If you can, you construct your Packet, discard the correct number of bytes from the beginning of the buffer and repeat. If or when you cannot extract an entire packet, you keep those incoming bytes there until the next time you read something from the socket successfully.
While you are at it, if you are doing datagram-based communication over a stream channel, I would recommend you to include a magic number at the beginning of each "packet" so that you can test that both ends of the connection are still synchronized. They may get out of sync if for some reason (a bug) one of them reads or writes the wrong number of bytes to/from the stream.

Can't you just read whatever number of bytes that are ready to be read, and feed all incoming bytes into a packet parsing state machine? That would mean treating the incoming (TCP) data stream like any other incoming data stream (via serial line, or USB, a pipe, or whatever...)
So you would have some Selector determining from which connection(s) there are incoming bytes to be read, and how many. Then you would (for each connection) read the available bytes, and then feed those bytes into a (connection specific) state machine instance (the reading and feeding could be done from the same class, though). This packet parsing state machine class would then spit out finished packets from time to time, and hand those over to whoever will handle those complete and parsed packets.
For an example packet format like
2 magic header bytes to mark the start
2 bytes of payload size (n)
n bytes of payload data
2 bytes of checksum
the state machine would have states like (try an enum, Java has those now, I gather)
wait_for_magic_byte_0,
wait_for_magic_byte_1,
wait_for_length_byte_0,
wait_for_length_byte_1,
wait_for_payload_byte (with a payload_offset variable counting),
wait_for_chksum_byte_0,
wait_for_chksum_byte_1
and on each incoming byte you can switch the state accordingly. If the incoming byte does not properly advance the state machine, discard the byte by resetting the state machine to wait_for_magic_byte_0.

Ignoring client disconnects and server shutdown for now, here's more or less traditional structure of a socket server:
Selector, handles sockets:
polls open sockets
if it's the server socket, create new Connection object
for each active client socket find the Connection, call it with event (read or write)
Connection (one per socket), handles I/O on one socket:
Communicates to Protocol via two queues, input and output
keeps two buffers, one for reading, one for writing, and respective offsets
on read event: read all available input bytes, look for message boundaries, put whole messages onto Protocol input queue, call Protocol
on write event: write the buffer, or if it's empty, take message form output queue into buffer, start writing it
Protocol (one per connection), handles application protocol exchange on one connection:
take message from input queue, parse application portion of the message
do the server work (here's where the state machine is - some messages are appropriate in one state, while not in the other), generate response message, put it onto output queue
That's it. Everything could be in a single thread. The key here is separation of responsibilities.
Hope this helps.

I think you're approaching the issue from a slightly wrong direction. Instead of thinking of packets, think of a data structure. That's what you're sending. Effectively, yes, it's an application layer packet, but just think of it as a data object. Then, at the lowest level, write a routine which will read off the wire, and output data objects. That will give you the abstraction layer I think you're looking for.

How do I read strings in J2ME?

I'm using the MIDP 2.0 (JSR 118) and I just noticed that there is no reader for strings in J2ME.
Does anyone know how you are supposed to read Strings from an InputStream or InputStreamReader in a platform independent way (i.e. between two java enabled cell phones of different models)?

Which profile are you using? The MID profile in JSR 118 specifies InputStreamReader (not StringReader, but that wouldn't help you read from an InputStream anyway).
EDIT: To reflect the change to the question :)
You use InputStreamReader.read(char[], int, int) and when you've read all you want to, create a new string from a char array. If you want to read a line at a time as you would from BufferedReader, you basically need to implement the functionality of BufferedReader yourself (keeping a buffer of "read but not consumed" chars) and keep reading until you hit a line break.

Alternatively have a look at DataInputStream.readUTF().
It does required that the string being read off the InputStream be encoded appropriately (as in by a corresponding DataOutputStream.writeUTF(String)) so it might not be what you're looking for - but it does work across different phones/models etc.

Well... I know this was a LONG time ago.
You need to do exactly what John said, and it is VERY simple. It almost took me 5 hours to figure this one out the first time...
I still wonder why j2ME didn't include something as essential as the BufferedReader method for sockets, it's not like the freakin cellphones will crash with it... and yes, I don't give a rat's ass if my app runs 1ms slower than it should.
(I'm just going to put the relevant code, I assume you know how to form classes and import the required libraries)
ServerSocketConnection listener
= (ServerSocketConnection)Connector.open("socket://:1235");
System.out.println("Waiting for connection...");
StreamConnection server = listener.acceptAndOpen();
InputStream is = server.openInputStream();
//Now comes the fake BufferedReader equivalent part
int ch = 0;
StringBuffer sb = new StringBuffer();
while ((ch = is.read()) != -1){
sb.append((char)ch);
if(sb.charAt(sb.length()-1) == 13 ) {
//Carriage return was received or ENTER was pressed
break; //Exit loop and print input
}
}
As you can see, the is.read() method will lock the thread till new input is received from the user ONE BYTE AT A TIME. This means if you use telnet to test, each keystroke will make the loop iterate once, hence, we simply concatenate char by char in a StringBuffer until char 13 is received.
System.out.println(sb.toString());
I hope this helps people trying to do a socket server on j2ME. I already crafted a fully functional multithreaded version of this for blackberry, in case anyone needs it.

Would you be able to provide an example of this?
You use InputStreamReader.read(char[], int, int) and when you've read all you want to, create a new string from a char array. If you want to read a line at a time as you would from BufferedReader, you basically need to implement the functionality of BufferedReader yourself (keeping a buffer of "read but not consumed" chars) and keep reading until you hit a line break.