We Keep Coding

Filter an InputStream line-by-line

I am retrieving large gzipped files from Amazon S3. I would like to be able to transform each line of these files on-the-fly and upload the output to another S3 bucket.
The upload API takes an InputStream as input.
S3Object s3object = s3.fetch(bucket, key);
InputStream is = new GZIPInputStream(s3object.getObjectContent());
// . . . ?
s3.putObject(new PutObjectRequest(bucket, key, is, metadata));
I believe that the most efficient way of doing this is to create my own custom input stream which transforms the original input stream into another input stream. I am not very familiar with this approach and curious to find out more.

The basic idea is as follows.
It's not terribly efficient but should get the job done.
public class MyInputStream extends InputStream {
private final BufferedReader input;
private final Charset encoding = StandardCharsets.UTF_8;
private ByteArrayInputStream buffer;
public MyInputStream(InputStream is) throws IOException {
input = new BufferedReader(new InputStreamReader(is, this.encoding));
nextLine();
}
#Override
public int read() throws IOException {
if (buffer == null) {
return -1;
}
int ch = buffer.read();
if (ch == -1) {
if (!nextLine()) {
return -1;
}
return read();
}
return ch;
}
private boolean nextLine() throws IOException {
String line;
while ((line = input.readLine()) != null) {
line = filterLine(line);
if (line != null) {
line += '\n';
buffer = new ByteArrayInputStream(line.getBytes(encoding));
return true;
}
}
return false;
}
#Override
public void close() throws IOException {
input.close();
}
private String filterLine(String line) {
// Filter the line here ... return null to skip the line
// For example:
return line.replace("ABC", "XYZ");
}
}
nextLine() pre-fills the line buffer with a (filtered) line. Then read() (called by the upload job) fetches bytes from the buffer one-by-one and calls nextLine() again to load the next line.
Use as:
s3.putObject(new PutObjectRequest(bucket, key, new MyInputStream(is), metadata));
A performance improvement could be to also implement the int read(byte[] b, int off, int len) method (if cpu use is high) and use a BufferedInputStream in case the S3 client doesn't internally use a buffer (I don't know).

new BufferedReader(is).lines()

BufferedReader messed up by different line seperators

I'm having a buffered reader streaming a file. There are two cases right now:
It is streaming a file generated on one PC, let's call it File1.
It is streaming a file generated on another Computer, let's call it File2.
I'm assuming my problem is caused by the EOLs.
BufferedReader does read both files, but for the File2, it reads an extra empty line for every new line.
Also, when I compare the line using line.equalsIgnoreCase("abc"), given that the line is "abc" it does not return true.
Use this code together with the two files provided in the two links to replicate the problem:
public class JavaApplication {
/**
* #param args the command line arguments
*/
public static void main(String[] args) throws IOException {
File file = new File("C:/Users/User/Downloads/html (2).htm");
BufferedReader in = new BufferedReader(new InputStreamReader(new FileInputStream(file), "UTF-8"));
String line = "";
while ((line = in.readLine()) != null) {
System.out.println(line);
}
}
File1,
File2
Note how the second file prints an empty line after each line...
I've been searching and trying and searching and trying, and couldn't come up with a solution.
Any ideas how to fix that? (Especially the compare thing?)

Works for me.
public class CRTest
{
static StringReader test = new StringReader( "Line 1\rLine 2\rLine 3\r" );
public static void main(String[] args) throws IOException {
BufferedReader buf = new BufferedReader( test );
for( String line = null; (line = buf.readLine()) != null; )
System.out.println( line );
}
}
Prints:
run:
Line 1
Line 2
Line 3
BUILD SUCCESSFUL (total time: 1 second)
As Joop said, I think you've mixed up which file isn't working. Please use the above skeleton to create an MCVE and show us exactly what file input isn't working for you.
Since you appear to have a file with reversed \r\n lines, here's my first attempt at a fix. Please test it, I haven't tried it yet. You need to wrap your InputStreamReader with this class, then wrap the BufferedReader on the outside like normal.
class CRFix extends Reader
{
private final Reader reader;
private boolean readNL = false;
public CRFix( Reader reader ) {
this.reader = reader;
}
#Override
public int read( char[] cbuf, int off, int len )
throws IOException
{
for( int i = off; i < off+len; i++ ) {
int c = reader.read();
if( c == -1 )
if( i == off ) return -1;
else return i-off-1;
if( c == '\r' && readNL ) {
readNL = false;
c = reader.read();
}
if( c == '\n' )
readNL = true;
else
readNL = false;
cbuf[i] = (char)c;
}
return len;
}
#Override
public void close()
throws IOException
{
reader.close();
}
}

Joop was right, after some more research it seems like, even though both files have specified a UTF-16 encoding in their header, one was encoded in UTF-16, and the other (File1) in UTF-8. This lead to the "double line effect".
Thanks for the effort that was put in answering this question.

infinite while loop using bufferedreader

I am having a code to store the log files to a SD card in android. The following is the code.
process = Runtime.getRuntime().exec("logcat "+ LOG_TAG +":D *:S");
BufferedReader bufferedReader = new BufferedReader(new InputStreamReader(process.getInputStream()));
StringBuilder log = new StringBuilder();
String line;
while ((line = bufferedReader.readLine()) != null) {
//do something over here
}
It is running an infinite loop.
Any help.

As logCat never ends, you might try to force an end when InputStream.available() == 0.
I did this using wrapping the original InputStream in an ImpatientInputStream.
As at the very first start available might be 0 because of the non-blocking nature, you might add a flag whether something read already.
BufferedReader bufferedReader = new BufferedReader(
new InputStreamReader(
new ImpatientInputStream(process.getInputStream())
));
public class ImpatientInputStream extends InputStream {
private final InputStream in;
private boolean eof;
public ImpatientInputStream(InputStream in) {
this.in = in;
}
#Override
public int read() throws IOException {
if (eof) {
return -1;
}
if (available() == 0) {
eof = true;
return -1;
}
return in.read();
}
}

logcat never stops, it waits for the next line to display, so readLine() blocks while waiting for the next input from the process.
You can however redirect the output of the command directly with -f filename as explained here.
This answer has everything you need.

You can provide timestamp/time period in while loop. Below sample will make this more clear
Sample :
while (System.currentTimeMillis() < end_time) {
//do your stuffs here
while ((line = bufferedReader.readLine()) != null) {
//do something over here
}
}

Java - reading file as binary with readLine

I have a Ruby code that reads file line-by-line and checks if it needs to read the next line to some block or it should handle that block and continue reading file parsing each line.
Here's it:
File.open(ARGV[0], 'rb') do |f|
fl = false
text = ''
f.readlines.each do |line|
if (line =~ /^end_block/)
fl = false
# parse text variable
end
text += line if fl == true
if (line =~ /^start_block/)
fl = true
end
end
end
E.g. i need the file to be opened for reading as binary and still i need a readLine method.
So, the question is: how can i do exactly the same with Groovy/Java?

You can use java.io.DataInputStream which provides both a readLine() method and readFully(byte[]) and read(byte[]) methods.
Warning: The JavaDoc for readLine says, it is deprecated and that the encoding might be inappropriate (read details in JavaDoc).
So think twice about your real requirements and if this is a suitable trade-off in your case.

If you have line formatted text, that's not binary IMHO. That's because true binary can have any byte, even new line and carriage return which would create false breaks in the code.
What you could mean is you have text where you want to read each byte without encoding or possibly mangling them. This is the same as using ISO-8859-1.
You can try
BufferedReader br = new BufferedReader(new InputStreamReader(
new FileInputStream(filename), "ISO-8859-1"));
StringBuilder sb = new StringBuilder();
String line;
boolean include = false;
while((line = br.readLine()) != null) {
if (line.startsWith("end_block"))
include = false;
else if (line.startsWith("start_block"))
include = true;
else if (include)
sb.append(line).append('\n'); // new lines back in.
}
br.close();
String text = sb.toString();

Maybe something like this:
public final class Read
{
private static final Pattern START_BLOCK = Pattern.compile("whatever");
private static final Pattern END_BLOCK = Pattern.compile("whatever");
public static void main(final String... args)
throws IOException
{
if (args.length < 1) {
System.err.println("Not enough arguments");
System.exit(1);
}
final FileReader r = new FileReader(args[0]);
final BufferedReader reader = new BufferedReader(r);
final StringBuilder sb = new StringBuilder();
boolean inBlock = false;
String line;
while ((line = reader.readLine()) != null) {
if (END_BLOCK.matcher(line).matches()) {
inBlock = false;
continue;
}
if (inBlock)
sb.append(line);
if (START_BLOCK.matcher(line).matches())
inBlock = true;
}
System.out.println(sb.toString());
System.exit(0);
}
}

Reading in from text file character by character

In Java, is there a way of reading a file (text file) in a way that it would only read one character at a time, rather than String by String. This is for the purpose of an extremely basic lexical analyzer, so you can understand why I'd want such a method. Thank you.

Here's a sample code for reading / writing one character at a time
public class CopyCharacters {
public static void main(String[] args) throws IOException {
FileReader inputStream = null;
FileWriter outputStream = null;
try {
inputStream = new FileReader("xanadu.txt");
outputStream = new FileWriter("characteroutput.txt");
int c;
while ((c = inputStream.read()) != -1) {
outputStream.write(c);
}
} finally {
if (inputStream != null) {
inputStream.close();
}
if (outputStream != null) {
outputStream.close();
}
}
}
}
Note, this answer was updated to copy the sample code from the Ref link, but I see this is essentially the same answer given below.
ref:
http://download.oracle.com/javase/tutorial/essential/io/charstreams.html

You can use the read method from the InputStreamReader class which reads one character from the stream and returns -1 when it reaches the end of the stream
public static void processFile(File file) throws IOException {
try (InputStream in = new FileInputStream(file);
Reader reader = new InputStreamReader(in)) {
int c;
while ((c = reader.read()) != -1) {
processChar((char) c); // this method will do whatever you want
}
}
}

You can read the whole file (if it is not much big) in the memory as string, and iterate on the string character by character

There are several possible solutions. Generally you can use any Reader from java.io package for reading characters, e.g.:
// Read from file
BufferedReader reader = new BufferedReader(new FileReader("file.txt"));
// Read from sting
BufferedReader reader = new BufferedReader(new StringReader("Some text"));