I'm trying to read from an input stream of a HttpURLConnection:
InputStream input = conn.getInputStream();
InputStreamReader isr = new InputStreamReader((input));
BufferedReader br = new BufferedReader(isr);
StringBuilder out = new StringBuilder("");
String output;
while ((output = br.readLine()) != null) {
out.append(output);
}
This does take too much time when the input stream contains a lot of data. Is it possible to optimize this?
Maybe this will be a bit faster, cause the new Stream API in Java 8 ist using internaly a parallel mechanism:
package testing;
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.net.HttpURLConnection;
import java.net.URL;
import java.util.stream.Stream;
public class StreamTest {
/**
* #param args the command line arguments
* #throws java.io.IOException
*/
public static void main(String[] args) throws IOException {
URL url = new URL("http://www.google.com");
HttpURLConnection conn = (HttpURLConnection) url.openConnection();
conn.setUseCaches(false);
if (conn.getResponseCode() == HttpURLConnection.HTTP_OK) {
BufferedReader br = new BufferedReader(new InputStreamReader(conn.getInputStream()));
Stream<String> s = br.lines();
s.parallel().forEach(System.out::println);
}
}
}
There's nothing slow about this code. You can read millions of lines a second with this, if the input arrives fast enough. Your time probably isn't spent reading the input stream at all, but in either blocking waiting for input or in appending to the StringBuilder.
But you shouldn't be doing this at all. Most files can be processed a line at a time or a record at a time. Compilers process them a token at a time, and there aren't many more complex file-processing tasks than compilation. It's possible.
In java inputstream we have method read(byte b[],off,len)
which reads the from the input stream into the given byte array.
Here off is the starting index of the array, len is the maximum number of byte to be read and b[] is the byte array.
Read method will attempt to read maximum of len number of bytes but this method returns number of actual byte read as many times i will fail to read the desired number of bytes.
Here is the example:-
FileInputStream i=new FileInputStream("file path");
FIleOutputStream o=new FileOutputStream("file path");
byte a[]=new byte[1024];
for(int j;(j=i.read(a,0,1024))!=-1;){
o.write(a,0,j);
}
Related
I had written some codes to read from a text file char by char and then print it to the screen,but the result had made me feel confused,here it is:
this is the code that i had written
import java.io.*;
import java.nio.charset.StandardCharsets;
public class learnIO
{
public static void main(String[] args) throws IOException{
var in = new InputStreamReader(new FileInputStream("test1.txt"), StandardCharsets.UTF_8);
while(in.read() != -1){
System.out.println((char)in.read());
}
}
}
the content and encoding scheme of the file:
file test1.txt
test1.txt: ASCII text
cat test1.txt
hello, world!
the result is:
e
l
,
w
r
d
some char had missed，Why did this happen？
return type of read method of InputStreamReader is int that takes 4 bytes
and char type is 2 bytes so casting int to char you skip 2 bytes
refer to https://docs.oracle.com/javase/7/docs/api/java/io/InputStreamReader.html
You need to use InputStreamReader inside BufferedReader as from the official oracle documentation it says that
An InputStreamReader is a bridge from byte streams to character streams: It reads bytes and decodes them into characters using a specified charset. The charset that it uses may be specified by name or maybe given explicitly, or the platform's default charset may be accepted.
Each invocation of one of an InputStreamReader's read() methods may cause one or more bytes to be read from the underlying byte-input stream.
To enable the efficient conversion of bytes to characters, more bytes may be read ahead from the underlying stream than are necessary to satisfy the current read operation.
For top efficiency, consider wrapping an InputStreamReader within a BufferedReader. For example:
BufferedReader in
= new BufferedReader(new InputStreamReader(System.in));
So the solution to your problem can be solved using the following code
try {
// Open the file that is the first
// command line parameter
FileInputStream fstream = new FileInputStream("hello.txt");
// Get the object of DataInputStream
DataInputStream in = new DataInputStream(fstream);
BufferedReader br = new BufferedReader(new InputStreamReader(in));
//Read File Line By Line
char c;
while ((c = (char) br.read()) != (char) -1) {
// Print the content on the console
String character = Character.toString(c);
System.out.println(character);
}
//Close the input stream
in.close();
} catch (Exception e) {//Catch exception if any
System.err.println("Error: " + e.getMessage());
}
I encountered this "bug" (quoting because I'm not sure if it's just me missing something really simple and obvious) when helping my GF code some simple android GUI.
This app fetches some data over the internet and displays charts that visualize the data.
I have some code like this:
import android.os.AsyncTask;
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.io.Reader;
import java.net.HttpURLConnection;
import java.net.URL;
import javax.net.ssl.HttpsURLConnection;
public class DownloadTask extends AsyncTask<String, Void, String> {
#Override
protected String doInBackground(String... urls) {
try {
return downloadUrl(urls[0]);
} catch (IOException e) {
return "Unable to retrieve web page. URL may be invalid.";
}
}
private String downloadUrl(String urlString) throws IOException {
InputStream stream = null;
try {
URL url = new URL(urlString);
HttpURLConnection conn = (HttpURLConnection) url.openConnection();
conn.setReadTimeout(10000);
conn.setConnectTimeout(15000);
conn.setRequestMethod("GET");
conn.setDoInput(true);
conn.connect();
stream = conn.getInputStream();
return readStream(stream, conn.getContentLength());
} finally {
if (stream != null) {
stream.close();
}
}
}
public String readStream(InputStream stream, int bufferLength) throws IOException {
char[] buffer = new char[bufferLength];
Reader reader = new BufferedReader(new InputStreamReader(stream, "ascii"));
reader.read(buffer, 0, bufferLength);
return new String(buffer);
}
}
I use the above code as a base class and implement onPostExecute in private classes inside other classes.
The problem is, when I get the result in onPostExecute, the string is not complete. Note: I wrote the server, so I can guarantee that the content length is correct.
The first couple hundred characters are correct, but at some point, I start to see unknown characters. I tried using a byte array to read the raw stream and saw the unknown characters are just 0s. So the stream looks like: [... 54 52 53 ... 0 0 0 0 0 0 0 ...]. Basically, correct bytes followed by all zeros.
I wonder why the stream is chopped off, and how to make it read the whole response correctly. Note: this "bug" happens on some phones very consistently and not so consistently on others.
Some more details about the data, it's pure text, comma or space separated lines, lines separated by \n (as I said above I wrote the server. I used to use commas, changed it to spaces just to test whether it's the problem). The data size is about 2000 bytes, not large at all I would assume.
Thanks for taking a look at this!
I wonder why the stream is chopped off, and how to make it read the
whole response correctly.
When you read from an InputStream that comes from the network, not all the bytes could be all available at the same time.
public String readStream(InputStream stream, int bufferLength) throws IOException {
StringBuilder builder = new StringBuilder();
int read = 0;
byte[] buffer = new byte[8 * 1024];
while ((read = stream.read(buffer)) > 0) {
builder.append(new String(buffer, 0, read));
}
return builder.toString();
}
Updated Question (to be more clear):
Is there a way to design the InputStream below such that BufferedReader#readLine() will return after reading the new line character?
In the example below, readLine() hangs forever even though the reader has read a new line because (presumably) it is waiting for the buffer to fill up. Ideally, readLine() would return after reading the new line character.
I know something like what I want is possible, because when you read from System.in using BufferedReader#readLine(), it does not wait for the buffer to fill up before returning.
import java.io.*;
import java.util.concurrent.BlockingQueue;
import java.util.concurrent.LinkedBlockingQueue;
public class Example {
private static final class MyInputStream extends InputStream {
public final BlockingQueue<String> lines = new LinkedBlockingQueue<>();
private InputStream current = null;
#Override
public int read() throws IOException {
try {
if(current == null || current.available() == 0)
current = new ByteArrayInputStream(lines.take().getBytes("UTF-8"));
return current.read();
}
catch(InterruptedException ex) {
return -1;
}
}
}
public static void main(String[] args) throws Exception {
MyInputStream myin = new MyInputStream();
myin.lines.offer("a line\n");
BufferedReader in = new BufferedReader(new InputStreamReader(myin));
System.out.println(in.readLine());
}
}
Also, if there is a better way to send a string to an InputStream, I'm open to suggestions.
Accepted Solution:
Based on a suggestion from Sotirios Delimanolis in one of the comments on his solution, I'm just going to used a PipedInputStream instead. I've coupled it to a PipedOutputStream, and BufferedReader#readLine() returns immediately as long as I call PipedOutputStream#flush() after sending a string that contains a new line character.
After updated question, the only way to get the BufferedReader to stop reading after the new line character is to set the buffer size to 1, which completely removes the need for a BufferedReader.
You'll have to write your own implementation.
A BufferedReader reads more bytes than required. In your case, that means it will read further than the new line character. For example, with the Oracle JVM, it will attempt to read 8192 bytes. Through your inheritance hierarchy, this
System.out.println(in.readLine());
will attempt to invoke your read() method 8192 times.
The first 6 calls will return a value, one for each of the characters in your String's byte array. The next one, will see
if(current == null || current.available() == 0)
current = new ByteArrayInputStream(lines.take().getBytes("UTF-8"));
and current.available() will return 0 since the ByteArrayInputStream has been fully consumed. It will then attempt to take from the BlockingQueue and block indefinitely.
Also, if there is a better way to send a string to an InputStream, I'm open to suggestions.
Well, instead of an InputStream you can try a BufferedReader, with something that looks like this:
public int read(String directory) throws Exception{
String line = "";
File file = new File(directory);
FileReader fr = new FileReader(file);
BufferedReader br = new BufferedReader(fr);
do{
lines.add(br.readLine());
while(br.readLine() != null);
br.close();
return Integer.parseInt(line);
}
BufferedReader br = new BufferedReader(new InputStreamReader(System.in));
System.in(Standard input stream)- gets the input from keyboard in bytes
InputStreamReader: Converts the bytes into Unicode characters/ converts the standard input into reader object to be used with BufferedReader
Finally BufferedReader: Used to read from character input stream(Input stream reader)
String c = br.ReadLine(); -- a method used to read characters from input stream and put them in the string in one go not byte by byte.
Is everything above right ? Please correct if anything wrong !
Nearly there, but this:
String c = br.readLine(); -- a method used to read characters from input stream and put them in the string in one go not byte by byte.
It reads characters from the input reader (BufferedReader doesn't know about streams) and returns a whole line in one go, not character by character. Think of it in layers, and "above" the InputStreamReader layer, the concept of "bytes" doesn't exist any more.
Also, note that you can read blocks of characters with a Reader without reading a line: read(char[], int, int) - the point of readLine() is that it will do the line ending detection for you.
(As noted in comments, it's also readLine, not ReadLine :)
What is the purpose of BufferedReader, explanation?
Bufferedreader is a java class, the following is the hierarchy of this class.
java.lang.Object ==> java.io.Reader ==> java.io.BufferedReader
Also, BufferedReader provides an efficient way to read content. Very Simple..
Let's have a look at the following example to understand.
import java.io.BufferedReader;
import java.io.FileReader;
public class Main {
public static void main(String[] args) {
BufferedReader contentReader = null;
int total = 0; // variable total hold the number that we will add
//Create instance of class BufferedReader
//FileReader is built in class that takes care of the details of reading content from a file
//BufferedReader is something that adds some buffering on top of that to make reading fom a file more efficient.
try{
contentReader = new BufferedReader(new FileReader("c:\\Numbers.txt"));
String line = null;
while((line = contentReader.readLine()) != null)
total += Integer.valueOf(line);
System.out.println("Total: " + total);
}
catch(Exception e)
{
System.out.println(e.getMessage());
}
finally{
try{
if(contentReader != null)
contentReader.close();
}
catch (Exception e)
{
System.out.println(e.getMessage());
}
}
}
}
I have problem while writing to the file. I want to write contents of my input file to output file but while writing to the file, I am getting NULL value written at the end of file.
What's the reason behind that?
My code is:
import java.io.BufferedReader;
import java.io.DataInputStream;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStreamReader;
public class FileReading {
/**
* #param args
* #throws IOException
*/
public static void main(String[] args) throws IOException {
// TODO Auto-generated method stub
FileInputStream fi=new
FileInputStream("E:\\Tejas\\NewData_FromNov\\New_Folder\\bb.txt");
DataInputStream di=new DataInputStream(fi);
BufferedReader buf=new BufferedReader(new InputStreamReader(di));
FileOutputStream fout=new FileOutputStream("E:\\Tejas\\NewData_FromNov\\New_Folder\\Out_bb.txt");
int ss;
byte[] input=new byte[500];
int len=input.length;
while((ss=di.read(input,0,len))!=-1)
{
System.out.println(ss);
//fout.write(ss);
fout.write(input,0,len);
}
fout.close();
}
}
You're always writing out the full buffer, even if you've only read part of it because the third argument to write is len (the length of the buffer) instead of ss (the number of bytes read). Your loop should look like this:
int bytesRead; // Easier to understand than "ss"
byte[] buffer = new byte[500];
while((bytesRead = di.read(buffer, 0, buffer.length)) != -1)
{
System.out.println(bytesRead);
fout.write(buffer, 0, bytesRead);
}
Additionally:
You should close both the input and output streams in finally blocks to ensure they're always closed (even if there's an exception).
You don't need a DataInputStream - just a FileInputStream is fine here.
You're not using your BufferedReader at all.
Consider using Guava or a similar third-party library which contains utility methods to do all of this.
The read method returns the number of actually read bytes, or -1 if the end of the stream has been reached. So you should only write ss bytes, and not len bytes:
while ((ss = di.read(input, 0, len)) != -1) {
System.out.println(ss);
fout.write(input, 0, ss);
}
Note thet the DataInputStream and the BufferedReader are completely unnecessary here.