Part of my application is given an InputStream and wants to do some processing on this to produce another InputStream.
try (
final BufferedReader inputReader = new BufferedReader(new InputStreamReader(inputStream, UTF_8), BUFFER_SIZE);
final Stream<String> resultLineStream = inputReader.lines().map(lineProcessor::processLine);
final InputStream resultStream = new ReaderInputStream(new StringStreamReader(resultLineStream), UTF_8);
) {
s3Client.putObject(targetBucket, s3File, resultStream, new ObjectMetadata());
} catch (IOException e) {
throw new RuntimeException("Exception", e);
}
I am using the new Java 8 BufferedReader.lines() to a Stream onto which I can easily map my processing function.
The only thing still lacking is class StringStreamReader() which is supposed to turn my Stream into a Reader from which Apache commons-io:ReaderInputStream can create an InputStream again. (The detour to readers and back seems reasonable to deal with encodings and line breaks.)
To be very clear, the code above assumes
public class StringStreamReader extends Reader {
public StringStreamReader(Stream<String> stringStream) { ... }
#Overwrite
public int read(char cbuf[], int off, int len) throws IOException { ... }
// possibly overwrite other methods to avoid bad performance or high resource-consumption
}
So is there any library that offers such a StringStreamReader class? Or this there another way to write the application code above without implementing a custom Reader or InputStream subclass?
You can do something like that:
PipedWriter writer = new PipedWriter();
PipedReader reader = new PipedReader();
reader.connect(writer);
strings.stream().forEach(string -> {
try {
writer.write(string);
writer.write("\n");
} catch (Exception e) {
e.printStackTrace();
}
});
But i guess you want some form of lazy processing. Stream api does not really help in that case, you need a dedicated Thread + some buffer to do that.
Related
I'm using the new java.net.http classes to handle asynchronous HTTP request+response exchanges, and I'm trying to find a way to have the BodySubscriber handle different encoding types such as gzip.
However, mapping a BodySubsriber<InputStream> so that the underlying stream is wrapped by a GZIPInputStream (when "Content-Encoding: gzip" is found in the response header) leads to a hang. No exceptions, just a total cessation of activity.
The code which maps the BodySubscriber looks like this:
private HttpResponse.BodySubscriber<InputStream> gzippedBodySubscriber(
HttpResponse.ResponseInfo responseInfo) {
return HttpResponse.BodySubscribers.mapping(
HttpResponse.BodySubscribers.ofInputStream(),
this::decodeGzipStream);
}
private InputStream decodeGzipStream(InputStream gzippedStream) {
System.out.println("Entered decodeGzipStream method.");
try {
InputStream decodedStream = new GZIPInputStream(gzippedStream);
System.out.println(
"Created GZIPInputStream to handle response body stream.");
return decodedStream;
} catch (IOException ex) {
System.out.println("IOException occurred while trying to create GZIPInputStream.");
throw new UncheckedIOException(ex);
}
}
Receiving an HTTP response which has "gzip" encoding leads to the console showing just this:
Entered EncodedBodyHandler.apply method.
Entered decodeGzipStream method.
Nothing more is seen, so the line after the call to the GZIPInputStream constructor is never executed.
Does anyone know why this attempt to wrap the InputStream from a BodySubscriber<InputStream> in a GZIPInputStream is hanging?
Note: the equivalent method for unencoded (raw text) HTTP response bodies contains simply a call to BodySubscribers.ofInputStream() with no mapping, and this allows the response to be received and displayed without problem.
EDIT: JDK-8217264 is fixed since JDK13
This is indeed a bug. I have logged JDK-8217264. I can suggest two work-arounds:
Workaround one
Do not use BodySubscribers.mapping - but transform the InputStream into a GZIPInputStream after getting the HttpResponse's body:
GZIPInputStream gzin = new GZIPInputStream(resp.getBody());
Workaround two
Have the mapping function return a Supplier<InputStream> instead, taking care not to create the GZIPInputStream until Supplier::get is called
static final class ISS implements Supplier<InputStream> {
final InputStream in;
GZIPInputStream gz;
ISS(InputStream in) {
this.in = in;
}
public synchronized InputStream get() {
if (gz == null) {
try {
gz = new GZIPInputStream(in);
} catch (IOException t) {
throw new UncheckedIOException(t);
}
}
return gz;
}
}
Encountered the exact same problem. I tried the example in the Javadoc of the BodySubscribers.mapping method. Same behavior, the application hangs without any errors.
Could be a bug, because this is an official example from the Javadoc.
public static <W> BodySubscriber<W> asJSON(Class<W> targetType) {
BodySubscriber<InputStream> upstream = BodySubscribers.ofInputStream();
BodySubscriber<W> downstream = BodySubscribers.mapping(
upstream,
(InputStream is) -> {
try (InputStream stream = is) {
ObjectMapper objectMapper = new ObjectMapper();
return objectMapper.readValue(stream, targetType);
} catch (IOException e) {
throw new UncheckedIOException(e);
}
});
return downstream;
} }
I am using piped output streams to convert OutputStream to InputStream because the AWS java sdk does not allow puting objects on S3 using OutputStreams
I'm using the code below, however, this will intermittently just hang. This code is in a web application. Currently there is no load on the application...I am just trying it out on my personal computer.
ByteArrayOutputStream os = new ByteArrayOutputStream();
PipedInputStream inpipe = new PipedInputStream();
final PipedOutputStream out = new PipedOutputStream(inpipe);
try {
String xmpXml = "<dc:description>somedesc</dc:description>"
JpegXmpRewriter rewriter = new JpegXmpRewriter();
rewriter.updateXmpXml(isNew1,os, xmpXml);
new Thread(new Runnable() {
public void run () {
try {
// write the original OutputStream to the PipedOutputStream
println "starting writeto"
os.writeTo(out);
out.close();
println "ending writeto"
} catch (IOException e) {
System.out.println("Some exception)
}
}
}).start();
ObjectMetadata metadata1 = new ObjectMetadata();
metadata1.setContentLength(os.size());
client.putObject(new PutObjectRequest("test-bucket", "167_sample.jpg", inpipe, metadata1));
}
catch (Exception e) {
System.out.println("Some exception")
}
finally {
isNew1.close()
os.close()
}
Instead of bothering with the complexities of starting another thread, instantiating two concurrent classes, and then passing data from thread to thread, all to solve nothing but a minor limitation in the provided JDK API, you should just create a simple specialization of the ByteArrayOutputStream:
class BetterByteArrayOutputStream extends ByteArrayOutputStream {
public ByteArrayInputStream toInputStream() {
return new ByteArrayInputStream(buf, 0, count);
}
}
This converts it to an input stream with no copying.
I'm trying to download a pdf file using URLConnection. Here's how I setup the connection object.
URL serverUrl = new URL(url);
urlConnection = (HttpURLConnection) serverUrl.openConnection();
urlConnection.setDoInput(true);
urlConnection.setRequestMethod("GET");
urlConnection.setRequestProperty("Content-Type", "application/pdf");
urlConnection.setRequestProperty("ENCTYPE", "multipart/form-data");
String contentLength = urlConnection.getHeaderField("Content-Length");
I obtained inputstream from the connection object.
bufferedInputStream = new BufferedInputStream(urlConnection.getInputStream());
And the output stream to write the file contents.
File dir = new File(context.getFilesDir(), mFolder);
if(!dir.exists()) dir.mkdir();
final File f = new File(dir, String.valueOf(documentName));
f.createNewFile();
final BufferedOutputStream bufferedOutputStream = new BufferedOutputStream(new FileOutputStream(f, true)); //true for appendMode
BlockingQueue is created so that threads performing read and write operations can access the queue.
final BlockingQueue<ByteArrayWrapper> blockingQueue = new ArrayBlockingQueue<ByteArrayWrapper>(MAX_VALUE,true);
final byte[] dataBuffer = new byte[MAX_VALUE];
Now created thread to read data from InputStream.
Thread readerThread = new Thread(new Runnable() {
#Override
public void run() {
try {
int count = 0;
while((count = bufferedInputStream.read(dataBuffer, 0, dataBuffer.length)) != -1) {
ByteArrayWrapper byteArrayWrapper = new ByteArrayWrapper(dataBuffer);
byteArrayWrapper.setBytesReadCount(count);
blockingQueue.put(byteArrayWrapper);
}
blockingQueue.put(null); //end of file
} catch(Exception e) {
e.printStackTrace();
} finally {
try {
bufferedInputStream.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
});
Now the writer thread reads those file contents.
Thread writerThread = new Thread(new Runnable() {
#Override
public void run() {
try {
while(true) {
ByteArrayWrapper byteWrapper = blockingQueue.take();
if(null == byteWrapper) break;
bufferedOutputStream.write(byteWrapper.getBytesRead(), 0, byteWrapper.getBytesReadCount());
}
bufferedOutputStream.flush();
} catch(Exception e) {
e.printStackTrace();
} finally {
try {
bufferedOutputStream.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
});
Finally, threads are started.
readerThread.start();
writerThread.start();
Theoretically it should read the file from InputStream and save it to the target file. However, in reality, it produces blank pdf file. At some other time, it shows invalid pdf format exception. File size matches with content length of the InputStream. Is there anything I'm missing?
I'm not familiar with ByteArrayWrapper. Does it just hold a reference to the array, like this?
public class ByteArrayBuffer {
final private byte[] data;
public ByteArrayBuffer(byte[] data) {
this.data = data;
}
public byte[] getBytesRead() {
return data;
}
/*...etc...*/
}
If so. that would be the problem: all of the ByteArrayWrapper objects are backed by the same array. Which is repeatedly overwritten by the writer. Even though BlockingQueue did the hard work of safely publishing each object from one thread to the other.
The simplest fix might be to make the ByteArrayWrapper effectively immutable i.e. don't change it after publishing it to another thread. Taking a copy of the array on construction would be simplest:
public ByteArrayWrapper(byte[] data) {
this.data = Arrays.copyOf(data, data.length);
}
One other problem is that "BlockingQueue does not accept null elements" (see BlockingQueue docs), and so the "end of input" sentinel value doesn't work. Replacing null with a
private static ByteArrayWrapper END = new ByteArrayWrapper(new byte[]{});
in the appropriate places will fix that.
By making those changes to a copy of the code I was able to retrieve a faithful copy of a PDF file.
Try to use Android DownloadManager (http://developer.android.com/reference/android/app/DownloadManager.html) it is used to handle long-running HTTP requests in the background.
Here you don't need to think about received bytes and the progress is displayed in the notification bar.
There is a good tutorial here: http://blog.vogella.com/2011/06/14/android-downloadmanager-example/
I met a problem with BufferedWriter when I write data to a single file with some threads.
I set the buffer size of the BufferedWriter, but no matter what number I set, it flushes the data to disk when the buffer is 8192 (the default buffer size), not the size I set (here is 16384). Is there a problem with my code?
This is how I'm constructing the BufferedWriter:
new BufferedWriter(new FileWriter(fileName, true), 16384);
This is the full code:
import java.io.BufferedWriter;
import java.io.File;
import java.io.FileWriter;
import java.io.IOException;
public class Test1 {
public static void main(String[] args) throws IOException {
for(int i =0;i<10;i++){
MyThread r = new MyThread();
Thread t = new Thread(r);
t.start();
}
}
}
class MyThread implements Runnable {
public void run() {
String s = "{addffffffkkkljlkj2015dd}\n";
BufferedWriter bw = null;
try {
bw = new BufferedWriter(new FileWriter(
"/Users/liaoliuqing/Downloads/1.txt", true),16384);
} catch (IOException e) {
e.printStackTrace();
}
for(int i =0 ; i<1000; i++){
try {
bw.write(String.format("%03d", i)+s);
//bw.flush();
} catch (IOException e) {
e.printStackTrace();
}
}
}
}
Is there a problem with my code?
A few. Mainly: potential IO and concurrency errors. File buffer size might be a lesser concern (and one you can't effectively deal with).
Trying to open already opened file. All your threads are trying to write into the same file (1.txt). That might be an issue. FileWriter documentation says:
Some platforms, in particular, allow a file to be opened for writing by only one FileWriter (or other file-writing object) at a time. In such situations the constructors in this class will fail if the file involved is already open.
Lines might be cut and mixed. If you have several threads with their respective buffers flushing at some point into the same output, you might not even need weird race-conditions or threads stopped right of the middle or a write operation to see your output corrupted.
As I solution (If your threads must share the same output) you can use a shared object with synchronized access to take care of actual writing. I implemented SafeAppender in my example, but probably there are better alternatives out there.
No flushing and closing buffers will mean (the tail of) your data will be lost (like tears in the rain). A finally block is usually good to take care of that.
Also, as stated by other users, BufferedWriter buffer size does not affect the buffer size in FileOutputStream (and so FileWriter). And it looks the java.io and java.nio APIs dont offer any way to mess with that. If you look at the Java library sources you might notice BufferedWriter buffer size just means the amount of chars you store before actually writing into the delegate output. The default size (8192) is optimal for most cases, and increasing it might mean more trouble (potentially losing more data) than benefits.
This is my code, if it serves you:
// http://stackoverflow.com/questions/32451526/how-to-set-the-buffer-size-on-a-bufferedwriter-over-a-filewriter
public class TestWriter {
public static class SafeAppender {
private BufferedWriter bw;
private int users = 0;
public SafeAppender(File f) throws IOException {
bw = new BufferedWriter(new FileWriter(f));
}
public synchronized void append(String s) throws IOException {
bw.write(s);
}
public synchronized void incrUsers() {
users ++;
}
public synchronized void decrUsers() {
if (--users <= 0) {
try {
bw.flush();
System.err.println("INFO-appender-flush()");
} catch (Throwable whatever) { /* log-if-you-care*/}
}
}
// Might be called by GC, or not
#Override protected void finalize() throws Throwable {
try {
bw.close();
System.err.println("INFO-appender-close()");
} catch (Throwable whatever) { /* log-if-you-care */}
super.finalize();
}
}
private static class MyRunnable implements Runnable {
final static String S = "{addffffffkkkljlkj2015dd}";
SafeAppender appender;
String threadId;
public MyRunnable (SafeAppender a, String tid) {
appender = a; threadId = tid;
}
public void run() {
appender.incrUsers();
try {
for(int i =0 ; i<1000; i++){
// NOTE: Not a good idea to printStackTrace if each line fails. Let thread fail
String line = String.format("%s-%03d-%s\n", threadId, i, S);
appender.append(line);
}
} catch (IOException e) {
System.err.printf("ERROR-%s-%s\n", threadId, e.toString());
} finally {
appender.decrUsers();
}
}
}
public static void main(String[] args) {
try {
File f = File.createTempFile("TestWriter", ".txt");
System.err.printf("INFO-main-Writing into %s\n", f.getCanonicalPath());
SafeAppender appender = new SafeAppender (f);
for(int i =0;i<10;i++){
MyRunnable r = new MyRunnable(appender, ""+i);
Thread t = new Thread(r);
t.start();
}
} catch (Throwable e) {
e.printStackTrace(System.err);
}
}
}
FileWriter actually uses its own fixed-size 1024 byte buffer. The BufferedWriter on the other hand, show that it uses and 8192 byte buffer size (default), which can be configured by the user to any other desired size.
And to further muddy the waters, the Java 6 implementation of OutputStreamWriter actually delegates to a StreamEncoder, which uses its own buffer with a default size of 8192 bytes. And the StreamEncoder buffer is user-configurable, although there is no way to access it directly through the enclosing OutputStreamWriter.
I solve the problem by using OutputStream, not writer, here is the code:
bw = new BufferedOutputStream(
new FileOutputStream(new File("/Users/liaoliuqing/Downloads/1.txt"),true),165537);
What you are seeing is not the size of the buffer BufferedWriter, but the size of the buffer used internally by FileWriter. Quoting from the Java Documentation (http://docs.oracle.com/javase/7/docs/api/java/io/FileWriter.html)
The constructors of this class assume that the default character encoding and the default byte-buffer size are acceptable. To specify these values yourself, construct an OutputStreamWriter on a FileOutputStream.
So if you wanted to have a fine grain control on when the data is actually written to the disk you should instantiate your BufferedWriter as
bw = new BufferedWriter(new OutputStreamWriter(new FileOutputStream(new File('my_file.txt),true)));
I have admin account which should be able to add many users to a .dat file. Then I want to retrieve all the objects from the .dat file into a list for further programming.
public class User implements Serializable { //get and set methods }
This is hwo I am writing each object to the .dat file
public void addNewUser() throws Exception {
User newUser=new User();
newUser.name="test";
newUser.position="admin";
FileOutputStream outStream = new FileOutputStream("Users.dat", true);
ObjectOutputStream objectOutputFile = new ObjectOutputStream(outStream);
// Write the object to the file.
objectOutputFile.writeObject(newUser);
// Close the file.
objectOutputFile.close();
}
How can retrieve all the objects from the .dat file into ArrayList??
public class displayUsers { **//what to do??** }
You can either write the list object and read it as list. But since you're writing user objects individually, you can do something like this -
ObjectInputStream ois = new ObjectInputStream(new FileInputStream("Users.dat"));
Object object = null;
while ((object = ois.readObject()) != null) {
if (object instanceof User) {
User user = (User) object;
list.add(user);
}
}
Of course, you would need to take care of exceptions (like EOFException).
Generally it is bad practice to concatenate individual ObjectOutputStreams in a file without adding any lengths or delimiters. So better write all objects in one pass (and use ObjectOutputStream.reset in case your process is long-running and you fear memory leaks (otherwise ObjectOutputStream will keep a reference to every object it serialized before) or add them to a List and write it.
If you have to write it in multiple passes, I'd suggest to write the individual objects to a ByteArrayOutputStream first, and then use DataOutputStream to write the array prefixed by its length. That way, you can use DataInputStream to get out the individual byte arrays and use ByteArrayInputStream to deserialize them.
In case this does not work, you can try this solution (depending on the lookahead used by ObjectInputStream, this might not work for more complex objects with custom serialization formats, though, so use at your own risk):
public static void displayUsers() throws Exception {
FileInputStream fiis = new FileInputStream("Users.dat");
InputStream fis = new FilterInputStream(fiis) {
#Override
public void close() throws IOException {
// ignore
}
};
try {
while (true) {
ObjectInputStream in = new ObjectInputStream(fis);
User user = (User) in.readObject();
in.close();
System.out.println(user.name + "/" + user.position);
}
} catch (EOFException ex) {
// done
}
fiis.close();
}
List<User> listOfUser = new ArrayList<User>();
ObjectInputStream input = null;
try {
while (true) {
input = new ObjectInputStream(new FileInputStream("Users.dat"));
listOfUser.add(input.readObject());
}
}
catch (ClassNotFoundException e) {
e.printStackTrace();
} catch (FileNotFoundException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
finally {
input.close();
}