I want to write multiple objects to a file, but the problem is that I dont have all the objects to write at once. I have to write one object and then close the file, and then maybe after sometime I want to add another object to the same file.
I am currently doing it as
FileOutputStream("filename", true)
so that it will append the object to the end of file and not overwrite it. But I get this error :
java.io.StreamCorruptedException: invalid type code: AC
any ideas how can I solve this issue ?
Thanks,
One option is to segment the file into individual messages. When you want to write a message, first serialize it to a ByteArrayOutputStream. Then open the file for appending with DataOutputStream - write the length with writeInt, then write the data.
When you're reading from the stream, you'd open it with DataInputStream, then repeatedly call readInt to find the length of the next message, then readFully to read the message itself. Put the message into ByteArrayInputStream and then deserialize from that.
Alternatively, use a nicer serialization format than the built-in Java serialization - I'm a fan of Protocol Buffers but there are lots of alternatives available. The built-in serialization is too brittle for my liking.
You can't append different ObjectOutputStreams to the same file. You would have to use a different form of serialization, or read the file in and write out all the objects plus the new objects to a new file.
You need to serialize/deserialize the List<T>. Take a look at this stackoverflow thread.
Related
I want to store a bunch of protobuf messages in a file, and read them later.
In java, I can just use 'writeDelimitedTo' and 'parseDelimitedFrom' to read and write to a file. However, I want to read it in Python, which only seems to have a 'ParseFromString' method.
Some SO questions are very similar, such as, Parsing Protocol Buffers, written in Java and read in Python, but that is only for a single message: not for multiple.
From the proto guide it is written that you need to deal yourself with the size of your message:
Streaming Multiple Messages
If you want to write multiple messages to a single file or stream, it
is up to you to keep track of where one message ends and the next
begins. The Protocol Buffer wire format is not self-delimiting, so
protocol buffer parsers cannot determine where a message ends on their
own. The easiest way to solve this problem is to write the size of
each message before you write the message itself. When you read the
messages back in, you read the size, then read the bytes into a
separate buffer, then parse from that buffer. (If you want to avoid
copying bytes to a separate buffer, check out the CodedInputStream
class (in both C++ and Java) which can be told to limit reads to a
certain number of bytes.)
https://developers.google.com/protocol-buffers/docs/techniques
A simple solution could be for you to serialize each proto in base64, on a new line in your file.
Doing so, it would be pretty easy on python to parse and use them.
I need to pass an InputStream to an object which reads data which I previously stored into a File. I'm assessing a more efficient approach than storing eveything into a File and then passing the FileInputStream. I'd like to do it on the fly.
May someone appoint me the correct approach to do that?
The idea would be passing a Custom InputStream which innerly calls every line I was going to store in the file. I guess I need buffering. I discard storing everything in a String and then build an InputStream on it, as we are in the same situation, waiting to output all the lines before rereading them again.
There already is a stream for this. It's the PipedInputStream. You'll need to have one thread write to the PipedOutputStream, and pass the PipedInputStream to the object that will be reading in another thread.
I'm working with a library that I have to provide an InputStream and a PrintStream. It uses the InputStream to gather data for processing and the PrintStream to provide results. I'm stuck using this library and its API cannot be altered.
There are two issues with this that I think have related solutions.
First, the data that needs to be read via the InputStream is not available upfront. Instead, the data is dynamically created by a different part of the application and given to my code as a String via method call. My code's job is to somehow allow the library to read this data through the InputStream provided as I get it.
Second, I need to somehow get the result that is written to the PrintStream and send it to another part of the application as a String. This needs to happen as immediately after the data is put in to the PrintStream as possible.
What it looks like I need are two stream objects that behave more or less like buffers. I need an InputStream that I can shove data in to whenever I have it and a PrintStream that I can grab it's contents whenever it has some. This seems a little awkward to me, but I'm not sure how else to do it.
I'm wondering if anything already exists that allows this kind of behavior or if there is a different (better) solution that will work in the situation I've described. The only thing I can come up with is to try to implement streams with this behavior, but that can become complicated fast (especially since the InputStream needs to block until data is available).
Any ideas?
Edit: To be clear, I'm not writing the library. I'm writing code that is supposed to provide the library with an InputStream to read data from and a PrintStream to write data to.
Looks like both streams need to be constantly reading/writing so you'll need two threads independent of each other. The pattern resembles JMS a little bit, in which case you're feeding information to a "queue" or "topic", and wait for it to be processed then put on a "output" queue/topic. This may introduce additional moving parts, but you could write a simple client to place info onto a JMS queue, then have a listener to just grab messages, and feed it to the input stream constantly. Then another piece of code to read from output stream, and do what you need with it.
Hope this helps.
I am sending a plain text file to the user through a servlet.
I am using flatworm framework to build the flat file. I receive the file in the browser but is empty. So i want start the debugging analysing the outputstream before being sent.
How i can read the response before i send it in the servlet? I think is the same thing that asking how can i transform an OutputStream to an InputStream.
I already saw solutions that always involve ByteArrayOutputStream , and as you know when i call in the servlet response.getOutputStream() it returns me an OutputStream and not a ByteArrayOutputStream.
There seems to be some confusion somewhere, though I'm not sure exactly where.
What can you do with an OutputStream? Why, you can write to it, and that's about it. That means that if you're given (or look up) an output stream, it's up to you to supply the data - which means you already have it.
Perhaps on the other hand, you're not directly calling write on the OutputStream yourself, but passing this stream into the flatworm library (which will in turn write output to it). In that case, there's your debugging "hook" right there - flatworm will write out the file to any output stream you send it. So in this case, instead of passing in the servlet's stream, you pass in a stream that you've created yourself.
That might be a ByteArrayOutputStream, which (after the flatworm method has returned) you can inspect to get the bytes written. At this point you could manually write them through to the response's output stream. Or maybe you need to do something slightly trickier and create your own stream wrapper which writes straight through to the underlying response stream but logs on the way - and pass this into flatworm.
The bottom line however is that if you're interacting with an output stream, then "your" code already has the data somewhere locally and it's just a matter of capturing/accessing that.
I am developing a socket application and my application needs to receive xml file over socket. The size of xml files received vary from 1k to 100k. I am now thinking of storing data that I received into a temporary file first, then pass it to the xml parser. I am not sure if it is a proper way to do it.
Another question is if I wanna do as mentioned above, should I pass file object or file path to xml parser?
Thanks in advance,
Regards
Just send it straight to the parser. That's what browsers do. Adding a temp file costs you time and space with no actual benefit.
Do you think it would work to put a BufferedReader around whatever input stream you have? It wouldn't put it into a temporary file, but it would let you hang onto that data. You can set whatever size BufferedReader you need.
Did you write your XML parser? If you didn't, what will it accept as a parameter? If you did write it, are you asking about efficiency. That is to say which object, the path or file, should your parser ask for to be most efficient?
You do not have to store the data from socket to any file. Just read whole the DataInputStream into a byte array and you can then do whatever you need. E.g. if needed create a String with the xml input to feed the parser. (I am assuming tcp sockets).
If there are preceding data you skip them so as to feed the actual xml data to the parser.