We Keep Coding

IOException problem with file loading - Java / [closed]

Closed. This question needs debugging details. It is not currently accepting answers.
Edit the question to include desired behavior, a specific problem or error, and the shortest code necessary to reproduce the problem. This will help others answer the question.
Closed 7 months ago.
Improve this question
How can i fix problem with load ArrayList from file?
Full code is on the git:
project github link
When I load data from file I'll get it back -> IOException
I'm learning to use the stream so I'm writing it to a file.
After the end of the program, I want to write the time the user has reached in the arraylist and list all the times with which the game was finished.
package sk.games.puzzle;
import java.io.*;
import java.util.*;
public class BestTimes implements Iterable<BestTimes.PlayerTime>, Serializable{
private static final String BESTTIME_DB = System.getProperty("user.home")
+ System.getProperty("file.separator")
+ "best.time";
private List<PlayerTime> playerTimes = new ArrayList<>();
public Iterator<PlayerTime> iterator() {
return playerTimes.iterator();
}
public void addTime(String name, int time){
playerTimes.add(new PlayerTime(name, time));
Collections.sort(playerTimes);
}
public void load(){
ObjectInputStream load = null;
try {
load = new ObjectInputStream(new FileInputStream(BESTTIME_DB));
playerTimes = (ArrayList<PlayerTime>) load.readObject();
} catch (FileNotFoundException e) {
System.err.println("fail nebola najdena db");
} catch (IOException e) {
System.err.println("fail nebola otvorena db");
} catch (ClassNotFoundException e) {
System.err.println("fail nebol najdeny zaznam");
} finally {
if (load != null) {
try {
load.close();
} catch (IOException e) {
//empty
}
}
}
}
public void save() {
ObjectOutputStream save = null;
try {
save = new ObjectOutputStream(new FileOutputStream(BESTTIME_DB));
save.writeObject(playerTimes);
} catch (FileNotFoundException e) {
System.err.println("fail db neexistuje");
} catch (IOException e) {
System.err.println("fail nepodarilo sa otvorit db");
} finally {
if (save != null) {
try {
save.close();
} catch (IOException e) {
//empty
}
}
}
}
#Override
public String toString() {
Formatter f = new Formatter();
for (int i = 0; i < playerTimes.size(); i++) {
PlayerTime pt = playerTimes.get(i);
f.format("%02d. %s - %ds.\n", i, pt.getName(), pt.getTime());
}
return f.toString();
}
public static class PlayerTime implements Comparable<PlayerTime> {
private final String name;
private final int time;
public PlayerTime(String name, int time) {
this.name = name;
this.time = time;
}
public String getName() {
return name;
}
public int getTime() {
return time;
}
#Override
public int compareTo(PlayerTime o){
return Integer.compare(this.time, o.getTime());
}
}
}

The problem is that your PlayerTime class is not serializable.
public static class PlayerTime implements Comparable<PlayerTime> { }
should be
public static class PlayerTime implements Comparable<PlayerTime> implements Serializable { }
It's not necessary to make BestTimes serializable unless you do write BestTimes object to file.

Spring ApplicationEvent onApplicationEvent method

Expected: When i run application in debug mode and pulling the endpoint, bytes still appear to be null however i did implement ApplicationEvent and passed ApplicationStartedEvent, then I have override onApplicationEvent and called my method there, which should lead to code execution once application started and bytes should already have a value. Have I missed something
public class FaqAttachment implements ApplicationListener<ApplicationStartedEvent> {
private final String fileName = "FAQ.pdf";
private byte[] bytes;
public Attachment asAttachment() {
return new Attachment(pdfToBytes(), fileName);
}
private byte[] pdfToBytes() {
if (bytes == null) {
try (FileInputStream inputStream = new FileInputStream(new File(ClassLoader.getSystemResource(fileName).getFile()))) {
this.bytes = ByteStreams.toByteArray(inputStream);
} catch (IOException e) {
throw new RuntimeException(e);
}
}
return bytes;
}
#Override
public void onApplicationEvent(ApplicationStartedEvent event) {
pdfToBytes();
}

This should work :
#Component
public class FaqAttachment {
private final String fileName = "FAQ.pdf";
private byte[] bytes;
public Attachment asAttachment() {
return new Attachment(pdfToBytes(), fileName);
}
private byte[] pdfToBytes() {
if (bytes == null) {
try (FileInputStream inputStream = new FileInputStream(new File(ClassLoader.getSystemResource(fileName).getFile()))) {
this.bytes = ByteStreams.toByteArray(inputStream);
} catch (IOException e) {
throw new RuntimeException(e);
}
}
return bytes;
}
#EventListener
public void onApplicationStartedEvent(ApplicationStartedEvent event) {
pdfToBytes();
}
}

File Transport between Server/Client

What kind of Service should I define for ".thrift"-file to use it later for my Program?
This File Transport should be between the Client and the Server and it should be "partly".
StreamFileService.thrift:
struct FileChunk {
1: binary data
2: i64 remaining
}
service StreamFileService {
FileChunk getBytes(1:string fileName, 2: i64 offset, 3: i32 size);
}
StreamFileClient.java:
public class StreamFileClient {
private int fileChunkSize = 16;
private String filePath;
public String getFilePath() {
return filePath;
}
public void setFilePath(String filePath) {
this.filePath = filePath;
}
private void invoke() {
try {
TTransport theClientTransport = new TFramedTransport(new TSocket(
"127.0.0.1", 7911));
TProtocol theProtocol = new TBinaryProtocol(theClientTransport);
StreamFileService.Client theClient = new StreamFileService.Client(
theProtocol);
theClientTransport.open();
filePath = "/home/output/output.pdf";
File theFile2 = new File(filePath);
theFile2.createNewFile();
FileInputStream stream = new FileInputStream(theFile2);
long currentPosition = 0;
FileChannel theFileChannel = stream.getChannel();
boolean again = true;
do {
FileChunk chunk2 = theClient.getBytes(filePath,
currentPosition, fileChunkSize);
currentPosition += fileChunkSize;
theFileChannel.write(chunk2.data);
if (chunk2.remaining == 0)
again = false;
} while (again);
stream.close();
} catch (TTransportException e) {
e.printStackTrace();
} catch (TException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (FileNotFoundException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
public static void main(String[] args) {
StreamFileClient theClient = new StreamFileClient();
theClient.invoke();
}
}
StreamFileServer.java:
public class StreamFileServer {
private void start() {
try {
TNonblockingServerTransport theServerSocket = new TNonblockingServerSocket(
7911);
StreamFileService.Processor theProcessor = new StreamFileService.Processor(
new StreamFileServiceImpl());
TServer theServer = new TNonblockingServer(
new TNonblockingServer.Args(theServerSocket)
.processor(theProcessor));
System.out.println("Server starting on port 7911...");
theServer.serve();
} catch (TTransportException e) {
e.printStackTrace();
}
}
public static void main(String[] args) {
StreamFileServer theFileServer = new StreamFileServer();
theFileServer.start();
}
}
StreamFileServiceImpl:
public class StreamFileServiceImpl implements StreamFileService.Iface {
public FileChunk getBytes(String filePath, long offset, int size)
throws TException {
File theFile = new File("/home/input/kl_12.pdf");
FileChunk chunk = new FileChunk();
try {
FileOutputStream stream = new FileOutputStream(theFile);
MappedByteBuffer buffer = stream.getChannel().map(
FileChannel.MapMode.READ_ONLY, offset, size);
chunk.data = buffer;
chunk.remaining = stream.getChannel().size() - offset - size;
stream.close();
} catch (FileNotFoundException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
return chunk;
}
}

Your code looks not so bad to me (not tested) and there is not much to change.
How about
typedef binary binar
service StreamFileService {
binar getBytes(1:string fileName, 2: i64 offset, 3: i32 size);
i64 getSize(1:string fileName)
}
I would also return a struct holding the bytes, but that's more or less my personal opinion.
struct FileChunk {
1: binary data
2: i64 remaining
}
service StreamFileService {
FileChunk getBytes(1:string fileName, 2: i64 offset, 3: i32 size);
}
The FileChunk struct can be easily extended, if such becomes necessary, e.g. in order to return additional metadata, like total size (especially if the size grows/shrinks over time), remaining bytes, indications about the data format, or the like. You don't have to do that, as you can easily extend the interface if such becomes necessary later. Matter of taste.

Has anyone played with NIO pipes to filter / intercept System.out?

As suggested here I would like to do that inside the selector loop. What I would really want is to read contents written to system out inside my selector loop.
EDIT1: I coded a complete solution just to find out that you CANNOT redirect GC logs by using System.setOut. It simply goes straight to the FD or something. Show stopper! Unless I redirect to a file and pipe this file into my selector. Lots of work! See here.

One way to do it would be as follows:
create a subclass of OutputStream that redirects its output to a Pipe's sink channel
redirect System.out using this class: System.setOut(new PrintStream(new MyOutputStream(pipe));
register the pipe's source channel with a selector and get whatever was written to System.out in the selector loop, i.e. the source channel's correpsonding SelectionKey is selected as readable()
The following immplementation is a naive but working implementation, which simply redirects to System.err everything that is written to System.out:
import java.io.*;
import java.nio.ByteBuffer;
import java.nio.channels.*;
import java.util.Iterator;
public class SystemOutPipe extends Thread {
public static void main(String[] args)
{
try {
SystemOutPipe sop = new SystemOutPipe();
sop.start();
System.out.println("This message should be redirected to System.err\nNow waiting 5 seconds ...");
Thread.sleep(5000L);
sop.setStopped(true);
sop.join();
} catch (Exception e) {
e.printStackTrace();
}
}
private Selector selector;
private Pipe pipe;
private boolean stopped = false;
public SystemOutPipe() throws IOException {
super("SystemOutPipe");
pipe = Pipe.open();
System.setOut(new PrintStream(new PipeOutputStream(pipe)));
selector = Selector.open();
pipe.source().configureBlocking(false);
pipe.source().register(selector, SelectionKey.OP_READ, ByteBuffer.allocate(1024));
}
#Override
public void run() {
try {
while (!isStopped()) {
int n = selector.select(1L);
if (n > 0) {
Iterator<SelectionKey> it = selector.selectedKeys().iterator();
while (it.hasNext()) {
SelectionKey key = it.next();
it.remove();
if (key.isReadable()) {
new ReadHandler(key).run();
}
}
}
}
} catch (Exception e) {
e.printStackTrace(); // writes to System.err !
}
}
public synchronized boolean isStopped() {
return stopped;
}
public synchronized void setStopped(final boolean stopped) {
this.stopped = stopped;
}
public class ReadHandler implements Runnable {
private final SelectionKey key;
public ReadHandler(final SelectionKey key) {
this.key = key;
}
#Override
public void run() {
ByteBuffer bbuf = (ByteBuffer) key.attachment();
ReadableByteChannel channel = (ReadableByteChannel) key.channel();
try
{
int count = 0;
do {
bbuf.clear();
count = channel.read(bbuf);
if (count > 0) System.err.write(bbuf.array(), 0, count);
} while(count > 0);
} catch (IOException e) {
e.printStackTrace();
key.cancel();
}
}
}
public class PipeOutputStream extends OutputStream {
private final Pipe pipe;
public PipeOutputStream(final Pipe pipe) {
this.pipe = pipe;
}
#Override
public void write(final int b) throws IOException {
write(new byte[] { (byte) b });
}
#Override
public void write(final byte[] b) throws IOException {
write(b, 0, b.length);
}
#Override
public void write(final byte[] b, final int off, final int len) throws IOException {
ByteBuffer bbuf = ByteBuffer.wrap(b, off, len);
bbuf.position(len);
bbuf.flip();
int count = 0;
while (count < len) {
int n = pipe.sink().write(bbuf);
if (n == 0) {
// let's wait a bit and not consume cpu
try {
Thread.sleep(1L);
} catch (InterruptedException e) {
throw new IOException(e);
}
}
else count += n;
}
}
}
}

What is the best way to write to a file in a parallel thread in Java?

I have a program that performs lots of calculations and reports them to a file frequently. I know that frequent write operations can slow a program down a lot, so to avoid it I'd like to have a second thread dedicated to the writing operations.
Right now I'm doing it with this class I wrote (the impatient can skip to the end of the question):
public class ParallelWriter implements Runnable {
private File file;
private BlockingQueue<Item> q;
private int indentation;
public ParallelWriter( File f ){
file = f;
q = new LinkedBlockingQueue<Item>();
indentation = 0;
}
public ParallelWriter append( CharSequence str ){
try {
CharSeqItem item = new CharSeqItem();
item.content = str;
item.type = ItemType.CHARSEQ;
q.put(item);
return this;
} catch (InterruptedException ex) {
throw new RuntimeException( ex );
}
}
public ParallelWriter newLine(){
try {
Item item = new Item();
item.type = ItemType.NEWLINE;
q.put(item);
return this;
} catch (InterruptedException ex) {
throw new RuntimeException( ex );
}
}
public void setIndent(int indentation) {
try{
IndentCommand item = new IndentCommand();
item.type = ItemType.INDENT;
item.indent = indentation;
q.put(item);
} catch (InterruptedException ex) {
throw new RuntimeException( ex );
}
}
public void end(){
try {
Item item = new Item();
item.type = ItemType.POISON;
q.put(item);
} catch (InterruptedException ex) {
throw new RuntimeException( ex );
}
}
public void run() {
BufferedWriter out = null;
Item item = null;
try{
out = new BufferedWriter( new FileWriter( file ) );
while( (item = q.take()).type != ItemType.POISON ){
switch( item.type ){
case NEWLINE:
out.newLine();
for( int i = 0; i < indentation; i++ )
out.append(" ");
break;
case INDENT:
indentation = ((IndentCommand)item).indent;
break;
case CHARSEQ:
out.append( ((CharSeqItem)item).content );
}
}
} catch (InterruptedException ex){
throw new RuntimeException( ex );
} catch (IOException ex) {
throw new RuntimeException( ex );
} finally {
if( out != null ) try {
out.close();
} catch (IOException ex) {
throw new RuntimeException( ex );
}
}
}
private enum ItemType {
CHARSEQ, NEWLINE, INDENT, POISON;
}
private static class Item {
ItemType type;
}
private static class CharSeqItem extends Item {
CharSequence content;
}
private static class IndentCommand extends Item {
int indent;
}
}
And then I use it by doing:
ParallelWriter w = new ParallelWriter( myFile );
new Thread(w).start();
/// Lots of
w.append(" things ").newLine();
w.setIndent(2);
w.newLine().append(" more things ");
/// and finally
w.end();
While this works perfectly well, I'm wondering:
Is there a better way to accomplish this?

Your basic approach looks fine. I would structure the code as follows:
import java.io.BufferedWriter;
import java.io.File;
import java.io.IOException;
import java.io.Writer;
import java.util.concurrent.BlockingQueue;
import java.util.concurrent.LinkedBlockingQueue;
import java.util.concurrent.TimeUnit;
public interface FileWriter {
FileWriter append(CharSequence seq);
FileWriter indent(int indent);
void close();
}
class AsyncFileWriter implements FileWriter, Runnable {
private final File file;
private final Writer out;
private final BlockingQueue<Item> queue = new LinkedBlockingQueue<Item>();
private volatile boolean started = false;
private volatile boolean stopped = false;
public AsyncFileWriter(File file) throws IOException {
this.file = file;
this.out = new BufferedWriter(new java.io.FileWriter(file));
}
public FileWriter append(CharSequence seq) {
if (!started) {
throw new IllegalStateException("open() call expected before append()");
}
try {
queue.put(new CharSeqItem(seq));
} catch (InterruptedException ignored) {
}
return this;
}
public FileWriter indent(int indent) {
if (!started) {
throw new IllegalStateException("open() call expected before append()");
}
try {
queue.put(new IndentItem(indent));
} catch (InterruptedException ignored) {
}
return this;
}
public void open() {
this.started = true;
new Thread(this).start();
}
public void run() {
while (!stopped) {
try {
Item item = queue.poll(100, TimeUnit.MICROSECONDS);
if (item != null) {
try {
item.write(out);
} catch (IOException logme) {
}
}
} catch (InterruptedException e) {
}
}
try {
out.close();
} catch (IOException ignore) {
}
}
public void close() {
this.stopped = true;
}
private static interface Item {
void write(Writer out) throws IOException;
}
private static class CharSeqItem implements Item {
private final CharSequence sequence;
public CharSeqItem(CharSequence sequence) {
this.sequence = sequence;
}
public void write(Writer out) throws IOException {
out.append(sequence);
}
}
private static class IndentItem implements Item {
private final int indent;
public IndentItem(int indent) {
this.indent = indent;
}
public void write(Writer out) throws IOException {
for (int i = 0; i < indent; i++) {
out.append(" ");
}
}
}
}
If you do not want to write in a separate thread (maybe in a test?), you can have an implementation of FileWriter which calls append on the Writer in the caller thread.

One good way to exchange data with a single consumer thread is to use an Exchanger.
You could use a StringBuilder or ByteBuffer as the buffer to exchange with the background thread. The latency incurred can be around 1 micro-second, doesn't involve creating any objects and which is lower using a BlockingQueue.
From the example which I think is worth repeating here.
class FillAndEmpty {
Exchanger<DataBuffer> exchanger = new Exchanger<DataBuffer>();
DataBuffer initialEmptyBuffer = ... a made-up type
DataBuffer initialFullBuffer = ...
class FillingLoop implements Runnable {
public void run() {
DataBuffer currentBuffer = initialEmptyBuffer;
try {
while (currentBuffer != null) {
addToBuffer(currentBuffer);
if (currentBuffer.isFull())
currentBuffer = exchanger.exchange(currentBuffer);
}
} catch (InterruptedException ex) { ... handle ... }
}
}
class EmptyingLoop implements Runnable {
public void run() {
DataBuffer currentBuffer = initialFullBuffer;
try {
while (currentBuffer != null) {
takeFromBuffer(currentBuffer);
if (currentBuffer.isEmpty())
currentBuffer = exchanger.exchange(currentBuffer);
}
} catch (InterruptedException ex) { ... handle ...}
}
}
void start() {
new Thread(new FillingLoop()).start();
new Thread(new EmptyingLoop()).start();
}
}

Using a LinkedBlockingQueue is a pretty good idea. Not sure I like some of the style of the code... but the principle seems sound.
I would maybe add a capacity to the LinkedBlockingQueue equal to a certain % of your total memory.. say 10,000 items.. this way if your writing is going too slow, your worker threads won't keep adding more work until the heap is blown.

I know that frequent write operations
can slow a program down a lot
Probably not as much as you think, provided you use buffering.