We Keep Coding

Console input no longer being echo'd after changing stdout

In my program I have changed stdout using System.setOut(); but after that, typing into the console no longer echoes the input back.
I tried the following, with no avail:
originalStdIn = System.in;
stdIn = new InputStream() {
#Override
public int available() throws IOException {
return originalStdIn.available();
}
#Override
public int read() throws IOException {
int chr = originalStdIn.read();
stdOut.write(chr);
return chr;
}
#Override
public int read(byte[] buffer) throws IOException {
return this.read(buffer, 0, buffer.length);
}
#Override
public int read(byte[] buffer, int off, int len) throws IOException {
int readResult = originalStdIn.read(buffer, off, len);
stdOut.write(buffer, off, readResult);
return readResult;
}
};
System.setIn(stdIn);
This is how I read input:
private void handleThread_func() {
Logging.didUpdateInWindow = true;
System.out.print(this.consoleInputPrefix);
while (!this.isClosed) {
try {
if (System.in.available() > 0) {
String input = "";
while (System.in.available() > 0) {
input += this.inputReader.readLine();
}
this.handleInput(input);
Logging.didUpdateInWindow = true;
System.out.print(this.consoleInputPrefix);
}
Thread.sleep(1);
} catch (Exception ex) {
if (ex instanceof InterruptedException) break;
Logging.logSevere("Unable to handle console input: " +
Utils.getExceptionStackTraceAsStr(ex));
}
}
}
I tried changing input += this.inputReader.readLine(); to input += (char)this.inputReader.read() but while it did echo, the input was read in the incorrect order

Slow performance when wrapping BufferedReader in custom FilterReader

While tinkering around with some I/O stuff, I made an interesting observation: my custom FilterReader seemed to have some unexpected performance overhead. To try and diagnose the issue, I threw together a simple performance test:
import java.io.*;
abstract class Test
{
public final long timeRun(Reader in) throws IOException
{
long start = System.nanoTime();
run(in);
long end = System.nanoTime();
return end - start;
}
protected abstract void run(Reader in) throws IOException;
}
class WrapInFilterTest extends Test
{
private class LetterFilterReader extends FilterReader
{
public LetterFilterReader(Reader in)
{
super(in);
}
#Override
public int read() throws IOException
{
int read;
while ((read = in.read()) != -1)
{
if (Character.isLetter(read))
break;
}
return read;
}
}
#Override
public void run(Reader in) throws IOException
{
try (Reader letterReader = new LetterFilterReader(in))
{
while (letterReader.read() != -1);
}
}
}
class RawReaderTest extends Test
{
#Override
public void run(Reader in) throws IOException
{
while (readLetter(in) != -1);
}
public int readLetter(Reader in) throws IOException
{
int read;
while ((read = in.read()) != -1)
{
if (Character.isLetter(read))
break;
}
return read;
}
}
public class PerformanceTest
{
public static void main(String[] args) throws IOException
{
String filePath = "/path/to/file.txt";
Test[] tests = new Test[] { new WrapInFilterTest(), new RawReaderTest() };
for (Test test : tests)
{
Reader r = new BufferedReader(new FileReader(filePath));
System.out.println(test.timeRun(r) + "ns");
}
}
}
In general, I've found that the custom filter approach can be as much as 3x slower than straight up reading from the buffered reader. However, it seems dependent on the file content. For example, if a file contains strictly letters, I've found that the custom filter approach actually performs marginally faster! What's going on?

Parquet Writer to buffer or byte stream

I have a java application that converts json messages to parquet format. Is there any parquet writer which writes to buffer or byte stream in java? Most of the examples, I have seen write to files.

TLDR; you will need to implement OutputFile, e.g. something along the line of:
import org.apache.parquet.io.OutputFile;
import org.apache.parquet.io.PositionOutputStream;
import java.io.BufferedOutputStream;
import java.io.IOException;
public class ParquetBufferedWriter implements OutputFile {
private final BufferedOutputStream out;
public ParquetBufferedWriter(BufferedOutputStream out) {
this.out = out;
}
#Override
public PositionOutputStream create(long blockSizeHint) throws IOException {
return createPositionOutputstream();
}
private PositionOutputStream createPositionOutputstream() {
return new PositionOutputStream() {
#Override
public long getPos() throws IOException {
return 0;
}
#Override
public void write(int b) throws IOException {
out.write(b);
}
};
}
#Override
public PositionOutputStream createOrOverwrite(long blockSizeHint) throws IOException {
return createPositionOutputstream();
}
#Override
public boolean supportsBlockSize() {
return false;
}
#Override
public long defaultBlockSize() {
return 0;
}
}
And your writer would be something like:
ParquetBufferedWriter out = new ParquetBufferedWriter();
try (ParquetWriter<Record> writer = AvroParquetWriter.
<Record>builder(out)
.withRowGroupSize(DEFAULT_BLOCK_SIZE)
.withPageSize(DEFAULT_PAGE_SIZE)
.withSchema(SCHEMA)
.build()) {
for (Record record : records) {
writer.write(record);
}
} catch (IOException e) {
throw new IllegalStateException(e);
}

I just also needed to write to a stream, so I completed the example given by naimdjon. The following works perfectly fine for me.
class ParquetBufferedWriter implements OutputFile {
private final BufferedOutputStream out;
public ParquetBufferedWriter(BufferedOutputStream out) {
this.out = out;
}
#Override
public PositionOutputStream create(long blockSizeHint) throws IOException {
return createPositionOutputstream();
}
private PositionOutputStream createPositionOutputstream() {
return new PositionOutputStream() {
int pos = 0;
#Override
public long getPos() throws IOException {
return pos;
}
#Override
public void flush() throws IOException {
out.flush();
};
#Override
public void close() throws IOException {
out.close();
};
#Override
public void write(int b) throws IOException {
out.write(b);
pos++;
}
#Override
public void write(byte[] b, int off, int len) throws IOException {
out.write(b, off, len);
pos += len;
}
};
}
#Override
public PositionOutputStream createOrOverwrite(long blockSizeHint) throws IOException {
return createPositionOutputstream();
}
#Override
public boolean supportsBlockSize() {
return false;
}
#Override
public long defaultBlockSize() {
return 0;
}
}

You need to write the data into temp file and then covert the data from file to input stream or buffer
something like this, first read the tempfile data
final InputStream targetStream = new DataInputStream(new FileInputStream(tmp1.getAbsoluteFile()));
StringWriter writer = new StringWriter();
String encoding = StandardCharsets.UTF_8.name();
IOUtils.copy(targetStream, writer, encoding);
System.out.println(writer);

FileOutputStream

public FileOutputStream(String name) throws FileNotFoundException {
this(name != null ? new File(name) : null, false);
}
public FileOutputStream(String name, boolean append)
throws FileNotFoundException
{
this(name != null ? new File(name) : null, append);
}
public FileOutputStream(File file) throws FileNotFoundException {
this(file, false);
}
public FileOutputStream(File file, boolean append)
throws FileNotFoundException
{
String name = (file != null ? file.getPath() : null);
SecurityManager security = System.getSecurityManager();
if (security != null) {
security.checkWrite(name);
}
if (name == null) {
throw new NullPointerException();
}
if (file.isInvalid()) {
throw new FileNotFoundException("Invalid file path");
}
this.fd = new FileDescriptor();
this.append = append;
this.path = name;
fd.incrementAndGetUseCount();
open(name, append);
}
public FileOutputStream(FileDescriptor fdObj) {
SecurityManager security = System.getSecurityManager();
if (fdObj == null) {
throw new NullPointerException();
}
if (security != null) {
security.checkWrite(fdObj);
}
this.fd = fdObj;
this.path = null;
this.append = false;
fd.incrementAndGetUseCount();
}
vate native void open(String name, boolean append)
throws FileNotFoundException;
private native void write(int b, boolean append) throws IOException;
public void write(int b) throws IOException {
Object traceContext = IoTrace.fileWriteBegin(path);
int bytesWritten = 0;
try {
write(b, append);
bytesWritten = 1;
} finally {
IoTrace.fileWriteEnd(traceContext, bytesWritten);
}
}
private native void writeBytes(byte b[], int off, int len, boolean append)
throws IOException;
public void write(byte b[]) throws IOException {
Object traceContext = IoTrace.fileWriteBegin(path);
int bytesWritten = 0;
try {
writeBytes(b, 0, b.length, append);
bytesWritten = b.length;
} finally {
IoTrace.fileWriteEnd(traceContext, bytesWritten);
}
}
public void close() throws IOException {
synchronized (closeLock) {
if (closed) {
return;
}
closed = true;
}
if (channel != null) {
nt useCount = fd.decrementAndGetUseCount();
if ((useCount <= 0) || !isRunningFinalize()) {
close0();
}
}
public final FileDescriptor getFD() throws IOException {
if (fd != null) return fd;
throw new IOException();
public FileChannel getChannel() {
synchronized (this) {
if (channel == null) {
channel = FileChannelImpl.open(fd, path, false, true, append, this);
fd.incrementAndGetUseCount();
}
return channel;
}
}
protected void finalize() throws IOException {
if (fd != null) {
if (fd == FileDescriptor.out || fd == FileDescriptor.err) {
flush();
} else {
runningFinalize.set(Boolean.TRUE);
try {
close();
} finally {
runningFinalize.set(Boolean.FALSE);
}
}
private native void close0() throws IOException;
private static native void initIDs();
static {
initIDs();
}
Stack Trace
java.io.FileNotFoundException: D:\webs\Softparam2\upload\1\1429175417820\cfm\CFM_test.xlsx (Le périphérique n’est pas prêt)
at java.io.FileOutputStream.open(Native Method)
at java.io.FileOutputStream.<init>(FileOutputStream.java:221)
at java.io.FileOutputStream.<init>(FileOutput)

FileNotFoundException is typically thrown when a file does not exist. I'd suggest you check the filepath.
As a general rule, it is a bad idea to use absolute filepaths. Instead, place the file required in the current working folder (can be found with System.getProperty("user.dir");) and simply call: FileInputStream("CFM_test.xlsx");

Is this LimitedInputStream correct?

I've written a class called LimitedInputStream. It wraps around an existing input stream to limit the number of bytes read from it to a specified length. It's meant as an alternative to:
byte[] data = readAll(length);
InputStream ins = new ByteArrayInputStream(data);
Which requires the extra buffer.
This is the class:
public static class LimitedInputStream extends InputStream {
private final InputStream ins;
private int left;
private int mark = -1;
public LimitedInputStream(InputStream ins, int limit) {
this.ins = ins;
left = limit;
}
public void skipRest() throws IOException {
ByteStreams.skipFully(ins, left);
left = 0;
}
#Override
public int read() throws IOException {
if (left == 0) return -1;
final int read = ins.read();
if (read > 0) left--;
return read;
}
#Override
public int read(byte[] b, int off, int len) throws IOException {
if (left == 0) return -1;
if (len > left) len = left;
final int read = ins.read(b, off, len);
if (read > 0) left -= read;
return read;
}
#Override
public int available() throws IOException {
final int a = ins.available();
return a > left ? left : a;
}
#Override
public void mark(int readlimit) {
ins.mark(readlimit);
mark = left;
}
#Override
public void reset() throws IOException {
if (!ins.markSupported()) throw new IOException("Mark not supported");
if (mark == -1) throw new IOException("Mark not set");
ins.reset();
left = mark;
}
#Override
public long skip(long n) throws IOException {
if (n > left) n = left;
long skipped = ins.skip(n);
left -= skipped;
return skipped;
}
}
Use case:
Object readObj() throws IOException {
int len = readInt();
final LimitedInputStream lis = new LimitedInputStream(this, len);
try {
return deserialize(new CompactInputStream(lis));
} finally {
lis.skipRest();
}
}
for (something) {
Object obj;
try {
obj = readObj();
} catch (Exception e) {
obj = null;
}
list.add(obj);
}
Could you code review my class for any serious bugs, e.g. possible mistakes in updating left?

Guava includes a LimitInputStream, so you may want to just use that.