We Keep Coding

Slow performance when wrapping BufferedReader in custom FilterReader

While tinkering around with some I/O stuff, I made an interesting observation: my custom FilterReader seemed to have some unexpected performance overhead. To try and diagnose the issue, I threw together a simple performance test:
import java.io.*;
abstract class Test
{
public final long timeRun(Reader in) throws IOException
{
long start = System.nanoTime();
run(in);
long end = System.nanoTime();
return end - start;
}
protected abstract void run(Reader in) throws IOException;
}
class WrapInFilterTest extends Test
{
private class LetterFilterReader extends FilterReader
{
public LetterFilterReader(Reader in)
{
super(in);
}
#Override
public int read() throws IOException
{
int read;
while ((read = in.read()) != -1)
{
if (Character.isLetter(read))
break;
}
return read;
}
}
#Override
public void run(Reader in) throws IOException
{
try (Reader letterReader = new LetterFilterReader(in))
{
while (letterReader.read() != -1);
}
}
}
class RawReaderTest extends Test
{
#Override
public void run(Reader in) throws IOException
{
while (readLetter(in) != -1);
}
public int readLetter(Reader in) throws IOException
{
int read;
while ((read = in.read()) != -1)
{
if (Character.isLetter(read))
break;
}
return read;
}
}
public class PerformanceTest
{
public static void main(String[] args) throws IOException
{
String filePath = "/path/to/file.txt";
Test[] tests = new Test[] { new WrapInFilterTest(), new RawReaderTest() };
for (Test test : tests)
{
Reader r = new BufferedReader(new FileReader(filePath));
System.out.println(test.timeRun(r) + "ns");
}
}
}
In general, I've found that the custom filter approach can be as much as 3x slower than straight up reading from the buffered reader. However, it seems dependent on the file content. For example, if a file contains strictly letters, I've found that the custom filter approach actually performs marginally faster! What's going on?

Pipe Broken with PipeInputStream with kubernetes-client exec()

I'm using the kubernetes-client to try copy a directory from a pod, but I'm doing something wrong with the input stream from stdout. I get a java.io.IOException: Pipe broken exception when it tries to read(). I'm pretty sure that no data flows at all. I'm half wondering if I need to read the InputStream on a separate thread or something?
The stream is created like this:
public InputStream copyFiles(String containerId,
String folderName) {
ExecWatch exec = client.pods().withName(containerId).redirectingOutput().exec("tar -C " + folderName + " -c");
// We need to wrap the InputStream so that when the stdout is closed, then the underlying ExecWatch is closed
// also. This will cleanup any Websockets connections.
ChainedCloseInputStreamWrapper inputStreamWrapper = new ChainedCloseInputStreamWrapper(exec.getOutput(), exec);
return inputStreamWrapper;
}
And the InputStream is processed in this function
void copyVideos(final String containerId) {
TarArchiveInputStream tarStream = new TarArchiveInputStream(containerClient.copyFiles(containerId, "/videos/"));
TarArchiveEntry entry;
boolean videoWasCopied = false;
try {
while ((entry = tarStream.getNextTarEntry()) != null) {
if (entry.isDirectory()) {
continue;
}
String fileExtension = entry.getName().substring(entry.getName().lastIndexOf('.'));
testInformation.setFileExtension(fileExtension);
File videoFile = new File(testInformation.getVideoFolderPath(), testInformation.getFileName());
File parent = videoFile.getParentFile();
if (!parent.exists()) {
parent.mkdirs();
}
OutputStream outputStream = new FileOutputStream(videoFile);
IOUtils.copy(tarStream, outputStream);
outputStream.close();
videoWasCopied = true;
LOGGER.log(Level.INFO, "{0} Video file copied to: {1}/{2}", new Object[]{getId(),
testInformation.getVideoFolderPath(), testInformation.getFileName()});
}
} catch (IOException e) {
LOGGER.log(Level.WARNING, getId() + " Error while copying the video", e);
ga.trackException(e);
} finally {
if (!videoWasCopied) {
testInformation.setVideoRecorded(false);
}
}
}
The InputStream Wrapper class is just there to close the ExecWatch at the end once the InputStream is closed, it looks like this:
private static class ChainedCloseInputStreamWrapper extends InputStream {
private InputStream delegate;
private Closeable resourceToClose;
public ChainedCloseInputStreamWrapper(InputStream delegate, Closeable resourceToClose) {
this.delegate = delegate;
this.resourceToClose = resourceToClose;
}
#Override
public int read() throws IOException {
return delegate.read();
}
public int available() throws IOException {
return delegate.available();
}
public void close() throws IOException {
logger.info("Shutdown called!");
delegate.close();
// Close our dependent resource
resourceToClose.close();
}
public boolean equals(Object o) {
return delegate.equals(o);
}
public int hashCode() {
return delegate.hashCode();
}
public int read(byte[] array) throws IOException {
return delegate.read(array);
}
public int read(byte[] array,
int n,
int n2) throws IOException {
return delegate.read(array, n, n2);
}
public long skip(long n) throws IOException {
return delegate.skip(n);
}
public void mark(int n) {
delegate.mark(n);
}
public void reset() throws IOException {
delegate.reset();
}
public boolean markSupported() {
return delegate.markSupported();
}
public String toString() {
return delegate.toString();
}
}

Turns out I had the tar command wrong, so it was causing a failure and the stdout PipeInputStream was dead locking. I managed to find a workaround for the deadlock. But the main reason for the failure was that I forgot to tell tar to actually do something! I at least needed a "." to include the current directory.

Apache Util.copyStream: Stop Stream

Is it possible to stop the bytesTransferred stream for the Apache Util.copyStream function?
long bytesTransferred = Util.copyStream(inputStream, outputStream, 32768, CopyStreamEvent.UNKNOWN_STREAM_SIZE, new CopyStreamListener() {
#Override
public void bytesTransferred(CopyStreamEvent event) {
bytesTransferred(event.getTotalBytesTransferred(), event.getBytesTransferred(), event.getStreamSize());
}
#Override
public void bytesTransferred(long totalBytesTransferred, int bytesTransferred,
long streamSize) {
try {
if(true) {
log.info('Stopping');
return; //Cancel
} else {
log.info('Still going');
}
} catch (InterruptedException e) {
// this should not happen!
}
}
});
In this case, what will happen is that I keep getting a Stopping message in my logs. I also tried throwing a new RuntileException instead of returning, and again I get endless Stopping messages. How would I cancel the bytesTransfered in this case?

You could try wrapping the input stream, and overriding the read methods to check for a stop flag. If set, throw an IOException. Example class.
/**
* Wrapped input stream that can be cancelled.
*/
public class WrappedStoppableInputStream extends InputStream
{
private InputStream m_wrappedInputStream;
private boolean m_stop = false;
/**
* Constructor.
* #param inputStream original input stream
*/
public WrappedStoppableInputStream(InputStream inputStream)
{
m_wrappedInputStream = inputStream;
}
/**
* Call to stop reading stream.
*/
public void cancelTransfer()
{
m_stop = true;
}
#Override
public int read() throws IOException
{
if (m_stop)
{
throw new IOException("Stopping stream");
}
return m_wrappedInputStream.read();
}
#Override
public int read(byte[] b) throws IOException
{
if (m_stop)
{
throw new IOException("Stopping stream");
}
return m_wrappedInputStream.read(b);
}
#Override
public int read(byte[] b, int off, int len) throws IOException
{
if (m_stop)
{
throw new IOException("Stopping stream");
}
return m_wrappedInputStream.read(b, off, len);
}
}
I am assuming that the file copying is running inside a thread. So you wrap your input stream with WrappedStoppableInputStream, and pass that to your copy function, to be used instead of the original input stream.

InputStream to servletInputStream

I have this InputStream:
InputStream inputStream = new ByteArrayInputStream(myString.getBytes(StandardCharsets.UTF_8));
How can I convert this to ServletInputStream?
I have tried:
ServletInputStream servletInputStream = (ServletInputStream) inputStream;
but do not work.
EDIT:
My method is this:
private static class LowerCaseRequest extends HttpServletRequestWrapper {
public LowerCaseRequest(final HttpServletRequest request) throws IOException, ServletException {
super(request);
}
#Override
public ServletInputStream getInputStream() throws IOException {
ServletInputStream servletInputStream;
StringBuilder jb = new StringBuilder();
String line;
String toLowerCase = "";
BufferedReader reader = new BufferedReader(new InputStreamReader(super.getInputStream()));
while ((line = reader.readLine()) != null) {
toLowerCase = jb.append(line).toString().toLowerCase();
}
InputStream inputStream = new ByteArrayInputStream(toLowerCase.getBytes(StandardCharsets.UTF_8));
servletInputStream = (ServletInputStream) inputStream;
return servletInputStream;
}
}
I´m trying to convert all my request to lowercase.

My advice: don't create the ByteArrayInputStream, just use the byte array you got from the getBytes method already. This should be enough to create a ServletInputStream.
Most basic solution
Unfortunately, aksappy's answer only overrides the read method. While this may be enough in Servlet API 3.0 and below, in the later versions of Servlet API there are three more methods you have to implement.
Here is my implementation of the class, although with it becoming quite long (due to the new methods introduced in Servlet API 3.1), you might want to think about factoring it out into a nested or even top-level class.
final byte[] myBytes = myString.getBytes("UTF-8");
ServletInputStream servletInputStream = new ServletInputStream() {
private int lastIndexRetrieved = -1;
private ReadListener readListener = null;
#Override
public boolean isFinished() {
return (lastIndexRetrieved == myBytes.length-1);
}
#Override
public boolean isReady() {
// This implementation will never block
// We also never need to call the readListener from this method, as this method will never return false
return isFinished();
}
#Override
public void setReadListener(ReadListener readListener) {
this.readListener = readListener;
if (!isFinished()) {
try {
readListener.onDataAvailable();
} catch (IOException e) {
readListener.onError(e);
}
} else {
try {
readListener.onAllDataRead();
} catch (IOException e) {
readListener.onError(e);
}
}
}
#Override
public int read() throws IOException {
int i;
if (!isFinished()) {
i = myBytes[lastIndexRetrieved+1];
lastIndexRetrieved++;
if (isFinished() && (readListener != null)) {
try {
readListener.onAllDataRead();
} catch (IOException ex) {
readListener.onError(ex);
throw ex;
}
}
return i;
} else {
return -1;
}
}
};
Adding expected methods
Depending on your requirements, you may also want to override other methods. As romfret pointed out, it's advisable to override some methods, such as close and available. If you don't implement them, the stream will always report that there are 0 bytes available to be read, and the close method will do nothing to affect the state of the stream. You can probably get away without overriding skip, as the default implementation will just call read a number of times.
#Override
public int available() throws IOException {
return (myBytes.length-lastIndexRetrieved-1);
}
#Override
public void close() throws IOException {
lastIndexRetrieved = myBytes.length-1;
}
Writing a better close method
Unfortunately, due to the nature of an anonymous class, it's going to be difficult for you to write an effective close method because as long as one instance of the stream has not been garbage-collected by Java, it maintains a reference to the byte array, even if the stream has been closed.
However, if you factor out the class into a nested or top-level class (or even an anonymous class with a constructor which you call from the line in which it is defined), the myBytes can be a non-final field rather than a final local variable, and you can add a line like:
myBytes = null;
to your close method, which will allow Java to free memory taken up by the byte array.
Of course, this will require you to write a constructor, such as:
private byte[] myBytes;
public StringServletInputStream(String str) {
try {
myBytes = str.getBytes("UTF-8");
} catch (UnsupportedEncodingException e) {
throw new IllegalStateException("JVM did not support UTF-8", e);
}
}
Mark and Reset
You may also want to override mark, markSupported and reset if you want to support mark/reset. I am not sure if they are ever actually called by your container though.
private int readLimit = -1;
private int markedPosition = -1;
#Override
public boolean markSupported() {
return true;
}
#Override
public synchronized void mark(int readLimit) {
this.readLimit = readLimit;
this.markedPosition = lastIndexRetrieved;
}
#Override
public synchronized void reset() throws IOException {
if (markedPosition == -1) {
throw new IOException("No mark found");
} else {
lastIndexRetrieved = markedPosition;
readLimit = -1;
}
}
// Replacement of earlier read method to cope with readLimit
#Override
public int read() throws IOException {
int i;
if (!isFinished()) {
i = myBytes[lastIndexRetrieved+1];
lastIndexRetrieved++;
if (isFinished() && (readListener != null)) {
try {
readListener.onAllDataRead();
} catch (IOException ex) {
readListener.onError(ex);
throw ex;
}
readLimit = -1;
}
if (readLimit != -1) {
if ((lastIndexRetrieved - markedPosition) > readLimit) {
// This part is actually not necessary in our implementation
// as we are not storing any data. However we need to respect
// the contract.
markedPosition = -1;
readLimit = -1;
}
}
return i;
} else {
return -1;
}
}

Try this code.
ByteArrayInputStream byteArrayInputStream = new ByteArrayInputStream(myString.getBytes(StandardCharsets.UTF_8));
ServletInputStream servletInputStream=new ServletInputStream(){
public int read() throws IOException {
return byteArrayInputStream.read();
}
}

You can only cast something like this:
ServletInputStream servletInputStream = (ServletInputStream) inputStream;
if the inputStream you are trying to cast is actually a ServletInputStream already. It will complain if it's some other implementation of InputStream. You can't cast an object to something it isn't.
In a Servlet container, you can get a ServletInputStream from a ServletRequest:
ServletInputStream servletInputStream = request.getInputStream();
So, what are you actually trying to do?
EDIT
I'm intrigued as to why you want to convert your request to lower-case - why not just make your servlet case-insensitive? In other words, your code to lower-case the request data can be copied into your servlet, then it can process it there... always look for the simplest solution!

Is this LimitedInputStream correct?

I've written a class called LimitedInputStream. It wraps around an existing input stream to limit the number of bytes read from it to a specified length. It's meant as an alternative to:
byte[] data = readAll(length);
InputStream ins = new ByteArrayInputStream(data);
Which requires the extra buffer.
This is the class:
public static class LimitedInputStream extends InputStream {
private final InputStream ins;
private int left;
private int mark = -1;
public LimitedInputStream(InputStream ins, int limit) {
this.ins = ins;
left = limit;
}
public void skipRest() throws IOException {
ByteStreams.skipFully(ins, left);
left = 0;
}
#Override
public int read() throws IOException {
if (left == 0) return -1;
final int read = ins.read();
if (read > 0) left--;
return read;
}
#Override
public int read(byte[] b, int off, int len) throws IOException {
if (left == 0) return -1;
if (len > left) len = left;
final int read = ins.read(b, off, len);
if (read > 0) left -= read;
return read;
}
#Override
public int available() throws IOException {
final int a = ins.available();
return a > left ? left : a;
}
#Override
public void mark(int readlimit) {
ins.mark(readlimit);
mark = left;
}
#Override
public void reset() throws IOException {
if (!ins.markSupported()) throw new IOException("Mark not supported");
if (mark == -1) throw new IOException("Mark not set");
ins.reset();
left = mark;
}
#Override
public long skip(long n) throws IOException {
if (n > left) n = left;
long skipped = ins.skip(n);
left -= skipped;
return skipped;
}
}
Use case:
Object readObj() throws IOException {
int len = readInt();
final LimitedInputStream lis = new LimitedInputStream(this, len);
try {
return deserialize(new CompactInputStream(lis));
} finally {
lis.skipRest();
}
}
for (something) {
Object obj;
try {
obj = readObj();
} catch (Exception e) {
obj = null;
}
list.add(obj);
}
Could you code review my class for any serious bugs, e.g. possible mistakes in updating left?

Guava includes a LimitInputStream, so you may want to just use that.