We Keep Coding

Pipe Broken with PipeInputStream with kubernetes-client exec()

I'm using the kubernetes-client to try copy a directory from a pod, but I'm doing something wrong with the input stream from stdout. I get a java.io.IOException: Pipe broken exception when it tries to read(). I'm pretty sure that no data flows at all. I'm half wondering if I need to read the InputStream on a separate thread or something?
The stream is created like this:
public InputStream copyFiles(String containerId,
String folderName) {
ExecWatch exec = client.pods().withName(containerId).redirectingOutput().exec("tar -C " + folderName + " -c");
// We need to wrap the InputStream so that when the stdout is closed, then the underlying ExecWatch is closed
// also. This will cleanup any Websockets connections.
ChainedCloseInputStreamWrapper inputStreamWrapper = new ChainedCloseInputStreamWrapper(exec.getOutput(), exec);
return inputStreamWrapper;
}
And the InputStream is processed in this function
void copyVideos(final String containerId) {
TarArchiveInputStream tarStream = new TarArchiveInputStream(containerClient.copyFiles(containerId, "/videos/"));
TarArchiveEntry entry;
boolean videoWasCopied = false;
try {
while ((entry = tarStream.getNextTarEntry()) != null) {
if (entry.isDirectory()) {
continue;
}
String fileExtension = entry.getName().substring(entry.getName().lastIndexOf('.'));
testInformation.setFileExtension(fileExtension);
File videoFile = new File(testInformation.getVideoFolderPath(), testInformation.getFileName());
File parent = videoFile.getParentFile();
if (!parent.exists()) {
parent.mkdirs();
}
OutputStream outputStream = new FileOutputStream(videoFile);
IOUtils.copy(tarStream, outputStream);
outputStream.close();
videoWasCopied = true;
LOGGER.log(Level.INFO, "{0} Video file copied to: {1}/{2}", new Object[]{getId(),
testInformation.getVideoFolderPath(), testInformation.getFileName()});
}
} catch (IOException e) {
LOGGER.log(Level.WARNING, getId() + " Error while copying the video", e);
ga.trackException(e);
} finally {
if (!videoWasCopied) {
testInformation.setVideoRecorded(false);
}
}
}
The InputStream Wrapper class is just there to close the ExecWatch at the end once the InputStream is closed, it looks like this:
private static class ChainedCloseInputStreamWrapper extends InputStream {
private InputStream delegate;
private Closeable resourceToClose;
public ChainedCloseInputStreamWrapper(InputStream delegate, Closeable resourceToClose) {
this.delegate = delegate;
this.resourceToClose = resourceToClose;
}
#Override
public int read() throws IOException {
return delegate.read();
}
public int available() throws IOException {
return delegate.available();
}
public void close() throws IOException {
logger.info("Shutdown called!");
delegate.close();
// Close our dependent resource
resourceToClose.close();
}
public boolean equals(Object o) {
return delegate.equals(o);
}
public int hashCode() {
return delegate.hashCode();
}
public int read(byte[] array) throws IOException {
return delegate.read(array);
}
public int read(byte[] array,
int n,
int n2) throws IOException {
return delegate.read(array, n, n2);
}
public long skip(long n) throws IOException {
return delegate.skip(n);
}
public void mark(int n) {
delegate.mark(n);
}
public void reset() throws IOException {
delegate.reset();
}
public boolean markSupported() {
return delegate.markSupported();
}
public String toString() {
return delegate.toString();
}
}

Turns out I had the tar command wrong, so it was causing a failure and the stdout PipeInputStream was dead locking. I managed to find a workaround for the deadlock. But the main reason for the failure was that I forgot to tell tar to actually do something! I at least needed a "." to include the current directory.

Timeout-based BufferedWriter flush

I'm using BufferedWriter with the default size of 8192 characters to write lines to a local file. The lines are read from socket inputstream using BufferedReader readLine method, blocking I/O.
Average line length is 50 characters. It all works well and fast enough (over 1 mln lines per second) however if the client stops writing, lines that are currently stored in BufferedWriter buffer won't be flushed to disk. In fact the buffered characters won't be flushed to disk until the client resumes writing or the connection is closed. This translates into a delay between the time line is transmitted by client and the time this line is committed to file, so long-tail latency goes up.
Is there a way to flush incomplete BufferedWriter buffer on timeout, e.g. within 100 milliseconds?

What about something like this? It's not a real BufferedWriter, but it's a Writer. It works by periodically checking on on the last writer to the underlying, hopefully unbuffered writer, then flushing the BufferedWriter if it's been longer than the timeout.
public class PeriodicFlushingBufferedWriter extends Writer {
protected final MonitoredWriter monitoredWriter;
protected final BufferedWriter writer;
protected final long timeout;
protected final Thread thread;
public PeriodicFlushingBufferedWriter(Writer out, long timeout) {
this(out, 8192, timeout);
}
public PeriodicFlushingBufferedWriter(Writer out, int sz, final long timeout) {
monitoredWriter = new MonitoredWriter(out);
writer = new BufferedWriter(monitoredWriter, sz);
this.timeout = timeout;
thread = new Thread(new Runnable() {
#Override
public void run() {
long deadline = System.currentTimeMillis() + timeout;
while (!Thread.interrupted()) {
try {
Thread.sleep(Math.max(deadline - System.currentTimeMillis(), 0));
} catch (InterruptedException e) {
return;
}
synchronized (PeriodicFlushingBufferedWriter.this) {
if (Thread.interrupted()) {
return;
}
long lastWrite = monitoredWriter.getLastWrite();
if (System.currentTimeMillis() - lastWrite >= timeout) {
try {
writer.flush();
} catch (IOException e) {
}
}
deadline = lastWrite + timeout;
}
}
}
});
thread.start();
}
#Override
public synchronized void write(char[] cbuf, int off, int len) throws IOException {
this.writer.write(cbuf, off, len);
}
#Override
public synchronized void flush() throws IOException {
this.writer.flush();
}
#Override
public synchronized void close() throws IOException {
try {
thread.interrupt();
} finally {
this.writer.close();
}
}
private static class MonitoredWriter extends FilterWriter {
protected final AtomicLong lastWrite = new AtomicLong();
protected MonitoredWriter(Writer out) {
super(out);
}
#Override
public void write(int c) throws IOException {
lastWrite.set(System.currentTimeMillis());
super.write(c);
}
#Override
public void write(char[] cbuf, int off, int len) throws IOException {
lastWrite.set(System.currentTimeMillis());
super.write(cbuf, off, len);
}
#Override
public void write(String str, int off, int len) throws IOException {
lastWrite.set(System.currentTimeMillis());
super.write(str, off, len);
}
#Override
public void flush() throws IOException {
lastWrite.set(System.currentTimeMillis());
super.flush();
}
public long getLastWrite() {
return this.lastWrite.get();
}
}
}

#copeg is right - flush it after every line. It is easy to flush it at time period but what is the sense to have only half record and not be able to proceed it?

You might apply Observer, Manager, and Factory patterns here and have a central BufferedWriterManager produce your BufferedWriters and maintain a list of active instances. An internal thread might wake periodically and flush the active instances. This might also be an opportunity for Weak references so there is no requirement for your consumers to explicitly free the object. Instead, the GC will do the work and your Manager simply needs to handle the case when its internal reference becomes null (i.e. when all strong references are dropped).

Don't try this complex scheme, it's too hard. Just reduce the size of the buffer, by specifying it when constructing the BufferedWriter. Reduce it till you find the balance between performance and latency that you need.

InputStream to servletInputStream

I have this InputStream:
InputStream inputStream = new ByteArrayInputStream(myString.getBytes(StandardCharsets.UTF_8));
How can I convert this to ServletInputStream?
I have tried:
ServletInputStream servletInputStream = (ServletInputStream) inputStream;
but do not work.
EDIT:
My method is this:
private static class LowerCaseRequest extends HttpServletRequestWrapper {
public LowerCaseRequest(final HttpServletRequest request) throws IOException, ServletException {
super(request);
}
#Override
public ServletInputStream getInputStream() throws IOException {
ServletInputStream servletInputStream;
StringBuilder jb = new StringBuilder();
String line;
String toLowerCase = "";
BufferedReader reader = new BufferedReader(new InputStreamReader(super.getInputStream()));
while ((line = reader.readLine()) != null) {
toLowerCase = jb.append(line).toString().toLowerCase();
}
InputStream inputStream = new ByteArrayInputStream(toLowerCase.getBytes(StandardCharsets.UTF_8));
servletInputStream = (ServletInputStream) inputStream;
return servletInputStream;
}
}
I´m trying to convert all my request to lowercase.

My advice: don't create the ByteArrayInputStream, just use the byte array you got from the getBytes method already. This should be enough to create a ServletInputStream.
Most basic solution
Unfortunately, aksappy's answer only overrides the read method. While this may be enough in Servlet API 3.0 and below, in the later versions of Servlet API there are three more methods you have to implement.
Here is my implementation of the class, although with it becoming quite long (due to the new methods introduced in Servlet API 3.1), you might want to think about factoring it out into a nested or even top-level class.
final byte[] myBytes = myString.getBytes("UTF-8");
ServletInputStream servletInputStream = new ServletInputStream() {
private int lastIndexRetrieved = -1;
private ReadListener readListener = null;
#Override
public boolean isFinished() {
return (lastIndexRetrieved == myBytes.length-1);
}
#Override
public boolean isReady() {
// This implementation will never block
// We also never need to call the readListener from this method, as this method will never return false
return isFinished();
}
#Override
public void setReadListener(ReadListener readListener) {
this.readListener = readListener;
if (!isFinished()) {
try {
readListener.onDataAvailable();
} catch (IOException e) {
readListener.onError(e);
}
} else {
try {
readListener.onAllDataRead();
} catch (IOException e) {
readListener.onError(e);
}
}
}
#Override
public int read() throws IOException {
int i;
if (!isFinished()) {
i = myBytes[lastIndexRetrieved+1];
lastIndexRetrieved++;
if (isFinished() && (readListener != null)) {
try {
readListener.onAllDataRead();
} catch (IOException ex) {
readListener.onError(ex);
throw ex;
}
}
return i;
} else {
return -1;
}
}
};
Adding expected methods
Depending on your requirements, you may also want to override other methods. As romfret pointed out, it's advisable to override some methods, such as close and available. If you don't implement them, the stream will always report that there are 0 bytes available to be read, and the close method will do nothing to affect the state of the stream. You can probably get away without overriding skip, as the default implementation will just call read a number of times.
#Override
public int available() throws IOException {
return (myBytes.length-lastIndexRetrieved-1);
}
#Override
public void close() throws IOException {
lastIndexRetrieved = myBytes.length-1;
}
Writing a better close method
Unfortunately, due to the nature of an anonymous class, it's going to be difficult for you to write an effective close method because as long as one instance of the stream has not been garbage-collected by Java, it maintains a reference to the byte array, even if the stream has been closed.
However, if you factor out the class into a nested or top-level class (or even an anonymous class with a constructor which you call from the line in which it is defined), the myBytes can be a non-final field rather than a final local variable, and you can add a line like:
myBytes = null;
to your close method, which will allow Java to free memory taken up by the byte array.
Of course, this will require you to write a constructor, such as:
private byte[] myBytes;
public StringServletInputStream(String str) {
try {
myBytes = str.getBytes("UTF-8");
} catch (UnsupportedEncodingException e) {
throw new IllegalStateException("JVM did not support UTF-8", e);
}
}
Mark and Reset
You may also want to override mark, markSupported and reset if you want to support mark/reset. I am not sure if they are ever actually called by your container though.
private int readLimit = -1;
private int markedPosition = -1;
#Override
public boolean markSupported() {
return true;
}
#Override
public synchronized void mark(int readLimit) {
this.readLimit = readLimit;
this.markedPosition = lastIndexRetrieved;
}
#Override
public synchronized void reset() throws IOException {
if (markedPosition == -1) {
throw new IOException("No mark found");
} else {
lastIndexRetrieved = markedPosition;
readLimit = -1;
}
}
// Replacement of earlier read method to cope with readLimit
#Override
public int read() throws IOException {
int i;
if (!isFinished()) {
i = myBytes[lastIndexRetrieved+1];
lastIndexRetrieved++;
if (isFinished() && (readListener != null)) {
try {
readListener.onAllDataRead();
} catch (IOException ex) {
readListener.onError(ex);
throw ex;
}
readLimit = -1;
}
if (readLimit != -1) {
if ((lastIndexRetrieved - markedPosition) > readLimit) {
// This part is actually not necessary in our implementation
// as we are not storing any data. However we need to respect
// the contract.
markedPosition = -1;
readLimit = -1;
}
}
return i;
} else {
return -1;
}
}

Try this code.
ByteArrayInputStream byteArrayInputStream = new ByteArrayInputStream(myString.getBytes(StandardCharsets.UTF_8));
ServletInputStream servletInputStream=new ServletInputStream(){
public int read() throws IOException {
return byteArrayInputStream.read();
}
}

You can only cast something like this:
ServletInputStream servletInputStream = (ServletInputStream) inputStream;
if the inputStream you are trying to cast is actually a ServletInputStream already. It will complain if it's some other implementation of InputStream. You can't cast an object to something it isn't.
In a Servlet container, you can get a ServletInputStream from a ServletRequest:
ServletInputStream servletInputStream = request.getInputStream();
So, what are you actually trying to do?
EDIT
I'm intrigued as to why you want to convert your request to lower-case - why not just make your servlet case-insensitive? In other words, your code to lower-case the request data can be copied into your servlet, then it can process it there... always look for the simplest solution!

Java Reader pre & post data

Is there a Reader class (JDK or library) I can use to decorate another Reader in such a way that the new reader returns "PREFIX" + everythong of innerReader + "POSTFIX"?
I want to decorate the file contents with a header and a footer before returning the Reader to the caller.

Not in the standard library, but take a look at http://ostermiller.org/utils/Concat.html
Looks promising, but I haven't used it myself.

I've built this on behalf of GreyBeardedGeek's post, maybe somebody can use it:
/**
* Utility <code>Reader</code> implementation which joins one or more other <code>Reader</code> to appear as one.
*/
public class CompositeReader extends Reader {
/** Logger. */
private final static Logger log = LoggerFactory.getLogger(CompositeReader.class);
/** List of readers (in order). */
private final Reader[] readers;
/** Current index. */
private int index;
/**
* #param readers ordered list of <code>Reader</code> to read from.
*/
public CompositeReader(final Reader... readers) {
checkArgument(readers.length > 0, "Argument readers must not be empty.");
this.readers = readers;
index = 0;
}
#Override
public int read(final char[] cbuf, final int off, final int len) throws IOException {
int read = 0;
while (read < len && index != readers.length) {
final Reader reader = readers[index];
final int readFromReader = reader.read(cbuf, off + read, len - read);
if (readFromReader == -1) {
++index;
} else {
read += readFromReader;
}
}
if (read == 0) {
return -1;
}
return read;
}
#Override
public void close() throws IOException {
IOException firstException = null;
for (final Reader reader : readers) {
try {
reader.close();
} catch (final IOException ex) {
if (firstException != null) {
log.warn("Multiple readers could not be closed, only first exception will be thrown.");
firstException = ex;
}
}
}
if (firstException != null) {
throw firstException;
}
}
}
Here you go :-)

Java - Capturing System.err.println or Capturing a PrintStream

Java Newbie question :
I need to capture the text being written to a printStream by a 3rd party component.
The PrintStream is defaulted to System.err, but can be changed to another PrintStream.
Looking through the docs, I couldn't find an easy way to direct the contents of a PrintStream to a string writer / buffer.
Can someone please assist?

PipedOutputStream pipeOut = new PipedOutputStream();
PipedInputStream pipeIn = new PipedInputStream(pipeOut);
System.setOut(new PrintStream(pipeOut));
// now read from pipeIn

import java.io.*;
public class Test {
public static void main(String[] args) {
FileOutputStream fos = null;
try {
fos = new FileOutputStream("errors.txt");
} catch(IOException ioe) {
System.err.println("redirection not possible: "+ioe);
System.exit(-1);
}
PrintStream ps = new PrintStream(fos);
System.setErr(ps);
System.err.println("goes into file");
}
}

You can create a PrintStream around any other OutputStream.
The simplest way to create one that goes to a buffer in memory would be:
PrintStream p = new PrintStream( new ByteArrayOutputStream() )
Then you could read and reset the contents of the byte array at whatever points you like.
Another possibility would be to use pipes.
InputStream third_party_output = new PipedInputStream();
PrintStream p = new PrintStream( new PipedOutputStream( third_party_output ) );
Then you could read from the third_party_output stream to get the text written by the library.

Are you looking for something like this?
OutputStream redirect = System.err;
PrintStream myPrintStream = new PrintStream(redirect);
myPrintStream.println("hello redirect");
If you can pass myPrintStream to the 3rd party application, you can redirect it anywhere you want.

I use the following class to log System.out and System.err to a set of rotating files (where xxx-001.log is the most recent). It contains a few call to utility methods, which you will need to implement before it will compile - they should be self-explanatory.
import java.io.*;
import java.lang.reflect.*;
public class LoggerOutputStream
extends OutputStream
{
// *****************************************************************************
// INSTANCE PROPERTIES
// *****************************************************************************
private FileOutputStream log=null; // the base output stream
private String fnmBase,fnmExt; // filename base, file extension
private int fnmCount,fnmLast; // count for filename index, last filename used
private int logSize,totWritten; // max log size, current number of bytes written
// *****************************************************************************
// INSTANCE CONSTRUCTORS/INIT/CLOSE/FINALIZE
// *****************************************************************************
public LoggerOutputStream(String baseFilename) throws IOException {
this(baseFilename,".log",2,1024000);
}
public LoggerOutputStream(String baseFilename, String extension) throws IOException {
this(baseFilename,extension,2,1024000);
}
public LoggerOutputStream(String baseFilename, String extension, int numberOfFiles, int maxFileSize) throws IOException {
fnmBase=baseFilename;
if(Character.isLetterOrDigit(fnmBase.charAt(fnmBase.length()-1))) { fnmBase=(fnmBase+"-"); }
fnmExt=extension;
if(!fnmExt.startsWith(".")) { fnmExt=('.'+fnmExt); }
fnmCount=numberOfFiles;
logSize=maxFileSize;
if(fnmCount>MAXLOGS) { fnmCount=MAXLOGS; }
fnmLast=0;
for(int xa=1; xa<=MAXLOGS; xa++) {
if(!new File(constructFilename(xa)).exists()) {
while((--xa)>fnmCount) { IoUtil.deleteFile(constructFilename(xa)); }
fnmLast=xa;
break;
}
}
log=null;
openFile(false);
if(numberOfFiles>MAXLOGS) { System.out.println("** Log File Count Limited To "+MAXLOGS); }
}
public void close() throws IOException {
close(false);
}
private void openFile(boolean ovrflw) throws IOException {
close(true);
if (fnmLast< fnmCount) { fnmLast++; }
else if(fnmLast==fnmCount) { IoUtil.deleteFile(constructFilename(fnmCount)); }
for(int xa=fnmLast; xa>0; xa--) { IoUtil.renameFile(constructFilename(xa-1),constructFilename(xa)); }
log=new FileOutputStream(constructFilename(1));
totWritten=0;
}
private String constructFilename(int index) {
return constructFilename(fnmBase,index,fnmExt);
}
private synchronized void close(boolean ovrflw) throws IOException {
if(log!=null) {
log.flush();
log.close();
log=null;
}
}
// *****************************************************************************
// INSTANCE METHODS - ACCESSORS
// *****************************************************************************
public String getFilename() {
return constructFilename(1);
}
public String getFilename(int idx) {
return constructFilename(idx);
}
public synchronized void cycleLogFile() throws IOException {
openFile(true);
}
// *****************************************************************************
// INSTANCE METHODS
// *****************************************************************************
public synchronized void flush() throws IOException {
if(log!=null) {
log.flush();
}
}
public synchronized void write(int val) throws IOException {
if(log!=null) {
log.write(val);
totWritten++;
if(val=='\n') {
if(totWritten>logSize) { openFile(true); }
else { log.flush(); }
}
}
}
public synchronized void write(byte[] bytes) throws IOException {
if(log!=null) {
log.write(bytes);
totWritten+=bytes.length;
if(bytes.length>0 && bytes[bytes.length-1]=='\n') {
if(totWritten>logSize) { openFile(true); }
else { log.flush(); }
}
}
}
public synchronized void write(byte[] bytes, int str, int len) throws IOException {
if(log!=null) {
log.write(bytes,str,len);
totWritten+=len;
if(bytes.length>(str+len-1) && bytes[str+len-1]=='\n') {
if(totWritten>logSize) { openFile(true); }
else { log.flush(); }
}
}
}
// *****************************************************************************
// STATIC PROPERTIES
// *****************************************************************************
static public final int MAXLOGS=999; // maximum log files allowed
// *****************************************************************************
// STATIC METHODS
// *****************************************************************************
static public String constructFilename(String bas, int idx, String ext) {
if(!bas.endsWith("-") && !bas.endsWith("_") && !bas.endsWith(".")) { bas=(bas+"-"); }
if(!ext.startsWith(".") ) { ext=('.'+ext); }
return (bas+TextUtil.raZeros(idx,3)+ext);
}
} /* END PUBLIC CLASS */