We Keep Coding

Pipe Broken with PipeInputStream with kubernetes-client exec()

I'm using the kubernetes-client to try copy a directory from a pod, but I'm doing something wrong with the input stream from stdout. I get a java.io.IOException: Pipe broken exception when it tries to read(). I'm pretty sure that no data flows at all. I'm half wondering if I need to read the InputStream on a separate thread or something?
The stream is created like this:
public InputStream copyFiles(String containerId,
String folderName) {
ExecWatch exec = client.pods().withName(containerId).redirectingOutput().exec("tar -C " + folderName + " -c");
// We need to wrap the InputStream so that when the stdout is closed, then the underlying ExecWatch is closed
// also. This will cleanup any Websockets connections.
ChainedCloseInputStreamWrapper inputStreamWrapper = new ChainedCloseInputStreamWrapper(exec.getOutput(), exec);
return inputStreamWrapper;
}
And the InputStream is processed in this function
void copyVideos(final String containerId) {
TarArchiveInputStream tarStream = new TarArchiveInputStream(containerClient.copyFiles(containerId, "/videos/"));
TarArchiveEntry entry;
boolean videoWasCopied = false;
try {
while ((entry = tarStream.getNextTarEntry()) != null) {
if (entry.isDirectory()) {
continue;
}
String fileExtension = entry.getName().substring(entry.getName().lastIndexOf('.'));
testInformation.setFileExtension(fileExtension);
File videoFile = new File(testInformation.getVideoFolderPath(), testInformation.getFileName());
File parent = videoFile.getParentFile();
if (!parent.exists()) {
parent.mkdirs();
}
OutputStream outputStream = new FileOutputStream(videoFile);
IOUtils.copy(tarStream, outputStream);
outputStream.close();
videoWasCopied = true;
LOGGER.log(Level.INFO, "{0} Video file copied to: {1}/{2}", new Object[]{getId(),
testInformation.getVideoFolderPath(), testInformation.getFileName()});
}
} catch (IOException e) {
LOGGER.log(Level.WARNING, getId() + " Error while copying the video", e);
ga.trackException(e);
} finally {
if (!videoWasCopied) {
testInformation.setVideoRecorded(false);
}
}
}
The InputStream Wrapper class is just there to close the ExecWatch at the end once the InputStream is closed, it looks like this:
private static class ChainedCloseInputStreamWrapper extends InputStream {
private InputStream delegate;
private Closeable resourceToClose;
public ChainedCloseInputStreamWrapper(InputStream delegate, Closeable resourceToClose) {
this.delegate = delegate;
this.resourceToClose = resourceToClose;
}
#Override
public int read() throws IOException {
return delegate.read();
}
public int available() throws IOException {
return delegate.available();
}
public void close() throws IOException {
logger.info("Shutdown called!");
delegate.close();
// Close our dependent resource
resourceToClose.close();
}
public boolean equals(Object o) {
return delegate.equals(o);
}
public int hashCode() {
return delegate.hashCode();
}
public int read(byte[] array) throws IOException {
return delegate.read(array);
}
public int read(byte[] array,
int n,
int n2) throws IOException {
return delegate.read(array, n, n2);
}
public long skip(long n) throws IOException {
return delegate.skip(n);
}
public void mark(int n) {
delegate.mark(n);
}
public void reset() throws IOException {
delegate.reset();
}
public boolean markSupported() {
return delegate.markSupported();
}
public String toString() {
return delegate.toString();
}
}

Turns out I had the tar command wrong, so it was causing a failure and the stdout PipeInputStream was dead locking. I managed to find a workaround for the deadlock. But the main reason for the failure was that I forgot to tell tar to actually do something! I at least needed a "." to include the current directory.

Parquet Writer to buffer or byte stream

I have a java application that converts json messages to parquet format. Is there any parquet writer which writes to buffer or byte stream in java? Most of the examples, I have seen write to files.

TLDR; you will need to implement OutputFile, e.g. something along the line of:
import org.apache.parquet.io.OutputFile;
import org.apache.parquet.io.PositionOutputStream;
import java.io.BufferedOutputStream;
import java.io.IOException;
public class ParquetBufferedWriter implements OutputFile {
private final BufferedOutputStream out;
public ParquetBufferedWriter(BufferedOutputStream out) {
this.out = out;
}
#Override
public PositionOutputStream create(long blockSizeHint) throws IOException {
return createPositionOutputstream();
}
private PositionOutputStream createPositionOutputstream() {
return new PositionOutputStream() {
#Override
public long getPos() throws IOException {
return 0;
}
#Override
public void write(int b) throws IOException {
out.write(b);
}
};
}
#Override
public PositionOutputStream createOrOverwrite(long blockSizeHint) throws IOException {
return createPositionOutputstream();
}
#Override
public boolean supportsBlockSize() {
return false;
}
#Override
public long defaultBlockSize() {
return 0;
}
}
And your writer would be something like:
ParquetBufferedWriter out = new ParquetBufferedWriter();
try (ParquetWriter<Record> writer = AvroParquetWriter.
<Record>builder(out)
.withRowGroupSize(DEFAULT_BLOCK_SIZE)
.withPageSize(DEFAULT_PAGE_SIZE)
.withSchema(SCHEMA)
.build()) {
for (Record record : records) {
writer.write(record);
}
} catch (IOException e) {
throw new IllegalStateException(e);
}

I just also needed to write to a stream, so I completed the example given by naimdjon. The following works perfectly fine for me.
class ParquetBufferedWriter implements OutputFile {
private final BufferedOutputStream out;
public ParquetBufferedWriter(BufferedOutputStream out) {
this.out = out;
}
#Override
public PositionOutputStream create(long blockSizeHint) throws IOException {
return createPositionOutputstream();
}
private PositionOutputStream createPositionOutputstream() {
return new PositionOutputStream() {
int pos = 0;
#Override
public long getPos() throws IOException {
return pos;
}
#Override
public void flush() throws IOException {
out.flush();
};
#Override
public void close() throws IOException {
out.close();
};
#Override
public void write(int b) throws IOException {
out.write(b);
pos++;
}
#Override
public void write(byte[] b, int off, int len) throws IOException {
out.write(b, off, len);
pos += len;
}
};
}
#Override
public PositionOutputStream createOrOverwrite(long blockSizeHint) throws IOException {
return createPositionOutputstream();
}
#Override
public boolean supportsBlockSize() {
return false;
}
#Override
public long defaultBlockSize() {
return 0;
}
}

You need to write the data into temp file and then covert the data from file to input stream or buffer
something like this, first read the tempfile data
final InputStream targetStream = new DataInputStream(new FileInputStream(tmp1.getAbsoluteFile()));
StringWriter writer = new StringWriter();
String encoding = StandardCharsets.UTF_8.name();
IOUtils.copy(targetStream, writer, encoding);
System.out.println(writer);

How to tie a PrintStream to the System.out and err streams

I'm trying to tie my PrintStream object to the console's output and error streams so that whatever I write there will also be written to my log file.
public static void tieOutputStreams(String fileName) {
try {
File output = new File(fileName);
FileWriter writer = new FileWriter(output);
writer.close();
outputStream = new TiedOutputStream(output);
}
catch (Exception e) {
e.printStackTrace();
}
System.setErr(outputStream);
System.setOut(outputStream);
}
Once I'm done writing, I could reset it back to the way things were.
public static void resetOutputStreams() {
outputStream.close();
System.setErr(System.err);
System.setOut(System.out);
}
TiedOutputStream class looks like this:
public class TiedOutputStream extends PrintStream {
public TiedOutputStream(File logFile) throws FileNotFoundException {
super(logFile);
}
#Override
public void print(Object obj) {
super.print(obj);
System.out.print(obj);
}
#Override
public PrintStream printf(String format, Object... args) {
super.printf(format, args);
System.out.printf(format, args);
return this;
}
#Override
public void println(Object args) {
super.println(args);
System.out.println(args);
}
}
And my main method:
public static void main(String[] args) {
try {
TieOutputStreams.tieOutputStreams("./sample.log");
System.out.println("Output console");
System.err.println("Error console");
float num = 1.123456f;
System.out.printf("A float: %.6f", num);
} catch (Exception e) {
e.printStackTrace();
}
finally {
TieOutputStreams.resetOutputStreams();
}
}
I want these statements to be printed on both my log file and the System consoles (out / err). For reasons I don't know, this isn't working. I appreciate all the answers and comments. Thanks in advance!
I know there is Log4j. But I want to do this anyway.

This doesn't work mainly because you didn't save the original System.out and because you didn't override println(String obj) When you call System.out.println("Output console"); you won't hit in the method you override because that one expects and object and there is a more specific method in PrintStream that expects a String argument
This seems to work:
public class TiedOutputStream extends PrintStream {
private final PrintStream sout;
private final PrintStream serr;
public TiedOutputStream(File logFile) throws FileNotFoundException {
super(logFile);
sout = System.out;//save standard output
serr = System.err;
}
#Override
public void print(Object obj) {
super.print(obj);
sout.print(obj);
}
#Override
public void println(String obj) {
super.println(obj);
sout.println(obj);
}
#Override
public PrintStream printf(String format, Object... args) {
super.printf(format, args);
sout.printf(format, args);
return this;
}
#Override
public void println(Object args) {
super.println(args);
sout.println(args);
}
}
Not sure why tieOutputStreams created that FileWriter
public static void tieOutputStreams(String fileName) {
try {
File output = new File(fileName);
outputStream = new TiedOutputStream(output);
System.setErr(outputStream);
System.setOut(outputStream);
} catch (Exception e) {
e.printStackTrace();
}
}
main method remains the same. You should update resetOutputStreams to restore to original out and err. I would override all print* method from PrintStream if I would use this.

Java pipes, why i get the write end dead exception

i'm trying to create a program using pipes that communicate between 2 threads (you can say chat between 2 threads), my problem here is when you write there is no problem, but when you read from the pipe, it throws exception with end dead end. I did a send & receive method but my receive should know the length of string is sent by sender method, i did another receive method in same name without knowing the String length sent.
My code is composed of 3 classes as shown below :
package pipes1;
import java.io.*;
public class Pipe
{
private PipedWriter writer;
private PipedReader reader;
public PipedWriter getWriter()
{
return writer;
}
public PipedReader getReader()
{
return reader;
}
public Pipe()
{
writer = new PipedWriter();
reader = new PipedReader();
}
}
========================================================
package pipes1;
import java.io.IOException;
public class Person
{
private String name; //name of person
private String msg1;
private String msg2;
private Pipe pipe;
public String getMsg1()
{
return msg1;
}
public String getMsg2()
{
return msg2;
}
public Pipe getPipe()
{
return pipe;
}
public String getName()
{
return name;
}
public Person(String name,Pipe pipe,String s1,String s2)
{
this.name = name;
this.msg1 = s1;
this.msg2 = s2;
this.pipe = pipe;
}
public void connection(Person x) throws Throwable
{
pipe.getReader().connect(x.pipe.getWriter());
}
public void closing() throws IOException
{
this.pipe.getReader().close();
this.pipe.getWriter().close();
}
public void send(String m) throws IOException
{
this.pipe.getWriter().write(m);
this.pipe.getWriter().flush();
}
public void recieve() throws IOException
{
int data = this.pipe.getReader().read();
while(data!=-1)
{
System.out.print((char)data);
data = this.pipe.getReader().read();
}
System.out.println("");
}
public void recieve(String m) throws IOException
{
int i = 0;
while(i<m.length())
{
System.out.print((char) this.pipe.getReader().read());
i++;
}
System.out.println("");
}
}
==================================================================
package pipes1;
public class Main
{
public static void main(String[] args) throws Throwable
{
Pipe p1 = new Pipe();
Pipe p2 = new Pipe();
Person alice = new Person("Alice",p1,"recieved,thanks","hi bob");
Person bob = new Person("Bob",p2,"hi alice","recieved, thanks");
alice.connection(bob);
bob.connection(alice);
Thread terminal1 = new Thread(new Runnable()
{
#Override
public void run()
{
try
{
bob.send(bob.getName()+":"+bob.getMsg1());
bob.recieve(alice.getName()+":"+alice.getMsg1());
bob.recieve(alice.getName()+":"+alice.getMsg2());
bob.send(bob.getName()+":"+bob.getMsg2());
bob.send("hi");
bob.send("hi");
}
catch (Throwable e)
{
System.out.println(e.getMessage());
}
}
});
//terminal of a
Thread terminal2 = new Thread(new Runnable()
{
#Override
public void run()
{
try
{
alice.recieve(bob.getName()+":"+bob.getMsg1());
alice.send(alice.getName()+":"+alice.getMsg1());
alice.send(alice.getName()+":"+alice.getMsg2());
alice.recieve(bob.getName()+":"+bob.getMsg2());
alice.recieve();
alice.recieve();
}
catch (Throwable e)
{
System.out.println(e.getMessage());
}
}
});
terminal1.start();
terminal2.start();
}
}
=================================================================
and the result is this :
Bob:hi alice
Alice:recieved,thanks
Alice:hi bob
Bob:recieved, thanks
hihiWrite end dead

A thread that wrote to a pipe ended without closing the pipe, leaving the pipe broken. A subsequent attempt to read from the PipedReader detected this and threw an IOException.
From the javadoc for the method PipedReader.read():
public int read()
throws IOException
...
Throws:
IOException - if the pipe is broken, unconnected, closed, or an I/O error occurs.
From the javadoc for PipedInputStream:
A pipe is said to be broken if a thread that was providing data bytes to the connected piped output stream is no longer alive.
I think you can avoid the error by adding bob.closing() in the first thread. (I haven't tested that.) Each writer thread should really close the pipe to which it's writing.

Can't resolve java.io.EOFException?

This is my first file
public class ObjectStream
{
public static void main(String args[]) throws FileNotFoundException, IOException
{
java.io.File f=new java.io.File("D:Employee.outt") ;
Employee e=new Employee("John");
Employee e1=new Employee("Mary");
Employee e2=new Employee("Christian");
OutputStream os=new java.io.FileOutputStream(f);
java.io.ObjectOutputStream oos= new ObjectOutputStream(os);
oos.writeObject(e);
oos.writeObject(e1);
oos.writeObject(e2);
}
}
This is my second file
public class Employee implements java.io.Serializable
{
private static int count=100;
private int eid;
private String ename;
public Employee()
{
count++;
this.eid=count;
}
public Employee(String ename)
{
this();
this.ename=ename;
}
public static int getCount() {
return count;
}
public static void setCount(int count) {
Employee.count = count;
}
public int getEid() {
return eid;
}
public void setEid(int eid) {
this.eid = eid;
}
public String getEname() {
return ename;
}
public void setEname(String ename) {
this.ename = ename;
}
}
This is my third file
public class MainClass
{
public static void main(String args[]) throws FileNotFoundException, IOException, ClassNotFoundException
{
File f=new File("D:Employee.outt");
byte data[]=new byte[(int)f.length()];
InputStream is=new java.io.FileInputStream(f);
java.io.ObjectInputStream ois=new java.io.ObjectInputStream(is);
Object o=ois.readObject();
while(o!=null)
{
Employee e=(Employee)o;
System.out.println(e.getEid());
System.out.println(e.getEname());
o=ois.readObject();
}
ois.close();
is.close();
}
}
I am trying to read objects stored in Employee.outt in via this third file but it is reading all the objects but at the end throwing
Exception in thread "main" java.io.EOFException.
I don't know how to resolve it.

while(o!=null)
This isn't a valid way to read an ObjectInputStream. The readObject() method only returns a null if you wrote a null. At end of stream it throws, guess what, an EOFException, so the correct way to read the stream is to loop calling `readObject() until you catch that exception, then break and close the stream.
at the end of file it is giving EOF exception
That's exactly what EOFException means.

You should close ObjectOutputStreamin ObjectStream class.
oos.writeObject(e);
oos.writeObject(e1);
oos.writeObject(e2);
oos.close();
In the MainClass you can use a try-finally block to close the inputstream.
try
{
InputStream is=new java.io.FileInputStream(f);
java.io.ObjectInputStream ois=new java.io.ObjectInputStream(is);
// READ logic here
}
finally
{
ois.close();
is.close();
}

You really should flush and close the object output stream in "first file".
You only need to close 'ois' in "third file". The other stream is automatically closed for you.
If you know how many objects you're going to write, change "first file" to write the number of objects following first, then change "third file" to read that number and only read that many objects. That way, you never get the EOF exception, because you stop reading when you've read the last object.
As EJP pointed out, you still have to catch the EOFException, because the file you're reading might be damaged.
In First File:
...
oos.writeInt(3);
oos.writeObject(e);
...
In Third File:
...
try {
int numObjects = ois.readInt();
for( int a = 0; a < numObjects; ++a ) {
...
}
} catch( EOFException e ) {
...
}