We Keep Coding

PostgreSQL CopyManager copyIn appears stuck doing nothing when using it with COPY FROM STDIN

I am trying to use PostgreSQL CopyManager copyIn functionality with COPY FROM STDIN as suggested in the docs for very fast copying from an InputStream into a database table. I am thinking of using this to continuously stream rows that are to be written to a table as and when i receive/process one. However the below quick and dirty sample code seems to be stuck on copyIn and does not write to the table.
Anyone knows what i am missing here or if my understanding is wrong?
import java.sql.*;
import java.util.concurrent.Callable;
import java.util.concurrent.Executors;
import java.util.concurrent.ExecutorService;
import java.io.PipedInputStream;
import java.io.PipedOutputStream;
import java.io.BufferedWriter;
import java.io.OutputStreamWriter;
import java.io.BufferedReader;
import java.io.InputStreamReader;
import org.postgresql.core.BaseConnection;
import org.postgresql.copy.CopyManager;
public class PGConnectTest {
public static void main(String[] args) {
try {
try (Connection connection = DriverManager.getConnection("jdbc:postgresql://XX.XX.XX.XX:9432/somedb", "someadmin", "somepassword");
BaseConnection pgcon = (BaseConnection)connection;
PipedInputStream is = new PipedInputStream();
BufferedReader br = new BufferedReader(new InputStreamReader(is));
PipedOutputStream os = new PipedOutputStream(is);
BufferedWriter bw = new BufferedWriter(new OutputStreamWriter(os));) {
ExecutorService executorService = Executors.newSingleThreadExecutor();
Callable callable = () -> {
Thread.sleep(3000);
String frmtStr = "%s\t{\"id\":%s, \"somefield\":\"%s\"}\n";
String row = null;
for(int i=1; i<10; i++) {
row = String.format(frmtStr, i, i, ("row"+i));
System.out.print(row);
bw.write(row);
}
bw.write("\n");
bw.flush();
System.out.println("WRITTEN!");
return true;
};
executorService.submit(callable);
System.out.println(connection);
CopyManager copyManager = new CopyManager(pgcon);
String copySql = "COPY dcm.testtbl FROM STDIN";
executorService.submit(() -> copyManager.copyIn(copySql, br));
Thread.sleep(10000);
System.out.println("QUITTING");
} catch (Exception e) {
throw e;
}
} catch(Exception ex) {
System.out.println(ex);
}
}
}
The schema of the table testtbl is below,
create table testtbl (
id integer primary key,
jsnclm jsonb
)
The console output is (it does NOT return and requires using CTRL+C to kill it),
C:\Users\ml410408\Documents\Useful Lookups\POSTGRESQL>java -cp ".;postgresql-42.2.18.jar" PGConnectTest
org.postgresql.jdbc.PgConnection#41975e01
1 {"id":1, "somefield":"row1"}
2 {"id":2, "somefield":"row2"}
3 {"id":3, "somefield":"row3"}
4 {"id":4, "somefield":"row4"}
5 {"id":5, "somefield":"row5"}
6 {"id":6, "somefield":"row6"}
7 {"id":7, "somefield":"row7"}
8 {"id":8, "somefield":"row8"}
9 {"id":9, "somefield":"row9"}
WRITTEN!
QUITTING
UPDATE:
Once i changed the format of the COPY sql command from the default TEXT to CSV and pass in csv records its no longer stuck but does nothing (meaning no records in the table) even though it returns unlike before.
import java.sql.*;
import java.util.concurrent.Callable;
import java.util.concurrent.Executors;
import java.util.concurrent.ExecutorService;
import java.io.PipedInputStream;
import java.io.PipedOutputStream;
import java.io.BufferedWriter;
import java.io.OutputStreamWriter;
import java.io.BufferedReader;
import java.io.InputStreamReader;
import org.postgresql.core.BaseConnection;
import org.postgresql.copy.CopyManager;
public class PGConnectTest {
public static void main(String[] args) {
try {
try (Connection connection = DriverManager.getConnection("jdbc:postgresql://XX.XX.XX.XX:9432/somedb", "someadmin", "somepassword");
BaseConnection pgcon = (BaseConnection)connection;
PipedInputStream is = new PipedInputStream();
BufferedReader br = new BufferedReader(new InputStreamReader(is));
PipedOutputStream os = new PipedOutputStream(is);
BufferedWriter bw = new BufferedWriter(new OutputStreamWriter(os));) {
ExecutorService executorService = Executors.newSingleThreadExecutor();
Callable callable = () -> {
Thread.sleep(3000);
String frmtStr = "%s,'{\"id\":%s,\"somefield\":\"%s\"}'\n";
String row = null;
for(int i=1; i<10; i++) {
row = String.format(frmtStr, i, i, ("row"+i));
System.out.print(row);
bw.write(row);
}
bw.write("\n");
bw.write("'\\.'\n");
System.out.println("'\\.'\n");
bw.flush();
os.flush();
System.out.println("WRITTEN!");
return true;
};
executorService.submit(callable);
System.out.println(connection);
CopyManager copyManager = new CopyManager(pgcon);
String copySql = "COPY dcm.testtbl FROM STDIN FORMAT CSV DELIMITER ','";
executorService.submit(() -> copyManager.copyIn(copySql, br));
Thread.sleep(5000);
System.out.println(br.ready());
while (br.ready()) {
System.out.println("LINE : " + br.readLine());
}
executorService.shutdown();
System.out.println("QUITTING");
} catch (Exception e) {
throw e;
}
System.out.println("QUITTING FINALLY");
} catch(Exception ex) {
System.out.println(ex);
}
}
}
Thanks

There seem to be a couple of different issues in there.
The program is hanging because the thread in the ExecutorService is keeping it alive; calling shutdown() after submitting the tasks causes it to terminate as expected.
The main reason nothing's being written is that copyIn() is throwing an exception: the trailing newline in the stream (bw.write("\n")) triggers an ERROR: invalid input syntax for integer: "" as it fails to find the id column.
Even then, it looks like this is still subject to some race conditions due to the timing of the resource cleanup. The copyIn() call will block until it reaches the end of its InputStream, and in the case of a PipedInputStream, the "end" is the point where the PipedOutputStream is closed. But after the stream is closed and the copyIn() call is unblocked, the input stream and the database connection are closed in quick succession, potentially before the copy has a chance to finalise. At best, it seems to successfully commit to the table, but then error out with a "Database connection failed when canceling copy operation".
To make sure that these resources aren't released while they're still in use:
Wait for the writer to complete
Close the OutputStream
Wait for the copier to complete
Close the InputStream / Connection
Waiting for the tasks to complete has the added benefit of propagating any exceptions to the main thread.
There's also a potential deadlock due to the newSingleThreadExecutor(): if the writer thread fills the pipe's buffer, it will block until the reader starts consuming the data, which will never happen if they're being executed sequentially. Using a newFixedThreadPool(2) should fix this.
With all that in mind:
public static void main(String[] args) {
ExecutorService executorService = Executors.newFixedThreadPool(2);
try {
try (Connection connection = DriverManager.getConnection("jdbc:postgresql://XX.XX.XX.XX:9432/somedb", "someadmin", "somepassword");
BaseConnection pgcon = (BaseConnection) connection;
PipedInputStream is = new PipedInputStream();
BufferedReader br = new BufferedReader(new InputStreamReader(is));
) {
Future write;
Future copy;
try (
PipedOutputStream os = new PipedOutputStream(is);
BufferedWriter bw = new BufferedWriter(new OutputStreamWriter(os))) {
write = executorService.submit(() -> {
String frmtStr = "%s\t{\"id\":%s, \"somefield\":\"%s\"}\n";
String row = null;
for (int i = 1; i < 1000; i++) {
row = String.format(frmtStr, i, i, ("row" + i));
System.out.print(row);
bw.write(row);
}
bw.flush();
System.out.println("WRITTEN!");
return true;
});
System.out.println(connection);
CopyManager copyManager = new CopyManager(pgcon);
String copySql = "COPY dcm.testtbl FROM STDIN";
copy = executorService.submit(() -> copyManager.copyIn(copySql, br));
System.out.println("QUITTING");
write.get();
}
copy.get();
}
} catch (Exception ex) {
System.out.println(ex);
} finally {
executorService.shutdown();
}
}

Java Chat Server (Socket) - how to write unit test?

I have written a Java Chat Server program.
This is a simple standalone program for Server.
I have to run this then run Client to get Chat working.
What are some possible Unit Test scenarios for the server program? Can anyone show me some example of unit test based on this code?
I have never written a unit test code before and I can't really think of what needs to be tested here.. I think testing Connection can be one but what else? (and how to?)
public class SimpleChatServer {
static final Logger logger = LogManager.getLogger(SimpleChatServer.class);
ArrayList<PrintWriter> clientOutputStreams;
private BufferedReader reader;
private Socket sock;
private ServerSocket serverSock;
public class ClientHandler implements Runnable{
public ClientHandler(Socket clientSocket){ // Socket Connection
try {
sock = clientSocket;
InputStreamReader isReader = new InputStreamReader(sock.getInputStream());
reader = new BufferedReader(isReader);
} catch(Exception ex) {
logger.trace(ex);
}
}
public void run() {
String message;
try {
while ((message = reader.readLine()) != null) {
System.out.println("read " + message);
tellEveryone(message);
}
} catch(Exception ex) {
logger.trace(ex);
}
} //close run
} //close ClientHandler
public static void main (String[] args) throws Exception
{
new SimpleChatServer().listen();
}
#SuppressWarnings("resource")
public void listen()
{
clientOutputStreams = new ArrayList<PrintWriter>();
try {
ServerSocket serverSock = new ServerSocket(8000); //port number 8000 was used
while(true) {
Socket clientSocket = serverSock.accept();
PrintWriter writer = new PrintWriter(clientSocket.getOutputStream());
clientOutputStreams.add(writer);
Thread t = new Thread(new ClientHandler(clientSocket));
t.start();
}
} catch (Exception ex) {
logger.trace("Server Error", ex);
} finally {
try
{
serverSock.close();
}
catch(Exception e){}
}
} // close go
public void tellEveryone(String message)
{
Iterator<PrintWriter> it = clientOutputStreams.iterator();
while(it.hasNext()) {
try {
PrintWriter writer = (PrintWriter) it.next();
writer.println(message);
writer.flush();
} catch (Exception ex) {
logger.trace(ex);
}
} // end while
} // close tellEveryone
}

I was going to crib an answer from Pragmatic Unit Testing, but suggest you just find a copy. At the very least you should consider whether results are right, whether your boundary conditions are correct, and if you can force error conditions.
Testing results often means making sure combinations of input get the expected results. Boundaries are reflected in the related "0, 1, many" rule, where you do silly stuff to see if your code has implicit boundaries that can be reached with bad, null or unexpected values.
For example, what happens if you pass huge Strings to your methods that take them? What about strings with weird Unicode chars in them? No line breaks?
Forcing error conditions means making sure things degrade gracefully and/or throw under the expected situation.
Think about your code as a brittle little appliance and then pretend a poo-flinging monkey, a 14-yr old hacker and your non-hacker grandmother (I know some exist) are all taking turns on it.

Threading, and why my program seems to get stuck in a single method

I am trying to learn some network programming, so I thought a good place to start would be with sockets and how to use them. Although it seems that I have hit a brick wall, but the issue does not have as much to do with sockets as it does with checking a socket for two things at (seemingly)the same time.
package com.redab.server;
import java.io.BufferedReader;
import java.io.DataInputStream;
import java.io.IOException;
import java.io.InputStreamReader;
import java.io.PrintWriter;
import java.net.ServerSocket;
import java.net.Socket;
public class server implements Runnable {
private final int portNumber = 4444;
private ServerSocket serverSocket;
private Socket clientSocket;
private Thread thread;
private PrintWriter out;
private BufferedReader in;
private BufferedReader stdIn;
private String incomingText;
private String outgoingText;
private Boolean isRunning = false;
public server() {
thread = new Thread(this, "serverThread");
try {
serverSocket = new ServerSocket(portNumber);
clientSocket = serverSocket.accept(); // Unless socket connection is made, probram will not proceed beyond this line.
System.out.println("connected");
out = new PrintWriter(clientSocket.getOutputStream(), true);
in = new BufferedReader(new InputStreamReader(clientSocket.getInputStream()));
stdIn = new BufferedReader(new InputStreamReader(System.in));
} catch (IOException e) {
System.out.println("Exception caught when trying to listen on port " + portNumber + " or listening for a connection");
System.out.println(e.getMessage());
}
}
private synchronized void start() {
thread.start();
isRunning = true;
}
private synchronized void stop() {
try {
thread.join();
isRunning = false;
} catch (InterruptedException e) {
e.printStackTrace();
}
}
public void run() {
System.out.println("running...");
while (isRunning) {
incoming();
outgoing();
}
}
private synchronized void incoming() {
System.out.println("Incoming");
try {
if ((incomingText = in.readLine()) != null) {
System.out.println(incomingText);
}
} catch (IOException e) {
e.printStackTrace();
}
}
private synchronized void outgoing() {
System.out.println("outgoing");
try {
if ((outgoingText = stdIn.readLine()) != null) {
out.println("Server: " + outgoingText);
System.out.println("Server: " + outgoingText);
}
} catch (IOException e) {
e.printStackTrace();
}
}
public static void main(String[] args) {
server server = new server();
server.start();
}
}
My problem is the following:
How do I make it so both the method incoming() and outgoing() is ran constantly when I execute the code?
I did google a bit and threads seems to be the solution, so I tried creating a thread which is supposed to run both methods for me. But I run into the same problem again, the code gets stuck in the incoming() method when I want it to simply check this statement ((incomingText = in.readLine()) != null) and then proceed to the method outgoing(). I suspect I might need two threads, one of which checks for incoming messages through the socket and the other checks for outgoing messages that are typed into the console(System.in).

I suspect I might need two threads, one of which checks for incoming messages through the socket and the other checks for outgoing messages that are typed into the console(System.in)
You are right, you need two threads, one per each task.

You might want to check if there is data available to read by using the in.ready() method first. If available, you can read the data using in.readLine(), else do nothing. Currently, in.readLine() blocks because there is no input available on the socket.

Java multiplayer server blocking io

I am currently developing a prototype for a game and I need a simple server to run it.
At this stage, I don't want to invest the time learning about all the different full-featured multiplayer game servers already there (smartfox, etc...)
I know how to develop a basic Server with Threads listening on Sockets but I have ran into a roadblock. Here's the run() function of the Thread
public void run() {
try {
out = new PrintWriter(mSocket1.getOutputStream(), true);
in = new BufferedReader( new InputStreamReader( mSocket1.getInputStream() ) );
String inputLine1 = null, outputLine;
out.println("hello");
out.flush();
while( (inputLine1 = in.readLine()) != null) {
outputLine = mGameControl.processInput(mPlayerNum, inputLine1);
out.println(outputLine);
out.flush();
if(outputLine.contentEquals("bye"))
break;
}
Terminate();
}
catch(IOException e) { e.printStackTrace(); }
}
Now my problem is that the thread is blocked waiting for input. I do have other similar Threads connected to other client which may result in information being dispatched to all clients...
How can I modify it so that a different Thread can interact with it and push info to the client?

Just write a synchronised public method which writes to your PrintWriter, and allow other threads to use it to send messages to your client. Call the same method from your read loop to avoid two threads writing at the same time.
Here's a tested example:
import java.io.*;
import java.net.ServerSocket;
import java.net.Socket;
public class SocketTest {
public static class Client implements Runnable {
private final BufferedReader in;
private final PrintWriter out;
public Client(Socket clientSocket) throws IOException {
out = new PrintWriter(clientSocket.getOutputStream(), true);
in = new BufferedReader( new InputStreamReader( clientSocket.getInputStream() ) );
}
public void run() {
send("Hello");
String inputLine1 = null, outputLine;
try {
while( (inputLine1 = in.readLine()) != null) {
outputLine = inputLine1.toLowerCase();
System.out.println(inputLine1);
send(outputLine);
if(outputLine.contentEquals("bye"))
break;
}
} catch (IOException e) {
e.printStackTrace();
}
}
public synchronized void send(String message) {
out.println(message);
out.flush();
}
}
public static void main(String[] args) throws IOException, InterruptedException {
ServerSocket s = new ServerSocket(5050);
Socket clientSocket = s.accept();
Client client = new Client(clientSocket);
Thread clientThread = new Thread(client);
clientThread.start();
int i = 1;
while (true) {
Thread.sleep(1000);
client.send("Tick " + (i++));
}
}
}

Use Netty to handle your connections and query treatments. Since I discovered that project, I never touched sockets directly anymore (except when writing C programs, etc.)
There are actually some examples to look at and the documentation is quite extensive. The project is very well alive since a couple of years already, and is not soon to die! There's a fairly large user base behind it.

If you only need to perform a non-blocking read on the socket, one of the simplests ways is to use available(). Call available and if there is data to read (bytes pending > 0), perform the read, any way, wait using Thread.sleep() and try to read again. This allow the thread to do while waiting for input data and cat react to external signals.
The use of selectors is encouraged when you need to perform high-performance non-blocking read using one thread and several sockets (java.nio.channels.Selector).

Run a sub process, provide input and output to it correctly in Java

I use Runtime exec() method to create a subprocess in Java. However, since the subprocess is an interactive program, I need to provide input to it as and when required by it. Also I need to show the output of the subprocess. How can I do this in the simplest possible way?
I was using a StreamGobbler to show the program output using process.getInputStream(). I, however, do not know how to identify when the program is waiting for input and when to provide it input using proc.getOutputStream. How can I do this?

You need to copy the input and output between the subprocess' streams and System streams (System.in, System.out and System.err). This is related to my recent quesion. The best solution I have found so far is:
import java.io.FileInputStream;
import java.io.FilterInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.lang.reflect.Field;
import java.nio.ByteBuffer;
import java.nio.channels.AsynchronousCloseException;
import java.nio.channels.FileChannel;
class StreamCopier implements Runnable {
private InputStream in;
private OutputStream out;
public StreamCopier(InputStream in, OutputStream out) {
this.in = in;
this.out = out;
}
public void run() {
try {
int n;
byte[] buffer = new byte[4096];
while ((n = in.read(buffer)) != -1) {
out.write(buffer, 0, n);
out.flush();
}
}
catch (IOException e) {
System.out.println(e);
}
}
}
class InputCopier implements Runnable {
private FileChannel in;
private OutputStream out;
public InputCopier(FileChannel in, OutputStream out) {
this.in = in;
this.out = out;
}
public void run() {
try {
int n;
ByteBuffer buffer = ByteBuffer.allocate(4096);
while ((n = in.read(buffer)) != -1) {
out.write(buffer.array(), 0, n);
out.flush();
}
out.close();
}
catch (AsynchronousCloseException e) {}
catch (IOException e) {
System.out.println(e);
}
}
}
public class Test {
private static FileChannel getChannel(InputStream in)
throws NoSuchFieldException, IllegalAccessException {
Field f = FilterInputStream.class.getDeclaredField("in");
f.setAccessible(true);
while (in instanceof FilterInputStream)
in = (InputStream)f.get((FilterInputStream)in);
return ((FileInputStream)in).getChannel();
}
public static void main(String[] args)
throws IOException, InterruptedException,
NoSuchFieldException, IllegalAccessException {
Process process = Runtime.getRuntime().exec("sh -i +m");
Thread outThread = new Thread(new StreamCopier(
process.getInputStream(), System.out));
outThread.start();
Thread errThread = new Thread(new StreamCopier(
process.getErrorStream(), System.err));
errThread.start();
Thread inThread = new Thread(new InputCopier(
getChannel(System.in), process.getOutputStream()));
inThread.start();
process.waitFor();
System.in.close();
outThread.join();
errThread.join();
inThread.join();
}
}
The tricky part here is to extract a channel from System.in. Without this you will not be able to interrupt the thread that reads input when the subprocess terminates.
This approach has a serious drawback: after closing System.in you can no longer read from it. The workaround that I'm currently using is to have a single input redirecting thread used for all subprocesses.

Ask yourself "How do I know when the program wants input when I run it from the command line"? You see what it prompts and enter data based on that prompt. The principle will be the same, except your code will need to interpret the program's output and provide the correct input.
To avoid reinventing the wheel, take a look at ExpectJ and/or Expect4J, which are Java implementations of the venerable *nix Expect tool, which is designed to handle this kind of programmatic interaction.