We Keep Coding

How can I know if there is no data to read in Netty ByteBuf?

I'm new to Netty. There is a problem about file transfer confusing me for days. I want to send image file from client to server.
The code below is executable. But only I shutdown server forcibly can I open received image file normally. Otherwise, it shows "It looks like you don't have permission to view this file. Check the permissions and try again". So I want to close fileoutputstream when there is no data in ByteBuf using ByteBuf.isReadable(), but the else block in method channelRead in ServerHandler never reach. It's useless.
Besides, if sending text file, it can be open normally when server is alive. I don't want to shutdown server every time after transfer. Please give me some suggestions to solve it.
This is FileClientHandler
public class FileClientHandler extends ChannelInboundHandlerAdapter {
private int readLength = 8;
#Override
public void channelActive(ChannelHandlerContext ctx) throws Exception {
sendFile(ctx.channel());
}
private void sendFile(Channel channel) throws IOException {
File file = new File("C:\\Users\\xxx\\Desktop\\1.png");
FileInputStream fis = new FileInputStream(file);
BufferedInputStream bis = new BufferedInputStream(fis);
for (;;) {
byte[] bytes = new byte[readLength];
int readNum = bis.read(bytes, 0, readLength);
// System.out.println(readNum);
if (readNum == -1) {
bis.close();
fis.close();
return;
}
sendToServer(bytes, channel, readNum);
}
}
private void sendToServer(byte[] bytes, Channel channel, int length)
throws IOException {
channel.writeAndFlush(Unpooled.copiedBuffer(bytes, 0, length));
}
}
This is FileServerHandler
public class FileServerHandler extends ChannelInboundHandlerAdapter {
private File file = new File("C:\\Users\\xxx\\Desktop\\2.png");
private FileOutputStream fos;
public FileServerHandler() {
try {
if (!file.exists()) {
file.createNewFile();
} else {
file.delete();
file.createNewFile();
}
fos = new FileOutputStream(file);
} catch (IOException e) {
e.printStackTrace();
}
}
#Override
public void channelRead(ChannelHandlerContext ctx, Object msg)
throws Exception {
try {
ByteBuf buf = (ByteBuf) msg;
if (buf.isReadable()) {
buf.readBytes(fos, buf.readableBytes());
fos.flush();
} else {
System.out.println("I want to close fileoutputstream!");
buf.release();
fos.flush();
fos.close();
}
} catch (Exception e) {
e.printStackTrace();
}
}
}

Fixing the server side
In the Netty world, there are multiple "events":
channelActive
channelRead
channelReadComplete
channelInactive
exceptionCaught
more...
Of these "events", you probably already knows what channelRead does (since your using it), but another one that you seem to need is the channelInactive. This one is called when the other endpoint shuts the connection down, and you can use it like this:
#Override
public void channelInactive(ctx) {
System.out.println("I want to close fileoutputstream!");
fos.close();
}
#Override
public void channelRead(ChannelHandlerContext ctx, Object msg)
throws Exception {
try {
ByteBuf buf = (ByteBuf) msg;
// if (buf.isReadable()) { // a buf should always be readable here
buf.readBytes(fos, buf.readableBytes());
// fos.flush(); // flushing is always done when closing
//} else {
// System.out.println("I want to close fileoutputstream!");
// buf.release(); // Should be placed in the finally block
// fos.flush();
// fos.close();
//}
} catch (Exception e) {
e.printStackTrace();
} finally {
buf.release(); // Should always be done, even if writing to the file fails
}
}
However, how does the server know the connection has shutdown? At the moment the client does not shut the server down, and instead keep running in the background forever keeping the connection alive.
Fixing the client side
To properly shutdown the connection from the client, we need to call channel.close(), however, we cannot directly insert this before the return line, as this causes a race condition between sending the data, and closing the connection in the network layer, potentially dropping data.
To properly handle these conditions, Netty uses a Future system that allows code to handle events after asynchronous actions happen.
Lucky for us, Netty already has a build in solution for this, we only need to wire it up. To wire this solution up to our code, we have to keep track of the latest ChannelFuture emitted by Netty's write method.
To properly implement this solution, we change sendToServer to return the result of the write method:
private ChannelFuture sendToServer(byte[] bytes, Channel channel, int length)
throws IOException {
return channel.writeAndFlush(Unpooled.copiedBuffer(bytes, 0, length));
}
Then we keep keep track of this return value, and add a listener containing Netty's build in solution when we want to close the connection:
ChannelFuture lastFuture = null;
for (;;) {
byte[] bytes = new byte[readLength];
int readNum = bis.read(bytes, 0, readLength);
// System.out.println(readNum);
if (readNum == -1) {
bis.close();
fis.close();
if(lastFuture == null) { // When our file is 0 bytes long, this is true
channel.close();
} else {
lastFuture.addListener(ChannelFutureListener.CLOSE);
}
return;
}
lastFuture = sendToServer(bytes, channel, readNum);
}

How do I create a file sending client/server with RSocket?

I can't seem to find any resources/tutorials on RSocket, other than just reading their code on GitHub, which I don't understand.
I have a file's path on my server: String serverFilePath;
I'd like to be able to download it from my client (using RSocket's Aeron implementation, preferably). Does anyone know how to do this using RSocket?
Thanks in advance.

I work on RSocket, and wrote a large portion of the java version including the Aeron transport.
I wouldn't recommend using the Aeron implementation currently. There's a couple ways you can send files:
Using a requestChannel to push the data to a remote server.
Use requestChannel or requestStream to stream bytes to a client.
Here's an example using requestStream:
public class FileCopy {
public static void main(String... args) throws Exception {
// Create a socket that receives incoming connections
RSocketFactory.receive()
.acceptor(
new SocketAcceptor() {
#Override
// Create a new socket acceptor
public Mono<RSocket> accept(ConnectionSetupPayload setup, RSocket sendingSocket) {
return Mono.just(
new AbstractRSocket() {
#Override
public Flux<Payload> requestStream(Payload payload) {
// Get the path of the file to copy
String path = payload.getDataUtf8();
SeekableByteChannel _channel = null;
try {
_channel = Files.newByteChannel(Paths.get(path), StandardOpenOption.READ);
} catch (IOException e) {
return Flux.error(e);
}
ReferenceCountUtil.safeRelease(payload);
SeekableByteChannel channel = _channel;
// Use Flux.generate to create a publisher that returns file at 1024 bytes
// at a time
return Flux.generate(
sink -> {
try {
ByteBuffer buffer = ByteBuffer.allocate(1024);
int read = channel.read(buffer);
buffer.flip();
sink.next(DefaultPayload.create(buffer));
if (read == -1) {
channel.close();
sink.complete();
}
} catch (Throwable t) {
sink.error(t);
}
});
}
});
}
})
.transport(TcpServerTransport.create(9090))
.start()
.subscribe();
String path = args[0];
String dest = args[1];
// Connect to a server
RSocket client =
RSocketFactory.connect().transport(TcpClientTransport.create(9090)).start().block();
File f = new File(dest);
f.createNewFile();
// Open a channel to a new file
SeekableByteChannel channel =
Files.newByteChannel(f.toPath(), StandardOpenOption.CREATE, StandardOpenOption.WRITE);
// Request a stream of bytes
client
.requestStream(DefaultPayload.create(path))
.doOnNext(
payload -> {
try {
// Write the bytes received to the new file
ByteBuffer data = payload.getData();
channel.write(data);
// Release the payload
ReferenceCountUtil.safeRelease(payload);
} catch (Exception e) {
throw new RuntimeException(e);
}
})
// Block until all the bytes are received
.blockLast();
// Close the file you're writing too
channel.close();
}
}

There is now a resumable file transfer example here
https://github.com/rsocket/rsocket-java/commit/d47629147dd1a4d41c7c8d5af3d80838e01d3ba5

Simple Proxy in Java : POST request exception

I wrote a simple HTTP proxy in java. The code is as follows:
public class SampleProxy {
public static void main(String[] args) throws IOException {
ServerSocket serverSocket = null;
boolean listening = true;
int port = 1234;
try {
serverSocket = new ServerSocket(port);
} catch (IOException e) {
System.out.println("Port Error");
System.exit(-1);
}
while (listening) {
new ProxyThread(serverSocket.accept()).start();
}
serverSocket.close();
}
}
class ProxyThread extends Thread {
private final Socket clientSocket;
public ProxyThread(Socket socket) {
this.clientSocket = socket;
}
public void run() {
try {
String parts[];
// Read request
InputStream incomingIS = clientSocket.getInputStream();
int incomingLen = incomingIS.available();
byte[] b = new byte[8196];
int len = incomingIS.read(b);
if (len > 0) {
System.out.println("The Request is : \n" + new String(b, 0, len) + "\n*********\n");
Socket socket = new Socket("localhost", 80);
OutputStream outgoingOS = socket.getOutputStream();
outgoingOS.write(b, 0, len);
OutputStream incomingOS = clientSocket.getOutputStream();
InputStream outgoingIS = socket.getInputStream();
int length;
//Read from server
byte[] b2=new byte[8196];
length = outgoingIS.available();
outgoingIS.read(b2);
incomingOS.write(b2, 0, b2.length);
incomingOS.close();
outgoingIS.close();
outgoingOS.close();
incomingIS.close();
socket.close();
} else {
incomingIS.close();
}
} catch (IOException e) {
e.printStackTrace();
} catch (Exception ex) {
Logger.getLogger(ProxyThread.class.getName()).log(Level.SEVERE, null, ex);
} finally {
try {
clientSocket.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
}
As seen, the client requests at port 1234, the proxy forwards the request to the server and gets the response which is then returned back to the client.
In normal scenarios, this works fine, but, I tried testing the same for high traffic scenarios using a shell script which sends multiple(around 50) POST requests, one after another and found out that for some requests, only the headers are getting received. The POST data is omitted but the Content-Length shows the appropriate length of the variables which ought to be sent. This does not happen for specific requests, but at random, maybe 3 or 4 out of 50 Requests.
The same error however, does not seem to exist for GET requests.
Any suggestions as to whats actually causing the error?

Usual mistakes here. Misuse of available(), and failure to actually implement the HTTP protocol correctly. You need to read RFC 2616 thoroughly, especially the parts about request and response length; and reread the Javadoc for InputStream.available(), where you will find a specific warning against using it as you are doing here.

send a file to a client after long wait

i've a java servlet that makes some reports. When a user choose a report it makes a query on a db and stream the xls report to the client. All in synchronous way. The problem is that sometimes i've a lot of records to fetch from the db and i would like to give a better user experience, allowing the user to do something else while the report is processing and popping out in some way the link when the process is finished. Is there a java library or some techniques to avoid the long waiting and achieve that goal?
Right now i've prepared a piece of code that in a asynchronous way completes the report and sends an email to the registered client, with the url from wich download the file, but it has to be replaced with something else because i can no longer communicate by email.
Thanks in advance

heres my take on this, i dont know of a single library that will exactly match you needs, youd probably need some custom development here.
I believe you have implemented async service that on completion sends
out an email for notification. Instead of sending out an email, let
that thread update a job table of some sort -- an entry in a db table
or some application/session scoped map.
Have a servlet/restful ws
expose that job table at some url. Poll the url at regular
intervals. Ajax poll is a standard feature in js libraries JQuery,
Prototype.
When you get a response that some report is complete, show
some popup or may be a facebook you-have-notification kind of thing
on the client side.
i have not considered authentication/authorization issues here, you need to take care of that as well.
Hope this helps

A multithreaded client server program to download my image files.
Since there are four files to download the client makes 4 connection attempts. This is not limited to 4 but the files sent by the FileServer will get repeated after the fourth attempt. The save dialog and file saving is done in different threads so as to not hamper the file downloading.
Here is the FileServer...
public class FileServer {
private final ExecutorService exec = Executors.newCachedThreadPool();
final String[] fileNames = {
"C:\\Users\\clobo\\Pictures\\Arpeggios\\Ex 1.jpg",
"C:\\Users\\clobo\\Pictures\\Arpeggios\\Ex 2.jpg",
"C:\\Users\\clobo\\Pictures\\Arpeggios\\Ex 3.jpg",
"C:\\Users\\clobo\\Pictures\\Arpeggios\\Ex 4.jpg"
};
public void start() throws IOException {
ServerSocket socket = new ServerSocket(7777);
System.out.println("Waiting for client message...");
while (!exec.isShutdown()) {
try {
for (final String fileName : fileNames){
final Socket conn = socket.accept();
exec.execute(new Runnable() {
public void run() {
sendFile(conn,fileName);
}
});
}
} catch (RejectedExecutionException e) {
if (!exec.isShutdown())
log("task submission rejected", e);
}
}
}
public void stop() {
System.out.println("Shutting down server...");
exec.shutdown();
}
private void log(String msg, Exception e) {
Logger.getAnonymousLogger().log(Level.WARNING, msg, e);
}
public void sendFile(Socket conn, String fileName) {
File myFile = new File(fileName);
if (!myFile.exists()) {
log("File does not exist!",null);
}
// file does exist
System.out.println(Thread.currentThread().getName());
System.out.println("AbsolutePath:" + myFile.getAbsolutePath());
System.out.println("length: " + myFile.length());
if (myFile.exists()) {
try {
ObjectOutputStream oos = new ObjectOutputStream(
conn.getOutputStream());
oos.writeObject(myFile);
oos.close();
} catch (IOException e) {
log("IOException Error", e);
e.printStackTrace();
}
}
}
public static void main(String[] args) throws IOException {
FileServer fs = new FileServer();
fs.start();
}
}
here is the FileServerClient...
public class FileServerClient {
private final ExecutorService exec = Executors.newCachedThreadPool();
Frame myFrame = new Frame();
List<File> fileList = new ArrayList<File>();
public void receiveFileFromServer() throws Exception{
Socket sock = null;
InputStream socketInputStream = null;
String host = "localhost";
int port = 7777;
for (int i=0;i<4;i++) {
sock = new Socket(host, port);
socketInputStream = sock.getInputStream();
System.out.println("Connection successful...");
// recieve the file
ObjectInputStream ois = new ObjectInputStream(socketInputStream);
// file from server is deserialized
final File myfile = (File) ois.readObject();
fileList.add(myfile);
// deserialized file properties
System.out.println("AbsolutePath: " + myfile.getAbsolutePath());
System.out.println("FileName:" + myfile.getName());
System.out.println("length" + myfile.length());
exec.execute(new Runnable() {
public void run() {
saveFile(myfile);
}
});
}
}
private void saveFile(File myfile) {
FileDialog fileDialog = new FileDialog(myFrame,
"Choose Destination for "+ myfile.getName(), FileDialog.SAVE);
fileDialog.setDirectory(null);
fileDialog.setFile("enter file name here");
fileDialog.setVisible(true);
String targetFileName = fileDialog.getDirectory()
+ fileDialog.getFile() + ".jpg";
System.out.println("File will be saved to: " + targetFileName);
copyBytes(myfile, targetFileName);
}
private void copyBytes(File originalFile, String targetFileName) {
try {
FileInputStream in = new FileInputStream(originalFile);
FileOutputStream out = new FileOutputStream(targetFileName);
int c;
while ((c = in.read()) != -1) {
out.write(c);
}
out.close();
in.close();
} catch (Exception e) {
log("IOException Error", e);
}
}
private void log(String msg, Exception e) {
Logger.getAnonymousLogger().log(Level.WARNING, msg, e);
}
public static void main(String[] args) throws Exception {
FileServerClient client = new FileServerClient();
client.receiveFileFromServer();
}
}

You could make an asynchronous request from the client. Lets assume that you client is an html page. When the user selects a report and clicks on 'submit' you could fire an ajax request with the report parameters (jquery can be useful for this). It would be good to keep a section on the user homepage that says something like 'prepared reports'. The client can then goto the prepared report section to download the report. As specified in the comments above, you may also have to implement a popup that informs the user that the requested report is ready. the popup is shown when the ajax requests returns successfully. However, the client may have logged out by the time the report finishes, so it may be a good idea to make the download link available again in the 'prepared reports' section when the user logs in.

Java: send more than one image file from client to server using socket

I want to send more than one image file from client to server for this I write the code in my application, but it will send only one image.
In client application one frame is there and in server application also there is a frame to start/stop the server.
One more problem is there when Client application send the image file then this image file shown on server computer but when I try to open this image file then nothing is there but when I close server application(server frame) then I am able to see the image.
code:
client site:
public void sendPhotoToServer(String str){ // str is image location
try {
InputStream input = new FileInputStream(str);
byte[] buffer=new byte[1024];
int readData;
while((readData=input.read(buffer))!=-1){
dos.write(buffer,0,readData); // dos is DataOutputStream
}
} catch (FileNotFoundException e) {
} catch (IOException e) {
}
}
In server side this code is running into thread:
public void run() {
while (true) {
try {
byte[] buffer = new byte[8192];
fis = new FileOutputStream("C:\\"+(s1++)+".jpg"); // fis is FileOutputStream
while ((count = in.read(buffer)) > 0){ //count is a integer and 'in' is InputStream
fis.write(buffer, 0, count);
fis.flush();
}
} catch (Exception e) {}
}
}
Problem:
only 1st image is copying which is send by the client.
I am able to see this Image only when I close the server application.
no exception is there and i call sendPhotoToServer method in other class consecutively to send all the image file as:
if (photoSourcePath != null) {
clientClass.sendPhotoToServer(photoSourcePath+"\\"+rowData.get(5));
}

Your server side should stop the thread when its job is done. The while loop just keeps running forever and keeps the stream open (that's why you see the image when you shut down the server, the threads only stops then).
Try changing the server side to this:
public void run() {
boolean processing = true;
while (processing) {
try {
byte[] buffer = new byte[8192];
fis = new FileOutputStream("C:\\" + (s1++) + ".jpg"); // fis is
// FileOutputStream
while ((count = in.read(buffer)) > 0) { // count is a integer
// and 'in' is
// InputStream
fis.write(buffer, 0, count);
fis.flush();
}
processing = false;
} catch (Exception e) {
processing = false;
}
}
}