I can't seem to find any resources/tutorials on RSocket, other than just reading their code on GitHub, which I don't understand.
I have a file's path on my server: String serverFilePath;
I'd like to be able to download it from my client (using RSocket's Aeron implementation, preferably). Does anyone know how to do this using RSocket?
Thanks in advance.
I work on RSocket, and wrote a large portion of the java version including the Aeron transport.
I wouldn't recommend using the Aeron implementation currently. There's a couple ways you can send files:
Using a requestChannel to push the data to a remote server.
Use requestChannel or requestStream to stream bytes to a client.
Here's an example using requestStream:
public class FileCopy {
public static void main(String... args) throws Exception {
// Create a socket that receives incoming connections
RSocketFactory.receive()
.acceptor(
new SocketAcceptor() {
#Override
// Create a new socket acceptor
public Mono<RSocket> accept(ConnectionSetupPayload setup, RSocket sendingSocket) {
return Mono.just(
new AbstractRSocket() {
#Override
public Flux<Payload> requestStream(Payload payload) {
// Get the path of the file to copy
String path = payload.getDataUtf8();
SeekableByteChannel _channel = null;
try {
_channel = Files.newByteChannel(Paths.get(path), StandardOpenOption.READ);
} catch (IOException e) {
return Flux.error(e);
}
ReferenceCountUtil.safeRelease(payload);
SeekableByteChannel channel = _channel;
// Use Flux.generate to create a publisher that returns file at 1024 bytes
// at a time
return Flux.generate(
sink -> {
try {
ByteBuffer buffer = ByteBuffer.allocate(1024);
int read = channel.read(buffer);
buffer.flip();
sink.next(DefaultPayload.create(buffer));
if (read == -1) {
channel.close();
sink.complete();
}
} catch (Throwable t) {
sink.error(t);
}
});
}
});
}
})
.transport(TcpServerTransport.create(9090))
.start()
.subscribe();
String path = args[0];
String dest = args[1];
// Connect to a server
RSocket client =
RSocketFactory.connect().transport(TcpClientTransport.create(9090)).start().block();
File f = new File(dest);
f.createNewFile();
// Open a channel to a new file
SeekableByteChannel channel =
Files.newByteChannel(f.toPath(), StandardOpenOption.CREATE, StandardOpenOption.WRITE);
// Request a stream of bytes
client
.requestStream(DefaultPayload.create(path))
.doOnNext(
payload -> {
try {
// Write the bytes received to the new file
ByteBuffer data = payload.getData();
channel.write(data);
// Release the payload
ReferenceCountUtil.safeRelease(payload);
} catch (Exception e) {
throw new RuntimeException(e);
}
})
// Block until all the bytes are received
.blockLast();
// Close the file you're writing too
channel.close();
}
}
There is now a resumable file transfer example here
https://github.com/rsocket/rsocket-java/commit/d47629147dd1a4d41c7c8d5af3d80838e01d3ba5
Related
I have an endpoint that its purpose is to receive a csv file, make a couple of changes to its name and then send this to a method to upload all the data it contains in a single file to Google Cloud as plain text.
The file can have more than 100,000 records, so when parsing it I have to save all the data in a variable and after that save it in Google Cloud. Today I can do it, but I am overwriting all the time the same file because I don't know how to indicate in the method to wait for the complete process of the subscribe and after that upload the file, so every time data is added to the array the file is uploaded again.
Although the method complies with the idea, I want to improve the performance of this, since to upload a file of only 2 mb with 100,000 records is taking approximately 15 minutes. Any idea?
private Storage storage;
private void uploadToGoogleCloudStorage(FilePart filePart, BlobInfo blobInfo) throws IOException {
try (ByteArrayOutputStream bos = new ByteArrayOutputStream()) {
filePart.content()
.subscribe(dataBuffer -> {
byte[] bytes = new byte[dataBuffer.readableByteCount()];
dataBuffer.read(bytes);
DataBufferUtils.release(dataBuffer);
try {
bos.write(bytes);
storage.createFrom(blobInfo, new ByteArrayInputStream(bos.toByteArray()));
} catch (IOException e) {
e.printStackTrace();
}
});
}
}
Finally i get the solution. I change the subscribe to map, then i get the last response from the flux and with that i subscribe the response to upload the data to google cloud store with the Storage interface (package from google to use their api)
private Storage storage;
private void uploadToGoogleCloudStorage(FilePart filePart, BlobInfo blobInfo) throws IOException {
try (ByteArrayOutputStream bos = new ByteArrayOutputStream()) {
filePart.content()
.map(dataBuffer -> {
byte[] bytes = new byte[dataBuffer.readableByteCount()];
dataBuffer.read(bytes);
DataBufferUtils.release(dataBuffer);
try {
bos.write(bytes);
} catch (IOException e) {
e.printStackTrace();
}
return bos;
}).last().subscribe(data -> {
try {
storage.createFrom(blobInfo, new ByteArrayInputStream(bos.toByteArray()));
} catch (IOException e) {
e.printStackTrace();
}
});
}
}
I'm trying to build a server client application using jetty. I have setup a jetty server and configured websockets. Sending text messages works fine between client and server. But how binary data as inputstream could be sent from client endpoint. I cannot find any snippets regarding websocket client. Below is what i have tried
ServerEndPoint:
#OnMessage
public void handleBinaryMessage(InputStream input, Session session) {
logger.info("onMessage::inputstream");
try {
byte[] buffer = new byte[2048];
try (OutputStream output = session.getBasicRemote().getSendStream())
{
int read;
while ((read = input.read(buffer)) >= 0)
output.write(buffer, 0, read);
}
} catch (IOException e) {
e.printStackTrace();
}
ClientEndpoint:
#OnOpen
public void onOpen(Session s) throws IOException {
logger.info("Client Connected ... " + s.getId());
this.session=s;
session.getBasicRemote().sendText("Ping from client");
// size of the file 200~500MB
File source= new File("/tmp/Setup.exe");
try(InputStream input = new FileInputStream(source)) {
session.getAsyncRemote().sendObject(input);
}
}
Any help is appreciated
EDIT:
I have modified clientendpoint and serverendpoint. Trying to send data as chunks but the zip file is partial or sometimes even very smaller than source file.
source size : 1.5gb
after writing data from buffer using stream : 20kb
#ClientEndpoint
private static void sendFileToRemote(File source) throws FileNotFoundException, IOException {
// TODO Auto-generated method stub
Session session=null;
final WsClient wc = new WsClient("ws://localhost:2714/events/","test");
session=wc.getSession();
try (
InputStream inputStream = new FileInputStream(source);
) {
byte[] chunk = new byte[102400];
int chunkLen = 0;
while ((chunkLen = inputStream.read(chunk)) != -1) {
session.getAsyncRemote().sendBinary(ByteBuffer.wrap(chunk, 0, chunkLen));
}
} catch (IOException ex) {
ex.printStackTrace();
}
}
#serverendpoint
#ServerEndpoint("/events/")
public class EventListener {
static Logger logger = Logger.getLogger(Initservice.class.getName());
private OutputStream os=null;
#OnOpen
public void Init(Session session) throws FileNotFoundException {
this.user_session = session;
logger.info("onOpen:: server" + session.getId());
this.os = new FileOutputStream(new File("/tmp/silicon_test.zip"));
logger.info("instantiate zip files");
}
#OnMessage
public void onMessage(Session session, ByteBuffer byteBuffer) throws IOException {
try {
os.write(byteBuffer.get());
} catch(Exception e) {
close();
logger.log(Level.SEVERE,"Exception occured in onMessage :: ", e);
throw e;
}
}
}
The code in your ServerEndpoint looks like it should work fine, however in your ClientEndpoint you are only sending text data to the ServerEndpoint and this can only be read by a server onMessage method configured to receive text messages.
Instead of using session.getRemoteEndpoint().sendText(...) you should use the method session.getRemoteEndpoint().sendBinary(...). This will send the data in binary frames instead of text frames and you will be able to receive it in your servers handleBinaryMessage method.
As for the session.getAsyncRemote().sendObject(input), to make this work you will also need to provide an Encoder.Binary or Encoder.BinaryStream in order to send the object as binary data.
Edit:
WebSocket is a message based protocol and you are sending your data from the file over multiple websocket messages. You could use session.getBasicRemote().sendBinary(ByteBuffer, boolean) to send partial messages and send all the data in the same message.
Or you could try something like this code which may be simpler.
try (InputStream inputStream = new FileInputStream(source))
{
try (OutputStream sendStream = session.getBasicRemote().getSendStream())
{
inputStream.transferTo(sendStream);
}
}
I'm new to Netty. There is a problem about file transfer confusing me for days. I want to send image file from client to server.
The code below is executable. But only I shutdown server forcibly can I open received image file normally. Otherwise, it shows "It looks like you don't have permission to view this file. Check the permissions and try again". So I want to close fileoutputstream when there is no data in ByteBuf using ByteBuf.isReadable(), but the else block in method channelRead in ServerHandler never reach. It's useless.
Besides, if sending text file, it can be open normally when server is alive. I don't want to shutdown server every time after transfer. Please give me some suggestions to solve it.
This is FileClientHandler
public class FileClientHandler extends ChannelInboundHandlerAdapter {
private int readLength = 8;
#Override
public void channelActive(ChannelHandlerContext ctx) throws Exception {
sendFile(ctx.channel());
}
private void sendFile(Channel channel) throws IOException {
File file = new File("C:\\Users\\xxx\\Desktop\\1.png");
FileInputStream fis = new FileInputStream(file);
BufferedInputStream bis = new BufferedInputStream(fis);
for (;;) {
byte[] bytes = new byte[readLength];
int readNum = bis.read(bytes, 0, readLength);
// System.out.println(readNum);
if (readNum == -1) {
bis.close();
fis.close();
return;
}
sendToServer(bytes, channel, readNum);
}
}
private void sendToServer(byte[] bytes, Channel channel, int length)
throws IOException {
channel.writeAndFlush(Unpooled.copiedBuffer(bytes, 0, length));
}
}
This is FileServerHandler
public class FileServerHandler extends ChannelInboundHandlerAdapter {
private File file = new File("C:\\Users\\xxx\\Desktop\\2.png");
private FileOutputStream fos;
public FileServerHandler() {
try {
if (!file.exists()) {
file.createNewFile();
} else {
file.delete();
file.createNewFile();
}
fos = new FileOutputStream(file);
} catch (IOException e) {
e.printStackTrace();
}
}
#Override
public void channelRead(ChannelHandlerContext ctx, Object msg)
throws Exception {
try {
ByteBuf buf = (ByteBuf) msg;
if (buf.isReadable()) {
buf.readBytes(fos, buf.readableBytes());
fos.flush();
} else {
System.out.println("I want to close fileoutputstream!");
buf.release();
fos.flush();
fos.close();
}
} catch (Exception e) {
e.printStackTrace();
}
}
}
Fixing the server side
In the Netty world, there are multiple "events":
channelActive
channelRead
channelReadComplete
channelInactive
exceptionCaught
more...
Of these "events", you probably already knows what channelRead does (since your using it), but another one that you seem to need is the channelInactive. This one is called when the other endpoint shuts the connection down, and you can use it like this:
#Override
public void channelInactive(ctx) {
System.out.println("I want to close fileoutputstream!");
fos.close();
}
#Override
public void channelRead(ChannelHandlerContext ctx, Object msg)
throws Exception {
try {
ByteBuf buf = (ByteBuf) msg;
// if (buf.isReadable()) { // a buf should always be readable here
buf.readBytes(fos, buf.readableBytes());
// fos.flush(); // flushing is always done when closing
//} else {
// System.out.println("I want to close fileoutputstream!");
// buf.release(); // Should be placed in the finally block
// fos.flush();
// fos.close();
//}
} catch (Exception e) {
e.printStackTrace();
} finally {
buf.release(); // Should always be done, even if writing to the file fails
}
}
However, how does the server know the connection has shutdown? At the moment the client does not shut the server down, and instead keep running in the background forever keeping the connection alive.
Fixing the client side
To properly shutdown the connection from the client, we need to call channel.close(), however, we cannot directly insert this before the return line, as this causes a race condition between sending the data, and closing the connection in the network layer, potentially dropping data.
To properly handle these conditions, Netty uses a Future system that allows code to handle events after asynchronous actions happen.
Lucky for us, Netty already has a build in solution for this, we only need to wire it up. To wire this solution up to our code, we have to keep track of the latest ChannelFuture emitted by Netty's write method.
To properly implement this solution, we change sendToServer to return the result of the write method:
private ChannelFuture sendToServer(byte[] bytes, Channel channel, int length)
throws IOException {
return channel.writeAndFlush(Unpooled.copiedBuffer(bytes, 0, length));
}
Then we keep keep track of this return value, and add a listener containing Netty's build in solution when we want to close the connection:
ChannelFuture lastFuture = null;
for (;;) {
byte[] bytes = new byte[readLength];
int readNum = bis.read(bytes, 0, readLength);
// System.out.println(readNum);
if (readNum == -1) {
bis.close();
fis.close();
if(lastFuture == null) { // When our file is 0 bytes long, this is true
channel.close();
} else {
lastFuture.addListener(ChannelFutureListener.CLOSE);
}
return;
}
lastFuture = sendToServer(bytes, channel, readNum);
}
I am making an android application that should send and receive some json files through a wifi direct connection (whenever another device is connected, they both trade all their json files).
This question is mostly about what would be the best practice, since I am fairly new both to android and java.
When a wifi direct connection is established one of the two devices (the group owner) becomes the server and opens a server socket; the other connects to said socket and sends one single json file to the server.
I want the client to send all his json files and then receive all the json files from the server, and I'm wondering how it should be done: how do I tell one file is over? Should I send the lenght of the file ahead, or make the client wait for an acknowledgement for when the server is done reading?
Can I signal "End of Data" (by closing the OutputStream?) to stop the reading loop on the receiving end and then start sending another file?
To have some context, this is currently the code I'm using:
Client side
#Override
protected String doInBackground(Void... params) {
Socket socket = new Socket();
try {
Log.d(TAG, "Opening client socket - ");
socket.bind(null);
socket.connect((new InetSocketAddress(mHost, Constants.PORT)), SOCKET_TIMEOUT);
Log.d(TAG, "Client socket - " + socket.isConnected());
OutputStream stream = socket.getOutputStream();
//There is only one file, so the loop here runs only once
File dir = Utils.graffitiDir(context);
for (File tmp : dir.listFiles()) {
FileInputStream in = new FileInputStream(tmp);
Utils.copyFileInOut(in, stream);
}
Log.d(TAG, "Client: Data written");
return "ok";
} catch (IOException e) {
Log.e(TAG, e.getMessage());
return null;
}
finally {
if (socket != null) {
if (socket.isConnected()) {
try {
socket.close();
} catch (IOException e) {
// Give up
e.printStackTrace();
}
}
}
}
}
Server side
#Override
protected String doInBackground(Void... params) {
try {
ServerSocket serverSocket = new ServerSocket(Constants.PORT);
byte buf[] = new byte[4];
String res;
Log.d(TAG, "Server: Socket opened");
Socket client = serverSocket.accept();
Log.d(TAG, "Server: connection done");
File f = new File(context.getFilesDir(), Constants.DIRECTORY);
File dirs = new File(f.getParent());
if (!dirs.exists())
dirs.mkdirs();
f.createNewFile();
File newf = new File(f, Calendar.getInstance().getTime().toString());
Log.d(TAG, "server: copying files " + f.toString());
InputStream inputstream = client.getInputStream();
//copyFile(inputstream);
Utils.copyFileInOut(inputstream, new FileOutputStream(newf));
serverSocket.close();
return newf.getAbsolutePath();
} catch (IOException e) {
Log.e(TAG, e.getMessage());
return null;
}
}
The copyFileInOut function:
public static boolean copyFileInOut(InputStream inputStream, OutputStream out) {
byte buf[] = new byte[1024];
int len;
long startTime=System.currentTimeMillis();
try {
while ((len = inputStream.read(buf)) != -1) {
out.write(buf, 0, len);
Log.d("copyfile", "I'm writing");
}
out.close();
inputStream.close();
long endTime=System.currentTimeMillis()-startTime;
Log.d("copyfile", "Time taken to transfer all bytes is : " + endTime);
} catch (IOException e) {
Log.d(TAG, e.toString());
return false;
}
return true;
}
As a sidenote, I've seen a couple of similar questions (Sending and receiving files on socket) where the answer suggested to send the length of the file ahead, but I feel like my situation is a bit different, and I don't have the necessary experience to find out what is the best solution. I apologize if this is an obvious question, but I couldn't find an answer by myself.
In your case you should
1. send the number of files you are going to send first,
2. then send length of a file and that file. repeat for all files.
after this the server can also use the same order of sending number of files, the size of a file, that file repeat for all files.
I need to transfer data from android client to c++ server using sockets and using this code.
I am sending "ACK" from Android client and reeving NUL pointer in the Server C++ and then the" ACK".
Receiving message[512] is representing like this:
Data :
messageString[0]= NUL
messageString[1]= A;
messageString[2]= C;
messageString[3]= K;
ACIII Values :
messageString[0]= 0
messageString[1]= 65;
messageString[2]= 67;
messageString[3]= 75;
Why am i receiving NUL in the receive in the c++ side.
As i am debugging and have realized that when i receive a message second time it is coming correct. But for the first time NUL is there for every new type of message.
Error Image:
https://drive.google.com/file/d/0B9Oz1-JlgLdTVVR6UUc4TGZtU2s/view?pli=1
Receive function on C++ server:
char messageString[512];
int result = recv(socket, messageString, strlen(messageString), 0);
Send in Andorid client :
public class AgentOutputChannel
{
Socket clientSocket = null;
OutputStream socketOutputStream = null;
static AgentOutputChannel outputChannel = null;
private AgentOutputChannel()
{
}
private AgentOutputChannel(Socket socket, OutputStream outStream)
{
clientSocket = socket;
socketOutputStream = outStream;
}
public static AgentOutputChannel GetOutputChannel(Socket socket, OutputStream outStream)
{
if(outputChannel == null)
{
outputChannel = new AgentOutputChannel(socket, outStream);
}
return outputChannel;
}
public void Send(byte[] message)
{
try
{
if(!clientSocket.isOutputShutdown())
{
socketOutputStream.write(message);
}
else
{
outputChannel = null;
}
}
catch (IOException e)
{
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
int result = recv(socket, messageString, strlen(messageString), 0);
should instead read
int result = recv(socket, messageString, sizeof(messageString), 0);
Also try printing the values of each byte of message right before
socketOutputStream.write(message);
to make sure that you aren't transmitting the null, ACK