We Keep Coding

REST Server to client communication

I'm developing a Java API for an Adndroid app in Spring. Right now my API is 100% REST and stateless. For the client to receive data, it must send a request first.
However, what I need is the server to send data to the to the client /not the client to the server fisrt/ whenever it is ready with it's task.
I think that some kind of session must be created between the two parties.
My question is: How can I achieve this functionality of the SERVER sending data to the CLIENT when it's ready with it's task? /It is unknown how long the task will take./
What kind of API should I develop for this purpose?
One idiotic workaround is sending a request to the server every n seconds but I'm seeking for a more intelligent approach.

There are multiple options available. You can choose what suits best for you.
Http Long Polling - In this, server holds the request until it's ready with its task (in your case). Here, you don't have to make multiple requests every few seconds (Which is Http Polling).
Server Sent Events - In this, server sends update to the client without long-polling. It is a standardized part of HTML 5 - https://www.w3.org/TR/eventsource/
Websockets - Well, websockets work in duplex mode and in this a persistent TCP connection is established. Once TCP connection is established, both server and client sends data to and fro. Supported by most modern browsers. You can check for Android Websocket Library like autobahn and Java websocket.
SockJs - I would recommend to go with this option instead of plain WebSocket. http://docs.spring.io/spring/docs/current/spring-framework-reference/htmlsingle/#websocket-fallback-sockjs-enable

Implementing a protocol exchanging four messages within a single persistent connection

I have a complete implementation of a protocol where four messages are exchanged between the client (a native Android application) and the server (a standalone Java server) in the following way using a persistent connection through Java sockets:
(client->server): message1
(server->client); message2
(client->server): message3
(server->client): message4
Between sending each message, both client and server have to do heavy mathematical (cryptographic) operations (pairing-based computations over elliptic curves).
This project works properly in my local development machine using sockets and mantaining opened this socket from message1 to the message4 between the Android app and the Java server. Now, I need to do the same with Google AppEngine, but since it does not allow opening sockets, I do not know how can I do it. I already checked the Channel and XMPP APIs, but I do not know whether my use-case applies to that APIs. Is it the right method using Channel and XMPP APIs from AppEngine? Is it possible to emulate the functionality implemented in my local machine through sockets on AppEngine?
Thank you for your response.

Google App Engine doesn't support persistent connections.
You will need to significantly re-design your protocol to run over HTTP.
As an example, message1 can be sent over an HTTP request, message2 can be returned with the matching HTTP response. At that point, your socket connection ends.
You'll have to issue a second HTTP request to open a new socket with message3, and you can return message4 with the second HTTP response.
You can "connect" the first and second HTTP request by using an HTTP session. A session is basically an id with extra data stored on the server side. You'd create the session in the first HTTP request, and pass it as a parameter to the second HTTP request. The server can look up the session id and the associated data when processing the second request.
You can find more info about sessions on SO: How to use session on Google app engine
The XMPP API will not help you, it's for communicating between the GAE server-side code and other XMPP clients that use HTTP as a communcation protocol.
The Channel API can be used to send data from the server->client, but it's actually implemented as an HTTP long poll. Multiple long HTTP requests are used, and you are not guaranteed to have a single socket that stays open; multiple sockets are opened and closed in the process. It will be more complicated that what I described above, and more expensive.

How to get Acknowlegement in TCP communication in Java

I have written a socket program in Java. Both server and client can sent/receive data to each other. But I found that if client sends data to server using TCP then internally TCP sends acknowledgement to the client once the data is received by the server. I want to detect or handle that acknowledgement. How can I read or write data in TCP so that I can handle TCP acknowledgement. Thanks.

This is simply not possible, even if you were programming in C directly against the native OS sockets API. One of the points of the sockets API is that it abstracts this away for you.
The sending and receiving of data at the TCP layer doesn't necessarily correlate with your Java calls to send or receive data. The data you send in one Java call may be broken into several pieces which may be buffered, sent and received independently, or even received out of order.
See here for more discussion about this.

Any data sent over a TCP socket is acknowledged in both directions. Data sent from client to server is the same as data sent from server to client as far as TCP wire communications and application signaling. As #Eric mentions, there is no way to get at that signaling.
It may be that you are talking about timing out while waiting for the response from the server. That you'd like to detect if a response is taking too long. Is it possible that the client's message is larger than the server's response so the buffering is getting in the way of the response but not the initial request? Have you tried to use non-blocking sockets?
You might want to take a look at the NIO code if you have not already done so. It has a number of classes that give you more fine grained control over socket communications.

This is not possible in pure Java since Java's network API all handles socket, which hides all the TCP details.
You need a protocol that can handle IP-layer data so you can get TCP headers. DLPI is the most popular API to do this,
http://www.opengroup.org/onlinepubs/9638599/chap1.htm
Unfortunately, there is not Java implementation of such network. You have to use native code through JNI to do this.

I want to detect or handle that acknowledgement.
There is no API for receiving or detecting the ACKs at any level above the protocol stack.
Rethink your requirement. Knowing that the data has got to the server isn't any use to an application. What you want to know is that the peer application has received it, in which case you have to get the peer application to acknowledge at the application protocol level.

Redirect a TCP connection

I have something like a proxy server (written in java) running between my clients and the actual video server (made in c++). Everything the clients send goes through this proxy and is then redirected to the server.
It is working fine, but I have some issues and think it would be better if I could make this proxy server only to listen to the clients requests and then somehow tell the server that a request has been made from the client side, and that it is supposed to create a connection with the client directly.
Basically in the TCP level what I want to happen is something like this:
1- whenever a client sends a SYN to my proxy, the proxy just sends a message to the real server telling the ip and port of the client.
2- The server would then send the corresponding SYN-ACK to the specified client creating a direct connection between client and server.
The proxy would then be just relaying the initial requests (but not the later data transfer) to the actual server. I just don't know if that is possible.
Thank you very much
Nelson R. Perez

That's very much the way some games (and Fog Creek CoPilot) do it, but it requires support on both the server and the client. Basically the proxy has to say to the client and server "try communicating with the directly on this ip and this port" and if they can't get through (because one or both is behind a NAT or firewall), they fall back to going through the proxy.
I found this good description of "peer to peer tcp hole punching" at http://www.brynosaurus.com/pub/net/p2pnat/

Does the proxy and server lives on the same machine? If so, you can pass the connection to the server using Socket Transfer or File Descriptor Passing. You can find examples in C here,
http://www.wsinnovations.com/softeng/articles/uds.html
If they are on the different machines, there is no way to pass connection to the server. However, it's possible to proxy the IP packets to server using VIP (Virtual IP). This is below socket so you have to use Link layer interface, like DLPI.

You don't have control of TCP handshake in userland like that. This is what firewalls/routers do but it all happens in the kernel. Take a look at the firewalling software for your platform - you might not even have to code anything.

web server sending command to a J2ME app

Can a J2ME app be triggered by a message from a remote web server. I want to perform a task at the client mobile phone as soon as the J2ME app running on it receives this message.
I have read of HTTP connection, however what I understand about it is a client based protocol and the server will only reply to client requests.
Any idea if there is any protocol where the server can send a command to the client without client initiating any request?. How about Socket/Stream based(TCP) or UDP interfaces?.

If the mobile device doesnt allow you to make TCP connections, and you are limited to HTTP requests, then you're looking at implementing "long polling".
One POST a http request and the web-server will wait as long time as possible (before things time out) to answer. If something arrives while the connection is idling it can receive it directly, if something arrives between long-polling requests it is queued until a request comes in.
If you can make TCP connections, then just set up a connection and let it stay idle. I have icq and irc applications that essentially just sit there waiting for the server to send it something.

You should see PushRegistry feature where you can send out an SMS to a specific number have the application started when the phone receives that SMS and then make the required HTTP connection or whatever. However, the downside of it is that you might have to sign the application to have it working on devices and you also need an SMS aggregator like SMSLib or Kannel

You can open socket connection and implement "Hide" (or "Minimize") functionality in your app. Call this to hide:
Display.getDisplay(MyMIDlet.instance).setCurrent(null);
Listen to the server in a loop, and if you receive some message, popup the applicaion by calling this from canvas:
Display.getDisplay(MyMIDlet.instance).setCurrent(this);
But it dosen't work on all devices.

Socket push are supported by j2me. But it could work only if your server could deliver data to your mobile phone. Most likely that operator gateway don't allow to do this.
Maybe it would be possible if your mobile has static external IP address - some operators could provide this for $$.