We Keep Coding

Webhook implementation | Doubts?

There is a client to my server, which is calling a GET API to know all updates regarding a particular thing very frequently (let's say once every 5 seconds). Due to this there is unnecessary network calls are landing on my server even if I don't an an update to share.
I reached a decision to replace above approach with web-hooks, where I'll call there POST API whenever I have an Update to share instead of letting them put unnecessary load on my server.
What I understand about Web-hooks:
A web-hook is like a reverse API which POSTs the updates to the server (The client which was calling our application frequently to know/GET updates).
My client has to implement a Web-hook client, which is basically a POST API, I'll be calling whenever there's an event/update to be shared.
I need to call their POST API using REST template wherever there's a new event.
What I don't understand:
Is there anything called a Webhook server? If yes, how do I create/implement it? Any references?
Is it just a reverse API, or is there anything special which both the server and the client needs to handle?

Webhooks are usually used to notify other service that some event occurred on your side. This is made by standard HTTP request sent to some URL.
There is no specific thing like Webhook Server. This is standard application that sends request to someone proactively.
Should both services communicating with webhooks handle something special - this depends on your architecture, there is no "standard" here. You can implement any sort of retry or mechanism to validate that the other side received the information. Im most cases it is assumed that webhooks should be idempotent, so if two same webhooks are sent, the other side should not repeat their action.

JavaEE 6 Servlet for long polling

I've tried to create async-servlet for long-polling, using this tutorial on my Glassfish server, however it didn't work well, the POST requests were too long (timed out) and GET returned sometimes void response with "Error parsing XML file" firebug message. Can somebody please provide simple async-servlet for handling long-polling requests?

It is better you move away from the servlet based approach if you want to implement a long polling solution. Netty is very good choice for such applications. Just build a simple HTTP instance with child.keepAlive option in bootstrap kept to true (server doesnt close off the client connections). That way you don't need to deal with the servlet containers time out and connection handling intricacies. Netty is an asynchronous IO framework and should give you great performance matrix.

SOAP web service callback architecture?

I am quite new to web services, JAX-WS etc. so maybe noob question...
So, I want to implement a web service to make two systems communicate. The "client" system is interested in events that are generated on the "server" system. But the "client system" is itself a server for a different app. The server is Java (WAR in tomcat). The client is .Net.
There should be just one client system, but several client processes inside the client system, each interested in distinct categories of events.
I will implement the server-side, and a test client. Somebody else will implement the .Net code.
The running sequence should be along this line :
Server is running...
Client initiates conversation, "registers" to the server, and requests some initial data.
Server keeps a list of registered clients' endpoints
In the server there is a listener that is notified when certain events happen. It will then go through the list of registered clients and forwards the event to each of them
At some point, the client can "unregister" no notify the server that it doesn't want to receive events any more.
First, does it sound like something reasonably doable ?
And is there a standard built-in mechanism, using SOAP (JAX-WS on the server, whatever is available with .Net n the client) - that the server can use to obtain a callback endpoint from the client ?
For example, I did something very similar using RMI, in this case the client can just send a remote reference to itself, that the server can just store ant refer to later.
Finally, is there a standard library to store endpoints references, make (collective) callbacks, and maybe keep the list up-to-date, removing the clients that don't respond so some "ping" call ?
Note for clarity : I need more than just asynchronous method with callback: one message from the client will generate many callback messages from server to client.

Seems you wish to implement a notification facility to inform arbitrary anonymous clients.
I suggest you first consider how you would pass the information using SOAP messages. Then you can consider how to achieve this using java - JAX-WS or additional non-standard libraries. The point is there may be significant limitations or assumptions required to transfer SOAP messages. E.g. firewalls might block your HTTP messages, clients might "just clients" with no ability to act in a server role to recieve SOAP notification requests
Note: An async callback mechanism is defined in JAX-WS 2.0, where the service obtains the endpoint reference of the client. This is the same sort of functionality provided by WebLogic/Fusion proprietary solution described by Deepak Bala. Websphere has a similar proprietary async solution. None of these meet your requirements, because they only allow a single response per request.
SOAP Options:
Proprietary SOAP messages - the "100% Do-It-Yourself Option"
You design full SOAP payload schemas and message exchange pattern.
You can push the notification from the server to the client if you know the client's SOAP endpoint address. The client can transfer it's SOAP endpoint address within original SOAP request payload. Sometime later the server can send a SOAP request to the client.
Problems/Assumptions: (1) need a SOAP/HTTP communication path for requests from server-to-client - not guaranteed when firewalls exist; (2) the client needs to be aware of your notification schema - in fact the client needs to act as a Service endpoint to recieve the SOAP notification request. That's two big assumptions IF you are trying to support arbitrary anonymous clients - that's not something SOAP "just supports" both ends need to design all this in detail. In fact to do this in a service typesafe manner, the client should actually declare it's own service WSDL interface so that you can invoke it. (3) As hinted earlier, many clients are "just clients" - they might not have a HTTP server to accept SOAP requests.
So for proprietary "push" notifications to work, both sides need to servers and both need to publish their SOA interfaces.
Alternatively, you can pull the notification to the client. The client can use a notification request to the server that is either blocking or polling. The server can respond with the notification or nothing or error.
Problems/Assumptions: (1) HTTP servers (i.e. the SOAP server) do not support blocking requests, as a rule, meaning you must poll; (2) the client needs to be aware of your notification schema - in fact the client needs to act as a Service endpoint to recieve the SOAP notification request. That's two very big assumptions for an arbitrary anonymous client - that's not something SOAP "just supports" both ends need to design all this in detail. In fact to do this in a service typesafe manner, the client should actually declare it's own service WSDL interface so that you can invoke it.
Same as above, but make include WS-addressing data in SOAP headers to inform either side of the other's endpoint address.
Basically the same Problems/Assumptions as the first option. WS-addressing addresses will help you intelligently route SOAP messages to the right URL address, but no more.
Use WS-notification
This spec was designed for your scenario.
WS-BaseNotification sub-standard would meet your needs. It provides WSDL port definitions for notification producers and consumers. It provides a WS- standards compliant solution for subscription from a consumer to a producer, and a notification from producers to consumers.
Problems/Limitations: (1) It does NOT define the format of notification payloads. The Notification payload is application-domain specific (proprietary design). The standard does not define any “standard” or “built-in” notification situations or messages. (2) It has the same problems with HTTP notifications passing through firewalls as mentioned above.
(3) WS-Notification is not a supported standard of Java EE/JAX-WS (but there are plenty of app servers, open source and commercial, that support it as an extension).
Use a message queuing solution (e.g. JMS) with traffic encapsulated in HTTP
This requires proprietary design of payloads passing between client and server and back - forming contracts between the two sides. An advantage is that the client can be a pure client, with a message listener invoked in a thread when a message is recieved.
Problems/Limitations: (1) It has the same problems with HTTP notifications passing through firewalls as mentioned above. (2) Is a do-it-yourself implementation. (3) Uses more technology then you currently use.
End Result:
You need at least part of your solution to be a proprietary design - the SOAP/WS standards do not meet your full requirements. In theory this proprietary design could leverage a product to provide much of the legwork, BUT the notification schema design would need to be created and integrated by you.
IF you wish to push notifications, you need some sort of contract for notifications passing to the client, the client needs to act as a SOA server, and you need the firewalls openned for your traffic. Most corporations disallow HTTP requests leaving a server and passing to a client - you normally need an extremely good reason to open firewall ports and even then, many corporations will disallow it...
IF you wish to have clients polling for notifications, you just need a basic WSDL interface on the server side that can be called frequently by clients.
Future Option: HTML5 Web Sockets
IF your client is a HTML5 app, then web sockets enabled servers can push traffic to the browser - and there is some chance corporations will open firewalls. SOAP messages could travel over HTTP web sockets, enabling you to push notifications.

Async clients are supported for WSDL based services through the use of polling and callbacks. In your case I think the requirement is relatively more complicated.
The Oracle fusion middleware doc page has a scenario outlined that will help you. It details a method that allows clients to send requests which generate a HTTP 202 (accepted) and the clients then wait for a response on message queues. In your case the scenario can be tweaked from the one shown below.
Initiate several response queues for each category of callback. The clients can filter them by supplying a client and category ID for the queue. This will serve as a callback mechanism for each client or the processes that are governed under each client. The MDB can be backed by a file store or DB store to ensure reliability and one-time delivery.
Of course you do not need Oracle fusionware to implement this. You can use RabbitMQ or Redis (with transactions) to acknowledge receipt of a message on the client. If your client wishes to un-register it make a call and stop listening to the queue.
I'm unaware of any industry standard that will fit your scenario, but I believe this solution should work well for you.

Have you considered the simpler approach of "pub-sub" using a messaging product ?
(Such as MQ, EMS, or ActiveMQ)
The requirements you describe does not seem to fit "classic" request/reply sync/async SOAP Web Service scenarios.
In a Pub/Sub solution, the client(s) subscribe to a topic once, and the publisher(s) (in your case, the Server) can post any number of relevant messages to all subscribers.
As a bonus, most messaging products include support for "durable subscribers", so the client can be off-line at times and receive all messages after re-connection.
In your case, the server could house a (free) ActiveMQ Server... Providing most of the feature you seem to seek.
If you go that way, I suggest you pick a JMS compliant product with support for .Net.

For those getting here from search engines:
You can use a WebHook for Web APIs nowadays, which works essentially as OP described.
In REST for example, the client will have an HTTP endpoint itself that is dedicated to receiving POST events/notifications from the actual server. The client registers his endpoint with the actual server by giving it an URI on his notification endpoint. From that point on, the actual server can POST to the client’s notification endpoint. It is essentially a convoluted callback, if you are familiar with async terminology.
Maybe Java or .Net have libraries for WebHooks by now.
That said, SSE and Websockets standards provide actual push and real-time messages while being compatible with HTTP and most browsers.
Long polling variations were also used in the past, and are now sometimes called Comet as an aggregate.

Web service and server->client notifications

For a web service application, I would like for the server to be able to notify the clients about some events. When a client is launched, he calls one of the WS methods to get some information it needs. Then the server, that stores this information, listens continuously for changes on these information and if there is a change, it notifies the concerned client.
I don't know if a web service is a good solution to my problem? I don't know how it may work concerning the TCP connections, since the server may notify a client after a very long time.
What would be the best architecture to solve this kind of issue?
Thanks
EDIT: I've looked at some discussions that propose to use Comet, but if you think there are simpler and more convenient solution, please let me know. Since I'm starting this project from scratch, I have no limitations.
I can also use a polling model where the clients periodically poll the server for the information they need, but then I need to take into account the load that this model may create on the server. I don't know if web services can support such a load when there are a lot of clients.
I've also looked at the asynchronous functionality provided by Servlet 3.0 but I don't know how it may solve my problem.

Without polling: sockets
With polling and webservices: u should use etag (html).
When a client polls he sends a request with an etag. webservice responds either with 200(ok) and data or 304(not modified). 304 has no body => less trafic

Instead of client polling the server, you could implement a callback method on the client so that when ever server want to publish some changes to the client, the server can use the callback method provided by the client.

I can think of one the two approaches below using web services solution:
Callback: When client invokes the server it leaves its call back url and a id, say correlation id. When the server wants to respond back to the client it will just use the call back url to notify. The Server can use a variety of approaches to process the request asynchronously. Your client need not be a webservice for this, but it should be capable of accepting requests (callback). It can be a servlet etc.
Polling: When client makes a request to the server it receives back a id, say requestid. After specified interval client polls the server with this request id to fetch a response. A reasonable timeout and polling interval based on the processing time would be required.

JAX-WS Asynchonous client techniques for calling web services

I am doing some study on SOAP based web services with JAX-WS, and i am finding very complicated to understand the part regarding to the clients.
If some one could give me a hand explaining me some of the topics, i would appreciate it a lot. I am using a SOA book and also google, but i am getting really confused, so that's why i think here i can get a more accurate answer, focused on what i need.
So we can communicate with web services in a synchronous and an asynchronous way. Synchronous communication has the lack that both sides have to wait for each other until the processing ends.
Asynchronous clients allows us to communicate in an asynchronous manner using 2 programming models: 1- Callback and 2- Polling
Doubt 1: Can someone please explain me in a simple way what are the differences, pros and cons of the 2 models(Callback and Pooling)
When talking about asynchronous communication i also understand that there should be a way for the producer(Web Service), to find the consumer(client) once the processing is finished.That is why we have addressing.
Doubt 2: Could you briefly explain how addressing works? And also How is Addressing related to callback and polling techniques?
Just one more thing. The Dispatch API is also confusing me a lot. Because they keep talking about dynamic clients.
Doubt 3: How is the Dispatch API related to Asynchronous communication if they are talking all the time about dynamic clients?
Doubt 4: The 2 usages modes of the Dispatch API(MESSAGE and PAYLOAD), what are used for, and how are they related to the Asynchronous communication model of JAXWS?

Can someone please explain me in a simple way what are the differences, pros and cons of the 2 models(Callback and Pooling)
Callback: implementation is not straight forward.
- Advantage: this approach is more elegant/systematic comapred to the polling approach.
- Disadvantage: the client should have a mechanism so that the server can call it back using the callback.
Polling: implementation is straight forward.
- Advantage: becuase of it's simplicity of techique, it is universal, even an ajax enabled webpage can use this technique to check if the update to the request is available.
- Disadvantage: huge wasteage of bandwidth, also server client timing issues, like how long should the server hold the response incase polling request from client did not yet come.
2.Could you briefly explain how addressing works? And also How is Addressing related to callback and polling techniques?
Addressing makes it possible for the webservice to understand the information which was shared previously between messaging systems and transport providers only. Consider this usecase, Initially transport used would be HTTP for both sending request as well as response.And then you might want to change a part of this transport, say while sending the response back,only to SMTP protocol. Incase you were using ws addressing, the whole information related to transport and addressing would be part of the soap envolope itself which would make it possible for your webservice to dynamically change the transport to response sending.
Webservices addressing can also be used for callback, from the server to the client. The service informs port type of the callback required by it in the WSDL. the client implements this port type and then informs the service, of the callback endpoint, using the WS addressing.
In simple words, WS addressing adds extra tags to the soap envolope which includes information which would be present in the transport headers only, like from address, to address, action name etc.
3.How is the Dispatch API related to Asynchronous communication if they are talking all the time about dynamic clients?
Future<?> response = dispatch.invokeAsync(T, AsyncHandler);
invokeAsync method is a polling method. The response, returns to the user immediately and may be polled for completion. In the meantime, the client program can do other work.The javax.xml.ws.Response implements the java.util.concurrent.Future interface that is included in J2SE 5.0. The Response object returns the actual response via its get method, which blocks if the response is not ready to be returned.
4.The 2 usages modes of the Dispatch API(MESSAGE and PAYLOAD), what are used for, and how are they related to the Asynchronous communication model of JAXWS?
Messaging modes are not directly related to asynchronous communication. The messaging modes define the amount of information sent in the web service request.
MESSAGE MODE can only be used if SOAP binding is used.