We Keep Coding

WCF MTOM/XOP Client Deserialization Error

This is one of those SO 'questions' that i've already answered, but am posting b/c there seems to be next to zero information out there based on a week of Googling.
TL;DR: WCF MTOM encoded BasicHttpBinding client to external/3rd part, non-.NET web service chokes on XOP processing of MTOM response - basically MTOM encoder seems to be expecting a base64 payload in binary element, but runs in to the ... directive and fails deserializing SOAP/XML to runtime object, thus throwing error in this question's Title.
Error: End element 'MyBinaryData' from namespace 'http://mynamespace' expected. Found element 'xop:Include' from namespace 'http://www.w3.org/2004/08/xop/'
As previously noted, there's not much out there on this topic, i'm guessing b/c MS wrote most of their WCF documentation based on service development, and not so much client (although there is some, to be fair).
I'm not going to go in to the nitty-gritty initial set up b/c i'm about to answer my own question, but i'll preface the answer by saying that this was much more akin to a default configuration of WCF MTOM than not.
Also, i know WCF is old, boring, and no longer actively developed by MS, but it is still supported and there are plenty of uses for it. I actually i didn't have much of a choice and had to find a way to make this work. This is why i'm sharing my findings with anyone else that has to deal with this kind of headache.

TL;DR: check http headers to see if service response is "Transfer-Encoding: chunked" (streamed) to you and if so, use transferMode="StreamedResponse" in your binding configuration.
So after Googling for days with no help, i spun up Fiddler for http traffic capture - this requires your WCF basic http binding configuration to proxy in to Fiddler (http://localhost:8888 by default, i think) and depending on where your target service resides you may or may not need to configure Fiddler's Gateway settings (corporate proxy, etc.).
This allowed me to see the raw text being sent between my client to/from their service; all payloads were coming in just fine, which meant, in my case, that the MTOM/XOP response from the service was being completely transmitted and that the WCF runtime was not interpreting the response correctly. Another critical thing i saw was that the Transfer-Encoding http header was "chunked" and there was no Content-Length header... this meant that the service was streaming the response, as opposed to a buffered response. Now a little side note: MS's WCF MTOM documentation has a call-out saying that you should always use "Buffered" as your transferMode in your binding configuration... but failed to mention that was really only applicable in Services, not necessarily clients!
So naturally, i simply went in to my config file, found system.serviceModel >> bindings >> basicHttpBinding collection, found my specific binding configuration and set transferMode="StreamedResponse" (because the 3rd party service was streaming my response back to my client).

NServiceBus and Java integration

We have several microservices, most of them written in C#, but one written in Java. We're using NServiceBus and RabbitMQ for communication between the .NET services. We have one case where one of the C# Components needs to talk to the Java component with a Request/Reply over RabbitMQ.
When using NServiceBus the reply Queue is labeled with AnyQueueName-1 where the -1 indicates that this is a Reply Queue.
Debugging the Java service, I can see that I'm able to send Messages from the C# Service to the Java service. Once the Java component is finished processing the Message, it's supposed to send back a Message. Here is where all the problems starts.
I can see that it's supposed to send the Message to the correct Queue. The name has been verified within the RabbitMQ Management Tool. But there is not activitiy in this Queue.
No exceptions are thrown on the Java side, which to me sounds like its able to send the message, but where does it go? I do not expect that NServiceBus and Java will work together perfectly, but at least I should see some activity in my expected queue.
The relevant Java-code looks like this:
Message responseMessage = rabbitTemplate.getMessageConverter().toMessage(response, responseProperties);
String replyTo = requestProperties.getReplyTo();
rabbitTemplate.convertAndSend(replyTo, responseMessage, cd);
Where the variable replyTo is the name of the queue that I expect the message to be pushed to.
Can anyone give me a push in the correct direction? Where should I debug next?
Thanks!

This issue is hard to identify without actuall access to the infrastructure.
It seems to me, considering that your Java client successfully publishes to the RabbitMQ broker, but the C# client doesn't react to the messages, that your message serialisation might be missing something.
A few things that you should check:
Take a look at the native integration with RabbitMQ sample
The NServiceBus documentation has a section on how to use native RabbitMQ integration.
You can download the sample and see how the particular guys got it to work.
NServiceBus uses specific header attributes within messages.
One lf these is the NServiceBus.EnclosedMessageTypes, which alows NServiceBus to identify the message type, more specifically to map it to the corresponding C# class implementing IMessage. Also the NServiceBus.MessageIntent header would be needed.
I suggest you check the NServiceBus Message Headers documentation, in your case specially the Reply Headers section. Make sure you add the NServiceBus required headers from your Java client as explained in the RabbitMQ API-Guide. I can't tell which of those headers are mandatory for NServiceBus to work, so trial and error is your friend.
Track Messages published by NServiceBus in RabbitMQ
Compare the messages published using NServiceBus with the ones you publish using your Java client. Here is a link for one possible solution on how to trace RabbitMQ messages and their payload. This way you might identify differences between the two message types and their content.
Use NServiceBus ServiceControl
One of the advantages of NServiceBus is the tools it brings with it. Set up an instance of ServiceControl and use the ServiceInsights to check messages and errors per endpoints. You might find your Java messages listened.
Generally this tooling is great for production environments and I can only recommend it.
If nothing gets it to work, another option for you would be to create a slim REST API in C#, as a publisher gateway, that would take the POST requests and publish the content using NServiceBus to your RabbitMQ. You would be able to reference your project containing your messages and use them as your models for the API.
A possible example:
POST https://busgateway.corporate.com/api/{endpoint}/send/{messagetype}
body:
{
"property": "value",
"publisher": "java client"
}
That way your Java client would send messages using http requests.

Sending mail from java code - the reliable way

It's a pretty common thing to have to send mails from your app and in most cases it's a real pain in the ... you know where.
So what I'm doing is taking Apache Commons Email (which is based on top of the "official" java mail api) and I'm sending out email in the most simple way there is, i.e. without authenticating to a smtp server. I just do a simple MX lookup on the destination hostname, get the MX servers and try to drop my message at the first one (whether the mail gets rejected or not is a completely different issue and I might sometime soon ask a further question about the whole mess up with return-path: vs. from: vs. reply-to: and the way these headers are (not) handled in java). Back to business... So I've just tried to drop my Message at the mail server with the least preference score.
Here is an example: I want to write to recipient#domain.com. The MX Lookup tells me that domain.com is aware of two MX servers and these are e.g. mail1.domain.com with preference 10 and mail2.domain.com with preference 20. The rfc way to do things is to go to the server with the least preference and drop the mail there. So that's what I do.
And finally my problem: What happens if that server is not available in some way or another? It's pretty simple - I go to the other server, but Apache Commons (and I suspect java mail api) doesn't allow me to do just that. The mail.smtp.host variable is rooted inside the props of the session in the message in the email. And I cannot get at it.
So what's the best way to handle this problem? Should I build my Email completely from the top with the new hostname (mail2), or is there some clever way to make this all work in java without much pain?

It sounds if you are trying to implement a partial mail server, not just merely sending an e-mail. Routing, relaying, caching and delivery retrials are operations implemented and offered by all mail servers and not usually done by mail clients.
What you should do is either use an (one!) existing mail server, which is configured for you to allow relaying outbound mails or if you don't have access to such a server (which I doubt), setup and operate your own server. You then configure this server in mail.smtp.host and forget all you've learned about DNS lookups, server priorities and your worries about what to do if none of the MX servers are reachable.

SOAP web service callback architecture?

I am quite new to web services, JAX-WS etc. so maybe noob question...
So, I want to implement a web service to make two systems communicate. The "client" system is interested in events that are generated on the "server" system. But the "client system" is itself a server for a different app. The server is Java (WAR in tomcat). The client is .Net.
There should be just one client system, but several client processes inside the client system, each interested in distinct categories of events.
I will implement the server-side, and a test client. Somebody else will implement the .Net code.
The running sequence should be along this line :
Server is running...
Client initiates conversation, "registers" to the server, and requests some initial data.
Server keeps a list of registered clients' endpoints
In the server there is a listener that is notified when certain events happen. It will then go through the list of registered clients and forwards the event to each of them
At some point, the client can "unregister" no notify the server that it doesn't want to receive events any more.
First, does it sound like something reasonably doable ?
And is there a standard built-in mechanism, using SOAP (JAX-WS on the server, whatever is available with .Net n the client) - that the server can use to obtain a callback endpoint from the client ?
For example, I did something very similar using RMI, in this case the client can just send a remote reference to itself, that the server can just store ant refer to later.
Finally, is there a standard library to store endpoints references, make (collective) callbacks, and maybe keep the list up-to-date, removing the clients that don't respond so some "ping" call ?
Note for clarity : I need more than just asynchronous method with callback: one message from the client will generate many callback messages from server to client.

Seems you wish to implement a notification facility to inform arbitrary anonymous clients.
I suggest you first consider how you would pass the information using SOAP messages. Then you can consider how to achieve this using java - JAX-WS or additional non-standard libraries. The point is there may be significant limitations or assumptions required to transfer SOAP messages. E.g. firewalls might block your HTTP messages, clients might "just clients" with no ability to act in a server role to recieve SOAP notification requests
Note: An async callback mechanism is defined in JAX-WS 2.0, where the service obtains the endpoint reference of the client. This is the same sort of functionality provided by WebLogic/Fusion proprietary solution described by Deepak Bala. Websphere has a similar proprietary async solution. None of these meet your requirements, because they only allow a single response per request.
SOAP Options:
Proprietary SOAP messages - the "100% Do-It-Yourself Option"
You design full SOAP payload schemas and message exchange pattern.
You can push the notification from the server to the client if you know the client's SOAP endpoint address. The client can transfer it's SOAP endpoint address within original SOAP request payload. Sometime later the server can send a SOAP request to the client.
Problems/Assumptions: (1) need a SOAP/HTTP communication path for requests from server-to-client - not guaranteed when firewalls exist; (2) the client needs to be aware of your notification schema - in fact the client needs to act as a Service endpoint to recieve the SOAP notification request. That's two big assumptions IF you are trying to support arbitrary anonymous clients - that's not something SOAP "just supports" both ends need to design all this in detail. In fact to do this in a service typesafe manner, the client should actually declare it's own service WSDL interface so that you can invoke it. (3) As hinted earlier, many clients are "just clients" - they might not have a HTTP server to accept SOAP requests.
So for proprietary "push" notifications to work, both sides need to servers and both need to publish their SOA interfaces.
Alternatively, you can pull the notification to the client. The client can use a notification request to the server that is either blocking or polling. The server can respond with the notification or nothing or error.
Problems/Assumptions: (1) HTTP servers (i.e. the SOAP server) do not support blocking requests, as a rule, meaning you must poll; (2) the client needs to be aware of your notification schema - in fact the client needs to act as a Service endpoint to recieve the SOAP notification request. That's two very big assumptions for an arbitrary anonymous client - that's not something SOAP "just supports" both ends need to design all this in detail. In fact to do this in a service typesafe manner, the client should actually declare it's own service WSDL interface so that you can invoke it.
Same as above, but make include WS-addressing data in SOAP headers to inform either side of the other's endpoint address.
Basically the same Problems/Assumptions as the first option. WS-addressing addresses will help you intelligently route SOAP messages to the right URL address, but no more.
Use WS-notification
This spec was designed for your scenario.
WS-BaseNotification sub-standard would meet your needs. It provides WSDL port definitions for notification producers and consumers. It provides a WS- standards compliant solution for subscription from a consumer to a producer, and a notification from producers to consumers.
Problems/Limitations: (1) It does NOT define the format of notification payloads. The Notification payload is application-domain specific (proprietary design). The standard does not define any “standard” or “built-in” notification situations or messages. (2) It has the same problems with HTTP notifications passing through firewalls as mentioned above.
(3) WS-Notification is not a supported standard of Java EE/JAX-WS (but there are plenty of app servers, open source and commercial, that support it as an extension).
Use a message queuing solution (e.g. JMS) with traffic encapsulated in HTTP
This requires proprietary design of payloads passing between client and server and back - forming contracts between the two sides. An advantage is that the client can be a pure client, with a message listener invoked in a thread when a message is recieved.
Problems/Limitations: (1) It has the same problems with HTTP notifications passing through firewalls as mentioned above. (2) Is a do-it-yourself implementation. (3) Uses more technology then you currently use.
End Result:
You need at least part of your solution to be a proprietary design - the SOAP/WS standards do not meet your full requirements. In theory this proprietary design could leverage a product to provide much of the legwork, BUT the notification schema design would need to be created and integrated by you.
IF you wish to push notifications, you need some sort of contract for notifications passing to the client, the client needs to act as a SOA server, and you need the firewalls openned for your traffic. Most corporations disallow HTTP requests leaving a server and passing to a client - you normally need an extremely good reason to open firewall ports and even then, many corporations will disallow it...
IF you wish to have clients polling for notifications, you just need a basic WSDL interface on the server side that can be called frequently by clients.
Future Option: HTML5 Web Sockets
IF your client is a HTML5 app, then web sockets enabled servers can push traffic to the browser - and there is some chance corporations will open firewalls. SOAP messages could travel over HTTP web sockets, enabling you to push notifications.

Async clients are supported for WSDL based services through the use of polling and callbacks. In your case I think the requirement is relatively more complicated.
The Oracle fusion middleware doc page has a scenario outlined that will help you. It details a method that allows clients to send requests which generate a HTTP 202 (accepted) and the clients then wait for a response on message queues. In your case the scenario can be tweaked from the one shown below.
Initiate several response queues for each category of callback. The clients can filter them by supplying a client and category ID for the queue. This will serve as a callback mechanism for each client or the processes that are governed under each client. The MDB can be backed by a file store or DB store to ensure reliability and one-time delivery.
Of course you do not need Oracle fusionware to implement this. You can use RabbitMQ or Redis (with transactions) to acknowledge receipt of a message on the client. If your client wishes to un-register it make a call and stop listening to the queue.
I'm unaware of any industry standard that will fit your scenario, but I believe this solution should work well for you.

Have you considered the simpler approach of "pub-sub" using a messaging product ?
(Such as MQ, EMS, or ActiveMQ)
The requirements you describe does not seem to fit "classic" request/reply sync/async SOAP Web Service scenarios.
In a Pub/Sub solution, the client(s) subscribe to a topic once, and the publisher(s) (in your case, the Server) can post any number of relevant messages to all subscribers.
As a bonus, most messaging products include support for "durable subscribers", so the client can be off-line at times and receive all messages after re-connection.
In your case, the server could house a (free) ActiveMQ Server... Providing most of the feature you seem to seek.
If you go that way, I suggest you pick a JMS compliant product with support for .Net.

For those getting here from search engines:
You can use a WebHook for Web APIs nowadays, which works essentially as OP described.
In REST for example, the client will have an HTTP endpoint itself that is dedicated to receiving POST events/notifications from the actual server. The client registers his endpoint with the actual server by giving it an URI on his notification endpoint. From that point on, the actual server can POST to the client’s notification endpoint. It is essentially a convoluted callback, if you are familiar with async terminology.
Maybe Java or .Net have libraries for WebHooks by now.
That said, SSE and Websockets standards provide actual push and real-time messages while being compatible with HTTP and most browsers.
Long polling variations were also used in the past, and are now sometimes called Comet as an aggregate.

how to implement Java socket communication on Tomcat

I currently have an TCP Java socket communication implementation in which I have a server that is listening to a port (let's say port 5478). Then I need an Android client to remotely connect to the Java server and send a SQL query, than will then be executed on the server side database and then I want to send a list of results back to the Android client (already implemented with a custom Java class named Result that implements Serializable). I do this by sending an ArrayList of Result to the Android client. The Java server is always listening to the port and supports multiple clients trough multiple Threads. How can I migrate this implementation to a more secure platform and what is the best way to do it? I don't need to respect HTTP protocol to afford this communication. Is Tomcat the best solution?
Thanks

I would use Servlet3.0 as part of tomcat.
Then from android you just have to send http requests to the server using a URL and the servlet can database them. You can also serialize the data as well if you need to.
I hope that answers your question.
~ Dan
//EDIT:
Once you have set up eclipse and tomcat, you can start writing servlets. First - you have to configure the server to use servlets for certain addresses, for example localhost:8080/myServlet - that means that anything you send to local host triggers the servlet. The code for your first servelet looks like this:
public class ExampServlet extends HttpServlet {
public void doPost(HttpServletRequest request,
HttpServletResponse response)
throws ServletException, IOException
{
Your doPost method is what gets called when you perform a http post request on the address the servlet is listening on. Then, all you have to do it put some code in to read the request to get the data out of the message body. Basically you read your request object that gets passed in, and you write to your response object to send the response back to the client. There are plenty of guides out there. I followed something like this to get started:
http://www.coreservlets.com/Apache-Tomcat-Tutorial/tomcat-7-with-eclipse.html
Hope that helps :)
~ Dan

Tomcat is an Servlet container + webserver. If you plan to move to tomcat then you are implicitly moving to http. And yes, if you want a secure communication .. you can create a soap based webservice(apache axis) and host it on https.

I'm not sure how mutch additional security tomcat is able to provide for your application. Two tings come to mind:
Enforcing authentication and some access rules. This is not too hart to implement and heavily depends on the rule quality. However it may help f you use it. It's often replaced by own imlpementations. However, to get securty you need encryption i.e. https. Or it's possible to steel the session and gain the rights bound to it.
Request to file mapping. This in fact somewhat more complicated. You shouldn't code this on your own. It's more complicated than it looks at first sight.
However, one of the biggest security wholes ever is directly executing code you got from somewhere. For example SQL statements. Ok it's secure as long as your databse rights are set perfectly...
Developing a securly encrypted protocol is not simple either.
However, the major win on switching to tomcat (or whatever) might be scaleability for free. And I think implementing servlets is much simpler than programming against sockets. And there are many great to tools fo working with http(s) though ven it might be more complicated than yours, it's pretty simple to deal with.
Unfortunately I can't answer our question. I don't know what's the best solution is. But I think there's at least some potential for wins.