I use Java SOAP web-service based on Axis2 1.4.1 to exchange data and attachments since years; no matter so far even if big attachments since the web-service implements MTOM.
This till last week, when a webservice WS client started sending huge attachments (around 1GB) as base64 inline text within the SOAP body (no MTOM no SwA).
I was confident that Axis2 attachment caching feature would solve also in this case, but this is not true and OutOfMemory occurs.
After some attempts my understanding is that attachments are cached to disk (and not kept in memory) only if the WS client sends them as MIME parts.If sent as base64 text within the SOAP body they are kept in memory.
I also tried to replace Axis2 with CXF (I'm prepared to change my WS framework if it is the only solution) but I experimented exactly the same behavior.
I'm wondering if I'm missing something or someone having my same issue found a solution.
I cant't believe that my webservice can manage big attachments only if the client implements MTOM.
Thanks in advance for any possible suggestion.
Related
This is one of those SO 'questions' that i've already answered, but am posting b/c there seems to be next to zero information out there based on a week of Googling.
TL;DR: WCF MTOM encoded BasicHttpBinding client to external/3rd part, non-.NET web service chokes on XOP processing of MTOM response - basically MTOM encoder seems to be expecting a base64 payload in binary element, but runs in to the ... directive and fails deserializing SOAP/XML to runtime object, thus throwing error in this question's Title.
Error: End element 'MyBinaryData' from namespace 'http://mynamespace' expected. Found element 'xop:Include' from namespace 'http://www.w3.org/2004/08/xop/'
As previously noted, there's not much out there on this topic, i'm guessing b/c MS wrote most of their WCF documentation based on service development, and not so much client (although there is some, to be fair).
I'm not going to go in to the nitty-gritty initial set up b/c i'm about to answer my own question, but i'll preface the answer by saying that this was much more akin to a default configuration of WCF MTOM than not.
Also, i know WCF is old, boring, and no longer actively developed by MS, but it is still supported and there are plenty of uses for it. I actually i didn't have much of a choice and had to find a way to make this work. This is why i'm sharing my findings with anyone else that has to deal with this kind of headache.
TL;DR: check http headers to see if service response is "Transfer-Encoding: chunked" (streamed) to you and if so, use transferMode="StreamedResponse" in your binding configuration.
So after Googling for days with no help, i spun up Fiddler for http traffic capture - this requires your WCF basic http binding configuration to proxy in to Fiddler (http://localhost:8888 by default, i think) and depending on where your target service resides you may or may not need to configure Fiddler's Gateway settings (corporate proxy, etc.).
This allowed me to see the raw text being sent between my client to/from their service; all payloads were coming in just fine, which meant, in my case, that the MTOM/XOP response from the service was being completely transmitted and that the WCF runtime was not interpreting the response correctly. Another critical thing i saw was that the Transfer-Encoding http header was "chunked" and there was no Content-Length header... this meant that the service was streaming the response, as opposed to a buffered response. Now a little side note: MS's WCF MTOM documentation has a call-out saying that you should always use "Buffered" as your transferMode in your binding configuration... but failed to mention that was really only applicable in Services, not necessarily clients!
So naturally, i simply went in to my config file, found system.serviceModel >> bindings >> basicHttpBinding collection, found my specific binding configuration and set transferMode="StreamedResponse" (because the 3rd party service was streaming my response back to my client).
I am trying to upload a file (and additional fields) using single REST POST request. This request has to be processed asynchronous and form has to be submitted using multipart-form content type (because beside file I am sending few other text fields with certain metadata).
I tried with Apache's org.apache.http.impl.nio.client.CloseableHttpAsyncClient but until now I didn't succeed (when request arrives on the server, fields from the form are not presented). When I do the same using org.apache.commons.httpclient.HttpClient request is processed without problems, so the issue is not on the server side.
Does anybody know how this async 'multipart-form' request could be made? If not possible with CloseableHttpAsyncClient, do you have some other library to suggest?
Thanks in advance...
I just want to give an update. I did some investigation and was able to fulfill my requirements with AsyncHttpClient (AHC)
It works very fast and is based on Netty Framework and NIO Java API.
I'm new to Apache NiFi, and I developed an application to send JSON data to a NiFi endpoint via HTTP POST request.
In Apache NiFi, I have a HandleHTTPRequest processor connected to a PutFile processor then a HandleHTTPResponse processor. I've also tried using a LogAttribute processor. The response code and HTTP message from the NiFi web service is working properly, but I'm trying to make sure my JSON payload is being sent properly. Post is enabled in the HTTPRequest handler. Yet, the only files it's generating are blank and 0kb in size. Any ideas?
Based on the comments above the conclusion is that there was no problem on the side of Nifi.
It's definitely an issue with the Java code, I used postman again after changing some local firewall configurations and it successfully logs any POST requests that are sent to it and puts it in a file
The key step towards resolving the issue was trying to connect with something like Postman as #Binary Nerd and #Brian Bende suggested.
My system:
Server: Java Soap Web Service generated by JAX WS 2.2.3 (-wsgen)
Client: iOS - Soap Web Service generated by SudzC
I am using SudzC on iOS to communicate with a Java web service. I want to upload NSData files from the iOS client to the Java Server or download byte[] files from the Java server to the client. For small amounts of data the web service runs fine, but when the data is greater than 4MB, there are problems. If i send a file which is greater than 4MB a HTTP Internal Error 500 occurred or everything get stuck and my application crashes.
Any suggestions? Perhaps, should I try something else than SudzC?
I know, that to send and receive large files, SOAP has an opportunity called MTOM.
This extracts the base64Binary data from the SOAP message and packages it as separate binary http attachments within the MIME message, in a similar manner to e-mail attachments.
So my problem is, how can i implement this option on the iOS client SudzC generated objective-C program code?
On the java server side the MTOM option is activated, but on the iOS client i have to implement this option?!
Perhaps someone can help?
I use WSClient++ to generate the classes and never had a problem.
http://wsclient.neurospeech.com/
I don't like SudzC, I have any problems when XML return has list of list.
I've used SudzC to upload larger files (20 + megabytes) so the issue probably isn't from SudzC. I remember having an issue with the file upload at the beginning aswell : the server didn't accept anything over X bytes and was returning an error.
However, what i have seen is that sudzC has a lot of issues with memory when uploading large files so i switched to wsdl2objc for file upload
I am using SAAJ at the client side for sending a soap request. I am also attaching a random binary string as an attachment to the SOAP request, using this API.
soapMessage.addAttachment(attachment);
When I sniff this request through wireshark, I can see my attachment outside the <SoapEnv>,
but when this request reaches the client side (which is implemented using the JBoss libraries), I only get the message and not the attachment.
I have not specificed anything in the WSDL related to the attachments.
I can't figure out what can be the problem.
Any pointers would be helpful.
Do you send along the right Content-Type, Content-Transfer-Encoding and Content-ID header values?
http://www.ws-i.org/Profiles/AttachmentsProfile-1.0.html#Value-space_of_Content-Id_Header
Maybe post the generated soap-message