Sending large files from C# client to Glassfish using MTOM and streaming - java

I'm working on a web service that must be able to receive potentially large files (up to 2~3GB).
For the moment I configured a very simple test web service that runs in Glassfish :
package ch.geste.MTOMTest;
import javax.activation.DataHandler;
import javax.jws.WebParam;
import javax.jws.WebService;
import javax.xml.ws.soap.MTOM;
#MTOM
#WebService(name="ServiceMTOM_Svc")
public class ServiceMTOM {
public int upload(#WebParam(name="filename") String filename, #WebParam(name="data") DataHandler data)
{
return 0;
}
}
On the client side (C#) I imported this web service, and the generated proxy method signature to call my upload function is :
public int upload(string filename, byte[] data)
My problem is that I want to stream the data variable, because it can be quite large. That's why on the server side I'm using the DataHandler class instead of a byte array.
But on client side, if I make a naive call to upload method like this (with a 3.8GB file) :
svc.upload(Path.GetFileName(path), File.ReadAllBytes(path));
I get an exception because the C# program tries to load the whole file into memory... I understand that, I would crash too :)
So I think I need to pass a Stream object to the proxy method but I don't know how.
Do you have any idea?
Oh by the way, in my app.config, I set the transferMode="Streamed" in my binding tag.
Many thanks,
Nicolas
EDIT : Crap, I made a duplicate of an unanswered question... really sorry.
See How to use Stream instead of byte[] in C# client with a Java MTOM webservice

Related

Is it possible to use Java WebSockets in Eclipse without Jetty?

I am trying to develop a web server using Java's WebSockets. I am using Eclipse with Tomcat 7, so the javax.websocket-jar file is already included.
I created a simple ServerEndpoint
import javax.websocket.OnMessage;
import javax.websocket.server.ServerEndpoint;
#ServerEndpoint("/login")
public class LoginServer {
#OnMessage
public String onMessage(String message) {
User foobar = new User("Foo", "Bar", "Baz");
return foobar.toString();
}
}
Then I opened the console in Firefox and tried to connect:
var ws = new WebSocket('ws://localhost:8080/MyServer/login');
This ends in an error message which says it can't connect (404).
So I searched the web and found this Jetty thing. I don't really want to add an unnecessary layer on top of my software, but I couldn't find a way to connect to my server.
Do I need to use Jetty when programming WebSockets in Eclipse? If so, why?
I fixed it. I wanted to have a minimal example so I wanted to add the methods bit by bit. But I needed the #OnOpen method. Now it works. (Embarrassing...)

how to get the list of jobs in Jenkins using java?

I have downloaded and configured Jenkins in a server, my problem is that i need to access Jenkins through Java to perform some process such as starting a job, returning the current working job and returning list of jobs in the server(all that using Json) i've tried several codes such as this but im getting no results, also i cant find a clear way to achieve that, is there is any clear API and example to do it?
You can use the Jenkins API over XML:
import org.dom4j.io.*;
import org.dom4j.*;
import java.net.*;
import java.util.*;
public class Main {
public static void main(String[] args) throws Exception {
URL url = new URL("http://your-hudson-server.local/hudson/api/xml");
Document dom = new SAXReader().read(url);
for( Element job : (List<Element>)dom.getRootElement().elements("job")) {
System.out.println(String.format("Job %s has status %s",
job.elementText("name"), job.elementText("color")));
}
}
}
A complete example (with sources) can be found here.
If these examples don't work, you might have problems with Jenkins Security (your client must provide login data before it can send the request)or with CSRF protection (you have to retrieve a token before the first request and add this token as a parameter to each request).

mailto URI truncated between Java.Desktop and Windows/MS outlook

I'm trying to create an automated error reporting tool for our Java desktop app. the idea is to make it as easy as possible for customers to send us error reports whenever our application crashes.
Using the Desktop.mail API, I am able to craft messages that can be easily edited and sent from our users, but I'm running into system limitations on several platforms (notably Windows 7 and MS Outlook, which most customers are using)
When I run the example code below, you'll notice that the email message that is displayed truncates the included stack trace. I believe this has something to do with a maximum length of either command lines or URIs in the underlying systems.
Is there a better way to craft an email from an error report that is not subject to this limitation?
import java.awt.Desktop;
import java.io.PrintWriter;
import java.io.StringWriter;
import java.net.URI;
import java.net.URLEncoder;
public class Scratchpad {
public static void main(String[] args) throws Exception {
try {
generateLongStackTrace();
} catch (Error e) {
URI uri = createMailURI(e);
// this will correctly pop up the system email client, but it will truncate the message
// after about 2K of data (this seems system dependent)
Desktop.getDesktop().mail(uri);
}
}
// Will eventually generate a really long stack overflow error
public static void generateLongStackTrace() throws Exception {
generateLongStackTrace();
}
public static URI createMailURI(Error e) throws Exception {
StringBuilder builder = new StringBuilder();
builder.append("mailto:foo#example.com?body=");
// encodes the stack trace in a mailto URI friendly form
String encodedStackTrace = URLEncoder.encode(dumpToString(e), "utf-8").replace("+", "%20");
builder.append(encodedStackTrace);
return new URI(builder.toString());
}
// Dumps the offending stack trace into a string object.
public static String dumpToString(Error e) {
StringWriter sWriter = new StringWriter();
PrintWriter writer = new PrintWriter(sWriter);
e.printStackTrace(writer);
writer.flush();
return sWriter.toString();
}
}
there are length limitations wrt admissible urls in ie and the length of a windows command line (see here, here, here and here) - i seems you run into one of these (though i admit that i have not rigorously checked).
however i think it's a plausible assumption that even if you could worm your way around the said limits the length of a generic transmission buffer between desktop applications (unless you use a dedicated api for remote controlling the target app) will be restricted somehow without a loophole.
therefore i'd suggest one of the following strategies:
distribution through a web server.
upload the data to be mailed to a web server instead using the html form file upload technique.
basically you have to forge a POST request a payload with content type set to 'multipart/form-data'. your content will need some wrapper data to conform syntactically with this mime type.
the actual transmission can be instigated by means of the WinHttpRequest COM object under windows or the curl command line program from everywhere else.
server side processing can be delegated to a suitable cgi handler which eg. might produce a (short) link to download the data fom the web server.
this link may be part of the http response to the upload request or you generate it client-side in the proper format for publishing it on the web server unaltered.
pro:
this scheme is feasible - i have repeatedly applied it in enterprise projects. data transmission can be secured through https.
con:
requires a web server to implement
send a mail using an attachment (for some details see here):
save the body of your message to some file on the desktop.
generate a mailto-link that references an attachment (instead of the bulk of your body)
any decent mail client will be able to show the attachment inline if it has some elementary mime type like 'text/plain'.
on windows platforms you set it by choosing the proper file extension ('.txt')
pro:
simple
con:
file system access on the client platform;
untested (at least by me)
good luck !

VB.NET Consuming web service written for glassfish: SoapHeaderException

Wrote a 'webservice' with Netbeans wizard, runs on glassfish. I added a reference using the wsdl to my .NET client, VB if it makes any difference.
I clearly have no idea what is going on, as I am encountering some brick walls.
The issue is a SoapHeaderException.
System.Web.Services.Protocols.SoapHeaderException: com/mysql/jdbc/Connection
at System.Web.Services.Protocols.SoapHttpClientProtocol.ReadResponse(
SoapClientMessage message, WebResponse response, Stream responseStream,
Boolean asyncCall)
at System.Web.Services.Protocols.SoapHttpClientProtocol.Invoke(
String methodName, Object[] parameters)
at WSClient.WSClient.localhost.DatabaseGateService.createCustomerTable(String xml)
in C:\Project\WSClient\Web References\localhost\Reference.vb:line 40
at WSClient.USAHWSClientConsumer.TestCustomer() in
C:\Project\WSClient\Client\WSConsumer.vb:line 22
The web service itself is simple:
#WebService()
public class DatabaseGate {
private MySQLManagerImp manager;
public DatabaseGate(){
manager = new MySQLManagerImp();
}
#WebMethod(operationName = "createCustomerTable")
public void createCustomerTable(#WebParam(name = "xml") String xml) {
manager.createCustomersTable(xml);
}
}
It takes an xml string, as I did not want to pass in an abomination of arguments.
I attempt to consume the service by simply instantiating the web reference:
Dim ws As localhost.DatabaseWS = New localhost.DatabaseWS
// Create the xml string
Dim qbCustomerQueryRS As String = qbQuery.GetCustomerQueryXML()
Dim processedCustomerXML As String =
customerResponseParser.GetAllCustomerDatabaseFriendlyXML(qbCustomerQueryRS)
ws.createCustomerTable(processedCustomerXML)
I've tried writing the string in a soap envelope, but still receive the same message. So passing a string is kaputt, as it should be; why would the WS know to parse a string, and simply instantiating and calling the method from the object as if it were local isn't working the way I think it does.
What is happening?
Sounds like the WSDL references com/mysql/jdbc/Connection, which is not a class known on the .NET side. If you have control over the Web Service, add annotations to avoid serialization of external class references (like com/mysql/jdbc/Connection). If you don't, simply download the WSDL to a text file, edit it manually to remove such classes/attributes, and re-create the reference pointing to the edited file. You can change the endpoints in Web.config later.
As it turns out, the reason I was receiving
System.Web.Services.Protocols.SoapHeaderException: com/mysql/jdbc/Connection
was similar to Diego's answer (thank you for pointing me in the right direction): A reference problem.
I assumed my WS deployment worked correctly because I had tested the method that executed, but only assuming the data I needed successfully transmitted over the wire.
Testing another angle using glassfish revealed:
Service invocation threw an exception with message : null; Refer to the server log for more details
Checking the server log, the answer was obvious:
java.lang.NoClassDefFoundError: com/mysql/jdbc/Connection
I had forgotten to add the jar to com/mysql/jdbc/Connection.

Can a web service return a stream?

I've been writing a little application that will let people upload & download files to me. I've added a web service to this applciation to provide the upload/download functionality that way but I'm not too sure on how well my implementation is going to cope with large files.
At the moment the definitions of the upload & download methods look like this (written using Apache CXF):
boolean uploadFile(#WebParam(name = "username") String username,
#WebParam(name = "password") String password,
#WebParam(name = "filename") String filename,
#WebParam(name = "fileContents") byte[] fileContents)
throws UploadException, LoginException;
byte[] downloadFile(#WebParam(name = "username") String username,
#WebParam(name = "password") String password,
#WebParam(name = "filename") String filename) throws DownloadException,
LoginException;
So the file gets uploaded and downloaded as a byte array. But if I have a file of some stupid size (e.g. 1GB) surely this will try and put all that information into memory and crash my service.
So my question is - is it possible to return some kind of stream instead? I would imagine this isn't going to be terribly OS independent though. Although I know the theory behind web services, the practical side is something that I still need to pick up a bit of information on.
Cheers for any input,
Lee
Yes, it is possible with Metro. See the Large Attachments example, which looks like it does what you want.
JAX-WS RI provides support for sending and receiving large attachments in a streaming fashion.
Use MTOM and DataHandler in the programming model.
Cast the DataHandler to StreamingDataHandler and use its methods.
Make sure you call StreamingDataHandler.close() and also close the StreamingDataHandler.readOnce() stream.
Enable HTTP chunking on the client-side.
Stephen Denne has a Metro implementation that satisfies your requirement. My answer is provided below after a short explination as to why that is the case.
Most Web Service implementations that are built using HTTP as the message protocol are REST compliant, in that they only allow simple send-receive patterns and nothing more. This greatly improves interoperability, as all the various platforms can understand this simple architecture (for instance a Java web service talking to a .NET web service).
If you want to maintain this you could provide chunking.
boolean uploadFile(String username, String password, String fileName, int currentChunk, int totalChunks, byte[] chunk);
This would require some footwork in cases where you don't get the chunks in the right order (Or you can just require the chunks come in the right order), but it would probably be pretty easy to implement.
When you use a standardized web service the sender and reciever do rely on the integrity of the XML data send from the one to the other. This means that a web service request and answer only are complete when the last tag was sent. Having this in mind, a web service cannot be treated as a stream.
This is logical because standardized web services do rely on the http-protocol. That one is "stateless", will say it works like "open connection ... send request ... receive data ... close request". The connection will be closed at the end, anyway. So something like streaming is not intended to be used here. Or he layers above http (like web services).
So sorry, but as far as I can see there is no possibility for streaming in web services. Even worse: depending on the implementation/configuration of a web service, byte[] - data may be translated to Base64 and not the CDATA-tag and the request might get even more bloated.
P.S.: Yup, as others wrote, "chuinking" is possible. But this is no streaming as such ;-) - anyway, it may help you.
I hate to break it to those of you who think a streaming web service is not possible, but in reality, all http requests are stream based. Every browser doing a GET to a web site is stream based. Every call to a web service is stream based. Yes, all. We don't notice this at the level where we are implementing services or pages because lower levels of the architecture are dealing with this for you - but it is being done.
Have you ever noticed in a browser that sometimes it can take a while to fetch a page - the browser just keeps cranking away showing the hourglass? That is because the browser is waiting on a stream.
Streams are the reason mime/types have to be sent before the actual data - it's all just a byte stream to the browser, it wouldn't be able to identify a photo if you didn't tell it what it was first. It's also why you have to pass the size of a binary before sending - the browser won't be able to tell where the image stops and the page picks up again.
It's all just a stream of bytes to the client. If you want to prove this for yourself, just get a hold of the output stream at any point in the processing of a request and close() it. You will blow up everything. The browser will immediately stop showing the hourglass, and will display a "cannot find" or "connection reset at server" or some other such message.
That a lot of people don't know that all of this stuff is stream based shows just how much stuff has been layered on top of it. Some would say too much stuff - I am one of those.
Good luck and happy development - relax those shoulders!
For WCF I think its possible to define a member on a message as stream and set the binding appropriately - I've seen this work with wcf talking to Java web service.
You need to set the transferMode="StreamedResponse" in the httpTransport configuration and use mtomMessageEncoding (need to use a custom binding section in the config).
I think one limitation is that you can only have a single message body member if you want to stream (which kind of makes sense).
Apache CXF supports sending and receiving streams.
One way to do it is to add a uploadFileChunk(byte[] chunkData, int size, int offset, int totalSize) method (or something like that) that uploads parts of the file and the servers writes it the to disk.
Keep in mind that a web service request basically boils down to a single HTTP POST.
If you look at the output of a .ASMX file in .NET , it shows you exactly what the POST request and response will look like.
Chunking, as mentioned by #Guvante, is going to be the closest thing to what you want.
I suppose you could implement your own web client code to handle the TCP/IP and stream things into your application, but that would be complex to say the least.
I think using a simple servlet for this task would be a much easier approach, or is there any reason you can not use a servlet?
For instance you could use the Commons open source library.
The RMIIO library for Java provides for handing a RemoteInputStream across RMI - we only needed RMI, though you should be able to adapt the code to work over other types of RMI . This may be of help to you - especially if you can have a small application on the user side. The library was developed with the express purpose of being able to limit the size of the data pushed to the server to avoid exactly the type of situation you describe - effectively a DOS attack by filling up ram or disk.
With the RMIIO library, the server side gets to decide how much data it is willing to pull, where with HTTP PUT and POSTs, the client gets to make that decision, including the rate at which it pushes.
Yes, a webservice can do streaming. I created a webservice using Apache Axis2 and MTOM to support rendering PDF documents from XML. Since the resulting files could be quite large, streaming was important because we didn't want to keep it all in memory. Take a look at Oracle's documentation on streaming SOAP attachments.
Alternately, you can do it yourself, and tomcat will create the Chunked headers. This is an example of a spring controller function that streams.
#RequestMapping(value = "/stream")
public void hellostreamer(HttpServletRequest request, HttpServletResponse response) throws CopyStreamException, IOException
{
response.setContentType("text/xml");
OutputStreamWriter writer = new OutputStreamWriter (response.getOutputStream());
writer.write("this is streaming");
writer.close();
}
It's actually not that hard to "handle the TCP/IP and stream things into your application". Try this...
class MyServlet extends HttpServlet
{
public void doGet(HttpServletRequest request, HttpServletResponse response)
{
response.getOutputStream().println("Hello World!");
}
}
And that is all there is to it. You have, in the above code, responded to an HTTP GET request sent from a browser, and returned to that browser the text "Hello World!".
Keep in mind that "Hello World!" is not valid HTML, so you may end up with an error on the browser, but that really is all there is to it.
Good Luck in your development!
Rodney

Categories