I am writing a web service using Java JDK 1.7 and Jersey Web service Framework. One of the things I need to provide is a way to allow authenticated clients to download certain large data files ( 1-3 GB). Ideally I would like this to be a pause and resume type downloadable option. I tried the jersey multi-part API and was able to get it to work on my client machine upto 400 MB but beyond that it ran into out-of memory issues. I am also worried that the server might fail when faced with simultaneous download requests. Any thoughts on how this can be done? Is Netty an option? Any pointers on how Netty can be integrated into a existing Jersey based web service? Are there other frame works available to help accomplish this? I do have to use java for the web service. Any pointers will be helpful.
If you are getting stuck on out-of-memory issues, you should check how you are handling the data you are downloading. If you are using Jersey's ClientResponse, make sure you are using getEntityInputStream() and not getEntity(). This way, you can stream the data, write it to file, and toss it aside, rather than letting it build up in the Java heap space.
I can't really speak about your simultaneous download concerns, but if you are using the web services framework, then it should be handled properly.
For both issues, more info on your specific implementation, especially code, will help you get a better response.
The server and the client must both support HTTP chunked encoding which allows one to stream data using HTTP. The code below should work with Jersey 2.11.
For downloading large files, try this on the server:
#GET
#Path("/files/{fileName}")
#Produces(MediaType.APPLICATION_OCTET_STREAM)
public StreamingOutput getFile(#PathParam("fileName") final String fileName) throws Exception {
//create instance of StreamingOutput here
return streamingOutput;
}
Try this for a client GET request using steams to download a file.
public String getFileReq(File outFile) throws IOException {
client = ClientBuilder.newClient(new ClientConfig());
client.property(ClientProperties.REQUEST_ENTITY_PROCESSING, "CHUNKED");
WebTarget target = client.target(URI)
OutputStream fileOutputStream = new FileOutputStream(outFile);
InputStream fileInputStream = target.request().get(InputStream.class);
writeFile(fileInputStream, fileOutputStream);
}
public static void writeFile(InputStream fileInputStream, OutputStream outputStream) throws IOException {
try {
byte[] buffer = new byte[1024];
int bytesRead;
while((bytesRead = fileInputStream.read(buffer)) !=-1) {
outputStream.write(buffer, 0, bytesRead);
}
fileInputStream.close();
outputStream.flush();
} catch (IOException e) {
e.printStackTrace();
} finally {
outputStream.close();
}
Related
I'm trying to build a HTTP server in Java out of curiosity.
I know that HTTP uses sockets underneath(correct me if i'm wrong). So started programming initially using ServerSocket class.
public class Server
{
public static void main(String[] args) throws IOException
{
System.out.println("Listening.....");
ServerSocket ss = new ServerSocket(80);
while(true)
{
Socket s = ss.accept();
Scanner sc = new Scanner(s.getInputStream());
while(sc.hasNextLine())
{
String line = sc.nextLine();
if(line.equals(""))
break;
else
System.out.println(line);
}
System.out.println("-------------------------------");
PrintStream ps = new PrintStream(s.getOutputStream());
ps.println("Hello from Server");
s.close();
ps.close();
sc.close();
}
}
}
(I'm using Thread in my actual code to serve multiple users. I've just provided the basic code.)
I'm getting all the headers from the web browser. But how can I send the files and images?
For, simple HTML I can read the file and use PrintStream to print it on the web browser.
But how can I send JavaScript, Images etc to the browser?
HTTP has a protocol to it, you need to follow that protocol. The HTTP 1.1 protocol spec still in wide use is RFC 2616 (though it has officially been replaced with newer RFCs 7230, 7231, 7232, 7233, 7234, and 7235).
In my answer to another question, I show the correct way to read an inbound HTTP request from a Java Socket directly.
When sending a reply back, you can use a PrintStream or PrintWriter to send the response HTTP headers. However, the body content is sent as raw bytes, based on the format specified by the Content-Type and Transfer-Encoding response headers. Typically, you would just send the raw bytes directly to the socket's OutputStream, or at least to a BufferedOutputStream attached to it. If you are sending a pre-existing file from disk, regardless of its type, you could just open an InputStream for the file and then copy its data directly to the socket's OutputStream. If you are generating data dynamically, then you would send the data to the socket's OutputStream using whatever intermediate classes are appropriate. Print... classes are only appropriate for textual data, not binary data, like images.
That being said, Java has its own HttpServer and HttpsServer classes. You should consider using them.
Basically the same way. You should "print" the raw bytes to the socket's OutputStream.
However, for the browser to be able to understand it, you need to shape your response according to the HTTP/1.1 protocol. Specifying a Content-Type header will tell the browser what it is receiving from you. Specifying a Content-Length header will tell the browser how many bytes it is receiving from you. Etc.
I am trying to send a PDF file I generated in my server to my client in my Jersey application. The generation goes just fine, I created the file on my local machine and there is no problem with it.
Problem arises when I try to send it by a stream. When I try to read the outputted file, it's completely empty (sometimes, it's just corrupted, same file same code, but the file is corrupeted instead of blank) however, the amount of page is correct, which is really strange. I then tried to use the stream to create a local file, which worked just fine, so I'm pretty sure the problem isn't with what I put in my stream, it's within my method of sending it to the client. I also tried with different, completly normal, pdf, and I have the same problem.
Some code sample
#Path("/produiretest/{id}")
#GET
#Produces("application/pdf")
public Response rapportStreamTest(#PathParam("id") Long id) throws Exception {
final StreamingOutput file = manager.produireRapportStreamingTest(id);
Response.ResponseBuilder response = Response.ok(file);
response.header("Content-Disposition", "attachment; filename=\"" + "previsualisationFicheSignaletique.pdf");
return response.build();
}
The StreamingOutput implement (even if I think that's correct)
StreamingOutput outStream = new StreamingOutput() {
#Override
public void write(OutputStream outputStream) {
try {
JasperExportManager.exportReportToPdfStream(jasperPrint, outputStream);
outputStream.flush();
outputStream.close();
}
catch (Exception ex) {
}
}
};
return outStream;
Sample from the response I get when I poke the server, obviously broken.
%PDF-1.4
%����
5 0 obj
<>stream
x���1 ��g
?�����
endstream
endobj
6 0 obj
<>]/Intent/Perceptual/Subtype/Image/Height 600/Filter/FlateDecode/Type/XObject/Width 600/SMask 5 0 R/Length 23357/BitsPerComponent 8>>stream
x���{pU��>pLB¥E#�� ��Bu8��ܔ!#% �=P�H)j�NsxAD8"IA��$��i�r�Ŋ0#�~������7#B.{��k=k������������I����_"�����nݺu͚5K�.�;w��I��}��aÆ�����ׯw��ݺu�ܹsǎ۷oߪU�f͚5n���뮫W�^bbb||�U?������__
___��!�s�G�O�p8v=DD�6�֯��{�����cƌy��G�ao�>U�nݸ���<�C��p�8T0�S���k)""�u�ȑ�7���L���R� (���WIDD�(..}{vvvfffZZZӦM=��LoB�P(�EC�P#����""�Ç���ӦM6lX�ػ�
PF%EaQ^���""���B����jƌ�
jܸ1{�
Don't really go where to go from now, any pointer would be appreciated.
Ok after a couple of phone calls I've managed to solve my problem that can be broken down in 2 parts :
1) The way our application structure works, our client apps doesn't directly poke the Tomcat server where the backend app is. We have a Node.js server that serves as a dispatcher for request. Apparently there is a problem within the Node that attempts to interpret the stream with the wrong encoding or something. Will have to contact the team that manages the Node.
2) For some reason the Postman Chrome app seems afflicted by the same problem as our Node server. Using the DHC plugin for Chrome for the backend testing fixed my problems.
So I made a mistake to assume the problem was within the server side of the code because Postman and Client-side gave the same problem, while in reality the result is in between and Postman seems to have a problem too. Not quite sure yet, will have to investigate why it works on DHC and not Postman later.
I'm trying to create an application that download and uploads large files, so I don't want the file contents to be stored in memory.
On the mvc controller side I'm using an http message converter that converts to / from InputStream
#Override
public InputStream read(Class<? extends InputStream> clazz, HttpInputMessage inputMessage) throws IOException,
HttpMessageNotReadableException {
return inputMessage.getBody();
}
#Override
public void write(InputStream t, MediaType contentType, HttpOutputMessage outputMessage) throws IOException,
HttpMessageNotWritableException {
try {
IOUtils.copy(t, outputMessage.getBody());
} finally {
IOUtils.closeQuietly(t);
}
}
This works well on the server side.
On the client (RestTemplate) side I tried to use the same converter, but I got an exception that the stream has been closed (probably closed when the request was completed).
Client side code:
ResponseEntity<InputStream> res = rest.getForEntity(url, InputStream.class);
// res.getBody() is closed
I've also tried to copy the input stream into a buffer and create a new ByteArrayInputStream and return it to the RestTemplate client and it worked well, however it does require that the data will be read into memory which doesn't suite my demands.
My question is how to keep the stream open until I process it without having to read it all into memory / file?
Any idea will be appreciated.
Regards, Shay
As far as I am aware, RestTemplate's getForEntity() is not an appropriate way to get an InputStream. It's a convenience for converting to and from entity classes, so presumably that's where your problem lies.
Since you are used to HttpInputMessage, why don't you use HttpInputMessage.getBody() on the client side as well? It gets you a nice InputStream, which would be ready for passing straight to an OutputStream such as HttpServletResponse.getOutputStream().
Check how Spring MVC handles large files upload with org.springframework.web.multipart.commons.CommonsMultipartResolver. It has a 'maxInMemorySize' that can help control the memory requirements. See this thread for using a multipart resolver with the REST template Sending Multipart File as POST parameters with RestTemplate requests
I have a GWT page where user enter data (start date, end date, etc.), then this data goes to the server via RPC call. On the server I want to generate Excel report with POI and let user save that file on their local machine.
This is my test code to stream file back to the client but for some reason I think it does not know how to stream file to the client when I'm using RPC:
public class ReportsServiceImpl extends RemoteServiceServlet implements ReportsService {
public String myMethod(String s) {
File f = new File("/excelTestFile.xls");
String filename = f.getName();
int length = 0;
try {
HttpServletResponse resp = getThreadLocalResponse();
ServletOutputStream op = resp.getOutputStream();
ServletContext context = getServletConfig().getServletContext();
resp.setContentType("application/octet-stream");
resp.setContentLength((int) f.length());
resp.setHeader("Content-Disposition", "attachment; filename*=\"utf-8''" + filename + "");
byte[] bbuf = new byte[1024];
DataInputStream in = new DataInputStream(new FileInputStream(f));
while ((in != null) && ((length = in.read(bbuf)) != -1)) {
op.write(bbuf, 0, length);
}
in.close();
op.flush();
op.close();
}
catch (Exception ex) {
ex.printStackTrace();
}
return "Server says: " + filename;
}
}
I've read somewhere on internet that you can't do file stream with RPC and I have to use Servlet for that. Is there any example of how to use Servlet and how to call that servlet from ReportsServiceImpl. Do I really need to make a servlet or it is possible to stream it back with my RPC?
You have to make a regular Servlet, you cannot stream binary data from ReportsServiceImpl. Also, there is no way to call the servlet from ReportsServiceImpl - your client code has to directly invoke the servlet.
On the client side, you'd have to create a normal anchor link with the parameters passed via the query string. Something like <a href="http://myserver.com/myservlet?parm1=value1&.."</a>.
On the server side, move your code to a standard Servlet, one that does NOT inherit from RemoteServiceServlet. Read the parameters from the request object, create the excel and send it back to the client. The browser will automatically popup the file download dialog box.
You can do that just using GWT RPC and Data URIs:
In your example, make your myMethod return the file content.
On the client side, format a Data URI with the file content received.
Use Window.open to open a file save dialog passing the formatted DataURI.
Take a look at this reference, to understand the Data URI usage:
Export to csv in jQuery
It's possible to get the binary data you want back through the RPC channel in a number of ways... uuencode, for instance. However, you would still have to get the browser to handle the file as a download.
And, based on your code, it appears that you are trying to trigger the standard browser mechanism for handling the given mime-type by modifying the response in the server so the browser will recognize it as a download... open a save dialog, for instance. To do that, you need to get the browser to make the request for you and you need the servlet there to handle the request. It can be done with rest urls, but ultimately you will need a serviet to do even that.
You need, in effect, to set a browser window URL to the URL that sends back the modified response object.
So this question (about streaming) is not really compatible with the code sample. One or the other (communication protocols or server-modified response object) approach has to be adjusted.
The easiest one to adjust is the communication method.
I have an old legacy java web application that I want to deploy on the same server as my asp.net-applications (running on IIS 7). And I need to have all applications running on port 80, so I can't just install two web servers on different ports.
The java-application is really simple, just a couple of serverlets (no JSP) with functionality pretty close to "Hello World". So I want the servlet engine to be as lightweight as possible. I consider Tomcat to be overkill for this.
Does such a product exist or I'm I stuck with Tomcat?
Have a look at Jetty. It can be invoked from a standard Main invocation, and handles servlet containers pretty well (GWT debugging is hosted in a Jetty environment, for example).
I've used this for debugging Lift applications, and been pretty impressed.
To forward requests through IIS to Jetty you can try mod_jk. The problem is that IIS and Java/JSP don't Just Work because IIS needs add-ons to support the loading of the VM and the reflection of JSP/Java content. There is an excellent article on how this can be done here.
I'm sure you don't want to get too exotic but if you don't find something to your liking, you could always run Tomcat on a different port and then do some reverse proxying and send everything heading to a specific context to your Tomcat install running on port XYZ, although admittedly I'm only familiar with this in the Apache world, and not the IIS world.
Check out JK, it has an Apache module and I think it also has an IIS module to connect with Tomcat (maybe Jetty, I don't know). This will allow you to run your app on Tomcat on some other port (even on some other host) and access it through your IIS.
http://tomcat.apache.org/connectors-doc/
If you use Java 6, there is a HTTP server built-in,
http://java.sun.com/javase/6/docs/jre/api/net/httpserver/spec/com/sun/net/httpserver/package-summary.html
Of course, this wouldn't be a good solution if you care about portability.
Use HTTP server that comes with java version 6 or more. Straight forward code.
As suggested above, REST can be implemented using Servlet technology.We cannot compare both.
Servelet is an adapter(Java) that converts http request to Java object/s and triggers a method(doGet/doPost) when ever the request Arrives and sends http response as per instructed in Java programatically.
public class server {
public static void main(String[] args) throws Exception {
//Start server
HttpServer server = HttpServer.create(new InetSocketAddress(8000, 0);
server.createContext("/monitor", new MyHandler());
server.createContext("/usage", new MyHandler());
server.setExecutor(java.util.concurrent.Executors.newCachedThreadPool()); //Uses New Thread every time
server.start();
System.out.println("Server Started at 8000...");
}
static class MyHandler implements HttpHandler {
#Override
public void handle(HttpExchange t) throws IOException {
URI uri = t.getRequestURI();
String path = uri.getPath();
InputStreamReader isr = new InputStreamReader(t.getRequestBody(), StandardCharsets.UTF_8);
BufferedReader br = new BufferedReader(isr);
int b;
StringBuilder buf = new StringBuilder(512);
while ((b = br.read()) != -1) {
buf.append((char) b);
}
br.close();
isr.close();
JSONObject body = null;
try {
body = new JSONObject(buf.toString());
} catch (Exception e) {
System.out.println(e.getMessage());
}
if (path.equals("/monitor")) {
//your Logic
//Use any other Library - I use GSON
Gson gson = new Gson();
String json = gson.toJson(POJO_OBJECT);
t.getResponseHeaders().set("Content-Type", "application/json");
t.getResponseHeaders().set("Access-Control-Allow-Origin", "*");
t.getResponseHeaders().set("Access-Control-Allow-Headers", "Origin, X-Requested-With, Content-Type, Accept");
t.sendResponseHeaders(200, json.length());
OutputStream os = t.getResponseBody();
os.write(json.getBytes());
os.close();
}
if (path.equals("/monitor")) {
//Your Logic
//Use any other Library - I use GSON
Gson gson = new Gson();
String json = gson.toJson(POJO_OBJECT);
//System.out.println(json);
t.getResponseHeaders().set("Content-Type", "application/json");
t.getResponseHeaders().set("Access-Control-Allow-Origin", "*");
t.getResponseHeaders().set("Access-Control-Allow-Headers", "Origin, X-Requested-With, Content-Type, Accept");
t.sendResponseHeaders(200, json.length());
OutputStream os = t.getResponseBody();
os.write(json.getBytes());
os.close();
}
}
}
}