Using Play Framework with Java (controller) and Twirl, I am trying to send a page that contains about 40MB of data to the caller, through a single HTTP GET request (using scala templates, nothing special here). The issue is that when I call the render method on the template, my application crashes with an OutOfMemoryError. I am 100% positive it is relative to page weigth VS available heap memory, I have confirmed it through testing. I think it is related to the fact that "To be able to compute the Content-Length header properly, Play must consume the whole response data and load its content into memory." (https://www.playframework.com/documentation/2.5.x/JavaStream). The Twirl templates render method seems to load the entire page into memory just the same anyway.
I have confirmed, as mentionned in the docs, that if I use an InputStream to output my own data, things will work smoothly and I can send as much data as I want (up to gigabytes of data if I want, it just works) back to the client simply by returning ok(myInputStream) in my controller method.
Now, what I would like to do is to get Twirl to render its template using streaming (providing me an InputStream or something similar). I have searched for solutions, read the docs to no avail. Am I looking at the wrong places, or it is just not available and possible by design, and I have no other choices than to rethink my template and requests/responses ?
Java 8, Play Framework 2.5.
Related
I have a JSP page using ajax as my front-end talking to a java springboot backend. I need to upload large files, which is currently working fine. The problem is that Cloudflare only allows 99MB at a time to be uploaded. I need multiple GB's uploaded at one time.
From what I understand it is possible to break the file into a chunks or BLOB and then send them through one piece at a time. It is fairly easy to find an example of this for the Ajax side of things, but I've yet to find anything for the correlating SpringBoot functionality.
Ultimately I'd also like to be able to stop and resume the uploads/downloads, so I think this would also be used in that instance?
Edit
It seems like Spring Batch might be the tool for this, but I'm still in the dark on how it would work with Ajax.
I need to convert a web page [which has not public access] to PDF or Image [preferably to PNG].
Web page contains set of charts and image. Most of the charts are populated through Ajax calls so there is a delay between page load and chart load.
I am looking answer for any of these questions:
1- I found set of snapshot api's but none of them support accessing my internal page. Since the web page I am trying to export is not public I need to be authenticated. Biggest problem is I cannot send request headers [such as session-id, cookie or other variables] along with these API's. It seems they don't support this kind of functionality.
2- I am not sure if I can do following: Login to my web page with HTTP client, add http headers, send get call and get HTML string. Then use one of the converters to convert it to PDF. What I am not sure is if it's possible to get proper PDF from the HTML string I got from http client since resources [css, js and etc] will be missing. I want my pdf/image looks exactly as it on the web site.
I really appreciate if you can help.
Thanks in advance,
ED
You're probably best of using wkhtmltopdf, which is a server-side tool and is easily installed.
There are two parameters you can use to wait for your Ajax to finish, try:
javascript-delay to influence the time the program waits for the JavaScript to finish
window-status to wait for a certain return code for the window
See the extensive manual for this program here
wkhtmltopdf generates a PDF and wkhtmltoimg generates an image, which is PNG (as you requested) by default.
Authentication is difficult because it involves security. Because the operation you are describing is unusual it is likely to result in all kinds of alarm bells going off. It is entirely possible to do but it is fraught, easy to get wrong and fragile in the face of security updates and code changes.
As such I'm going to suggest an alternate method which is one we often recommend for ABCpdf (on which I work). Yes we support standard authentication methods but the beauty of this approach is that it is robust and is applicable to other solutions (eg Java based) and novel authentication methods.
Typically you just want a PDF of the current page. The easiest way to do this is snaffle the HTML. The way you do this rather depends on your environment. For example under ASP.NET you can obtain the HTML of the current page using the HttpResponse.Filter property or by overriding the Render method of the page. The way you do it will depend on what you're coding in.
Then you need to save this HTML to a file and present it to your solution via a 'file://' protocol URL. Now obviously at this point any relative links will be broken but this is easily fixed by dropping in a BASE tag that references the place they are located.
Generally the types of resources referenced by an server-side page are static. So if you can create a tag that references the actual files rather than a web site, you will bypass any authentication for access to these resources.
That still leaves the AJAX based problems which are another can of worms. The render delay method is something we have supported for many years (from before AJAX was around) however it is not terribly reliable because you just don't know how long to wait.
Much better is a tighter link into the JavaScript via a callback you can use to determine if the page is loaded. I don't think ABCpdf is going to be appropriate for you since it is .NET but I would certainly encourage you to look for a Java based solution that uses this type of more sophisticated approach.
I have a situation where in I write to a text file programmatically using java and simultaneously I read from the same file using jQuery.
The problem I face is jQuery is unable to find the updated content whenever a content is written into the text file via java.
I have Googled a lot but the only results I find are for java and java processing and not for java and javascript (i.e A Client side and Server side)
I am not sure if this is even possible.
More about the question:
I write into the file the crawling results using java and I am trying to display the same using javascript (jQuery.post() method).
JAVA
A multi-threaded crawling program that crawls a website and does some functionality. I am trying to write some content into a text file using the same java program as and when the crawling happens. The content I write mostly are the details about which thread is getting invoked and what is the current link that is being crawled.
The reason I write this in the text file is I need to show the output in the UI so that people looking at the UI will understand what happens.
Writing happens perfectly as expected.
JAVASCRIPT (jQUERY)
This using the
jQuery.get or post ("sample.txt", function (result) {
$("#someID").html(result);
});
It reads from the text file normally but when java and javascript both are trying to access the file, It is the java that dominates leaving javascript behind thus jQuery is unable to fetch the updated content as and when it happens.
I guess this explanation is more than sufficient to make people understand what exactly my problem is !
On the whole, java and javascript try to access the same file at the same time. So there comes this issue.
Any help is appreciated.
Thanks in advance
I think the file is cached. Easiest thing is to request the file by different urls. Try something like "sample.txt?rnd="+Math.rand()
There can be synchronization problems and your data will be corrupted.
I have a question, is it must be done with Ajax? I think you are trying to figure out about
Ajax push and pull
This is not very easy to do and I wouldn't really recommend it. However, there is a better technology called websocket. So what you can do is, client can submit request to the server to write data into a file then server can send back updated content to the client. Moreover, this is much better than achieving the same objective through numerous amount of HTTP requests.
Additionally, if you want the crossbrowser compatibility, have a look at http://socket.io/
Thanks for all those who were trying to help me out.
I have finally come up with a solution. I, instead of using jquery post to directly read from file, am using another jsp file that reads the file contents and prints using out.println on screen, and after which I am using jQuery post to get the content written by that jsp file. Hence the synchronization problem is avoided.
Here is more about my explanation:
Earlier I had
java program -> Text File <- javascript (jQuery post) // Resulted in synchronization problem where in javascript was not able to access the updated content.
Now
java program -> Text file <- JSP file <- javascript (jQuery post) // Avoided the synchronization problem as that file is accessed by the same server side language. After that jQuery reads the content printed by JSP page.
After many changes, finally came up with one good working solution.
Thanks all.
I'm developing a download-anything app for Android and it works fine in most cases.
I have come across sites that have URLs with a long hash signature (it seems) at the end of it. But the standard video app for Android, and my web browser is able to play it directly, streaming.
I have no clue as to how to stream this to a file (progressive download?), which should be possible. The URL paramater after '?' is used for something. As Jessica pointed out the URL below is probably used for RTMP streams with rtmp://....
URL example (host domain edited out):
http://blush.im.54ca3830.919727.x.yesitisporn.com/videos/3gp/d/b/f/
filthysite.com_dbf7f0a9c3913d4d0e09a36fe8ab3aba.mp4?e=1348368010&ri=1024&rs=85&h=c81c6707b13714ac65b651ba2939d94a
In the URL above there is a link to an mp4 video file. Trying to download it with this shorter URL does not work: http://blush.im.54ca3830.919727.x.yesitisporn.com/videos/3gp/d/b/f/
filthysite.com_dbf7f0a9c3913d4d0e09a36fe8ab3aba.mp4. Returns an empty document.
Since popular video apps and browsers pick up these types of HTTP links just fine for playback; there should be a standard way of getting the byte stream and write it to file. Thanks for any help!
In response to the question as originallly posed:
It is quite common to add URL parameters, splitting the url from the parameters with a question-mark, and seperating the parameters with ampersands. Take the substring on everything up to the first non-esecaped question-mark in the url, if a question-mark is present, otherwise use the entire string.
Based on new feedback:
Like I said in my comment, and as confirmed by your tests without the parameters, I think you're barking up the wrong tree to try to change the URL. I would suspect the reason you can't save these specific streams is there is something different about the file format or server configuration that is different than the ones that work. In particular, my first thought would be that perhaps those URLs are served by a Streaming Server (Example: Icecast), and not a normal file-based HTTP server. Advanatages of a streaming server include being able to on the fly serve different bandwidth versions of the streams, and instant seeking to any part of the file and so forth, but the downside for people trying to build download anything applications is those servers don't send the data as a single file, they send the data in chunks--trying not to get too crazy technical, basically, a chunk might have the first frame plus a bunch of diffs for what's on the video in the next several frames and the audio, repeat. As it does this it can throttle what quality to send depending on the latency it's seeing or the resolution of your screen, or resize what it sends if you resize the window and so forth. This sort of streaming works particularly well for live events, but it is not without its advantages for recorded events as well--particularly random seeking. To complicate the matter of capturing the data, some streaming servers actually transmit the video data via RTMP, RTSP, or MMS protocols instead of over HTTP. HTTP Pseudo-streaming or straight HTTP downloads is going to be a lot easier to save than streaming via RTMP. Some streaming types you pretty much have to recreate the file from the individual packets or transcode it from what plays on the screen as it plays in real time. So you may need to spend some time learning about different streaming protocols to figure out the best way to save the specific stream you're looking at.
I want to create a page where the top half contains start stop buttons and in the bottom half i want to write content from the server. the buttons invoke functions on the server and the server does some computing and generates timely messages which need to be written to the bottom half of the page.
Possible ways of doing
1. AJAX
2. DWR
3. HTML5
Let me know which method is better and how can i do it.
AJAX
Means "Making an HTTP request without leaving the page". This will let you get content from the server. To write it you need to do DOM manipulation. There are no shortage of Ajax and DOM tutorials on the web
DWR
Is a library to help with Ajax (although not one I've heard of before).
HTML5
Is a now meaningless buzzword. Of all the meanings it has, only the ones which include "Ajax and DOM manipulation" will help, and that's already covered above.
Look up a tutorial on xmlhttprequest. I know there is a good explanation of it here Quirksmode. I am not sure where your current skill set is using HTML, server side programming, XML, and Javascript are, but you will need to have a minimum understanding of all of those in order to successfully implement AJAX.
There are lots of resources on the web, Google