I have the most basic problem ever. The user wants to export some data which is around 20-70k records and can take from 20-40 seconds to execute and the file can be around 5-15MB.
Currently my code is as such:
User clicks a button which makes an API call to a Java Lambda
AWS Lambda Handler calls a method to get the data from DB and generate excel file using Apache POI
Set Response Headers and send the file as XLSX in the response body
I am now faced with two bottlenecks:
API Gateway times out after 29 seconds; if file takes longer to
generate it will not work and user get 504 in the browser
Response from lambda can only be 6MB, if file is bigger the user will
get 413/502 in the browser
What should be my approach to just download A GENERATED RUNTIME file (not pre-built in s3) using AWS?
If you want to keep it simple (no additional queues or async processing) this is what I'd recommend to overcome the two limitations you describe:
Use the new AWS Lambda Endpoints. Since that option doesn't use the AWS API Gateway, you shouldn't be restricted to the 29-sec timeout (not 100% sure about this).
Write the file to S3, then get a temporary presigned URL to the file and return a redirect (HTTP 302) to the client. This way you won't be restricted to the 6MB response size.
Here are the possible options for you.
Use Javascript skills to rescue. Accept the request from browser/client and immediately respond from server that your file preparation is in progress. Meanwhile continue preparing the file in the background (sperate job). Using java script, keep polling the status of file using separate request. Once the file is ready return it back.
Smarter front-end clients use web-sockets to solve such problems.
In case DB query is the culprit, cache the data on server side, if possible, for you.
When your script takes more than 30s to run on your server then you implement queues, you can get help from this tutorial on how to implement queues using SQS or any other service.
https://mikecroft.io/2018/04/09/use-aws-lambda-to-send-to-sqs.html
Once you implement queues your timeout issue will be solved because now you are fetching your big data records in the background thread on your server.
Once the excel file is ready in the background then you have to save it in your s3 bucket or hard disk on your server and create a downloadable link for your user.
Once the download link is created you will send that to your user via email. In this case, you should have your user email.
So the summary is Apply queue -> send a mail with the downloadable file.
Instead of some sophisticated solution (though that would be interesting).
Inventory. You will split the Excel in portions of say 10 k rows. Calculate the number of docs.
For every Excel generation called you have a reduced work load.
Whether e-mail, page with links, using a queue you decide.
The advantage is staying below e-mail limits, response time-outs, denial of service.
(In Excel one could also create a master document, but I have no experience.)
Related
I have a Java Play Framework 2.5 project where periodically I have to update the registries of our customer. I can't do a simple SQL insert/update because I to do some logic and operation on every single row.
I ended up uploading on my website an xlsx file (asynchronous way) with JS and then working on it with Apache POI.
My only problem is that I don't know how to inform the user who uploaded the file on the progress of the processing of the file.
I'll like to simple show every 10% a message with the percentage.
I've searched on the net, I think the best way is to use websockets with Akka (BroadcastHub), but i'm only finding examples in Scala or for 2-way communication that send output only when an input is received.
If you know a different way to do it you are welcome!
I suggest you look at https://github.com/playframework/play-java-websocket-example : It gives an example of a websocket and an actor interaction.
In your case, you would create a named actor (useractor-someid), which manages the mutations. The controller then would reference the same actor (or have an other actor send to useractor-someid), so that it can fetch the status. I suggest you use a JsonNode flow, so you can send / receive JSON in the websocket connection.
I have a UI being developed in HTML5 and angular JS , which makes a webservice call to REST endpoint developed in Java ,
As our REST endpoint is accepting a file so we have throttled the file request so as accept only given amount of file request from the client .
Now if the threshold is reached my question is it a good design to throw error back to client say "memory threshold reached" back to client or Do i send some generic error back to client as the consumer of this end point is other consumers as well from different clients
The way we currently handle this in our application is to actually do a client side check on the file first, before even making the REST call. Should the file selected exceed the threshold size which the API can tolerate, then we give an appropriate error message right there to this effect.
With this check in place, the only way that a file too large to be handled could be uploaded would be if a given user were to somehow hack around the UI, or if someone were using a really old or non supported browser. Both of these are probably rare cases, so any generic error message stating the REST call failed should be sufficient.
Here is a link to a tutorial which covers the HTML 5 file API: https://www.html5rocks.com/en/tutorials/file/dndfiles/
It is possible to capture a local file's size directly in your JavaScript code. This information can then be used to display a message, block an upload, etc. depending on how you want to handle the size. Note that you cannot get the file's local path on the computer, because that would be a security problem.
I have a servlet that accepts large (up to 4GB) binary file uploads. The submitted file is transmitted as the body of an HTTP POST.
The servlet has to perform some time-consuming processing as it receives the file, and it has to finish doing that before sending the response. As a result, it can appear to a fast client that the server has hung because the client can be waiting for a minute or two after sending the the last few bytes before getting the response.
Is there a way either within Tomcat or within the servlet API to throttle back the speed at which the server accepts the file? I would like it to appear to the client that the server is accepting the file at (for example) 10MB/second rather than it accepting the file at 50MB/second and then taking a few minutes after receiving the body to return a response.
Thanks.
I'm extending on the comment of Mark Thomas here because I feel that this is worth being an answer (or the answer), rather than a comment. Mark, let me know if you want to convert the comment yourself and I'll happily delete mine.
John, you're trying to solve your problem in a way that imposes severe limitations: What's the bandwidth that you want to throttle to? What happens when the server is upgraded to a beefier CPU and can process more quickly? What if multiple uploads happen at the same time?
You probably want to have an upload of 4G in as quick a time as possible - imagine the connection going down in the middle - in a web application this typically means you'll have to restart the upload from the beginning. Thus you should decouple your processing from the upload procedure as much as possible.
You also don't mention the file format that gets uploaded: If it happens to be a zip file, note that the server can't do anything with the file until it's fully transmitted, as zip files have the directory of contents at their end. (this might be old knowledge, but at least the old spec had it this way. Someone correct me if this changed)
So: The proper way: Accept the file for processing, signal that you received it and are processing. If you like: Implement Ajax updates once you're done. In the simplest case: "click here to see if processing finished" or frequently reload the page. Anything works and everything is better than throttling throughput on this layer.
We have one web application(With Spring, hibernate and MySQL as a Database) in which multiple users can store the heavy videos(pre-recorded or record from application itself) on server at same time.
In that scenario, server load would be definitely more there. We are assuming there would be 500-2000 users in the application.
So what strategy i should use to reduce the load from server and make the response time faster.
1) Storing the videos on our server(With large Disk Space), and using the ActiveMQ/RabbitMQ mechanisms for File Upload and download in the Queues.
2) Storing the videos on some third party server(like YouTube,vimeo etc) that will upload all the videos on one central account. I had recently check this thing with you tube and vimeo but they require the end user login credentials for each upload. And i don;t want in my application that end-users to provide their credentials before each upload.
Is there any other way to reduce the work load and make the response time better for simultaneously upload on server, then please guide.
Thanks In Advance,
Arun
Multi servers can help.
On a single server:
If you use a single core processor - only ONE client will get served.
If you use a multi core processor and you are oppening a new thread for a new connection - only #ofCores clients will get served, and even that is not correct because your local memory might run out before your os will save the data to your local hard disk (which has one bus), so serving 500-2000 clients leads you to a multi server solution.
In a server-side application running on Tomcat, I am generating full HTML pages (with header) based on random user-requested sites pulled down from the Internet. The client-side application uses asynchronous callbacks for requesting processing of a particular web page. Since processing can take a while, I want to inform the user about progress via polling, hence the callbacks.
On server-side, after the web page is retrieved, it is processed and an "enhanced" version is created. Then this version has to go back to the user.
Displaying the page as part of the page of the client-side application is not an option.
Currently, the server generates a temporary file and sends back a link to it. This is clearly suboptimal.
The next best solution I can come up with inolves creating a caching-DB that stores the HTML content together with its md5-sums or sha1-ids and then sends back a link to a servlet, with the hash-ID as an argument. The servlet then requests the site from the caching-DB.
Is there any better solution? If not, which DB-backend would you propose? I'm thinking of SQLite. Part of the problem to be solved is: how do I push a page <html> to </html> back to client side?
If true persistence isn't required how about using something more temporal like memcached instead of SQL? Calling semantics are pretty clean and easy - and of course you can expire the data manually, ttl, or # restart.
Instead of creating a temporary file, filling it up, and then sending a link, you can create a memory buffer, fill it up, and then send that as the response (serve it with mime-type 'text/html'). If you don't want to send page-buffers immediately, you can save them for later in the user's session. If you're worried of taking up too much memory that way, you may want to keep only a certain number of page-buffers around in memory, and write the rest to disk for later retrieval. Using a DB sounds like overkill (after all, there's no relational information involved) - but it would solve the caching problem nicely.