We have one web application(With Spring, hibernate and MySQL as a Database) in which multiple users can store the heavy videos(pre-recorded or record from application itself) on server at same time.
In that scenario, server load would be definitely more there. We are assuming there would be 500-2000 users in the application.
So what strategy i should use to reduce the load from server and make the response time faster.
1) Storing the videos on our server(With large Disk Space), and using the ActiveMQ/RabbitMQ mechanisms for File Upload and download in the Queues.
2) Storing the videos on some third party server(like YouTube,vimeo etc) that will upload all the videos on one central account. I had recently check this thing with you tube and vimeo but they require the end user login credentials for each upload. And i don;t want in my application that end-users to provide their credentials before each upload.
Is there any other way to reduce the work load and make the response time better for simultaneously upload on server, then please guide.
Thanks In Advance,
Arun
Multi servers can help.
On a single server:
If you use a single core processor - only ONE client will get served.
If you use a multi core processor and you are oppening a new thread for a new connection - only #ofCores clients will get served, and even that is not correct because your local memory might run out before your os will save the data to your local hard disk (which has one bus), so serving 500-2000 clients leads you to a multi server solution.
Related
I have the most basic problem ever. The user wants to export some data which is around 20-70k records and can take from 20-40 seconds to execute and the file can be around 5-15MB.
Currently my code is as such:
User clicks a button which makes an API call to a Java Lambda
AWS Lambda Handler calls a method to get the data from DB and generate excel file using Apache POI
Set Response Headers and send the file as XLSX in the response body
I am now faced with two bottlenecks:
API Gateway times out after 29 seconds; if file takes longer to
generate it will not work and user get 504 in the browser
Response from lambda can only be 6MB, if file is bigger the user will
get 413/502 in the browser
What should be my approach to just download A GENERATED RUNTIME file (not pre-built in s3) using AWS?
If you want to keep it simple (no additional queues or async processing) this is what I'd recommend to overcome the two limitations you describe:
Use the new AWS Lambda Endpoints. Since that option doesn't use the AWS API Gateway, you shouldn't be restricted to the 29-sec timeout (not 100% sure about this).
Write the file to S3, then get a temporary presigned URL to the file and return a redirect (HTTP 302) to the client. This way you won't be restricted to the 6MB response size.
Here are the possible options for you.
Use Javascript skills to rescue. Accept the request from browser/client and immediately respond from server that your file preparation is in progress. Meanwhile continue preparing the file in the background (sperate job). Using java script, keep polling the status of file using separate request. Once the file is ready return it back.
Smarter front-end clients use web-sockets to solve such problems.
In case DB query is the culprit, cache the data on server side, if possible, for you.
When your script takes more than 30s to run on your server then you implement queues, you can get help from this tutorial on how to implement queues using SQS or any other service.
https://mikecroft.io/2018/04/09/use-aws-lambda-to-send-to-sqs.html
Once you implement queues your timeout issue will be solved because now you are fetching your big data records in the background thread on your server.
Once the excel file is ready in the background then you have to save it in your s3 bucket or hard disk on your server and create a downloadable link for your user.
Once the download link is created you will send that to your user via email. In this case, you should have your user email.
So the summary is Apply queue -> send a mail with the downloadable file.
Instead of some sophisticated solution (though that would be interesting).
Inventory. You will split the Excel in portions of say 10 k rows. Calculate the number of docs.
For every Excel generation called you have a reduced work load.
Whether e-mail, page with links, using a queue you decide.
The advantage is staying below e-mail limits, response time-outs, denial of service.
(In Excel one could also create a master document, but I have no experience.)
I've built an app where users can upload their avatars. I used the paperclip gem and everything works fine on my local machine. On Heroku everything works fine until server restart. Then every uploaded images disappear. Is it possible to keep them on the server?
Notice: I probably should use services such as Amazon S3 or Google Cloud. However each of those services require credit card or banking account information, even if you want to use a free mode. This is a small app just for my portfolio and I would rather avoid sending that information.
No, this isn't possible. Heroku's filesystem is ephemeral and there is no way to make it persistent. You will lose your uploads every time your dyno restarts.
You must use an off-site file storage service like Amazon S3 if you want to store files long-term.
(Technically you could store your images directly in your database, e.g. as a bytea in Postgres, but I strongly advise against that. It's not very efficient and then you have to worry about how to provide the saved files to the browser. Go with S3 or something similar.)
In my webapp users can download the files among themselves. If a user A has shared a file F , then user B after connecting to A can download the file F from A. Till now each user makes a simple HTTP connection like :xxx.xxx.xxx.xxx/FileList with another user. The file resides on the local hard disk of each user. So that a user can download a file there were two options in my mind.
As the user shares a file,copy that file into the web-app directory of the server,so that the download link becomes as simple as Click to download.
Run a separate FTP server on each node.
I don't know which one of these is a better option but the first one seems very simple to me. What are the ways each client can share the files,without having to copy the stuff somewhere in the webapp directory. How in this case I can use a P2P protocol ?
NOTE : I am using Tomcat 7.
Real P2P is impossible without opening a listening socket on the client machine (that imposes you have to install something on client machine).
If you don't want to STORE the files on the server, I would rather recommend a "connection server", which serves as a gateway between the two users. User A will upload, user B will download at the same time, all you need is to make the bytebuffer in memory. The downloaded bytes can be dropped.
You can write a small client-side program in any language for updating the available files, and receiving the upload request from the server side (also execute the upload)
I would recommend using TCP sockets for upload to the server side, this way you have direct control over the uploaded bytes (streams).
There are some interesting technical issues here (blocking streams, metadata (filename, length, createdate, ...), data consistency, error handling, etc.) that should be taken into consideration. Nice task.
I don't recommend FTP because you cannot control the authentication and authorisation (who can see the files).
I have an SQLite Database on a webserver. I would like to access the database from a typical Java Desktop Application. Presently, I'm doing this thing...
Download the Database file to a local directory, perform the queries as necessary.
But, I'm unable to perform any update queries on this. How can I do this. [ On the actual database]
Another question is, to directly access the database from web in java (is this possible), make direct queries, updates anything etc,.
How can I achieve this type?
I've written code for connection of Java to SQLite and is working pretty fine, if the db file is in local directory. What changes or anything I have to do to establish a link to the file on webserver without having to download the database file.?
Thanks in advance...
CL. is right saying that if you need to access from desktop applications to a web database, SQLite is not an appropriate choice.
Using SQLite is fine in small web sites, applications where your data have to be accessed from and only from the web site itself; but if you need to access your data from - say - your desktop, without downloading the data file, you can't achieve that with SQLite and HTTP.
An appropriate choice for your web application would be MySQL or other client/server database, so that you could be able to connect to the database service from any place other than your web application, provided server access rules set permit that (e.g. firewalls, granted authentication, etc.).
In your usage scenario, you would be facing several orders of problems.
1) Security
You would be forced to violate the safety principle saying that database files must be protected from direct web exposure; in fact, to access your web SQLite database file from your desktop you would be forced to expose it directly, and this is wrong, as anyone would be able to download it and access your data, which by definition must be accessible just by you.
2) Updatability without downloading
Using HTTP to gain access to the database file can only lead to the requested resource download, because HTTP is a stateless protocol, so when you request GET or even POST access to the database, the web server would provide it to you in one solution, full stop.
In extreme synthesis, no chance to directly write back changes to the database file.
3) Updatability with downloading
You could download your file with a HTTP GET request, read data, make changes and the rest, but what if your online database changes in the meanwhile? Data consistency would be easily compromised.
There could be a way
If you give up using HTTP for your desktop application access to the database, then you could pick FTP (provided you have access credentials to the resource).
FTP lets you read data from and write data to files, so - on Linux - you could use FUSE to mount a remote FTP sharing and access it just like if it was connected to your local file system (see this article, for example).
In synthesis, you:
Create a mount point (i.e. a local directory) for FTP sharing
Use curlftpfs to link the remote FTP resource to your mount point
Access to this directory from your application as if it was a conventional directory
This way you could preserve security, keeping the database file from being exposed on the web, and you would be able to access it from your desktop application.
That said, please consider that concurrent access by several processes (desktop app + webserver instance) to a single database file could lead to problems (see this SO post to have an idea). Keep that in mind before architecting your solution.
Finally, in your usage scenario my suggestion is to program some server-side web service or REST interface that, under authentication, let you interact with the database file performing the key operations you need.
It is safe, reliable and "plastic" enough to let you do whatever you want.
EDIT:
MySQL is widely used for web sites or web applications, as it is fast, quite scalable and reasonably reliable. Activating MySQL server is a little bit OT on StackOverflow and quite long-winded to report, but in that case you could google around to find plenty of articles discussing such topic for your operating system of choice.
Then use MySQL JDBC driver to access the database from your Java desktop application.
If your idea is to stick with SQLite, though, you could basically prepare four web endpoints:
http://yourwebsite.com/select
http://yourwebsite.com/insert
http://yourwebsite.com/update
http://yourwebsite.com/delete
(Notice I specified "http", but you could consider moving to SSL encrypted http connection, a.k.a. "https", find details here and here. I don't know which webserver are you running, but still googling a little bit should point you to a good resource to properly configure https.)
Obviously you could add any endpoint you like for any kind of operation, even a more generic execute, but play my game just for a while.
Requests towards those endpoints are POST, and every endpoint receives proper parameters such as:
table name
fields
where clause
... and the like, but most important is security, so you have to remember 2 things:
1. Sign every request. You could achieve this defining a secret operation key (a string which is known to your client and you server but never travels in clear text), and using it in a hashing function to produce a digest which is sent together with other parameters as an incontrovertible proof for the server that that request it's receiving comes from a genuine source. This avoids you to send username and password in every request, which would introduce the problem of password encryption if you don't use https, and involves that the server has to be able to reconstruct the same signature for the same request using the same algorithm. (I flew over this thing at 400Mph, but the topic is too large to be correctly treated here. Anyways I hope this could point you in the right direction.)
2. Properly escape request parameters. "Sanitize" parameters someone calls it, and I think the metaphor is correct. Generally speaking this process involves some filtering operations performed by the server's endpoint, but it basically could be written as "use prepared statements for your queries". If you don't it could be likely that some malicious attacker injects SQL code in requests to exploit your server in some manner.
SQLite is an embedded database and assumes that the database file is directly accessible.
Your application is not an appropriate use of SQLite.
You should use a client/server database.
In any case, you should never make a database directly accessible on the internet;
the data should go through a web service.
What would be a scalable file upload/download system/database?
I'm building a website where users can login, upload images that are private, but truly private. I can't upload them to a map on the harddisk of a server, since that would not scale (what happend if we add more servers?) and it wouldn't be private since everyone could go:
http://127.372.171.33/images/private_picture.png
and download the file.
I am building the project in Play Framework (scala/java)
How do websites like flickr handle these kind of things? Do they put them in a database? And what kind of database would be suitable for this situation?
Thanks for help
I can't tell you how those big sites handle it but putting those images into a database might be one way.
Another way would be to put the files into a virtual filesystem that spans a cluster of servers or distribute them onto different servers and just don't make the directories that contain the images visible to the webserver. Thus nobody should be able to open the image just using the server and the path on that server.
To actually deliver the images you could them implement some streaming service that sends a bytestream to the browser for display (like the webservers would do as well). This service could first check the download permissions for the requested image.