I am curious to know, how to show any image or document which is stored on a private S3 bucket and I need to show it on UI to some authorized users. Just like facebook and instagram do it, to secure their images. I can not put images on a public bucket otherwise anyone can view and download it and it won't be secure. I was thinking about some solutions but every solution has it's own pros and cons. I think about below solutions:
Solution 1:
I can put all the images in a private S3 bucket and an auth token can be associated with image url as a query param. This image url would be passed to image server and image server will validate the auth token. If it is a valid one then, image would be fetched from private S3 bucket to image server and downloaded to client in byte array.
This approach has multiple disadvantages like:
Image has to travel from S3 bucket to image server and again image server to client device. It will add more latency and this would be increased with size of image.
This approach is not scalable. If I think that 100 image read requests are coming in a second and every image has a size of 5MB, then approx 500MB content would present in the memory every time. And with the time when traffic would be on it's peak, server goes down.
I can think to cache images for faster response but image caching will take a lot of space and for the request of every server above problems would occur.
Solution 2
We can think to bypass image server and try to download image from the S3 bucket directly to the client device. If user is logged in, then client can download images using secret key of private bucket. Again this approach has some advantages:
we have to configure secret key of private bucket in client devices like Android, IOS, web, etc. There are high chances to leak secret key from the frontend resources.
In future if I think to replicate image content for faster service in different geographical locations, then it is hard to maintain different secret key for each bucket.
Apart from that other solution which can serve our purpose?
Solution 1 is good, but you don't need a server... Amazon S3 can do it for you.
When your application wants to provide a link to a private object, it can generate an Amazon S3 pre-signed URLs, which is a time-limited URL that provides access to a private object. It can be included in your HTML page (eg <img src="...">) or even as a direct URL.
The beauty of this approach is that your app can generate a pre-signed URL in a few lines of code (no call to AWS required) and then Amazon S3 will do all the serving for you. This means there is less load on your application server and it is highly scalable.
Related
I've built an app where users can upload their avatars. I used the paperclip gem and everything works fine on my local machine. On Heroku everything works fine until server restart. Then every uploaded images disappear. Is it possible to keep them on the server?
Notice: I probably should use services such as Amazon S3 or Google Cloud. However each of those services require credit card or banking account information, even if you want to use a free mode. This is a small app just for my portfolio and I would rather avoid sending that information.
No, this isn't possible. Heroku's filesystem is ephemeral and there is no way to make it persistent. You will lose your uploads every time your dyno restarts.
You must use an off-site file storage service like Amazon S3 if you want to store files long-term.
(Technically you could store your images directly in your database, e.g. as a bytea in Postgres, but I strongly advise against that. It's not very efficient and then you have to worry about how to provide the saved files to the browser. Go with S3 or something similar.)
I am trying to upload large files (more than 1 GB) on amazon S3 using Java
I am using AWS S3 multipart upload to upload large files in chunks.
https://docs.aws.amazon.com/AmazonS3/latest/dev/HLuploadFileJava.html
I am using also uploading the files in chunks from the frontend.
So, the file being uploaded will be temporarily uploaded on the server in chunks and it will be uploaded on S3 in chunks.
Now the problem is that this method puts a huge load on the server since this consumes server space temporarily. If multiple users are trying to upload large files at the same time then it will create an issue.
Is there any way of directly uploaded files from the user's system to amazon S3 in chunks without storing the file on server temporarily?
If upload the files via frontend directly then there a major risk of keys getting exposed.
You should leverage the upload directly from client with Signed URL
There are plenty documentation for this
AWS SDK Presigned URL + Multipart upload
The presigned URLs are useful if you want your user/customer to be able to upload a specific object to your bucket, but you don't require them to have AWS security credentials or permissions.
You could also be interested in limiting the size that user is able to upload
Limit Size Of Objects While Uploading To Amazon S3 Using Pre-Signed URL
Think about signed URL as a temporary credential for client to access a specific S3 location. These credential expire in a short time so there is less security concern, but do remember to restrict the access of the signed URLs appropriately
I am currently developing my first Java based RESTful service that should be run on Heroku. This service manages some objects that have associated images. As Heroku is not able to store this images (apart from storing them in a database), I thought of using an external Content Delivery Network (CDN) like Amazon CloudFront. My first attempt to implement this would be as followed:
Encode the image (base64) on the client side.
Send the encoded image encapsulated in Json to the server side.
Decode the image on the server side.
Process the image (scale, create thumbnails, etc.).
Store the image in Amazon's CloudFront using the AWS SDK for Java.
Store a link to the image with the associated object in a postgreSQL database.
Now my question is, if this is the way to go, or if there is a better way to do this and if this GitHub project is a good point to start. If this helps to give an answer - the images are used within desktop and mobile as well as in web-applications.
Store the image in Amazon's CloudFront using the AWS SDK for Java.
Er, CloudFront doesn't store things, it caches things. If you want it stored, you need to put the image on S3 and pay for it every month.
Encode the image (base64) on the client side
Er, why? Just do a PUT or multipart-mime POST, and send the file as bytes.
Send the encoded image encapsulated in Json to the server side
Again, there's no reason to encode it, you can send the metadata + data in the same POST easily without encoding.
Store a like to the image with the associated object in a postgreSQL database
Storing images in a database is an anti-pattern. It makes the database harder to backup, slower to query, etc. Generally, you want to put the image in S3, and store the path in the database.
What would be a scalable file upload/download system/database?
I'm building a website where users can login, upload images that are private, but truly private. I can't upload them to a map on the harddisk of a server, since that would not scale (what happend if we add more servers?) and it wouldn't be private since everyone could go:
http://127.372.171.33/images/private_picture.png
and download the file.
I am building the project in Play Framework (scala/java)
How do websites like flickr handle these kind of things? Do they put them in a database? And what kind of database would be suitable for this situation?
Thanks for help
I can't tell you how those big sites handle it but putting those images into a database might be one way.
Another way would be to put the files into a virtual filesystem that spans a cluster of servers or distribute them onto different servers and just don't make the directories that contain the images visible to the webserver. Thus nobody should be able to open the image just using the server and the path on that server.
To actually deliver the images you could them implement some streaming service that sends a bytestream to the browser for display (like the webservers would do as well). This service could first check the download permissions for the requested image.
I have an application running on GAE/J that streams video from AWS S3.
I need a solution for protecting the video from being stolen and I found that pre-signed URLs might be it (??).
How can I create pre-signed URLs from GAE/J or there's a better solution to secure the videos?
thanks
I need a solution for protecting the
video from being stolen and I found
that pre-signed URLs might be it (??).
What you're asking for is impossible. Pre-signed URLs will limit the availability of the file to a certain window of time, after which the link will stop working, but there's no way to allow someone to download something with a video player, but not store it to their computer.
Nick, that is not quite right. You can securely stream video with Amazons protected Flash rtmp video streaming service, using Cloudfront. Nobody will be able to download your files. There are tutorials about how to do it on the net. For instance here. You can also go straight to the cloudfront documentation. (Can't post the link here as I'm new to SO and can only post one link).