I just learned from the console of google app engine that there is a 1000 file per directory limit. This is a real bummer and completely unexpected. I may have to change a lot of what was designed for my application.
The application has some entities on the datastore and these entities may have a link to an image. originally I expected to upload all the images together with the application as static files. But there could be 10,000 images or more.
What solution do you recommend? using another service for serving the images and just providing the link to this other service in the entity? Is there a service inside google apps for this purpose?
You can use Google Cloud Storage for your images. You can also consider a Blobstore service within the App Engine itself.
It is a better idea than uploading them as static files, considering how many images you have. There is also a limit on the total size of an app code that you may run into. Plus, it will be easier to upload new versions.
Related
I created a web application using Java Spring, Hibernate, with JSP's. It's basically a blog I made from scratch, so there's an interface where I can add a new blog post with pictures.
I use Heroku to host and deploy the app.
When I want to post images in a blog post, I do:
And that image is currently stored in a resource folder right outside of 'WEB-INF'. Except, as I add more blog posts, I will have a lot of large sized pictures. I don't know where I should store these images and how I should access it. I thought of storing it in the file system, but wouldn't I have to re-deploy the app on heroku each time I add more images? Another option was having a public dropbox folder and linking the images there, but that would open up my entire dropbox directory to the public. Is there a better way to do all of this, especially managing so many large pictures?
Saving files in file system is not a scalable solution. If you add new servers for your web app the images won't be displayed correctly for any request.
You have to save your images in a distributed environment. It can be a LOB column on the database (Postgres for example) or an Amazon S3 storage system.
The access type is very different but both systems are a good option for your images. If they are big in size, S3 is the best option.
Since from the 4 days i have been trying to find out the path for the uploaded file. I think it wont possible. Can any one tell me how to get the uploaded file path in java web application. Is there any external API to get the uploaded file path? And my project is google app engine type project. Please some one answer it.
As you can't write to the file system it's likely you can't do whatever it is you are trying to do. So you need to use one of the storage options available instead, likely GCS.
https://developers.google.com/appengine/docs/java/googlecloudstorageclient/
Google Cloud Storage is useful for storing and serving large files.
Additionally, Cloud Storage offers the use of access control lists
(ACLs), and the ability to resume upload operations if they're
interrupted, and many other features. (The GCS client library makes
use of this resume capability automatically for your app, providing
you with a robust way to stream data into GCS.)
I'm trying to work on an app which uses GTFS. This may seems like a stupid question but I couldn't find any answer to it.
The GTFS for Israel, a rather small country with not so many buses infrastructure, is around 120 MB zipped file.
Right now the only possible way I could think of for getting it working is to download the file, but downloading 120 MB using the phone could take quite a long time. Sure you can do this only once and save it in a database on the phone, but it still requires downloading 120 MB.
Since it is zipped, I can't unzip it over the server and than just get the txt files..
So basically I'm asking, How can I get the information to the phone, without downloading the zipped file?
I've seen and used apps which uses that same GTFS file, and they load up really fast, even on the first load..
I hope you understand my issue, not sure how to explain it better.
Thanks!
P.s I would make an iPhone app too, and it's the same issue, hence the iPhone tag
One approach might be to preprocess the GTFS data during your app development. You could load it into a SQLite database, and use Core Data to get the data you need out of the file at runtime. This also gives you an opportunity to include only the data that you actually need for your app - it doesn't make sense to ask users to download extra data that they won't need.
Use protocol binary format (pbf) formely google and now open source. It is compact and very fast searchable, so no need to decompress it on a device and load it into a database on that device because pbf acts as a database. Just include pbf library in your code to query it. Of course you have to compress it once before distributing the data online.
I have a problem I've been dealing with lately. My application asks its users to upload videos, to be shared with a private community. They are teaching videos, which are not always optimized for web quality to start with. The problem is, many of the videos are huge, way over the 50 megs I've seen in another question. In one case, a video was over a gig, and the only solution I had was to take the client's video from box.net, upload it to the video server via FTP, then associate it with the client's account by updating the database manually. Obviously, we don't want to deal with videos this way, we need it to all be handled automatically.
I've considered using either the box.net or dropbox API to facilitate large uploads, but would rather not go that way if I don't have to. We're using PHP for the main logic of the site, though I'm comfortable with many other languages, especially Python, but including Java, C++, or Perl. If I have to dedicate a whole server or server instance to handling the uploads, I will.
I'd rather do the client-side using native browser JavaScript, instead of Flash or other proprietary tech.
What is the final answer to uploading huge files though the web, by handling the server response in PHP or any other language?
It is possible to raise the limits in Apache and PHP to handle files of this size. The basic HTTP upload mechanism does not offer progressive information, however, so I would usually consider this acceptable only for LAN-type connections.
The normal alternative is to locate a Flash or Javascript uploader widget. These have the bonus that they can display progressive information and will integrate well with a PHP-based website.
For php http://php.net/manual/en/features.file-upload.php
Note the ini files changes in the first comment.
Edit: Assuming you are running into timeout issues.
i am using Google app engine for my development, my project involves around 60 PDfs to be available for users to download.
when i try to upload the project by clicking deploy button in eclipse i get the error app limit exceeded.
i just want to know if i try to use the paid account is there is a different in the application size in paid account or not?
as far as i know its 150 MB for now
You should use Blobstore service to store your PDF files and keep application only for files needed by your application logic and presentation, not data. Here is description of the Blobstore:
The Blobstore API allows your app to serve data objects, called blobs,
that are much larger than the size allowed for objects in the
Datastore service. Blobs are created by uploading a file through an
HTTP request. Typically, your apps will do this by presenting a form
with a file upload field to the user. When the form is submitted, the
Blobstore creates a blob from the file's contents and returns an
opaque reference to the blob, called a blob key, which you can later
use to serve the blob.
All good advice above, try to avoid putting content like that in your code. My app hit this issue and only has about 10MB of code/images/resources. What takes up a lot of space is the GWT compiling of 15 permutations of your app.
One thing that helped me, was changing my GWT javascript generation output style from Details to Obfuscated, resulting in much smaller code. You can also limit the number of permutations being created.
https://developers.google.com/web-toolkit/doc/1.6/FAQ_DebuggingAndCompiling#Can_I_speed_up_the_GWT_compiler?
According to http://code.google.com/intl/de/appengine/docs/quotas.html#Deployments the applications may not exceed 10 MB.
upto 10MB data u can upload to ur app engine
see following link
http://code.google.com/appengine/docs/quotas.html