Attempts to deploy my application to appengine have failed because of the hard limit on uploads i.e. 10,000.
My application is using external libraries and constants in 2 other languages. Please refer to the following snapshot:
GWT.Async blocks have been placed in necessary positions in the project.
Following compile time options are used:
-localWorkers 3 -XfragmentCount 10
But the problem is when I upload the project to appengine I get the following exception:
**
java.io.IOException: Applications are limited to 10000 files, you have
34731
**
I am aware that I can cut down on the file count by reducing the cross browser compatibility or by reducing locales. But that won't be a practical approach while deploying
So please suggest me some alternatives.
Another thing I wish to mention is the project extensively uses VerticalPanel/HorizontalPanel/FlexTable/DialogBox in most of its screens. I am not sure if this has something to do with this problem.
I'm afraid this will happen to me also, I had that problem at the middle of the project, so I limited browsers to chrome and ff. But when I'll have to really deploy, this could be an issue.
An application is limited to 10,000 uploaded files per version. Each file is limited to a maximum size of 32 megabytes. Additionally, if the total size of all files for all versions exceeds the initial free 1 gigabyte, then there will be a $0.13 per GB per month charge.
https://developers.google.com/appengine/docs/quotas#Deployments
The solution could be to deploy each language as an application, if your data is not related together between languages
Sounds like you might also be deploying all of your gwt classes along with your application.
When I was a heavy appengine user, I was sure to jars all my uploaded classes (and not include any non-shared gwt code). You might want to $ find . -n "*.class" | wc -l to count how many classes you are sending.
Jarring up your classes beforehand will make 15000 class files = 1 jar file.
It just sucks to make huge jars since you'll need to redeploy the whole jar on every change. Better to have lots of small jars. ;)
What I did is to put all the GWT generated files into a ZIP and serve them with a servlet.
To optimize things a bit I put every file in memcache after dezipping.
Related
An application I have uses Java Agents with need large jar Libraries (the biggest one is PDFBox - all in all 11MB). They were running for 3 years without any issue with the jars in jvm/lib/ext.
During an upgrade to Domino 9.0.1FP6 the administrator forgot to reinstall the jars in jvm/lib/ext - with obvious repercussions. (Such an annoyance that IBM just completely replaces the whole jvm sometimes without being gentle to the jars)
Upon request, I changed the code by including the jars directly into the Java Agents. Things worked well for 2-3 days, and now we're getting OutOfMemory errors.
As far as I understand it, the jars get loaded onto the Java Heap when the agents get started, but the garbage collection is working slower than the continuous loading of the jars into the heap. I couldn't find any precise documentation by IBM on this matter.
We've increased JavaMaxHeapSize in the notes.ini of the servers but that didn't bring the expected results.
I'm dismissing the possibility that I have forgotten a recycle() in my code because it run beforehand with no memory leaks for three years.
I have thought of the possibility of running a separate Agent that checks total memory usage and then runs Sytem.gc() but I'm not convinced since I have no guarantee that the garbage collector will actually fire.
Apart from the obvious move of putting back the jars in jvm/lib/ext, is there an alternative that I haven't considered?
And is there anywhere some sort of documentation about how these classes get loaded into the Heap, and whether there's a possibility that the jars erroneously are not recognized as being garbage-collectible?
It's a memory leak bug - see http://www-01.ibm.com/support/docview.wss?uid=swg1LO49880 for details.
You need to go back to placing the jar files in jvm/lib/ext.
We use Eclipse to develop and Websphere 7.0 to run our code. We use a plugin for Eclipse which allows you run run a copy of WAS locally and integrated with Eclipse. This works but it's not great. Each time I have to make a change I have to run my Ant script (5 mins.) then do my install into WAS and restart the app in WAS (another 5 mins). This is not efficient (actually it's downright annoying!)
What's the best setup Websphere allows, ignoring for a min what is easy to do etc. I want to know what to aim for. Obviously I would love to be able to save in eclipse then go to my browser and refresh the screen, am I mad?
I have looked at this about using Websphere's native Ant, and it is one avenue but I'm sure there's better.
Can I use jpda like I do when starting Tomcat and develop directly in the code? I will still need Ant when doing a full re-build as I have some Java file generation to do but for the in-process builds???
Thanks-in-advance for any and all help.
There is not just one straightforward single best answer as it depends a lot on the application you are doing.
If you rely on ant to make the build and that takes five minutes for any update you do then it may be hard to get away from that. You may not have to do it like that, you may be able to do incremental compilations (only compiling the updated classes) and that should be a lot quicker (but since I don't know anything about your application it is hard for me to tell you that this is a way forward for you). The updates you make in WAS and if that requires a total restart of the application depends on many different factors.
If I update only a jsp file it usually does not require a full application restart, but if you start to update the classes and definitely when you update your web.xml or something like that, a full restart of the application may be required.
The time it takes for you to restart your application depends a lot on what actions your application will perform when you start it up. If you have an application that takes five minutes to start up, then every restart will cost a lot. Is there something you can do to reduce the restart time in certain configurations? Can you divide your application into separate deployables to reduce the startup time for each individual application?
It all depends on your needs and requirements.
We used to have long startup times for our local servers, but since we had upgraded disks (SSD-disks) and enough memory the server startup time has been reduced by 80-90%.
I have made two rather small applications in java- and c-language. The application is a financial calculator (Black & Scholes) where one can calculate the call- and put-price given the parameters assetprice, strikeprice, volatility, time and interest rate.
I somehow expected the apps to be approximately the same size in kilobytes - but I was surprized when I discovered that the c-file is much heavier:
c-program (exefile): 450 KB
java-program (jar): 11.7 KB
That is - when the c-program is almost o.5 MB the jar-file is as little as 11.7 KB.
How could this be explained?
(the firt image shows the c-app and the last is the java app)
Your trying to compare apples and oranges. Java java is an interpreted language compiling bytecode and using libraries available within the jvm. c compiles into machine code and includes static libraries but not dynamic shared libraries acailable by the os.
So, this depends on many factors and the specific project.
C apps also depend on compiler used and settings. Eg gcc will affect size differently than ms visual studio and a debug mode compile will be way larger than a non debug. Also optimizations and obfuscators play a role.
Without knowing all the details I would argue that the c-program is self-contained, i.e. you can run it on virtually any Windows machine.
To run Java applications, you need the JVM installed, which brings a lot of functionality (like... the String type, or Math libraries), so these do not need to be included in your deployed jar file.
The c file has incompiled a lot of runtime and other overhead, while java is almost interpreted. See, c is standalone and java needs the rontime
I am getting killed on performance with file / folder deletes in Java.
The code is quite old and I am wondering if Java 7 (which I upgraded to) actually offers performance improvements, or just another syntax. (I don't want to retool everything unless there is a benefit). I regularly need to extract large ZIPs and then delete the contents and the recursion time is brutal.
I am also stuck on Windows.
Thanks
I would to suggest to use some kind of jar already provided by community.
For example, common-io.x-x.jar, spring-core.jar
Eg, org.apache.commons.io.FileUtils;
FileUtils.copyDirectory(from, to);
FileUtils.deleteDirectory(childDir);
FileUtils.forceDelete(springConfigDir);
FileUtils.writeByteArrayToFile(file, data);
org.springframework.util.FileSystemUtils;
FileSystemUtils.copyRecursively(from, to);
FileSystemUtils.deleteRecursively(dir);
File IO is very dependant on the performance of your hardware. Many HDD can perform 80 - 120 IOPS per second. If you want to open a file you can read up to 120 files per second. To delete a file it can require two updates or up to 60 files deleted per second. With these constraints there is almost nothing you can do in software which will make any difference.
If you have an SSD however, these can do 80,000 to 230,000 IOPS per second (more than a thousand fold increase) At this point what you do software might make a difference, but as you are dealing with compressed files, it most like that CPU will be your bottleneck at this point.
First please dont overlook because you might think it as common question, this is not. I know how to find out size of file and directory using file.length and Apache FileUtils.sizeOfDirectory.
My problem is, in my case files and directory size is too big (in hundreds of mb). When I try to find out size using above code (e.g. creating file object) then my program becomes so much resource hungry and slows down the performance.
Is there any way to know the size of file without creating object?
I am using
for files File file1 = new file(fileName); long size = file1.length();
and for directory, File dir1 = new file (dirPath); long size = fileUtils.sizeOfDirectiry(dir1);
I have one parameter which enables size computing. If parameter is false then it goes smoothly. If false then program lags or hangs.. I am calculating size of 4 directory and 2 database files.
File objects are very lightweight. Either there is something wrong with your code, or the problem is not with the file objects but with the HD access necessary for getting the file size. If you do that for a large number of files (say, tens of thousands), then the harddisk will do a lot of seeks, which is pretty much the slowest operation possible on a modern PC (by several orders of magnitude).
A File is just a wrapper for the file path. It doesn't matter how big the file is only its file name.
When you want to get the size of all the files in a directory, the OS needs to read the directory and then lookup each file to get its size. Each access takes about 10 ms (because that's a typical seek time for a hard drive) So if you have 100,000 file it will take you about 17 minutes to get all their sizes.
The only way to speed this up is to get a faster drive. e.g. Solid State Drives have an average seek time of 0.1 ms but it would still take 10 second or more to get the size of 100K files.
BTW: The size of each file doesn't matter because it doesn't actually read the file. Only the file entry which has it s size.
EDIT: For example, if I try to get the sizes of a large directory. It is slow at first but much faster once the data is cached.
$ time du -s /usr
2911000 /usr
real 0m33.532s
user 0m0.880s
sys 0m5.190s
$ time du -s /usr
2911000 /usr
real 0m1.181s
user 0m0.300s
sys 0m0.840s
$ find /usr | wc -l
259934
The reason the look up is so fast the fist time is that the files were all installed at once and most of the information is available continuously on disk. Once the information is in memory, it takes next to no time to read the file information.
Timing FileUtils.sizeOfDirectory("/usr") take under 8.7 seconds. This is relatively slow compared with the time it takes du, but it is processing around 30K files per second.
An alterative might be to run Runtime.exec("du -s "+directory); however, this will only make a few seconds difference at most. Most of the time is likely to be spent waiting for the disk if its not in cache.
We had a similar performance problem with File.listFiles() on directories with large number of files.
Our setup was one folder with 10 subfolders each with 10,000 files.
The folder was on a network share and not on the machine running the test.
We were using a FileFilter to only accept files with known extensions or a directory so we could recourse down the directories.
Profiling revealed that about 70% of the time was spent calling File.isDirectory (which I assume Apache is calling). There were two calls to isDirectory for each file (one in the filter and one in the file processing stage).
File.isDirectory was slow cause it had to hit the network share for each file.
Reversing the order of the check in the filter to check for valid name before valid directory saved a lot of time, but we still needed to call isDirectory for the recursive lookup.
My solution was to implement a version of listFiles in native code, that would return a data structure that contained all the metadata about a file instead of just the filename like File does.
This got rid of the performance problem but added a maintenance problem of having to native code maintained by Java developers (lucking we only supported one OS).
I think that you need to read the Meta-Data of a file.
Read this tutorial for more information. This might be the solution you are looking for:
http://download.oracle.com/javase/tutorial/essential/io/fileAttr.html
Answering my own question..
This is not the best solution but works in my case..
I have created a batch script to get the size of the directory and then read it in java program. It gives me less execution time when number of files in directory are more then 1L (That is always in my case).. sizeOfDirectory takes around 30255 ms and with batch script i get 1700 ms.. For less number of files batch script is costly.
I'll add to what Peter Lawrey answered and add that when a directory has a lot of files inside it (directly, not in sub dirs) - the time it takes for file.listFiles() it extremely slow (I don't have exact numbers, I know it from experience). The amount of files has to be large, several thousands if I remember correctly - if this is your case, what fileUtils will do is actually try to load all of their names at once into memory - which can be consuming.
If that is your situation - I would suggest restructuring the directory to have some sort of hierarchy that will ensure a small number of files in each sub-directory.