We Keep Coding

Tomcat. dynamic extend heap memory

I have a task to automaticly extend heap memory for our tomcat application. I know it isn't possible by JDK features, and I'm looking for any dirty(or not) hack to make that kinda feature. The main reason of that demand is minimal configuration of tomcat before start demo version of our application. In other word no user of our application will have any access to configuration of JVM/tomcat.
Application required ~1024M of heap memory, default value for tomcat8 is 256M, which isn't appropriate for our goals.
At this moment I have two possible solutions:
An a .sh/.bat script which will configure tomcat. pros - it will do the work, cons - it's another point of configuration demo stand(copying of a script)
An wrapper for our application, which goes in the same war file and configure tomcat if required. pros - it will do the work, and no new configuration point(Just install tomcat and copy the war file),
Is there another way to do that in more... common and simple way?
EDIT The main goal is to make installation of our application in following steps:
Install tomcat
Copy war file
Start tomcat
no any additional configuration, just copy war and start tomcat

That is commonly solved by wrapping the installation and configuration of tomcat in an installation script. Pros: the end user just download the installer and runs it; cons: the installation script must be tailored for the final environment (one for Windows, one for Linux) and can be hard to write. A sometimes simpler way is to provide a zip containing a readme.txt file and a install.bat (or install.sh)
An alternate way if the configuration is really complex is to directly configure a VM (VMDK is a de facto standard), and let the users install the virtualizer they prefere.

How are Java applications deployed in the "real world"?

As a novice to the world of Java programming, this question has always boggled my mind. I first believed that all Java files were compacted into applets and then ran, but I soon realized that this isn't always the case. Could someone explain to me how we actually interweave our Java applications into a real product of everyday life?
TL;DR: How do we implement our code for practical usage?

It depends on the application. There are many options depending on how you want your users to use your app. Usually it's packaged as a jar or a specialized jar (war, ear).
In theory, you could zip the raw directory structure with your .class files in it and provide a shell script/instructions that run the java command for the user. I don't recommend this because it's kind of unprofessional and requires you to maintain a shell script for each OS you want to be able to run the program on.
Jar files are used to package libraries but you can also have a manifest file in it that says, "When someone double clicks/executes this, run this class". That class can start up a GUI or be a headless task that responds to the parameters, etc.
You can have applets, like you said. These programs are run in the user's browser.
You can have a war file, which is a way to package a web application. You give this to a web server and it knows how to deploy it so that you can visit the web pages. An example web server/container is tomcat or jetty.
You can have an ear file which can contain other war files inside it. This is used for applications that need other parts of the javaee functionality (ejbs, jms queues, etc.). An example of an application server is jboss or glassfish.
There's also java web start apps. These are apps you can run by visiting a webpage, but they get downloaded to your computer and run on the user's computer (instead of on the server's backend, like in a war/ear).
There's also javafx. I don't know anything about that though. By skimming the FAQ, it appears to be Java's answer to Adobe's Flex. You configure UI components with an xml configuration. I'm not sure what format JavaFX apps use, but it does say, "Deploy on the desktop or in the browser".
As Sotirios Delimanolis mentioned in a comment below, you can build these files with build systems like Ant or Maven. You can also build them "by hand" with the tools that come with the java/javaee sdk. For example, you should have a jar command in your path if you installed the sdk. Here are some details of these build systems:
Maven
High level (you tell it what to build, not how to build it)
Much more than just a build system. It also has dependency management, etc.
Opinionated (it uses convention over configuration, each config file generates 1 artifact, etc.)
Ant
Low level (you tell it how to build things)
Flexible
Config files can do whatever you want, build as many artifacts as you want
Easy to learn
SDK tools
Always up to date. EG: Very rarely, maven/ant may not be able to set a configuration option
Difficult to remember commands
Very low level
By itself, not repeatable (EG: unless you build a script, you will have to type the jar command yourself each time)

Applets never really caught on and are very rarely used nowadays.
Simple applications can be deployed as "executable" JAR files , which are basically ZIP archives with additional metadata that tells the JVM which class contains the main method to run. They can be run on the command line using the -jar option, or in most desktop environments by double-clicking (this requires a JVM to be installed as well).
Desktop applications can be deployed via Java Web Start or installers like IzPack or Install4J, but Java desktop applications are not very common either.
Most Java software nowadays runs only on servers (web servers or app servers). They are typically deployed as WAR or EAR files, which are also ZIP archives containing classes and other resources. These applications then run inside a server component following the Servlet or EJB standards.

If the application is mean to run on a client, it is packaged as an executable JAR, then further packaged as an Application Bundle (Mac), maybe wrapped in an exe (Windows), or paired with an executable script that will launch the JAR and set any required VM arguments.
If it is part of a web application, then it will be packaged as a WAR or EAR and placed into the appropriate location on the web server.
If it is simply a library, then it is usually packaged as a JAR (non-executable) and distributed as such for integration into larger projects.
applets and then ran, but I soon realized that this isn't always the case
Actually, applets are rare nowadays and their use is discouraged.

Create an executable jar, a war which is dropped into a web server or a library that is used by another project that is one of the previous two.

Can not put anything into /var/lib/tomcat7/webapps/

I was starting on JAVA web development today and encountered some problems, I installed my tomcat7 on my ubuntu machine. Now when I browse to //localhost:8080, I get the default welcome page saying:
This is the default Tomcat home page. It can be found on the local filesystem at: /var/lib/tomcat7/webapps/ROOT/index.html
Tomcat7 veterans might be pleased to learn that this system instance of Tomcat is installed with CATALINA_HOME in /usr/share/tomcat7 and CATALINA_BASE in /var/lib/tomcat7, following the rules from /usr/share/doc/tomcat7-common/RUNNING.txt.gz.
But strangely when I try echo $CATALINA_HOME nothing shows up.
Also I can not copy/create anything in the default /var/lib/tomcat7/ROOT though it's just a matter of providing few permissions but I was wondering whether it is the right way to do it?,
What I would like to do is create a separate directory in my home where I can put my web application and tomcat can read hem from there. Is there a way to do it? In apache I can do it by changing the document-root and directory but I don't know how to do it for tomcat

But strangely when I try echo $CATALINA_HOME nothing shows up.
This is because the packaged version sets CATALINA_HOME just prior to launching Tomcat, after reading it from a configuration file (typically somewhere in /etc).
Also I can not copy/create anything in the default
/var/lib/tomcat7/ROOT though it's just a matter of providing few
permissions but I was wondering whether it is the right way to do it?
The permissions problem has to do with you not being root (or the Tomcat user). Most packaged Tomcat installations (deb or RPM) tend to install with a specific user in mind, and copying stuff in as a different sometimes won't work.
/usr/share/tomcat7 is your CATALINA_HOME directory, and it has links to the other directories, such as /var/lib/tomcat7/webapps, /etc/tomcat7, etc. You shouldn't copy directly into a web application, you should package the web application into a WAR file and "deploy" it. The advantages are numerous.
What I would like to do is create a separate directory in my home
where I can put my web application and tomcat can read hem from there.
Is there a way to do it?
Yes, one is created when "deploying a web app". Look to the standard Tomcat7 documentation, and consider installing the "manager" web application if you like a non-command line interface. Now that you know what "installation" of a web app is called, it will likely be an easier task.
In apache I can do it by changing the document-root and directory but
I don't know how to do it for tomcat
Tomcat has a different, slightly more restrictive set of requirements for a document-root. You need to learn it, and just come to terms with the idea that it's never going to be the same. Basically under the "webapps" directory, is a WAR file or a expanded directory that came from a WAR file. Editing in-place is not a good idea for Tomcat, as the CGI-equivalents are not read from disk each time they are ran, they are linked into the memory of Tomcat. This means that a change in the directory might not affect your web application, which is good enough reason to not bother changing the on-disk files for a deployed web application.
Modify the web application, repackage it, and update the deployment. It's really the "right" way to go with Tomcat.

Give permission 777 to the webroot folder
sudo chmod -R 777 Webroot
After moving to the tomcat folder

How do I create an Installer for a Java Application?

This is still awhile down the road for me but for my Project Implementation class we have to create a program and then distribute it. I have written an application in Java and from the specification I have made in the previous class (Project Design) my application will need to be platform-independent.
For mac and linux the user can just run the jar file from the terminal, but for windows I would like to have the Application installed to the path user chooses (default: C:\Program Files(x86)\NameOfApplication), Create a desktop shortcut (if the user wishes to have one), install under the start menu (if the user wants it to) and then also show up in the add\remove programs list.
Is there any easy way to do this?
Is it any harder if I did decided to create an installer for mac and linux?
Thanks in Advance.

You can create an installer with NSIS, even for a Java application.
You might also consider distributing your application via Java Web Start.

There are opensource installer generators for java. I have never used one before. Here is a good resource of links
I recommend using Java Web Start.
It has several advantages.
Available for all major desktop platforms
Single distribution for all JWS-enabled platforms
Code-signing and sandboxing
Versioning and incremental updates
Automatic installation of JREs and optional packages
It has one major disadvantage.
Internet connectivity is required if JWS, JRE, and/or an Optional
Package is not present on the system
Have a look here and here

Install4j does what you want, although you have to pay for it. Personally, I am not aware of any free alternatives. You can make installers for Linux and Mac OS as well.

you can use Exe4J, see http://www.ej-technologies.com/products/exe4j/overview.html

You can do most of that using standard JNLP:
http://docs.oracle.com/javase/1.4.2/docs/guide/jws/developersguide/syntax.html
You make a JNLP file that takes the executable JAR from some local (or remote) location and creates a Desktop icon for it (of your chosing). Only difference is that the actual JAR will be placed in the JDK's jar cache directory (not in a directory of your choice - I don't think the user would care much).
The huge advantage with this is that if you make a JNLP that installs the jar from a remote location, you can remotely upload a new version of the jar to that location, and when the user next accesses the jar locally, your latest version will be downloaded and placed into local cache.
Also I recommend you use a smart "fat JAR" builder, which packages all dependency jars inside the executable jar. Eclipse IDE has a way to export a project in this format (and also adds the necessary class-loader so that all works ok from on fat jar).

If your target OS is windows I highly recommend Advanced Installer. It's very very easy to use and will let you create your own native microsoft installer (.msi) with specific target Java VM and a bunch of useful windows features, even in the free version. Note you can also include a private jre into the package.
http://www.advancedinstaller.com/top-freeware-features.html
If you want a "package one deploy everywhere" solution then IzPack is the way to go, platform independant, free and open source.
http://izpack.org/
Depending on the complexity of your project Java Web Start could be a very good option, it's very simple configure and maintain but it relies on the browser's java plugin and believe me... most users DON'T like being warned about certificates and risks everytime they launch an application.

How to avoid copying 40M of java lib's within a WAR when the WAR's size is 41M?

At the moment my build process consists of repackaging the war file with all required java libraries under WEB-INF/lib and then copying the war file to development/demo/production server to be redeployed by tomcat.
The packaged war file's size is about 41M and it has at the moment something like 40M of external java libraries. There has to be a better way. How have you solved this issue?
My development machine is a windows box with Eclipse as my IDE and Ant as my build tool. The servers are all linux boxes with Tomcat 5.5.
Should I maybe add the jar files to the war package at server side?

I can see what you are saying, and have had the same frustration with some of our webapps, but for consistency sake, I would suggest you keep things as they are. If copying the libraries to the tomcat/lib directory you may run into issues with the system classpath vs. the webapp classpath.
By keeping things as they are you are sure you are deploying the same stuff in development/demo as you are in production. Life sucks when you are manually tweaking stuff in production, or you have some crazy error because you forgot to update XYZ.jar from version 1.6 to 1.6.1_b33 in production and now it's behaving differently than what you believe is the exact same code on demo.
When working with something vital enough to have dev/demo/production systems, I think consistency is much more of an issue than .war file size.

We use the rsync tool for this (in our case, using cygwin under windows) to copy the deployments to the servers (which run linux). We use exploded WAR/EAR files (i.e. a directory structure called MyApp.war rather than a zip file called MyApp.war), rsync will only transfer the files that have changed.
In general use, rsync will transfer our 30-40 megabyte exploded EARs in about 5 seconds.

Tomcat has a shared/lib directory, which is the proper place for global application dependencies. However, these will be visible to all applications, which will affect dependency management and may have consequences for things like static variables. I'm not sure if you can configure anything better in Tomcat.
An alternative is to switch to a more sophisticated web container. For example, WebSphere Application Server Community Edition (a blue-washed version of Geronimo) supports per-asset libraries. Other free and commercial servers support this, too. I know WebSphere Application Server does and I'm pretty sure you can do it in Glassfish.

#McDowell, when mentioning those J2EE servers, you should precise that they are J2EE servers(servlet container + the rest).
Like #digitaljoel, I suggest to keep things like they are. It looks like you haven't done much web application deployment yet. The issues that you'll have are not worth the price(version conflicts, deployment errors, etc.).

Can you add the non-changing Jars to the Java Library Path on the server side, and only include the regularly changing Jars in your WAR?

you could include the external java libraries in the Tomcat/lib directory. That way they stay on the server.

You could just deploy as a JAR file, replicate your deployment environment locally and just copy over the files that have changed and the jar itself. The pathing is the only real issue.
Or you could look into setting up an EAR.

I work with the 'exploded web application' in the development servers, and occasionally in production as well. The deployment process (based on ANT) updates the JARs in WEB-INF/lib with our packages. Only in the development server, we activate Tomcat reloading which takes care of restarting the application when something changes. You should assign some extra permanent memory to these Tomcats and have a way to restart the server, as the reloading may crash Tomcat from time to time.
I know it's a weird configuration, but I can't understand how constantly repackaging the 30MB (and growing) of our typical application could do any better. May one day the development descriptor allow for external references to libraries which the container may download and cache. ??
Excuse my poor English.

What you need is a version control tool and a build process.
Use CSV,SVN,GIT or whatever fit you to keep your source under control.
use a build tool to build your application : Maven,ant,...
Now, when you want to deploy your application on your server ,you just have to commit your updates on your computer,update your source on your server,build your application and deploy it from the server.
This way , the server will just have to load your modifications and it should be much faster.