We Keep Coding

What to test after changing only the source and target version

We have been following the official Java migration guide to upgrade our application from Java 6 to Java 8. Unfortunately we haven't set source and target version to 1.8 to prevent the usage of new language features.
So currently we are compiling and executing our application with JDK 1.8 / JRE 1.8 but set source and target level to 1.6.
Nevertheless, we now even want to level up the source and target level for our applications to 1.8.
Does setting those properties only changes the allowed features for the compilation and the binary format of the classes or does the configuration changes the semantics of the application? We expect that there shouldn't be any known issues or incompatibility when updating the source and target version to a higher version, when staying on the same JRE for execution and JDK for compilation.

Does setting those properties only changes the allowed features for the compilation and the binary format of the classes or does the configuration changes the semantics of the application?
It can actually change the semantics of the Java language. For example, the meaning of #Overrides changed between Java 5 and Java 6. (I can't recall any changes like that between Java 6 and Java 8, but ....)
And of course:
There will be library bug fixes.
There will be cases where methods or classes are re-implemented (e.g. Arrays.sort and HashMap) with differences in unspecified aspects of library behavior.
Changes to the JIT compiler can lead to native code being optimized differently which can lead to timing or memory model-related regressions in (your) buggy multi-threaded code.
In short, even though upgrading from one version to the next ... or just changing the --source or --target is usually innocuous, it can sometimes lead to problems that need to be fixed.
So ... test everything ... thoroughly. Don't assume that it will all just work when you roll our the newer version.
Of course, it is advisable to read the lists of bug fixes and incompatibilities in the respective release notes. But be aware that the lists may be incomplete, or that you won't realize that some fix impacts your code. (Who knows / remembers every implementation detail of an large application written many years go?) And be aware that the fixed bug lists in a new major release are incremental from a previous patch release.
The counterpoint is that, you can't just defer upgrading because. The longer you put it off, the harder it becomes and the larger your codebase's technical debt becomes. And you may find yourself having to do the upgrade in a panic; e.g. due to an unpatched security problem, an OS support issue, and so on.

According to Compatibility Guidelines at the source level, JDK 8 is strongly compatible with previous versions however there're corner cases (explained in the guide).
What you refer to as "changes to the semantics" is a result of not only behavioural incompatibilities, including changes in API implementation (for example this bug which is fixed in 8), but also the platform itself.
Therefore your code compiles and runs, but it might have bugs.
Bottom line, only your test cases can guarantee the smooth transition.
Update
Changing source compatibility to 8 does not break compile of existing source which is in lower level, e.g. 6.
Changing binary compatibility changes the version of the generated bytecode, i.e. compiled .class files, and obviously you compatible runtime environment.
JRE is backward compatible when deals with bytecode. You can run the new bytecode (e.g. your compiled source) along with old bytecode (e.g. your dependencies).
You might experience some semantic changes due to different platform and API implementations, including runtime libraries.
Some of the semantic changes are not easy to catch even with test cases (my personal was experience with Java Crypto API and SSL).

A slightly related to the topic of target and source. This depends on if your application is library which can be included in other projects. For example prior to AS3 you could not include libaries which where compiled with target and source > 1.7 although the code semantics were exactly the same. Even if the library did not include any Java 8 features e.g. streams.
So that might be a "check mark".

Old projects compatible with Java 7

My old projects use Java 6 (1.6), and I don't know when I update (Java 7), they can run fine ?

There is an official list of known incompatibilities between java 6 and java 7 from Oracle (including descriptions of both binary and source-level incompatibilities in public APIs).
Also you can look at the independent analysis of API changes in the Java API Tracker project: http://abi-laboratory.pro/java/tracker/timeline/jre/
The report is generated by the japi-compliance-checker tool.

They should do, yes. Java has a reasonably strong history of backward compatibility. However, if these are in any way important projects you should still perform a thorough test pass before deploying anywhere production-like.

There shouldn't be any compatibility differences as the JVM is basically the same. However it is early days so there may be subtle differences which cause a problem which people are not yet aware of.
e.g. Eclipse looks at the Supplier in the java.exe on Windows and sets the command line arguments differently for different suppliers. It has a problem with Java 6 update 22 because Oracle wanted to change it from "Sun" to "Oracle". I believe this has been changed so it is "Oracle" in Java 7 (but still "Sun" for Java 6)
My point being, that if you write generic Java code, you shouldn't have a problem. However, if you are doing something a bit unusual, you are likely to need to re-test your application.

As was already stated backward compatibility is a very important aspect in new Java releases, so in general there should be no problems in switching to a newer Java version. In this case, however, Java 7 seems to have a few bugs in the new hotspot compiler optimizations. The Apache Software Foundation has issued a warning that their products Lucene and Solr are affected by these bugs.
http://lucene.apache.org/#28+July+2011+-+WARNING%3A+Index+corruption+and+crashes+in+Apache+Lucene+Core+%2F+Apache+Solr+with+Java+7
The affected loop optimizations can be switched off by starting java with -XX:-UseLoopPredicate.

AFAIS here, there's no Java 6 features which get deprecated in Java 7 so yes, your project should run fine.

How many people still use JRE 1.5 or older?

Is there some statistic on how widely people use various JRE's? I'm asking, because I created a program (using JDK 1.6), and found that it would not work on JRE 1.5 and older. So, do I need to bother making it compatible, or maybe the percent of JRE 1.5 out there is too small?

Here is some stats:
http://www.statowl.com/java.php
Don't really know how it's calculated... I think come from users browser, but don't know which websites.

You need to have your JDK emit bytecode for older JVMs with the -target SDK_VERSION argument to javac. Bytecode from one version of the JVM is not compatible with an older release of the JVM.
Thanks to comment
The above only works if you are not using functionality that older JVMs don't have, for example, if you took advantage of the Java Desktop API, which was introduced in 1.6, you wouldn't be able to target an older JVM anyway.
I won't throw statistics your way, I don't really know what your application is/does/who it's intended for, but there are companies and people who still use older JDKs - perhaps due to heavy investment in some specific release of the JVM and it would require significant time and testing to ensure that moving to a newer version does not break their existing software/code, or perhaps some software strictly mandates it. I worked with software from a company (which shall remain unnamed) that ONLY wanted a specific patch revision of an older JVM - their software refused to work on anything newer.

Um yes plenty of enterprises use earlier versions of Java. You simply need to decide what your lowest level target JVM is. Don't forget you can use the -source & -target parameters to specify the type of source code and byte code compatibility with earlier versions.

I don't know of any statistics, but for what is worth I am certain people use Java 1.5. If your app is geared toward general public, I don't think you need to bother, but in specialized environments it might be.

If your application will be for consumers, you will probably be fine relying on 1.6.
Most places that are still running 1.5 are for business applications that have not been updated.

If you're not using any Java 1.6 specific features, you can pass a command line argument to javac to target the 1.5 framework. The argument to add is -target 1.5. Obviously this wouldn't work if you're using any new features shipped with 1.6.

Security issues of compiling against older JDKs

Are there any known security issues with compiling against an older JDK? For instance lets say Oracle decides to change something in one of their implementations that was to close a security vulnerability. If we compile against the older JDK would we still be vulnerable?
EDIT
We are running against JDK 6 and we compile against Java 5, which is the crux of the issue I was attempting to get at for any one else whom sees this question. For the sake of completeness lets say that in the ant build we target 1.5
EDIT 2
Additionally there is the issue of calls in the application that potentially have private APIs/implementations that are not directly accessed.

Aside from some very hypothetical niche situation that hasn't ever happened yet, the only thing that matters is the version with which the application is executed.
I don't think there are any known security issues that have been "fixed" with #deprecated, because that wouldn't really be appropriate.
There are two issues at play here:
1) You can have your Java compiler produce binaries that are binary compatible with older versions. That affects the language features available to you.
2) You can compile against the Java system libraries of an older version, and that will affect what methods/classes are available, as they are always adding new ones.
But neither of these affects the security of your application. What matters is the runtime version with which the application is executed.
Observe that security problems are fixed between updates, not the major versions, which introduce changes in the language itself. For example, Java 6 is currently at update 21. Java 6 update 19 fixed stuff that was vulnerable in Java 6 update 18.
When Java 6 update 19 was released, updates were released for Java 1.5 and Java 1.4, to fix the same issues in Java 1.5 (update 24) and Java 1.4 (update 26). See the security baseline table here: http://www.oracle.com/technetwork/java/javase/6u19-141078.html

If you compile against JDK 1.5:
Users that are using JRE 1.5 will be vulnerable
Users that are using JRE 1.6 will not be vulnerable
The application will run using the libraries of the users's runtime. Since the vulnerabilities are fixed in Oracle's library, which is now being called by your application, the security vulnerability will be alleviated.
However, since you compile against 1.5, your users will be able to use 1.5. Do anything you can to avoid the vulnerability; if you can't avoid it, detect the JRE version at startup and issue an alert if apporpriate.

If Oracle changes an API (not an implementation) to promote security, they will probably add an #Deprecated annotation but preserve backward compatibility for some time. The annotation will trigger a warning in a newer compiler, but not in yours, and not in the compiled program. (Unless Oracle decides to throw an exception or log a message from the deprecated method.)

I would say that those unfixed vulnerabilities are still there, and you and your app are still at risk.
It'd be the same issue as running with an old, unpatched OS or browser.
It's a good incentive to migrate up to newer JVMs.

Consequences of running a Java Class file on different JREs?

What are the consequences of running a Java class file compiled in JDK 1.4.2 on JRE 1.6 or 1.5?

The Java SE 6 Compatibility page lists the compatibility of Jave SE 6 to Java SE 5.0. Furthermore, there is a link to Incompatibilities in J2SE 5.0 (since 1.4.2) as well. By looking at the two documents, it should be possible to find out whether there are any incomapatibilities of programs written under JDK 1.4.2 and Java SE 6.
In terms of the binary compatibility of the Java class files, the Java SE 6 Compatibility page has the following to say:
Java SE 6 is upwards binary-compatible
with J2SE 5.0 except for the
incompatibilities listed below. Except
for the noted incompatibilities, class
files built with version 5.0 compilers
will run correctly in JDK 6.
So, in general, as workmad3 noted, Java class files compiled on a older JDK will still be compatible with the newest version. Furthermore, as noted by Desty, any changes to the API are generally deprecated rather than removed.
From the Source Compatibilities section:
Deprecated APIs are interfaces that
are supported only for backwards
compatibility. The javac compiler
generates a warning message whenever
one of these is used, unless the
-nowarn command-line option is used. It is recommended that programs be
modified to eliminate the use of
deprecated APIs, although there are no
current plans to remove such APIs
entirely from the system with the
exception of JVMDI and JVMPI.
There is a long listing of performance improvements in the Java SE 6 Performance White Paper.

Java classes are forward compatible , e.g. classes generated using 1.5 compiler will be loaded and executed successfully without any problems on JRE 1.6. Generally your classes genereated by today java compilers will be compatible with future JREs (for example Java7)
The inverse does not hold : you can not run classes generated by 1.6 on older JREs (1.3, 1.4, etc).

Java compilers specify source and target compliance levels. This way, you can compile for any JRE from any other higher-versioned JRE. You need to make sure to use these compliance levels because there are API differences between JREs. For example, JRE 1.5 introduced StringBuilder at the compiler level. This means any time you do:
String s = "string1" + "string2";
The compiler changes it to:
String s = new StringBuilder("string1").append("string2").toString();
Obviously, this will break with a NoClassDefFoundError when you attempt to construct the StringBuilder.

Theoretically, nothing. The JVM is supposedly backwards compatible. Myself, I've never had a problem in that direction.

Depends entirely on what parts of the java library you are using. It could be anything from 'absolutely fine, no difference whatsoever' to 'OMG!! WHY HAS IT JUST FORMATTED MY HARD DRIVE??' (Well, perhaps not this second one, but it serves to support the point of it going from nothing to possibly bad :)).
Your class could also pick up on bug fixes in the library as well, which would mean niggling bugs disappear (or could be introduced depending on if you were relying on buggy behaviour or not).
AFAIK though, the java bytecode is backwards compatible so you shouldn't get any issues with it just not doing anything.

One positive consequence is that the 1.4 classes will still take advantage of speed improvements made to the JVM (although not necesarily improvements made to library classes).

just ran into a problem like this myself. I was writing code that should work with 1.6 but the college had 1.3 installed. Lots of methods just don't work i.e
input = ""+ JOptionPane.showInputDialog(null,"Enter a four digit number to " + (b?"encrypt":"decrypt")+".",(b?"4086":"5317"));
wouldn't work but
input = ""+ JOptionPane.showInputDialog(null,"Enter a four digit number to " + (b?"encrypt":"decrypt")+".");
would. the inputdialog method that accepts three agruments doesn't seam to exist in 1.3.
this is just a long winded way of saying working with 1.6 api on 1.3 results in head slamming incidents.

It should work. I don't remember encountering any problems with it, except when parts of the Java API are deprecated, in which case it'll explain what they are anyway and you can hopefully write a workaround.
Of course, running a class file compiled with JDK 1.6 in JRE 1.5 would cause a problem - even a JRE only minor build revisions older will throw an error.