Given a Java class file (ClassName.class) with bytecode version X is there a general way to convert this class file from being represented in bytecode version X to being represented in bytecode version Y?
Assumptions:
The source code is not available. The class file is the only available representation of the class.
The class file is heavily obfuscated, so decompiling the class with say jad or similar program and then recompiling it with "-target ..." does not work.
Updates after initial post:
Update #1: Futhermore, assume that bytecode version X and bytecode version Y are sufficiently close so that all instructions used by the class (currently in bytecode version X) also exists in version Y.
For downgrading you can have a look at various methods to get Java 5/6 code running in Java 1.3/1.4. See my anwser to related question Backport Java 5/6 features to Java 1.4?
You could use Apache BCEL
The Byte Code Engineering Library is
intended to give users a convenient
possibility to analyze, create, and
manipulate (binary) Java class files
BCEL gives you the possibility of reading in a class file of a given version, manipulating it, generating a new class file stream, and then loading that into the VM using the low-level ClassLoader API. Very fiddly, no doubt, and I doubt this will let you downgrade the version as easily as you could prograde.
No. While later versions of Java will be able to execute that bytecode, you can't upgrade it: Later versions of the class files have a different format.
You can't downgrade it either because there is no way to replace the missing bytecodes by other constructs in older versions of Java.
[EDIT] IIRC, Sun added at least a single new bytecode instruction for every major version of Java. This is usually the reason for major Java releases: Changes to the bytecode.
That said, just try your luck and change the major version of the class file and see if your newer VM will load it. I doubt it will work for any complex example but you might be lucky.
Use retroweaver
Related
We have been following the official Java migration guide to upgrade our application from Java 6 to Java 8. Unfortunately we haven't set source and target version to 1.8 to prevent the usage of new language features.
So currently we are compiling and executing our application with JDK 1.8 / JRE 1.8 but set source and target level to 1.6.
Nevertheless, we now even want to level up the source and target level for our applications to 1.8.
Does setting those properties only changes the allowed features for the compilation and the binary format of the classes or does the configuration changes the semantics of the application? We expect that there shouldn't be any known issues or incompatibility when updating the source and target version to a higher version, when staying on the same JRE for execution and JDK for compilation.
Does setting those properties only changes the allowed features for the compilation and the binary format of the classes or does the configuration changes the semantics of the application?
It can actually change the semantics of the Java language. For example, the meaning of #Overrides changed between Java 5 and Java 6. (I can't recall any changes like that between Java 6 and Java 8, but ....)
And of course:
There will be library bug fixes.
There will be cases where methods or classes are re-implemented (e.g. Arrays.sort and HashMap) with differences in unspecified aspects of library behavior.
Changes to the JIT compiler can lead to native code being optimized differently which can lead to timing or memory model-related regressions in (your) buggy multi-threaded code.
In short, even though upgrading from one version to the next ... or just changing the --source or --target is usually innocuous, it can sometimes lead to problems that need to be fixed.
So ... test everything ... thoroughly. Don't assume that it will all just work when you roll our the newer version.
Of course, it is advisable to read the lists of bug fixes and incompatibilities in the respective release notes. But be aware that the lists may be incomplete, or that you won't realize that some fix impacts your code. (Who knows / remembers every implementation detail of an large application written many years go?) And be aware that the fixed bug lists in a new major release are incremental from a previous patch release.
The counterpoint is that, you can't just defer upgrading because. The longer you put it off, the harder it becomes and the larger your codebase's technical debt becomes. And you may find yourself having to do the upgrade in a panic; e.g. due to an unpatched security problem, an OS support issue, and so on.
According to Compatibility Guidelines at the source level, JDK 8 is strongly compatible with previous versions however there're corner cases (explained in the guide).
What you refer to as "changes to the semantics" is a result of not only behavioural incompatibilities, including changes in API implementation (for example this bug which is fixed in 8), but also the platform itself.
Therefore your code compiles and runs, but it might have bugs.
Bottom line, only your test cases can guarantee the smooth transition.
Update
Changing source compatibility to 8 does not break compile of existing source which is in lower level, e.g. 6.
Changing binary compatibility changes the version of the generated bytecode, i.e. compiled .class files, and obviously you compatible runtime environment.
JRE is backward compatible when deals with bytecode. You can run the new bytecode (e.g. your compiled source) along with old bytecode (e.g. your dependencies).
You might experience some semantic changes due to different platform and API implementations, including runtime libraries.
Some of the semantic changes are not easy to catch even with test cases (my personal was experience with Java Crypto API and SSL).
A slightly related to the topic of target and source. This depends on if your application is library which can be included in other projects. For example prior to AS3 you could not include libaries which where compiled with target and source > 1.7 although the code semantics were exactly the same. Even if the library did not include any Java 8 features e.g. streams.
So that might be a "check mark".
The compiler display warnings if you use Sun's proprietary Java classes. I'm of the opinion that it's generally a bad idea to use these classes. I read this somewhere. However, aside from the warnings are there any fundamental reasons why you should not use them?
Because they are internal APIs: they are subject to change in a undocumented or unsupported way and they are bound to a specific JRE/JDK (Sun in your case), limiting portability of your programs.
Try to avoid uses of such APIs, always prefer a public documented and specified class.
The JDK 6 Documentation includes a link titled Note About sun.* Packages. This is a document from the Java 1.2 docs, so references to sun.* should be treated as if they said com.sun.*
The most important points from it are:
The classes that Sun includes with the
Java 2 SDK, Standard Edition, fall
into package groups java.*, javax.*,
org.* and sun.*. All but the sun.*
packages are a standard part of the
Java platform and will be supported
into the future. In general, packages
such as sun.*, that are outside of the
Java platform, can be different across
OS platforms (Solaris, Windows, Linux,
Macintosh, etc.) and can change at any
time without notice with SDK versions
(1.2, 1.2.1, 1.2.3, etc). Programs
that contain direct calls to the sun.*
packages are not 100% Pure Java.
and
Each company that implements the Java
platform will do so in their own
private way. The classes in sun.* are
present in the SDK to support the Sun
implementation of the Java platform:
the sun.* classes are what make the
Java platform classes work "under the
covers" for the Sun Java 2 SDK. These
classes will not in general be present
on another vendor's Java platform. If
your Java program asks for a class
"sun.package.Foo" by name, it may fail
with ClassNotFoundError, and you will
have lost a major advantage of
developing in Java.
Try running your code with a non-Sun JVM and see what happens...
(Your code will fail with a ClassNotFound exception)
Yes, because nobody guarantees that these classes or API will be the same with the next Java release and I bet it's not guaranteed that those classes are available in Java versions from other vendors.
So you couple your code to special Java version and loose at least portability.
Sun's proprietary Java classes are part of their Java implementation not part of the Java API their use is undocumented and unsupported. Since they are internal they can be changed at any time for any reason that the team working the Sun JVM decides.
Also Sun's Java implementation is not the only one out there! Your code would not be able portable to JVMs from other vendors like Oracle/BEA and IBM.
Here is Oracle's answer: Why Developers Should Not Write Programs That Call 'sun' Packages
I recently had a case that showed a real-world problem you can hit when you use these classes: we had code that would not compile because a method it was using on a sun.* class simply did not exist in OpenJDK on Ubuntu. So I guess when using these classes you can no longer say things like 'this works with Java 5', because it will only work on a certain Java implementation.
It seems to me that the Java Compiler API allows to compile at runtime a class, writing its output (the .class file) to the file system. However, in-memory compilation is not supported. Is this correct ? or is possible to use this API to compile a class in memory (from a String) and instantiating such class afterwards ?
(I know I can compile the class to the file system and load it afterwards with a custom class loader, but I am wondering if I can compile it in memory, without passing by the file system).
Is there another alternative mechanism to do such in memory compilation using the J2SE only?
BeanShell (I library that can do what I want) mentions in its web page that it may be included in the J2SE at "some point in the future", however, the status of its JSR is "Dormant" (whatever does it mean).
Update:
Ideally, I would like to know if this can be done with the J2SE only (or if there are any expected enhancements to J2SE that will allow me to do this in, for example, Java 8). However, tips about how to do that requirying the JDK to be installed are also appreciated (thanks Evgeniy).
It is possible if you have JDK, java complier is in tools.jar which comes with JDK only. See http://docs.oracle.com/javase/7/docs/api/javax/tools/package-summary.html
Java Compiler API .. in-memory compilation is not supported. Is this correct?
No. The STBC uses the JavaCompiler to do exactly that.
..and instantiating such class afterwards?
The STBC does not go as far as trying to load/run the class, but I believe it should be possible. I imagine it might require a custom 'in memory' class loader though.
..the JavaCompiler API can do that independently if the JDK is installed or not?
From the page..
System Requirements
STBC will run on any computer with a version 1.6+ Java Plug-In* JDK (AKA SDK).
* The API that STBC uses is merely a public interface to the compiler in the tools.jar that is distributed only with JDKs (though the 'public JRE' of the JDK also seems to acquire a tools.jar). ..
there are several sites out providing Java source code.
Unluckily, these does not refer to the Java native binaries.
In short: The source code of native functions is not shown.
An example is StrictMath.floor.
The only thing provided is the raw Java declaration
public static native double floor(double a);
What I am interested in is the actual c/c++ code. How did they actually implement it?
The OpenJDK project contains the full source. The source for jdk 6 is roughly equivalent to what is in the Sun/Oracle jdk. The source for jdk 7 should be almost exactly equivalent.
You can see the native code generated if you use the debug build of the OpenJDK.
StringMath.floor() is turned into a single machine instruction. i.e. some "native" methods are actually inlined by the JVM. Not sure how that helps you though. ;)
See notes of fistp http://stereopsis.com/FPU.html
We are stuck with Java2SE v1.4 till the end of 2010. That's really nasty, but we can't help it. What options do we have to use some of the new features already now? I can think of several ways like
changing the bytecode, e.g. using Retrotranslator or Retroweaver.
backport of libraries, e.g. Concurrent Backport, but this does not help for generics.
emulation of Java 5 features, e.g. checked Collections, Varargs with helper methods, etc.
changing source code by precompilation, stripping all 1.5 stuff before final compilation, e.g. using Declawer can do this.
I am most interested in very positive experience with it in production environments using Weblogic and "real" stuff.
Thanks for your answers. Here is the summary of all relevant answers and my own research.
Changing the bytecode: The Retros
This is done by the "retro"-tools: Retrotranslator, Retroweaver and JBossRetro. Retrotranslator seems to be the most mature
and active of them tool. These tools scan all classes and change the bytecode to remove Java 5 and 6 features. Many Java5 features are supported, some
by using 3rd party backport libraries. This option is most popular and there is some positive feedback from users. Experiments showed that it's working as
expected. See a short overview on developerworks.
Pro: You can develop entirely in Java 5, build modules and all kind of JARs. In the end you just transform all classes to Java 1.4 and package your EAR.
This is easily done with Retrotranslator's Maven integration (org.codehaus.mojo:retrotranslator-maven-plugin).
Con: Conservative environments do not allow changed bytecode to be deployed. The result of the retro-step is not visible to any coder and can't be approved.
The second problem is fear: There might be some cryptic production problem and the retro-code is another step that might be blamed for that. App-server vendors
might refuse help due to changed bytecode. So nobody wants to take responsibility to use it in production. As this is rather a policital than a technical
problem, so I see no solution. It has happened to us, so I was looking for further options :-(
Compiling Java5 to Java 1.4: jsr14
There is an unsupported option, javac -source 1.5 and -target jsr14 which compiles the Java5 source to valid Java 1.4 bytecode. Most features like
varargs or extended for loop are translated by the compiler anyway. Generics and annotations are stripped. Enums are not supported and I don't know
about autoboxing, as the valueOf methods were mostly introduced in Java5.
Con: Only byte code is translated, library usage is not changed. So you have to be careful not to use Java5 specific APIs (but could use Backports).
Further you have to build all modules at the same time, because for development time you propably want Java5 code with generic and annotation information.
So you have to build the entire project from scratch for Java 1.4 production.
Changing Source back to Java 1.4: Declawer
As answered in a related question, there is Declawer, a compiler extension, that works for generics and varargs, but not for enhanced for loop or
autoboxing. The generated source "is a little funky, but not too bad".
Pro: The generated source is available and can be reviewed. In worst case fixes can be made in this source. There is no "magic", because the source
is valid Java. Some people even use JAD (Java decompiler) to get the Java 1.4 source again. The output of Jad readable is readable if you compile with debug
information and don't use inner classes.
Con: Similar to -target jsr14, you need an extra step in the deployment. Same problems with libraries, too.
Changing Source back to Java 1.4: by hand
Several answers suggested doing it by hand. For an automatic, repeating build process this is of course not useful, but for one-time changes it's
reasonable. Just automate what is possible. Maybe look at Antlr for creating a home-grown conversion tool.
Backported Libraries:
The problem is, that Java5 also ships new libraries, that are not available in older JREs, see related question. Fortunately there are several
backported libraries that give you some functionality of Java5, but can't simulate language features, like generics.
Annotations, discussed at TSS
Concurrent
com.sun.net.httpserver (Java 6 to 5)
Gif writing (Java 6 to 5)
Start your own backport project ;-)
You might copy classes you need from the JDK or other libraries, but most likely they are related to other classes.
Emulating Java5 features in Java 1.4 code:
I was thinking about some things you might do to make your life easier and still staying with Java 1.4. The most important features are typesafe collections,
here are some ideas:
Instead of using generics you can create your own typesafe containers with some template.
Add a typesafe Iterator (which is no Iterator any more).
Add asList methods that allows 1,2,...,n arguments and an array of them (to simulate varargs).
Methods for varargs (converting 1,...,n arguments to arrays) and valueOf can be put in some helper class.
sourcecode precompilation, stripping
all 1.5 stuff before final compilation
and deployment. Are there any tools
which can do this?
Yes. They're called Retrotranslator or Retroweaver. Apart from Generics (which only exist for the compiler's sake anyway), you cannot simply "strip 1.5 stuff". Enums (and maybe also some other features) have to be replaced with functionally equivalent code. Which is exactly what those tools do.
You can code with JDK 1.5 features and target JDK 1.4 at compile time. See available Javac options. However, most libraries are now using JDK 1.5 code, so you'll be stuck with old libs.
It worth noting that while Java 1.4 has been EOL for some time. Java 5.0 will be EOL Oct 8th, 2009. If anyone is promising you Java 5.0 by 2010, I would ask why?!
To simulate annotations in java 1.4 you can use http://xdoclet.sourceforge.net/xdoclet/index.html