When the lombok jar file is opened in Intellij, all files other than the annotations end with .SCL.lombok (e.g. HandleAccessors.SCL.lombok). I was just wondering what the reason for this was and how it's handled.
The reason for it
Lombok has a public API - the stuff you're supposed to interact with. That'd be, for example, the #lombok.Getter annotation. Those are just class files in that jar, the aim is simply: add that jar to your classpath and your IDE autocomplete dialogs and the like will automatically start suggesting these, as per design.
But, lombok also has lots of classes that just 'make it tick', these aren't meant for public consumption. Things like lombok.eclipse.HandleGetter, which is the implementation for handling the #Getter annotation inside the eclipse agent. There is no point or purpose to referring to this class anywhere, in any project - it's an internal lombok thing. If we just stuck that jar file into the jar, and you typed Handle and hit your IDE's autocomplete shortcut key, you'd still get the suggestion.
Similarly, we ship a few dependencies straight into lombok.jar - it's a 'shaded jar' (a jar with all deps included), though we don't have many, keeping lombok.jar a nice small size. Still, ASM (a bytecode manipulation library) is in it, and that is fairly popular.
The standard shading solution offered by most shading tools is to prefix something to the name. ASM's org.objectweb.asm.AnnotationVisitor class would become org.projectlombok.shading.org.objectweb.asm.AnnotationVisitor. Point is, your IDE doesn't know that, and if you ALSO use asm in your project (where you also use lombok), and you want AnnotationVisitor thus you type AnnV and hit cmd+space or whatnot, your IDE suggests both. That's ugly and we'd like to avoid this.
Hence, we built our own shader, and it works by not having class files in the first place. This way, IDEs and any other automated tool doesn't even know either our ASM classes, or our implementation details, even exists. The only files that such tools (such as your IDE) sees are the types you're meant to see: lombok.Builder, lombok.extern.slf4j.Slf4j, lombok.experimental.UtilityClass, etcetera.
How does it work
Java's classloader architecture is abstracted: You can make your own. The primitives offered by a class loader is simply this: "Convert this byte array containing bytecode (i.e. the contents of a class file) into a Class<?> definition", and the primitives that you're supposed to implement when you write your own classloader is twofold:
Here is a resource key, such as "/com/foo/load.png". Please provide me an InputStream with this data.
Here is a fully qualified class name, such as "com.foo.MyApp". Please provide me with a Class<?> instance representing it.
Out of the box, java ships with a default classloader. This default classloader answers these questions by checking your CLASSPATH - which can be provided in various ways (via the jar manifest's Class-Path entry, or via the -cp argument to the JVM executable, or the CLASSPATH environment variable), and scanning each entry on the classpath for the resource requested, capable of reading the file system as well as opening jar files.
But that's just a classloader. One implementation of the general principle that's baked into java. You can write your own. You can write a classloader that generates resources on the fly, or that loads them from a network.
Or, as lombok does, that loads them by opening its own jar and looking for .SCL.lombok files.
Thus, lombok works like this: When you launch it, the 'entrypoint' (the class containing public static void main - or in lombok's case, for javac mode it's the annotation processor entrypoint and for eclipse it's agentmain), we 'hide' it from you using some fancy trickery: agentmain does not need to be in a public class (it can't be .SCL.lombok files - our classloader isn't available yet, we need to bootstrap that up first!). annotation processors do have to be in a public class, but, it's a public class inside a package private class, thus, just about every IDE knows it's 'invisible' and won't show it, but javac's annotation runner accepts it.
From there, we register a classloader that is capable of loading classes by way of reading in an .SCL.lombok file, and this lets us hide everything else we want to hide.
I want to develop lombok and this is getting in the way!
No need; just clone our repo, run ant eclipse or ant intellij, and off you go. There is no way to extend lombok without first forking it; we'd like lombok to be able to be extensible without it, but that would be far more complicated than simply not doing the .SCL.lombok thing. Eclipse runs on top of equinox, a runtime modularization system, and making that work properly requires all sorts of stuff that would make 'just toss some extra handlers on the classpath' not a feasible route to extending lombok in the first place.
Related
While setting up a project as a template for Slick2d based projects following the instructions here:Slick2d wiki using the provided code for testing setup here at run-time I keep getting a giant block of sealing errors. My thought is that this problem stems from the version of ljgwl.jar in both libraries, however Slick requires both in order to function properly. How can I resolve this?
Package sealing is a Java feature implemented in part in the JAR file format. It is discussed in several places, including in Oracle's Java Tutorial, but the bottom line is that when package sealing is enabled in a Jar's manifest, no classes belonging to that package can appear in any other JAR file.
My thought is that this problem stems from the version of ljgwl.jar in both libraries, however Slick requires both in order to function properly.
I'm not sure what you mean by "library", as that is not a Java concept. I suspect, however, that you're trying to say that you somehow have ljgwl.jar files from two separate sources, and you've put both into your project classpath. That would indeed be a problem, and more than just for package sealing. You can, in fact, be thankful for the sealing errors, for they may have saved you from subtler, more difficult to diagnose runtime errors.
How can I resolve this?
You should have only one copy of LWJGL in your classpath, regardless of any requirements enforced by package sealing, and regardless of how the classes are packaged in jar files. It looks like the Slick2D distribution may come with a copy of LWJGL -- in that case, it's probably wisest to use that one. As long as it's in your classpath (as it must be anyway for Slick2D to use it), any class anywhere in your application can use it.
It gets tricky if you need to contend with a inconsistent requirements for LWJGL version, or if you have obtained a JAR that incorporates the LWJGL classes along with something else, but that doesn't change the bottom line: you must choose one version of LWJGL, use that version exclusively within your application, and include only one copy of it in your classpath.
I'm digging into the source code of the deeplearning for java recently. There is such a class NeuralNetConfiguration in which there are tons of fields that all requires getters and setters. The NeuralNetConfiguration.java source code does not provide any, however.
When I open this project in IntelliJ, ctrl click on the usage of this class, which are methods mostly like, NeuralNetConfiguration.getNInput() or NeuralNetConfiguration.getKernelSize(), the IDE direct me to the compiled class file in which all the getters are defined for each of the field in this class.
Just wonder how this is done since I'm a new bee to java. Posts I found about java reflect suggest that reflect can not add method to a method to a class unless you wrote your own classloader. I check the deep learning for java project and I don't think they have done that.
What bothers me too from time to time is, IntelliJ starts to report errors that those getFields methods could not be resolved since they are not in the source file at all, especially after my building the project using IntelliJ instead of using mvn command line.
The magic happens with the #Data annotation on the class. This annotation is from Project Lombok. There is probably an annotation processor somewhere that hooks into the compiling process and generates these methods.
Can I use ClassLoader's definePackage to override some packages from inside a jar?
For example, the application currently contains "javax.xml.bind" from abc.jar. If I call ClassLoader.definePackage(def.jar), in which the def.jar contains another version of javax.xml.bind, can I replace the classpath for the entire application to point to that of def.jar? Thanks.
No, you definitely can not use ClassLoader.definePackage to "override" some packages from inside a jar.
If I understand correctly, you want to make your JVM load any class under javax.xml.bind from def.jar while all other ones from abc.jar. In this case you can (in my personal order of preference):
1) Put def.jar before abc.jar in the CLASSPATH. This requires that no class you want loaded from abc.jar is present in def.jar.
2) Unzip def.jar, abc.jar, or both, and remove any conflicting classes so it is really irrelevant which jar comes first in the CLASSPATH. Then re-zip them. Or you can do this only on one jar and put it before the other.
3) Use a configurable classloader (sorry, no public domain one that I know of; let me know if you find one). This could be an interesting topic for an OS project, except that several initiatives with similar (but much broader) objectives are already ongoing, some at the core of the language.
4) Create a classloader for this purpose, probably extending the default one.
MATLAB is configured to search its static java class path before searching the user-modifiable dynamic path. Unfortunately, the static path contains quite a number of very old public libraries, so if you are trying to use a new version you may end up loading the wrong implementation and get errors.
For instance, the static path contains an old copy of the google-collections.jar, which has long been supplanted by Google's guava library and which has some of the same class names (e.g. com.google.common.base.Objects). As a result, if you invoke a Guava method that uses a newer method of one of such a class, you will end up getting surprising NoSuchMethodErrors because the google-collections jar is found first.
As of R2012b, MATLAB lets you specify additional jars to add to the static path by putting a javaclasspath.txt file in your preferences folder, but that adds jars to the end of the path, and doesn't let you override jars that are built into MATLAB.
So what is the best way around this?
I got an official response from Mathworks:
As of MATLAB R2013a (also in R2012b), classes can be added to the front of the static Java class path by including the following line in javaclasspath.txt:
<before>
Any directory that is after this line in javaclasspath.txt will be added to the front of the static Java class path. This is an undocumented use of javaclasspath.txt as of R2013a.
But overall in MATLAB, the ability to add classes to the front of the static Java classpath is not available through javaclasspath.txt in MATLAB 8.0 (R2012b).
MATLAB searches for classpath.txt in the following order:
In the startup directory. As of MATLAB 8.0 (R2012b) a warning will be shown if the file is found there and it will be ignored.
In the first directory on the MATLABPATH environment variable. (This environment variable is used in the bin/matlab shell script on Linux and in general is not used by the end-user).
In the toolbox/local directory.
Although the MATLABPATH environment variable of point 2 is normally not used by end-users we can use it in a workaround to allow reading a custom classpath.txt outside of the toolbox/local directory.
On Windows:
You will need to create the MATLABPATH environment variable. The first directory on it should be your directory with the custom classpath.txt AND you will also need to add the toolbox\local directory as second option. So from a cmd prompt you could do:
set MATLABPATH=c:\Users\user\Documents\myMATLABClasspath;c:\Program Files\MATLAB\R2012b
\toolbox\local
matlab.exe
One hack that appears to work is to add the jar to the top of the classpath.txt file that can be found in your MATLAB installations toolbox/local folder. Unfortunately, this is automatically generated and may get rewritten at some unspecified time, such as when you install new toolboxes, so this approach would require you to have some way to notice when this happens and reapply the hack.
If you're distributing a jar that's intended to be used with matlab, it may be better to use proguard as described at http://code.google.com/p/guava-libraries/wiki/UsingProGuardWithGuava.
If you specify that all of your classes and their (public) fields and methods are to be preserved and include guava as a program jar (not a library), then it will rename all of guava's methods and update your compiled bytecode to reference the new names.
It seems a bit hackish, but depending on the audience, it may be significantly easier than teaching your users about static vs. dynamic classpath, and it won't break any matlab code that depends on the old behavior.
Instead of obfuscating the package as suggested by #user2443532, I have found it easier to "shade" the conflicting package instead of obfuscating it - unless you actually need obfuscation. One easy way to do this is to build your package using Maven and use the maven-shade-plugin. Internal calls are modified automatically, so you don't need to modify any of the Java code.
Direct calls from Matlab will need to be modified - for example, calls to com.opensource.Class become shaded.com.opensource.Class.
For more info on shading, see What is the maven-shade-plugin used for, and why would you want to relocate Java packages?
I am using Eclipse and I have got two libraries included in my project, foo.jar and bar.jar. In both JARs there is a class FooBar.java that includes the method getFoobar(Object xy).
Now I would like to load the method getFoobar(Object xy) either from foo.jar or from bar.jar on the basis of a properties file:
config.properties:
choice=foo
If choice==foo then the method of Foo.jar shall be picked, elsewise the method getFoobar(Object xy) from Bar.java. To make things more complicated the method getFoobar(Object xy) has in its method declaration objects loaded from another JAR which is included in foo.jar (and bar.jar respectively).
So my question is. How can I get the methods of the JARs respectively by Reflections? I have not yet found a solution. Is there a way to solve this issue?
Thank you.
Emrah
You can only load one or the other methods without getting complicated with class loaders. A simple workaround is to rename the package of one or both libraries with jarjar This allows you to have everything in foo.jar start with foo. and everything is bar.jar start with bar. and this avoids any confusion. I have seen this used to be able to load several version of Xerces based on configuration (and use them concurrently in the same app)
Write your ClassLoader that reads the property. Alternatively, you can use -D option in the jvm arguments
-Dchoice=foo
and in the code that loads the class from foo.jar or from bar.jar you can switch on this property value:
String choice = System.getProperty("choice");
if (Main.CHOICE_FOO.equalsIgnoreCase(choice)) {...}
...
OSGI can give you a similar capability.
How can I get the methods of the JARs respectively by Reflections?
You can't.
Reflection does not allow you to select between the two versions of the class. In fact, as far as the JVM is concerned, there is only one version; i.e. the one that appears earliest on the classpath.
You can solve this problem by defining multiple classloaders, each with a different classpath. However, there are complications:
If you manage to load both versions of the class into a running application, you will find that instances of the respective classes have different types, and you won't be able to convince the JVM otherwise.
When your application loads another class that statically depends on one of these classes, it will bind to the version of the class that is on the classpath of the dependent classes classloader. And you can't change that. So, uses of the class name in declarations / typecasts / etc in the dependent class will refer to the version of the class found by the dependent classes classloader, not the other one.
The upshot is that you can't use these same-named classes like regular classes ... especially if both versions need to be loaded in the same JVM. It is a better idea to give the two versions of the class different names. You don't gain anything by making the class names the same.