Essentially I'm asking something similar to this: How do I get a list of Java class dependencies for a main class? but I'd like to do it using a Java API. I'd prefer not to exec jdeps and scrape the output.
Extra points if it uses https://github.com/classgraph/classgraph since our project already uses that library and it is awesome for scanning classes without actually instantiating everything it processes.
Oh, and this would be restricted to working on JDK 8.
[Update]
Here is a bit more background. Let's say I have a bundled application. Perhaps a Spring Boot app or something that can be deployed to a cloud platform like Cloud Foundry or Heroku. The app probably contains framework jars (like Spring), utility jars (logging, templating, json/xml processing, etc.), application domain classes and then all necessary transitive dependencies. Assume this is all either bundled into an uber-jar or referenced via a classpath.
My use case would be: Given a class, and the uber-jar/classpath, what is the subset of jars (out of that uber-jar) that my given class requires as dependencies to instantiate it?
Turns out that I can do what I want with classgraph. The key is to set enableInterClassDependencies. Given a class, the following will process all dependencies, of that class, and determine which jars contain the relevant dependent classes:
import java.net.URI;
import java.util.HashSet;
import java.util.Map;
import java.util.Set;
import io.github.classgraph.ClassGraph;
import io.github.classgraph.ClassInfo;
import io.github.classgraph.ClassInfoList;
import io.github.classgraph.ScanResult;
public class DependencyFinder {
private final String clazz;
public DependencyFinder(String clazz) {
this.clazz = clazz;
}
public Set<URI> process() {
ScanResult scanResult = new ClassGraph()
.whitelistPackages()
.enableInterClassDependencies()
.scan();
ClassInfo rootClass = scanResult.getClassInfo(clazz);
Map<ClassInfo, ClassInfoList> dependencyMap = scanResult.getClassDependencyMap();
Set<URI> results = new HashSet<>();
Set<ClassInfo> seen = new HashSet<>();
accumulateJars(new HashSet<>(dependencyMap.get(rootClass)), dependencyMap, results, seen);
return results;
}
private void accumulateJars(Set<ClassInfo> roots, Map<ClassInfo, ClassInfoList> dependencies, Set<URI> accumulated, Set<ClassInfo> seen) {
Set<ClassInfo> nextRoots = new HashSet<>();
for (ClassInfo info : roots) {
if (seen.contains(info)) {
continue;
}
accumulated.add(info.getClasspathElementURI());
seen.add(info);
nextRoots.addAll(dependencies.get(info));
}
if (nextRoots.size() > 0) {
accumulateJars(nextRoots, dependencies, accumulated, seen);
}
}
}
Related
How you do enable Objectify to automatically discover your annotated Entity classes at runtime?
I want to register my classes at application start up and not have to hard code them into a class as recommended in the documentation.
Solution
This is no runtime reflection cost solution because it does all the reflection at compile/build/package time.
import com.google.appengine.api.ThreadManager;
import com.googlecode.objectify.ObjectifyFactory;
import com.googlecode.objectify.ObjectifyService;
import com.googlecode.objectify.annotation.Entity;
import org.reflections.Reflections;
import org.reflections.util.ClasspathHelper;
import org.reflections.util.ConfigurationBuilder;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import javax.annotation.Nonnull;
import javax.servlet.ServletContextEvent;
import javax.servlet.ServletContextListener;
import java.util.HashSet;
import java.util.Set;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
/**
* This class processes the classpath for classes with the #Entity or #Subclass annotations from Objectify
* and registers them with the ObjectifyFactory, it is multi-threaded and works very fast!
*/
public class ObjectifyLoaderContextListener implements ServletContextListener
{
private static final Logger L = LoggerFactory.getLogger(ObjectifyLoaderContextListener.class);
private final Set<Class<?>> entities;
public ObjectifyLoaderContextListener()
{
this.entities = new HashSet<>();
}
#Override
public void contextInitialized(#Nonnull final ServletContextEvent sce)
{
final ExecutorService es = Executors.newCachedThreadPool(ThreadManager.currentRequestThreadFactory());
cb.setExecutorService(es);
final Reflections r = Reflections.collect();
this.entities.addAll(r.getTypesAnnotatedWith(Entity.class));
es.shutdown();
final ObjectifyFactory of = ObjectifyService.factory();
for (final Class<?> cls : this.entities)
{
of.register(cls);
L.debug("Registered {} with Objectify", cls.getName());
}
}
#Override
public void contextDestroyed(#Nonnull final ServletContextEvent sce)
{
/* this is intentionally empty */
}
}
Updates to this class will be available in this gist.
Here's an alternative solution building on the accepted answer.
Summary
If you want to eliminate all production runtime cost from your service (even the relatively small one of picking up the XML artifacts which are then consumed by the #collect API), you can move the cost to the unit testing stage while still retaining 100% safety in prod of all entities being registered. See end of answer for code.
Motivation & Details
This has several benefits as outlined below, though whether they are worth it is up to your use case. To me personally, it isn't whether any of these are ever going to be a problem -- it's more that given the option to avoid doing anything unnecessary in production (and deleting code from production paths), I would always take it, as long as it doesn't entail extra complexity, weird design costs, etc. In this case there are none. This is a text-book approach of moving runtime registration guarantees from prod to testing.
Note that GAE instances may get restarted arbitrarily many times. Other considerations include how sensitive your app is to service startup times, how large the generated Reflection XML artifacts are (a function of your jar dependencies and impl details of the Reflection 3rd party library), and how efficiently the library parses them. While you can mitigate performance risk with multi-threading, google still charges you for CPU time used.
The relative benefits of converting this to a unit test solution are:
you eliminate the cost of parsing XML resources and reconstructing them into Reflection metadata each time your service is restarted (may happen arbitrarily many times)
you eliminate the risk of changing your project JAR dependencies at some future times (dependencies tend to grown over time for non-trivial projects) and forgetting to re-assess new size of XML resources and their parsing; this may start adding more sizeable overhead to your service restart time gradually, which is hard to pinpoint but affects users (contributing to the classic death-by-a-thousand-cuts)
you eliminate a runtime prod dependency on a 3rd library which is always good (the dependency will remain for your unit testing modules). For instance, if a future version of the library suddenly regresses on XML parsing speeds, you will need to deal with that in prod. It's a risk that can be eliminated.
Whether these considerations remain purely academic depends on your use case and risk/cost tolerance.
Caveat: if for some reason your release rollout process does not mandate passing unit tests, this clearly isn't the right approach. But I don't think that would be the case for many people.
Code
I've omitted the outer test class for readability (assume all code lives in a Test class)
#Test
public void testObjectifyRegistration() {
final ObjectifyFactory ofy = Ofy.factory(); // your backend-specific ofy entry point where you've done your registrations
for (final Class<?> cls : allEntities()) {
verifyRegistrationState(ofy, cls, /*expected to be registered?*/!cls.equals(UnregisteredUnittestEntity.class));
}
}
private Iterable<Class<?>> allEntities() {
final ExecutorService es = Executors.newCachedThreadPool();
try {
final ConfigurationBuilder cb = new ConfigurationBuilder();
cb.setUrls(ClasspathHelper.forPackage(""));
cb.setExecutorService(es);
return new HashSet<>(new Reflections(cb).getTypesAnnotatedWith(Entity.class));
} finally {
es.shutdown();
}
}
private void verifyRegistrationState(final ObjectifyFactory ofy, final Class<?> cls, final boolean shouldBeRegistered) {
RuntimeException missingRegistrationExc = null;
try {
ofy.getMetadata(cls); // per today's API contract, this method throws if cls has not been registered with objectify
} catch (RuntimeException exc) {
missingRegistrationExc = exc;
}
if (shouldBeRegistered && missingRegistrationExc != null)
throw missingRegistrationExc;
else if (!shouldBeRegistered && missingRegistrationExc == null)
fail("ObjectifyFactory#getMetadata expected to throw upon unregistered entity, check for new contract of method and update test");
}
#Entity // self-test: used to validate correctness of the unit test itself
private static class UnregisteredUnittestEntity {
#VisibleForTesting
public UnregisteredUnittestEntity() {
}
}
And below is an example failure when a class annotated with #Entity has not been registered with Objectify.
java.lang.IllegalArgumentException: No class '[xxx].UserImpl' was
registered at
com.googlecode.objectify.impl.Registrar.getMetadataSafe(Registrar.java:120)
at
com.googlecode.objectify.ObjectifyFactory.getMetadata(ObjectifyFactory.java:210)
at
[xxx].OfyRegistrationTest.verifyRegistrationState(OfyRegistrationTest.java:56)
at
[xxx].OfyRegistrationTest.testOfyRegistration(OfyRegistrationTest.java:37)
I'm creating a Java library for using in other Java projects. The projects use Repast Symphony and my library does so too (so i'm afraid this error is being caused by some conflict). Everything builds fine, but when I run a the Repast simulation, it throws java.lang.NoClassDefFoundError: repast/simphony/context/Context
I tried exporting my library as a jar, importing the project directly and adding the library to my project's classpath, to no avail. What can I be doing wrong?
This Context class is being used in both my library and my projects. The following is a snippet of it use in two classes:
// MyContextBulder.java
// This file is in my project
// This class is called by Repast first
import repast.simphony.context.Context;
import repast.simphony.dataLoader.ContextBuilder;
import mylibrary.core.DF;
import mylibrary.core.DF.MyContext;
public class MyContextBuilder implements ContextBuilder<Object> {
#Override
public Context<Object> build(Context<Object> context) {
context.setId("test");
DF.setContext((MyContext) context);
// Create agent
new MyAgent();
// Add the agent to the Repast context.
// context.add(t);
return context;
}
}
// DF.java
// This file is in my library
import java.util.Collection;
import java.util.HashMap;
import java.util.Iterator;
import org.apache.commons.collections15.Predicate;
import repast.simphony.context.Context;
import repast.simphony.context.ContextListener;
import repast.simphony.space.projection.Projection;
import repast.simphony.util.collections.IndexedIterable;
import repast.simphony.valueLayer.ValueLayer;
import mylibrary.Agent;
/**
* This static class provides the Directory Facilitator Service
* and is used to send messages to agents
* and to keep a directory of all agents in the application.
* Agents use the static method send(ACLMessage) to send a message
* to one or more agents. The ACLMessage object contains
* the receiver agent and the sender (so the receiver can reply back).
*
* This class needs to be setup initially before registering new agents.
* To do that, simply call setContext(...);
* #author joaolopes
*
*/
public class DF {
private static int lastAID = 0; // Just to help generate new identifiers
private static HashMap<Integer, Agent> agents; // Contains all agents
/**
* The Repast context that contains all
* scheduled Repast objects.
*/
private static MyContext context = null;
/**
* Registers the agent in the directory and returns its
* AID (freshly generated for it). If the agent is already
* registered, returns its current AID.
* #param agent The agent to be registered
* #return The AID generated for the agent.
*/
public static int registerAgent(Agent agent) {
// If this agent is already in the hashMap,
// just return its key.
if (getAgents().containsValue(agent)) {
return agent.getAID();
}
// Quick way to find a new ID for this agent
// that is not in use at the moment.
while (getAgents().containsKey(lastAID)) {
lastAID++;
}
// The agent must know their own ID.
agent.setAID(lastAID);
agents.put(lastAID, agent);
System.err.println(context.toString());
context.add(agent);
return lastAID;
}
public static void setContext(MyContext c){
context = c;
}
}
Editing to add relevant info from the comments:
I don't import the repast JAR directly in my projects as I do in my library. Repast Symphony is installed in Eclipse as a plugin, so I created "Repast Projects" that include all Repast libraries. Therefore, I'm unable to remove the specific JAR that is causing the possible conflict of classes.
Exactly as you said. This error should be the conflict between the same classes in a jar. If you are using an IDE try to clean the build and rebuild again.
And also I would suggest you to use only one symphony library jar. Multiple class definitions always leads to ambiguity for the JVM class loader.
Try not to use symphony jar in the importing project since you already have it in your exported jar. After importing your lib, there should no errors.
Try this and let me know how it goes.
I suggest that you use an build tool. Something like Maven. Then maven with the right plugin will fix this problem for you. All you need to do, is to tell Maven that you need a particular jar file. Then a magic will occur, and you will have a well working jar-file to distribute
java.lang.NoClassDefFoundError is thrown when the JVM tries to run the application. Typical cases is when you got one jar-file as "interface". Then you got other jar-file that implement that interface.
So what you need to do, is that have the Repast jar inside your jars classpath. So that your program can find the right class you want to use.
I cannot find an answer in the Google docs. As I understand from the docs, the #Api annotation is applied to a class to indicate that it is part of the Endpoint API and the #ApiMethod then indicates which methods of that class is part of the Cloud API.
However, even if a method is not annotated with #ApiMethod the Google App Engine Cloud Endpoints Builder still includes that method as part of the Cloud API.
How can I exclude a method from the API? If it is not possible, is a good alternative then to pass the received API parameters to a separate object (which is a field of the annotated class) that do contain the required method?
I include the following code which is a class annotated with #Api for clarification (see comment on top of ggetStr method):
package com.barcodeapp.www.app;
import java.util.ArrayList;
import java.util.List;
import com.google.api.server.spi.config.Api;
#Api(
name = "cetest",
version = "v1",
scopes = {EndpointsConstants.EMAIL_SCOPE},
clientIds = {EndpointsConstants.WEB_CLIENT_ID, EndpointsConstants.ANDROID_CLIENT_ID, com.google.api.server.spi.Constant.API_EXPLORER_CLIENT_ID},
audiences = {EndpointsConstants.ANDROID_AUDIENCE}
)
public class CloudTest {
public List<String> list() {
List<String> strs = new ArrayList<String>();
strs.add("a"); strs.add("b");
return strs;
}
/* THE FOLLOWING METHOD NEEDS TO BE EXCLUDED FROM CLOUD API */
public String ggetStr() {
return "abc";
}
}
EDIT: I have included the library .jar file in the WAR/WEB-INF/lib folder - no difference.
Below is the two classes for completeness:
package com.cloudtest.lib;
public class ClassInLibProj {
}
and
package com.cloudtest.my;
public class ClassInLocalProj {
}
Thanks.
I may be wrong, but it seems that the only way currently for your method not to be included is to make that private in your class.
This is because:
If your method is annotated with APIMethod, then it is included
If you method is not annotated with APIMethod but it is public, it is taken too. This is likely because of the API annotation at the top.
There is an issue raised : https://code.google.com/p/googleappengine/issues/detail?id=10372&thanks=10372&ts=1386300958
I'm looking a utility method so that given a class will return the full classpath required to run this class externally. This means the jar the class is in as well as all jars (or folders) of classes that it uses.
UPDATE: there are tools that analyze .class files to find dependencies. This is not what I'm looking for. I'm looking for something that uses Java's reflection API on an already loaded class. I'll settle for something that analyzes byte code, if it goes recursively into classes it finds through the class loader
Reflection will not help you a lot for this one. You will need to analyse the byte code to find dependencies.
UPDATE:
Alright then. I am using a library that I made years ago, that you can download here.
The following code:
package classdep;
import java.io.InputStream;
import java.net.URL;
import java.util.ArrayList;
import java.util.List;
import org.jedo.classfile.ClassFile;
import org.jedo.classfile.ConstantPool;
public class Main {
public static void main(String[] args) {
try {
ClassLoader cl = Thread.currentThread().getContextClassLoader();
List<String> classes = new ArrayList<String>();
classes.add(args[0].replace('.', '/'));
for (int i = 0; i < classes.size(); ++i) {
String className = classes.get(i);
URL url = cl.getResource(className + ".class");
if (url == null) {
System.out.println("--- class not found " + className);
} else {
System.out.println(url);
ClassFile classFile = new ClassFile();
InputStream in = url.openStream();
try {
classFile.load(in);
} finally {
in.close();
}
ConstantPool cp = classFile.getConstantPool();
for (String name: cp.getClassNames()) {
if (!classes.contains(name)) {
classes.add(name);
}
}
}
}
} catch (Exception e) {
e.printStackTrace();
}
}
}
Will give you all the dependencies of a class. When applied to org.jedo.classfile.ClassFile, it produces the following output:
file:/D:/projects/casagrande/jedo/build/classes/org/jedo/classfile/ClassFile.class
file:/D:/projects/casagrande/jedo/build/classes/org/jedo/classfile/ConstantPool.class
file:/D:/projects/casagrande/jedo/build/classes/org/jedo/classfile/FieldInfo.class
file:/D:/projects/casagrande/jedo/build/classes/org/jedo/classfile/MethodInfo.class
file:/D:/projects/casagrande/jedo/build/classes/org/jedo/classfile/AttributeInfo.class
jar:file:/C:/Program%20Files/Java/jdk1.6.0_18/jre/lib/rt.jar!/java/io/File.class
jar:file:/C:/Program%20Files/Java/jdk1.6.0_18/jre/lib/rt.jar!/java/io/FileInputStream.class
jar:file:/C:/Program%20Files/Java/jdk1.6.0_18/jre/lib/rt.jar!/java/io/DataInputStream.class
jar:file:/C:/Program%20Files/Java/jdk1.6.0_18/jre/lib/rt.jar!/java/io/StreamCorruptedException.class
jar:file:/C:/Program%20Files/Java/jdk1.6.0_18/jre/lib/rt.jar!/java/io/FileOutputStream.class
jar:file:/C:/Program%20Files/Java/jdk1.6.0_18/jre/lib/rt.jar!/java/io/DataOutputStream.class
jar:file:/C:/Program%20Files/Java/jdk1.6.0_18/jre/lib/rt.jar!/java/lang/StringBuilder.class
jar:file:/C:/Program%20Files/Java/jdk1.6.0_18/jre/lib/rt.jar!/java/lang/StringBuffer.class
jar:file:/C:/Program%20Files/Java/jdk1.6.0_18/jre/lib/rt.jar!/java/lang/Object.class
jar:file:/C:/Program%20Files/Java/jdk1.6.0_18/jre/lib/rt.jar!/java/io/IOException.class
...
Followed by a lot of system classes. You need to filter out system classes, and parse the other urls to extract either the .jar file if it is a jar: url, or the directory if it is a file: url.
I don't think this is possible. Certainly, the reflection APIs don't support it.
You can find out a classes classloader, but you cannot find out:
which of the classloader's possible JAR files and directories contained the class,
what the static dependencies of the class are, or
what the dynamic dependencies of the class are; e.g. what it or its dependants loaded using Class.forName().
Actually, this overstates things somewhat:
You can in theory figure out which classes came from which JARs if you can find out what the classpath is. There are possibly ways to dig this out of a classloader.
You can in theory figure out what a classes dependants are, but you been to dig around in the class'es bytecode file using (for instance) BCEL to find this out.
You can in theory figure out what was dynamically loaded if you are prepared to write your own classloader. It may be possible to link this back to the class that initiated the loading using some hairy analysis of the stack frames.
But this is all extremely complicated, and I'd expect it to be unreliable under certain circumstances.
There are cases when you cannot determine this prior to using your class.
This cannot always be known. For instance, a class can be dynamically created at run time and then loaded with a custom ClassLoader.
I do not believe Java stores this information.
I have a java project I want to create which will be built on top of some vendor APIs. The APIs connect to servers running said vendor's software and perform various actions.
I have 2 different versions of the servers supported by 2 different API versions. Any changes to the API's are in internal implementation only. I.E. The classes, interfaces, methods, etc. available to me in the older version exist in the newer version. Therefore the code I write should compile and run with either API version. There is a version number in the API presented to the servers when using the API to connect that prevents you from using a different version API on that particular server.
Is there a way to switch JAR files on the fly at runtime? (something like a c/c++ DLL??)
If switching API versions at runtime isn't possible, what is the most elegant way to handle the problem. Build the code 2x (one for each api version)?
I hope I'm missing something but approach 2 doesn't seem ideal. Here's a more concrete example of why:
package org.myhypotheticalwrapper.analyzer;
import org.myhypothetical.worker;
import org.myhypothetical.comparator;
public class Analyzer {
Worker w1 = new Worker();
Worker w2 = new Worker();
Comparator c = new Comparator(w1.connectAndDoStuff(),w2.connectAndDoStuff());
c.generateReport();
}
This is my dilema. I want w1 to be built with the old API and w2 be built with the new API so they can connect to the appropriate servers. Other than the API's they sit on top of, they are the same (identical code). Do I have to create two uniquely named Class types for W1 and W2 even though their code is identical, simply to accommodate different API versions? It seems like that could get unwieldy fast, if I had many API versions that I wanted to interact with.
Any suggestions and comments greatly appreciated.
-new guy
The easiest is probably having a classloader loading in classes not in the default classpath.
From http://www.exampledepot.com/egs/java.lang/LoadClass.html
// Convert File to a URL
URL url = file.toURL(); // file:/c:/myclasses/
URL[] urls = new URL[]{url};
// Create a new class loader with the directory
ClassLoader cl = new URLClassLoader(urls);
// Load in the class; MyClass.class should be located in
// the directory file:/c:/myclasses/com/mycompany
Class cls = cl.loadClass("com.mycompany.MyClass");
You can't really change out a class file once it's been loaded, so there's really no way to replace a class at runtime. Note that projects like JavaRebel get around this with some clever use of instrumentation via the javaagent - but even what you can do with that is limited.
From the sounds of it you just need to have two parallel implementations in your environment at the same time, and don't need to reload classes at runtime. This can be accomplished pretty easily. Assume your runtime consists of the following files:
analyzer.jar - this contains the analyzer / test code from above
api.jar - this is the common forward-facing api code, e.g. interfaces
api-impl-v1.jar - this is the older version of the implementation
api-impl-v2.jar - this is the newer version of the implementation
Assume your worker interface code looks like this:
package com.example.api;
public interface Worker {
public Object connectAndDoStuff();
}
And that your implementations (both in v1 and v2) look like this:
package com.example.impl;
import com.example.api.Worker;
public class WorkerImpl implements Worker {
public Object connectAndDoStuff() {
// do stuff - this can be different in v1 and v2
}
}
Then you can write the analyzer like this:
package com.example.analyzer;
import com.example.api.Worker;
public class Analyzer {
// should narrow down exceptions as needed
public void analyze() throws Exception {
// change these paths as need be
File apiImplV1Jar = new File("api-impl-v1.jar");
File apiImplV2Jar = new File("api-impl-v2.jar");
ClassLoader apiImplV1Loader =
new URLClassLoader(new URL[] { apiImplV1Jar.toURL() });
ClassLoader apiImplV2Loader =
new URLClassLoader(new URL[] { apiImplV2Jar.toURL() });
Worker workerV1 =
(Worker) apiImplV1Loader.loadClass("com.example.impl.WorkerImpl")
.newInstance();
Worker workerV2 =
(Worker) apiImplV2Loader.loadClass("com.example.impl.WorkerImpl").
.newInstance();
Comparator c = new Comparator(workerV1.connectAndDoStuff(),
workerV2.connectAndDoStuff());
c.generateReport();
}
}
To run the analyzer you would then include analyzer.jar and api.jar in the classpath, but leave out the api-impl-v1.jar and api-impl-v2.jar.
You need to make sure that the classes are in different packages. You can't import two jar files with the same package listing and expect one to be recognized over the other. If they are in different packages you can use:
com.foo.Worker w1;
com.bar.Worker w2;
Your worker needs to have a delegate that implements an interface. You will have to write two delegates, one for the old api, one for the new. Choose which delegate to instantiate at runtime. Something like:
public class Worker {
private WorkerDelegate delegate;
public void foo() { delegate.foo(); }
public Object bar(){ return delegate.bar(); }
public Enum API{v1,v2};
public Worker(API api) {
try { switch(api){
case v1: delegate = Class.forName("my.v1.impl").newInstance(); break
case v2: delegate = Class.forName("my.v2.impl").newInstance(); break
}
}catch(...){throw new Error(e);}
}
}
More implementations can be added later with ease.