We Keep Coding

How to use URLClassloader in an auto-update jar launcher?

I've come across many posts about these two topics: Auto-Updating and URLClassloaders. I'll start with the auto updating goal. I found this post here that talks about a 2 jar system. One jar that launches the main app jar: From Stephen C:
The launcher could be a Java application that creates a classloader for the new JAR, loads an entrypoint class and calls some method on it. If you do it this way, you have to watch for classloader storage leaks, but that's not difficult. (You just need to make sure that no objects with classes loaded from the JAR are reachable after you relaunch.)
This is the approach I'm taking, but I'm open to other ideas if they prove easier and/or more reliable. The Coordinator has posted some pretty cool launcher code to which I plan on incorporating some of this reload type code in my launcher, but first I need to get it to work.
My issue is that my main app jar has many other dependencies, and I cannot get some of those classes to load despite the fact that all the jars have been added to the URL's array. This brings up the second topic URLClassloader.
Side Note for future readers: When passing a URL to the URLClassloader that is a directory, a helpful note that would have saved me (an embarrassingly large) amount of time is that the contents of the directory must be .class files! I was originally pointing to my dependent jar directory, no good.
Context for the code below, my launcher jar resides in the same directory as my app jar, which is why I'm using user.dir. I will probably change this, but for now the code works and gets far enough into my app's code to request a connection to a sqlite database before failing.
Launcher:
public class Launcher {
public static void main(String[] args) {
try {
String userdir = System.getProperty("user.dir");
File parentDir = new File(userdir);
ArrayList<URL> urls = getJarURLs(parentDir);
URL[] jarURLs = new URL[urls.size()];
int index = 0;
for (URL u : urls) {
System.out.println(u.toString());
jarURLs[index] = u;
index ++;
}
URLClassLoader urlCL = new URLClassLoader(jarURLs);
Class<?> c = urlCL.loadClass("main.AppStart");
Object [] args2 = new Object[] {new String[] {}};
c.getMethod("main", String[].class).invoke(null, args2);
urlCL.close();
} catch (Exception e1) {
e1.printStackTrace();
}
}
public static ArrayList<URL> getJarURLs(File parentDir) throws MalformedURLException {
ArrayList<URL> list = new ArrayList<>();
for (File f : parentDir.listFiles()) {
if (f.isDirectory()) {
list.addAll(getJarURLs(f));
} else {
String name = f.getName();
if (name.endsWith(".jar")) {
list.add(f.toURI().toURL());
}
}
}
return list;
}
}
Here's an example of the URL output added to the array:
file:/C:/my/path/to/dependent/jars/sqlite-jdbc-3.32.3.2.jar
file:/C:/my/path/to/main/app.jar
file: ... [10 more]
The URLClassloader seems to work well enough to load my main method in app.jar. The main executes a some startup type stuff, before attempting to load a login screen. When the request is made to get the user info database, my message screen loads and displays (<-this is important for later)
the stacktrace containing:
java.sql.SQLException: No suitable driver found for jdbc:sqlite:C:\...\users.db
I understand that this is because that jar is not on the class path, but it's loaded via the class loader, so why can't it find the classes from the jar? From this post JamesB suggested adding Class.forName("org.sqlite.JDBC"); before the connection request. I rebuilt the app jar with this line of code and it worked!
The weird thing that happened next, is that my message screen class can no longer be found even though earlier it loaded and displayed correctly. The message screen is a class inside my main app.jar and not in a dependent jar, which is why I'm baffled. Am I going to have to add Class.forName before every instance of any of my classes? That seems rude..
So what could I be doing wrong with the class loader? Why does it load some classes and not others despite that fact that all the jars have been added to the URL array?
Some other relative info: My app works perfectly as intended when launched from windows command line when the classpath is specified: java -cp "main-app.jar;my/dependent/jar/directory/*" main.AppStart. It's only when I try launching the app via this classloader that I have these issues.
By the way, is this java command universal? Will it work on all operating systems with java installed? If so, could I not just scrap this launcher, and use a process builder to execute the above command? Bonus points for someone who can tell me how to execute the command from a jre packaged with my app, as that's what I plan on doing so the user does not have to download Java.

EDIT
I figured out one of the answers to one of the questions below. Turns out, I didn't need to do any of the code below. My main method loads a login screen but after it's loaded it returns back to the AppLauncher code, thus closing the URLClassLoader! Of course, at that point any requested class will not be found as the loader has been closed! What an oof! Hopefully I will save someone a headache in the future...
Original
Well, after more time, effort, research, and effective use of Eclipse's debugging tool, I was able to figure out what I needed to do to resolve my issues.
So the first issue was my JDBC driver was never registered when passing the jars to the URLClassloader. This is the part I sorta don't understand, so advisement would be welcomed, but there is a static block in the JDBC class that registers the driver so it can be used by DriverManager see code below. Loading the class is what executes that static block, hence why calling Class.forName works.
static {
try {
DriverManager.registerDriver(new JDBC());
} catch (SQLException e) {
e.printStackTrace();
}
}
What I don't understand, is how class loading works if jars are specified via the class path. The URLClassLoader doesn't load any of those classes until they are called, and I never directly work with the JDBC class, thus no suitable driver exception, but are all the classes specified via the classpath loaded initially? Seems that way for static blocks to execute.
Anyhow, to resolve my other issue with some of my app's classes not being found I had to implement my own classloader. I get what I did and how it works well, but still don't understand why I had to do it. All of my jars were loaded to the original URLClassloader so if I could find them and the files within, why couldn't it do it?
Basically, I had to override the findClass and findResource methods to return jarEntry information that I had to store. I hope this code helps someone!
public class SBURLClassLoader extends URLClassLoader {
private HashMap<String, Storage> map;
public SBURLClassLoader(URL[] urls) {
super(urls);
map = new HashMap<>();
try {
storeClasses(urls);
} catch (ClassNotFoundException e) {
e.printStackTrace();
}
}
private void storeClasses(URL[] urls) throws ClassNotFoundException {
for (URL u : urls) {
try {
JarFile jarFile = new JarFile(new File(u.getFile()));
Enumeration<JarEntry> e = jarFile.entries();
while (e.hasMoreElements()) {
JarEntry jar = e.nextElement();
String entryName = jar.getName();
if (jar.isDirectory()) continue;
if (!entryName.endsWith(".class")) {
//still need to store these non-class files as resources
//let code continue to store entry un-altered
} else {
entryName = entryName.replace(".class", "");
entryName = entryName.replace("/", ".");
}
map.put(entryName, new Storage(jarFile, jar));
System.out.println(entryName);
}
} catch (IOException e) {
e.printStackTrace();
}
}
}
#Override
protected Class<?> findClass(String name) throws ClassNotFoundException {
Class<?> c = null;
try {
c = super.findClass(name);
} catch (ClassNotFoundException e) {
Storage s = map.get(name);
try {
InputStream in = s.jf.getInputStream(s.je);
int len = in.available();
c = defineClass(name, in.readAllBytes(), 0, len);
resolveClass(c);
} catch (IOException e1) {
e1.printStackTrace();
}
if (c == null) throw e;
}
return c;
}
#Override
public URL findResource(String name) {
URL url = super.findResource(name);
if (url == null) {
Storage s = map.get(name);
if (s != null) {
try {
url = new URL("jar:"+s.base.toString() + "!/" + name);
} catch (Exception e) {
e.printStackTrace();
}
}
}
return url;
}
private class Storage {
public JarFile jf;
public JarEntry je;
public URL base;
public Storage(JarFile jf, JarEntry je) {
this.jf = jf;
this.je = je;
try {
base = Path.of(jf.getName()).toUri().toURL();
} catch (MalformedURLException e) {
e.printStackTrace();
}
}
}
}

Issue with sapjco3 driver

I've written a Spring MVC (Spring framework 4.1.1) java 1.8 application that successfully connects to SAP using the sapjco3.jar driver, and I've accomplished this using the CustomDestinationDataProvider technique. I then use this drive to call RFCs in my SAP R/3 system. The java code is executed via api call from an AngularJS front end application.
Something that I've discovered occuring about 5% of the time that the call to SAP happens is the following error occurs:
NestedServletException: Handler processing failed; nested exception is
java.lang.Error: java.lang.IllegalStateException: DestinationDataProvider
already registered
Here's the contents of my CustomDestinationDataProvider.java file:
public class CustomDestinationDataProvider {
public class MyDestinationDataProvider implements DestinationDataProvider {
private DestinationDataEventListener eL;
private HashMap<String, Properties> secureDBStorage = new HashMap<String, Properties>();
public Properties getDestinationProperties(String destinationName) {
try {
Properties p = secureDBStorage.get(destinationName);
if(p!=null) {
if(p.isEmpty())
throw new DataProviderException(DataProviderException.Reason.INVALID_CONFIGURATION, "destination configuration is incorrect", null);
return p;
}
return null;
} catch(RuntimeException re) {
throw new DataProviderException(DataProviderException.Reason.INTERNAL_ERROR, re);
}
}
public void setDestinationDataEventListener(DestinationDataEventListener eventListener) {
this.eL = eventListener;
}
public boolean supportsEvents() {
return true;
}
public void changeProperties(String destName, Properties properties) {
synchronized(secureDBStorage) {
if(properties==null) {
if(secureDBStorage.remove(destName)!=null)
eL.deleted(destName);
} else {
secureDBStorage.put(destName, properties);
eL.updated(destName); // create or updated
}
}
}
}
public ArrayList<MaterialBean> executeAvailabilityCall(Properties connectProperties, String searchString) {
String destName = "ABAP_AS";
SAPDAO sapDAO = new SAPDAO();
ArrayList<MaterialBean> searchResults = new ArrayList<MaterialBean>();
MyDestinationDataProvider myProvider = new MyDestinationDataProvider();
JCoDestination dest;
try {
com.sap.conn.jco.ext.Environment.registerDestinationDataProvider(myProvider);
} catch(IllegalStateException providerAlreadyRegisteredException) {
}
myProvider.changeProperties(destName, connectProperties);
try {
dest = JCoDestinationManager.getDestination(destName);
searchResults = sapDAO.searchAvailability(dest, searchString);
} catch(JCoException e) {
e.printStackTrace();
} catch (Exception e) {
e.printStackTrace();
}
myProvider.changeProperties(destName, null);
try {
com.sap.conn.jco.ext.Environment.unregisterDestinationDataProvider(myProvider);
} catch(IllegalStateException providerAlreadyRegisteredException) {
throw new Error(providerAlreadyRegisteredException);
}
return searchResults;
} // end method executeAvailabilityCall()
} // end class CustomDestinationProvider()
My guess is that multiple api calls are occuring at the same time, and once the first query registers the destination data provider, the subsequent queries, which try to also register the destination data provider, fail because they are using the same value for 'destName' in the executeAvailabilityCall method.
Upon first glace, it seems to me like I should use a dynamic value for the destName variable instead of just using "ABAP_AS" for all queries. In other words, I should change the following line:
String destName = "ABAP_AS";
to something like this:
String destName = "ABAP_AS_" + LocalDateTime.now();
This would guarantee a unique value for the destName variable, thus a unique destination provider name.
Any thoughts on the wisdom of trying this? If this is not a good idea, what other solution would be worth exploring?

Yes, you should use multiple unique destination names for your various logon Properties configuration sets. Your class MyDestinationDataProvider is already implemented that way. But why putting a timestamp into the destination name? Why not simply using a destination name schema like "TargetSystem_<SID>_with_<username>"?
Regarding your exception, simply register MyDestinationDataProvider only once and do not permanently register and unregister it. This is not how JCo expects this to be implemented. Quote from the JCo JavaDoc at com.sap.conn.jco.ext.DestinationDataProvider:
Only one implementation of DestinationDataProvider can be registered.
For registering another implementation the infrastructure has first to
unregister the implementation that is currently registered. It is not
recommended to permanently exchange DestinationDataProvider
registrations. The one registered instance should globally manage all
destination configurations for the whole infrastructure environment.

Google App Engine - cannot resolve method 'execute()', but project compiles

I'm writing an Android client that works with a Google App Engine backend. The client compiles and runs just fine, but the execute() method is yet unidentified.
My AsyncTask's doInBackground(...):
#Override
protected String doInBackground(Pair<Context, String>... params) {
context = params[0].first;
if(mService == null) {
DatastoreRequests.Builder builder = new DatastoreRequests.Builder(AndroidHttp.newCompatibleTransport(), new AndroidJsonFactory(), null)
.setRootUrl(myRootUrl);
mService = builder.build();
}
try {
return String.valueOf(mService.sayHi(params[0].second).execute()); // 'execute' is red colored
} catch (IOException e) {
return e.getMessage();
}
}
sayHi:
public DatastoreRequests.SayHi sayHi(String name) throws IOException {
DatastoreRequests.SayHi result = new DatastoreRequests.SayHi(name);
this.initialize(result);
return result;
}
As I mentioned, the code does run, and for example, when removing the String.valueOf() method - the build fails. So, why is it unrecognized? Any explanation for such behavior and how to solve it? Thanks.
I'm working with IntelliJ IDEA 2016.1.1.

Mocking/Testing HTTP Get Request

I'm trying to write unit tests for my program and use mock data. I'm a little confused on how to intercept an HTTP Get request to a URL.
My program calls a URL to our API and it is returned a simple XML file. I would like the test to instead of getting the XML file from the API online to receive a predetermined XML file from me so that I can compare the output to the expected output and determine if everything is working correctly.
I was pointed to Mockito and have been seeing many different examples such as this SO post, How to use mockito for testing a REST service? but it's not becoming clear to me how to set it all up and how to mock the data (i.e., return my own xml file whenever the call to the URL is made).
The only thing I can think of is having another program made that's running locally on Tomcat and in my test pass a special URL that calls the locally running program on Tomcat and then return the xml file that I want to test with. But that just seems like overkill and I don't think that would be acceptable. Could someone please point me in the right direction.
private static InputStream getContent(String uri) {
HttpURLConnection connection = null;
try {
URL url = new URL(uri);
connection = (HttpURLConnection) url.openConnection();
connection.setRequestMethod("GET");
connection.setRequestProperty("Accept", "application/xml");
return connection.getInputStream();
} catch (MalformedURLException e) {
LOGGER.error("internal error", e);
} catch (IOException e) {
LOGGER.error("internal error", e);
} finally {
if (connection != null) {
connection.disconnect();
}
}
return null;
}
I am using Spring Boot and other parts of the Spring Framework if that helps.

Part of the problem is that you're not breaking things down into interfaces. You need to wrap getContent into an interface and provide a concrete class implementing the interface. This concrete class will then
need to be passed into any class that uses the original getContent. (This is essentially dependency inversion.) Your code will end up looking something like this.
public interface IUrlStreamSource {
InputStream getContent(String uri)
}
public class SimpleUrlStreamSource implements IUrlStreamSource {
protected final Logger LOGGER;
public SimpleUrlStreamSource(Logger LOGGER) {
this.LOGGER = LOGGER;
}
// pulled out to allow test classes to provide
// a version that returns mock objects
protected URL stringToUrl(String uri) throws MalformedURLException {
return new URL(uri);
}
public InputStream getContent(String uri) {
HttpURLConnection connection = null;
try {
Url url = stringToUrl(uri);
connection = (HttpURLConnection) url.openConnection();
connection.setRequestMethod("GET");
connection.setRequestProperty("Accept", "application/xml");
return connection.getInputStream();
} catch (MalformedURLException e) {
LOGGER.error("internal error", e);
} catch (IOException e) {
LOGGER.error("internal error", e);
} finally {
if (connection != null) {
connection.disconnect();
}
}
return null;
}
}
Now code that was using the static getContent should go through a IUrlStreamSource instances getContent(). You then provide to the object that you want to test a mocked IUrlStreamSource rather than a SimpleUrlStreamSource.
If you want to test SimpleUrlStreamSource (but there's not much to test), then you can create a derived class that provides an implementation of stringToUrl that returns a mock (or throws an exception).

The other answers in here advise you to refactor your code to using a sort of provider which you can replace during your tests - which is the better approach.
If that isn't a possibility for whatever reason you can install a custom URLStreamHandlerFactory that intercepts the URLs you want to "mock" and falls back to the standard implementation for URLs that shouldn't be intercepted.
Note that this is irreversible, so you can't remove the InterceptingUrlStreamHandlerFactory once it's installed - the only way to get rid of it is to restart the JVM. You could implement a flag in it to disable it and return null for all lookups - which would produce the same results.
URLInterceptionDemo.java:
public class URLInterceptionDemo {
private static final String INTERCEPT_HOST = "dummy-host.com";
public static void main(String[] args) throws IOException {
// Install our own stream handler factory
URL.setURLStreamHandlerFactory(new InterceptingUrlStreamHandlerFactory());
// Fetch an intercepted URL
printUrlContents(new URL("http://dummy-host.com/message.txt"));
// Fetch another URL that shouldn't be intercepted
printUrlContents(new URL("http://httpbin.org/user-agent"));
}
private static void printUrlContents(URL url) throws IOException {
try(InputStream stream = url.openStream();
BufferedReader reader = new BufferedReader(new InputStreamReader(stream))) {
String line;
while((line = reader.readLine()) != null) {
System.out.println(line);
}
}
}
private static class InterceptingUrlStreamHandlerFactory implements URLStreamHandlerFactory {
#Override
public URLStreamHandler createURLStreamHandler(final String protocol) {
if("http".equalsIgnoreCase(protocol)) {
// Intercept HTTP requests
return new InterceptingHttpUrlStreamHandler();
}
return null;
}
}
private static class InterceptingHttpUrlStreamHandler extends URLStreamHandler {
#Override
protected URLConnection openConnection(final URL u) throws IOException {
if(INTERCEPT_HOST.equals(u.getHost())) {
// This URL should be intercepted, return the file from the classpath
return URLInterceptionDemo.class.getResource(u.getHost() + "/" + u.getPath()).openConnection();
}
// Fall back to the default handler, by passing the default handler here we won't end up
// in the factory again - which would trigger infinite recursion
return new URL(null, u.toString(), new sun.net.www.protocol.http.Handler()).openConnection();
}
}
}
dummy-host.com/message.txt:
Hello World!
When run, this app will output:
Hello World!
{
"user-agent": "Java/1.8.0_45"
}
It's pretty easy to change the criteria of how you decide which URLs to intercept and what you return instead.

The answer depends on what you are testing.
If you need to test the processing of the InputStream
If getContent() is called by some code that processes the data returned by the InputStream, and you want to test how the processing code handles specific sets of input, then you need to create a seam to enable testing. I would simply move getContent() into a new class, and inject that class into the class that does the processing:
public interface ContentSource {
InputStream getContent(String uri);
}
You could create a HttpContentSource that uses URL.openConnection() (or, better yet, the Apache HttpClientcode).
Then you would inject the ContentSource into the processor:
public class Processor {
private final ContentSource contentSource;
#Inject
public Processor(ContentSource contentSource) {
this.contentSource = contentSource;
}
...
}
The code in Processor could be tested with a mock ContentSource.
If you need to test the fetching of the content
If you want to make sure that getContent() works, you could create a test that starts a lightweight in-memory HTTP server that serves the expected content, and have getContent() talk to that server. That does seem overkill.
If you need to test a large subset of the system with fake data
If you want to make sure things work end to end, write an end to end system test. Since you indicated you use Spring, you can use Spring to wire together parts of the system (or to wire the entire system, but with different properties). You have two choices
Have the system test start a local HTTP server, and when you have your test create your system, configure it to talk to that server. See the answers to this question for ways to start the HTTP server.
Configure spring to use a fake implementation of ContentSource. This gets you slightly less confidence that everything works end-to-end, but it will be faster and less flaky.

Java Webdav File Synchronization

I have a cloud storage at strato namely hidrive. It uses the webdav protocol. Note that it's based on HTTP. The client application they provide is poor and buggy so I tried various other tools for synchronization but none just worked the way I need it.
I'm therefore trying to implement it in Java using the Sardine project. Is there any code for hard-copying a local source folder to an external cloud folder? I haven't found anything in that direction.
The following code is supposed to upload the file...
Sardine sardine = SardineFactory.begin("username", "password");
InputStream fis = new FileInputStream(new File("some/file/test.txt"));
sardine.put("https://webdav.hidrive.strato.com/users/username/Backup", fis);
... but throws an exception instead:
Exception in thread "main" com.github.sardine.impl.SardineException: Unexpected response (301 Moved Permanently)
at com.github.sardine.impl.handler.ValidatingResponseHandler.validateResponse(ValidatingResponseHandler.java:48)
at com.github.sardine.impl.handler.VoidResponseHandler.handleResponse(VoidResponseHandler.java:34)
at com.github.sardine.impl.handler.VoidResponseHandler.handleResponse(VoidResponseHandler.java:1)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:218)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:160)
at com.github.sardine.impl.SardineImpl.execute(SardineImpl.java:828)
at com.github.sardine.impl.SardineImpl.put(SardineImpl.java:755)
at com.github.sardine.impl.SardineImpl.put(SardineImpl.java:738)
at com.github.sardine.impl.SardineImpl.put(SardineImpl.java:726)
at com.github.sardine.impl.SardineImpl.put(SardineImpl.java:696)
at com.github.sardine.impl.SardineImpl.put(SardineImpl.java:689)
at com.github.sardine.impl.SardineImpl.put(SardineImpl.java:682)
at com.github.sardine.impl.SardineImpl.put(SardineImpl.java:676)
Printing out the folders in that directory works so the connection/ authentication did succeed:
List<DavResource> resources = sardine.list("https://webdav.hidrive.strato.com/users/username/Backup");
for (DavResource res : resources)
{
System.out.println(res);
}
Please either help me fix my code or link me to some file synchronization library that works for my purpose.

Sardine uses (internally) HttpClient. There is similar question here where you can find an answer Httpclient 4, error 302. How to redirect?.

Try converting the InputStream obj into byte array before you call put(). Something like the below,
byte[] fisByte = IOUtils.toByteArray(fis);
sardine.put("https://webdav.hidrive.strato.com/users/username/Backup", fisByte);
It worked for me. Let me know.

I had to extend the "org.apache.http.impl.client.LaxRedirectStrategy" and also the getRedirect() Method of org.apache.http.impl.client.DefaultRedirectStrategy with a treatment of the needed methods: PUT, MKOL, etc. . By default only GET is redirected.
It looks like this:
private static final String[] REDIRECT_METHODS = new String[] { HttpGet.METHOD_NAME, HttpPost.METHOD_NAME, HttpHead.METHOD_NAME, HttpPut.METHOD_NAME, HttpDelete.METHOD_NAME, HttpMkCol.METHOD_NAME };
isRedirectable-Method
for (final String m : REDIRECT_METHODS) {
if (m.equalsIgnoreCase(method)) {
System.out.println("isRedirectable true");
return true;
}
}
return method.equalsIgnoreCase(HttpPropFind.METHOD_NAME);
getRedirect-Method:
final URI uri = getLocationURI(request, response, context);
final String method = request.getRequestLine().getMethod();
if (method.equalsIgnoreCase(HttpHead.METHOD_NAME)) {
return new HttpHead(uri);
} else if (method.equalsIgnoreCase(HttpGet.METHOD_NAME)) {
return new HttpGet(uri);
} else if (method.equalsIgnoreCase(HttpPut.METHOD_NAME)) {
HttpPut httpPut = new HttpPut(uri);
httpPut.setEntity(((HttpEntityEnclosingRequest) request).getEntity());
return httpPut;
} else if (method.equalsIgnoreCase("MKCOL")) {
return new HttpMkCol(uri);
} else if (method.equalsIgnoreCase("DELETE")) {
return new HttpDelete(uri);
} else {
final int status = response.getStatusLine().getStatusCode();
if (status == HttpStatus.SC_TEMPORARY_REDIRECT) {
return RequestBuilder.copy(request).setUri(uri).build();
} else {
return new HttpGet(uri);
}
}
That worked for me.