We Keep Coding

Java ClassLoading from client's memory

I am developing a client-server application using Java Sockets where at some point the client has to download some classes from the server (through the already bound Socket) and use them. I am able to read the class file bytes and send them to the client. So then, the client has to use a ClassLoader to load the classes from memory. So, in my opinion, the problem is not really related to Sockets, but it is about custom class loading (you will see why I say that, later in this post).
My setup is as follows: the client project with a single package named client with a single class in it, named LiveClientTest and then the server project with a single package named server with 3 classes in it: ClientMain, LiveServerTest (the entry point) and MyStrings. In short, all server-side code is under the package server and all client-side under the client. Each of the two packages is in a different project also.
The problem occurs when the custom ClassLoader of the client (named LiveClientTest.MemoryClassLoader) tries to load a non-static nested class (named MyStrings.NestedNonStaticSubclass) which refers to its enclosing class (named MyStrings) before constructing the object (of type MyStrings.NestedNonStaticSubclass). Although the classes compile fine, the error appears at runtime, while loading the class MyStrings.NestedNonStaticSubclass with the custom ClassLoader. I know it sounds weird, but you can see what I mean if you take a look at the code.
Server side code:
LiveServerTest class (entry point):
package server;
import java.io.DataOutputStream;
import java.io.File;
import java.io.IOException;
import java.io.OutputStream;
import java.net.ServerSocket;
import java.net.Socket;
import java.net.URISyntaxException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.Objects;
import javax.swing.JOptionPane;
public class LiveServerTest {
//Convert class binary name representation to file path:
public static String classNameToResourcePath(final String className) {
return className.replace('.', File.separatorChar) + ".class";
}
//Get directory location of the given class (packages not included):
public static Path getDirectoyPath(final Class clazz) throws URISyntaxException {
return new File(clazz.getProtectionDomain().getCodeSource().getLocation().toURI()).toPath();
}
//Get absolute file location of the given class:
public static Path getClassFilePath(final Class c) throws URISyntaxException {
return getDirectoyPath(c).resolve(classNameToResourcePath(c.getName()));
}
public static void main(final String[] args) {
ServerSocket srv = null;
try {
srv = new ServerSocket(Integer.parseInt(Objects.requireNonNull(JOptionPane.showInputDialog(null, "Enter host port number:"))));
System.out.println("Waiting for client connection...");
try (final Socket sck = srv.accept();
final OutputStream os = sck.getOutputStream()) {
srv.close();
srv = null;
//These are the classes we need the client to load:
final Class[] clientClasses = new Class[] {
ClientMain.class,
MyStrings.class,
MyStrings.NestedStatic.class,
MyStrings.NestedNonStaticSubclass.class
};
System.out.println("Sending all client classes' bytes to client...");
final DataOutputStream dos = new DataOutputStream(os);
dos.writeInt(clientClasses.length);
for (final Class clazz: clientClasses) {
final byte[] contents = Files.readAllBytes(getClassFilePath(clazz));
dos.writeUTF(clazz.getName());
dos.writeInt(contents.length);
dos.write(contents);
}
System.out.println("Main application now starts...");
//Here would go the server side code for the client-server application.
}
}
catch (final IOException | URISyntaxException | RuntimeException x) {
JOptionPane.showMessageDialog(null, x.toString());
}
finally {
System.out.println("Done.");
try { if (srv != null) srv.close(); } catch (final IOException iox) {}
}
}
}
ClientMain class:
package server;
import java.io.DataInputStream;
import java.net.Socket;
public class ClientMain {
//This method is called by the client to start the application:
public static void main(final Socket sck,
final DataInputStream dis,
final ClassLoader loader,
final String[] args) {
System.out.println("Running...");
//Here would go the client side code for the client-server application.
//Just test that all the classes are loaded successfully:
System.out.println(new MyStrings("A", "B", "C").new NestedNonStaticSubclass().getFunction().apply(2)); //Should print "C".
}
}
MyStrings class:
package server;
import java.util.function.IntFunction;
public class MyStrings {
public static class NestedStatic {
private final IntFunction<String> f;
public NestedStatic(final IntFunction<String> f) {
this.f = f;
}
public IntFunction<String> getFunction() {
return f;
}
}
//This class produces the error when loaded:
public class NestedNonStaticSubclass extends NestedStatic {
public NestedNonStaticSubclass() {
super(i -> getString(i)); //Here we refer to MyStrings object before constructing the NestedNonStaticSubclass object.
}
}
private final String[] array;
public MyStrings(final String... array) {
this.array = array.clone();
}
public String getString(final int i) {
return array[i];
}
}
Client side code:
package client;
import java.io.DataInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.lang.reflect.InvocationTargetException;
import java.net.Socket;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.Iterator;
import java.util.Objects;
import javax.swing.JOptionPane;
public class LiveClientTest {
public static class MemoryClassLoader extends ClassLoader {
//The key is the name of the class, and value is the compiled class file bytes:
private final HashMap<String, byte[]> classByteCode = new HashMap<>();
#Override
public /*synchronized*/ Class findClass(final String name) throws ClassNotFoundException {
try {
return super.findClass(name);
}
catch (final ClassNotFoundException cnfx) {
if (!classByteCode.containsKey(name))
throw new ClassNotFoundException(name);
final byte[] byteCode = classByteCode.get(name);
return defineClass(name, byteCode, 0, byteCode.length);
}
}
//Try to load all classes that are downloaded (with readClass):
ArrayList<String> loadClasses() {
final ArrayList<String> classNames = new ArrayList<>(classByteCode.keySet());
int oldSize;
do {
oldSize = classNames.size();
final Iterator<String> classNamesIter = classNames.iterator();
while (classNamesIter.hasNext()) {
final String className = classNamesIter.next();
try {
loadClass(className);
classNamesIter.remove();
}
catch (final ClassNotFoundException x) {
}
}
}
while (classNames.size() < oldSize); //If we reached a point where there are some classes that can not be loaded, then quit...
return classNames; //Return the classes that failed to be loaded (if any) (should be empty).
}
//Read class bytes from supplied DataInputStream:
void readClass(final DataInputStream dis) throws IOException {
final String name = dis.readUTF();
final byte[] contents = new byte[dis.readInt()];
int i = 0, n = dis.read(contents);
//Make sure all 'contents.length' multitude of bytes are read:
while (n >= 0 && (i + n) < contents.length) {
i += n;
n = dis.read(contents, i, contents.length - i);
}
if (/*n < 0 ||*/ (i + n) != contents.length)
throw new IOException("Unfinished class input (" + name + " - " + contents.length + ").");
classByteCode.put(name, contents);
}
}
public static void main(final String[] args) {
try {
final String host = Objects.requireNonNull(JOptionPane.showInputDialog(null, "Enter host name or address:"));
final int port = Integer.parseInt(Objects.requireNonNull(JOptionPane.showInputDialog(null, "Enter host port number:")));
try (final Socket sck = new Socket(host, port);
final InputStream is = sck.getInputStream()) {
final DataInputStream dis = new DataInputStream(is);
final MemoryClassLoader loader = new MemoryClassLoader();
//Download all classes and put them into the class loader:
System.out.println("Downloading...");
for (int i = dis.readInt(); i > 0; --i)
loader.readClass(dis);
//Load all downloaded classes from the class loader:
System.out.println("Loading...");
System.out.println("Failed to load: " + loader.loadClasses() + '.');
//Call main method in main downloaded class:
loader
.loadClass("server.ClientMain") //package is from server side.
.getDeclaredMethod("main", Socket.class, DataInputStream.class, ClassLoader.class, String[].class)
.invoke(null, sck, dis, loader, args);
}
}
catch (final IOException | ClassNotFoundException | NoSuchMethodException | IllegalAccessException | InvocationTargetException | RuntimeException x) {
x.printStackTrace();
JOptionPane.showMessageDialog(null, x);
}
}
}
Client side output:
Downloading...
Loading...
Exception in thread "main" java.lang.ClassFormatError: Illegal field name "server.MyStrings$this" in class server/MyStrings$NestedNonStaticSubclass
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
at java.lang.ClassLoader.defineClass(ClassLoader.java:642)
at client.LiveClientTest$MemoryClassLoader.findClass(LiveClientTest.java:30)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at client.LiveClientTest$MemoryClassLoader.loadClasses(LiveClientTest.java:44)
at client.LiveClientTest.main(LiveClientTest.java:89)
So my question is:
Why does the code fail with a ClassFormatError, what does that mean, and how to avoid it in this particular scenario?
My question is not:
What alternatives exist? (such as using a URLClassLoader, or alternative ways of class loading from memory other than a custom ClassLoader, etc...)
How to reproduce:
I am using JRE 1.8.0_251 (and I would like this to work for 1.8), so I think you must put the source files in different projects (one for the client and one for the server) in order to make sure that the client does not already have direct visibility of the server's classes while class-loading them.
Run the server's main class server.LiveServerTest and input a port number for the server in the dialog that pops up. Then, run the client's main class client.LiveClientTest and enter localhost for the host (first dialog that pops up) and then the port number of the server (second dialog that pops up).
The stack trace will be in your CLI (through System.err) and not in a GUI.
The code will not work if you build the projects into jar files, but it is written to work for plain class files into directories, for simplicity. For example, in NetBeans, don't build into jar files, but rather click Run File for each entry point.
The built classes should have their file's .class extention in lower case.
Warning:
If you put the source files in different packages but in the same project, or even worse in the same package, then the class loading might succeed without errors because:
I am using the default constructor of the ClassLoader class in my LiveClientTest.MemoryClassLoader class, which means the parent class loader is the system class loader.
The LiveClientTest.MemoryClassLoader.findClass method first searches the parent ClassLoader and then, if that fails, it searches the downloaded classes. To my knowledge, this is the suggested way of implementing this, mainly because the ClassLoader class (which is the parent class of my LiveClientTest.MemoryClassLoader class) caches already defined classes.
References:
How to load JAR files dynamically at Runtime?
In which scenarios the remote class loading are needed?
Does the Java ClassLoader load inner classes?
Java - Get a list of all Classes loaded in the JVM
Java: How to load a class (and its inner classes) that is already on the class path?
Create new ClassLoader to reload Class
How to use classloader to load class file from server to client
Custom Java classloader and inner classes
ClassFormatError in java 8?
JVM Invalid Nested Class Name?
https://bugs.openjdk.java.net/browse/JDK-8145051
https://www.programming-books.io/essential/java/implementing-a-custom-classloader-0f0fe95cf7224c668e631a671eef3b94
https://www.baeldung.com/java-classloaders
https://www.infoworld.com/article/2077260/learn-java-the-basics-of-java-class-loaders.html
https://www.oracle.com/technical-resources/articles/javase/classloaders.html
https://www.ibm.com/developerworks/library/j-onejar/index.html
I am new to class loading, so please don't take my words for granted.
Huge post, because of the divided code of the MRE. Sorry. I tried to make the code as minimal as possible.

Loading class from bin folder

I would like to use .class file from bin folder in my code - convert it to bytes, but have no idea how to get to it. I have bin/example.class and I need to load it and check how many bytes does my class have.
I found something like:
public class MyClassLoader extends ClassLoader{
public MyClassLoader(){
super(MyClassLoader.class.getClassLoader());
}
}
But it doesn't seem to help, it must be some extremely easy way to do this. It looks really easy and whole internet try to push me into writing thousand lines of classLoader Code.
EDIT: My java file is compiled programatically and .class file is created programatically, so I can't just refer to it's name, it's also somewhere else in workspace.
Some hints?

Just add the bin folder to your class path!
To get the number of bytes, get the resource URL, convert to a File object and query the size.
Example:
package test;
import java.io.File;
import java.net.URISyntaxException;
import java.net.URL;
public class Example {
public static final String NAME = Example.class.getSimpleName() + ".class";
public static void main(String[] args) throws URISyntaxException {
URL url = Example.class.getResource(NAME);
long size = new File(url.toURI().getPath()).length();
System.out.printf("The size of file '%s' is %d bytes\n", NAME, size);
}
}
Will output:
The size of file 'Example.class' is 1461 bytes

You could do something like this:
public class MyClassLoader extends ClassLoader {
protected synchronized Class<?> loadClass(String name, boolean resolve)
throws ClassNotFoundException {
try {
return super.loadClass(name, resolve);
}
catch (ClassNotFoundException e) {
// TODO: test, if you can load the class with
// the given name. if not, rethrow the exception!
byte[] b = loadClassData(name);
return defineClass(name, b, 0, b.length);
}
}
private byte[] loadClassData(String name) {
// TODO: read contents of your file to byte array
}
}

JBoss 7: how to dynamically load jars

I am using JBoss 7 (dependency loading was changed in this version).
My war-application uploads to server jars and need to use classes inside of them, but it gets ClassNotFoundException.
So I can't find a way to add jar-dependencies to modules dynamically - MANIFEST.MF, jboss-deployment-structure.xml are static way of doing this.

Just rephrasing the question to make sure I it correctly;
You want to be able to upload an arbitrary jar file to the server and then use the contained classes/resources in the JVM? Without restarting the JVM and/or editing your configuration ofcourse.
If that's the case, then you should load the jar into a classloader (chaining your current classloader if needed) and then load the class from there.
Assuming you store the jar-file physically on the server you could for example do something like:
public static Class<?> loadClass(String className, String jarFileLocation)
throws MalformedURLException, ClassNotFoundException {
URL jarUrl = new File(jarFileLocation).toURI().toURL();
ClassLoader classLoader = new URLClassLoader(new URL[] {jarUrl }, MyClass.class.getClassLoader());
return classLoader.loadClass(className);
}
public static Object executeMethodOndClass(String methodName, Class<?>[] parameterTypes,
Object[] parameters, String className, String jarFileLocation)
throws MalformedURLException, ClassNotFoundException, IllegalAccessException, InstantiationException,
NoSuchMethodException, InvocationTargetException {
Class<?> loadedClass = loadClass(className, jarFileLocation);
Method method = loadedClass.getMethod(methodName, parameterTypes);
Object instance = loadedClass.newInstance();
return method.invoke(instance, parameters);
}
Ps. this is crude code, I didn't even compile or test it; it should work, but nothing more then that and there is the chance I overlooked something or made a typo ;-)
Pps. allowing custom jar files to be uploaded and classes from it to be executed does bring a number of (security) risks with it.

#Rage: This question on stackoverflow asked earlier might give you some inputs how to organize jars: be it your own or third-party jars.

Try this (I've grabbed it somewhere on the Internet):
import java.io.File;
import java.io.IOException;
import java.lang.reflect.Method;
import java.net.URL;
import java.net.URLClassLoader;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
public final class ClassPathHacker {
private static final Class<?>[] PARAMS = new Class<?>[] { URL.class };
private static final Logger LOG_CPH = LoggerFactory.getLogger(ClassPathHacker.class);
private ClassPathHacker() {}
public static void addFile(final String pFileName) throws IOException {
final File myFile = new File(pFileName);
ClassPathHacker.addFile(myFile);
}
public static void addFile(final File pFile) throws IOException {
ClassPathHacker.addURL(pFile.toURI().toURL());
}
public static void addURL(final URL pFileUrl) throws IOException {
/* variables definition */
final URLClassLoader sysLoader = (URLClassLoader) ClassLoader.getSystemClassLoader();
final Class<?> sysClass = URLClassLoader.class;
try {
final Method myMethod = sysClass.getDeclaredMethod("addURL", PARAMS);
myMethod.setAccessible(true);
myMethod.invoke(sysLoader, new Object[] { pFileUrl });
} catch (final Exception exc) {
ClassPathHacker.LOG_CPH.error(exc.getLocalizedMessage(), exc);
throw new IOException(exc.getLocalizedMessage());
}
}
}
Together with this method:
private static void hackClassPath(final File myData) {
if (myData.isDirectory()) {
/* block variables */
final File[] myList = myData.listFiles();
/* hacking classpath... */
for (final File tmp : myList) {
try {
ClassPathHacker.addFile(tmp.getAbsolutePath());
MainApplication.MAIN_LOG.trace("{} added to classpath",
tmp.getAbsolutePath());
} catch (final IOException iOE) {
MainApplication.MAIN_LOG.error(iOE.getLocalizedMessage(),
iOE);
}
}
}
}
And with this sample call:
MainApplication.hackClassPath(new File("test/data"));
MainApplication.hackClassPath(new File("data"));
A bit hacky, but maybe it works... it runtime adds all JAr files available in the data or test/data directory to the running classpath.

How do I correctly load resources from a network directory in the java classpath?

Our java application relies on some resources which are available on a network share. This network share is located on the classpath, and the resources are read at runtime using MyClass.class.getResourceAsStream("/myfile.jpg").
java -Djava.class.path=\\myserver\myshare:C:\myjar.jar MainClass
When the share is available at startup, everything runs smoothly. Image and properties files which are located in the share can be read using getResourceAsStream(). However, if the share is not online when the application starts, even if the share comes online before any resources are read, they cannot be read using getResourceAsStream().
Doing some digging using eclispse + decompiler, I noticed one difference. The default classloader inherits from URLClassLoader, and its ucp member (URLClassPath) contains a list of URLClassPath.Loader instances. In the first scenario, it contains a URLClassPath.FileLoader and a URLClassPath.JarLoader. In the second scenario, it only contains a jar loader.
It's like java determines that the classpath entry is invalid and completely discards it.
Why is this? How can I avoid it?
Update
I am unable to change the mechanism by which we are loading resources because of a few reasons:
There are far too many areas which currently load files this way for me change at the moment
There are situations where by the resource is actually being loaded by a third party component
I have no problem creating a custom class loader, I just need some guidance on how to do it.
I tried with this, but was unable to get expected results:
import java.net.URL;
import java.net.URLClassLoader;
public class MyUrlClassLoader extends URLClassLoader {
public MyUrlClassLoader(ClassLoader parent) {
super(new URL[0], parent);
System.out.println("MyUrlClassLoader ctor");
}
#Override
protected Class<?> findClass(String name) throws ClassNotFoundException {
System.out.println("url find class " + name);
return super.findClass(name);
}
#Override
public Class<?> loadClass(String name) throws ClassNotFoundException {
System.out.println("url load class " + name);
return super.loadClass(name);
}
#Override
public URL getResource(String name) {
System.out.println("url get resource " + name);
return super.getResource(name);
}
}
import java.net.URL;
public class ClassLoaderMain {
public static void main(String[] args) throws ClassNotFoundException {
URL url = ClassLoaderMain.class.getResource("/myfile.txt");
System.out.print("Loaded? ");
System.out.println(url != null);
System.out.println(ClassLoaderMain.class.getClassLoader().toString());
System.out.println(MyUrlClassLoader.class.getClassLoader().toString());
System.out.println(FakeClass.class.getClassLoader().toString());
}
}
When I run java -cp . -Djava.system.class.loader=MyUrlClassLoader ClassLoaderMain
This outputs:
MyUrlClassLoader ctor
url load class java.lang.System
url load class java.nio.charset.Charset
url load class java.lang.String
url load class ClassLoaderMain
Loaded? true
sun.misc.Launcher$AppClassLoader#923e30
sun.misc.Launcher$AppClassLoader#923e30
sun.misc.Launcher$AppClassLoader#923e30
So my class loader is being created, and load class is being called, but it doesn't appear to be the class loader for the classes it is loading?

I ended up resolving this by creating my own ClassLoader, deriving from URLClassLoader.
import java.io.File;
import java.net.MalformedURLException;
import java.net.URI;
import java.net.URL;
import java.net.URLClassLoader;
import java.util.ArrayList;
import java.util.List;
public class CustomClassLoader extends URLClassLoader {
public CustomClassLoader(ClassLoader parent) {
// System classloader will filter inaccessible URLs.
// Force null parent to avoid using system classloader.
super(createURLReferences(), null);
}
/**
* Build an array of URLs based on the java.class.path property.
* #return An array of urls to search for classes.
*/
private static URL[] createURLReferences() {
String classpath = System.getProperty("java.class.path");
String[] classpathEntries = classpath.split(System.getProperty("path.separator"));
List<URL> urls = new ArrayList<URL>();
for (String classpathEntry : classpathEntries) {
File classpathFile = new File(classpathEntry);
URI uri = classpathFile.toURI();
try {
URL url = uri.toURL();
urls.add(url);
} catch (MalformedURLException e) {
System.out.println("Ignoring classpath entry: " + classpathEntry);
}
}
return urls.toArray(new URL[urls.size()]);
}
}

Adding files to java classpath at runtime [duplicate]

This question already has answers here:
How to load JAR files dynamically at Runtime?
(20 answers)
Closed 7 years ago.
Is it possible to add a file (not necessarily a jar file) to java classpath at runtime.
Specifically, the file already is present in the classpath, what I want is whether I can add a modified copy of this file to the classpath.
Thanks,

You can only add folders or jar files to a class loader. So if you have a single class file, you need to put it into the appropriate folder structure first.
Here is a rather ugly hack that adds to the SystemClassLoader at runtime:
import java.io.IOException;
import java.io.File;
import java.net.URLClassLoader;
import java.net.URL;
import java.lang.reflect.Method;
public class ClassPathHacker {
private static final Class[] parameters = new Class[]{URL.class};
public static void addFile(String s) throws IOException {
File f = new File(s);
addFile(f);
}//end method
public static void addFile(File f) throws IOException {
addURL(f.toURL());
}//end method
public static void addURL(URL u) throws IOException {
URLClassLoader sysloader = (URLClassLoader) ClassLoader.getSystemClassLoader();
Class sysclass = URLClassLoader.class;
try {
Method method = sysclass.getDeclaredMethod("addURL", parameters);
method.setAccessible(true);
method.invoke(sysloader, new Object[]{u});
} catch (Throwable t) {
t.printStackTrace();
throw new IOException("Error, could not add URL to system classloader");
}//end try catch
}//end method
}//end class
The reflection is necessary to access the protected method addURL. This could fail if there is a SecurityManager.

Try this one on for size.
private static void addSoftwareLibrary(File file) throws Exception {
Method method = URLClassLoader.class.getDeclaredMethod("addURL", new Class[]{URL.class});
method.setAccessible(true);
method.invoke(ClassLoader.getSystemClassLoader(), new Object[]{file.toURI().toURL()});
}
This edits the system class loader to include the given library jar. It is pretty ugly, but it works.

The way I have done this is by using my own class loader
URLClassLoader urlClassLoader = (URLClassLoader) ClassLoader.getSystemClassLoader();
DynamicURLClassLoader dynalLoader = new DynamicURLClassLoader(urlClassLoader);
And create the following class:
public class DynamicURLClassLoader extends URLClassLoader {
public DynamicURLClassLoader(URLClassLoader classLoader) {
super(classLoader.getURLs());
}
#Override
public void addURL(URL url) {
super.addURL(url);
}
}
Works without any reflection

You coud try java.net.URLClassloader with the url of the folder/jar where your updated class resides and use it instead of the default classloader when creating a new thread.

Yes I believe it's possible but you might have to implement your own classloader. I have never done it but that is the path I would probably look at.

yes, you can. it will need to be in its package structure in a separate directory from the rest of your compiled code if you want to isolate it. you will then just put its base dir in the front of the classpath on the command line.

My solution:
File jarToAdd = new File("/path/to/file");
new URLClassLoader(((URLClassLoader) ClassLoader.getSystemClassLoader()).getURLs()) {
#Override
public void addURL(URL url) {
super.addURL(url);
}
}.addURL(jarToAdd.toURI().toURL());