We Keep Coding

Classloading from package spawns NoClassDefFoundError

I have a small program that attempts to allow plugins via class files copied to a specific ext directory.
The program is derived from https://javaranch.com/journal/200607/Plugins.html and I have attempted to simplify it and add on a directory scanning ability to scan packages and directories that the original code lacks.
When running the original code, it works. When I add on my directory and package scanning capability and test it on a demo package, it fails. Below are the samples.
The directory layout of the system accepting dynamically loaded class files as plugins:
testpack-+
|
+---PluginDemo.java
|
+---PluginFunction.java
The test plugin's directory layout:
b-+
|
+---Fibonacci.java
testpack-+
|
+---PluginFunction.java
The PluginDemo code with custom class loader:
package testpack;
import java.io.DataInputStream;
import java.io.File;
import java.io.FileInputStream;
import java.util.*;
public class PluginDemo extends ClassLoader {
static String pluginsDir = "ext";
static File basePluginDir = null;
File directory;
static List plugins;
public static void main(String args[]) {
PluginDemo demo = new PluginDemo();
basePluginDir = new File(System.getProperty("user.dir") + File.separator + pluginsDir);
demo.getPlugins(pluginsDir, "");
}
PluginDemo() {
plugins = new ArrayList();
}
protected void getPlugins(String directory, String parent) {
File dir = new File(System.getProperty("user.dir") + File.separator + directory);
if (dir.exists() && dir.isDirectory()) {
String[] files = dir.list();
for (int i = 0; i < files.length; i++) {
try {
// Allows recursive targetting of nested directories
String newTargetFile = System.getProperty("user.dir") + File.separator + directory + File.separator
+ files[i];
System.out.println("Targetting: " + newTargetFile);
File currentTarget = new File(newTargetFile);
if (currentTarget.isDirectory()) {
String newDirectoryTarget = directory + File.separator + files[i];
getPlugins(newDirectoryTarget, files[i]);
}
if (!files[i].endsWith(".class"))
continue;
String childFile = parent + File.separator + files[i].substring(0, files[i].indexOf("."));
Class c = loadClass(childFile);
Class[] intf = c.getInterfaces();
for (int j = 0; j < intf.length; j++) {
if (intf[j].getName().equals("PluginFunction")) {
PluginFunction pf = (PluginFunction) c.newInstance();
plugins.add(pf);
continue;
}
}
} catch (Exception ex) {
System.err.println("File " + files[i] + " does not contain a valid PluginFunction class.");
ex.printStackTrace();
}
}
}
}
public Class loadClass(String name) throws ClassNotFoundException {
return loadClass(name, true);
}
public Class loadClass(String classname, boolean resolve) throws ClassNotFoundException {
try {
Class c = findLoadedClass(classname);
if (c == null) {
try {
c = findSystemClass(classname);
} catch (Exception ex) {
}
}
if (c == null) {
String filename = classname.replace('.', File.separatorChar) + ".class";
// Create a File object. Interpret the filename relative to the
// directory specified for this ClassLoader.
File baseDir = new File(System.getProperty("user.dir"));
File f = new File(baseDir, PluginDemo.pluginsDir + File.separator + filename);
int length = (int) f.length();
byte[] classbytes = new byte[length];
DataInputStream in = new DataInputStream(new FileInputStream(f));
in.readFully(classbytes);
in.close();
c = defineClass(classname, classbytes, 0, length);
}
if (resolve)
resolveClass(c);
return c;
} catch (Exception ex) {
throw new ClassNotFoundException(ex.toString());
}
}
}
The PluginFunction interface code:
package testpack;
public interface PluginFunction {
// let the application pass in a parameter
public void setParameter (int param);
// retrieve a result from the plugin
public int getResult();
// return the name of this plugin
public String getPluginName();
// can be called to determine whether the plugin
// aborted execution due to an error condition
public boolean hasError();
}
The Fibonacci.java code:
package b;
import testpack.PluginFunction;
public class Fibonacci implements PluginFunction {
int parameter = 0;
boolean hasError = false;
public boolean hasError() {
return hasError;
}
public void setParameter (int param) {
parameter = param;
}
public int getResult() {
hasError = false;
return fib(parameter);
}
protected int fib (int n) {
if (n < 0) {
hasError = true;
return 0;
}
if (n == 0)
return 0;
else if (n == 1)
return 1;
else
return fib(n-1) + fib(n-2);
}
public String getPluginName() {
return "Fibonacci";
}
}
The output with errors:
Targetting: C:\Users\Administrator\eclipse-workspace\TestPluginSystem\ext\b\Fibonacci.class
Exception in thread "main" java.lang.NoClassDefFoundError: b\Fibonacci (wrong name: b/Fibonacci)
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
at java.lang.ClassLoader.defineClass(ClassLoader.java:642)
at testpack.PluginDemo.loadClass(PluginDemo.java:89)
at testpack.PluginDemo.loadClass(PluginDemo.java:65)
at testpack.PluginDemo.getPlugins(PluginDemo.java:47)
at testpack.PluginDemo.getPlugins(PluginDemo.java:40)
at testpack.PluginDemo.main(PluginDemo.java:19)
I would need help to get this package and directory scanning capable classloader working. Thanks.

Looking at the error and method ClassLoader.defineClass, I think the name parameter must have . as package separators, not / or \.
In your code in getPlugins the childFile is constructed using File.separator
String childFile = parent + File.separator + files[i].substring(0, files[i].indexOf("."));
Class c = loadClass(childFile);

skip duplicate file using java

Can anyone help me to skip file having extension "read" in my code ?
I have two files in my folder:
123.csv
123.csv.read
After execution every csv file is converted into ".csv.read", but if the same file comes again, that file should be skipped.
Like this file (123.csv.read) has been processed already, so if same new file(123.csv) comes, I want to be skipped that file.
In my code below, after 123.csv file is processed, the folder has only one file 123.csv.read. break is not behaving as I was expecting.
context.Str = ((String)globalMap.get("tFileList_1_CURRENT_FILEPATH"));
String extension = context.Str.substring(context.Str.lastIndexOf(".") + 1);
if (extension.equals("read"))
{
break;
}
else {
System.out.println("Good File to Process");
}

public static void listFile(final String folder, final String ext) {
ExtFilter filter = new ExtFilter(ext);
File dir = new File(folder);
if (dir.isDirectory() == false) {
System.out.println("Directory does not exists : " + FindFileExtension.FILE_DIR);
return;
}
// list out all the file name and filter by the extension
String[] list = dir.list(filter);
if (list.length == 0) {
System.out.println("no files end with : " + ext);
return;
}
for (String file : list) {
String temp = new StringBuffer(FindFileExtension.FILE_DIR).append(File.separator).append(file).toString();
System.out.println("file : " + temp);
// do your stuff here this file is not processed
}
}
public static class ExtFilter implements FilenameFilter {
private String ext;
public ExtFilter(final String ext) {
this.ext = ext;
}
public boolean accept(final File dir, final String name) {
return (name.endsWith(this.ext));
}
}
You can do something like that,it might help you

You can try this:
For example 123.csv file came again, then you check this if exist in read folder
if(!new File(123.csv+".read").exist()) {
// if this file is not exist, then it means that this has not been processed
// process the file
} else {
// do some other staff
}
Edit: Or you can try this
public static void main(String[] args) throws Exception {
File dir = new File("your_path");
File[] processedFiles = dir.listFiles(new FileFilter() {
#Override
public boolean accept(File pathname) {
return pathname.getName().contains("read");
}
});
List<File> files = Arrays.asList(processedFiles);
File[] noneProcessedFiles = dir.listFiles(new FileFilter() {
#Override
public boolean accept(File pathname) {
return !pathname.getName().contains("read");
}
});
for (File file : noneProcessedFiles) {
if (!files.stream().findAny().get().getName().contains(file.getName())) {
// process the file....
System.out.println("Not found ... " + file.getName());
} else {
// do some other staff....
System.out.println("Fount the file");
}
}
}

getResources() return enumeration with only 1 object

I try to get the list of files from classpath (web-inf/classes/dir).
Enumeration<URL> en = getClass().getClassLoader().getResources("dir");
However, instead of elements in this folder, the only element is the folder itself. Is there a way I can get the list of files, or the only way is to access file one by one. Because when I try to refer to some file in the same folder, I can easily access its contents.

You can't browse a directory on the classpath recursively.
See this post: How to use ClassLoader.getResources() correctly?
If you know you're looking at a JAR file however, you might open it directly as an archive and browse the files.
Somebody came up with a useful answer for this question: Get all of the Classes in the Classpath
You could adapt this code to browse through all JAR files and directories on your classpath and apply some filter for your directory name yourself. The example will list all classes from the gnu.trove.* package:
import java.io.File;
import java.util.Enumeration;
import java.util.jar.JarEntry;
import java.util.jar.JarFile;
public class Test
{
public static void main(String[] args)
{
final String dirname = "gnu.trove.";
findClasses(new Visitor<String>() {
#Override
public boolean visit(String classname)
{
if (classname.startsWith(dirname)) {
System.out.println(classname);
}
return true;
}
});
}
public interface Visitor<T>
{
public boolean visit(T t);
}
public static void findClasses(Visitor<String> visitor)
{
String classpath = System.getProperty("java.class.path");
String[] paths = classpath.split(System.getProperty("path.separator"));
String javaHome = System.getProperty("java.home");
File file = new File(javaHome + File.separator + "lib");
if (file.exists()) {
findClasses(file, file, true, visitor);
}
for (String path : paths) {
file = new File(path);
if (file.exists()) {
findClasses(file, file, true, visitor);
}
}
}
private static boolean findClasses(File root, File file,
boolean includeJars, Visitor<String> visitor)
{
if (file.isDirectory()) {
for (File child : file.listFiles()) {
if (!findClasses(root, child, includeJars, visitor)) {
return false;
}
}
} else {
if (file.getName().toLowerCase().endsWith(".jar") && includeJars) {
JarFile jar = null;
try {
jar = new JarFile(file);
} catch (Exception ex) {
}
if (jar != null) {
Enumeration<JarEntry> entries = jar.entries();
while (entries.hasMoreElements()) {
JarEntry entry = entries.nextElement();
String name = entry.getName();
int extIndex = name.lastIndexOf(".class");
if (extIndex > 0) {
if (!visitor.visit(name.substring(0, extIndex)
.replace("/", "."))) {
return false;
}
}
}
}
} else if (file.getName().toLowerCase().endsWith(".class")) {
if (!visitor.visit(createClassName(root, file))) {
return false;
}
}
}
return true;
}
private static String createClassName(File root, File file)
{
StringBuffer sb = new StringBuffer();
String fileName = file.getName();
sb.append(fileName.substring(0, fileName.lastIndexOf(".class")));
file = file.getParentFile();
while (file != null && !file.equals(root)) {
sb.insert(0, '.').insert(0, file.getName());
file = file.getParentFile();
}
return sb.toString();
}
}
Here is some code for ignoring JAR files and just going through the files on directories on your classpath:
import java.io.File;
public class Test
{
public static void main(String[] args)
{
findClasses(new Visitor<String>() {
#Override
public boolean visit(String classname)
{
// apply your filtering here
System.out.println(classname);
return true;
}
});
}
public interface Visitor<T>
{
public boolean visit(T t);
}
public static void findClasses(Visitor<String> visitor)
{
String classpath = System.getProperty("java.class.path");
String[] paths = classpath.split(System.getProperty("path.separator"));
for (String path : paths) {
File file = new File(path);
// Ignore JAR files, just go through directories on the classpath
if (file.isFile()) {
continue;
}
findFiles(file, file, visitor);
}
}
private static boolean findFiles(File root, File file,
Visitor<String> visitor)
{
if (file.isDirectory()) {
for (File child : file.listFiles()) {
if (!findFiles(root, child, visitor)) {
return false;
}
}
} else {
if (!visitor.visit(createRelativePath(root, file))) {
return false;
}
}
return true;
}
private static String createRelativePath(File root, File file)
{
StringBuffer sb = new StringBuffer();
sb.append(file.getName());
file = file.getParentFile();
while (file != null && !file.equals(root)) {
sb.insert(0, '/').insert(0, file.getName());
file = file.getParentFile();
}
return sb.toString();
}
}

How to find out where a class originates from at runtime?

I'm having this weird issue where a class from some transitive dependency keeps showing up at runtime, shadowing a newer version of the class from the (correct) first level dependency, even though I thought I made sure that I excluded the older version from all other dependencies I declare (this is in a Maven/IntelliJ setup)
More specifically, at runtime the app fails with a NoClassDefFoundError, since during class loading a wrong version of the owning class is loaded, which has a field of a type that does not exist in newer versions of the library that class is defined in. To illustrate:
// lib.jar:wrong-version
class Owner {
private SomeType f;
}
// lib.jar:new-version
class Owner {
private OtherType f;
}
At runtime, the class loader finds a reference to the symbol Owner and attempts to load the version that has SomeType, which in return does not exist anymore. This is even though I excluded wrong-version where ever I could spot it.
I also ran mvn dependency:tree to see if the old version is still being pulled in somewhere, but it's not!
In order to further debug this, I was wondering if there is a way to find out where a class loader was reading a specific class from, i.e. which file? Is that possible? Or even better, build a list of origins where a certain symbol is defined, in case it's defined more than once?
Sorry if this is vague, but the problem is rather nebulous.

The following code will search the whole classpath for a particular class. With no arguments it will dump every class it finds and then you can pipe to grep or redirect to a file. It looks inside jars...
Usage: WhichClass or WhichClass package.name (note no .class)
Apologies for the lack of comments ...
import java.io.File;
import java.io.IOException;
import java.util.Enumeration;
import java.util.StringTokenizer;
import java.util.Vector;
import java.util.zip.ZipEntry;
import java.util.zip.ZipFile;
public final class WhichClass {
private WhichClass() {
}
static Vector<String> scratchVector;
public static void main(final String[] argv) {
Vector v;
if ((argv.length == 0) || "-all".equals(argv[0])) {
v = findClass(null);
} else {
v = findClass(argv[0]);
}
for (int i = 0; i < v.size(); i++) {
System.out.println(v.elementAt(i));
}
}
static String className(final String classFile) {
return classFile.replace('/', '.').substring(0, classFile.length() - ".class".length());
}
static Vector findClass(final String className) {
if (className != null) {
scratchVector = new Vector<String>(5);
} else {
scratchVector = new Vector<String>(5000);
}
findClassInPath(className, setupBootClassPath());
findClassInPath(className, setupClassPath());
return scratchVector;
}
static void findClassInPath(final String className, final StringTokenizer path) {
while (path.hasMoreTokens()) {
String pathElement = path.nextToken();
File pathFile = new File(pathElement);
if (pathFile.isDirectory()) {
try {
if (className != null) {
String pathName = className.replace('.', System.getProperty("file.separator").charAt(0)) + ".class";
findClassInPathElement(pathName, pathElement, pathFile);
} else {
findClassInPathElement(className, pathElement, pathFile);
}
} catch (IOException e) {
e.printStackTrace();
}
} else if (pathFile.exists()) {
try {
if (className != null) {
String pathName = className.replace('.', '/') + ".class";
ZipFile zipFile = new ZipFile(pathFile);
ZipEntry zipEntry = zipFile.getEntry(pathName);
if (zipEntry != null) {
scratchVector.addElement(pathFile + "(" + zipEntry + ")");
}
} else {
ZipFile zipFile = new ZipFile(pathFile);
Enumeration entries = zipFile.entries();
while (entries.hasMoreElements()) {
String entry = entries.nextElement().toString();
if (entry.endsWith(".class")) {
String name = className(entry);
scratchVector.addElement(pathFile + "(" + entry + ")");
}
}
}
} catch (IOException e) {
System.err.println(e + " while working on " + pathFile);
}
}
}
}
static void findClassInPathElement(final String pathName, final String pathElement, final File pathFile)
throws IOException {
String[] list = pathFile.list();
for (int i = 0; i < list.length; i++) {
File file = new File(pathFile, list[i]);
if (file.isDirectory()) {
findClassInPathElement(pathName, pathElement, file);
} else if (file.exists() && (file.length() != 0) && list[i].endsWith(".class")) {
String classFile = file.toString().substring(pathElement.length() + 1);
String name = className(classFile);
if (pathName != null) {
if (classFile.equals(pathName)) {
scratchVector.addElement(file.toString());
}
} else {
scratchVector.addElement(file.toString());
}
}
}
}
static StringTokenizer setupBootClassPath() {
String classPath = System.getProperty("sun.boot.class.path");
String separator = System.getProperty("path.separator");
return new StringTokenizer(classPath, separator);
}
static StringTokenizer setupClassPath() {
String classPath = System.getProperty("java.class.path");
String separator = System.getProperty("path.separator");
return new StringTokenizer(classPath, separator);
}
}

If you know the fully qualified name of the class, say somelib.Owner, you can try calling the following in your code:
public void foo() {
URL url = somelib.Owner.class.getClassLoader().getResource("somelib/Owner.class");
System.out.println(url);
}

How to copy all files with certain extensions from a directory and sub directories?

I know how to copy a file from one directory to another, what I would like help on is copy a file with .jpg or .doc extension.
So for example.
Copy all files from D:/Pictures/Holidays
Scanning all folders in the above path and transfer all jpg's to a destination provided.

This works, but the file 'copy(File file, File outputFolder)' method could be enhanced for large files:
package net.bpfurtado.copyfiles;
import java.io.File;
import java.io.FileFilter;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.InputStream;
import java.io.OutputStream;
public class CopyFilesFromType
{
public static void main(String[] args)
{
new CopyFilesFromType().copy("jpg", "C:\\Users\\BrunoFurtado\\Pictures", "c:/temp/photos");
}
private FileTypeOrFolderFilter filter = null;
private void copy(final String fileType, String fromPath, String outputPath)
{
filter = new FileTypeOrFolderFilter(fileType);
File currentFolder = new File(fromPath);
File outputFolder = new File(outputPath);
scanFolder(fileType, currentFolder, outputFolder);
}
private void scanFolder(final String fileType, File currentFolder, File outputFolder)
{
System.out.println("Scanning folder [" + currentFolder + "]...");
File[] files = currentFolder.listFiles(filter);
for (File file : files) {
if (file.isDirectory()) {
scanFolder(fileType, file, outputFolder);
} else {
copy(file, outputFolder);
}
}
}
private void copy(File file, File outputFolder)
{
try {
System.out.println("\tCopying [" + file + "] to folder [" + outputFolder + "]...");
InputStream input = new FileInputStream(file);
OutputStream out = new FileOutputStream(new File(outputFolder + File.separator + file.getName()));
byte data[] = new byte[input.available()];
input.read(data);
out.write(data);
out.flush();
out.close();
input.close();
} catch (Exception e) {
throw new RuntimeException(e);
}
}
private final class FileTypeOrFolderFilter implements FileFilter
{
private final String fileType;
private FileTypeOrFolderFilter(String fileType)
{
this.fileType = fileType;
}
public boolean accept(File pathname)
{
return pathname.getName().endsWith("." + fileType) || pathname.isDirectory();
}
}
}

Use a FileFilter when listing files.
In this case, the filter would select directories and any file type of interest.
Here is a quick example (crudely hacked out of another project) of gaining a list of types of files in a directory structure.
import java.io.*;
import java.util.ArrayList;
class ListFiles {
public static void populateFiles(File file, ArrayList<File> files, FileFilter filter) {
File[] all = file.listFiles(filter);
for (File f : all) {
if (f.isDirectory()) {
populateFiles(f,files,filter);
} else {
files.add(f);
}
}
}
public static void main(String[] args) throws Exception {
String[] types = {
"java",
"class"
};
FileFilter filter = new FileTypesFilter(types);
File f = new File("..");
ArrayList<File> files = new ArrayList<File>();
populateFiles(f, files, filter);
for (File file : files) {
System.out.println(file);
}
}
}
class FileTypesFilter implements FileFilter {
String[] types;
FileTypesFilter(String[] types) {
this.types = types;
}
public boolean accept(File f) {
if (f.isDirectory()) return true;
for (String type : types) {
if (f.getName().endsWith(type)) return true;
}
return false;
}
}

Use following file walker tree class to do that
static class TreeCopier implements FileVisitor<Path> {
private final Path source;
private final Path target;
private final boolean preserve;
private String []fileTypes;
TreeCopier(Path source, Path target, boolean preserve, String []types) {
this.source = source;
this.target = target;
this.preserve = preserve;
this.fileTypes = types;
}
#Override
public FileVisitResult preVisitDirectory(Path dir, BasicFileAttributes attrs) {
// before visiting entries in a directory we copy the directory
// (okay if directory already exists).
CopyOption[] options = (preserve)
? new CopyOption[]{COPY_ATTRIBUTES} : new CopyOption[0];
Path newdir = target.resolve(source.relativize(dir));
try {
Files.copy(dir, newdir, options);
} catch (FileAlreadyExistsException x) {
// ignore
} catch (IOException x) {
System.err.format("Unable to create: %s: %s%n", newdir, x);
return SKIP_SUBTREE;
}
return CONTINUE;
}
#Override
public FileVisitResult visitFile(Path file, BasicFileAttributes attrs) {
String fileName = file.toFile().getName();
boolean correctType = false;
for(String t: fileTypes) {
if(fileName.endsWith(t)){
correctType = true;
break;
}
}
if(correctType)
copyFile(file, target.resolve(source.relativize(file)), preserve);
return CONTINUE;
}
#Override
public FileVisitResult postVisitDirectory(Path dir, IOException exc) {
// fix up modification time of directory when done
if (exc == null && preserve) {
Path newdir = target.resolve(source.relativize(dir));
try {
FileTime time = Files.getLastModifiedTime(dir);
Files.setLastModifiedTime(newdir, time);
} catch (IOException x) {
System.err.format("Unable to copy all attributes to: %s: %s%n", newdir, x);
}
}
return CONTINUE;
}
#Override
public FileVisitResult visitFileFailed(Path file, IOException exc) {
if (exc instanceof FileSystemLoopException) {
System.err.println("cycle detected: " + file);
} else {
System.err.format("Unable to copy: %s: %s%n", file, exc);
}
return CONTINUE;
}
static void copyFile(Path source, Path target, boolean preserve) {
CopyOption[] options = (preserve)
? new CopyOption[]{COPY_ATTRIBUTES, REPLACE_EXISTING}
: new CopyOption[]{REPLACE_EXISTING};
if (Files.notExists(target)) {
try {
Files.copy(source, target, options);
} catch (IOException x) {
System.err.format("Unable to copy: %s: %s%n", source, x);
}
}
}
}
and call it using following two lines
String []types = {".java", ".form"};
TreeCopier tc = new TreeCopier(src.toPath(), dest.toPath(), false, types);
Files.walkFileTree(src.toPath(), tc);
.java and .form file types are not omitted to copy and passed as String array parameter, src.toPath() and dest.toPath() are source and destination paths, false is used to specify not to preserve previous files and overwrite them
if you want reverse that is to consider only these remove not and use as
if(!correctType)

Take a look at the listFiles methods from the File class:
Link 1
Link 2

You could try this code:
public class MyFiler implements FileNameFilter{
bool accept(File file, String name){
if(name.matches("*.jpg");
}
}
public void MassCopy(){
ArrayList<File> filesToCopy = new ArrayList<File>();
File sourceDirectory = new File("D:/Pictures/Holidays");
String[] toCopy = sourceDirectory.list(new MyFilter());
for(String file : toCopy){
copyFileToDestination(file);
}
}