Dynamically compiling and Running a Hadoop job from another Java File - java

I am trying to write a Java file that receives the source code of a MapReduce job, compiles it dynamically and runs the job on a Hadoop cluster. To reach this, I have written 3 methods called compile(), makeJAR() and run_Hadoop_Job(). Everything works fine with the compilation and creation of the JAR file. However, when the job is submitted to Hadoop, as soon as the job starts, it faces problem with finding required Mapper/Reducer classes and throws a ClassNotFoundException for both the Mapper_Class and Reducer_Class *(java.lang.ClassNotFoundException: reza.rCloud.Mapper_Reducer_Classes$Mapper_Class.class)* . I know that there should be something wrong with how I have referenced the required Mapper/Reducer classes but I was not able to figure it out after several. Any help/suggestion on how to solve the issue is highly appreciated.
Regarding the details of the project: I have a file called "rCloud_test/src/reza/Mapper_Reducer_Classes.java" that contains the source code for Mapper_Class and Reducer_Class. This file is ultimately received during the runtime but for now I copied the Hadoop WordCount example in it and store it locally in the same folder as my main class file: rCloud_test/src/reza/Platform2.java.
Here below you can see the main() method of the Platform2.java which is my main class for this project:
public static void main(String[] args){
System.out.println("Code Execution Started");
String className = "Mapper_Reducer_Classes";
Platform2 myPlatform = new Platform2();
//step 1: compile the received class file dynamically:
boolean compResult = myPlatform.compile(className);
System.out.println(className + ".java compilation result: "+compResult);
//step 2: make a JAR file out of the compiled file:
if (compResult) {
compResult = myPlatform.makeJAR("jar_file", myPlatform.compilation_Output_Folder);
System.out.println("JAR creation result: "+compResult);
}
//step 3: Now let's run the Hadoop job:
if (compResult) {
compResult = myPlatform.run_Hadoop_Job(className);
System.out.println("Running on Hadoop result: "+compResult);
}
The method that is causing me all the problems is the run_Hadoop_Job() which is as below:
private boolean run_Hadoop_Job(String className){
try{
System.out.println("*Starting to run the code on Hadoop...");
String[] argsTemp = { "project_test/input", "project_test/output" };
Configuration conf = new Configuration();
conf.set("fs.default.name", "hdfs://localhost:54310");
conf.set("mapred.job.tracker", "localhost:54311");
conf.set("mapred.jar", jar_Output_Folder + "/jar_file"+".jar");
conf.set("libjars", required_Execution_Classes);
//THIS IS WHERE IT CAN'T FIND THE MENTIONED CLASSES, ALTHOUGH THEY EXIST BOTH ON DISK
// AND IN THE CREATED JAR FILE:??????
System.out.println("Getting Mapper/Reducer package name: " +
Mapper_Reducer_Classes.class.getName());
conf.set("mapreduce.map.class", "reza.rCloud.Mapper_Reducer_Classes$Mapper_Class");
conf.set("mapreduce.reduce.class", "reza.rCloud.Mapper_Reducer_Classes$Reducer_Class");
Job job = new Job(conf, "Hadoop Example for dynamically and programmatically compiling-running a job");
job.setJarByClass(Platform2.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class);
FileInputFormat.addInputPath(job, new Path(argsTemp[0]));
FileSystem fs = FileSystem.get(conf);
Path out = new Path(argsTemp[1]);
fs.delete(out, true);
FileOutputFormat.setOutputPath(job, new Path(argsTemp[1]));
//job.submit();
System.out.println("*and now submitting the job to Hadoop...");
System.exit(job.waitForCompletion(true) ? 0 : 1);
System.out.println("Job Finished!");
} catch (Exception e) {
System.out.println("****************Exception!" );
e.printStackTrace();
return false;
}
return true;
}
if needed, here's the source code for the compile() method:
private boolean compile(String className) {
String fileToCompile = JOB_FOLDER + "/" +className+".java";
JavaCompiler compiler = ToolProvider.getSystemJavaCompiler();
FileOutputStream errorStream = null;
try{
errorStream = new FileOutputStream(JOB_FOLDER + "/logs/Errors.txt");
} catch(FileNotFoundException e){
//if problem creating the file, default wil be console
}
int compilationResult =
compiler.run( null, null, errorStream,
"-classpath", required_Compilation_Classes,
"-d", compilation_Output_Folder,
fileToCompile);
if (compilationResult == 0) {
//Compilation is successful:
return true;
} else {
//Compilation Failed:
return false;
}
}
and the source code for makeJAR() method:
private boolean makeJAR(String outputFileName, String inputDirectory) {
Manifest manifest = new Manifest();
manifest.getMainAttributes().put(Attributes.Name.MANIFEST_VERSION,
"1.0");
JarOutputStream target = null;
try {
target = new JarOutputStream(new FileOutputStream(
jar_Output_Folder+ "/"
+ outputFileName+".jar" ), manifest);
add(new File(inputDirectory), target);
} catch (Exception e) { return false; }
finally {
if (target != null)
try{
target.close();
} catch (Exception e) { return false; }
}
return true;
}
private void add(File source, JarOutputStream target) throws IOException
{
BufferedInputStream in = null;
try
{
if (source.isDirectory())
{
String name = source.getPath().replace("\\", "/");
if (!name.isEmpty())
{
if (!name.endsWith("/"))
name += "/";
JarEntry entry = new JarEntry(name);
entry.setTime(source.lastModified());
target.putNextEntry(entry);
target.closeEntry();
}
for (File nestedFile: source.listFiles())
add(nestedFile, target);
return;
}
JarEntry entry = new JarEntry(source.getPath().replace("\\", "/"));
entry.setTime(source.lastModified());
target.putNextEntry(entry);
in = new BufferedInputStream(new FileInputStream(source));
byte[] buffer = new byte[1024];
while (true)
{
int count = in.read(buffer);
if (count == -1)
break;
target.write(buffer, 0, count);
}
target.closeEntry();
}
finally
{
if (in != null)
in.close();
}
}
and finally the fixed parameters used for accessing the files:
private String JOB_FOLDER = "/Users/reza/My_Software/rCloud_test/src/reza/rCloud";
private String HADOOP_SOURCE_FOLDER = "/Users/reza/My_Software/hadoop-0.20.2";
private String required_Compilation_Classes = HADOOP_SOURCE_FOLDER + "/hadoop-0.20.2-core.jar";
private String required_Execution_Classes = required_Compilation_Classes + "," +
"/Users/reza/My_Software/ActorFoundry_dist_ver/lib/commons-cli-1.1.jar," +
"/Users/reza/My_Software/ActorFoundry_dist_ver/lib/commons-logging-1.1.1.jar";
public String compilation_Output_Folder = "/Users/reza/My_Software/rCloud_test/dyn_classes";
private String jar_Output_Folder = "/Users/reza/My_Software/rCloud_test/dyn_jar";
As a result of running the Platform2, the structure of the project on disk looks as below:
rCloud_test/classes/reza/rCloud/Platform2.class: contain the Platform2 class
rCloud_test/dyn_classes/reza/rCloud/ contains the classes for Mapper_Reducer_Classes.class, Mapper_Reducer_Classes$Mapper_Class.class, and Mapper_Reducer_Classes$Reducer_Class.class
rCloud_test/dyn_jar/jar_file.jar contains the created jar file
REVSED: here's the source code for the rCloud_test/src/reza/rCloud/Mapper_Reducer_Classes.java:
package reza.rCloud;
import java.io.IOException;
import java.lang.InterruptedException;
import java.util.StringTokenizer;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.util.GenericOptionsParser;
public class Mapper_Reducer_Classes {
/**
* The map class of WordCount.
*/
public static class Mapper_Class
extends Mapper<Object, Text, Text, IntWritable> {
private final static IntWritable one = new IntWritable(1);
private Text word = new Text();
public void map(Object key, Text value, Context context)
throws IOException, InterruptedException {
StringTokenizer itr = new StringTokenizer(value.toString());
while (itr.hasMoreTokens()) {
word.set(itr.nextToken());
context.write(word, one);
}
}
}
/**
* The reducer class of WordCount
*/
public static class Reducer_Class
extends Reducer<Text, IntWritable, Text, IntWritable> {
public void reduce(Text key, Iterable<IntWritable> values, Context context)
throws IOException, InterruptedException {
int sum = 0;
for (IntWritable value : values) {
sum += value.get();
}
context.write(key, new IntWritable(sum));
}
}
}

Try to set them by using the setClass() method :
conf.setClass("mapreduce.map.class",
Class.forName("reza.rCloud.Mapper_Reducer_Classes$Mapper_Class"),
Mapper.class);
conf.setClass("mapreduce.reduce.class",
Class.forName("reza.rCloud.Mapper_Reducer_Classes$Reducer_Class"),
Reducer.class);

Related

Getting java.lang.NoClassDefFoundError executing .war generated by Gradle

I have a .war generated by Gradle, in Eclipse it doesn't show any problems and the project generates the .war without errors, but when I run the war I get java.lang.NoClassDefFoundError: com/sun/grizzly/http/servlet/ServletAdapter:
INFO: Using classpath: file:/tmp/app_dir/apex/WEB-INF/lib/grizzly-comet-1.9.18-o.jar:file:/tmp/app_dir/apex/WEB-INF/lib/grizzly-http-servlet-1.9.18-o.jar:file:/tmp/app_dir/apex/WEB-INF/lib/grizzly-http-1.9.18-o.jar:file:/tmp/app_dir/apex/WEB-INF/lib/grizzly-utils-1.9.18-o.jar:file:/tmp/app_dir/apex/WEB-INF/lib/grizzly-rcm-1.9.18-o.jar:file:/tmp/app_dir/apex/WEB-INF/lib/commons-fileupload-1.2.1.jar:file:/tmp/app_dir/apex/WEB-INF/lib/xmlparserv2-11.2.0.jar:file:/tmp/app_dir/apex/WEB-INF/lib/grizzly-compat-1.9.18-o.jar:file:/tmp/app_dir/apex/WEB-INF/lib/commons-logging-1.1.1.jar:file:/tmp/app_dir/apex/WEB-INF/lib/javaee-web-api-7.0.jar:file:/tmp/app_dir/apex/WEB-INF/lib/sun.misc.BASE64Decoder.jar:file:/tmp/app_dir/apex/WEB-INF/lib/poi-3.6-20091214.jar:file:/tmp/app_dir/apex/WEB-INF/lib/web-ajp-3.1.jar:file:/tmp/app_dir/apex/WEB-INF/lib/je-4.0.103.jar:file:/tmp/app_dir/apex/WEB-INF/lib/grizzly-cometd-1.9.18-o.jar:file:/tmp/app_dir/apex/WEB-INF/lib/xdb-11.2.0.jar:file:/tmp/app_dir/apex/WEB-INF/lib/grizzly-servlet-deployer-1.9.18-o.jar:file:/tmp/app_dir/apex/WEB-INF/lib/grizzly-portunif-1.9.18-o.jar:file:/tmp/app_dir/apex/WEB-INF/lib/ucp.jar:file:/tmp/app_dir/apex/WEB-INF/lib/json-simple-1.1.1.jar:file:/tmp/app_dir/apex/WEB-INF/lib/ojdbc8.jar:file:/tmp/app_dir/apex/WEB-INF/lib/grizzly-servlet-webserver-1.9.18-o.jar:file:/tmp/app_dir/apex/WEB-INF/lib/ojmisc.jar:file:/tmp/app_dir/apex/WEB-INF/lib/grizzly-framework-1.9.18-o.jar:file:/tmp/app_dir/apex/WEB-INF/lib/apex.jar
SEVERE: com/sun/grizzly/http/servlet/ServletAdapter
java.lang.NoClassDefFoundError: com/sun/grizzly/http/servlet/ServletAdapter
at ____bootstrap.____Bootstrap.newServer(____Bootstrap.java:57)
at ____bootstrap.____Bootstrap.startServer(____Bootstrap.java:125)
at ____bootstrap.____Bootstrap._start(____Bootstrap.java:37)
at ____bootstrap.____Bootstrap.start(____Bootstrap.java:139)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at ____embedded.____EntryPoint.invoke(____EntryPoint.java:273)
at ____embedded.____EntryPoint.main(____EntryPoint.java:96)
Even though it seems I have this class available:
On build process I'm moving 2 directories from src/main/java/ to the root of the .war, maybe this is causing the issue?
I have to move those because this war can be executed by command line.
My gradle.build:
plugins {
// Apply the java-library plugin for API and implementation separation.
id 'war'
}
sourceCompatibility = 1.8
targetCompatibility = 1.8
repositories {
// Use Maven Central for resolving dependencies.
mavenCentral()
}
dependencies {
implementation fileTree('libs')
implementation 'com.sun.grizzly:grizzly-project:1.9.18b'
implementation 'com.sun.grizzly:grizzly-servlet-deployer:1.9.18-o'
}
war {
manifest {
attributes(
'Main-Class': '____embedded.____EntryPoint'
)
}
from 'build/classes/java/main/'
}
tasks.register('moveEntryPoint') {
description 'Copies entrypoint to root dir.'
doLast {
ant.move file: "${buildDir}/classes/java/main/____embedded/",
todir: "${buildDir}/____embedded/"
}
}
I run the war like this: java -jar app.war params
Code of where error happens:
package ____bootstrap;
import java.awt.Desktop;
import java.io.Console;
import java.io.File;
import java.io.PrintStream;
import java.net.URI;
import java.util.Arrays;
public class ____Bootstrap {
// ERROR HAPPENS HERE
protected Deployer newServer(Realm realm, String apexImages) {
return new Deployer(apexImages, realm);
}
private void info(String text) {
LOG.println("INFO: " + text);
}
private boolean isEmpty(char[] text) {
boolean isWhitespace = true;
for (int i = 0; i < text.length; i++) {
if (!Character.isWhitespace(text[i])) {
isWhitespace = false;
break;
}
}
return isWhitespace;
}
private void patchXmlLibraries() {
setIfAvailable("javax.xml.transform.TransformerFactory", "com.sun.org.apache.xalan.internal.xsltc.trax.TransformerFactoryImpl");
setIfAvailable("javax.xml.parsers.SAXParserFactory", "com.sun.org.apache.xerces.internal.jaxp.SAXParserFactoryImpl");
setIfAvailable("javax.xml.parsers.DocumentBuilderFactory", "com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderFactoryImpl");
}
private String[] readCredentials(Console c, String role, String defaultName) {
char[] first = new char[0];
try {
String username = c.readLine("Enter a username for the %s [%s]: ", new Object[] { role, defaultName }).trim();
if (username.isEmpty())
username = defaultName;
boolean match = false;
boolean empty = true;
while (!match) {
first = c.readPassword("Enter a password for %s: ", new Object[] { username });
if (isEmpty(first)) {
c.format("Password must not be empty or whitespace.%n", new Object[0]);
continue;
}
char[] second = c.readPassword("Confirm password for %s: ", new Object[] { username });
match = Arrays.equals(first, second);
if (!match)
c.format("Passwords do not match. Try Again.%n", new Object[0]);
Arrays.fill(second, ' ');
}
return new String[] { username, new String(first) };
} finally {
Arrays.fill(first, ' ');
}
}
private Realm realm(File securityCredentials) {
return FileCredentialsStore.realm("APEX Listener Manager", securityCredentials);
}
private void setIfAvailable(String property, String className) {
try {
Class.forName(className);
System.setProperty(property, className);
} catch (Exception e) {}
}
private void severe(Throwable t) {
LOG.println("SEVERE: " + t.getMessage());
t.printStackTrace(LOG);
}
private Deployer startServer(File war, Realm realm, Integer port, int ajpPort) {
String apexImages = System.getProperty("apex.images");
Deployer ws = newServer(realm, apexImages);
if (war != null) {
ws.setLocations(war.getAbsolutePath());
} else {
ws.setLocations("");
}
ws.setPort(port.intValue());
if (ajpPort > 0)
ws.setAJPPort(Integer.valueOf(ajpPort));
return ws;
}
public static void start(File home, File war, Integer port, Integer ajpPort) throws Exception {
____Bootstrap instance = new ____Bootstrap();
instance._start(home, war, port, ajpPort.intValue());
}
}
It generates some classpath stuff here:
package ____embedded;
import java.io.BufferedInputStream;
import java.io.BufferedOutputStream;
import java.io.Closeable;
import java.io.Console;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.FileOutputStream;
import java.io.FilenameFilter;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.io.PrintStream;
import java.lang.reflect.Method;
import java.net.MalformedURLException;
import java.net.URISyntaxException;
import java.net.URL;
import java.net.URLClassLoader;
import java.nio.Buffer;
import java.nio.ByteBuffer;
import java.nio.channels.Channels;
import java.nio.channels.ReadableByteChannel;
import java.nio.channels.WritableByteChannel;
import java.util.ArrayList;
import java.util.List;
import java.util.Properties;
import java.util.jar.JarEntry;
import java.util.jar.JarFile;
import java.util.jar.JarInputStream;
public class ____EntryPoint {
private static final String APEX_ERASE_PROP = "apex.erase";
static final String APEX_HOME_PROP = "apex.home";
private static final String APEX_IMAGES_PROP = "apex.images";
static final String APEX_PORT_PROP = "apex.port";
static final String APEX_AJP_PORT_PROP = "apex.ajp";
private static final String APEX_PROPERTIES = "apex.properties";
private static final String BOOTSTRAP = "____bootstrap.____Bootstrap";
private static final int BUFFER_SIZE = 65536;
private static final String DEFAULT_PORT = "8080";
public static void main(String[] args) {
try {
if (args.length == 1 && args[0].equals("--help")) {
usage(System.out);
} else {
File war = file(war());
info("Starting: " + war.getAbsolutePath() + "\n See: 'java -jar " + war.getName() + " --help' for full range of configuration options");
String portProp = System.getProperty("apex.port");
if (portProp == null) {
System.setProperty("apex.port", "8080");
portProp = "8080";
}
Integer port = Integer.valueOf(Integer.parseInt(portProp));
File home = DEFAULT_HOME;
String apexHome = System.getProperty("apex.home");
if (apexHome != null)
home = new File(apexHome);
Properties props = apexProperties(home);
if (!props.isEmpty())
for (Object p : props.keySet()) {
String prop = p.toString();
String value = props.getProperty(prop);
String existing = System.getProperty(prop);
if (value != null && existing == null)
System.setProperty(prop, value);
}
String ajpPortText = System.getProperty("apex.ajp");
int ajpPort = -1;
if (ajpPortText != null)
ajpPort = Integer.parseInt(ajpPortText);
String erase = System.getProperty("apex.erase");
if (Boolean.parseBoolean(erase)) {
info("Erasing: " + home.getAbsolutePath());
delete(home);
}
info("Extracting to: " + home.getAbsolutePath());
File warFolder = new File(home, "apex");
File warLib = new File(warFolder, "WEB-INF/lib");
if (warFolder.exists())
removeOldFiles(war, warFolder);
extractWar(war(), warFolder);
apexImages(home);
ClassLoader classloader = classloader(new File[] { new File(warFolder, PKG), warLib });
Thread.currentThread().setContextClassLoader(classloader);
invoke(classloader, "____bootstrap.____Bootstrap", "start", new Object[] { home, warFolder, port, Integer.valueOf(ajpPort) });
}
} catch (Throwable t) {
severe(t);
}
}
private static void apexImages(File home) throws IOException {
String apexImages = System.getProperty("apex.images");
Console c = System.console();
File apexProperties = new File(home, "apex.properties");
if (apexImages == null && !apexProperties.exists() && c != null) {
boolean exists = false;
while (!exists) {
apexImages = c.readLine("Enter the path to the directory containing the APEX static resources\n\t Example: /Users/myuser/apex/images \n\t or press Enter to skip: ", new Object[0]).trim();
if (apexImages.isEmpty()) {
String again = c.readLine("Do you want to be prompted to specifiy this path next time? y/n [y]: ", new Object[0]).trim();
if ("n".equalsIgnoreCase(again)) {
apexProperties.getParentFile().getAbsoluteFile().mkdirs();
apexProperties.createNewFile();
}
apexImages = null;
break;
}
File f = new File(apexImages);
exists = f.exists();
if (!exists)
c.format("%s does not exist or is not accessible, please enter another path:%n", new Object[] { f.getAbsolutePath() });
}
}
if (apexImages != null) {
System.setProperty("apex.images", apexImages);
Properties props = apexProperties(home);
String existing = props.getProperty("apex.images");
if (existing == null || !existing.equals(apexImages)) {
props.setProperty("apex.images", apexImages);
OutputStream out = null;
try {
out = new FileOutputStream(apexProperties);
props.store(out, (String)null);
} finally {
close(new Closeable[] { out });
}
}
}
}
static Properties apexProperties(File home) throws FileNotFoundException, IOException {
Properties props = new Properties();
File apexProperties = new File(home, "apex.properties");
if (apexProperties.exists()) {
InputStream propsStream = null;
try {
propsStream = new BufferedInputStream(new FileInputStream(apexProperties));
props.load(propsStream);
} finally {
close(new Closeable[] { propsStream });
}
}
return props;
}
private static ClassLoader classloader(File... folders) throws MalformedURLException {
ClassLoader scope = new URLClassLoader(jars(folders));
return scope;
}
private static void close(Closeable... items) {
for (Closeable close : items) {
if (close != null)
try {
close.close();
} catch (IOException e) {
throw new RuntimeException(e);
}
}
}
private static final boolean delete(File file) {
File failed = null;
if (file.isDirectory())
for (File f : file.listFiles()) {
if (!delete(f)) {
failed = f;
break;
}
}
if (failed == null &&
!file.delete())
failed = file;
return (failed == null);
}
private static long drain(InputStream is, OutputStream os) throws IOException {
if (is != null && os != null) {
ReadableByteChannel source = Channels.newChannel(is);
WritableByteChannel destination = Channels.newChannel(os);
return drain(source, destination);
}
return -1L;
}
private static long drain(ReadableByteChannel source, WritableByteChannel destination) throws IOException {
long length = 0L;
ByteBuffer buffer = ByteBuffer.allocate(65536);
while (source.read(buffer) != -1) {
((Buffer) buffer).flip();
length += buffer.limit();
destination.write(buffer);
buffer.compact();
}
((Buffer) buffer).flip();
while (buffer.hasRemaining())
destination.write(buffer);
return length;
}
private static boolean exists(JarFile jar, File root, File child) {
String path = child.getAbsolutePath().substring(root.getAbsolutePath().length() + 1);
return (jar.getEntry(path) != null);
}
private static void extract(InputStream in, File folder) throws IOException {
JarInputStream jar = null;
OutputStream out = null;
try {
jar = new JarInputStream(in);
JarEntry entry;
while ((entry = jar.getNextJarEntry()) != null) {
File target = new File(folder, entry.getName());
if (isNewer(entry, target)) {
if (entry.isDirectory()) {
target.mkdirs();
continue;
}
target.getParentFile().mkdirs();
out = new BufferedOutputStream(new FileOutputStream(target));
drain(jar, out);
close(new Closeable[] { out });
out = null;
}
}
} finally {
close(new Closeable[] { out, jar, in });
}
}
private static void extractWar(URL war, File folder) throws IOException {
InputStream in = null;
try {
in = war.openStream();
extract(in, folder);
} finally {
close(new Closeable[] { in });
}
}
private static File file(URL url) {
File file;
try {
file = new File(url.toURI());
} catch (URISyntaxException e) {
file = new File(url.getPath());
}
return file;
}
private static void info(String text) {
LOG.println("INFO: " + text);
}
private static void invoke(ClassLoader scope, String clazz, String method, Object... args) throws Exception {
Class<?>[] types = new Class[args.length];
for (int i = 0; i < args.length; i++)
types[i] = args[i].getClass();
Class<?> c = scope.loadClass(clazz);
Method m = c.getMethod(method, types);
m.invoke(null, args);
}
private static boolean isNewer(JarEntry entry, File target) {
boolean exists = target.exists();
long jarModified = entry.getTime();
long targetModified = target.lastModified();
return (!exists || jarModified == -1L || jarModified > targetModified);
}
private static URL[] jars(File... folders) throws MalformedURLException {
StringBuilder classpath = new StringBuilder();
List<URL> allUrls = new ArrayList<URL>();
for (File folder : folders) {
if (folder.exists()) {
URL[] urls = jars(folder);
if (urls.length == 0) {
allUrls.add(folder.toURI().toURL());
} else {
for (URL url : urls) {
allUrls.add(url);
classpath.append(url.toExternalForm());
classpath.append(":");
}
}
}
}
info("Using classpath: " + classpath.toString());
return allUrls.<URL>toArray(new URL[allUrls.size()]);
}
private static URL[] jars(File folder) throws MalformedURLException {
File[] jars = folder.listFiles(new FilenameFilter() {
public boolean accept(File dir, String name) {
name = name.toLowerCase();
return (name.endsWith(".zip") || name.endsWith(".jar"));
}
});
URL[] urls = new URL[(jars == null) ? 0 : jars.length];
StringBuilder classpath = new StringBuilder();
for (int i = 0; i < urls.length; i++) {
urls[i] = jars[i].toURI().toURL();
classpath.append(urls[i].toExternalForm());
if (i < jars.length)
classpath.append(':');
}
return urls;
}
private static void removeOldFiles(File war, File folder) throws IOException {
JarFile jar = new JarFile(war);
File root = folder;
removeOldFiles(jar, root, folder);
}
private static void removeOldFiles(JarFile jar, File root, File folder) {
File[] children = folder.listFiles();
for (File child : children) {
if (child.isDirectory())
removeOldFiles(jar, root, child);
if (!exists(jar, root, child))
child.delete();
}
}
private static void severe(Throwable t) {
LOG.println("SEVERE: " + t.getMessage());
t.printStackTrace(LOG);
}
private static void usage(PrintStream out) {
out.println("java [options] -jar " + file(war()).getName() + " [--help]");
out.println(" Options: ");
out.println(" -Dapex.home=/path/to/apex : Path to the folder used to store the");
out.println(" web container runtime, defaults to:");
out.println(" ${java.io.tmpdir}/apex ");
out.println(" -Dapex.port=nnnn : HTTP listen port, default 8080");
out.println(" -Dapex.ajp=nnnn : AJP (mod_jk) listen port, default none");
out.println(" If an AJP Port is specified then HTTP access is disabled");
out.println(" -Dapex.images=/images/location : Path to the folder containing static");
out.println(" resources required by APEX");
out.println(" -Dapex.erase=true : Erase the contents of ${apex.home} ");
out.println(" before launching");
out.println(" --help : Print this usage message");
}
private static URL war() {
return CLASS.getProtectionDomain().getCodeSource().getLocation();
}
private static final Class<____EntryPoint> CLASS = ____EntryPoint.class;
private static final File DEFAULT_HOME = new File(System.getProperty("java.io.tmpdir") + "/apex");
private static final PrintStream LOG = System.out;
private static final String PKG = CLASS.getPackage().getName();
private static final String START = "start";
}
As I understand I get an error when it tries to call a method on a class that imports the Class it can't find.

javax.tools.JavaCompiler compile into byte[] [duplicate]

This question already has answers here:
How do you dynamically compile and load external java classes? [duplicate]
(2 answers)
Closed 5 years ago.
I'm using the JavaCompiler from the javax.tools package (JDK 1.7) to compile some stuff on the fly, like this:
compiler.run(null, null, "-cp", paths, "path/to/my/file.java");
It works but I would like to do it all in memory (e.g. pass a string with the code, not the source file, and get the byte code back not a .class file). I found that extending the InputStream and OutputStream parameters is no use since it's probably just the same as in the console. Do you know a way to make the run method work like this? Or do you know a confirmed way to do this with the getTask() method? (extending the FileManager looks easy but isn't that easy :)
I've run the above code in Mac OS Java 7. None of them works. So i wrote one
https://github.com/trung/InMemoryJavaCompiler
StringBuilder source = new StringBuilder()
.append("package org.mdkt;\n")
.append("public class HelloClass {\n")
.append(" public String hello() { return \"hello\"; }")
.append("}");
Class<?> helloClass = InMemoryJavaCompiler.compile("org.mdkt.HelloClass", source.toString());
I think this here might be of help it basically shows how to compile Java source from memory (the string is located in the class).
It uses the PrinterWriter and StringWriter to write the source to a String/in memory and then uses the JavaCompiler class (since JDK 6) to compile and run the program:
import javax.tools.Diagnostic;
import javax.tools.DiagnosticCollector;
import javax.tools.JavaCompiler;
import javax.tools.JavaCompiler.CompilationTask;
import javax.tools.JavaFileObject;
import javax.tools.SimpleJavaFileObject;
import javax.tools.ToolProvider;
import java.io.File;
import java.io.IOException;
import java.io.PrintWriter;
import java.io.StringWriter;
import java.lang.reflect.InvocationTargetException;
import java.net.URI;
import java.net.URL;
import java.net.URLClassLoader;
import java.util.Arrays;
public class CompileSourceInMemory {
public static void main(String args[]) throws IOException {
JavaCompiler compiler = ToolProvider.getSystemJavaCompiler();
DiagnosticCollector<JavaFileObject> diagnostics = new DiagnosticCollector<JavaFileObject>();
StringWriter writer = new StringWriter();
PrintWriter out = new PrintWriter(writer);
out.println("public class HelloWorld {");
out.println(" public static void main(String args[]) {");
out.println(" System.out.println(\"This is in another java file\");");
out.println(" }");
out.println("}");
out.close();
JavaFileObject file = new JavaSourceFromString("HelloWorld", writer.toString());
Iterable<? extends JavaFileObject> compilationUnits = Arrays.asList(file);
CompilationTask task = compiler.getTask(null, null, diagnostics, null, null, compilationUnits);
boolean success = task.call();
for (Diagnostic diagnostic : diagnostics.getDiagnostics()) {
System.out.println(diagnostic.getCode());
System.out.println(diagnostic.getKind());
System.out.println(diagnostic.getPosition());
System.out.println(diagnostic.getStartPosition());
System.out.println(diagnostic.getEndPosition());
System.out.println(diagnostic.getSource());
System.out.println(diagnostic.getMessage(null));
}
System.out.println("Success: " + success);
if (success) {
try {
URLClassLoader classLoader = URLClassLoader.newInstance(new URL[] { new File("").toURI().toURL() });
Class.forName("HelloWorld", true, classLoader).getDeclaredMethod("main", new Class[] { String[].class }).invoke(null, new Object[] { null });
} catch (ClassNotFoundException e) {
System.err.println("Class not found: " + e);
} catch (NoSuchMethodException e) {
System.err.println("No such method: " + e);
} catch (IllegalAccessException e) {
System.err.println("Illegal access: " + e);
} catch (InvocationTargetException e) {
System.err.println("Invocation target: " + e);
}
}
}
}
class JavaSourceFromString extends SimpleJavaFileObject {
final String code;
JavaSourceFromString(String name, String code) {
super(URI.create("string:///" + name.replace('.','/') + Kind.SOURCE.extension),Kind.SOURCE);
this.code = code;
}
#Override
public CharSequence getCharContent(boolean ignoreEncodingErrors) {
return code;
}
}
If you have a look at the reference link you will find a few more other examples too
Reference:
Compiling from Memory
This is a class that compiles entirely in memory.
I've taken (almost) the entirety of this from http://javapracs.blogspot.de/2011/06/dynamic-in-memory-compilation-using.html by Rekha Kumari (June 2011). Though this version is more than 100 lines shorter and has considerably more features (but no docs:P).
It can compile multiple classes at once, which is the only way to compile classes that depend on each other. If you wonder about the class "CompilerFeedback": I was working on a tiny Java IDE for coding-games where I needed that. I'm including it here because I assume that you want to do something with this compiler, and the predigestion might help with that. (I realize that some of the code in the CompilerFeedback class is complete crap. It was recycled from a years old attempt.
There's also a utility method, not needed for compilation, that derives the full class name from a class' source code (incl. package name, if it's provided). Very useful for calling the compiler, which does need this information.
DEMO CLASS:
import java.util.ArrayList;
import java.util.List;
public class Demo {
public static void main(final String[] args) {
final InMemoryCompiler.IMCSourceCode cls1source;
final InMemoryCompiler.IMCSourceCode cls2source;
final StringBuilder sb = new StringBuilder();
sb.append("package toast;\n");
sb.append("public class DynaClass {\n");
sb.append(" public static void main(final String[] args) {");
sb.append(" System.out.println(\"Based massively on the work of Rekha Kumari, http://javapracs.blogspot.de/2011/06/dynamic-in-memory-compilation-using.html\");\n");
sb.append(" System.out.println(\"This is the main method speaking.\");\n");
sb.append(" System.out.println(\"Args: \" + java.util.Arrays.toString(args));\n");
sb.append(" final Test test = new Test();\n");
sb.append(" }\n");
sb.append(" public String toString() {\n");
sb.append(" return \"Hello, I am \" + ");
sb.append("this.getClass().getSimpleName();\n");
sb.append(" }\n");
sb.append("}\n");
cls1source = new InMemoryCompiler.IMCSourceCode("toast.DynaClass", sb.toString());
sb.setLength(0);
sb.append("package toast;\n");
sb.append("public class Test {\n");
sb.append(" public Test() {\n");
sb.append(" System.out.println(\"class Test constructor reporting in.\");\n");
sb.append(" System.out.println(new DynaClass());\n");
sb.append(" }\n");
sb.append("}\n");
cls2source = new InMemoryCompiler.IMCSourceCode("toast.Test", sb.toString());
final List<InMemoryCompiler.IMCSourceCode> classSources = new ArrayList<>();
classSources.add(cls1source);
classSources.add(cls2source);
final InMemoryCompiler uCompiler = new InMemoryCompiler(classSources);
final CompilerFeedback compilerFeedback = uCompiler.compile();
System.out.println("\n\nCOMPILER FEEDBACK: " + compilerFeedback);
if (compilerFeedback != null && compilerFeedback.success) {
try {
System.out.println("\nTOSTRING DEMO:");
uCompiler.runToString(cls1source.fullClassName);
} catch (Exception e) {
e.printStackTrace();
}
try {
System.out.println("\nMAIN DEMO:");
uCompiler.runMain(cls1source.fullClassName, new String[] { "test1", "test2" });
} catch (Exception e) {
e.printStackTrace();
}
}
System.exit(0);
}
}
COMPILER CLASS:
import javax.tools.*;
import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.io.OutputStream;
import java.lang.reflect.InvocationTargetException;
import java.lang.reflect.Method;
import java.net.URI;
import java.security.SecureClassLoader;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
/**
* MASSIVELY based on http://javapracs.blogspot.de/2011/06/dynamic-in-memory-compilation-using.html by Rekha Kumari
* (June 2011)
*/
final public class InMemoryCompiler {
final public static class IMCSourceCode {
final public String fullClassName;
final public String sourceCode;
/**
* #param fullClassName Full name of the class that will be compiled. If the class should be in some package,
* fullName should contain it too, for example: "testpackage.DynaClass"
* #param sourceCode the source code
*/
public IMCSourceCode(final String fullClassName, final String sourceCode) {
this.fullClassName = fullClassName;
this.sourceCode = sourceCode;
}
}
final public boolean valid;
final private List<IMCSourceCode> classSourceCodes;
final private JavaFileManager fileManager;
public InMemoryCompiler(final List<IMCSourceCode> classSourceCodes) {
this.classSourceCodes = classSourceCodes;
final JavaCompiler compiler = ToolProvider.getSystemJavaCompiler();
if (compiler == null) {
fileManager = null;
valid = false;
System.err.println("ToolProvider.getSystemJavaCompiler() returned null! This program needs to be run on a system with an installed JDK.");
return;
}
valid = true;
fileManager = new ForwardingJavaFileManager<JavaFileManager>(compiler.getStandardFileManager(null, null, null)) {
final private Map<String, ByteArrayOutputStream> byteStreams = new HashMap<>();
#Override
public ClassLoader getClassLoader(final Location location) {
return new SecureClassLoader() {
#Override
protected Class<?> findClass(final String className) throws ClassNotFoundException {
final ByteArrayOutputStream bos = byteStreams.get(className);
if (bos == null) {
return null;
}
final byte[] b = bos.toByteArray();
return super.defineClass(className, b, 0, b.length);
}
};
}
#Override
public JavaFileObject getJavaFileForOutput(final Location location, final String className, final JavaFileObject.Kind kind, final FileObject sibling) throws IOException {
return new SimpleJavaFileObject(URI.create("string:///" + className.replace('.', '/') + kind.extension), kind) {
#Override
public OutputStream openOutputStream() throws IOException {
ByteArrayOutputStream bos = byteStreams.get(className);
if (bos == null) {
bos = new ByteArrayOutputStream();
byteStreams.put(className, bos);
}
return bos;
}
};
}
};
}
public CompilerFeedback compile() {
if (!valid) {
return null;
}
final List<JavaFileObject> files = new ArrayList<>();
for (IMCSourceCode classSourceCode : classSourceCodes) {
URI uri = null;
try {
uri = URI.create("string:///" + classSourceCode.fullClassName.replace('.', '/') + JavaFileObject.Kind.SOURCE.extension);
} catch (Exception e) {
// e.printStackTrace();
}
if (uri != null) {
final SimpleJavaFileObject sjfo = new SimpleJavaFileObject(uri, JavaFileObject.Kind.SOURCE) {
#Override
public CharSequence getCharContent(final boolean ignoreEncodingErrors) {
return classSourceCode.sourceCode;
}
};
files.add(sjfo);
}
}
final DiagnosticCollector<JavaFileObject> diagnostics = new DiagnosticCollector<>();
final JavaCompiler compiler = ToolProvider.getSystemJavaCompiler();
if (files.size() > 0) {
final JavaCompiler.CompilationTask task = compiler.getTask(null, fileManager, diagnostics, null, null, files);
return new CompilerFeedback(task.call(), diagnostics);
} else {
return null;
}
}
public void runToString(final String className) throws InstantiationException, IllegalAccessException, ClassNotFoundException {
if (!valid) {
return;
}
final Class<?> theClass = getCompiledClass(className);
final Object instance = theClass.newInstance();
System.out.println(instance);
}
public void runMain(final String className, final String[] args) throws IllegalAccessException, ClassNotFoundException, NoSuchMethodException, InvocationTargetException {
if (!valid) {
return;
}
final Class<?> theClass = getCompiledClass(className);
final Method mainMethod = theClass.getDeclaredMethod("main", String[].class);
mainMethod.invoke(null, new Object[] { args });
}
public Class<?> getCompiledClass(final String className) throws ClassNotFoundException {
if (!valid) {
throw new IllegalStateException("InMemoryCompiler instance not usable because ToolProvider.getSystemJavaCompiler() returned null: No JDK installed.");
}
final ClassLoader classLoader = fileManager.getClassLoader(null);
final Class<?> ret = classLoader.loadClass(className);
if (ret == null) {
throw new ClassNotFoundException("Class returned by ClassLoader was null!");
}
return ret;
}
}
COMPILERFEEDBACK CLASS:
import javax.tools.Diagnostic;
import javax.tools.DiagnosticCollector;
import javax.tools.JavaFileObject;
import javax.tools.SimpleJavaFileObject;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
import java.util.Locale;
final public class CompilerFeedback {
final public boolean success;
final public List<CompilerMessage> messages = new ArrayList<>();
public CompilerFeedback(final Boolean success, final DiagnosticCollector<JavaFileObject> diagnostics) {
this.success = success != null && success;
for (Diagnostic<? extends JavaFileObject> diagnostic : diagnostics.getDiagnostics()) {
messages.add(new CompilerMessage(diagnostic));
}
}
public String toString() {
final StringBuilder sb = new StringBuilder();
sb.append("SUCCESS: ").append(success).append('\n');
final int iTop = messages.size();
for (int i = 0; i < iTop; i++) {
sb.append("\n[MESSAGE ").append(i + 1).append(" OF ").append(iTop).append("]\n\n");
// sb.append(messages.get(i).toString()).append("\n");
// sb.append(messages.get(i).toStringForList()).append("\n");
sb.append(messages.get(i).toStringForDebugging()).append("\n");
}
return sb.toString();
}
final public static class CompilerMessage {
final public Diagnostic<? extends JavaFileObject> compilerInfo;
final public String typeOfProblem;
final public String typeOfProblem_forDebugging;
final public String multiLineMessage;
final public int lineNumber;
final public int columnNumber;
final public int textHighlightPos_lineStart;
final public int textHighlightPos_problemStart;
final public int textHighlightPos_problemEnd;
final public String sourceCode;
final public String codeOfConcern;
final public String codeOfConcernLong;
CompilerMessage(final Diagnostic<? extends JavaFileObject> diagnostic) {
final JavaFileObject sourceFileObject = diagnostic.getSource();
String sourceCodePreliminary = null;
if (sourceFileObject instanceof SimpleJavaFileObject) {
final SimpleJavaFileObject simpleSourceFileObject = (SimpleJavaFileObject) sourceFileObject;
try {
final CharSequence charSequence = simpleSourceFileObject.getCharContent(false);
sourceCodePreliminary = charSequence.toString();
} catch (IOException e) {
e.printStackTrace();
}
}
if (sourceCodePreliminary == null) {
sourceCode = "[SOURCE CODE UNAVAILABLE]";
} else {
sourceCode = sourceCodePreliminary;
}
compilerInfo = diagnostic;
typeOfProblem = diagnostic.getKind().name();
typeOfProblem_forDebugging = "toString() = " + diagnostic.getKind().toString() + "; name() = " + typeOfProblem;
lineNumber = (int) compilerInfo.getLineNumber();
columnNumber = (int) compilerInfo.getColumnNumber();
final int sourceLen = sourceCode.length();
textHighlightPos_lineStart = (int) Math.min(Math.max(0, diagnostic.getStartPosition()), sourceLen);
textHighlightPos_problemStart = (int) Math.min(Math.max(0, diagnostic.getPosition()), sourceLen);
textHighlightPos_problemEnd = (int) Math.min(Math.max(0, diagnostic.getEndPosition()), sourceLen);
final StringBuilder reformattedMessage = new StringBuilder();
final String message = diagnostic.getMessage(Locale.US);
final int messageCutOffPosition = message.indexOf("location:");
final String[] messageParts;
if (messageCutOffPosition >= 0) {
messageParts = message.substring(0, messageCutOffPosition).split("\n");
} else {
messageParts = message.split("\n");
}
for (String s : messageParts) {
String s2 = s.trim();
if (s2.length() > 0) {
boolean lengthChanged;
do {
final int lBeforeReplace = s2.length();
s2 = s2.replace(" ", " ");
lengthChanged = (s2.length() != lBeforeReplace);
} while (lengthChanged);
reformattedMessage.append(s2).append("\n");
}
}
codeOfConcern = sourceCode.substring(textHighlightPos_problemStart, textHighlightPos_problemEnd);
codeOfConcernLong = sourceCode.substring(textHighlightPos_lineStart, textHighlightPos_problemEnd);
if (!codeOfConcern.isEmpty()) {
reformattedMessage.append("Code of concern: \"").append(codeOfConcern).append('\"');
}
multiLineMessage = reformattedMessage.toString();
}
public String toStringForList() {
if (compilerInfo == null) {
return "No compiler!";
} else {
return compilerInfo.getCode();
}
}
public String toStringForDebugging() {
final StringBuilder ret = new StringBuilder();
ret.append("Type of problem: ").append(typeOfProblem_forDebugging).append("\n\n");
ret.append("Message:\n").append(multiLineMessage).append("\n\n");
ret.append(compilerInfo.getCode()).append("\n\n");
ret.append("line number: ").append(lineNumber).append("\n");
ret.append("column number: ").append(columnNumber).append("\n");
ret.append("textHighlightPos_lineStart: ").append(textHighlightPos_lineStart).append("\n");
ret.append("textHighlightPos_problemStart: ").append(textHighlightPos_problemStart).append("\n");
ret.append("textHighlightPos_problemEnd: ").append(textHighlightPos_problemEnd).append("\n");
return ret.toString();
}
#Override
public String toString() {
// return compilerInfo.getMessage(Locale.US);
return typeOfProblem + ": " + multiLineMessage + "\n";
}
}
}
UTILITY METHOD (Not needed for the three classes further up.):
final public static String PREFIX_CLASSNAME = "class ";
final public static String PREFIX_PACKAGENAME = "package ";
final public static String CHARSET_JAVAKEYWORDENDERS = " \n[](){}<>;,\"\\/*+-=%!&?#:";
/**
* #return e.g. "com.dreamspacepresident.TestClass" if the source's first root level "class" (I'm talking about {}
* hierarchy.) is named "TestClass", and if the "package" name is "com.dreamspacepresident". Null is returned if
* sourceCode is null or does not provide a class name. (Mind that the parsing is done in a quite crappy way.)
*/
public static String deriveFullClassNameFromSource(final String sourceCode) {
if (sourceCode == null) {
return null;
}
final int firstBr = sourceCode.indexOf('{');
if (firstBr >= 0) {
// DETERMINE CLASS NAME
final int firstClass = sourceCode.indexOf(PREFIX_CLASSNAME);
if (firstClass < firstBr) {
String className = sourceCode.substring(firstClass + PREFIX_CLASSNAME.length(), firstBr).trim();
final int classNameEnd = indexOfAnyOfThese(className, CHARSET_JAVAKEYWORDENDERS);
if (classNameEnd >= 0) {
className = className.substring(0, classNameEnd);
}
if (!className.isEmpty()) {
// DETERMINE PACKAGE NAME
String packageName = null;
final int firstPackage = sourceCode.indexOf(PREFIX_PACKAGENAME);
if (firstPackage >= 0 && firstPackage < firstBr && firstPackage < firstClass) {
packageName = sourceCode.substring(firstPackage + PREFIX_PACKAGENAME.length(), firstBr).trim();
final int packageNameEnd = indexOfAnyOfThese(packageName, CHARSET_JAVAKEYWORDENDERS);
if (packageNameEnd >= 0) {
packageName = packageName.substring(0, packageNameEnd);
}
}
return (packageName != null && !packageName.isEmpty() ? packageName + "." : "") + className;
}
}
}
return null;
}
/**
* Looks for the first occurrence of ANY of the given characters, which is easier than using a bunch of
* String.indexOf() calls.
*
* #return -1 if not found, otherwise the String index of the first hit
*/
public static int indexOfAnyOfThese(final String text, final String characters) {
if (text != null && !text.isEmpty() && characters != null && !characters.isEmpty()) {
final int lenT = text.length();
final int lenC = characters.length();
for (int i = 0; i < lenT; i++) {
final char c = text.charAt(i);
for (int ii = 0; ii < lenC; ii++) {
if (c == characters.charAt(ii)) {
return i;
}
}
}
}
return -1;
}
I wrote a library to do this a few years ago. It takes a String which can contain nested classes, compiles them and optionally loads them into the current class loader (so you don't need an additional class loader) If the JVM is running in debug mode it will write the generated code to a file so you can step through your generated code.
http://vanillajava.blogspot.co.uk/2010_11_01_archive.html
To paraphrase the example from erolagnab you can do
StringBuilder sourceCode = new StringBuilder();
sourceCode.append("package org.mdkt;\n")
.append("public class HelloClass {\n")
.append(" public String hello() { return \"hello\"; }")
.append("}");
Class<?> helloClass = CACHED_COMPILER.compile("org.mdkt.HelloClass",
sourceCode.toString());
Update, the source is available here https://github.com/OpenHFT/Java-Runtime-Compiler
And you can obtain the latest build via maven http://search.maven.org/#browse%7C842970587
A longer example.
// this writes the file to disk only when debugging is enabled.
CachedCompiler cc = CompilerUtils.DEBUGGING ?
new CachedCompiler(new File(parent, "src/test/java"), new File(parent, "target/compiled")) :
CompilerUtils.CACHED_COMPILER;
String text = "generated test " + new Date();
Class fooBarTeeClass = cc.loadFromJava("eg.FooBarTee", "package eg;\n" +
'\n' +
"import eg.components.BarImpl;\n" +
"import eg.components.TeeImpl;\n" +
"import eg.components.Foo;\n" +
'\n' +
"public class FooBarTee{\n" +
" public final String name;\n" +
" public final TeeImpl tee;\n" +
" public final BarImpl bar;\n" +
" public final BarImpl copy;\n" +
" public final Foo foo;\n" +
'\n' +
" public FooBarTee(String name) {\n" +
" // when viewing this file, ensure it is synchronised with the copy on disk.\n" +
" System.out.println(\"" + text + "\");\n" +
" this.name = name;\n" +
'\n' +
" tee = new TeeImpl(\"test\");\n" +
'\n' +
" bar = new BarImpl(tee, 55);\n" +
'\n' +
" copy = new BarImpl(tee, 555);\n" +
'\n' +
" // you should see the current date here after synchronisation.\n" +
" foo = new Foo(bar, copy, \"" + text + "\", 5);\n" +
" }\n" +
'\n' +
" public void start() {\n" +
" }\n" +
'\n' +
" public void stop() {\n" +
" }\n" +
'\n' +
" public void close() {\n" +
" stop();\n" +
'\n' +
" }\n" +
"}\n");
// add a debug break point here and step into this method.
FooBarTee fooBarTee = new FooBarTee("test foo bar tee");
Foo foo = fooBarTee.foo;
assertNotNull(foo);
assertEquals(text, foo.s);
I wanted:
in-memory compilation of a java file (useful for Java as a scripting language)
No additional dependencies (easy to setup)
Implementation in as low number of files as possible (easy to incorporate in a project)
You can try it first here: http://ideone.com/cu1GhE#view_edit_box
The following code is based on Rekha Kumari code:
Main.java
package com.mycompany.java;
//import org.slf4j.Logger;
//import org.slf4j.LoggerFactory;
import java.io.PrintWriter;
import java.io.StringWriter;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
public class Main {
//private static final Logger logger = LoggerFactory.getLogger(Main.class);
public static void main(String[] args) {
try {
StringWriter writer = new StringWriter();
PrintWriter out = new PrintWriter(writer);
out.println("package com.mycompany.script;");
out.println("");
out.println("public class HelloWorld {");
out.println(" public static void main(String args[]) {");
out.println(" System.out.println(\"This is in another java file\");");
out.println(" }");
out.println("}");
out.close();
String fullName = "com.mycompany.script.HelloWorld";
String src = writer.toString();
DynamicCompiler uCompiler = new DynamicCompiler(fullName, src);
uCompiler.compile();
uCompiler.run();
} catch (Exception e) {
//logger.error("Exception:", e);
System.out.print("Exception");
}
}
}
DynamicCompiler.java
package com.mycompany.java;
//import org.slf4j.Logger;
//import org.slf4j.LoggerFactory;
import javax.tools.*;
import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.io.OutputStream;
import java.lang.reflect.InvocationTargetException;
import java.net.URI;
import java.security.SecureClassLoader;
import java.util.ArrayList;
import java.util.List;
// Based on: http://javapracs.blogspot.cz/2011/06/dynamic-in-memory-compilation-using.html
public class DynamicCompiler {
//private static final Logger logger = LoggerFactory.getLogger(DynamicCompiler.class);
private JavaFileManager fileManager;
private String fullName;
private String sourceCode;
public DynamicCompiler(String fullName, String srcCode) {
this.fullName = fullName;
this.sourceCode = srcCode;
this.fileManager = initFileManager();
}
public JavaFileManager initFileManager() {
if (fileManager != null)
return fileManager;
else {
JavaCompiler compiler = ToolProvider.getSystemJavaCompiler();
fileManager = new
ClassFileManager(compiler
.getStandardFileManager(null, null, null));
return fileManager;
}
}
public void compile() {
JavaCompiler compiler = ToolProvider.getSystemJavaCompiler();
List<JavaFileObject> files = new ArrayList<>();
files.add(new CharSequenceJavaFileObject(fullName, sourceCode));
compiler.getTask(
null,
fileManager,
null,
null,
null,
files
).call();
}
public void run() throws InstantiationException, IllegalAccessException, ClassNotFoundException {
try {
fileManager
.getClassLoader(null)
.loadClass(fullName)
.getDeclaredMethod("main", new Class[]{String[].class})
.invoke(null, new Object[]{null});
} catch (InvocationTargetException e) {
System.out.print("InvocationTargetException");
//logger.error("InvocationTargetException:", e);
} catch (NoSuchMethodException e) {
System.out.print("NoSuchMethodException ");
//logger.error("NoSuchMethodException:", e);
}
}
public class CharSequenceJavaFileObject extends SimpleJavaFileObject {
/**
* CharSequence representing the source code to be compiled
*/
private CharSequence content;
public CharSequenceJavaFileObject(String className, CharSequence content) {
super(URI.create("string:///" + className.replace('.', '/') + Kind.SOURCE.extension), Kind.SOURCE);
this.content = content;
}
public CharSequence getCharContent(boolean ignoreEncodingErrors) {
return content;
}
}
public class ClassFileManager extends ForwardingJavaFileManager {
private JavaClassObject javaClassObject;
public ClassFileManager(StandardJavaFileManager standardManager) {
super(standardManager);
}
#Override
public ClassLoader getClassLoader(Location location) {
return new SecureClassLoader() {
#Override
protected Class<?> findClass(String name) throws ClassNotFoundException {
byte[] b = javaClassObject.getBytes();
return super.defineClass(name, javaClassObject.getBytes(), 0, b.length);
}
};
}
public JavaFileObject getJavaFileForOutput(Location location, String className, JavaFileObject.Kind kind, FileObject sibling) throws IOException {
this.javaClassObject = new JavaClassObject(className, kind);
return this.javaClassObject;
}
}
public class JavaClassObject extends SimpleJavaFileObject {
protected final ByteArrayOutputStream bos =
new ByteArrayOutputStream();
public JavaClassObject(String name, Kind kind) {
super(URI.create("string:///" + name.replace('.', '/')
+ kind.extension), kind);
}
public byte[] getBytes() {
return bos.toByteArray();
}
#Override
public OutputStream openOutputStream() throws IOException {
return bos;
}
}
}
I'd like to introduce my solution which runs well in production.
Here are the three source code files.
MemoryJavaCompiler.java
package me.soulmachine.compiler;
import java.io.IOException;
import java.io.PrintWriter;
import java.io.Writer;
import java.lang.reflect.Method;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import javax.tools.*;
/**
* Simple interface to Java compiler using JSR 199 Compiler API.
*/
public class MemoryJavaCompiler {
private javax.tools.JavaCompiler tool;
private StandardJavaFileManager stdManager;
public MemoryJavaCompiler() {
tool = ToolProvider.getSystemJavaCompiler();
if (tool == null) {
throw new RuntimeException("Could not get Java compiler. Please, ensure that JDK is used instead of JRE.");
}
stdManager = tool.getStandardFileManager(null, null, null);
}
/**
* Compile a single static method.
*/
public Method compileStaticMethod(final String methodName, final String className,
final String source)
throws ClassNotFoundException {
final Map<String, byte[]> classBytes = compile(className + ".java", source);
final MemoryClassLoader classLoader = new MemoryClassLoader(classBytes);
final Class clazz = classLoader.loadClass(className);
final Method[] methods = clazz.getDeclaredMethods();
for (final Method method : methods) {
if (method.getName().equals(methodName)) {
if (!method.isAccessible()) method.setAccessible(true);
return method;
}
}
throw new NoSuchMethodError(methodName);
}
public Map<String, byte[]> compile(String fileName, String source) {
return compile(fileName, source, new PrintWriter(System.err), null, null);
}
/**
* compile given String source and return bytecodes as a Map.
*
* #param fileName source fileName to be used for error messages etc.
* #param source Java source as String
* #param err error writer where diagnostic messages are written
* #param sourcePath location of additional .java source files
* #param classPath location of additional .class files
*/
private Map<String, byte[]> compile(String fileName, String source,
Writer err, String sourcePath, String classPath) {
// to collect errors, warnings etc.
DiagnosticCollector<JavaFileObject> diagnostics =
new DiagnosticCollector<JavaFileObject>();
// create a new memory JavaFileManager
MemoryJavaFileManager fileManager = new MemoryJavaFileManager(stdManager);
// prepare the compilation unit
List<JavaFileObject> compUnits = new ArrayList<JavaFileObject>(1);
compUnits.add(fileManager.makeStringSource(fileName, source));
return compile(compUnits, fileManager, err, sourcePath, classPath);
}
private Map<String, byte[]> compile(final List<JavaFileObject> compUnits,
final MemoryJavaFileManager fileManager,
Writer err, String sourcePath, String classPath) {
// to collect errors, warnings etc.
DiagnosticCollector<JavaFileObject> diagnostics =
new DiagnosticCollector<JavaFileObject>();
// javac options
List<String> options = new ArrayList<String>();
options.add("-Xlint:all");
// options.add("-g:none");
options.add("-deprecation");
if (sourcePath != null) {
options.add("-sourcepath");
options.add(sourcePath);
}
if (classPath != null) {
options.add("-classpath");
options.add(classPath);
}
// create a compilation task
javax.tools.JavaCompiler.CompilationTask task =
tool.getTask(err, fileManager, diagnostics,
options, null, compUnits);
if (task.call() == false) {
PrintWriter perr = new PrintWriter(err);
for (Diagnostic diagnostic : diagnostics.getDiagnostics()) {
perr.println(diagnostic);
}
perr.flush();
return null;
}
Map<String, byte[]> classBytes = fileManager.getClassBytes();
try {
fileManager.close();
} catch (IOException exp) {
}
return classBytes;
}
}
MemoryJavaFileManager.java
package me.soulmachine.compiler;
import java.io.ByteArrayOutputStream;
import java.io.File;
import java.io.FilterOutputStream;
import java.io.IOException;
import java.io.OutputStream;
import java.net.URI;
import java.nio.CharBuffer;
import java.util.HashMap;
import java.util.Map;
import javax.tools.FileObject;
import javax.tools.ForwardingJavaFileManager;
import javax.tools.JavaFileManager;
import javax.tools.JavaFileObject;
import javax.tools.JavaFileObject.Kind;
import javax.tools.SimpleJavaFileObject;
/**
* JavaFileManager that keeps compiled .class bytes in memory.
*/
#SuppressWarnings("unchecked")
final class MemoryJavaFileManager extends ForwardingJavaFileManager {
/** Java source file extension. */
private final static String EXT = ".java";
private Map<String, byte[]> classBytes;
public MemoryJavaFileManager(JavaFileManager fileManager) {
super(fileManager);
classBytes = new HashMap<>();
}
public Map<String, byte[]> getClassBytes() {
return classBytes;
}
public void close() throws IOException {
classBytes = null;
}
public void flush() throws IOException {
}
/**
* A file object used to represent Java source coming from a string.
*/
private static class StringInputBuffer extends SimpleJavaFileObject {
final String code;
StringInputBuffer(String fileName, String code) {
super(toURI(fileName), Kind.SOURCE);
this.code = code;
}
public CharBuffer getCharContent(boolean ignoreEncodingErrors) {
return CharBuffer.wrap(code);
}
}
/**
* A file object that stores Java bytecode into the classBytes map.
*/
private class ClassOutputBuffer extends SimpleJavaFileObject {
private String name;
ClassOutputBuffer(String name) {
super(toURI(name), Kind.CLASS);
this.name = name;
}
public OutputStream openOutputStream() {
return new FilterOutputStream(new ByteArrayOutputStream()) {
public void close() throws IOException {
out.close();
ByteArrayOutputStream bos = (ByteArrayOutputStream)out;
classBytes.put(name, bos.toByteArray());
}
};
}
}
public JavaFileObject getJavaFileForOutput(JavaFileManager.Location location,
String className,
Kind kind,
FileObject sibling) throws IOException {
if (kind == Kind.CLASS) {
return new ClassOutputBuffer(className);
} else {
return super.getJavaFileForOutput(location, className, kind, sibling);
}
}
static JavaFileObject makeStringSource(String fileName, String code) {
return new StringInputBuffer(fileName, code);
}
static URI toURI(String name) {
File file = new File(name);
if (file.exists()) {
return file.toURI();
} else {
try {
final StringBuilder newUri = new StringBuilder();
newUri.append("mfm:///");
newUri.append(name.replace('.', '/'));
if(name.endsWith(EXT)) newUri.replace(newUri.length() - EXT.length(), newUri.length(), EXT);
return URI.create(newUri.toString());
} catch (Exception exp) {
return URI.create("mfm:///com/sun/script/java/java_source");
}
}
}
}
MemoryClassLoader.java
package me.soulmachine.compiler;
import java.io.File;
import java.net.MalformedURLException;
import java.net.URL;
import java.net.URLClassLoader;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import java.util.StringTokenizer;
/**
* ClassLoader that loads .class bytes from memory.
*/
final class MemoryClassLoader extends URLClassLoader {
private Map<String, byte[]> classBytes;
public MemoryClassLoader(Map<String, byte[]> classBytes,
String classPath, ClassLoader parent) {
super(toURLs(classPath), parent);
this.classBytes = classBytes;
}
public MemoryClassLoader(Map<String, byte[]> classBytes, String classPath) {
this(classBytes, classPath, ClassLoader.getSystemClassLoader());
}
public MemoryClassLoader(Map<String, byte[]> classBytes) {
this(classBytes, null, ClassLoader.getSystemClassLoader());
}
public Class load(String className) throws ClassNotFoundException {
return loadClass(className);
}
public Iterable<Class> loadAll() throws ClassNotFoundException {
List<Class> classes = new ArrayList<Class>(classBytes.size());
for (String name : classBytes.keySet()) {
classes.add(loadClass(name));
}
return classes;
}
protected Class findClass(String className) throws ClassNotFoundException {
byte[] buf = classBytes.get(className);
if (buf != null) {
// clear the bytes in map -- we don't need it anymore
classBytes.put(className, null);
return defineClass(className, buf, 0, buf.length);
} else {
return super.findClass(className);
}
}
private static URL[] toURLs(String classPath) {
if (classPath == null) {
return new URL[0];
}
List<URL> list = new ArrayList<URL>();
StringTokenizer st = new StringTokenizer(classPath, File.pathSeparator);
while (st.hasMoreTokens()) {
String token = st.nextToken();
File file = new File(token);
if (file.exists()) {
try {
list.add(file.toURI().toURL());
} catch (MalformedURLException mue) {}
} else {
try {
list.add(new URL(token));
} catch (MalformedURLException mue) {}
}
}
URL[] res = new URL[list.size()];
list.toArray(res);
return res;
}
}
Explanations:
In order to represent a Java source file in memory instead of disk, I defined a StringInputBuffer class in the MemoryJavaFileManager.java.
To save the compiled .class files in memory, I implemented a class MemoryJavaFileManager. The main idea is to override the function getJavaFileForOutput() to store bytecodes into a map.
To load the bytecodes in memory, I have to implement a customized classloader MemoryClassLoader, which reads bytecodes in the map and turn them into classes.
Here is a unite test.
package me.soulmachine.compiler;
import org.junit.Test;
import java.io.IOException;
import java.lang.reflect.InvocationTargetException;
import java.lang.reflect.Method;
import java.util.ArrayList;
import static org.junit.Assert.assertEquals;
public class MemoryJavaCompilerTest {
private final static MemoryJavaCompiler compiler = new MemoryJavaCompiler();
#Test public void compileStaticMethodTest()
throws ClassNotFoundException, InvocationTargetException, IllegalAccessException {
final String source = "public final class Solution {\n"
+ "public static String greeting(String name) {\n"
+ "\treturn \"Hello \" + name;\n" + "}\n}\n";
final Method greeting = compiler.compileStaticMethod("greeting", "Solution", source);
final Object result = greeting.invoke(null, "soulmachine");
assertEquals("Hello soulmachine", result.toString());
}
}
Reference
JavaCompiler.java from Cloudera Morphlines
How to create an object from a string in Java (how to eval a string)?
InMemoryJavaCompiler
Java-Runtime-Compiler
[动态的Java - 无废话JavaCompilerAPI中文指南]
Compile and Run Java Source Code in Memory.
String fileToCompile = ;
JavaCompile compiler = ToolProvider.getSystemJavaCompiler();
if(
compiler.run(null, null, null, "PACKAGE_NAME" + java.io.File.separator +"CLASS_NAME.java") == 0
)
System.out.println("Compilation is successful");
else
System.out.println("Compilation Failed");

Google SpeechClient io.grpc.StatusRuntimeException: UNAVAILABLE: Credentials failed to obtain metadata

I'm making an application with Google SpeechClient that has the requirements to set a GOOGLE_APPLICATION_CREDENTIALS environment variable that, once set, you can use the voice to text api.
My application is required to run in linux and windows. In linux it runs perfectly, however, on windows, when running the project, it throws an exception com.google.api.gax.rpc.UnavailableException: "io.grpc.StatusRuntimeException: UNAVAILABLE: Credentials failed to obtain metadata" when trying to run this thread
package Controller.Runnables;
import Controller.GUI.VoxSpeechGUIController;
import Model.SpokenTextHistory;
import com.google.api.gax.rpc.ClientStream;
import com.google.api.gax.rpc.ResponseObserver;
import com.google.api.gax.rpc.StreamController;
import com.google.cloud.speech.v1.*;
import com.google.protobuf.ByteString;
import javax.sound.sampled.AudioFormat;
import javax.sound.sampled.AudioSystem;
import javax.sound.sampled.DataLine;
import javax.sound.sampled.TargetDataLine;
import java.io.IOException;
import java.util.ArrayList;
public class SpeechRecognizerRunnable implements Runnable{
private VoxSpeechGUIController controller;
public SpeechRecognizerRunnable(VoxSpeechGUIController voxSpeechGUIController) {
this.controller = voxSpeechGUIController;
}
#Override
public void run() {
MicrofoneRunnable micrunnable = MicrofoneRunnable.getInstance();
Thread micThread = new Thread(micrunnable);
ResponseObserver<StreamingRecognizeResponse> responseObserver = null;
try (SpeechClient client = SpeechClient.create()) {
ClientStream<StreamingRecognizeRequest> clientStream;
responseObserver =
new ResponseObserver<StreamingRecognizeResponse>() {
ArrayList<StreamingRecognizeResponse> responses = new ArrayList<>();
public void onStart(StreamController controller) {}
public void onResponse(StreamingRecognizeResponse response) {
try {
responses.add(response);
StreamingRecognitionResult result = response.getResultsList().get(0);
// There can be several alternative transcripts for a given chunk of speech. Just
// use the first (most likely) one here.
SpeechRecognitionAlternative alternative = result.getAlternativesList().get(0);
String transcript = alternative.getTranscript();
System.out.printf("Transcript : %s\n", transcript);
String newText = SpokenTextHistory.getInstance().getActualSpeechString() + " " + transcript;
SpokenTextHistory.getInstance().setActualSpeechString(newText);
controller.setLabelText(newText);
}
catch (Exception ex){
System.out.println(ex.getMessage());
ex.printStackTrace();
}
}
public void onComplete() {
}
public void onError(Throwable t) {
System.out.println(t);
}
};
clientStream = client.streamingRecognizeCallable().splitCall(responseObserver);
RecognitionConfig recognitionConfig =
RecognitionConfig.newBuilder()
.setEncoding(RecognitionConfig.AudioEncoding.LINEAR16)
.setLanguageCode("pt-BR")
.setSampleRateHertz(16000)
.build();
StreamingRecognitionConfig streamingRecognitionConfig =
StreamingRecognitionConfig.newBuilder().setConfig(recognitionConfig).build();
StreamingRecognizeRequest request =
StreamingRecognizeRequest.newBuilder()
.setStreamingConfig(streamingRecognitionConfig)
.build(); // The first request in a streaming call has to be a config
clientStream.send(request);
try {
// SampleRate:16000Hz, SampleSizeInBits: 16, Number of channels: 1, Signed: true,
// bigEndian: false
AudioFormat audioFormat = new AudioFormat(16000, 16, 1, true, false);
DataLine.Info targetInfo =
new DataLine.Info(
TargetDataLine.class,
audioFormat); // Set the system information to read from the microphone audio
// stream
if (!AudioSystem.isLineSupported(targetInfo)) {
System.out.println("Microphone not supported");
System.exit(0);
}
// Target data line captures the audio stream the microphone produces.
micrunnable.targetDataLine = (TargetDataLine) AudioSystem.getLine(targetInfo);
micrunnable.targetDataLine.open(audioFormat);
micThread.start();
long startTime = System.currentTimeMillis();
while (!micrunnable.stopFlag) {
long estimatedTime = System.currentTimeMillis() - startTime;
if (estimatedTime >= 55000) {
clientStream.closeSend();
clientStream = client.streamingRecognizeCallable().splitCall(responseObserver);
request =
StreamingRecognizeRequest.newBuilder()
.setStreamingConfig(streamingRecognitionConfig)
.build();
startTime = System.currentTimeMillis();
} else {
request =
StreamingRecognizeRequest.newBuilder()
.setAudioContent(ByteString.copyFrom(micrunnable.sharedQueue.take()))
.build();
}
clientStream.send(request);
}
} catch (Exception e) {
System.out.println(e);
}
} catch (IOException e) {
e.printStackTrace();
}
}
}
I've been working hard for hours and have not found a solution that solves my problem.
It is worth mentioning that the environment variable is being set correctly.
Has anyone ever had this problem with Google? What should I do to fix this?
This is my envirounment variable creator:
PS: I`ve already tried use all google alternatives to validate credentials, but all return me errors.
package Controller.Autentication;
import java.io.*;
import java.lang.reflect.Field;
import java.util.Collections;
import java.util.HashMap;
import java.util.Map;
public class GoogleAuthentication {
private static final String GOOGLE_APPLICATION_CREDENTIALS = "GOOGLE_APPLICATION_CREDENTIALS";
private static final String VoxSpeechFolder = ".vox";
private static final String GoogleAuthenticationJsonFile = "VoxAuthentication.json";
public static void setupGoogleCredentials() {
String directory = defaultDirectory();
directory += File.separator+VoxSpeechFolder;
File voxPath = new File(directory);
if (!voxPath.exists()) {
voxPath.mkdirs();
}
ClassLoader classLoader = new GoogleAuthentication().getClass().getClassLoader();
File srcFile = new File(classLoader.getResource(GoogleAuthenticationJsonFile).getFile());
if(srcFile.exists()){
try {
String voxDestPath = defaultDirectory() + File.separator + VoxSpeechFolder +File.separator+ GoogleAuthenticationJsonFile;
File destFile = new File(voxDestPath);
copyFile(srcFile,destFile);
} catch (IOException e) {
e.printStackTrace();
}
}
try {
Map<String,String> googleEnv = new HashMap<>();
String path = defaultDirectory() +File.separator+ VoxSpeechFolder +File.separator+ GoogleAuthenticationJsonFile;
googleEnv.put(GOOGLE_APPLICATION_CREDENTIALS, path);
setGoogleEnv(googleEnv);
} catch (Exception e) {
e.printStackTrace();
}
}
static void copyFile(File sourceFile, File destFile)
throws IOException {
InputStream inStream ;
OutputStream outStream ;
System.out.println(destFile.getPath());
if(destFile.createNewFile()){
inStream = new FileInputStream(sourceFile);
outStream = new FileOutputStream(destFile);
byte[] buffer = new byte[1024];
int length;
while ((length = inStream.read(buffer)) > 0){
outStream.write(buffer, 0, length);
}
inStream.close();
outStream.close();
}
}
static String defaultDirectory()
{
String OS = getOperationSystem();
if (OS.contains("WIN"))
return System.getenv("APPDATA");
else if (OS.contains("MAC"))
return System.getProperty("user.home") + "/Library/Application "
+ "Support";
else if (OS.contains("LINUX")) {
return System.getProperty("user.home");
}
return System.getProperty("user.dir");
}
static String getOperationSystem() {
return System.getProperty("os.name").toUpperCase();
}
protected static void setGoogleEnv(Map<String, String> newenv) throws Exception {
try {
Class<?> processEnvironmentClass = Class.forName("java.lang.ProcessEnvironment");
Field theEnvironmentField = processEnvironmentClass.getDeclaredField("theEnvironment");
theEnvironmentField.setAccessible(true);
Map<String, String> env = (Map<String, String>) theEnvironmentField.get(null);
env.putAll(newenv);
Field theCaseInsensitiveEnvironmentField = processEnvironmentClass.getDeclaredField("theCaseInsensitiveEnvironment");
theCaseInsensitiveEnvironmentField.setAccessible(true);
Map<String, String> cienv = (Map<String, String>) theCaseInsensitiveEnvironmentField.get(null);
cienv.putAll(newenv);
} catch (NoSuchFieldException e) {
Class[] classes = Collections.class.getDeclaredClasses();
Map<String, String> env = System.getenv();
for(Class cl : classes) {
if("java.util.Collections$UnmodifiableMap".equals(cl.getName())) {
Field field = cl.getDeclaredField("m");
field.setAccessible(true);
Object obj = field.get(env);
Map<String, String> map = (Map<String, String>) obj;
map.clear();
map.putAll(newenv);
}
}
}
String genv = System.getenv(GOOGLE_APPLICATION_CREDENTIALS);
System.out.println(genv);
}
}

Fork/Join combine with FileChannel to copy file

Recently I am making an exercise using Java 7 FORK/JOIN framework and FileChannel to copy a file. Here is my code (Test.java):
import java.io.File;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.FileOutputStream;
import java.io.IOException;
import java.nio.channels.FileChannel;
import java.nio.file.Files;
import java.nio.file.Paths;
import java.util.ArrayList;
import java.util.concurrent.ForkJoinPool;
import java.util.concurrent.RecursiveTask;
import java.util.concurrent.TimeUnit;
import java.util.logging.Level;
import java.util.logging.Logger;
public class Test {
private ArrayList<FileProcessor> processors = new ArrayList<FileProcessor>();
public Test(){
String outputDir = "C:\\temp";
if (!Files.isDirectory(Paths.get(outputDir))) {
System.out.println("this is not a path");
} else {
try {
//start copying file
ForkJoinPool pool = new ForkJoinPool();
int numberOfThread = 2;
File file = new File("C:\\abc.cdm");
long length = file.length();
long lengthPerCopy = (long)(length/numberOfThread);
long position = 0L;
for (int i = 0; i < numberOfThread; i++) {
FileProcessor processor = null;
if (i == numberOfThread - 1) {
//the last thread
processor = new FileProcessor("abc.cdm", "C:\\abc.cdm", "C:\\temp", position, length - position);
} else {
processor = new FileProcessor("abc.cdm", "C:\\abc.cdm", "C:\\temp", position, lengthPerCopy);
position = position + lengthPerCopy + 1;
}
processors.add(processor);
pool.execute(processor);
}
do {
System.out.printf("******************************************\n");
System.out.printf("Main: Parallelism: %d\n", pool.getParallelism());
System.out.printf("Main: Active Threads: %d\n", pool.getActiveThreadCount());
System.out.printf("Main: Task Count: %d\n", pool.getQueuedTaskCount());
System.out.printf("Main: Steal Count: %d\n", pool.getStealCount());
System.out.printf("******************************************\n");
try
{
TimeUnit.SECONDS.sleep(1);
} catch (InterruptedException e)
{
e.printStackTrace();
}
} while (!isDone()); //when all the thread not been done
pool.shutdown();
System.out.println("copy done");
} catch (Exception ex) {
//out an error here...
}
}
}
private boolean isDone(){
boolean res = false;
for (int i = 0; i < processors.size(); i++) {
res = res || processors.get(i).isDone();
}
return res;
}
public static void main(String args[]) {
Test test = new Test();
}
class FileProcessor extends RecursiveTask<Integer>
{
private static final long serialVersionUID = 1L;
private long copyPosition;
private long copyCount;
FileChannel source = null;
FileChannel destination = null;
//Implement the constructor of the class to initialize its attributes
public FileProcessor(String fileName, String filePath, String outputPath, long position, long count) throws FileNotFoundException, IOException{
this.copyPosition = position;
this.copyCount = count;
this.source = new FileInputStream(new File(filePath)).getChannel().position(copyPosition);
this.destination = new FileOutputStream(new File(outputPath + "/" + fileName), true).getChannel().position(copyPosition);
}
#Override
protected Integer compute()
{
try {
this.copyFile();
} catch (IOException ex) {
Logger.getLogger(FileProcessor.class.getName()).log(Level.SEVERE, null, ex);
}
return new Integer(0);
}
private void copyFile() throws IOException {
try {
destination.transferFrom(source, copyPosition, copyCount);
}
finally {
if (source != null) {
source.close();
}
if (destination != null) {
destination.close();
}
}
}
}
}
I run my code,
if number of threads is 1, the file is copied exactly, but when number of theads is 2, file "C:\abc.cdm" is 77KB (78335), but after copied, file "C:\temp\abc.cdm" is just (39KB).
Where did I get wrong, please tell me??
Update: My problem has been solves
The problem is in isDone method, it must be:
boolean res = true;
for (int i = 0; i < processors.size(); i++) {
res = res && processors.get(i).isDone();
}
return res;
Also edit the following lines of codes:
File file = new File(selectedFile[i].getPath());
long length = file.length();
new RandomAccessFile("C:\\temp\abc.cdm", "rw").setLength(length);
This is just a practice for FORK/JOIN usage!
Your isDone() method was indeed wrong and you corrected it in the original question. But there is another issue in the FileProcessor. You assume that setting the position on the destination past the end of the file will automatically grow the file when you transfer to it. This is not the case.
Your first segment will always write because the write position is 0 and the file's length cannot be less than zero. That was the 39K you saw, which is roughly half of the total file size. The second segment never got written.
In order to get your code to run, you can do the following at the start:
File file = new File("C:\\abc.cdm");
long length = file.length();
new RandomAccessFile("C:\\temp\\abc.cdm", "rw").setLength(length);`

Parse zip or Jar project

I need to return all the packages, classes ... that a java project (zip/jar) contains. I guess QDox can do that. I found that class : http://www.jarvana.com/jarvana/view/com/ning/metrics.serialization-all/2.0.0-pre5/metrics.serialization-all-2.0.0-pre5-jar-with-dependencies-sources.jar!/com/thoughtworks/qdox/tools/QDoxTester.java?format=ok
package com.thoughtworks.qdox.tools;
import com.thoughtworks.qdox.JavaDocBuilder;
import com.thoughtworks.qdox.directorywalker.DirectoryScanner;
import com.thoughtworks.qdox.directorywalker.FileVisitor;
import com.thoughtworks.qdox.directorywalker.SuffixFilter;
import com.thoughtworks.qdox.parser.ParseException;
import java.io.BufferedReader;
import java.io.File;
import java.io.FileInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.io.PrintStream;
import java.util.Enumeration;
import java.util.zip.ZipEntry;
import java.util.zip.ZipFile;
/**
* Tool for testing that QDox can parse Java source code.
*
* #author Joe Walnes
*/
public class QDoxTester {
public static interface Reporter {
void success(String id);
void parseFailure(String id, int line, int column, String reason);
void error(String id, Throwable throwable);
}
private final Reporter reporter;
public QDoxTester(Reporter reporter) {
this.reporter = reporter;
}
public void checkZipOrJarFile(File file) throws IOException {
ZipFile zipFile = new ZipFile(file);
Enumeration entries = zipFile.entries();
while (entries.hasMoreElements()) {
ZipEntry zipEntry = (ZipEntry) entries.nextElement();
InputStream inputStream = zipFile.getInputStream(zipEntry);
try {
verify(file.getName() + "!" + zipEntry.getName(), inputStream);
} finally {
inputStream.close();
}
}
}
public void checkDirectory(File dir) throws IOException {
DirectoryScanner directoryScanner = new DirectoryScanner(dir);
directoryScanner.addFilter(new SuffixFilter(".java"));
directoryScanner.scan(new FileVisitor() {
public void visitFile(File file) {
try {
checkJavaFile(file);
} catch (IOException e) {
// ?
}
}
});
}
public void checkJavaFile(File file) throws IOException {
InputStream inputStream = new FileInputStream(file);
try {
verify(file.getName(), inputStream);
} finally {
inputStream.close();
}
}
private void verify(String id, InputStream inputStream) {
try {
JavaDocBuilder javaDocBuilder = new JavaDocBuilder();
javaDocBuilder.addSource(new BufferedReader(new InputStreamReader(inputStream)));
reporter.success(id);
} catch (ParseException parseException) {
reporter.parseFailure(id, parseException.getLine(), parseException.getColumn(), parseException.getMessage());
} catch (Exception otherException) {
reporter.error(id, otherException);
}
}
public static void main(String[] args) throws IOException {
if (args.length == 0) {
System.err.println("Tool that verifies that QDox can parse some Java source.");
System.err.println();
System.err.println("Usage: java " + QDoxTester.class.getName() + " src1 [src2] [src3]...");
System.err.println();
System.err.println("Each src can be a single .java file, or a directory/zip/jar containing multiple source files");
System.exit(-1);
}
ConsoleReporter reporter = new ConsoleReporter(System.out);
QDoxTester qDoxTester = new QDoxTester(reporter);
for (int i = 0; i < args.length; i++) {
File file = new File(args[i]);
if (file.isDirectory()) {
qDoxTester.checkDirectory(file);
} else if (file.getName().endsWith(".java")) {
qDoxTester.checkJavaFile(file);
} else if (file.getName().endsWith(".jar") || file.getName().endsWith(".zip")) {
qDoxTester.checkZipOrJarFile(file);
} else {
System.err.println("Unknown input <" + file.getName() + ">. Should be zip, jar, java or directory");
}
}
reporter.writeSummary();
}
private static class ConsoleReporter implements Reporter {
private final PrintStream out;
private int success;
private int failure;
private int error;
private int dotsWrittenThisLine;
public ConsoleReporter(PrintStream out) {
this.out = out;
}
public void success(String id) {
success++;
if (++dotsWrittenThisLine > 80) {
newLine();
}
out.print('.');
}
private void newLine() {
dotsWrittenThisLine = 0;
out.println();
out.flush();
}
public void parseFailure(String id, int line, int column, String reason) {
newLine();
out.println("* " + id);
out.println(" [" + line + ":" + column + "] " + reason);
failure++;
}
public void error(String id, Throwable throwable) {
newLine();
out.println("* " + id);
throwable.printStackTrace(out);
error++;
}
public void writeSummary() {
newLine();
out.println("-- Summary --------------");
out.println("Success: " + success);
out.println("Failure: " + failure);
out.println("Error : " + error);
out.println("Total : " + (success + failure + error));
out.println("-------------------------");
}
}
}
It contains a method called checkZipOrJarFile(File file), maybe it could help me. but I can't find any examples or tutorials on how to use it.
Any help is welcomed.
QDox cannot do that for you, unfortunately (I came here looking for QDox support for jars). The source code for QDox that you list above is only for testing if the classes in the given jar can be parsed successfully by QDox. That code does, however, give you a clue on how to use standard java apis to do what you want: enumerate over classed in a jar.
Here's some code I'm using (which I cribbed from another SO answer here: analyze jar file programmatically)
// Your jar file
JarFile jar = new JarFile(jarFile);
// Getting the files into the jar
Enumeration<? extends JarEntry> enumeration = jar.entries();
// Iterates into the files in the jar file
while (enumeration.hasMoreElements()) {
ZipEntry zipEntry = enumeration.nextElement();
// Is this a class?
if (zipEntry.getName().endsWith(".class")) {
// Relative path of file into the jar.
String className = zipEntry.getName();
// Complete class name
className = className.replace(".class", "").replace("/", ".");
// Load class definition from JVM - you may not need this, but I want to introspect the class
try {
Class<?> clazz = getClass().getClassLoader().loadClass(className);
// ... I then go on to do some intropsection
If you don't actually want to introspect the class as I do, you can stop at getName(). Also, if you specifically want to find packages, you could use zipEntry.isDirectory() on your zip entries.

Categories