We Keep Coding

java Gherkin parser stream does not release file locks

I am using Gherkin parser to parse feature files and returning the list of Gherkin documents see the function below:
import io.cucumber.gherkin.Gherkin;
import io.cucumber.messages.IdGenerator;
import io.cucumber.messages.Messages;
import io.cucumber.messages.Messages.Envelope;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
import java.util.ArrayList;
import java.util.List;
import java.util.stream.Collectors;
import java.util.stream.Stream;
public class GherkinUtils {
private static final Logger LOG = LogManager.getLogger(GherkinUtils.class);
public static ArrayList<Messages.GherkinDocument> getGherkinDocumentsFromFiles() {
IdGenerator idGenerator = new IdGenerator.Incrementing();
ArrayList<Messages.GherkinDocument> listOfGherkinDocuments = new ArrayList<>();
String pathFolderFrameworkFeatures = SettingsUtils.getPathFolderFrameworkFeatures();
List<String> listOfPathsForFeatureFiles = FileUtils.getAllFilePathsFromFolder(pathFolderFrameworkFeatures);
try (Stream<Envelope> dataStream = Gherkin.fromPaths(listOfPathsForFeatureFiles, false, true, false, idGenerator)){
List<Envelope> envelopes = dataStream.collect(Collectors.toList());
for (Envelope env : envelopes) {
Messages.GherkinDocument gherkinDocument = env.getGherkinDocument();
listOfGherkinDocuments.add(gherkinDocument);
}
} catch (Exception e) {
LOG.error("Error occurred while trying to read the feature files", new Exception(e));
}
FileUtils.renameAllFeatureFiles("b");
return listOfGherkinDocuments;
}
}
Just before the return statement, you can see the function that will update the name for all feature files just to check if they are not locked.
The problem is that only the first file is always renamed and the rest of them are always locked.
If I will place the rename function at the top, then all the files are successfully renamed...
My understanding is that the try statement will automatically close the stream. Also, I tried to close it manually inside the try block but the results are the same.
What am I missing? How can I make it to release the file locks?
Update 1:
This exact line is making the files (except the first one to be locked):
List<Envelope> envelopes = dataStream.collect(Collectors.toList());
Here is the file name update function definition in case you want to test it:
public static void renameAllFeatureFiles(String fileName) {
String pathFeaturesFolder = SettingsUtils.getPathFolderFrameworkFeatures();
List<String> pathList = FileUtils.getAllFilePathsFromFolder(pathFeaturesFolder);
int counter = 0;
for (String path : pathList) {
counter ++;
File file = new File(path);
File newFile = new File(pathFeaturesFolder + "\\" + fileName +counter+".feature");
System.out.println("File: " + path + " locked: " + !file.renameTo(newFile));
}
}
And here is a sample feature file content:
Feature: Test
Scenario: test 1
Given User will do something
And User will do something
Update 2:
Tried with separate thread using javafx Task, still the same issue :(
Except for one file (this is really strange) all files are locked...
public static void runInNewThread() {
// define the execution task that will run in a new thread
Task<Void> newTask = new Task<>() {
#Override
protected Void call() {
ArrayList<Messages.GherkinDocument> listOfGherkinDocuments = GherkinUtils.getGherkinDocumentsFromFiles();
return null;
}
};
// run the task in a new thread
Thread th = new Thread(newTask);
th.setDaemon(true);
th.start();
}

For now, I have used workaround with creating copies of the specific files and using parser on the copies to prevent locking of the original versions...

Get Random File from Folder - Java [duplicate]

I am looking for a way to get a list of all resource names from a given classpath directory, something like a method List<String> getResourceNames (String directoryName).
For example, given a classpath directory x/y/z containing files a.html, b.html, c.html and a subdirectory d, getResourceNames("x/y/z") should return a List<String> containing the following strings:['a.html', 'b.html', 'c.html', 'd'].
It should work both for resources in filesystem and jars.
I know that I can write a quick snippet with Files, JarFiles and URLs, but I do not want to reinvent the wheel. My question is, given existing publicly available libraries, what is the quickest way to implement getResourceNames? Spring and Apache Commons stacks are both feasible.

Custom Scanner
Implement your own scanner. For example:
(limitations of this solution are mentioned in the comments)
private List<String> getResourceFiles(String path) throws IOException {
List<String> filenames = new ArrayList<>();
try (
InputStream in = getResourceAsStream(path);
BufferedReader br = new BufferedReader(new InputStreamReader(in))) {
String resource;
while ((resource = br.readLine()) != null) {
filenames.add(resource);
}
}
return filenames;
}
private InputStream getResourceAsStream(String resource) {
final InputStream in
= getContextClassLoader().getResourceAsStream(resource);
return in == null ? getClass().getResourceAsStream(resource) : in;
}
private ClassLoader getContextClassLoader() {
return Thread.currentThread().getContextClassLoader();
}
Spring Framework
Use PathMatchingResourcePatternResolver from Spring Framework.
Ronmamo Reflections
The other techniques might be slow at runtime for huge CLASSPATH values. A faster solution is to use ronmamo's Reflections API, which precompiles the search at compile time.

Here is the code
Source: forums.devx.com/showthread.php?t=153784
import java.io.File;
import java.io.IOException;
import java.util.ArrayList;
import java.util.Collection;
import java.util.Enumeration;
import java.util.regex.Pattern;
import java.util.zip.ZipEntry;
import java.util.zip.ZipException;
import java.util.zip.ZipFile;
/**
* list resources available from the classpath # *
*/
public class ResourceList{
/**
* for all elements of java.class.path get a Collection of resources Pattern
* pattern = Pattern.compile(".*"); gets all resources
*
* #param pattern
* the pattern to match
* #return the resources in the order they are found
*/
public static Collection<String> getResources(
final Pattern pattern){
final ArrayList<String> retval = new ArrayList<String>();
final String classPath = System.getProperty("java.class.path", ".");
final String[] classPathElements = classPath.split(System.getProperty("path.separator"));
for(final String element : classPathElements){
retval.addAll(getResources(element, pattern));
}
return retval;
}
private static Collection<String> getResources(
final String element,
final Pattern pattern){
final ArrayList<String> retval = new ArrayList<String>();
final File file = new File(element);
if(file.isDirectory()){
retval.addAll(getResourcesFromDirectory(file, pattern));
} else{
retval.addAll(getResourcesFromJarFile(file, pattern));
}
return retval;
}
private static Collection<String> getResourcesFromJarFile(
final File file,
final Pattern pattern){
final ArrayList<String> retval = new ArrayList<String>();
ZipFile zf;
try{
zf = new ZipFile(file);
} catch(final ZipException e){
throw new Error(e);
} catch(final IOException e){
throw new Error(e);
}
final Enumeration e = zf.entries();
while(e.hasMoreElements()){
final ZipEntry ze = (ZipEntry) e.nextElement();
final String fileName = ze.getName();
final boolean accept = pattern.matcher(fileName).matches();
if(accept){
retval.add(fileName);
}
}
try{
zf.close();
} catch(final IOException e1){
throw new Error(e1);
}
return retval;
}
private static Collection<String> getResourcesFromDirectory(
final File directory,
final Pattern pattern){
final ArrayList<String> retval = new ArrayList<String>();
final File[] fileList = directory.listFiles();
for(final File file : fileList){
if(file.isDirectory()){
retval.addAll(getResourcesFromDirectory(file, pattern));
} else{
try{
final String fileName = file.getCanonicalPath();
final boolean accept = pattern.matcher(fileName).matches();
if(accept){
retval.add(fileName);
}
} catch(final IOException e){
throw new Error(e);
}
}
}
return retval;
}
/**
* list the resources that match args[0]
*
* #param args
* args[0] is the pattern to match, or list all resources if
* there are no args
*/
public static void main(final String[] args){
Pattern pattern;
if(args.length < 1){
pattern = Pattern.compile(".*");
} else{
pattern = Pattern.compile(args[0]);
}
final Collection<String> list = ResourceList.getResources(pattern);
for(final String name : list){
System.out.println(name);
}
}
}
If you are using Spring Have a look at PathMatchingResourcePatternResolver

Using Reflections
Get everything on the classpath:
Reflections reflections = new Reflections(null, new ResourcesScanner());
Set<String> resourceList = reflections.getResources(x -> true);
Another example - get all files with extension .csv from some.package:
Reflections reflections = new Reflections("some.package", new ResourcesScanner());
Set<String> resourceList = reflections.getResources(Pattern.compile(".*\\.csv"));

So in terms of the PathMatchingResourcePatternResolver this is what is needed in the code:
#Autowired
ResourcePatternResolver resourceResolver;
public void getResources() {
resourceResolver.getResources("classpath:config/*.xml");
}

If you use apache commonsIO you can use for the filesystem (optionally with extension filter):
Collection<File> files = FileUtils.listFiles(new File("directory/"), null, false);
and for resources/classpath:
List<String> files = IOUtils.readLines(MyClass.class.getClassLoader().getResourceAsStream("directory/"), Charsets.UTF_8);
If you don't know if "directoy/" is in the filesystem or in resources you may add a
if (new File("directory/").isDirectory())
or
if (MyClass.class.getClassLoader().getResource("directory/") != null)
before the calls and use both in combination...

The most robust mechanism for listing all resources in the classpath is currently to use this pattern with ClassGraph, because it handles the widest possible array of classpath specification mechanisms, including the new JPMS module system. (I am the author of ClassGraph.)
List<String> resourceNames;
try (ScanResult scanResult = new ClassGraph().acceptPaths("x/y/z").scan()) {
resourceNames = scanResult.getAllResources().getNames();
}

The Spring framework's PathMatchingResourcePatternResolver is really awesome for these things:
private Resource[] getXMLResources() throws IOException
{
ClassLoader classLoader = MethodHandles.lookup().getClass().getClassLoader();
PathMatchingResourcePatternResolver resolver = new PathMatchingResourcePatternResolver(classLoader);
return resolver.getResources("classpath:x/y/z/*.xml");
}
Maven dependency:
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-core</artifactId>
<version>LATEST</version>
</dependency>

This should work (if spring is not an option):
public static List<String> getFilenamesForDirnameFromCP(String directoryName) throws URISyntaxException, UnsupportedEncodingException, IOException {
List<String> filenames = new ArrayList<>();
URL url = Thread.currentThread().getContextClassLoader().getResource(directoryName);
if (url != null) {
if (url.getProtocol().equals("file")) {
File file = Paths.get(url.toURI()).toFile();
if (file != null) {
File[] files = file.listFiles();
if (files != null) {
for (File filename : files) {
filenames.add(filename.toString());
}
}
}
} else if (url.getProtocol().equals("jar")) {
String dirname = directoryName + "/";
String path = url.getPath();
String jarPath = path.substring(5, path.indexOf("!"));
try (JarFile jar = new JarFile(URLDecoder.decode(jarPath, StandardCharsets.UTF_8.name()))) {
Enumeration<JarEntry> entries = jar.entries();
while (entries.hasMoreElements()) {
JarEntry entry = entries.nextElement();
String name = entry.getName();
if (name.startsWith(dirname) && !dirname.equals(name)) {
URL resource = Thread.currentThread().getContextClassLoader().getResource(name);
filenames.add(resource.toString());
}
}
}
}
}
return filenames;
}

My way, no Spring, used during a unit test:
URI uri = TestClass.class.getResource("/resources").toURI();
Path myPath = Paths.get(uri);
Stream<Path> walk = Files.walk(myPath, 1);
for (Iterator<Path> it = walk.iterator(); it.hasNext(); ) {
Path filename = it.next();
System.out.println(filename);
}

With Spring it's easy. Be it a file, or folder, or even multiple files, there are chances, you can do it via injection.
This example demonstrates the injection of multiple files located in x/y/z folder.
import org.springframework.beans.factory.annotation.Value;
import org.springframework.core.io.Resource;
import org.springframework.stereotype.Service;
#Service
public class StackoverflowService {
#Value("classpath:x/y/z/*")
private Resource[] resources;
public List<String> getResourceNames() {
return Arrays.stream(resources)
.map(Resource::getFilename)
.collect(Collectors.toList());
}
}
It does work for resources in the filesystem as well as in JARs.

Used a combination of Rob's response.
final String resourceDir = "resourceDirectory/";
List<String> files = IOUtils.readLines(Thread.currentThread().getClass().getClassLoader().getResourceAsStream(resourceDir), Charsets.UTF_8);
for (String f : files) {
String data = IOUtils.toString(Thread.currentThread().getClass().getClassLoader().getResourceAsStream(resourceDir + f));
// ... process data
}

I think you can leverage the [Zip File System Provider][1] to achieve this. When using FileSystems.newFileSystem it looks like you can treat the objects in that ZIP as a "regular" file.
In the linked documentation above:
Specify the configuration options for the zip file system in the java.util.Map object passed to the FileSystems.newFileSystem method. See the [Zip File System Properties][2] topic for information about the provider-specific configuration properties for the zip file system.
Once you have an instance of a zip file system, you can invoke the methods of the [java.nio.file.FileSystem][3] and [java.nio.file.Path][4] classes to perform operations such as copying, moving, and renaming files, as well as modifying file attributes.
The documentation for the jdk.zipfs module in [Java 11 states][5]:
The zip file system provider treats a zip or JAR file as a file system and provides the ability to manipulate the contents of the file. The zip file system provider can be created by [FileSystems.newFileSystem][6] if installed.
Here is a contrived example I did using your example resources. Note that a .zip is a .jar, but you could adapt your code to instead use classpath resources:
Setup
cd /tmp
mkdir -p x/y/z
touch x/y/z/{a,b,c}.html
echo 'hello world' > x/y/z/d
zip -r example.zip x
Java
import java.io.IOException;
import java.net.URI;
import java.nio.file.FileSystem;
import java.nio.file.FileSystems;
import java.nio.file.Files;
import java.util.Collections;
import java.util.stream.Collectors;
public class MkobitZipRead {
public static void main(String[] args) throws IOException {
final URI uri = URI.create("jar:file:/tmp/example.zip");
try (
final FileSystem zipfs = FileSystems.newFileSystem(uri, Collections.emptyMap());
) {
Files.walk(zipfs.getPath("/")).forEach(path -> System.out.println("Files in zip:" + path));
System.out.println("-----");
final String manifest = Files.readAllLines(
zipfs.getPath("x", "y", "z").resolve("d")
).stream().collect(Collectors.joining(System.lineSeparator()));
System.out.println(manifest);
}
}
}
Output
Files in zip:/
Files in zip:/x/
Files in zip:/x/y/
Files in zip:/x/y/z/
Files in zip:/x/y/z/c.html
Files in zip:/x/y/z/b.html
Files in zip:/x/y/z/a.html
Files in zip:/x/y/z/d
-----
hello world

Neither of answers worked for me even though I had my resources put in resources folders and followed the above answers. What did make a trick was:
#Value("file:*/**/resources/**/schema/*.json")
private Resource[] resources;

Expanding on Luke Hutchinsons answer above, using his ClassGraph library, I was able to easily get a list of all files in a Resource folder with almost no effort at all.
Let's say that in your resource folder, you have a folder called MyImages. This is how easy it is to get a URL list of all the files in that folder:
import io.github.classgraph.ClassGraph;
import io.github.classgraph.ResourceList;
import io.github.classgraph.ScanResult;
public static LinkedList<URL> getURLList (String folder) {
LinkedList<URL> urlList = new LinkedList<>();
ScanResult scanResult = new ClassGraph().enableAllInfo().scan();
ResourceList resources = scanResult.getAllResources();
for (URL url : resources.getURLs()) {
if (url.toString().contains(folder)) {
urlList.addLast(url);
}
}
return urlList;
}
Then you simply do this:
LinkedList<URL> myURLFileList = getURLList("MyImages");
The URLs can then be loaded into streams or use Apache's FileUtils to copy the files somewhere else like this:
String outPath = "/My/Output/Path";
for(URL url : myURLFileList) {
FileUtils.copyURLToFile(url, new File(outPath, url.getFile()));
}
I think ClassGraph is a pretty slick library for making tasks like this very simple and easy to comprehend.

Based on #rob 's information above, I created the implementation which I am releasing to the public domain:
private static List<String> getClasspathEntriesByPath(String path) throws IOException {
InputStream is = Main.class.getClassLoader().getResourceAsStream(path);
StringBuilder sb = new StringBuilder();
while (is.available()>0) {
byte[] buffer = new byte[1024];
sb.append(new String(buffer, Charset.defaultCharset()));
}
return Arrays
.asList(sb.toString().split("\n")) // Convert StringBuilder to individual lines
.stream() // Stream the list
.filter(line -> line.trim().length()>0) // Filter out empty lines
.collect(Collectors.toList()); // Collect remaining lines into a List again
}
While I would not have expected getResourcesAsStream to work like that on a directory, it really does and it works well.

Pass parameters to a maven mojo invoked programmatically

Here I have my Maven Mojo:
import org.apache.maven.plugin.AbstractMojo;
import org.apache.maven.plugin.MojoExecutionException;
import org.apache.maven.plugins.annotations.Mojo;
import org.apache.maven.shared.invoker.*;
import java.util.Collections;
#Mojo(name = "run")
public class RunMojo extends AbstractMojo {
#Override
public void execute() throws MojoExecutionException {
InvocationRequest request = new DefaultInvocationRequest();
request.setGoals(Collections.singletonList("myplugin:mygoal"));
// need to set parameters to pass to the goal
Invoker invoker = new DefaultInvoker();
try {
invoker.execute(request);
} catch (MavenInvocationException e) {
e.printStackTrace();
}
}
}
And I need to invoke a second Mojo passing some parameters as I do when defining the plugin inside the pom.xml, as follow.
<build>
<plugins>
<plugin>
<artifactId>myPlugin</artifactId>
<groupId>myGroupId</groupId>
<version>myVersion</version>
<configuration>
<param1>value1</param1>
<param2>value2</param2>
<param3>value3</param3>
</configuration>
</plugin>
</plugins>
</build>
Any solution?

It might be an older question, but found myself in this situation recently and taught I'd show how I created a new plugin where which wraps another plugin.
I've ended up using the mojo-executor plugin (https://github.com/TimMoore/mojo-executor) . Even though it is not actively maintained, it does the job quite well. Here is an example of how I executed the html2pdf-maven-plugin (https://bitbucket.org/prunge/html2pdf-maven-plugin/wiki/Home) from my own maven plugin.
The generatePdf(...) method is called from the execute() method of your Mojo class. The getMvnPlugin(...) method is where I specified the plugin groupId and ArtifactId and the execution details. The getPluginConfiguration(...) method is where the plugin configuration is made, specifying the parameters of the html2pdf-maven-plugin.
private void generatePdf(File inputFilePath, String outputFile) {
String inputDirectory = inputFilePath.getParent();
String inputFileName = inputFilePath.getName();
try {
Plugin html2pdfPlugin = getMvnPlugin(outputFile, inputFile, inputFileName);
MojoExecutor.ExecutionEnvironment executionEnvironment = MojoExecutor.executionEnvironment(mavenProject, mavenSession, pluginManager);
MojoExecutor.executeMojo(html2pdfPlugin, "html2pdf", getPluginConfiguration(outputFile, inputDirectory, inputFileName), executionEnvironment);
} catch (Exception ex) {
String errorMessage = String.format("Failed to generate PDF file [%s] from html file [%s].", outputFile, inputFile);
getLog().error(errorMessage, ex);
}
}
private Plugin getMvnPlugin(String outputFile, String inputDirectory, String inputFileName) {
Plugin plugin = new Plugin();
plugin.setGroupId("au.net.causal.maven.plugins");
plugin.setArtifactId("html2pdf-maven-plugin");
plugin.setVersion("2.0");
PluginExecution pluginExecution = new PluginExecution();
pluginExecution.setGoals(Collections.singletonList("html2pdf"));
pluginExecution.setId("generate-pdf");
pluginExecution.setPhase("generate-resources");
plugin.setExecutions(Collections.singletonList(pluginExecution));
plugin.setConfiguration(getPluginConfiguration(outputFile, inputDirectory, inputFileName));
return plugin;
}
private Xpp3Dom getPluginConfiguration(String outputFile, String inputDirectory, String inputFileName) {
MojoExecutor.Element outputFileElement = new MojoExecutor.Element("outputFile", outputFile);
MojoExecutor.Element includeElement = new MojoExecutor.Element("include", "**/" + inputFileName);
MojoExecutor.Element includesElement = new MojoExecutor.Element("includes", includeElement);
MojoExecutor.Element directoryElement = new MojoExecutor.Element("directory", inputDirectory);
MojoExecutor.Element htmlFileSetElement = new MojoExecutor.Element("htmlFileSet", directoryElement, includesElement);
MojoExecutor.Element htmlFileSetsElement = new MojoExecutor.Element("htmlFileSets", htmlFileSetElement);
return MojoExecutor.configuration(outputFileElement, htmlFileSetsElement);
}
The mavenProject, mavenSession and pluginManager just have to be injected in the Mojo class using the org.apache.maven.plugins.annotations.Component.
#Component
private MavenProject mavenProject;
#Component
private MavenSession mavenSession;
#Component
private BuildPluginManager pluginManager;

How to Run PDI Transformation with Database from Java?

I am trying to run a PDI transformation involving database (any database, but noSQL one are more preferred) from Java.
I've tried using mongodb and cassandradb and got missing plugins, I've already asked here: Running PDI Kettle on Java - Mongodb Step Missing Plugins, but no one replied yet.
I've tried switching to SQL DB using PostgreSQL too, but it still doesn't work. From the research I did, I think it was because I didn't connect the database from the Java thoroughly, yet I haven't found any tutorial or direction that works for me. I've tried following directions from this blog : http://ameethpaatil.blogspot.co.id/2010/11/pentaho-data-integration-java-maven.html : but still got some problems about repository (because I don't have any and there seems to be required).
The transformations are fine when I run it from Spoon. It only failed when I run it from Java.
Can anyone help me how to run PDI transformation involving database? Where did I go wrong?
Is anyone ever succeeded in running PDI transformation from involving either noSQL and SQL database? what DB did you use?
I'm sorry if I asked too many questions, I am so desperate. any kind of information will be very appreciated. Thank you.

Executing PDI Jobs from Java is pretty straight forward. You just need to import all the necessary jar files (for the databases) and then call in the kettle class. The best way is obviously to use "Maven" to control the dependency. In the maven pom.xml file, just call the database drivers.
A Sample Maven file would be something like below, assuming you are using pentaho v5.0.0GA and Database as PostgreSQL:
<dependencies>
<!-- Pentaho Kettle Core dependencies development -->
<dependency>
<groupId>pentaho-kettle</groupId>
<artifactId>kettle-core</artifactId>
<version>5.0.0.1</version>
</dependency>
<dependency>
<groupId>pentaho-kettle</groupId>
<artifactId>kettle-dbdialog</artifactId>
<version>5.0.0.1</version>
</dependency>
<dependency>
<groupId>pentaho-kettle</groupId>
<artifactId>kettle-engine</artifactId>
<version>5.0.0.1</version>
</dependency>
<dependency>
<groupId>pentaho-kettle</groupId>
<artifactId>kettle-ui-swt</artifactId>
<version>5.0.0.1</version>
</dependency>
<dependency>
<groupId>pentaho-kettle</groupId>
<artifactId>kettle5-log4j-plugin</artifactId>
<version>5.0.0.1</version>
</dependency>
<!-- The database dependency files. Use it if your kettle file involves database connectivity. -->
<dependency>
<groupId>postgresql</groupId>
<artifactId>postgresql</artifactId>
<version>9.1-902.jdbc4</version>
</dependency>
You can check my blog for more. It works for database connections.
Hope this helps :)

I had the same problem in a application using the pentaho libraries. I resolved the problem with this code:
The singleton to init Kettle:
import org.pentaho.di.core.KettleEnvironment;
import org.pentaho.di.core.exception.KettleException;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
/**
* Inicia as configurações das variáveis de ambiente do kettle
*
* #author Marcos Souza
* #version 1.0
*
*/
public class AtomInitKettle {
private static final Logger LOGGER = LoggerFactory.getLogger(AtomInitKettle.class);
private AtomInitKettle() throws KettleException {
try {
LOGGER.info("Iniciando kettle");
KettleJNDI.protectSystemProperty();
KettleEnvironment.init();
LOGGER.info("Kettle iniciado com sucesso");
} catch (Exception e) {
LOGGER.error("Message: {} Cause {} ", e.getMessage(), e.getCause());
}
}
}
And the code that saved me:
import java.io.File;
import java.util.Properties;
import org.pentaho.di.core.Const;
import org.pentaho.di.core.exception.KettleException;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
public class KettleJNDI {
private static final Logger LOGGER = LoggerFactory.getLogger(KettleJNDI.class);
public static final String SYS_PROP_IC = "java.naming.factory.initial";
private static boolean init = false;
private KettleJNDI() {
}
public static void initJNDI() throws KettleException {
String path = Const.JNDI_DIRECTORY;
LOGGER.info("Kettle Const.JNDI_DIRECTORY= {}", path);
if (path == null || path.equals("")) {
try {
File file = new File("simple-jndi");
path = file.getCanonicalPath();
} catch (Exception e) {
throw new KettleException("Error initializing JNDI", e);
}
Const.JNDI_DIRECTORY = path;
LOGGER.info("Kettle null > Const.JNDI_DIRECTORY= {}", path);
}
System.setProperty("java.naming.factory.initial", "org.osjava.sj.SimpleContextFactory");
System.setProperty("org.osjava.sj.root", path);
System.setProperty("org.osjava.sj.delimiter", "/");
}
public static void protectSystemProperty() {
if (init) {
return;
}
System.setProperties(new ProtectionProperties(SYS_PROP_IC, System.getProperties()));
if (LOGGER.isInfoEnabled()) {
LOGGER.info("Kettle System Property Protector: System.properties replaced by custom properies handler");
}
init = true;
}
public static class ProtectionProperties extends Properties {
private static final long serialVersionUID = 1L;
private final String protectedKey;
public ProtectionProperties(String protectedKey, Properties prprts) {
super(prprts);
if (protectedKey == null) {
throw new IllegalArgumentException("Properties protection was provided a null key");
}
this.protectedKey = protectedKey;
}
#Override
public synchronized Object setProperty(String key, String value) {
// We forbid changes in general, but do it silent ...
if (protectedKey.equals(key)) {
if (LOGGER.isDebugEnabled()) {
LOGGER.debug("Kettle System Property Protector: Protected change to '" + key + "' with value '" + value + "'");
}
return super.getProperty(protectedKey);
}
return super.setProperty(key, value);
}
}
}

I think your problem is with connection of data base. You can configure in transformation and do not need use JNDI.
public class DatabaseMetaStep {
private static final Logger LOGGER = LoggerFactory.getLogger(DatabaseMetaStep.class);
/**
* Adds the configurations of access to the database
*
* #return
*/
public static DatabaseMeta createDatabaseMeta() {
DatabaseMeta databaseMeta = new DatabaseMeta();
LOGGER.info("Carregando informacoes de acesso");
databaseMeta.setHostname("localhost");
databaseMeta.setName("stepName");
databaseMeta.setUsername("user");
databaseMeta.setPassword("password");
databaseMeta.setDBPort("port");
databaseMeta.setDBName("database");
databaseMeta.setDatabaseType("MonetDB"); // sql, MySql ...
databaseMeta.setAccessType(DatabaseMeta.TYPE_ACCESS_NATIVE);
return databaseMeta;
}
}
Then you need set the databaseMeta to Transmeta
DatabaseMeta databaseMeta = DatabaseMetaStep.createDatabaseMeta();
TransMeta transMeta = new TransMeta();
transMeta.setUsingUniqueConnections(true);
transMeta.setName("ransmetaNeame");
List<DatabaseMeta> databases = new ArrayList<>();
databases.add(databaseMeta);
transMeta.setDatabases(databases);

I tried your code with a "tranformation without jndi" and works!
But I needed add this repository in my pom.xml:
<repositories>
<repository>
<id>pentaho-releases</id>
<url>http://repository.pentaho.org/artifactory/repo/</url>
</repository>
</repositories>
Also when I try with a datasource I have this error : Cannot instantiate class: org.osjava.sj.SimpleContextFactory [Root exception is java.lang.ClassNotFoundException: org.osjava.sj.SimpleContextFactory]
Complete log here:
https://gist.github.com/eb15f8545e3382351e20.git
[FIX] : Add this dependency :
<dependency>
<groupId>pentaho</groupId>
<artifactId>simple-jndi</artifactId>
<version>1.0.1</version>
</dependency>
After that a new error occurs:
transformation_with_jndi - Dispatching started for transformation [transformation_with_jndi]
Table input.0 - ERROR (version 5.0.0.1.19046, build 1 from 2013-09-11_13-51-13 by buildguy) : An error occurred, processing will be stopped:
Table input.0 - Error occured while trying to connect to the database
Table input.0 - java.io.File parameter must be a directory. [D:\opt\workspace-eclipse\invoke-ktr-jndi\simple-jndi]
Complete log : https://gist.github.com/jrichardsz/9d74c7263f3567ac4b45
[EXPLANATION] This is due to in
KettleEnvironment.init();
https://github.com/jrichardsz/pentaho-pdi-spoon-usefull-templates/blob/master/running-etl-transformation-using-java/researching-pentaho-classes/KettleEnvironment.java
There is a inicialization :
if (simpleJndi) {
JndiUtil.initJNDI();
}
And in JndiUtil:
String path = Const.JNDI_DIRECTORY;
if ((path == null) || (path.equals("")))
https://github.com/jrichardsz/pentaho-pdi-spoon-usefull-templates/blob/master/running-etl-transformation-using-java/researching-pentaho-classes/JndiUtil.java
And in Const class :
public static String JNDI_DIRECTORY = NVL(System.getProperty("KETTLE_JNDI_ROOT"), System.getProperty("org.osjava.sj.root"));
https://github.com/jrichardsz/pentaho-pdi-spoon-usefull-templates/blob/master/running-etl-transformation-using-java/researching-pentaho-classes/Const.java
So wee need set this variable KETTLE_JNDI_ROOT
[FIX] A small change in your example : Just add this
System.setProperty("KETTLE_JNDI_ROOT", jdbcPropertiesPath);
before
KettleEnvironment.init();
A complete example based in your code :
import java.io.File;
import org.pentaho.di.core.KettleEnvironment;
import org.pentaho.di.core.exception.KettleException;
import org.pentaho.di.trans.Trans;
import org.pentaho.di.trans.TransMeta;
public class ExecuteSimpleTransformationWithJndiDatasource {
public static void main(String[] args) {
String resourcesPath = (new File(".").getAbsolutePath())+"\\src\\main\\resources";
String ktr_path = resourcesPath+"\\transformation_with_jndi.ktr";
//KETTLE_JNDI_ROOT could be the simple-jndi folder in your pdi or spoon home.
//in this example, is the resources folder
String jdbcPropertiesPath = resourcesPath;
try {
/**
* Initialize the Kettle Enviornment
*/
System.setProperty("KETTLE_JNDI_ROOT", jdbcPropertiesPath);
KettleEnvironment.init();
/**
* Create a trans object to properly assign the ktr metadata.
*
* #filedb: The ktr file path to be executed.
*
*/
TransMeta metadata = new TransMeta(ktr_path);
Trans trans = new Trans(metadata);
// Execute the transformation
trans.execute(null);
trans.waitUntilFinished();
// checking for errors
if (trans.getErrors() > 0) {
System.out.println("Erroruting Transformation");
}
} catch (KettleException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
For a complete example check my github channel:
https://github.com/jrichardsz/pentaho-pdi-spoon-usefull-templates/tree/master/running-etl-transformation-using-java/invoke-transformation-from-java-jndi/src/main/resources

How to list the files inside a JAR file?

I have this code which reads all the files from a directory.
File textFolder = new File("text_directory");
File [] texFiles = textFolder.listFiles( new FileFilter() {
public boolean accept( File file ) {
return file.getName().endsWith(".txt");
}
});
It works great. It fills the array with all the files that end with ".txt" from directory "text_directory".
How can I read the contents of a directory in a similar fashion within a JAR file?
So what I really want to do is, to list all the images inside my JAR file, so I can load them with:
ImageIO.read(this.getClass().getResource("CompanyLogo.png"));
(That one works because the "CompanyLogo" is "hardcoded" but the number of images inside the JAR file could be from 10 to 200 variable length.)
EDIT
So I guess my main problem would be: How to know the name of the JAR file where my main class lives?
Granted I could read it using java.util.Zip.
My Structure is like this:
They are like:
my.jar!/Main.class
my.jar!/Aux.class
my.jar!/Other.class
my.jar!/images/image01.png
my.jar!/images/image02a.png
my.jar!/images/imwge034.png
my.jar!/images/imagAe01q.png
my.jar!/META-INF/manifest
Right now I'm able to load for instance "images/image01.png" using:
ImageIO.read(this.getClass().getResource("images/image01.png));
But only because I know the file name, for the rest I have to load them dynamically.

CodeSource src = MyClass.class.getProtectionDomain().getCodeSource();
if (src != null) {
URL jar = src.getLocation();
ZipInputStream zip = new ZipInputStream(jar.openStream());
while(true) {
ZipEntry e = zip.getNextEntry();
if (e == null)
break;
String name = e.getName();
if (name.startsWith("path/to/your/dir/")) {
/* Do something with this entry. */
...
}
}
}
else {
/* Fail... */
}
Note that in Java 7, you can create a FileSystem from the JAR (zip) file, and then use NIO's directory walking and filtering mechanisms to search through it. This would make it easier to write code that handles JARs and "exploded" directories.

Code that works for both IDE's and .jar files:
import java.io.*;
import java.net.*;
import java.nio.file.*;
import java.util.*;
import java.util.stream.*;
public class ResourceWalker {
public static void main(String[] args) throws URISyntaxException, IOException {
URI uri = ResourceWalker.class.getResource("/resources").toURI();
Path myPath;
if (uri.getScheme().equals("jar")) {
FileSystem fileSystem = FileSystems.newFileSystem(uri, Collections.<String, Object>emptyMap());
myPath = fileSystem.getPath("/resources");
} else {
myPath = Paths.get(uri);
}
Stream<Path> walk = Files.walk(myPath, 1);
for (Iterator<Path> it = walk.iterator(); it.hasNext();){
System.out.println(it.next());
}
}
}

erickson's answer worked perfectly:
Here's the working code.
CodeSource src = MyClass.class.getProtectionDomain().getCodeSource();
List<String> list = new ArrayList<String>();
if( src != null ) {
URL jar = src.getLocation();
ZipInputStream zip = new ZipInputStream( jar.openStream());
ZipEntry ze = null;
while( ( ze = zip.getNextEntry() ) != null ) {
String entryName = ze.getName();
if( entryName.startsWith("images") && entryName.endsWith(".png") ) {
list.add( entryName );
}
}
}
webimages = list.toArray( new String[ list.size() ] );
And I have just modify my load method from this:
File[] webimages = ...
BufferedImage image = ImageIO.read(this.getClass().getResource(webimages[nextIndex].getName() ));
To this:
String [] webimages = ...
BufferedImage image = ImageIO.read(this.getClass().getResource(webimages[nextIndex]));

I would like to expand on acheron55's answer, since it is a very non-safe solution, for several reasons:
It doesn't close the FileSystem object.
It doesn't check if the FileSystem object already exists.
It isn't thread-safe.
This is somewhat a safer solution:
private static ConcurrentMap<String, Object> locks = new ConcurrentHashMap<>();
public void walk(String path) throws Exception {
URI uri = getClass().getResource(path).toURI();
if ("jar".equals(uri.getScheme()) {
safeWalkJar(path, uri);
} else {
Files.walk(Paths.get(path));
}
}
private void safeWalkJar(String path, URI uri) throws Exception {
synchronized (getLock(uri)) {
// this'll close the FileSystem object at the end
try (FileSystem fs = getFileSystem(uri)) {
Files.walk(fs.getPath(path));
}
}
}
private Object getLock(URI uri) {
String fileName = parseFileName(uri);
locks.computeIfAbsent(fileName, s -> new Object());
return locks.get(fileName);
}
private String parseFileName(URI uri) {
String schemeSpecificPart = uri.getSchemeSpecificPart();
return schemeSpecificPart.substring(0, schemeSpecificPart.indexOf("!"));
}
private FileSystem getFileSystem(URI uri) throws IOException {
try {
return FileSystems.getFileSystem(uri);
} catch (FileSystemNotFoundException e) {
return FileSystems.newFileSystem(uri, Collections.<String, String>emptyMap());
}
}
There's no real need to synchronize over the file name; one could simply synchronize on the same object every time (or make the method synchronized), it's purely an optimization.
I would say that this is still a problematic solution, since there might be other parts in the code that use the FileSystem interface over the same files, and it could interfere with them (even in a single threaded application).
Also, it doesn't check for nulls (for instance, on getClass().getResource().
This particular Java NIO interface is kind of horrible, since it introduces a global/singleton non thread-safe resource, and its documentation is extremely vague (a lot of unknowns due to provider specific implementations). Results may vary for other FileSystem providers (not JAR). Maybe there's a good reason for it being that way; I don't know, I haven't researched the implementations.

So I guess my main problem would be, how to know the name of the jar where my main class lives.
Assuming that your project is packed in a Jar (not necessarily true!), you can use ClassLoader.getResource() or findResource() with the class name (followed by .class) to get the jar that contains a given class. You'll have to parse the jar name from the URL that gets returned (not that tough), which I will leave as an exercise for the reader :-)
Be sure to test for the case where the class is not part of a jar.

I've ported acheron55's answer to Java 7 and closed the FileSystem object. This code works in IDE's, in jar files and in a jar inside a war on Tomcat 7; but note that it does not work in a jar inside a war on JBoss 7 (it gives FileSystemNotFoundException: Provider "vfs" not installed, see also this post). Furthermore, like the original code, it is not thread safe, as suggested by errr. For these reasons I have abandoned this solution; however, if you can accept these issues, here is my ready-made code:
import java.io.IOException;
import java.net.*;
import java.nio.file.*;
import java.nio.file.attribute.BasicFileAttributes;
import java.util.Collections;
public class ResourceWalker {
public static void main(String[] args) throws URISyntaxException, IOException {
URI uri = ResourceWalker.class.getResource("/resources").toURI();
System.out.println("Starting from: " + uri);
try (FileSystem fileSystem = (uri.getScheme().equals("jar") ? FileSystems.newFileSystem(uri, Collections.<String, Object>emptyMap()) : null)) {
Path myPath = Paths.get(uri);
Files.walkFileTree(myPath, new SimpleFileVisitor<Path>() {
#Override
public FileVisitResult visitFile(Path file, BasicFileAttributes attrs) throws IOException {
System.out.println(file);
return FileVisitResult.CONTINUE;
}
});
}
}
}

Here is an example of using Reflections library to recursively scan classpath by regex name pattern augmented with a couple of Guava perks to to fetch resources contents:
Reflections reflections = new Reflections("com.example.package", new ResourcesScanner());
Set<String> paths = reflections.getResources(Pattern.compile(".*\\.template$"));
Map<String, String> templates = new LinkedHashMap<>();
for (String path : paths) {
log.info("Found " + path);
String templateName = Files.getNameWithoutExtension(path);
URL resource = getClass().getClassLoader().getResource(path);
String text = Resources.toString(resource, StandardCharsets.UTF_8);
templates.put(templateName, text);
}
This works with both jars and exploded classes.

Here's a method I wrote for a "run all JUnits under a package". You should be able to adapt it to your needs.
private static void findClassesInJar(List<String> classFiles, String path) throws IOException {
final String[] parts = path.split("\\Q.jar\\\\E");
if (parts.length == 2) {
String jarFilename = parts[0] + ".jar";
String relativePath = parts[1].replace(File.separatorChar, '/');
JarFile jarFile = new JarFile(jarFilename);
final Enumeration<JarEntry> entries = jarFile.entries();
while (entries.hasMoreElements()) {
final JarEntry entry = entries.nextElement();
final String entryName = entry.getName();
if (entryName.startsWith(relativePath)) {
classFiles.add(entryName.replace('/', File.separatorChar));
}
}
}
}
Edit:
Ah, in that case, you might want this snippet as well (same use case :) )
private static File findClassesDir(Class<?> clazz) {
try {
String path = clazz.getProtectionDomain().getCodeSource().getLocation().getFile();
final String codeSourcePath = URLDecoder.decode(path, "UTF-8");
final String thisClassPath = new File(codeSourcePath, clazz.getPackage().getName().repalce('.', File.separatorChar));
} catch (UnsupportedEncodingException e) {
throw new AssertionError("impossible", e);
}
}

Just to mention that if you are already using Spring, you can take advantage of the PathMatchingResourcePatternResolver.
For instance to get all the PNG files from a images folder in resources
ClassLoader cl = this.getClass().getClassLoader();
ResourcePatternResolver resolver = new PathMatchingResourcePatternResolver(cl);
Resource[] resources = resolver.getResources("images/*.png");
for (Resource r: resources){
logger.info(r.getFilename());
// From your example
// ImageIO.read(cl.getResource("images/" + r.getFilename()));
}

A jar file is just a zip file with a structured manifest. You can open the jar file with the usual java zip tools and scan the file contents that way, inflate streams, etc. Then use that in a getResourceAsStream call, and it should be all hunky dory.
EDIT / after clarification
It took me a minute to remember all the bits and pieces and I'm sure there are cleaner ways to do it, but I wanted to see that I wasn't crazy. In my project image.jpg is a file in some part of the main jar file. I get the class loader of the main class (SomeClass is the entry point) and use it to discover the image.jpg resource. Then some stream magic to get it into this ImageInputStream thing and everything is fine.
InputStream inputStream = SomeClass.class.getClassLoader().getResourceAsStream("image.jpg");
JPEGImageReaderSpi imageReaderSpi = new JPEGImageReaderSpi();
ImageReader ir = imageReaderSpi.createReaderInstance();
ImageInputStream iis = new MemoryCacheImageInputStream(inputStream);
ir.setInput(iis);
....
ir.read(0); //will hand us a buffered image

Given an actual JAR file, you can list the contents using JarFile.entries(). You will need to know the location of the JAR file though - you can't just ask the classloader to list everything it could get at.
You should be able to work out the location of the JAR file based on the URL returned from ThisClassName.class.getResource("ThisClassName.class"), but it may be a tiny bit fiddly.

Some time ago I made a function that gets classess from inside JAR:
public static Class[] getClasses(String packageName)
throws ClassNotFoundException{
ArrayList<Class> classes = new ArrayList<Class> ();
packageName = packageName.replaceAll("\\." , "/");
File f = new File(jarName);
if(f.exists()){
try{
JarInputStream jarFile = new JarInputStream(
new FileInputStream (jarName));
JarEntry jarEntry;
while(true) {
jarEntry=jarFile.getNextJarEntry ();
if(jarEntry == null){
break;
}
if((jarEntry.getName ().startsWith (packageName)) &&
(jarEntry.getName ().endsWith (".class")) ) {
classes.add(Class.forName(jarEntry.getName().
replaceAll("/", "\\.").
substring(0, jarEntry.getName().length() - 6)));
}
}
}
catch( Exception e){
e.printStackTrace ();
}
Class[] classesA = new Class[classes.size()];
classes.toArray(classesA);
return classesA;
}else
return null;
}

public static ArrayList<String> listItems(String path) throws Exception{
InputStream in = ClassLoader.getSystemClassLoader().getResourceAsStream(path);
byte[] b = new byte[in.available()];
in.read(b);
String data = new String(b);
String[] s = data.split("\n");
List<String> a = Arrays.asList(s);
ArrayList<String> m = new ArrayList<>(a);
return m;
}

There are two very useful utilities both called JarScan:
www.inetfeedback.com/jarscan
jarscan.dev.java.net
See also this question: JarScan, scan all JAR files in all subfolders for specific class

The most robust mechanism for listing all resources in the classpath is currently to use this pattern with ClassGraph, because it handles the widest possible array of classpath specification mechanisms, including the new JPMS module system. (I am the author of ClassGraph.)
How to know the name of the JAR file where my main class lives?
URI mainClasspathElementURI;
try (ScanResult scanResult = new ClassGraph().whitelistPackages("x.y.z")
.enableClassInfo().scan()) {
mainClasspathElementURI =
scanResult.getClassInfo("x.y.z.MainClass").getClasspathElementURI();
}
How can I read the contents of a directory in a similar fashion within a JAR file?
List<String> classpathElementResourcePaths;
try (ScanResult scanResult = new ClassGraph().overrideClasspath(mainClasspathElementURI)
.scan()) {
classpathElementResourcePaths = scanResult.getAllResources().getPaths();
}
There are lots of other ways to deal with resources too.

One more for the road that's a bit more flexible for matching specific filenames because it uses wildcard globbing. In a functional style this could resemble:
import java.io.IOException;
import java.net.URISyntaxException;
import java.nio.file.FileSystem;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.util.function.Consumer;
import static java.nio.file.FileSystems.getDefault;
import static java.nio.file.FileSystems.newFileSystem;
import static java.util.Collections.emptyMap;
/**
* Responsible for finding file resources.
*/
public class ResourceWalker {
/**
* Globbing pattern to match font names.
*/
public static final String GLOB_FONTS = "**.{ttf,otf}";
/**
* #param directory The root directory to scan for files matching the glob.
* #param c The consumer function to call for each matching path
* found.
* #throws URISyntaxException Could not convert the resource to a URI.
* #throws IOException Could not walk the tree.
*/
public static void walk(
final String directory, final String glob, final Consumer<Path> c )
throws URISyntaxException, IOException {
final var resource = ResourceWalker.class.getResource( directory );
final var matcher = getDefault().getPathMatcher( "glob:" + glob );
if( resource != null ) {
final var uri = resource.toURI();
final Path path;
FileSystem fs = null;
if( "jar".equals( uri.getScheme() ) ) {
fs = newFileSystem( uri, emptyMap() );
path = fs.getPath( directory );
}
else {
path = Paths.get( uri );
}
try( final var walk = Files.walk( path, 10 ) ) {
for( final var it = walk.iterator(); it.hasNext(); ) {
final Path p = it.next();
if( matcher.matches( p ) ) {
c.accept( p );
}
}
} finally {
if( fs != null ) { fs.close(); }
}
}
}
}
Consider parameterizing the file extensions, left an exercise for the reader.
Be careful with Files.walk. According to the documentation:
This method must be used within a try-with-resources statement or similar control structure to ensure that the stream's open directories are closed promptly after the stream's operations have completed.
Likewise, newFileSystem must be closed, but not before the walker has had a chance to visit the file system paths.

Just a different way of listing/reading files from a jar URL and it does it recursively for nested jars
https://gist.github.com/trung/2cd90faab7f75b3bcbaa
URL urlResource = Thead.currentThread().getContextClassLoader().getResource("foo");
JarReader.read(urlResource, new InputStreamCallback() {
#Override
public void onFile(String name, InputStream is) throws IOException {
// got file name and content stream
}
});