We Keep Coding

Cannot load custom File System on Flink's shadow jar

I needed some metadata on my S3 objects, so I had to override the S3 file system provided by flink.
I followed this guide to the letter and now I have a custom file system which works on my local machine, when I run my application in the IDE.
Now I am trying to use it on a local kafka cluster OR on my docker deployment, and I keep getting this error Could not find a file system implementation for scheme 's3c'. The scheme is not directly supported by Flink and no Hadoop file system to support this scheme could be loaded.
I package my application using shadowJar, using the following configuration:
shadowJar {
configurations = [project.configurations.flinkShadowJar]
mainClassName = "dev.vox.collect.delivery.Application"
mergeServiceFiles()
}
I have my service file in src/main/resources/META-INF/services/org.apache.flink.core.fs.FileSystemFactory that contains a single line with the namespace and name of my factory :dev.vox.collect.delivery.filesystem.S3CFileSystemFactory
If I unzip my shadowJar I can see in its org.apache.flink.core.fs.FileSystemFactory file it has both my factory and the others declared by Flink, which should be correct:
dev.vox.collect.delivery.filesystem.S3CFileSystemFactory
org.apache.flink.fs.s3hadoop.S3FileSystemFactory
org.apache.flink.fs.s3hadoop.S3AFileSystemFactory
When I use the S3 file system provided by flink everything works, it is just mine that does not.
I am assuming the service loader is not loading my factory, either because it does not find it or because it is not declared correctly.
How can I make it work? Am I missing something?

Java EE WebApp ServiceLoader loading external jar as plugin

I have a very simple POC setup where I deploy a JEE7 webapp on a wildfly 9.
Via a jaxRs Resource endpoint I can trigger a "plugin loader".
The PluginLoader does use a directory and scans for jar files in the directory, which URLs then will be fed into a URLClassLoader.
Afterwards I use the ServiceLoader to load implmementations of a simple interface from those URLs.
When the ServiceLoader starts iterating over the found implementations, I get this error:
Caused by: java.util.ServiceConfigurationError: com.test.MyIface: Provider com.test.MyImpl not a subtype
The structure is also very simple:
MyIface.jar is the interface.
MyImpl.jar is a implementation of MyIface, while it contains a META-INF/services file with the correct naming and content for MyIface..
The webapp itself only knows MyIFace of course.
In JavaSE using a simple main entry point and invoking the loader from there, everything works.
In JavaEE the services file seems to be ignored though..at least that is what I get from the exception.
I put it in src/main/resources/META-INF/services
and in src/main/resource/WEB-INF/classes/META-INF/services (as I read that already in context with SPI and webapps)

In order for this to work, the following 2 steps must be followed:
Instantiate a ClassLoader (the stock URLClassLoader will do) that knows both the targeted jar AND the classloader of the web application.
It needs to know the targeted jar obviously to load the service implementation
It needs to have the classloader of the web app as parent so that all the classloaders share the interface class; otherwise, even if the custom classloader loads the interface, you will run to ClassCastExceptions like "MyIface is not an instance of MyIface"
Specify the classloader you created using the ServiceLoader.load(Class, ClassLoader) method

Using Spring #PropertySource in a Maven submodule

In a Spring-boot application, I was having a single module and I was able to inject a configuration file, e.g. "my.properties", that was located in src/main/resources as follows:
#Configuration
#PropertySource("/my.properties")
public class MyConf{
}
Everything was ok, but then I created submodules and now I moved that configuration file in a submodule. When I start the main application I gedt the following exception:
org.springframework.beans.factory.BeanDefinitionStoreException: Failed to parse configuration class [com.myapp.MainApplication]; nested exception is java.io.FileNotFoundException: Could not open ServletContext resource [/home/jeanvaljean/workspace/mainmodule/secondarymodule/my.properties]
As I see, I can solve the issue by writing
#PropertySource("/src/main/resources/my.properties")
Doing this, the path is correct and the file can be loaded.
Anyway, that is an horrible solution, and I'm pretty sure that there is a more elegant and flexible alternative. Any solution?

Spring has a few different implementations of how to find a resource. By using the prefix classpath: you are telling Spring to search for the resource in all the classpath, rather than in the classes that are bundled with your application.
Depending on the ApplicationContenxt, Spring will use a different default Resource class. It looks like in your case, Spring was instantiating a FileSystemResource, which only finds files available on the filesystem with either relative or absolute paths (but not inside jars!). My rule of thumb is to never prefix something if it's in the same module/component/jar, and always prefix it with classpath: if I know it's in a different module/component/jar (some people get mad at this :).
You can read a more in the Spring Documentation - Resources

Failed to import bean definitions from URL location [classpath:applicationContext-core.xml]

I'm working on a java project with spring on eclipse using Maven, and running on a Tomcat server v6.0. Everything was working fine since yesterday morning.
Here his my problem : I'm building my project, I got a build success. Then I start my Tomcat server and got this error :
org.springframework.beans.factory.parsing.BeanDefinitionParsingException: Configuration problem: Failed to import bean definitions from URL location [classpath:applicationContext- core.xml]
Offending resource: ServletContext resource [/WEB-INF/applicationContext.xml]; nested exception is org.springframework.beans.factory.BeanDefinitionStoreException: IOException parsing XML document from class path resource [applicationContext-core.xml]; nested exception is java.io.FileNotFoundException: class path resource [applicationContext-core.xml] cannot be opened because it does not exist
I found out similar problem on some website but none of them give me a solution that worked for me.
It seems that eclipse isn't able to find applicationContext-core.xml when I'm doing this :
<import resource="classpath:applicationContext-core.xml" />
However, I do have the needed jar file nad-core-0.0.1-SNAPSHOT.jar in WEB-INF/lib containing applicationContext-core.xml.
I even tried to add it manually to the classpath but I was still having the same problem.
I keep on looking for a solution, when suddendly it work again once after restarting Eclipse and building while Eclipse was still updating indexes and my project was having this strange status Hg status pending instead of default. Surprised by this result I decide to build again my project after restarting Eclipse and I got the error again and I enable to make it work again. It's quite annoying...
This looks to be a really random problem.
Thanks a lot for your help :)

As you've not specified you web application structure. I assume you've a simple web application at hand with the following structures
webapp
WEB-INF/classes/applicationContext.xml
WEB-INF/lib/nad-core-0.0.1-SNAPSHOT.jar/applicationContext-core.xml
Application context.xml refers to the applicationContext-core.xml file using the import tag. I did encounter a similar situation in my web application, here're the check lists that you should go through and may be one of them can apply to your situation.
Check the generated snapshot jar file for the applicationContext-core.xml file and make sure it is in the root directory of the jar. As silly as it sounds, this was the root cause of the issue I faced in my deployment.
Make sure your Maven Pom.xml file is configured to include this XML file from the resources folder. You can use the resource tags in the build phase of Maven to package them within the jar file itself.
You can try removing the import tag from application context.xml file and instead load both of them from Spring's webapplication context itself.
Add a context loader listener class from spring org.springframework.web.context.ContextLoaderListener
Add context-param contextConfigLocation with value classpath:applicationContext-core.xml,classpath:applicationContext.xml. Spring has the ability to dynamically sort out the dependencies before initiating the bean factory.
Hope this check list helps.

I get pretty much the same config, six years later, I got the same error.
I also restart Eclipse, and it solved the issue.

Grails config of Spring beans in different files

Grails have cofig for spring bean called resources.groovy. And as i understand from docs it allows you to include another file, using loadBeans(%path%)
I'm tried with this:
println 'loading application config ...'
// Place your Spring DSL code here
beans = {
loadBeans("classpath:security") //i'm tried with "spring/security" and "spring/security.groovy" also
}
but when grails is running it log following error:
Caused by: org.springframework.beans.factory.parsing.BeanDefinitionParsingException: Configuration problem: Error evaluating bean definition script: class path resource [security] cannot be opened because it does not exist
Offending resource: class path resource [security]; nested exception is java.io.FileNotFoundException: class path resource [security] cannot be opened because it does not exist
at grails.spring.BeanBuilder.loadBeans(BeanBuilder.java:470)
at grails.spring.BeanBuilder.loadBeans(BeanBuilder.java:424)
at resources$_run_closure1.doCall(resources.groovy:13)
at resources$_run_closure1.doCall(resources.groovy)
... 45 more
Script security.groovy is exists at grails-app/conf/spring and compiled by grails maven plugin into target/classes/security.class.
Directory target/resources/spring is empty at this time
How i can configure Grails or grails-maven-plugin to copy this config files, not compile it into classes?
p.s. this problem also presents when i try to include config scripts using grails.config.locations = [ %path% ] inside conf/Config.groovy, my groovy scripts compiles into classes and because of it, grails config builder can't find them :(

Did you try:
println 'loading application config ...'
// Place your Spring DSL code here
beans = {
loadBeans("classpath:*security.groovy")
}
(this should load all Groovy files on the classpath ending with security.groovy and parse them into bean definitions).
Update: Found an interesting thread with this message as reference and my understanding is that one trick is to use ant in scripts/_Events.groovy to copy the .groovy file to the classesDirPath dir and then simply use:
beans = {
// load spring-beans for db-access via spring-jdbc-template
loadBeans('security.groovy')
// load some other spring-beans
...
}
This looks like a hack to get things working in both the war and when running run-app though. Not sure how things "should" be done (if this even makes sense).