We Keep Coding

PyFlink "pipeline.classpaths" vs $FLINK_HOME/lib

What is the difference between class loading classes passed as part of PyFlink pipeline.classpath config and putting them into a $FLINK_HOME\lib directory?
When I want to use flink-sql-connector-kafka-*.jar it works fine just passing it using pipeline.classpath but when I want to use something that has some external dependencies like flink-avro-*.jar that needs avro-*.jar jars. It seems to load flink-avro-*.jar but it looks like it fails to load avro-*.jar and throws:
java.lang.NoClassDefFoundError: Could not initialize class org.apache.avro.SchemaBuilder
When I am dding avro-*.jar to $FLINK_HOME\lib it works just fine.

NoClassDefFoundError and ClassNotFoundException are different
java.lang.ClassNotFoundException This exception indicates that the
class was not found on the classpath. This indicates that we were
trying to load the class definition, and the class did not exist on
the classpath.
java.lang.NoClassDefFoundError This exception indicates that the JVM
looked in its internal class definition data structure for the
definition of a class and did not find it. This is different than
saying that it could not be loaded from the classpath. The point is,
a NoClassDefFoundError is not necessarily a classpath problem.
flink-sql-avro-*.jar is a shaded jar which will relocate the path of org.apache.flink:flink-avro org.apache.avro:avro
Judging from the NoClassDefFoundError, there may be a conflict between the avro version dependencies

Possible to ignore configuration classes on the classpath?

I have a Spring Boot application that works as expected when ran with embedded tomcat, but I noticed that if I try to run it from an existing tomcat instance that I'm using with a previous project then it fails with a NoClassDefFoundError for a class that I don't use anywhere in my application.
I noticed in the /lib directory I had a single jar that contained a few Spring annotated classes, so as a test I cleaned out the /lib directory which resolved the issue. My assumption is that Spring is seeing some of the configurations/beans/imports on the classpath due to them existing in the /lib directory and either trying to autoconfigure something on its own, or is actually trying to instantiate some of these classes.
So then my question is - assuming I can't always fully control the contents of everything on the classpath, how can I prevent errors like this from occurring?
EDIT
For a little more detail - the class not being found is DefaultCookieSerializer which is part of the spring-session-implementation dependency. It is pulled into one of the classes in the jar located in /lib, but it is not any part of my application.

Check for features provided by #EnableAutoConfiguration. You can explicitly configure set of auto-configuration classes for your application. This tutorial can be a good starting point.

You can remove the #SpringBootApplication annotation from the main class and replace it with an #ComponentScan annotation and an #Import annotation that explicitly lists only the configuration classes you want to load. For example, in a Spring boot MVC app that uses metrics, web client, rest template, Jackson, etc, I was able to replace the #SpringBootApplication annotation with below code and get it working exactly as it was before, with all functional tests passing:
#Import({ MetricsAutoConfiguration.class,
InfluxMetricsExportAutoConfiguration.class,
ServletWebServerFactoryAutoConfiguration.class,
DispatcherServletAutoConfiguration.class,
WebMvcAutoConfiguration.class,
JacksonAutoConfiguration.class,
WebClientAutoConfiguration.class,
RestTemplateAutoConfiguration.class,
RefreshAutoConfiguration.class,
ValidationAutoConfiguration.class
})
#ComponentScan

The likely culprit of mentioned exception are incompatible jars on the classpath.
As we don't know with what library you have the issue we cant tell you the exact reason, but the situation looks like that:
One of Spring-Boot autoconfiguration classes is being triggered by the presence of class on the classpath
Trigerred configuration tries to create some bean of class that is not present in the jar you have (but it is in the specific version mentioned in the Spring BOM)
Version incompatibilities may also cause MethodNotFound exceptions.
That's one of the reasons why it is good practice not to run Spring Boot applications inside the container (make jar not war), but as a runnable jar with an embedded container.
Even before Spring Boot it was preferred to take account of libraries being present on runtime classpath and mark them as provided inside your project. Having different versions of the library on a classpath may cause weird ClassCastExceptions where on both ends names match, but the rest doesn't.
You could resolve specific cases by disabling autoconfiguration that causes your issue. You can do that either by adding exclude to your #SpringBootApplication or using a property file.
Edit:
If you don't use very broad package scan (or use package name from outside of your project in package scan) in your Spring Boot application it is unlikely that Spring Boot simply imports configuration from the classpath.
As I have mentioned before it is rather some autoconfiguration that is being triggered by existence of a class in the classpath.
Theoretical solution:
You could use maven shade plugin to relocate all packages into your own package space: see docs.
The problems is you'd have face:
Defining very broad relocation pattern that would exclude JEE classes that need to be used so that container would know how to run your application.
Relocation most likely won't affect package names used as strings in the Spring Boot annotations (like annotations #PackageScan or #ConditionalOnClass). As far as I know it is not implemented yet. You'd have to implement that by yourself - maybe as some kind of shade plugin resource processor.
When relocating classes you'd have to replace package names in all relevant configuration located in the jars. Possibly also merge some of those.
You'd also have to take into account how libraries that you use, or spring uses use package names or files.
This is definitely not a trivial tasks with many traps ahead. But if done right, then it would possibly allow you to disregard what is on the containers classpath. Spring Boot would also look for classes in relocated packages, and you wouldn't have those in ordinary jars.

Resolving to correct dependencies when loading classes with custom classloaders

So I have this generic backend server that loads shaded jars in memory and then loads it through a custom Classloader.
E.g.
MyClass class = c.newInstance();
It works fine until the shaded Jar dependencies conflicts with the server classes.
E.g.
Server contains (with Custom Classloader):
com.fasterxml.jackson.jackson-databind:2.6.0
While the shaded jar contains
com.fasterxml.jackson.jackson-databind:2.9.9
When the method in the class that requires the said library e.g. class.doSomeThing(); it throws an error Caused by java.lang.NoSuchFieldError: because the loaded jackson-databind is 2.6.0 instead of 2.9.9
The question here is when the class is loaded from the shaded jar is there a way to make sure that the shaded dependencies are the ones used?

The question here is when the class is loaded from the shaded jar is there a way to make sure that the shaded dependencies are the ones used?
If you are using the default Class loader then the order of resolution will work as the order of the classpath. Within your code you can use
System.out.println(System.getProperty("system.class.path").replaceAll(":", "\n"));
And inspect the classpath. Usually such runtime environment (for example apache spark) has such features to allow you to prepend the classpath. You can check with your runtime server environment for such feature.

module-info.java compile fail with maven-compiler-plugin and automatic modules

I'm using v3.7.0 of the plugin as required and JDK 9.0.1. I have added two requires statements, each referring to a jar in the class path (automatic module). The module-info.java compiles successfully in Eclipse after I moved the jars to Modulepath. However, Maven gives me a compiler error saying one of them is missing (strangely, not the first one which is just one line before). I tried to check the automatic module name but I get an error from the commands just for this jar. What does this error mean and how do I fix it so that I can discover the proper module name?
I replaced my username in the output below. The jar in question does use a ServiceLoader but is not compiled with Java 9.
computerName:Commander-java username$ jar --file=/Users/username/.m2/repository/com/username/rcf/1.0/rcf-1.0.jar --describe-module
Unable to derive module descriptor for: /Users/username/.m2/repository/com/username/rcf/1.0/rcf-1.0.jar
Provider class com.username.rcf.server.TestCmdChain not in module
computerName:Commander-java username$ java -p /Users/username/.m2/repository/com/username/rcf/1.0/rcf-1.0.jar --list-modules
Error occurred during initialization of boot layer
java.lang.module.FindException: Unable to derive module descriptor for /Users/username/.m2/repository/com/username/rcf/1.0/rcf-1.0.jar
Caused by: java.lang.module.InvalidModuleDescriptorException: Provider class com.username.rcf.server.TestCmdChain not in module
The answer in How to deal with java keywords in auto generated module names in Java 9? has a different error related to using a Java identifier in the module name. The automatic jar name for my module should just be rcf since the jar name is rcf-1.0.jar. The error I'm getting is different also.

While deriving module description the contents of any
META-INF/services configuration files are mapped to provides
declarations.
The packages scanned for the services are the ones containing class files.
Also, the package name for individual classes is derived from their fully qualified name. From the shared logs com.username.rcf.server shall be the expected package name for the service to be provided and this shall turn into
provides x.y.z.TestCmdChainInterface with com.username.rcf.server.TestCmdChain
Seems like there is no such package com.username.rcf.server existing in your module.

Case sensitivity issue in Spring application

I have been given a Java Spring application to containerize. I am using a Tomcat8/Java8 base image. When it boots, it fails to deploy with the following error:
Caused by: org.springframework.context.annotation.ConflictingBeanDefinitionException: Annotation-specified bean name 'controller3P' for bean class [com.******.******.brand3P.controller.Controller3P] conflicts with existing, non-compatible bean definition of same name and class [com.******.******.Brand3P.controller.Controller3P]
The problem here seems to that Spring is finding 2 conflicting classes, differentiated by "brand3P" and "Brand3P".
However, in the source code, I can find no reference to a class "Brand3P", only a class "brand3P".
Can anyone explain what Spring is doing here?
(Be nice, I'm Ops, not Dev)

Package name should be all lower cased, checkout here oracle doc
Package name is not included in spring definition name.
My guess is someone changed the package name in Git, because git is by default case insensitive, so you will run into this situation, if it is this case, remove the file and create it again will fix this problem.
Hi Ops, please work with Devs together ;)

Turns out it was an issue with the build in Jenkins. The code had changed (Brand3P -> brand3P) but the older class name was hanging around in the Jenkins workspace, and being bundled into the Docker image.
We deleted the workspace, re-ran the build, and all was fine.