I'm new to Java, but I have to use it to do a small WebSocket related project.
So, I installed JDK 1.8.0 and NetBeans 8.1 on my CentOS 7 in a VirtualBox.
I added the tyrus-standalone-client-jdk 1.12 plug-in in the pom.xml to make the standalone Websocket client, and it built fine. However, I ran into the error below:
[root#cet7 ~]# java -jar "/root/NetBeansProjects/Switchclient/target/Switchclient-1.0-SNAPSHOT.jar"
Exception in thread "main" java.lang.NoClassDefFoundError: javax/websocket/ContainerProvider
at org.sample.switchclient.Switchclient.main(Switchclient.java:21)
Caused by: java.lang.ClassNotFoundException: javax.websocket.ContainerProvider
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 1 more
[root#cet7 ~]# java -version
java version "1.8.0_65"
Java(TM) SE Runtime Environment (build 1.8.0_65-b17)
Java HotSpot(TM) 64-Bit Server VM (build 25.65-b01, mixed mode)
I did a bit more searching and found that the "fully qualified classname of the container implementation of ContainerProvider must be listed in the META-INF/services/javax.websocket.ContainerProvider file in the implementation JAR file" for the ServiceLoader API according to Oracle documentation. So, I added the serviceloader-maven-plugin to the pom.xml. The result was that it did generate the META-INF/services/javax.websocket.ContainerProvider file, but without any content, and the runtime error continued to persist. I tried to modify the contents bellow manually and re-pack it into a JAR but it did not worked:
org.glassfish.tyrus.container.inmemory.InMemoryContainerProvider
org.glassfish.tyrus.client.ClientManager
I've attached the Java file and the pom.xml. I've worked for hours and haven't a clue what the issue is, so any response to this thread will be appreciated.
Thank you very much.
===========LIST1: pom.xml===========
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>org.sample</groupId>
<artifactId>Switchclient</artifactId>
<version>1.0-SNAPSHOT</version>
<packaging>jar</packaging>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
<version>2.6</version>
<configuration>
<archive>
<manifest>
<addClasspath>true</addClasspath>
<mainClass>org.sample.switchclient.Switchclient</mainClass>
</manifest>
</archive>
</configuration>
</plugin>
<plugin>
<groupId>eu.somatik.serviceloader-maven-plugin</groupId>
<artifactId>serviceloader-maven-plugin</artifactId>
<version>1.0.6</version>
<configuration>
<services>
<param>javax.websocket.ContainerProvider</param>
</services>
</configuration>
<executions>
<execution>
<goals>
<goal>generate</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
<dependencies>
<dependency>
<groupId>org.glassfish.tyrus.bundles</groupId>
<artifactId>tyrus-standalone-client-jdk</artifactId>
<version>1.12</version>
</dependency>
</dependencies>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<maven.compiler.source>1.8</maven.compiler.source>
<maven.compiler.target>1.8</maven.compiler.target>
</properties>
</project>
===========LIST2: Switchclient.java===========
package org.sample.switchclient;
import java.net.URI;
import javax.websocket.ClientEndpoint;
import javax.websocket.ContainerProvider;
import javax.websocket.OnMessage;
import javax.websocket.Session;
import javax.websocket.WebSocketContainer;
#ClientEndpoint
public class Switchclient {
#OnMessage
public void onRemoteMessage (String message) {
System.out.println("Received msg: "+message);
}
public static void main(String[] args) {
WebSocketContainer container = null;
Session session = null;
try{
container = ContainerProvider.getWebSocketContainer();
session = container.connectToServer (Switchclient.class, URI.create("ws://localhost:8080/Switchserver/"));
}catch (Exception e) {
e.printStackTrace();
}
}
}
Basically, Tyrus requires Java EE. It's the reason you have to list a lot of dependencies in pom.xml. If you use Java SE and want to keep your project small, use another different WebSocket client library that depends on only Java SE. For example, nv-websocket-client (mine).
Just add the following dependency to pom.xml,
<dependency>
<groupId>com.neovisionaries</groupId>
<artifactId>nv-websocket-client</artifactId>
<version>1.13</version>
</dependency>
then try:
import com.neovisionaries.ws.client.*;
public class Switchclient
{
public static void main(String[] args) throws Exception
{
WebSocket websocket = new WebSocketFactory()
.createSocket("ws://localhost:8080/Switchserver/")
.addListener(new WebSocketAdapter() {
#Override
public void onTextMessage(WebSocket ws, String message) {
System.out.println("Received msg: " + message);
}
})
.connect();
// Don't forget to call disconnect() after use.
// websocket.disconnect();
}
}
I'm not sure what exactly caused the problem since I kept trying and problems kept jumping out during the past day. But finally here is it:
Client dependencies:
<dependency>
<groupId>javax.websocket</groupId>
<artifactId>javax.websocket-client-api</artifactId>
<version>1.1</version>
</dependency>
<dependency>
<groupId>org.glassfish.tyrus.bundles</groupId>
<artifactId>tyrus-standalone-client</artifactId>
<version>1.12</version>
</dependency>
<dependency>
<groupId>org.glassfish.tyrus</groupId>
<artifactId>tyrus-container-grizzly-client</artifactId>
<version>1.12</version>
</dependency>
at a first glance it seems javax.websocket-client-api should be enough but finally cyber said that ContainerProvider is not impediment.
Then, all built OK. (with different java codes from my original post, I tried a lot on including the source codes, but codes themselves don't matter that match here while the environment setup matters. They mostly based on the examples of Tyrus 1.9 user guide however.)
And the run from the NetBeans by maven was OK, but when I went to use "java -jar Switchclient.jar", same/similar problem jumped out saying problem with "Endpoint".
Finally (as a last try) I copied all those tar files included in the classpath (witch was generated by maven-jar-plugin by specifying "<addClasspath>true<addClasspath>" into one directory, and also copied the generated jar file in, then it worked:
[root#cet7 neededjars]# ls
grizzly-framework-2.3.22.jar tyrus-client-1.12.jar
grizzly-http-2.3.22.jar tyrus-container-grizzly-client-1.12.jar
grizzly-http-server-2.3.22.jar tyrus-core-1.12.jar
javax.websocket-api-1.1.jar tyrus-spi-1.12.jar
javax.websocket-client-api-1.1.jar tyrus-standalone-client-1.12.jar
Switchclient-1.1-SNAPSHOT.jar
[root#cet7 neededjars]# java -jar Switchclient-1.1-SNAPSHOT.jar
Received message: Hello world
That's it, dirty and but worked. and I'm at a new start. Again, I'm really new to java (one of those non-hard-tech guys, just pick it up in case of need); and it showed me the complicity of the community based development, especially in the case the technology is relatively new. dependencies and pitfall everywhere. That's is one part of the nature I guess...
Related
My project builds successfully with groovy-eclipse-compiler, but fails without groovy-eclipse-compiler (using just javac). The build fails with an error message as given below (reported in a test class, while mocking an invocation)
java: reference to getFileResource is ambiguous
In order to debug the issue, I created a project with minimal files (given below). Though in project we have groovy source also, but I have not included them here to keep the code minimal.
The code is also pushed to git and is available at https://github.com/kaushalkumar/project-debug
My Doubt: The reported issue looks to be legitimate and I feel that groovy-eclipse-compiler must also fail, but it seems that the error is ignored. I am trying to understand what make groovy compiler to ignore it. Is it an issue in groovy compiler?
src/main/java/pkg1/IStrategy.java
package pkg1;
import java.util.Map;
public interface IStrategy {
Map<String, Object> getEnvMap();
}
src/main/java/pkg1/SharedResourceHelper.java
package pkg1;
import java.io.File;
import java.io.IOException;
import java.util.Map;
public class SharedResourceHelper {
public static File getFileResource(final String resourceName, final IStrategy strategy) throws IOException {
return getFileResource(resourceName, strategy.getEnvMap());
}
public static File getFileResource(final String resourceName, final Map<String, Object> envConfig) throws IOException {
return null;
}
}
src/test/java/pkg1/StrategyTest.java
package pkg1;
import pkg1.SharedResourceHelper;
import org.easymock.EasyMock;
import org.powermock.api.easymock.PowerMock;
import org.powermock.core.classloader.annotations.PrepareForTest;
import org.junit.Test;
import org.powermock.modules.junit4.PowerMockRunner;
import org.junit.runner.RunWith;
import java.io.File;
#PrepareForTest({SharedResourceHelper.class})
#RunWith(PowerMockRunner.class)
public class StrategyTest {
#Test
#PrepareForTest({SharedResourceHelper.class})
public void testGetFileResource() throws Exception {
PowerMock.mockStatic(SharedResourceHelper.class);
EasyMock.expect(SharedResourceHelper.getFileResource(EasyMock.anyString(), EasyMock.anyObject())).andReturn(File.createTempFile("tmp", "s"));
// EasyMock.expect(SharedResourceHelper.getFileResource("test", null)).andReturn(File.createTempFile("tmp", "s"));
}
}
/pom.xml
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>project.debug</groupId>
<artifactId>project</artifactId>
<packaging>jar</packaging>
<version>1.0-SNAPSHOT</version>
<dependencies>
<dependency>
<groupId>org.powermock</groupId>
<artifactId>powermock-api-easymock</artifactId>
<version>2.0.7</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.powermock</groupId>
<artifactId>powermock-module-junit4</artifactId>
<version>2.0.7</version>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.3</version>
<configuration>
<compilerId>groovy-eclipse-compiler</compilerId>
<source>1.8</source>
<target>1.8</target>
</configuration>
<dependencies>
<dependency>
<groupId>org.codehaus.groovy</groupId>
<artifactId>groovy-eclipse-compiler</artifactId>
<version>2.9.2-01</version>
</dependency>
<dependency>
<groupId>org.codehaus.groovy</groupId>
<artifactId>groovy-eclipse-batch</artifactId>
<version>2.4.3-01</version>
</dependency>
</dependencies>
</plugin>
</plugins>
</build>
</project>
Java version - 1.8.0_231
Maven - 3.6.2
OS - Mac 10.15.6
groovy-eclipse-compiler - 2.9.2-01
groovy-eclipse-batch - 2.4.3-01
You reference "SharedResourceHelper.getFileResource(EasyMock.anyString(), EasyMock.anyObject())" is indeed ambiguous. If you add a typecast before "EasyMock.anyObject()" you could disambiguate. And EasyMock probably provides an "any" method that you can pass a type into as well.
groovy-eclipse-compiler is based upon ecj (eclipse compiler for java) and not javac, so there are bound to be differences. It may also be that ecj has a different default error/warning level for this particular case. If you feel this should be an error, you can file a JDT bug at bugs.eclipse.org.
eric-milles gave some direction to further explore this. His input is available at https://github.com/groovy/groovy-eclipse/issues/1157.
Based on his comment, we explored the history of https://github.com/groovy/groovy-eclipse/blob/master/extras/groovy-eclipse-batch-builder/build.properties and found that the compilation issue was between 2.4.12-01 (compilation works) and 2.4.12-02 (compilation breaks- as expected), which was part of release 2.9.2.
The change happened on Aug 10, 2017 (13c1c2a#diff-c8c111c3afb6080ae6b32148caaf6a0a), with comment as "Remove codehaus references". The jdt.patch.target was targeted for e44 which is Luna. This was same for both the files.
I invested some time in exploring https://github.com/eclipse/eclipse.jdt.core, to figure out how compiler behaviour could have altered, but could not get much. Though I am not very sure, but I feel that change in groovy-eclipse-batch (between 2.4.12-01 and 2.4.12-02) might be the cause of this.
Having invested this much time, I feel that it is not worth to further debug on this to figure out the root cause as the issue is already fixed in next version(s) [2.4.12-02 and beyond].
This question already has answers here:
Unable to derive module descriptor: Provider {class X} not in module
(2 answers)
Closed 1 year ago.
Error occurred during initialization of boot layer
java.lang.module.FindException: Unable to derive module descriptor for C:\Users\admin\eclipse-workspace\Testing\lib\selenium-server-standalone.jar
Caused by: java.lang.module.InvalidModuleDescriptorException: Provider class org.eclipse.jetty.http.Http1FieldPreEncoder not in module
package Testing;
import com.thoughtworks.selenium.DefaultSelenium;
import com.thoughtworks.selenium.Selenium;
public class Testing {
public static void main(String[] args) throws InterruptedException
{
Selenium selenium= new DefaultSelenium("localhost",4444,"firefox","http://www.calculator.net");
selenium.start();
selenium.open("/");
selenium.windowMaximize();
selenium.click("xpath=.//*[#id=''hl3']/li[3]/a");
Thread.sleep(4000);
selenium.focus("id=cpar1");
selenium.type("css=input[id=\"cpar1\"]", "10");
selenium.focus("id=cpar2");
selenium.type("css=input[id=\"cpar2\"]", "50");
(selenium).click("xpath=.//*[#id='content']/table[1]/tbody/tr[2]/td/input[2]");
// verify if the result is 5
Thread.sleep(4000);
String result = selenium.getText("xpath=.//*[#id='content']/p[2]/font/b");
//String result = selenium.getValue("xpath=.//*[#id='cpar3']");
System.out.println("Result:"+result);
if (result.equals("5")/*== "5"*/){
System.out.println("Pass");
}
else{
System.out.println("Fail");
}
}
}
I would recommend reconsidering using Selenium Remote Control as it is quite outdated approach which is no longer supported, current stable version of Selenium Java client is 3.141.59 and it provides WebDriver API which is a W3C Standard as of now.
Once you implement option 1 get rid of those Thread.sleep() as it's a some form of a performance anti-pattern, go for Explicit Wait instead, check out How to use Selenium to test web applications using AJAX technology for comprehensive explanation and code examples.
It's better to use a dependency management solution like Apache Maven which will automatically detect and download your project transitive dependencies. A relevant pom.xml file would be something like:
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.example</groupId>
<artifactId>selenium</artifactId>
<version>1.0-SNAPSHOT</version>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<configuration>
<source>8</source>
<target>8</target>
</configuration>
</plugin>
</plugins>
</build>
<dependencies>
<dependency>
<groupId>org.seleniumhq.selenium</groupId>
<artifactId>selenium-java</artifactId>
<version>3.141.59</version>
</dependency>
</dependencies>
</project>
I am trying to test out some code that works with Java Doc, it is used under the maven-javadoc-plugin. I am trying to get it to work under jdk11. I am after an implementation of RootDoc which I can use when running tests.
Currently the tests use EasyDoclet which gives me a RootDoc like so:
EasyDoclet easyDoclet = new EasyDoclet(new File("dir"), "com.foo.bar");
RootDoc rootDoc = easyDoclet.getRootDoc()
However I could not get this to work under jdk11.
The first issue I had was tools.jar is missing so I changed my pom.xml to have:
<dependency>
<groupId>org.seamless</groupId>
<artifactId>seamless-javadoc</artifactId>
<version>1.1.1</version>
<exclusions>
<exclusion>
<groupId>com.sun</groupId>
<artifactId>tools</artifactId>
</exclusion>
</exclusions>
</dependency>
<!-- maybe this will get what ever was in tools.jar -->
<dependency>
<groupId>com.github.olivergondza</groupId>
<artifactId>maven-jdk-tools-wrapper</artifactId>
<version>0.1</version>
</dependency>
This lead to many instances of:
java.lang.NoClassDefFoundError: com/sun/tools/javadoc/PublicMessager
The PublicMessager class seems to exist to make public some constructors, I am not sure why it exists under the com.sun.tools package. I tried to make a copy of this class:
public static class PublicMessager extends
com.sun.tools.javadoc.main.Messager {
public PublicMessager(Context context, String s) {
super(context, s);
}
public PublicMessager(Context context, String s, PrintWriter printWriter, PrintWriter printWriter1, PrintWriter printWriter2) {
super(context, s, printWriter, printWriter1, printWriter2);
}
}
And the error message changes to:
java.lang.IllegalAccessError: superclass access check failed: class com.fun.javadoc.FooBar$PublicMessager (in unnamed module #0x4abdb505) cannot access class com.sun.tools.javadoc.main.Messager (in module jdk.javadoc) because module jdk.javadoc does not export com.sun.tools.javadoc.main to unnamed module #0x4abdb50
I exposed jdk.javadoc to the unnamed module using:
<build>
<plugins>
<plugin>
<artifactId>maven-surefire-plugin</artifactId>
<configuration>
<argLine>-Dfile.encoding=UTF-8</argLine>
<argLine>--add-opens=jdk.javadoc/com.sun.tools.javadoc.main=ALL-UNNAMED</argLine>
</configuration>
</plugin>
</plugins>
</build>
This meant that my custom version of PublicMessager would no longer have the errors shown however the version from seamless under com.sun.tools could not be found. I made my own version of EasyDoclet which used my PublicMessager however it turned out that the following two classes are missing:
import com.sun.tools.javadoc.JavadocTool;
import com.sun.tools.javadoc.ModifierFilter;
At this point I am not sure what to do. halp!
Perhaps an alternative would be to instead find the jdk11 equivalent of RootDoc which I think is DocletEnvironment and then some how get an implementation of that, I have no idea how to get an implementation of DocletEnvironment.
I will start by mentioning that I have tried all the suggestion in similar topics and nothing worked for me, so please, this is not a duplicate question.
The issue that I am having is as follows -
I am trying to run a sample java application on spark, using spark streaming and kafka. I have added all the required dependencies:
<dependencies>
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-library</artifactId>
<version>2.11.12</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.11</artifactId>
<version>2.3.0</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming_2.11</artifactId>
<version>2.3.0</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming-kafka-0-10_2.11</artifactId>
<version>2.3.0</version>
</dependency>
</dependencies>
After deploying the Jar on a server where I would like to run my application (I have already set up the environment with spark, kafka, and created the relevant topic) I am trying to spark-submit it and getting the following error:
Exception in thread "main" java.lang.NoClassDefFoundError: org.apache.spark.streaming.kafka010.LocationStrategies
at JavaWordCount.main(JavaWordCount.java:47)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:90)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
at java.lang.reflect.Method.invoke(Method.java:508)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:879)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:197)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:227)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:136)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: org.apache.spark.streaming.kafka010.LocationStrategies
at java.net.URLClassLoader.findClass(URLClassLoader.java:609)
at java.lang.ClassLoader.loadClassHelper(ClassLoader.java:924)
at java.lang.ClassLoader.loadClass(ClassLoader.java:869)
at java.lang.ClassLoader.loadClass(ClassLoader.java:852)
It seems that the workers cannot identify the dependencies as part of the environment. I did some research online, and many suggest to create an assembly JAR with maven-shade-plugin. So I also tried to maven-package the jar this way, but still no success.
For reference, here is where the app is failing:
// Configure Spark to connect to Kafka running on local machine
Map<String, Object> kafkaParams = new HashMap<>();
kafkaParams.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG,"localhost:9092");
kafkaParams.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG,
"org.apache.kafka.common.serialization.StringDeserializer");
kafkaParams.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG,
"org.apache.kafka.common.serialization.StringDeserializer");
kafkaParams.put(ConsumerConfig.GROUP_ID_CONFIG,"group1");
kafkaParams.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG,"latest");
kafkaParams.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG,true);
//Configure Spark to listen messages in topic test
Collection<String> topics = Arrays.asList("wordCount");
SparkConf conf = new SparkConf().setMaster("local[2]").setAppName("SparkKafkaWordCount");
//Read messages in batch of 30 seconds
JavaStreamingContext jssc = new JavaStreamingContext(conf, Durations.seconds(30));
// Start reading messages from Kafka and get DStream
final JavaInputDStream<ConsumerRecord<String, String>> stream =
KafkaUtils.createDirectStream(jssc, LocationStrategies.PreferConsistent(),
ConsumerStrategies.<String,String>Subscribe(topics,kafkaParams));
In the last line above, the class LocationStrategies is not recognized, even though I have added the right dependency to the pom.xml
Any ideas how to fix this issue?
add this to your pom.xml
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-assembly-plugin</artifactId>
<configuration>
<archive>
<manifest>
<mainClass>your main class </mainClass>
</manifest>
</archive>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
</configuration>
</plugin>
and then build the jar with
mvn clean compile assembly:single
You should get two jars in the target directory, one without dependencies and one with dependencies (your-jar-1.0-jar-with-dependencies.jar)
I do not get this error when I include following jars under --jar in spark-spark-submit command:
sql-kafka-0-10_2.11-2.2.1.jar
kafka-clients-0.10.1.0.jar
spark-streaming-kafka-0-10_2.11-2.2.1.jar
spark-streaming-kafka-0-10-assembly_2.11-2.2.1.jar
spark-streaming-kafka_2.11-1.6.3.jar
Even I faced the same issue, didn't get much help from googling but what I've understood after reading many threads, the dependencies which are mentioned in pom.xml file has scope as "provided" which means we need to specify the dependent jar files during the time of execution. Also all the examples inside Apache Spark package was compiled as a single jar file and we need to specify the class path to execute the required module. Download the necessary jar files which you've mentioned in pom.xml and execute like this
spark-submit --jars kafka-clients-1.1.0.jar,spark-streaming_2.11-2.3.0.jar,spark-streaming-kafka-0-10_2.11-2.3.0.jar --class org.apache.spark.examples.streaming.JavaDirectKafkaWordCount target/original-spark-examples_2.11-2.4.0-SNAPSHOT.jar <brokerip:port> <topic>
But before that you need to re-write the consumer properties in the java file or else you will get error saying configuration is missing
kafkaParams.put("bootstrap.servers", brokers);
kafkaParams.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
kafkaParams.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
kafkaParams.put("group.id", "<group_id>");
I would like to introduce Byte Buddy to my company and I have prepared a demo for my colleagues. Since we use Spring a lot, I thought the best example would be instrumentation of SpringBoot application. I have decided to add logs to RestController methods.
Instrumented application is a simple SpringBoot Hello World example:
#RestController
public class HelloController {
private static final String template = "Hello, %s!";
#RequestMapping("/hello")
public String greeting(
#RequestParam(value = "name", defaultValue = "World") String name) {
return String.format(template, name);
}
#RequestMapping("/browser")
public String showUserAgent(HttpServletRequest request) {
return request.getHeader("user-agent");
}
}
And here is my Byte Buddy agent:
public class LoggingAgent {
public static void premain(String agentArguments,
Instrumentation instrumentation) {
install(instrumentation);
}
public static void agentmain(String agentArguments,
Instrumentation instrumentation) {
install(instrumentation);
}
private static void install(Instrumentation instrumentation) {
createAgent(RestController.class, "greeting")
.installOn(instrumentation);
}
private static AgentBuilder createAgent(
Class<? extends Annotation> annotationType, String methodName) {
return new AgentBuilder.Default().type(
ElementMatchers.isAnnotatedWith(annotationType)).transform(
new AgentBuilder.Transformer() {
#Override
public DynamicType.Builder<?> transform(
DynamicType.Builder<?> builder,
TypeDescription typeDescription,
ClassLoader classLoader) {
return builder
.method(ElementMatchers.named(methodName))
.intercept(
MethodDelegation
.to(LoggingInterceptor.class)
.andThen(
SuperMethodCall.INSTANCE));
}
});
}
}
Interceptor logs method execution:
public static void intercept(#AllArguments Object[] allArguments,
#Origin Method method) {
Logger logger = LoggerFactory.getLogger(method.getDeclaringClass());
logger.info("Method {} of class {} called", method.getName(), method
.getDeclaringClass().getSimpleName());
for (Object argument : allArguments) {
logger.info("Method {}, parameter type {}, value={}",
method.getName(), argument.getClass().getSimpleName(),
argument.toString());
}
}
When executed with -javaagent parameter this example works well. When however I try to load the agent on the running JVM with Attach API:
VirtualMachine vm = VirtualMachine.attach(args[0]);
vm.loadAgent(args[1]);
vm.detach();
I've got the following exception on the first logging attempt:
Exception in thread "ContainerBackgroundProcessor[StandardEngine[Tomcat]]" java.lang.IncompatibleClassChangeError: Class ch.qos.logback.classic.spi.ThrowableProxy does not implement the requested interface ch.qos.logback.classic.spi.IThrowableProxy
at ch.qos.logback.classic.pattern.ThrowableProxyConverter.subjoinExceptionMessage(ThrowableProxyConverter.java:180)
at ch.qos.logback.classic.pattern.ThrowableProxyConverter.subjoinFirstLine(ThrowableProxyConverter.java:176)
at ch.qos.logback.classic.pattern.ThrowableProxyConverter.recursiveAppend(ThrowableProxyConverter.java:159)
at ch.qos.logback.classic.pattern.ThrowableProxyConverter.throwableProxyToString(ThrowableProxyConverter.java:151)
at org.springframework.boot.logging.logback.ExtendedWhitespaceThrowableProxyConverter.throwableProxyToString(ExtendedWhitespaceThrowableProxyConverter.java:35)
at ch.qos.logback.classic.pattern.ThrowableProxyConverter.convert(ThrowableProxyConverter.java:145)
at ch.qos.logback.classic.pattern.ThrowableProxyConverter.convert(ThrowableProxyConverter.java:1)
at ch.qos.logback.core.pattern.FormattingConverter.write(FormattingConverter.java:36)
at ch.qos.logback.core.pattern.PatternLayoutBase.writeLoopOnConverters(PatternLayoutBase.java:114)
at ch.qos.logback.classic.PatternLayout.doLayout(PatternLayout.java:141)
at ch.qos.logback.classic.PatternLayout.doLayout(PatternLayout.java:1)
at ch.qos.logback.core.encoder.LayoutWrappingEncoder.doEncode(LayoutWrappingEncoder.java:130)
at ch.qos.logback.core.OutputStreamAppender.writeOut(OutputStreamAppender.java:187)
at ch.qos.logback.core.OutputStreamAppender.subAppend(OutputStreamAppender.java:212)
at ch.qos.logback.core.OutputStreamAppender.append(OutputStreamAppender.java:100)
at ch.qos.logback.core.UnsynchronizedAppenderBase.doAppend(UnsynchronizedAppenderBase.java:84)
at ch.qos.logback.core.spi.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:48)
at ch.qos.logback.classic.Logger.appendLoopOnAppenders(Logger.java:270)
at ch.qos.logback.classic.Logger.callAppenders(Logger.java:257)
at ch.qos.logback.classic.Logger.buildLoggingEventAndAppend(Logger.java:421)
at ch.qos.logback.classic.Logger.filterAndLog_0_Or3Plus(Logger.java:383)
at ch.qos.logback.classic.Logger.log(Logger.java:765)
at org.slf4j.bridge.SLF4JBridgeHandler.callLocationAwareLogger(SLF4JBridgeHandler.java:221)
at org.slf4j.bridge.SLF4JBridgeHandler.publish(SLF4JBridgeHandler.java:303)
at java.util.logging.Logger.log(Unknown Source)
at java.util.logging.Logger.doLog(Unknown Source)
at java.util.logging.Logger.logp(Unknown Source)
at org.apache.juli.logging.DirectJDKLog.log(DirectJDKLog.java:181)
at org.apache.juli.logging.DirectJDKLog.error(DirectJDKLog.java:147)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1352)
at java.lang.Thread.run(Unknown Source)
I run the example on 64-bit HotSpot with Java8:
java version "1.8.0_112"
Java(TM) SE Runtime Environment (build 1.8.0_112-b15)
Java HotSpot(TM) 64-Bit Server VM (build 25.112-b15, mixed mode)
Byte Buddy version is 1.4.32. Here is agent maven configuration:
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>pl.halun.demo.bytebuddy</groupId>
<artifactId>byte-buddy-agent-demo</artifactId>
<version>1.0</version>
<properties>
<jdk.version>1.8</jdk.version>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
</properties>
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>1.4.1.RELEASE</version>
</parent>
<dependencies>
<dependency>
<groupId>net.bytebuddy</groupId>
<artifactId>byte-buddy</artifactId>
<version>1.4.32</version>
</dependency>
<dependency>
<groupId>ch.qos.logback</groupId>
<artifactId>logback-classic</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>javax.servlet</groupId>
<artifactId>javax.servlet-api</artifactId>
<scope>provided</scope>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<configuration>
<source>${jdk.version}</source>
<target>${jdk.version}</target>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-assembly-plugin</artifactId>
<configuration>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
<finalName>${project.artifactId}-${project.version}-full</finalName>
<appendAssemblyId>false</appendAssemblyId>
<archive>
<manifestEntries>
<Premain-Class>pl.halun.demo.bytebuddy.logging.LoggingAgent</Premain-Class>
<Agent-Class>pl.halun.demo.bytebuddy.logging.LoggingAgent</Agent-Class>
<Can-Redefine-Classes>true</Can-Redefine-Classes>
<Can-Retransform-Classes>true</Can-Retransform-Classes>
<Can-Set-Native-Method-Prefix>true</Can-Set-Native-Method-Prefix>
</manifestEntries>
</archive>
</configuration>
<executions>
<execution>
<id>assemble-all</id>
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
And here is pom file for the instrumented application:
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>pl.halun.demo.bytebuddy.instrumented.app</groupId>
<artifactId>byte-buddy-agent-demo-instrumented-app</artifactId>
<version>1.0</version>
<packaging>jar</packaging>
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>1.4.1.RELEASE</version>
</parent>
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
</dependencies>
<properties>
<java.version>1.8</java.version>
</properties>
<build>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
</plugin>
</plugins>
</build>
<repositories>
<repository>
<id>spring-releases</id>
<url>https://repo.spring.io/libs-release</url>
</repository>
</repositories>
<pluginRepositories>
<pluginRepository>
<id>spring-releases</id>
<url>https://repo.spring.io/libs-release</url>
</pluginRepository>
</pluginRepositories>
From my point of view it is very valuable option to add logs on the running server and I hate to loose this part of demo. I tried to experiment with different redefinition strategies, but until now nothing seems to work.
What you observe is a classical version conflict, I think. Spring Boot most likely comes with a version of ThrowableProxy that is not compatible to the version that is added with the Java agent. When loading a Java agent at runtime, Spring's version is already loaded whereas the startup attachment prepends the agent-bundled version on the class path where the version of your agent is loaded.
Java agents are typically added to the class path. This is also where your Spring boot application is living. You need to make sure that a Java agent does not contain dependencies that are incompatible with your application's dependencies or you need to shade all dependencies to avoid such conflicts.
There is however another problem: When writing a Java agent that is attached at runtime, you meet additional constraints on most JVMs where on HotSpot, you are not allowed to change the class file format of any class that is already loaded. There is also a chance that your class is already loaded where currently, no effect would be visible as you do not enable retransformation.
A runtime-capable agent would need to use the Advice component which inlines code into target code rather then using the classical delegation model:
class MyAdvice {
#Advice.OnMethodEnter
static void intercept(#Advice.BoxedArguments Object[] allArguments,
#Advice.Origin Method method) {
Logger logger = LoggerFactory.getLogger(method.getDeclaringClass());
logger.info("Method {} of class {} called", method.getName(), method
.getDeclaringClass().getSimpleName());
for (Object argument : allArguments) {
logger.info("Method {}, parameter type {}, value={}",
method.getName(), argument.getClass().getSimpleName(),
argument.toString());
}
}
}
You can use the above advice class by registering it as a visitor. Such visitors only apply to declared methods, i.e. not to inherited methods and inline their code into existing methods. This way, the logging will not be visible on the call stack and it also becomes legal to retransform already loaded classes:
new AgentBuilder.Default()
.disableClassFormatChanges()
.with(AgentBuilder.RedefinitionStrategy.RETRANSFORMATION)
.type(isAnnotatedWith(annotationType))
.transform(new AgentBuilder.Transformer() {
#Override
public DynamicType.Builder<?> transform(
DynamicType.Builder<?> builder,
TypeDescription typeDescription,
ClassLoader classLoader) {
return builder.visit(Advice.to(MyAdvice.class).on(named(methodName)));
}
});
As for the attachment, look into the byte-buddy-agent project which allows you to call:
ByteBuddyAgent.attach(agentJar, processId);
The above helper supports other VMs where the attachment API often lives in a different namespace.
Update: Here is the problem with Spring Boot. Spring Boot creates custom class loaders that have the system class loader (class path) as their parent. These class loaders consider classes from the system class loader first. When you add the agent, the entire Spring boot app is both on the class loader and in these child class loaders. A class like IThrowableProxy now exists twice in two class loaders but is not considered to be equal by the JVM. Depending on the state of the VM, some classes might already be linked to the original IThrowableProxy whereas other classes are loaded after the agent was attached and get linked to the new IThrowableProxy from the agent. Both classes are not equal and the error that you see is thrown where the VM complains that the class does not implement the correct IThrowableProxy (but the previous one). If the agent is attached at start up, this problem does not exist as the class path's IThrowableProxy is always loaded.
This is not an easy error to fix, in the end, Byte Buddy cannot help you with such class path issues and Spring Boot is quite free in its interpretation of the class loader contract. The easiest way would be to not use Spring Boot types in your agent. You can still match the annotation with for example
isAnnotatedWith(named("org.springframework.web.bind.annotation.RestController"))
The question is how you can communicate with Spring Boot. One work-arround would be to add all shared classes to the class path on start up. Typically, I do avoid the usage of shared classes altogether but only use them in the Advice classes where the code is inlined in the class loader of the target application. Simply set the Spring Boot dependency in provided scope, the advice code itself is never executed.