We Keep Coding

Cannot start kafka connect distributed using windows bat file

UPDATE:
I have taken a look at the kafka.jar mentioned - which seems to be added to the classpath by the CMD files that run the distributed worker - kafka-run-class.bat in particular and it is less than 1kb so doesn't seem to be a valid JAR - so maybe that first error is a red herring. But that aside if it runs on, the distributed worker just spits out a load of errors about ClassNotFoundException, then goes on to say every config value provided in my distributed properties config file is not recognised, and doesn't try and load any plugins
ORIGINAL question
I am simply trying a very simple test of starting a Kafka Connect distributed worker, no plugins yet - just trying to get the basic worker going.
My distributed config is pretty stock
bootstrap.servers=GBV04303950:9092,GBV04303950:9082
group.id=connect-cluster
key.converter=org.apache.kafka.connect.json.JsonConverter
value.converter=org.apache.kafka.connect.json.JsonConverter
key.converter.schemas.enable=true
value.converter.schemas.enable=true
internal.key.converter=org.apache.kafka.connect.json.JsonConverter
internal.value.converter=org.apache.kafka.connect.json.JsonConverter
internal.key.converter.schemas.enable=false
internal.value.converter.schemas.enable=false
offset.storage.topic=connect-offsets
offset.storage.replication.factor=2
config.storage.topic=connect-configs
config.storage.replication.factor=2
status.storage.topic=connect-status
status.storage.replication.factor=2
However I get the following error
C:\kafka\ClusterOne\connect-worker_1>.\bin\windows\connect-distributed.bat .\etc\kafka-connect-replicator\replicator-connect-distributed.properties
[2019-03-22 10:53:51,832] WARN could not create Dir using jarFile from url file:/C:/kafka/ClusterOne/connect-worker_1/share/java/kafka/kafka.jar. skip
ping. (org.reflections.Reflections)
java.util.zip.ZipException: error in opening zip file
at java.util.zip.ZipFile.open(Native Method)
at java.util.zip.ZipFile.<init>(Unknown Source)
at java.util.zip.ZipFile.<init>(Unknown Source)
at java.util.jar.JarFile.<init>(Unknown Source)
at java.util.jar.JarFile.<init>(Unknown Source)
at org.reflections.vfs.Vfs$DefaultUrlTypes$1.createDir(Vfs.java:216)
at org.reflections.vfs.Vfs.fromURL(Vfs.java:99)
at org.reflections.vfs.Vfs.fromURL(Vfs.java:91)
at org.reflections.Reflections.scan(Reflections.java:240)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader$InternalReflections.scan(DelegatingClassLoader.java:412)
at org.reflections.Reflections$1.run(Reflections.java:198)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
The only reason I can think of for this is my local drive is locked down and doesn't let you create folders unless you have Elevated rights. I am running this from a CMD prompt that was started with elevated rights but as I've noticed, this doesn't always translate down to passing those rights on to processes spawned from within the CMD prompt. But I'm not sure how I can provide elevated rights to the java process that's trying to unpack the JAR.
Also, that error is followed by a long stream of ClassNotFoundException, I assume because the above error means JARs cannot be unpacked to load all the required classes
eg
[2019-03-22 10:56:04,567] WARN could not get type for name org.apache.kafka.common.utils.MockTime from any class loader (org.reflections.Reflections)
org.reflections.ReflectionsException: could not get type for name org.apache.kafka.common.utils.MockTime
at org.reflections.ReflectionUtils.forName(ReflectionUtils.java:390)
at org.reflections.Reflections.expandSuperTypes(Reflections.java:381)
at org.reflections.Reflections.<init>(Reflections.java:126)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader$InternalReflections.<init>(DelegatingClassLoader.java:404)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanPluginPath(DelegatingClassLoader.java:304)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanUrlsAndAddPlugins(DelegatingClassLoader.java:242)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initPluginLoader(DelegatingClassLoader.java:190)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initLoaders(DelegatingClassLoader.java:183)
at org.apache.kafka.connect.runtime.isolation.Plugins.<init>(Plugins.java:61)
at org.apache.kafka.connect.cli.ConnectDistributed.startConnect(ConnectDistributed.java:90)
at org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:77)
Caused by: java.lang.ClassNotFoundException: org.apache.kafka.common.utils.MockTime
at java.net.URLClassLoader.findClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at org.reflections.ReflectionUtils.forName(ReflectionUtils.java:388)
... 10 more
[2019-03-22 10:56:04,706] WARN could not get type for name com.google.gson.JsonDeserializer from any class loader (org.reflections.Reflections)
org.reflections.ReflectionsException: could not get type for name com.google.gson.JsonDeserializer
at org.reflections.ReflectionUtils.forName(ReflectionUtils.java:390)
at org.reflections.Reflections.expandSuperTypes(Reflections.java:381)
at org.reflections.Reflections.<init>(Reflections.java:126)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader$InternalReflections.<init>(DelegatingClassLoader.java:404)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanPluginPath(DelegatingClassLoader.java:304)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanUrlsAndAddPlugins(DelegatingClassLoader.java:242)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initPluginLoader(DelegatingClassLoader.java:190)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initLoaders(DelegatingClassLoader.java:183)
at org.apache.kafka.connect.runtime.isolation.Plugins.<init>(Plugins.java:61)
at org.apache.kafka.connect.cli.ConnectDistributed.startConnect(ConnectDistributed.java:90)
at org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:77)
Caused by: java.lang.ClassNotFoundException: com.google.gson.JsonDeserializer
at java.net.URLClassLoader.findClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at org.reflections.ReflectionUtils.forName(ReflectionUtils.java:388)
... 10 more
[2019-03-22 10:56:04,857] WARN could not get type for name org.scalatest.junit.JUnitSuite from any class loader (org.reflections.Reflections)
org.reflections.ReflectionsException: could not get type for name org.scalatest.junit.JUnitSuite
at org.reflections.ReflectionUtils.forName(ReflectionUtils.java:390)
at org.reflections.Reflections.expandSuperTypes(Reflections.java:381)
at org.reflections.Reflections.<init>(Reflections.java:126)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader$InternalReflections.<init>(DelegatingClassLoader.java:404)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanPluginPath(DelegatingClassLoader.java:304)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanUrlsAndAddPlugins(DelegatingClassLoader.java:242)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initPluginLoader(DelegatingClassLoader.java:190)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initLoaders(DelegatingClassLoader.java:183)
at org.apache.kafka.connect.runtime.isolation.Plugins.<init>(Plugins.java:61)
at org.apache.kafka.connect.cli.ConnectDistributed.startConnect(ConnectDistributed.java:90)
at org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:77)
Caused by: java.lang.ClassNotFoundException: org.scalatest.junit.JUnitSuite
at java.net.URLClassLoader.findClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at org.reflections.ReflectionUtils.forName(ReflectionUtils.java:388)
... 10 more
Can anyone suggest if they think my theory about elevated rights is why this isn't working or how I could get around this?

Inside the share/java/kafka folder there is a kafka.jar about 26 bytes.
If you look at it with a text editor, you can see, that it is a link.
Copy the linked jar to kafka.jar.
On windows, you should also set the plugin.path to an absolute path.
You will get a few errors, while starting, that is due to missing aws and other jars.

Flink: NoClassDefFoundError when runnig a Table API query

I have a streaming Table API query and want to write the resulting Table into a .csv file using the table.writetoSink(sink) method.
When I run the program, I got the following error.
Exception in thread "main" java.lang.NoClassDefFoundError: org/codehaus/commons/compiler/CompileException
at org.apache.calcite.sql2rel.SqlToRelConverter.convertQuery(SqlToRelConverter.java:580)
at org.apache.flink.api.table.FlinkPlannerImpl.rel(FlinkPlannerImpl.scala:102)
at org.apache.flink.api.table.BatchTableEnvironment.sql(BatchTableEnvironment.scala:132)
at table_streaming_test.main(table_streaming_test.java:59)
Caused by: java.lang.ClassNotFoundException: org.codehaus.commons.compiler.CompileException
at java.net.URLClassLoader$1.run(Unknown Source)
at java.net.URLClassLoader$1.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
... 4 more
I have also included the commons-compiler.jar file in my project.

Flink's Table and SQL API is built on Apache Calcite which depends on commons-compiler from org.codehaus.janino.
The error message suggests that the commons-compiler dependency is not included in your application. One approach is to built a fat jar containing all dependencies of your application. Flink's quickstart Maven architypes (Java and Scala) provide a blueprint for that. Another approach is to add all required dependencies to the ./lib folder of your Flink setup.

Successfully implementing external classes in Eclipse. (Java)

I have been trying to use this external class folder which was given from my programming lecturer a few months back. I followed people's explanations of how to add via "configure build class", and the application somewhat recognises the class because a shortlist of some of the methods are given. It also does not return any errors prior to runtime but once it's ran I get this:
Exception in thread "main" java.lang.NoClassDefFoundError: Console
at lab3.q2.main(q2.java:15)
Caused by: java.lang.ClassNotFoundException: Console
at java.net.URLClassLoader$1.run(Unknown Source)
at java.net.URLClassLoader$1.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
... 1 more
This is really frustrating as it works from the linux computers in my college without issue. The class file is just a variation of the scanner that we use for user i/o, I have it attached.
Console.class file

Java EWS throws NoClassDefFound

I want to read the Outlook calendar. So I'm started first to try to connect to the server. I found http://code.msdn.microsoft.com/exchange-ews-java-api-12-1a5a1143 and want just to get the connect and try around.
I found a example and tried this:
import microsoft.exchange.webservices.data.*;
...
public void test(){
ExchangeService es = new ExchangeService();
ExchangeCredentials credentials = new WebCredentials("user", "Password");
}
It crashes by Exchange Service:
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/commons/httpclient/HttpConnectionManager
at ExchangeTest.test(ExchangeTest.java:15)
at Window.main(Window.java:19)
Caused by: java.lang.ClassNotFoundException: org.apache.commons.httpclient.HttpConnectionManager
at java.net.URLClassLoader$1.run(Unknown Source)
at java.net.URLClassLoader$1.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
... 2 more
I tried to import a library but didn't got the correct one or the fault is something else. Can you tell me which library I need or where/how I can fix it. Thanks.
I'm using a Win7.

You need this library The Commons HttpClient and you have to add this library to the build path of your project.

SSL using RSAL Toolkit not encapsulated by a jar Error

I'm running a 3rd party app which fails with the following stacktrace when trying to create a SSL connection to a server:
The root exception is this:
Caused by: java.lang.SecurityException: Toolkit not encapsulated by a jar.
at com.rsa.jcm.f.hq.a(Unknown Source)
at com.rsa.jcm.f.jg.b(Unknown Source)
at com.rsa.crypto.jcm.ModuleLoader.a(Unknown Source)
at com.rsa.crypto.jcm.ModuleLoader.load(Unknown Source)
... 43 more
Has anyone seen this occur when connecting over SSL that uses RSA? I've worked the google and there isn't anything obvious as to why this would happen.
This particular error happens on a Mac, and I've got a windows version of the same app which works correctly. Main difference being they bundle the JRE in the windows version, so I checked various security policies etc but there are no great differences in the JRE on windows vs the JRE on my mac.
The full stacktrace looks like this:
Caused by: java.lang.Error: Problem loading module.
at com.rsa.cryptoj.o.ju.g(Unknown Source)
at com.rsa.cryptoj.o.ju.c(Unknown Source)
at com.rsa.cryptoj.o.gd.a(Unknown Source)
at com.rsa.cryptoj.o.dm.b(Unknown Source)
at com.rsa.cryptoj.o.dm.<clinit>(Unknown Source)
at com.rsa.cryptoj.o.me.newInstance(Unknown Source)
at javax.crypto.Cipher.chooseProvider(Cipher.java:845)
at javax.crypto.Cipher.init(Cipher.java:1348)
at sun.security.ssl.CipherBox.<init>(CipherBox.java:175)
at sun.security.ssl.CipherBox.newCipherBox(CipherBox.java:208)
at sun.security.ssl.CipherSuite$BulkCipher.newCipher(CipherSuite.java:467)
at sun.security.ssl.CipherSuite$BulkCipher.isAvailable(CipherSuite.java:507)
at sun.security.ssl.CipherSuite$BulkCipher.isAvailable(CipherSuite.java:485)
at sun.security.ssl.CipherSuite.isAvailable(CipherSuite.java:190)
at sun.security.ssl.SSLContextImpl.getApplicableCipherSuiteList(SSLContextImpl.java:342)
at sun.security.ssl.SSLContextImpl.getDefaultCipherSuiteList(SSLContextImpl.java:300)
at sun.security.ssl.SSLSocketImpl.init(SSLSocketImpl.java:576)
at sun.security.ssl.SSLSocketImpl.<init>(SSLSocketImpl.java:512)
at sun.security.ssl.SSLSocketFactoryImpl.createSocket(SSLSocketFactoryImpl.java:73)
at com.somevendor.client.common.spring.remoting.http.e.createSocket(Unknown Source)
at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707)
at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361)
at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387)
at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:346)
at com.somevendor.client.common.spring.remoting.http.SecureHttpInvokerRequestExecutor.executePostMethod(Unknown Source)
at org.springframework.remoting.httpinvoker.CommonsHttpInvokerRequestExecutor.doExecuteRequest(CommonsHttpInvokerRequestExecutor.java:140)
at org.springframework.remoting.httpinvoker.AbstractHttpInvokerRequestExecutor.executeRequest(AbstractHttpInvokerRequestExecutor.java:136)
at org.springframework.remoting.httpinvoker.HttpInvokerClientInterceptor.executeRequest(HttpInvokerClientInterceptor.java:192)
at org.springframework.remoting.httpinvoker.HttpInvokerClientInterceptor.executeRequest(HttpInvokerClientInterceptor.java:174)
at org.springframework.remoting.httpinvoker.HttpInvokerClientInterceptor.invoke(HttpInvokerClientInterceptor.java:142)
... 7 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
... 39 more
Caused by: java.lang.SecurityException: Toolkit not encapsulated by a jar.
at com.rsa.jcm.f.hq.a(Unknown Source)
at com.rsa.jcm.f.jg.b(Unknown Source)
at com.rsa.crypto.jcm.ModuleLoader.a(Unknown Source)
at com.rsa.crypto.jcm.ModuleLoader.load(Unknown Source)
... 43 more

Answering as I worked it out (5 years ago..). This exception happens when there is whitespace in the fully qualified path to the jar file that is been executed.
So this works:
/Users/stringy05/app/app.jar
But if you use a path with a space:
/Volumes/Macintosh HD/Users/stringy05/app/app.jar
Then it fails. When this originally happened I had a soft link in /Users/stringy05 to another hard disk, so either java or the OS was resolving the link path to the real path and it would fail.