We Keep Coding

Kafka-Connect add SQL JAR file to classpath

I am trying to deploy a connect-standalone job to stream from an mssql server however am facing an issue (Kafka-Connect is part of my Ambari deployment, not docker). This is the properties file I am using:
name=JdbcSourceConnector
connector.class=io.confluent.connect.jdbc.JdbcSourceConnector
connection.user=ue
connection.password=pw
tasks.max=1
connection.url=jdbc:sqlserver://servername
topic.prefix=iblog
query=SELECT * FROM IB_WEBLOG_DUMMY_small
value.converter=org.apache.kafka.connect.json.JsonConverter
key.converter=org.apache.kafka.connect.json.JsonConverter
poll.interval.ms=5000
table.poll.interval.ms=120000
mode=incrementing
incrementing.column.name=ID
I have added the jar file sqljdbc42.jar to /usr/share/java
and have run export CLASSPATH=/usr/share/java/*
however I still run into the error Failed to find any class that implements Connector and which name matches io.confluent.connect.jdbc.JdbcSourceConnector
Am I doing anything wrong or can I check something else?

Kafka-Connect is part of my Ambari deployment
That would imply you are using Hortonworks installation
You need to
git clone https://github.com/confluentinc/kafka-connect-jdbc/
Checkout a release branch that ideally matches your Kafka version. For example branch v3.1.2 is Kafka 0.10.1.1
mvn clean package will generate some folders in target/ of that project
SCP those files to all Kafka Connect workers in your cluster into /usr/hdp/current/kafka/.../share/java/kafka-connect-jdbc (create this, if not exists)
Restart Kafka processes to pick up the new CLASSPATH settings
You may need some extra Confluent packages that JDBC connect depends on

You need to include the kafka-connect-jdbc jar file, which contains the io.confluent.connect.jdbc.JdbcSourceConnector class.
If you are using maven, you can add it as a dependency:
[Add the following repo to your project if you haven't done so yet.]
<repository>
<id>confluent</id>
<url>http://packages.confluent.io/maven/</url>
</repository>
After this, add the following dependency:
<dependency>
<groupId>io.confluent</groupId>
<artifactId>kafka-connect-jdbc</artifactId>
<version>3.3.0 (or whatever version you want)</version>
</dependency>
https://github.com/confluentinc/kafka-connect-jdbc/issues/356

I too had the same problem. with Couchbase connector not found
ERROR Stopping after connector error (org.apache.kafka.connect.cli.ConnectStandalone:113) java.util.concurrent.ExecutionException: org.apache.kafka.connect.errors.ConnectException: Failed to find any class that implements Connector and which name matches com.couchbase.connect.kafka.CouchbaseSourceConnector
Setting classpath was losing the existing classpath and I couldn't append as the classpath
I moved the required jar file from kafka-connect-couchase/*.jar files to /path/kafka_verison/libs/
libs is A folder where all the jar file stored.

I met the same issue, I resolved it by running connect-standalone in the root folder of confluent, in my case this was: /opt/confluent-5.0.1

How to get maven-project install directory from pom.xml

Is there a way to get a maven project install directory from property in the pom.xml? Something like lets say ${project.install.directory}, I know there is ${project.build.directory} which points to target folder, but I need this C:\Users\user\.m2\repository\com\demo\1.0-SNAPSHOT.

I don't think that there is one property, that would do exactly, what you want.
However you can try to glue together these properties to make yourself a "project.install.directory" property:
${settings.localRepository} will point to your local repository - "C:\Users\user.m2\repository"
${project.groupId}
${project.artifactId}
${project.version}
${project.build.finalName}

Combining/Resolving Two Repository Types in Gradle

In my Gradle project, I need to define two types of repositories: a flat directory and a Maven repository (a local Nexus server).
I can get both working separately, but can't get them to play nicely together. Ideally, I would have Gradle look at the local directory first, then at the Maven repository.
Current Setup
I have defined the repositories in my build.gradle like this:
repositories {
flatDir dirs: "${rootProject.projectDir}/libs"
flatDir dirs: "${rootProject.projectDir}/test.libs"
maven {
credentials {
username nexus_username
password nexus_password
}
url "http://nexus-server/nexus/content/groups/public"
}
}
In the libs (and test.libs) directory, the jar file names may or may not have versioning (but when using a flatDir repository, I believe that is irrelevant):
libs\activation.jar
libs\imap.jar
....
libs\<closed_source_framework>.jar
....
libs\gson-2.2.4.jar
libs\stax-utils.jar
.....
The reason I can't use our local Nexus server for everything is because of <closed_source_framework>.jar; most of the dependancies in the libs folder come packaged with that distribution, and I can't reliably get the version information to pull them from Nexus.
Now, one of the other teams is publishing their jars to the Nexus server and I'd like to be able to pull their jars from Nexus, so I have (re)defined my dependencies in my build.gradle:
dependencies {
// Grab other-team jar files from Nexus server
compile "other-team-group:other-team-jar-1:version"
compile "other-team-group:other-team-jar-2:version"
compile "other-team-group:other-team-jar-3:version"
// Grab everything else from 'flatDir'
compile name: 'activation'
compile name: 'imap'
...
compile name: 'gson-2.2.4'
compile name: 'stax-utils'
.....
}
The Problem
So now comes my problem. I had expected Gradle would search the repositories in the order specified in my build.gradle; meaning that it would look to the local libs folder first and then go to Nexus if it can't find it locally. What I'm seeing instead is that Gradle is looking at the Nexus server for the jar files already in the local libs folder. Obviously, this is slowing down my build (I have ~30 dependencies defined).
Some Info
Output from gradle properties command, to show repository information:
.....
repositories: [org.gradle.api.internal.artifacts.repositories.DefaultFlatDirArtifactRepository_Decorated#18814b1b, org.gradle.api.internal.artifacts.repositories.DefaultMavenArtifactRepository_Decorated#6dff028]
.....
Output from gradle --info compileJava, to show that Gradle is doing a lookup to Nexus:
.....
// Successfully find the other team jar files in Nexus, this is okay
Download http://nexus-server/nexus/content/groups/public/other-team-group/other-team-jar-1/version/other-team-jar-1-version.pom
Download http://nexus-server/nexus/content/groups/public/other-team-group/other-team-jar-2/version/other-team-jar-2-version.pom
Download http://nexus-server/nexus/content/groups/public/other-team-group/other-team-jar-3/version/other-team-jar-3-version.pom
.....
// Continues looking in Nexus for jar files that should be found in local libs folder
Resource missing. [HTTP GET: http://nexus-server/nexus/content/groups/public//activation//activation-.pom]
Resource missing. [HTTP HEAD: http://nexus-server/nexus/content/groups/public//activation//activation-.jar]
Resource missing. [HTTP GET: http://nexus-server/nexus/content/groups/public///imap//imap-.pom]
Resource missing. [HTTP HEAD: http://nexus-server/nexus/content/groups/public//imap//imap-.jar]
.....
Bottom Line
How can I get Gradle to stop looking at the Maven repository for jar files that I know it will only find locally?

I also posted this question over on the Gradle forums. I have copy/pasted the solution below.
Gradle will prefer an artifact with an pom/ivy descriptor over an
artifact without. I think this is why gradle continues searching after
it finds a match in the flatDir repository. This may or may not solve
your problem, but you could use a FileCollectionDependency instead of
a ModuleDependency.
Eg:
ext {
libs = "${rootProject.projectDir}/libs"
testLibs = "${rootProject.projectDir}/test.libs"
}
dependencies {
compile files("${libs}/activation.jar", "${libs}/imap.jar")
compile files("${testLibs}/gson-2.2.4.jar")
...
}

An SPI class of type org.apache.lucene.codecs.Codec with name 'Lucene54' does not exist

With lucene-core-5.5.2 i am facing problem a in weblogic server. standalone search application works but when i deploy as WEB APP it is failing with below error
Exception type is 'java.lang.ExceptionInInitializerError'. Runtime error: java.lang.IllegalArgumentException: An SPI class of type org.apache.lucene.codecs.Codec with name 'Lucene54' does not exist. You need to add the corresponding JAR file supporting this SPI to your classpath. The current classpath supports the following names: []
I tried creating folder structure under classes/ as META-INF/services/ added all files from lucene-core-5.5.2.jar META-INF\services\ directory also created jar file for META-INF\services\ and added in classpath but it doesn't recognize META-INF/services to load SPI
Any help would be really appreciated.

Please add following file in
Folder : META-INF/services/
File :org.apache.lucene.codecs.Codec
Text :org.apache.lucene.codecs.lucene54.Lucene54Codec
Please review the solution with detailed description at https://anwaarlabs.wordpress.com/2017/02/25/lucene-an-spi-class-of-type-org-apache-lucene-codecs-codec-with-name-does-not-exist/

I got this building with gradle using a proposed solution to build a "fat jar" (executable jar containing all dependent jars) here.
But it didn't work: I got this obscure error to do with Lucene, but not when testing or building or running normally, only when building a fat jar.
My solution was to use shadow jar: code from gradle.build:
buildscript {
repositories { jcenter() }
dependencies { // fatjar
classpath 'com.github.jengelman.gradle.plugins:shadow:1.2.4' }
}
apply plugin: 'com.github.johnrengelman.shadow'
shadowJar {
baseName = project.name
classifier = null
version = project.version
}

I added the below code into configuration tags
<transformers>
<transformer implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
<resource>META-INF/services/org.apache.lucene.codecs.Codec</resource>
<resource>META-INF/services/org.apache.lucene.codecs.PostingsFormat</resource>
</transformer>
</transformers>
And also I add lucene-core dependency at the top of the dependencies tag
<dependency>
<groupId>org.apache.lucene</groupId>
<artifactId>lucene-core</artifactId>
<version>5.5.0</version>
</dependency>

its is fixed i added lucene jars in weblogic app server classpath and it is working as expected. i dont know why it is not detecting from my applications lib folder as its also on class path but looks like somehow SPI needs jars and META-INF on app servers classpath

Make sure that the lucene jar that you have included in your project is of the same version as that of lucene codec mentioned in error. E.g. if the error states LuceneCodec62 then you should have lucene-6.x.x jar included in your project.

eclipse : Can not find the tag library descriptor for "/struts-tags"

I am following the steps given in
http://www.tutorialspoint.com/struts_2/struts_quick_guide.htm
for a sample struts 2 application
but I am getting following error message :
Can not find the tag library descriptor for "/struts-tags
I followed several helps :
Error - Can not find the tag library descriptor for "/struts-tags"
Cannot find the tag library descriptor for /WEB-INF/struts-html.tld in Struts
still the error persists.. what shall I do?
Thanks

This message can be produced if the struts2-core-[VERSION].jar is not marked as an Export Dependency.
To correct this:
Project Properties > Java Build Path > Order and Export
Then mark struts2-core-[VERSION].jar as an Export Dependency (or, if included via Ivy, make sure the Ivy library is marked as an Export Dependency).

Have you added the struts2-core-.jar to the libraries of the project? Clean the project. The error will go away.