We Keep Coding

NiFI :- Failed to create Extension Definition for CONTROLLER_SERVICE org.apache.nifi.record.sink.lookup.RecordSinkServiceLookup

I'm trying to use DBCPConnectionPool service in my custom processor. So, how to use in-built controller services (which are already available on NiFi) in our custom processors.
Here are my properties in *-nar/->pom.xml
<dependencies>
<dependency>
<groupId>org.sample.com</groupId>
<artifactId>nifi-customprocessor-processors</artifactId>
<version>1.0</version>
</dependency>
<dependency>
<groupId>org.apache.nifi</groupId>
<artifactId>nifi-standard-services-api-nar</artifactId>
<version>1.14.0</version>
<type>nar</type>
</dependency>
</dependencies>
Here are my dependencies in *processors/->pom.xml
<dependency>
<groupId>org.apache.nifi</groupId>
<artifactId>nifi-api</artifactId>
</dependency>
<dependency>
<groupId>org.apache.nifi</groupId>
<artifactId>nifi-utils</artifactId>
<version>1.14.0</version>
</dependency>
<dependency>
<groupId>org.apache.nifi</groupId>
<artifactId>nifi-mock</artifactId>
<version>1.14.0</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-simple</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.nifi</groupId>
<artifactId>nifi-dbcp-service</artifactId>
<version>1.14.0</version>
</dependency>
<dependency>
<groupId>org.apache.nifi</groupId>
<artifactId>nifi-dbcp-service-api</artifactId>
<version>1.14.0</version>
</dependency>
And here is my propertydescriptor for DBCPConnetionPool service :-
public static final PropertyDescriptor DBCPConnectionPool_SERVICE = new PropertyDescriptor.Builder()
.name("DBCPConnectionPoolLookup Service")
.description("The Controller Service to use in order to establish a connection")
.required(true)
.dynamic(true)
.identifiesControllerService(DBCPService.class)
.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
.build();
and in my OnTrigger method I didn't used it as :-
DBCPService DBCPService = context.getProperty(DBCPConnectionPool_SERVICE)
.asControllerService(DBCPService.class);
Connection con = DBCPService.getConnection();
and was trying to use this con object as in regular passion createStatement->executeQuery.
When I tried to package it using mvn clean install it started throwing error as : A required class was missing while executing org.apache.nifi:nifi-nar-maven-plugi
n:1.3.1:nar: org/apache/nifi/record/sink/RecordSinkService
then I added the following dependencies in processors/->pom.xml
<dependency>
<groupId>org.apache.nifi</groupId>
<artifactId>nifi-record-sink-service</artifactId>
<version>1.14.0</version>
</dependency>
<dependency>
<groupId>org.apache.nifi</groupId>
<artifactId>nifi-record-sink-api</artifactId>
<version>1.14.0</version>
</dependency>
This time it was new error as :- Failed to create Extension Definition for CONTROLLER_SERVICE org.apache.nifi.record.sink.lookup.RecordSinkServiceLookup: Null
PointerException
I also checked by commenting out code which I have written, to check the way I'm using that property was wrong but, even that didn't worked.:-
DBCPService DBCPService = context.getProperty(DBCPConnectionPool_SERVICE)
.asControllerService(DBCPService.class);
Connection con = DBCPService.getConnection();
Can someone help me out in order to use DBCPConnectionPoolService controller service in my custom processor?

I had the same issues the other day and I solved it by using AbstractControllerService instead of DBCPConnectionPool therefore not needing org.apache.nifi.dbcp.DBCPService anymore. Therefore my code would look like this:
MyService DBCPService = context.getProperty(DBCPConnectionPool_SERVICE)
.asControllerService(MyService.class);
Connection con = DBCPService.getConnection();
MyService is the custom interface that extends ControllerService.(you have to use it when you also define the PropertyDescriptor DBCPConnectionPool_SERVICE)
And StandardMyService the class that extends AbstractControllerService and implements MyService and getConnection() from it.

cannot create EMRclient - Sdkclientexception

ive been trying to create a cluster using the Emrclient
EmrClient emrClient = EmrClient.builder().
region(Region.US_EAST_1).
credentialsProvider(EnvironmentVariableCredentialsProvider.create()).
build();
when i try to run the code, i get an SDKexception:
Exception in thread "main" software.amazon.awssdk.core.exception.SdkClientException: Unable to load an HTTP implementation from any provider in the chain. You must declare a dependency on an appropriate HTTP implementation or pass in an SdkHttpClient explicitly to the client builder.
at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:98)
at software.amazon.awssdk.core.internal.http.loader.DefaultSdkHttpClientBuilder.lambda$buildWithDefaults$1(DefaultSdkHttpClientBuilder.java:49)
at java.base/java.util.Optional.orElseThrow(Optional.java:408)
at software.amazon.awssdk.core.internal.http.loader.DefaultSdkHttpClientBuilder.buildWithDefaults(DefaultSdkHttpClientBuilder.java:43)
at software.amazon.awssdk.core.client.builder.SdkDefaultClientBuilder.lambda$resolveSyncHttpClient$5(SdkDefaultClientBuilder.java:269)
at java.base/java.util.Optional.orElseGet(Optional.java:369)
at software.amazon.awssdk.core.client.builder.SdkDefaultClientBuilder.resolveSyncHttpClient(SdkDefaultClientBuilder.java:269)
at software.amazon.awssdk.core.client.builder.SdkDefaultClientBuilder.finalizeSyncConfiguration(SdkDefaultClientBuilder.java:220)
at software.amazon.awssdk.core.client.builder.SdkDefaultClientBuilder.syncClientConfiguration(SdkDefaultClientBuilder.java:153)
at software.amazon.awssdk.services.emr.DefaultEmrClientBuilder.buildClient(DefaultEmrClientBuilder.java:27)
at software.amazon.awssdk.services.emr.DefaultEmrClientBuilder.buildClient(DefaultEmrClientBuilder.java:22)
at software.amazon.awssdk.core.client.builder.SdkDefaultClientBuilder.build(SdkDefaultClientBuilder.java:124)
at task2.App.main(App.java:27)
after searching online in the aws docs, i found that to create the EMRclient we need to make the builder look like this:
EmrClient emrClient = EmrClient.builder().
region(Region.US_EAST_1).
credentialsProvider(EnvironmentVariableCredentialsProvider.create()).
httpClient(ApacheHttpClient.builder().build()).
build();
but no matter what i do, my IDE doesnt recognize the class ApacheHttpClient,
ive tried to import : import software.amazon.awssdk.http.apache.ApacheHttpClient;
but it doesnt recognize it even though i added it to my POM:
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>emr</artifactId>
<version>2.13.23</version>
</dependency>
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>ec2</artifactId>
<version>2.13.23</version>
</dependency>
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>s3</artifactId>
<version>2.13.23</version>
</dependency>
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>apache-client</artifactId>
<version>2.13.23</version>
<scope>test</scope>
</dependency>
<!-- https://mvnrepository.com/artifact/software.amazon.awssdk/apache-client -->
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>apache-client</artifactId>
<version>2.13.23</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpcore</artifactId>
<version>4.4</version>
</dependency>
was wondering if anyone has encountered this and was able to fix it?
thanks in advance

Uncaught exception from servlet java.lang.RuntimeException: Driver org.hsqldb.jdbc.JDBCDriver claims to not accept jdbcUrl, jdbc:mysql:///iworks_db

I am working on a Java Google App Engine app, and when I deploy my app and open it on my browser I get the above error. iworks_db is the name of my database, and for some reason my app fails to connect to it. I'm using the guide found here: https://cloud.google.com/sql/docs/mysql/connect-app-engine.
My createConnectionPool method:
private DataSource createConnectionPool() {
HikariConfig config = new HikariConfig();
config.setJdbcUrl(String.format("jdbc:mysql:///%s", DB_NAME));
config.setUsername(DB_USER);
config.setPassword(DB_PASS);
config.setDriverClassName("org.hsqldb.jdbc.JDBCDriver");// This line does NOT exist on the guide
// I added it because I was getting "failed
// to get driver instance" error
config.addDataSourceProperty("socketFactory", "com.google.cloud.sql.mysql.SocketFactory");
config.addDataSourceProperty("cloudSqlInstance", CLOUD_SQL_CONNECTION_NAME);
config.addDataSourceProperty("useSSL", "false");
config.setMaximumPoolSize(5);
config.setMinimumIdle(5);
config.setConnectionTimeout(10000); // 10 seconds
config.setIdleTimeout(600000); // 10 minutes
config.setMaxLifetime(1800000); // 30 minutes
DataSource pool = new HikariDataSource(config);
return pool;
}
The dependencies in my pom.xml file:
<!-- Compile/runtime dependencies -->
<dependency>
<groupId>com.google.appengine</groupId>
<artifactId>appengine-api-1.0-sdk</artifactId>
<version>1.9.59</version>
</dependency>
<dependency>
<groupId>javax.servlet</groupId>
<artifactId>javax.servlet-api</artifactId>
<version>3.1.0</version>
<type>jar</type>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>jstl</groupId>
<artifactId>jstl</artifactId>
<version>1.2</version>
</dependency>
<dependency>
<groupId>com.zaxxer</groupId>
<artifactId>HikariCP</artifactId>
<version>3.3.1</version>
</dependency>
<dependency>
<groupId>org.hsqldb</groupId>
<artifactId>hsqldb</artifactId>
<version>2.3.2</version>
</dependency>
Any help would be much appreciated!

Your URL is a mysql url while your driver is HyperSQL DataBase (hsqldb). That is the reason you get the exception regarding not acceptable jdbcUrl.
Your DB is Google Cloud mysql, which I believe is the case as the rest of your code use mysql factory (com.google.cloud.sql.mysql.SocketFactory), then change the driver class name to following:
config.setDriverClassName("com.mysql.jdbc.GoogleDriver");
Also replace the hsldb maven dependency with
<dependency>
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
<version>8.0.18</version>
</dependency>
<dependency>
<groupId>com.google.cloud.sql</groupId>
<artifactId>mysql-socket-factory-connector-j-8</artifactId>
<version>1.0.14</version>
</dependency>
The com.google.cloud.sql.mysql.SocketFactory is provided in the mysql-socket-factory-connector-j-8 maven artifact.
Update:
You could also try not to use the DriverClassName at all.
Try adding the above two dependencies and remove the setDriverClassName line.

Google-Cloud: Jetty ALPN/NPN has not been properly configured

Getting exception whilst using Google Pubsub to list topics, my web application is running on tomcat.
public static List<String> listTopics(GcpCredentials gcCredentials, String project) throws GCPException, IOException
{
List<String> topics = new ArrayList<>();
TopicAdminClient client = getTopicClient(gcCredentials);
ProjectName projectName = ProjectName.create(project);
ListTopicsPagedResponse response = client.listTopics(projectName);
for (Topic topic :response.iterateAll())
{
topics.add(topic.getNameAsTopicName().getTopic());
}
return topics;
}`
Exception:
java.lang.IllegalArgumentException: Jetty ALPN/NPN has not been properly configured.
at io.grpc.netty.GrpcSslContexts.selectApplicationProtocolConfig(GrpcSslContexts.java:174)
at io.grpc.netty.GrpcSslContexts.configure(GrpcSslContexts.java:151)
at io.grpc.netty.GrpcSslContexts.configure(GrpcSslContexts.java:139)
at io.grpc.netty.GrpcSslContexts.forClient(GrpcSslContexts.java:109)
at io.grpc.netty.NettyChannelBuilder.createProtocolNegotiatorByType(NettyChannelBuilder.java:335)
at io.grpc.netty.NettyChannelBuilder.createProtocolNegotiator(NettyChannelBuilder.java:308)
at io.grpc.netty.NettyChannelBuilder$NettyTransportFactory$DynamicNettyTransportParams.getProtocolNegotiator(NettyChannelBuilder.java:499)
at io.grpc.netty.NettyChannelBuilder$NettyTransportFactory.newClientTransport(NettyChannelBuilder.java:448)
at io.grpc.internal.CallCredentialsApplyingTransportFactory.newClientTransport(CallCredentialsApplyingTransportFactory.java:61)
at io.grpc.internal.InternalSubchannel.startNewTransport(InternalSubchannel.java:209)
at io.grpc.internal.InternalSubchannel.obtainActiveTransport(InternalSubchannel.java:186)
at io.grpc.internal.ManagedChannelImpl$SubchannelImplImpl.obtainActiveTransport(ManagedChannelImpl.java:806)
at io.grpc.internal.GrpcUtil.getTransportFromPickResult(GrpcUtil.java:568)
at io.grpc.internal.DelayedClientTransport.reprocess(DelayedClientTransport.java:296)
at io.grpc.internal.ManagedChannelImpl$LbHelperImpl$5.run(ManagedChannelImpl.java:724)
at io.grpc.internal.ChannelExecutor.drain(ChannelExecutor.java:87)
at io.grpc.internal.ManagedChannelImpl$LbHelperImpl.runSerialized(ManagedChannelImpl.java:715)
at io.grpc.internal.ManagedChannelImpl$NameResolverListenerImpl.onUpdate(ManagedChannelImpl.java:752)
at io.grpc.internal.DnsNameResolver$1.run(DnsNameResolver.java:174)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

I have observed this problem with Netty version 4.1.15.Final but not with 4.1.13.Final. Check your transitive dependencies. I.e Spring Boot references Netty.
What I added to POM to make it work with Spanner API version 0.22.0-beta:
<properties>
<v.netty>4.1.13.Final</v.netty>
</properties>
...
<dependencyManagement>
<dependencies>
<dependency>
<groupId>io.netty</groupId>
<artifactId>netty-codec</artifactId>
<version>${v.netty}</version>
</dependency>
<dependency>
<groupId>io.netty</groupId>
<artifactId>netty-codec-http</artifactId>
<version>${v.netty}</version>
</dependency>
<dependency>
<groupId>io.netty</groupId>
<artifactId>netty-codec-http2</artifactId>
<version>${v.netty}</version>
</dependency>
<dependency>
<groupId>io.netty</groupId>
<artifactId>netty-handler</artifactId>
<version>${v.netty}</version>
</dependency>
<dependency>
<groupId>io.netty</groupId>
<artifactId>netty-common</artifactId>
<version>${v.netty}</version>
</dependency>
<dependency>
<groupId>io.netty</groupId>
<artifactId>netty-handler-proxy</artifactId>
<version>${v.netty}</version>
</dependency>
<dependency>
<groupId>io.netty</groupId>
<artifactId>netty-transport</artifactId>
<version>${v.netty}</version>
</dependency>
<dependency>
<groupId>io.netty</groupId>
<artifactId>netty-resolver</artifactId>
<version>${v.netty}</version>
</dependency>
<dependency>
<groupId>io.netty</groupId>
<artifactId>netty-codec-socks</artifactId>
<version>${v.netty}</version>
</dependency>
<dependency>
<groupId>io.netty</groupId>
<artifactId>netty-buffer</artifactId>
<version>${v.netty}</version>
</dependency>
</dependencies>
</dependencyManagement>
If the problem persists, or if it is not an option, plase run your JVM with appropriate bootclasspath entry, like:
java -Xbootclasspath/p:/tmp/alpn-boot-8.1.11.v20170118.jar -cp ...
Make sure to replace /tmp/alpn-boot-8.1.11.v20170118.jar with location of alpn jar that matches your JVM version as listed on this page: https://www.eclipse.org/jetty/documentation/9.4.x/alpn-chapter.html

There is a possible solution on github: "Exception when configuring SSL: "Jetty ALPN/NPN has not been properly configured."
The recommended steps, quoting from the above document:
1) Edit catalina.sh
/usr/local/Cellar/tomcat/8.5.3/libexec/bin/catalina.sh
2) At line 103, add the following
CATALINA_OPTS="-javaagent:/Library/Tomcat/lib/jetty-alpn-agent-2.0.6.jar"
3) Restart tomcat
../bin/shutdown.sh ../bin/startup.sh

Neo4j and Gremlin from Java

I am trying to use gremlin to interact with my Neo4j community database. I am working in eclipse using maven.
Following the documentation in http://tinkerpop.apache.org/docs/3.1.0-incubating/#neo4j-gremlin i set up my dependencies to:
<dependencies>
<dependency>
<groupId>org.neo4j</groupId>
<artifactId>neo4j-cypher</artifactId>
<version>2.2.0</version>
</dependency>
<dependency>
<groupId>org.neo4j</groupId>
<artifactId>neo4j-ha</artifactId>
<version>2.2.0</version>
</dependency>
<dependency>
<groupId>org.apache.tinkerpop</groupId>
<artifactId>gremlin-core</artifactId>
<version>3.1.0-incubating</version>
</dependency>
<dependency>
<groupId>org.apache.tinkerpop</groupId>
<artifactId>neo4j-gremlin</artifactId>
<version>3.1.0-incubating</version>
</dependency>
<dependency>
<groupId>org.neo4j</groupId>
<artifactId>neo4j-tinkerpop-api-impl</artifactId>
<version>0.1-2.2</version>
</dependency>
<dependency>
<groupId>org.neo4j</groupId>
<artifactId>neo4j-tinkerpop-api</artifactId>
<version>0.1</version>
</dependency>
</dependencies>
After executing:
package test.gremlin.maven;
import org.apache.tinkerpop.gremlin.neo4j.structure.Neo4jGraph;
import org.apache.tinkerpop.gremlin.structure.Graph;
public class TestGremlin {
public static void main(String[] args) {
Graph graph = Neo4jGraph.open("/tmp/gremlintest");
}
}
i get the following error:
Exception in thread "main" java.lang.ClassCastException: org.neo4j.function.Functions$$Lambda$24/252651381 cannot be cast to org.neo4j.helpers.Function
at org.neo4j.helpers.Settings$DefaultSetting.apply(Settings.java:846)
at org.neo4j.kernel.configuration.ConfigurationValidator.validate(ConfigurationValidator.java:50)
at org.neo4j.kernel.configuration.Config.replaceSettings(Config.java:204)
at org.neo4j.kernel.configuration.Config.<init>(Config.java:92)
at org.neo4j.kernel.impl.factory.PlatformModule.<init>(PlatformModule.java:124)
at org.neo4j.kernel.impl.factory.GraphDatabaseFacadeFactory.createPlatform(GraphDatabaseFacadeFactory.java:177)
at org.neo4j.kernel.impl.factory.GraphDatabaseFacadeFactory.newFacade(GraphDatabaseFacadeFactory.java:124)
at org.neo4j.kernel.impl.factory.CommunityFacadeFactory.newFacade(CommunityFacadeFactory.java:40)
at org.neo4j.kernel.impl.factory.GraphDatabaseFacadeFactory.newFacade(GraphDatabaseFacadeFactory.java:108)
at org.neo4j.graphdb.factory.GraphDatabaseFactory.newDatabase(GraphDatabaseFactory.java:130)
at org.neo4j.graphdb.factory.GraphDatabaseFactory$1.newDatabase(GraphDatabaseFactory.java:118)
at org.neo4j.graphdb.factory.GraphDatabaseBuilder.newGraphDatabase(GraphDatabaseBuilder.java:185)
at org.neo4j.tinkerpop.api.impl.Neo4jFactoryImpl.newGraphDatabase(Neo4jFactoryImpl.java:44)
at org.neo4j.tinkerpop.api.Neo4jFactory$Builder.open(Neo4jFactory.java:32)
at org.apache.tinkerpop.gremlin.neo4j.structure.Neo4jGraph.<init>(Neo4jGraph.java:131)
at org.apache.tinkerpop.gremlin.neo4j.structure.Neo4jGraph.open(Neo4jGraph.java:145)
at org.apache.tinkerpop.gremlin.neo4j.structure.Neo4jGraph.open(Neo4jGraph.java:154)
at test.gremlin.maven.TestGremlin.main(TestGremlin.java:10)

Can you try to use the latest release: which is 0.3-2.3.3
see: https://github.com/neo4j-contrib/neo4j-tinkerpop-api-impl/tree/0.3-2.3.3