Errors trying to start ZetaSQL planner - java

I'm trying to run a Beam pipeline with SQL transforms, parsed with ZetaSQL. I begin with setting options with
options.setPlannerName("org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLQueryPlanner");
When I try creating my SqlTransform with any given query, I get
java.util.ServiceConfigurationError: org.apache.beam.repackaged.sql.com.google.zetasql.ClientChannelProvider: Provider org.apache.beam.repackaged.sql.com.google.zetasql.JniChannelProvider could not be instantiated
at java.util.ServiceLoader.fail (ServiceLoader.java:232)
at java.util.ServiceLoader.access$100 (ServiceLoader.java:185)
at java.util.ServiceLoader$LazyIterator.nextService (ServiceLoader.java:384)
at java.util.ServiceLoader$LazyIterator.next (ServiceLoader.java:404)
at java.util.ServiceLoader$1.next (ServiceLoader.java:480)
at org.apache.beam.repackaged.sql.com.google.zetasql.ClientChannelProvider.loadChannel (ClientChannelProvider.java:31)
...
at org.apache.beam.sdk.extensions.sql.SqlTransform.expand (SqlTransform.java:82)
at org.apache.beam.sdk.Pipeline.applyInternal (Pipeline.java:539)
at org.apache.beam.sdk.Pipeline.applyTransform (Pipeline.java:473)
at org.apache.beam.sdk.values.PCollection.apply (PCollection.java:357)
...
I've added the following relevant dependencies to my POM in maven:
<dependency>
<groupId>org.apache.beam</groupId>
<artifactId>beam-runners-google-cloud-dataflow-java</artifactId>
<version>2.15.0</version>
</dependency>
<dependency>
<groupId>org.apache.beam</groupId>
<artifactId>beam-sdks-java-extensions-sql</artifactId>
<version>2.16.0</version>
</dependency>
Is there something else I'm missing here?

Unfortunately the ZetaSQL planner will today not work from a MAC, or older versions of Linux. Please see the comment from Rui in:
ZetaSQL Sample Using Apache beam
Looks like this PR maybe useful here ( I have not dug deeply into this to confirm):
https://github.com/google/zetasql/pull/3
As a workaround could you try on a newer version of linux? Maybe in a container?

This is resolved in the latest version of ZetaSQL, which will be used with Beam 2.21. You can also try pulling down a newer version of ZetaSQL (at least 2020.03.2):
<dependency>
<groupId>com.google.zetasql</groupId>
<artifactId>zetasql-jni-channel</artifactId>
<version>2020.03.2</version>
</dependency>

Related

WARNING: Unable to find an exact match for CDP version 107, so returning the closest version found: 106

Can anyone help me to solve this issue?
I have Google Chrome version 107.0.5304.6, and I'm using WebDriverManager to get the same chromeDriver version. I download Google Chrome version 106 and run my tests and this issue didn't happen, but my Google Chrome updates automatically from version 106 to 107 and the warning happens again.
I had a similar issue and resolved using the following steps:
Step1: Add the following devtools dependency:
<dependency>
<groupId>org.seleniumhq.selenium</groupId>
<artifactId>selenium-devtools-v88</artifactId>
<version>4.0.0-beta-1</version>
</dependency>
Step2: change v88 to the Chrome Version you have on your machine. e.g., v106 or v107
Step3: change the beta version to the latest one. e.g., 4.0.0-beta-1 to 4.5.3.
just download the chrome driver version 107 and update in your dependency folder
https://chromedriver.storage.googleapis.com/index.html?path=107.0.5304.62/
this resolves the issue
As of Selenium CHANGELOG CDP version 107 is supported starting selenium v4.6.0
you just need to update selenium java version to 4.6.0 or later
<properties>
<selenium.version>4.6.0</selenium.version>
</properties>
<dependency>
<groupId>org.seleniumhq.selenium</groupId>
<artifactId>selenium-java</artifactId>
<version>${selenium.version}</version>
</dependency>

Informix java.lang.ClassNotFoundException: com.informix.jdbc.IfxDirectConnection

I'm trying to install and connect to a local Informix 14 for testing on Windows 10. My maven POM has:
<dependency>
<groupId>com.ibm.informix</groupId>
<artifactId>informix-jdbc-complete</artifactId>
<version>4.50.4.1</version>
<scope>test</scope>
</dependency>
I'm trying direct connection
database.url = jdbc:informix-direct://sysmaster
database.driver=com.informix.jdbc.IfxDirectConnection
But I'm getting class not found.
Any ideas?
You shouldn't use 'jdbc:informix-direct:'. That was reserved for UDR written in Java.
Use "jdbc:informix-sqli", something like:
Class.forName("com.informix.jdbc.IfxDriver");
conn = DriverManager.getConnection("jdbc:informix-sqli://420ito:9088/stores7:INFORMIXSERVER=ids1410;user=informix;password=mypassword;");

Apache Beam Streaming from Pub/Sub to ElasticSearch

I'm writing a java streaming pipeline with Apache Beam that reads messages from Google Cloud PubSub and should write them into an ElasticSearch instance. Currently, I'm using the direct runner, but the plan is to deploy the solution on Google Cloud Dataflow.
First of all, I wrote a pipeline that reads from PubSub and writes to text files and it works. Then, I sat up the ElasticSearch instance and also this works. I wrote some documents with curl and it was easy.
Then, when I tried to perform the write with Beam's ElasticSearch connector, I started to get some error. Actually, I get ava.lang.NoSuchMethodError: org.elasticsearch.client.RestClient.performRequest, in spite of the fact that I added the dependency on my pom.xml file.
What I'm doing is essentially this:
messages.apply(
"TwoMinWindow",
Window.into(FixedWindows.of(new Duration(120*1000)))
).apply(
"ElasticWrite",
ElasticsearchIO.write()
.withConnectionConfiguration(
ElasticsearchIO.ConnectionConfiguration
.create(new String[]{"http://xxx.xxx.xxx.xxx:9200"}, "streaming_data", "string")
.withUsername("xxxx")
.withPassword("xxxxxxxx")
)
);
Using the DirectRunner, I'm able to connect to PubSub, but I get an exception when the pipeline tries to connect with the ElasticSearch instance:
java.lang.NoSuchMethodError: org.elasticsearch.client.RestClient.performRequest(Ljava/lang/String;Ljava/lang/String;[Lorg/apache/http/Header;)Lorg/elasticsearch/client/Response;
at org.apache.beam.sdk.util.UserCodeException.wrap (UserCodeException.java:34)
at org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO$Write$WriteFn$DoFnInvoker.invokeSetup (Unknown Source)
at org.apache.beam.sdk.transforms.reflect.DoFnInvokers.tryInvokeSetupFor (DoFnInvokers.java:50)
at org.apache.beam.runners.direct.DoFnLifecycleManager$DeserializingCacheLoader.load (DoFnLifecycleManager.java:104)
at org.apache.beam.runners.direct.DoFnLifecycleManager$DeserializingCacheLoader.load (DoFnLifecycleManager.java:91)
at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache$LoadingValueReference.loadFuture (LocalCache.java:3528)
at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache$Segment.loadSync (LocalCache.java:2277)
at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache$Segment.lockedGetOrLoad (LocalCache.java:2154)
at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache$Segment.get (LocalCache.java:2044)
at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache.get (LocalCache.java:3952)
at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache.getOrLoad (LocalCache.java:3974)
at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache$LocalLoadingCache.get (LocalCache.java:4958)
at org.apache.beam.runners.direct.DoFnLifecycleManager.get (DoFnLifecycleManager.java:61)
at org.apache.beam.runners.direct.ParDoEvaluatorFactory.createEvaluator (ParDoEvaluatorFactory.java:129)
at org.apache.beam.runners.direct.ParDoEvaluatorFactory.forApplication (ParDoEvaluatorFactory.java:79)
at org.apache.beam.runners.direct.TransformEvaluatorRegistry.forApplication (TransformEvaluatorRegistry.java:169)
at org.apache.beam.runners.direct.DirectTransformExecutor.run (DirectTransformExecutor.java:117)
at java.util.concurrent.Executors$RunnableAdapter.call (Executors.java:511)
at java.util.concurrent.FutureTask.run (FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:624)
at java.lang.Thread.run (Thread.java:748)
Caused by: java.lang.NoSuchMethodError: org.elasticsearch.client.RestClient.performRequest(Ljava/lang/String;Ljava/lang/String;[Lorg/apache/http/Header;)Lorg/elasticsearch/client/Response;
at org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO.getBackendVersion (ElasticsearchIO.java:1348)
at org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO$Write$WriteFn.setup (ElasticsearchIO.java:1200)
What I added in the pom.xml is :
<dependency>
<groupId>org.apache.beam</groupId>
<artifactId>beam-sdks-java-io-google-cloud-platform</artifactId>
<version>${beam.version}</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.elasticsearch.client/elasticsearch-rest-client -->
<dependency>
<groupId>org.elasticsearch.client</groupId>
<artifactId>elasticsearch-rest-client</artifactId>
<version>${elastic.version}</version>
</dependency>
I'm stuck with this problem and I don't know how to solve it. If I use a JestClient, I'm able to connect to ElasticSearch without any issue.
Have you any suggestion?
You are using a newer version of RestClient that does not have the method performRequest(String, Header). If you look at the latest source code, you can see that the method takes a Request now, whereas in older versions there were methods that took Strings and Headers.
These methods were deprecated and then removed from the code on September 1, 2018.
Either change your code to use the newer Elastic Search library, or specify an older version of the library (it needs to be before 7.0.x, e.g. 6.8.4) that is compatible with your code.

EWS(Exchange WebService) API not working in Open JDK 11 as well in Zulu Java 11

I have been using EWS(Exchange Web Service) for reading emails from a shared email box in Java 8.
After upgrading to Java 11 (Open JDK 11/Zulu Java 11), facing exceptions while reading emails.
This is the Exception:
java.lang.NoClassDefFoundError: javax/xml/ws/http/HTTPException at
Microsoft.exchange.webservices.data.core.ExchangeService.internalFindFolders(ExchangeService.java:370)
~[ews-java-api-2.0.jar:na]
Adding the below dependency in your pom.xml will help in reading emails from a shared mailbox.
<dependency>
<groupId>javax.xml.ws</groupId>
<artifactId>jaxws-api</artifactId>
<version>2.2.1</version>
</dependency>

jzmq vs jeromq for getting ZeroMQ up and running, jeromq works but not jzmq, but WHY?

It took me a day to solve this point so maybe it saves your time in case you are planning on setting up a CentOS 6.4 with ZeroMQ and java binding.
Installed jzmq, libzmq, zmq-4.x from the source from github for the basic hello world example. Still kept getting what almost drove me crazy:
Exception in thread "main" java.lang.UnsatisfiedLinkError: org.zeromq.ZMQ$Socket.nativeInit()V
at org.zeromq.ZMQ$Socket.nativeInit(Native Method)
at org.zeromq.ZMQ$Socket.<clinit>(ZMQ.java:435)
at org.zeromq.ZMQ$Context.socket(ZMQ.java:365)
at samples.zeromq.HWClient.main(HWClient.java:15)
Maven dependency:
<dependency>
<groupId>org.zeromq</groupId>
<artifactId>jzmq</artifactId>
<version>3.0.1</version>
</dependency>
But before giving up completely(!) I changed it to:
<dependency>
<groupId>org.zeromq</groupId>
<artifactId>jeromq</artifactId>
<version>0.3.3</version>
</dependency>
And everything was solved!
I suppose something is wrong with the JINI wrapping of jzmq. Any ideas why this happens? And potentially how to solve it?
As #Claude mentioned, libzmq might not be on the library path.
A solution can be found here.

Categories