We Keep Coding

ES Query Exception in Storm Crawler

I am using following packages
Apache zookeeper 3.4.14
Apache storm 1.2.3
Apache Maven 3.6.2
ElasticSearch 7.2.0 (hosted locally)
Java 1.8.0_252
aws ec2 medium instance with 4GB ram
I have used this command to increase the virtual memory for jvm(Earlier it was showing error for jvm not having enough memory)
sysctl -w vm.max_map_count=262144
I have created maven package with -
mvn archetype:generate -DarchetypeGroupId=com.digitalpebble.stormcrawler -
DarchetypeArtifactId=storm-crawler-elasticsearch-archetype -DarchetypeVersion=LATEST
Command used for submitting topology
storm jar target/newscrawler-1.0-SNAPSHOT.jar org.apache.storm.flux.Flux --local es-crawler.flux --sleep 30000
when i run this command, it shows my topology is submitted sucessfully, and in elasticsearch status index it shows FETCH_ERROR and also the url from seeds.txt
content index shows no hits in elasticsearch
In worker.log file there were many exceptions of following type-
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:1.8.0_252]
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:714) ~[?:1.8.0_252]
at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvent(DefaultConnectingIOReactor.java:174) [stormjar.jar:?]
at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents(DefaultConnectingIOReactor.java:148) [stormjar.jar:?]
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:351) [stormjar.jar:?]
at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:221) [stormjar.jar:?]
at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64) [stormjar.jar:?]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_252]
2020-06-12 10:31:14.635 c.d.s.e.p.AggregationSpout Thread-46-spout-executor[17 17] [INFO] [spout #7] Populating buffer with nextFetchDate <= 2020-06-12T10:30:50Z
2020-06-12 10:31:14.636 c.d.s.e.p.AggregationSpout Thread-32-spout-executor[19 19] [INFO] [spout #9] Populating buffer with nextFetchDate <= 2020-06-12T10:30:50Z
2020-06-12 10:31:14.636 c.d.s.e.p.AggregationSpout pool-13-thread-1 [ERROR] [spout #7] Exception with ES query
There are following logs in worker.log related to elasticsearch
'Suppressed: org.elasticsearch.client.ResponseException: method [POST], host [http://localhost:9200], URI [/status/_search?typed_keys=true&ignore_unavailable=false&expand_wildcards=open&allow_no_indices=true&preference=_shards%3A1&ignore_throttled=true&search_type=query_then_fetch&batched_reduce_size=512&ccs_minimize_roundtrips=true], status line [HTTP/1.1 503 Service Unavailable]
{"error":{"root_cause":[{"type":"cluster_block_exception","reason":"blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];"}],"type":"cluster_block_exception","reason":"blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];"},"status":503}
'
'
Suppressed: org.elasticsearch.client.ResponseException: method [POST], host [http://localhost:9200], URI [/status/_search?typed_keys=true&ignore_unavailable=false&expand_wildcards=open&allow_no_indices=true&preference=_shards%3A8&ignore_throttled=true&search_type=query_then_fetch&batched_reduce_size=512&ccs_minimize_roundtrips=true], status line [HTTP/1.1 503 Service Unavailable]
{"error":{"root_cause":[],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[]},"status":503}
'
I have checked health of shards, they are in green status.
Earlier i was using Java 11 , with which i was not able to submit topology so i shifted to java 8.
Now topology is submitted sucessfully, but no data is injected in Elasticsearch.
I want to know if there is a problem with version imcompatibility between java and elasticsearch or with any oher package.

Use an absolute path for the seed file and run it in remote mode. The local mode should be used mostly for debugging.
The sleep parameter is (I think) in milliseconds. The command above means that the topology will run for 30 seconds only, which doesn't give it much time to do anything.

Why some VPN clients break Java debugging and how to work around the issue?

I am using IntelliJ to develop a Scala project.
Due to client-server architecture of the system, my integration tests have to be run with the following settings in build.sbt:
fork in IntegrationTest := true
javaOptions in (IntegrationTest) ++= Seq("-Djdk.logging.allowStackWalkSearch=true", "-XX:PermSize=256M", "-XX:MaxPermSize=512M", "-Xmx1024m")
// for attaching with debugger to the processes under test
javaOptions in (IntegrationTest) += "-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5005"
Normally everything works fine - tests are being run, debugging works.
Sometimes I need to establish a VPN connection to access some resources of my employee company. I'm using Check Point Endpoint Security VPN (that's the officially recommended software of my employee and I'm not sure if anything else would work).
So, if I happen to be connected to the VPN and then run the integration tests, then SBT console starts getting stuck right after:
Listening for transport dt_socket at address: 5005
The exact message is:
Listening for transport dt_socket at address: 5025
[error] Uncaught exception when running tests: java.net.ConnectException: Connection timed out: connect
[trace] Stack trace suppressed: run last project/it:test for the full output.
Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8
When I run last project/it:test (I have to reload SBT console first because it has stuck in (busy) > state) I see this:
[debug] javaOptions: List(-Djdk.logging.allowStackWalkSearch=true, -XX:PermSize=256M, -XX:MaxPermSize=512M, -Xmx1024m, -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5005)
[debug] Forking tests - parallelism = false
[debug] Create a single-thread test executor
[error] Uncaught exception when running tests: java.net.ConnectException: Connection timed out: connect
Sometimes when I disconnect the VPN and run the tests again, it starts working. But often disconnecting VPN doesn't help and I have to reboot my computer.
I have tried some less dramatic solutions - restarting IDE, killing all java and javaw processes, looking at netstat results to see anything still using the port 5005, changing the port to 5025 in build.sbt and reloading SBT console... nothing works, only reboot and only until the next time I need to connect to VPN.
That's a nightmare. I don't want to reboot my machine each time after I connect to VPN.
Is there any solution to this? Any Java flags? Any Windows network stack settings? Any VPN settings?

Cannot connect to Wildfly

I have just installed Wildfly and I tried to connect it :
\wildfly-11.0.0.Final\bin>jboss-cli.bat -c
But gives me follow error :
Failed to connect to the controller: The controller is not available
at localhost:9990: java.net.ConnectException: WFLYPRT0053: Could not
connect to remote+http://localhost:9990. The connection failed:
WFLYPRT0053: Could not connect to remote+http://localhost:9990. The
connection failed: Connection refused: no further information
I tried a lot of solutions but it's not working for me.

With Widlfly running, i.e. (standalone.bat), use the --controller option to define where it is:
jboss-cli.bat -c --controller=localhost:9990

Got the same error on wildfly version 16
Error
Failed to connect to the controller: The controller is not available at localhost:: java.net.ConnectException: WFLYPRT0053: Could not connect to remote+http://localhost:. The connection failed: WFLYPRT0053: Could not connect to remote+http://localhost:. The connection failed: Connection refused
And following done and resolved successfully
Step 01
Comment (or you can remove) following line from /bin/jboss-cli.xml
default-protocol use-legacy-override="true">remote+https</default-protocol
Correct protocol Ex:
<default-protocol use-legacy-override="true">remote+http</default-protocol>
<!-- The default controller to connect to when 'connect' command is executed w/o arguments -->
<default-controller>
<protocol>remote+http</protocol>
<host>localhost</host>
<port>9990</port>
</default-controller>
Step 02
In my case i have alredy created a Administrative user hence, I have statup the CLI with following commnad
./jboss-cli.sh --user="<user>" --password="<password>" --controller=remote+http:<your IP>:<port> --connect
Example :
./jboss-cli.sh --user="Admin" --password="Password" --controller=remote+http://19.199.115.172:9990 --connect
Make sure your wildfly is up and running. If you have used different port for the admin console it should be added .

Jboss must be running while doing this. I was trying to do this while my jboss was not up n running... then i got this error message. So boot up jboss and try again.

The most common case is that there is a mismatch between the default controller defined in jboss-cli.xml and the management port, configured in the standalone.xml/domain.xml. Out of the box, they should converge on localhost:9990. Therefore, verify if you changed any of the two files. Other than that, it could be a firewall/network issue.
See also: Cannot connect to WildFly with CLI

spark standalone cluster slave unable to connect slave to master

i have the bin for spark-1.6.0-bin-hadoop2.6 im having issue trying to connect the slave to the master
so far i have tried(on ubuntu 14.04 live usb):
apt-get purge and install openssh-client and server on both systems
I have stated explicitly the ip address of the master in the spark url for the worker
spark://< master ip>:7077 and also tried changing the SPARK_MASTER_IP in /conf/spark-env.sh the worker executes but the log has the following error
im assuming there must be some ssh setup involved but i have tried ssh-keygen and ssh-copy-id # it also doesnt give any reuslts
16/02/22 07:49:16 INFO Worker: Connecting to master 192.168.0.208:7077...
16/02/22 07:49:16 WARN Worker: Failed to connect to master 192.168.0.208:7077
java.io.IOException: Failed to connect to /192.168.0.208:7077
at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:216)
at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:167)
at org.apache.spark.rpc.netty.NettyRpcEnv.createClient(NettyRpcEnv.scala:200)
at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:187)
at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:183)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.ConnectException: Connection refused: /192.168.0.208:7077
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:740)
at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:224)
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:289)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:528)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
... 1 more
16/02/22 07:49:27 INFO Worker: Retrying connection to master (attempt # 2)
i am however able to open the master webUI by typing :8080 on my browser . i am also able to access the webUI of the slave from the master . im almost at the point of giving in so please helllppp.

Make sure that each master and worker has a firewall exception to allow connections for all other workers and masters.
Here's a simplified example from one of our master machines (master0):
$iptables -L
...
ACCEPT all -- worker0.company.com master0.company.com
ACCEPT all -- worker1.company.com master0.company.com
ACCEPT all -- master1.company.com master0.company.com
...
Of course you can also use IPs instead of hostnames.

Websphere MQ client deploy to Jboss Fuse

Error appear when MQConnectionFactory try to create connection.
At the same time I'm able to send\receive message from standalone application.
jboss stack trace.
Caused by: com.ibm.mq.MQException: JMSCMQ0001: JMSCMQ0001, 2, MQCC_FAILED, 2195, MQRC_UNEXPECTED_ERROR
at com.ibm.msg.client.wmq.common.internal.Reason.createException(Reason.java:209)
... 24 more
Caused by: com.ibm.mq.jmqi.JmqiException: CC=2;RC=2195;AMQ9204: Connection to host 'hostname(1450)' rejected. [1=com.ibm.mq.jmqi.JmqiException[CC=2;RC=2195],3=hostname(1450),5=WMQThreadPool.enqueue]
at com.ibm.mq.jmqi.remote.api.RemoteFAP.jmqiConnect(RemoteFAP.java:2053)
at com.ibm.mq.jmqi.remote.api.RemoteFAP.jmqiConnect(RemoteFAP.java:1226)
at com.ibm.msg.client.wmq.internal.WMQConnection.<init>(WMQConnection.java:346)
... 23 more
Caused by: com.ibm.mq.jmqi.JmqiException: CC=2;RC=2195
at com.ibm.msg.client.wmq.common.internal.WMQThreadPool.enqueue(WMQThreadPool.java:109)
at com.ibm.mq.jmqi.remote.impl.RemoteConnection.setUpAsyncMode(RemoteConnection.java:1554)
at com.ibm.mq.jmqi.remote.impl.RemoteConnection.initSess(RemoteConnection.java:1348)
at com.ibm.mq.jmqi.remote.impl.RemoteConnection.connect(RemoteConnection.java:727)
at com.ibm.mq.jmqi.remote.impl.RemoteConnectionSpecification.getSessionFromNewConnection(RemoteConnectionSpecification.java:400)
at com.ibm.mq.jmqi.remote.impl.RemoteConnectionSpecification.getSession(RemoteConnectionSpecification.java:299)
at com.ibm.mq.jmqi.remote.impl.RemoteConnectionPool.getSession(RemoteConnectionPool.java:164)
at com.ibm.mq.jmqi.remote.api.RemoteFAP.jmqiConnect(RemoteFAP.java:1598)
... 25 more
Caused by: com.ibm.msg.client.commonservices.CSIException: JMSCS0002
at com.ibm.msg.client.commonservices.workqueue.PIWorkQueueManager.enqueueItem(PIWorkQueueManager.java:67)
at com.ibm.msg.client.commonservices.workqueue.WorkQueueManager.enqueue(WorkQueueManager.java:225)
at com.ibm.msg.client.commonservices.workqueue.WorkQueueManager.enqueue(WorkQueueManager.java:194)
at com.ibm.msg.client.wmq.common.internal.WMQThreadPool.enqueue(WMQThreadPool.java:91)
Query manager log:
----- amqrmrsa.c : 898 --------------------------------------------------------
24.03.2015 17:09:06 - Process(21296.261) User(mqm) Program(amqrmppa)
Host(localserver-name) Installation(Installation1)
VRMF(7.5.0.2) QMgr(manager-name)
AMQ9209: Connection to host 'jboss-short-hostname (jboss-ip)' for channel
'channel-name' closed.
EXPLANATION:
An error occurred receiving data from 'jboss-short-hostname (jboss-ip)' over TCP/IP.
The connection to the remote host has unexpectedly terminated.
The channel name is 'channel-name'; in some cases it cannot be determined and so
is shown as '????'.
ACTION:
Tell the systems administrator.
----- amqccita.c : 3843 -------------------------------------------------------
24.03.2015 17:09:06 - Process(21296.261) User(mqm) Program(amqrmppa)
Host(localserver-name) Installation(Installation1)
VRMF(7.5.0.2) QMgr(manager-name)
AMQ9999: Channel 'channel-name' to host 'jboss-short-hostname (jboss-ip)' ended
abnormally.
EXPLANATION:
The channel program running under process ID 21296 for channel 'channel-name'
ended abnormally. The host name is 'jboss-short-hostname (jboss-ip)'; in some cases
the host name cannot be determined and so is shown as '????'.
ACTION:
Look at previous error messages for the channel program in the error logs to
determine the cause of the failure. Note that this message can be excluded
completely or suppressed by tuning the "ExcludeMessage" or "SuppressMessage"
attributes under the "QMErrorLog" stanza in qm.ini. Further information can be
found in the System Administration Guide.
----- amqrmrsa.c : 898 --------------------------------------------------------
About my test stand:
Fuse Fabric (7.2.0.redhat-024)
IBM WebSphere MQ Version : 7.5.0.2
Java 1.7
Server WebSphere MQ and jboss installed on different servers.
I installed as features following jars:
com.ibm.mq.osgi.directip_7.5.0.2.jar
com.ibm.mq.osgi.java_7.5.0.2.jar
com.ibm.msg.client.osgi.commonservices.j2se_7.5.0.2.jar
com.ibm.msg.client.osgi.jms.prereq_7.5.0.2.jar
com.ibm.msg.client.osgi.jms_7.5.0.2.jar
com.ibm.msg.client.osgi.nls_7.5.0.2.jar
com.ibm.msg.client.osgi.wmq.nls_7.5.0.2.jar
com.ibm.msg.client.osgi.wmq.prereq_7.5.0.2.jar
com.ibm.msg.client.osgi.wmq_7.5.0.2.jar
My code snippet:
MQQueueConnectionFactory cf = new MQQueueConnectionFactory();
// Config
cf.setHostName("hostname");
cf.setPort("port");
cf.setTransportType(JMSC.MQJMS_TP_CLIENT_MQ_TCPIP);
cf.setChannel("channel-name");
cf.setQueueManager("manager-name");
MQQueueConnection connectionDpc = (MQQueueConnection) cf.createQueueConnection("user", "pass");

Please try using the IBM MQ 7.5.0.6 (or later) OSGI bundles. The 7.5.0.2 bundles are known to have this issue.