I'm using the HiveMQ library in my Java Spring application to connect to a Mosquitto instance as I find it more user-friendly compared to the Paho client. But something is going wrong with the automatic reconnection. From time to time the connection is lost and the application doesn't succeed in reconnecting (see logs 1). This can also be triggered by restarting the Mosquitto broker itself (see logs 2).
This is my client builder code with additional logging in the disconnect to check if the credentials are still correct:
client = MqttClient.builder()
.useMqttVersion5()
.identifier(identifier)
.serverHost(host)
.serverPort(port)
.sslWithDefaultConfig()
// https://www.hivemq.com/blog/hivemq-mqtt-client-features/reconnect-handling/
.automaticReconnectWithDefaultConfig()
.addDisconnectedListener(context -> logger.error("MQTT user {} with identifier {} on {}:{} has disconnected reason: {}",
username, identifier, host, port, context.getCause().getMessage()))
.buildAsync();
client.connectWith()
.simpleAuth()
.username(username)
.password(password.getBytes())
.applySimpleAuth()
.cleanStart(false)
.keepAlive(60)
.send();
1/ This is shown in my logs after the connection has been lost by the application itself:
2022-03-16 02:10:33.502 ERROR 1 --- [client.mqtt-1-2] MqttConfig : MQTT user *** with identifier SERVICE on ***:8883 has disconnected reason: Timeout while waiting for PINGRESP
2022-03-16 02:11:25.090 ERROR 1 --- [client.mqtt-1-2] MqttConfig : MQTT user *** with identifier SERVICE on ***:8883 has disconnected reason: CONNECT failed as CONNACK contained an Error Code: NOT_AUTHORIZED.
2022-03-16 02:12:27.200 ERROR 1 --- [client.mqtt-1-2] MqttConfig : MQTT user *** with identifier SERVICE on ***:8883 has disconnected reason: Timeout while waiting for CONNACK
2/ This is the logs after the broker has been restarted, some expected time-outs, but also in the end "not authorized":
2022-03-16 10:17:37.178 ERROR 1 --- [client.mqtt-1-2] MqttConfig : MQTT user *** with identifier SERVICE on ***:8883 has disconnected reason: Server closed connection without DISCONNECT.
2022-03-16 10:17:48.441 ERROR 1 --- [client.mqtt-1-2] MqttConfig : MQTT user *** with identifier SERVICE on ***:8883 has disconnected reason: io.netty.channel.ConnectTimeoutException: connection timed out: ***/***:8883
2022-03-16 10:18:00.747 ERROR 1 --- [client.mqtt-1-2] MqttConfig : MQTT user *** with identifier SERVICE on ***:8883 has disconnected reason: io.netty.channel.ConnectTimeoutException: connection timed out: ***/***:8883
2022-03-16 10:18:10.625 ERROR 1 --- [client.mqtt-1-2] MqttConfig : MQTT user *** with identifier SERVICE on ***:8883 has disconnected reason: io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: No route to host: ***/***:8883
2022-03-16 10:18:26.845 ERROR 1 --- [client.mqtt-1-2] MqttConfig : MQTT user *** with identifier SERVICE on ***:8883 has disconnected reason: io.netty.channel.ConnectTimeoutException: connection timed out: ***/***:8883
2022-03-16 10:18:42.584 ERROR 1 --- [client.mqtt-1-2] MqttConfig : MQTT user *** with identifier SERVICE on ***:8883 has disconnected reason: CONNECT failed as CONNACK contained an Error Code: NOT_AUTHORIZED.
In both cases, the connection is back to normal with an application restart.
any ideas?
It appears that your question is answered in this issue:
If you set the username and password on the connect call, they will not be stored and reused when the client reconnects (for security reasons).
The following code (from the issue linked above) demonstrates the approach:
Mqtt3Client.builder()
.identifier("ePCR mobile-" + currentTimeMillis())
.serverHost(config.getHost())
.serverPort(config.getPort())
.automaticReconnectWithDefaultConfig()
.simpleAuth()
.username(config.getUsername())
.password(config.getPassword())
.applySimpleAuth()
.buildRx();
I am migrating from a self-hosted Elasticsearch FusionAuth search to an AWS Elasticsearch Service solution.
I have a new FusionAuth app EC2 instance reading from the in-use database that is configured to use the new Elasticsearch service.
On triggering a reindex from the new app instance I see that only around 60k or 62.5k documents are being written to the new index when I am expecting roughly 6mil.
I see no errors from AWS's Elasticsearch Service and in the app's logs I can see: (endpoint intentionally omitted)
Feb 13, 2020 10:18:46.116 AM INFO io.fusionauth.api.service.search.ElasticSearchClientProvider - Connecting to FusionAuth Search Engine at [https://vpc-<<omitted>>.eu-west-1.es.amazonaws.com]
13-Feb-2020 11:19:55.176 INFO [http-nio-9011-exec-3] org.apache.coyote.http11.Http11Processor.service Error parsing HTTP request header
Note: further occurrences of HTTP header parsing errors will be logged at DEBUG level.
java.lang.IllegalArgumentException: Invalid character found in method name. HTTP method names must be tokens
at org.apache.coyote.http11.Http11InputBuffer.parseRequestLine(Http11InputBuffer.java:430)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:684)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:808)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1498)
"/usr/local/fusionauth/logs/fusionauth-app.log" [readonly] 43708L, 4308629C 42183,1 96%
at io.fusionauth.api.service.search.client.domain.documents.IndexUser.<init>(IndexUser.java:79)
at io.fusionauth.api.service.search.ElasticsearchSearchEngine.lambda$index$1(ElasticsearchSearchEngine.java:140)
at java.util.ArrayList.forEach(ArrayList.java:1257)
at io.fusionauth.api.service.search.ElasticsearchSearchEngine.index(ElasticsearchSearchEngine.java:140)
at io.fusionauth.api.service.user.ReindexRunner$ReindexWorker.run(ReindexRunner.java:101)
at java.lang.Thread.run(Thread.java:748)
Exception in thread "Thread-14" java.lang.NullPointerException
at io.fusionauth.api.service.search.client.domain.documents.IndexUser.<init>(IndexUser.java:79)
at io.fusionauth.api.service.search.ElasticsearchSearchEngine.lambda$index$1(ElasticsearchSearchEngine.java:140)
at java.util.ArrayList.forEach(ArrayList.java:1257)
at io.fusionauth.api.service.search.ElasticsearchSearchEngine.index(ElasticsearchSearchEngine.java:140)
at io.fusionauth.api.service.user.ReindexRunner$ReindexWorker.run(ReindexRunner.java:101)
at java.lang.Thread.run(Thread.java:748)
Exception in thread "Thread-13" java.lang.NullPointerException
Exception in thread "Thread-11" java.lang.NullPointerException
Exception in thread "Thread-12" java.lang.NullPointerException
Feb 18, 2020 10:23:29.064 AM INFO io.fusionauth.api.service.user.ReindexRunner - Reindex completed in [86797] ms or [86] seconds.
Although there are some exceptions there is also an "Reindex completed" INFO log at the end.
Not knowing the ins-and-outs of Elasticsearch I'm also not sure where to start in investigating a NullPointerException.
It looks like the re-index operation is taking an exception which is likely the cause of the truncated index.
Exception in thread "Thread-14" java.lang.NullPointerException
at io.fusionauth.api.service.search.client.domain.documents.IndexUser.<init>(IndexUser.java:79)
This code makes an assumption that you have a username or email address. This should be enforced by the FusionAuth APIs. But in this case, for this exception to occur the email and username are both NULL.
How did you get users into the db, using the Import API, User API, or something else?
In theory you should find at least one user with a NULL value for the email and username.
This query - or similar - should find the offending user, then we need to identify how this user was added to FusionAuth.
SELECT email, username from identities WHERE email IS NULL OR username IS NULL
I am using the AS400 Nagios Plugin in order to manage AS400 servers. For one server I get the following return (Exception):
java.lang.NumberFormatException: For input string: "01 % S"
Full return:
Exception in thread "main" java.lang.NumberFormatException: For input string: "01 % S"
at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
at java.lang.Integer.parseInt(Integer.java:492)
at java.lang.Integer.<init>(Integer.java:677)
at check_as400.parseWrkSysSts(check_as400.java:1048)
at check_as400.parse(check_as400.java:626)
at check_as400.main(check_as400.java:470)
Command:
./check_as400 -H xxx.xxx.xxx.xxx -v JOBS -w 21000 -c 23000
I already found a similar problem and solution in the Nagios Exchange Forum but without access or knowledge regarding AS400 I do not have any idea what to do:
https://exchange.nagios.org/directory/Plugins/Hardware/Server-Hardware/IBM/Check_as400/details (see Owner's reply).
Any idea how to solve this problem?
Thank you in advance
EDIT: Additional output with the -d Option
Establishing connection to server...done.
Logging in...
waiting for screen...
waiting for token IBM CORP...
token received.
sending login information for QPGMR...
waiting for login to process...
waiting for token ===>...
responding to allocated to another job message...
token received.
Login completed.
Sending command (0)...
waiting for token ===>...
token received.
Command sent.
Parsing results...
Exception in thread "main" java.lang.NumberFormatException: For input st ring: "01 % S"
at java.lang.NumberFormatException.forInputString(NumberFormatEx ception.java:65)
at java.lang.Integer.parseInt(Integer.java:492)
at java.lang.Integer.<init>(Integer.java:677)
at check_as400.parseWrkSysSts(check_as400.java:1048)
at check_as400.parse(check_as400.java:626)
at check_as400.main(check_as400.java:470)
Thanks to #WillRyan: The Problem is that the server which causes the exceptions had an older version of AS400 (< V6R1). Therefor the ":"-signs which greps the wanted information did not correctly match.
I'm getting an exception from a Pig script and haven't been able to nail down the cause. I'm fairly new to Pig & have searched for
various topics based on the exception I'm getting but haven't been
able to find anything meaningful. From the grunt shell & log I've
looked for different variations of these - unable to read pigs
manifest file java.lang.NegativeArraySizeException: -1 ERROR 1066:
Unable to open iterator for alias F. Backend error : -1
I'm using Hadoop version 2.0.0-cdh4.6.0 & Pig version 0.11.0, running
from the Grunt shell.
My Pig script reads a file, does some manipulation on the data
(including calling a Java UDF), joins to an HBase table, then DUMPs
the output. Pretty simple. I can DUMP the intermediate result (alias
B) and the data looks fine.
I've tested the Java function from Pig using the same input file and
have seen it return values as I'd expect, and I've tested the function
locally outside the Pig script. The Java function is provided a number
of days from 01-01-1900 & uses joda-time v2.7 to return a Datetime.
Initially, the UDF was accepting a tuple as input. I've tried changing
the UDF input format to Byte and most recently String and casting to
Datetime in Pig upon returning, but am still getting the same error.
When I change my Pig script merely to not call the UDF it works fine.
The NegativeArray error sounds like the data is out of whack for the
Dump, possibly from some kind of format issue, but I don't see how.
Pig script
A = LOAD 'tst2_SplitGroupMax.txt' using PigStorage(',')
as (id:bytearray, year:int, doy:int, month:int, dayOfMonth:int,
awh_minTemp:double, awh_maxTemp:double,
nws_minTemp:double, nws_maxTemp:double,
wxs_minTemp:double, wxs_maxTemp:double,
tcc_minTemp:double, tcc_maxTemp:double
) ;
register /import/pool2/home/NA1000APP-TPSDM/ejbles/Test-0.0.1-SNAPSHOT-jar-with-dependencies.jar;
B = FOREACH A GENERATE id as msmtid, SUBSTRING(id,0,8) as gridid, SUBSTRING(id,9,20) as msmt_days,
year, doy, month, dayOfMonth,
CONCAT(CONCAT(CONCAT((chararray)year,'-'),CONCAT((chararray)month,'-')),(chararray)dayOfMonth) as msmt_dt,
ToDate(monutil.geoloc.GridIDtoDatetime(id)) as func_msmt_dt,
awh_minTemp, awh_maxTemp,
nws_minTemp, nws_maxTemp,
wxs_minTemp, wxs_maxTemp,
tcc_minTemp, tcc_maxTemp
;
E = LOAD 'hbase://wxgrid_detail' using org.apache.pig.backend.hadoop.hbase.HBaseStorage
('loc:country, loc:fips, loc:l1 ,loc:l2, loc:latitude, loc:longitude',
'-loadKey=true -caster=HBaseBinaryConverter')
as (wxgrid:bytearray, country:chararray, fips:chararray, l1:chararray, l2:chararray,
latitude:double, longitude:double);
F = join B by gridid, E by wxgrid;
DUMP F; --- This is where I get the exception
Here's an excerpt from what's returned in the Grunt shell -
2015-06-15 12:23:24,204 [main] WARN
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- Ooops! Some job has failed! Specify -stop_on_failure if you want Pig to stop immediately on failure. 2015-06-15 12:23:24,205 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- job job_201502081759_916870 has failed! Stop running all dependent jobs 2015-06-15 12:23:24,205 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 100% complete 2015-06-15 12:23:24,221 [main] ERROR org.apache.pig.tools.pigstats.SimplePigStats - ERROR: -1 2015-06-15
12:23:24,221 [main] ERROR org.apache.pig.tools.pigstats.PigStatsUtil -
1 map reduce job(s) failed! 2015-06-15 12:23:24,223 [main] WARN
org.apache.pig.tools.pigstats.ScriptState - unable to read pigs
manifest file 2015-06-15 12:23:24,224 [main] INFO
org.apache.pig.tools.pigstats.SimplePigStats - Script Statistics:
HadoopVersion PigVersion UserId StartedAt FinishedAt
Features
2.0.0-cdh4.6.0 na1000app-tpsdm 2015-06-15 12:22:39 2015-06-15 12:23:24 HASH_JOIN
Failed!
Failed Jobs: JobId Alias Feature Message Outputs
job_201502081759_916870 A,B,E,F HASH_JOIN Message: Job failed!
hdfs://nameservice1/tmp/temp-238648079/tmp-1338617620,
Input(s): Failed to read data from "hbase://wxgrid_detail" Failed to
read data from
"hdfs://nameservice1/user/na1000app-tpsdm/tst2_SplitGroupMax.txt"
Output(s): Failed to produce result in
"hdfs://nameservice1/tmp/temp-238648079/tmp-1338617620"
Counters: Total records written : 0 Total bytes written : 0 Spillable
Memory Manager spill count : 0 Total bags proactively spilled: 0 Total
records proactively spilled: 0
Job DAG: job_201502081759_916870
2015-06-15 12:23:24,224 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- Failed! 2015-06-15 12:23:24,234 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1066: Unable to open iterator
for alias F. Backend error : -1 Details at logfile:
/import/pool2/home/NA1000APP-TPSDM/ejbles/pig_1434388844905.log
And here's the log -
Backend error message
--------------------- java.lang.NegativeArraySizeException: -1 at org.apache.hadoop.hbase.util.Bytes.readByteArray(Bytes.java:148) at
org.apache.hadoop.hbase.mapreduce.TableSplit.readFields(TableSplit.java:133)
at
org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:73)
at
org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:44)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigSplit.readFields(PigSplit.java:233)
at
org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:73)
at
org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:44)
at org.apache.hadoop.mapred.MapTask.getSplitDetails(MapTask.java:356)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:640)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:330) at
org.apache.hadoop.mapred.Child$4.run(Ch
Pig Stack Trace
--------------- ERROR 1066: Unable to open iterator for alias F. Backend error : -1
org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable
to open iterator for alias F. Backend error : -1 at
org.apache.pig.PigServer.openIterator(PigServer.java:828) at
org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:696)
at
org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:320)
at
org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:194)
at
org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:170)
at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:69) at
org.apache.pig.Main.run(Main.java:538) at
org.apache.pig.Main.main(Main.java:157) at
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597) at
org.apache.hadoop.util.RunJar.main(RunJar.java:208) Caused by:
java.lang.NegativeArraySizeException: -1 at
org.apache.hadoop.hbase.util.Bytes.readByteArray(Bytes.java:148) at
org.apache.hadoop.hbase.mapreduce.TableSplit.readFields(TableSplit.java:133)
at
org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:73)
at
org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:44)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigSplit.readFields(PigSplit.java:233)
at
org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:73)
at
org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:44)
at org.apache.hadoop.mapred.MapTask.getSplitDetails(MapTask.java:356)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:640)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:330)
I'm trying to connect to remote HDFS cluster. I've read some documentation and getting started's but didn't find a best solution how to do that.
Situation: I have HDFS on xxx-something.com. I can connect to it via SSH and everything works.
But what I'm trying to do, get the files from it to my local machine.
What I've done:
I've created core-site.xml in my conf folder (I'm creating Play! application). There I've changed fs.default.name config to hdfs://xxx-something.com:8020 (not sure about the port).
Then I'm trying to launch a simple test:
val conf = new Configuration()
conf.addResource(new Path("conf/core-site.xml"))
val fs = FileSystem.get(conf)
val status = fs.listStatus(new Path("/data/"))
And I'm getting errors:
13:56:09.012 [specs2.DefaultExecutionStrategy1] WARN org.apache.hadoop.conf.Configuration - conf/core-site.xml:a attempt to override final parameter: fs.trash.interval; Ignoring.
13:56:09.012 [specs2.DefaultExecutionStrategy1] WARN org.apache.hadoop.conf.Configuration - conf/core-site.xml:a attempt to override final parameter: hadoop.tmp.dir; Ignoring.
13:56:09.013 [specs2.DefaultExecutionStrategy1] WARN org.apache.hadoop.conf.Configuration - conf/core-site.xml:a attempt to override final parameter: fs.checkpoint.dir; Ignoring.
13:56:09.022 [specs2.DefaultExecutionStrategy1] DEBUG org.apache.hadoop.fs.FileSystem - Creating filesystem for hdfs://xxx-something.com:8021
13:56:09.059 [specs2.DefaultExecutionStrategy1] DEBUG org.apache.hadoop.conf.Configuration - java.io.IOException: config()
at org.apache.hadoop.conf.Configuration.<init>(Configuration.java:226)
at org.apache.hadoop.conf.Configuration.<init>(Configuration.java:213)
at org.apache.hadoop.security.SecurityUtil.<clinit>(SecurityUtil.java:53)
at org.apache.hadoop.net.NetUtils.<clinit>(NetUtils.java:62)
Thanks in advance!
Update:
probably the port was wrong. Now I set it to 22, I'm still getting same errors, but after 3 times it does say:
14:01:01.877 [specs2.DefaultExecutionStrategy1] DEBUG org.apache.hadoop.ipc.Client - Connecting to xxx-something.com/someIp:22
14:01:02.187 [specs2.DefaultExecutionStrategy1] DEBUG org.apache.hadoop.ipc.Client - IPC Client (47) connection to xxx-something.com/someIp:22 from britva sending #0
14:01:02.188 [IPC Client (47) connection to xxx-something.com/someIp:22 from britva] DEBUG org.apache.hadoop.ipc.Client - IPC Client (47) connection to xxx-something.com/someIp:22 from britva: starting, having connections 1
14:01:02.422 [IPC Client (47) connection to xxx-something.com/someIp:22 from britva] DEBUG org.apache.hadoop.ipc.Client - IPC Client (47) connection to xxx-something.com/someIp:22 from britva got value #1397966893
And afterwards:
Call to xxx-something.com/someIp:22 failed on local exception: java.io.EOFException
java.io.IOException: Call to xxx-something.com/someIp:22 failed on local exception: java.io.EOFException
at org.apache.hadoop.ipc.Client.wrapException(Client.java:1103)
at org.apache.hadoop.ipc.Client.call(Client.java:1071)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225)
at com.sun.proxy.$Proxy1.getProtocolVersion(Unknown Source)
at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:396)
at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:379)
at org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:118)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:222)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:187)
at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:89)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1328)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:65)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1346)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:244)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:122)
at HdfsSpec$$anonfun$1$$anonfun$apply$3.apply(HdfsSpec.scala:33)
at HdfsSpec$$anonfun$1$$anonfun$apply$3.apply(HdfsSpec.scala:17)
at testingSupport.specs2.MyNotifierRunner$$anon$2$$anon$1.executeBody(MyNotifierRunner.scala:16)
at testingSupport.specs2.MyNotifierRunner$$anon$2$$anon$1.execute(MyNotifierRunner.scala:16)
Caused by: java.io.EOFException
at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:807)
at org.apache.hadoop.ipc.Client$Connection.run(Client.java:745)
What does it mean?
You'll need to find the fs.default.name property in the $HADOOP_HOME/conf/core-site.xml on the server running the Name Node (HDFS master) to get the correct port. It might be 8020, or it could be something else. That's what you should use. Make sure there's no firewall between you and the server that disallows connections on the port.