I am embedding the apache ftpserver in my java web app, I have tried a couple different things and everytime I try to connect to the server via FTP, it says Authentication failed. I have attached my code, and also the console output when I try to do the logging in. Does anyone have any suggestions? I tried running this on OS X and Centos Linux.
PropertiesUserManagerFactory userManagerFactory = new PropertiesUserManagerFactory();
userManagerFactory.setFile(new File("/ftpdata/myusers.properties"));
//userManagerFactory.setPasswordEncryptor(new SaltedPasswordEncryptor());
userManagerFactory.setPasswordEncryptor(new ClearTextPasswordEncryptor());
UserManager um = userManagerFactory.createUserManager();
BaseUser user = new BaseUser();
user.setName("andy");
user.setPassword("12345678");
user.setHomeDirectory("/ftpdata/");
um.save(user);
FtpServerFactory serverFactory = new FtpServerFactory();
ListenerFactory factory = new ListenerFactory();
factory.setPort(2225);
serverFactory.addListener("default", factory.createListener());
FtpServer server = serverFactory.createServer();
server.start();
12:03:28,768 INFO [STDOUT] 12:03:28,768 INFO [DefaultFtpServer] FTP server started
12:04:14,200 INFO [STDOUT] 12:04:14,200 INFO [FtpLoggingFilter] CREATED
12:04:14,208 INFO [STDOUT] 12:04:14,208 INFO [FtpLoggingFilter] OPENED
12:04:14,228 INFO [STDOUT] 12:04:14,228 INFO [FtpLoggingFilter] RECEIVED: USER andy
12:04:14,230 INFO [STDOUT] 12:04:14,230 INFO [FtpLoggingFilter] SENT: 220 Service ready for new user.
12:04:14,232 INFO [STDOUT] 12:04:14,232 INFO [FtpLoggingFilter] SENT: 331 User name okay, need password for andy.
12:04:14,233 INFO [STDOUT] 12:04:14,233 INFO [FtpLoggingFilter] RECEIVED: PASS *****
12:04:14,236 INFO [STDOUT] 12:04:14,236 WARN [PASS] User failed to log in
12:04:14,736 INFO [STDOUT] 12:04:14,736 WARN [PASS] Login failure - andy
12:04:14,740 INFO [STDOUT] 12:04:14,740 INFO [FtpLoggingFilter] RECEIVED: QUIT
12:04:14,746 INFO [STDOUT] 12:04:14,746 INFO [FtpLoggingFilter] SENT: 530 Authentication failed.
12:04:14,746 INFO [STDOUT] 12:04:14,746 INFO [FtpLoggingFilter] SENT: 221 Goodbye.
12:04:14,747 INFO [STDOUT] 12:04:14,746 INFO [FtpLoggingFilter] CLOSED
my question may sound stupid but
i can see in the log
12:04:14,233 INFO [STDOUT] 12:04:14,233 INFO [FtpLoggingFilter] RECEIVED: PASS *****
These are 5 stars , your password length is 8 :D
I can see that you pass a property file, is there any chance that you have a password in there?
At last, you create a UserManager, you dont have to pass this object somewhere?
Related
The Kafka service I deployed on the server passed the test a few days ago and was running normally. Later, no other operations were carried out.
Then the service call failed today. I went online and looked at it. It's normal these days. Just this afternoon, the service hung up. Looking at the log, it seems to say that the Kafka service is restarted automatically, and then conflicts with the ID of zookeeper.
Restart the service. Everything is normal. Google doesn't have a good solution. What's the reason?
The following is the relevant log, please help to see, thank you
[2021-11-06 14:43:29,139] WARN Unexpected exception (org.apache.zookeeper.server.NIOServerCnxn)
EndOfStreamException: Unable to read additional data from client, it probably closed the socket: address = /159.89.4.236:59168, session = 0x0
at org.apache.zookeeper.server.NIOServerCnxn.handleFailedRead(NIOServerCnxn.java:163)
at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:326)
at org.apache.zookeeper.server.NIOServerCnxnFactory$IOWorkRequest.doWork(NIOServerCnxnFactory.java:522)
at org.apache.zookeeper.server.WorkerService$ScheduledWorkRequest.run(WorkerService.java:154)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
[2021-11-06 15:00:18,301] INFO Completed load of Log(dir=/tmp/kafka-logs/__consumer_offsets-46, topicId=dPs0Kq4ERHKDaR440Nt8qA, topic=__consumer_offsets, partition=46, highWatermark=0, lastStableOffset=0, logStartOffset=0, logEndOffset=0) with 1 segments in 20ms (53/53 loaded in /tmp/kafka-logs) (kafka.log.LogManager)
[2021-11-06 15:00:18,303] INFO Loaded 53 logs in 2350ms. (kafka.log.LogManager)
[2021-11-06 15:00:18,303] INFO Starting log cleanup with a period of 300000 ms. (kafka.log.LogManager)
[2021-11-06 15:00:18,304] INFO Starting log flusher with a default period of 9223372036854775807 ms. (kafka.log.LogManager)
[2021-11-06 15:00:18,838] INFO [BrokerToControllerChannelManager broker=0 name=forwarding]: Starting (kafka.server.BrokerToControllerRequestThread)
[2021-11-06 15:00:19,148] INFO Updated connection-accept-rate max connection creation rate to 2147483647 (kafka.network.ConnectionQuotas)
[2021-11-06 15:00:19,165] INFO Awaiting socket connections on 0.0.0.0:9092. (kafka.network.Acceptor)
[2021-11-06 15:00:19,237] INFO [SocketServer listenerType=ZK_BROKER, nodeId=0] Created data-plane acceptor and processors for endpoint : ListenerName(PLAINTEXT) (kafka.network.SocketServer)
[2021-11-06 15:00:19,282] INFO [BrokerToControllerChannelManager broker=0 name=alterIsr]: Starting (kafka.server.BrokerToControllerRequestThread)
[2021-11-06 15:00:19,400] INFO [ExpirationReaper-0-DeleteRecords]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2021-11-06 15:00:19,400] INFO [ExpirationReaper-0-Fetch]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2021-11-06 15:00:19,400] INFO [ExpirationReaper-0-Produce]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2021-11-06 15:00:19,409] INFO [ExpirationReaper-0-ElectLeader]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2021-11-06 15:00:19,451] INFO [LogDirFailureHandler]: Starting (kafka.server.ReplicaManager$LogDirFailureHandler)
[2021-11-06 15:00:19,530] INFO Creating /brokers/ids/0 (is it secure? false) (kafka.zk.KafkaZkClient)
[2021-11-06 15:00:19,566] ERROR Error while creating ephemeral at /brokers/ids/0, node already exists and owner '72057654275276801' does not match current session '72057654275276802' (kafka.zk.KafkaZkClient$CheckedEphemeral)
[2021-11-06 15:00:19,572] ERROR [KafkaServer id=0] Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists
at org.apache.zookeeper.KeeperException.create(KeeperException.java:126)
at kafka.zk.KafkaZkClient$CheckedEphemeral.getAfterNodeExists(KafkaZkClient.scala:1904)
at kafka.zk.KafkaZkClient$CheckedEphemeral.create(KafkaZkClient.scala:1842)
at kafka.zk.KafkaZkClient.checkedEphemeralCreate(KafkaZkClient.scala:1809)
at kafka.zk.KafkaZkClient.registerBroker(KafkaZkClient.scala:96)
at kafka.server.KafkaServer.startup(KafkaServer.scala:319)
at kafka.Kafka$.main(Kafka.scala:109)
at kafka.Kafka.main(Kafka.scala)
[2021-11-06 15:00:19,596] INFO [KafkaServer id=0] shutting down (kafka.server.KafkaServer)
[2021-11-06 15:00:19,597] INFO [SocketServer listenerType=ZK_BROKER, nodeId=0] Stopping socket server request processors (kafka.network.SocketServer)```
I am using ZIPKINS for distributed tracing the problem is when I am trying to test ZIPKINS by sending 10 request at a time to that service from other service by using loop, checked the UI for the logs of that, I had got only 2 logs i.e for first and last, I haven't received logs of the remaining requests. Can you help in figuring out what is the problem in that. Trace ids and span ids are generated for all the request, I am unable to see that logs for the same. Logs that are received:
2020-03-04 17:38:57.379 INFO [,7c8075c14691f988,43521ecc69b84d84,true] 10576 --- [nio-8081-exec-7] c.i.f.service.ProducerServiceImpl : Received Message ='ServiceInvocation [communicationID=COMM_0121, 2020-03-04 17:38:57.438 INFO [,7552e8c3d87d013a,89769451aafec094,false] –
10576 --- [nio-8081-exec-8] c.i.f.service.ProducerServiceImpl : Received Message ='ServiceInvocation [communicationID=COMM_0122, 2020-03-04 17:38:57.519 INFO [,79f38c25211dfab8,49ea12575eab0bcf,false] 10576 --- [nio-8081-exec-2] c.i.f.service.ProducerServiceImpl : Received –
Message ='ServiceInvocation [communicationID=COMM_0123, 2020-03-04 17:38:57.626 INFO [,294da34664fac032,ad98ed1fbce485df,false] 10576 --- [io-8081-exec-10] c.i.f.service.ProducerServiceImpl : Received Message ='ServiceInvocation [communicationID=COMM_0124, 2020-03-04 17:38:57.879 INFO [,8763a2ca3d6dfc44,9871d046cd7eacf1,false] 10576 --- [nio-8081-exec-1] c.i.f.service.ProducerServiceImpl : Received Message ='ServiceInvocation [communicationID=COMM_0125, 2020-03-04 17:38:57.923 INFO [,be1e3a490e114e92,2435ee34d215459c,false] –
10576 --- [nio-8081-exec-6] c.i.f.service.ProducerServiceImpl : Received Message ='ServiceInvocation [communicationID=COMM_0126, 2020-03-04 17:38:57.980 INFO [,21855ca20670de31,6213a3fdc0a23189,false] 10576 --- [nio-8081-exec-3] c.i.f.service.ProducerServiceImpl : Received Message ='ServiceInvocation [communicationID=COMM_0127, 2020-03-04 17:38:58.043 INFO [,4d9795e7d2dbf50c,21f83b3384381833,false] 10576 --- [nio-8081-exec-4] c.i.f.service.ProducerServiceImpl : Receive
Log format is: [application name, traceId, spanId, export]
So the last value true/false is actually the export value which means:
Export – This property is a boolean that indicates whether or not this log was exported to an aggregator like Zipkin. Zipkin is beyond the scope of this article but plays an important role in analyzing logs created by Sleuth.
As export values are false, zipkin is not receiving those values. That is expected behavior.
Root Cause: Some of the logs are exporting and some are not is because of sampler rate. Not all of the logs are meant to be sent. Anyway if you want all of the logs to be sampled try adding this property:
spring.sleuth.sampler.probability=1.0
Reference: sleuth-not-sending-trace-information-to-zipkin
I'm trying to create Kafka topics using Java. But I get a Exception in thread "main" java.lang.NoSuchFieldError: DEFAULT_SSL_PRINCIPAL_MAPPING_RULES and I can't fix it.
My goal is to create a topic so that when I run my Kafka server to display my topics using this command bin/kafka-topics.sh --list --bootstrap-server localhost:9092, I can actually see my topic in the list.
(the command is from Kafka's official website https://kafka.apache.org/quickstart)
I looked up to this problem How to create a Topic in Kafka through Java which actually inspired my code, but not only it doesn't really help, but it seems like it uses deprecated classes and methods.
I tried to use what I believed to be more recent classes such as ZooKeeperClient, KafkaZkClient and AdminZkClient, but from what I understand, the method adminZkClient.createTopic(topic, noOfPartitions, noOfReplication, topicConfiguration, RackAwareMode.Disabled$.MODULE$); is what brings the exception. I don't know what it is about that function that creates the exception, whether I forgot something from my application.properties file or something else.
Here's my code
import java.util.Properties;
import org.I0Itec.zkclient.ZkClient;
import org.I0Itec.zkclient.ZkConnection;
import org.apache.kafka.common.utils.Time;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.data.jpa.repository.config.EnableJpaAuditing;
import kafka.admin.AdminUtils;
import kafka.admin.RackAwareMode;
import kafka.utils.ZkUtils;
import kafka.zk.AdminZkClient;
import kafka.zk.KafkaZkClient;
import kafka.zookeeper.ZooKeeperClient;
import kafka.utils.*;
#SpringBootApplication
#EnableJpaAuditing
public class ChatApplication {
#SuppressWarnings("deprecation")
public static void main(String[] args) {
try {
String zookeeperHost = "127.0.0.1:2181";
int sessionTimeOutInMs = 15 * 1000;
int connectionTimeOutInMs = 10 * 1000;
ZooKeeperClient zooKeeperClient = new ZooKeeperClient(zookeeperHost, sessionTimeOutInMs, connectionTimeOutInMs, 2, Time.SYSTEM, "BytesInPerSec", "BytesOutPerSec");
KafkaZkClient kafkaZkClient = new KafkaZkClient(zooKeeperClient, true, Time.SYSTEM);
String topic = "superTopic";
int noOfPartitions = 2;
int noOfReplication = 1;
Properties topicConfiguration = new Properties();
AdminZkClient adminZkClient = new AdminZkClient(kafkaZkClient);
adminZkClient.createTopic(topic, noOfPartitions, noOfReplication, topicConfiguration, RackAwareMode.Disabled$.MODULE$);
} catch (Exception ex) {
ex.printStackTrace();
}
}
//SpringApplication.run(ChatApplication.class, args);
}
Here's the output I get :
06:50:06.796 [main] INFO kafka.utils.Log4jControllerRegistration$ - Registered kafka:type=kafka.Log4jController MBean
06:50:07.101 [main] INFO kafka.zookeeper.ZooKeeperClient - [ZooKeeperClient] Initializing a new session to 127.0.0.1:2181.
06:50:07.116 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:zookeeper.version=3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03, built on 06/29/2018 00:39 GMT
06:50:07.116 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:host.name=kunta
06:50:07.116 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.version=11.0.3
06:50:07.116 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.vendor=Oracle Corporation
06:50:07.116 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.home=/usr/lib/jvm/java-11-openjdk-amd64
06:50:07.116 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.class.path=/home/robscientist/STS-WORKSPACE/poc_chat/target/classes:/home/robscientist/.m2/repository/org/springframework/boot/spring-boot-starter-web/2.1.3.RELEASE/spring-boot-starter-web-2.1.3.RELEASE.jar:/home/robscientist/.m2/repository/org/springframework/boot/spring-boot-starter/2.1.3.RELEASE/spring-boot-starter-2.1.3.RELEASE.jar:/home/robscientist/.m2/repository/org/springframework/boot/spring-boot-starter-logging/2.1.3.RELEASE/spring-boot-starter-logging-2.1.3.RELEASE.jar:/home/robscientist/.m2/repository/ch/qos/logback/logback-classic/1.2.3/logback-classic-1.2.3.jar:/home/robscientist/.m2/repository/ch/qos/logback/logback-core/1.2.3/logback-core-1.2.3.jar:/home/robscientist/.m2/repository/org/apache/logging/log4j/log4j-to-slf4j/2.11.2/log4j-to-slf4j-2.11.2.jar:/home/robscientist/.m2/repository/org/apache/logging/log4j/log4j-api/2.11.2/log4j-api-2.11.2.jar:/home/robscientist/.m2/repository/org/slf4j/jul-to-slf4j/1.7.25/jul-to-slf4j-1.7.25.jar:/home/robscientist/.m2/repository/javax/annotation/javax.annotation-api/1.3.2/javax.annotation-api-1.3.2.jar:/home/robscientist/.m2/repository/org/yaml/snakeyaml/1.23/snakeyaml-1.23.jar:/home/robscientist/.m2/repository/org/springframework/boot/spring-boot-starter-json/2.1.3.RELEASE/spring-boot-starter-json-2.1.3.RELEASE.jar:/home/robscientist/.m2/repository/com/fasterxml/jackson/datatype/jackson-datatype-jsr310/2.9.8/jackson-datatype-jsr310-2.9.8.jar:/home/robscientist/.m2/repository/com/fasterxml/jackson/module/jackson-module-parameter-names/2.9.8/jackson-module-parameter-names-2.9.8.jar:/home/robscientist/.m2/repository/org/springframework/boot/spring-boot-starter-tomcat/2.1.3.RELEASE/spring-boot-starter-tomcat-2.1.3.RELEASE.jar:/home/robscientist/.m2/repository/org/apache/tomcat/embed/tomcat-embed-core/9.0.16/tomcat-embed-core-9.0.16.jar:/home/robscientist/.m2/repository/org/apache/tomcat/embed/tomcat-embed-el/9.0.16/tomcat-embed-el-9.0.16.jar:/home/robscientist/.m2/repository/org/apache/tomcat/embed/tomcat-embed-websocket/9.0.16/tomcat-embed-websocket-9.0.16.jar:/home/robscientist/.m2/repository/org/hibernate/validator/hibernate-validator/6.0.14.Final/hibernate-validator-6.0.14.Final.jar:/home/robscientist/.m2/repository/javax/validation/validation-api/2.0.1.Final/validation-api-2.0.1.Final.jar:/home/robscientist/.m2/repository/org/jboss/logging/jboss-logging/3.3.2.Final/jboss-logging-3.3.2.Final.jar:/home/robscientist/.m2/repository/com/fasterxml/classmate/1.4.0/classmate-1.4.0.jar:/home/robscientist/.m2/repository/org/springframework/spring-web/5.1.5.RELEASE/spring-web-5.1.5.RELEASE.jar:/home/robscientist/.m2/repository/org/springframework/spring-webmvc/5.1.5.RELEASE/spring-webmvc-5.1.5.RELEASE.jar:/home/robscientist/.m2/repository/org/springframework/spring-aop/5.1.5.RELEASE/spring-aop-5.1.5.RELEASE.jar:/home/robscientist/.m2/repository/org/springframework/spring-expression/5.1.5.RELEASE/spring-expression-5.1.5.RELEASE.jar:/home/robscientist/.m2/repository/io/debezium/debezium-core/0.9.5.Final/debezium-core-0.9.5.Final.jar:/home/robscientist/.m2/repository/com/fasterxml/jackson/core/jackson-core/2.9.8/jackson-core-2.9.8.jar:/home/robscientist/.m2/repository/io/debezium/debezium-core/0.9.5.Final/debezium-core-0.9.5.Final-tests.jar:/home/robscientist/.m2/repository/org/springframework/spring-jdbc/5.1.5.RELEASE/spring-jdbc-5.1.5.RELEASE.jar:/home/robscientist/.m2/repository/org/springframework/spring-beans/5.1.5.RELEASE/spring-beans-5.1.5.RELEASE.jar:/home/robscientist/.m2/repository/org/springframework/spring-core/5.1.5.RELEASE/spring-core-5.1.5.RELEASE.jar:/home/robscientist/.m2/repository/org/springframework/spring-jcl/5.1.5.RELEASE/spring-jcl-5.1.5.RELEASE.jar:/home/robscientist/.m2/repository/org/springframework/spring-tx/5.1.5.RELEASE/spring-tx-5.1.5.RELEASE.jar:/home/robscientist/.m2/repository/org/springframework/boot/spring-boot-starter-websocket/2.1.3.RELEASE/spring-boot-starter-websocket-2.1.3.RELEASE.jar:/home/robscientist/.m2/repository/org/springframework/spring-messaging/5.1.5.RELEASE/spring-messaging-5.1.5.RELEASE.jar:/home/robscientist/.m2/repository/org/springframework/spring-websocket/5.1.5.RELEASE/spring-websocket-5.1.5.RELEASE.jar:/home/robscientist/.m2/repository/org/springframework/boot/spring-boot-starter-data-jpa/2.1.3.RELEASE/spring-boot-starter-data-jpa-2.1.3.RELEASE.jar:/home/robscientist/.m2/repository/org/springframework/boot/spring-boot-starter-aop/2.1.3.RELEASE/spring-boot-starter-aop-2.1.3.RELEASE.jar:/home/robscientist/.m2/repository/org/aspectj/aspectjweaver/1.9.2/aspectjweaver-1.9.2.jar:/home/robscientist/.m2/repository/org/springframework/boot/spring-boot-starter-jdbc/2.1.3.RELEASE/spring-boot-starter-jdbc-2.1.3.RELEASE.jar:/home/robscientist/.m2/repository/com/zaxxer/HikariCP/3.2.0/HikariCP-3.2.0.jar:/home/robscientist/.m2/repository/javax/transaction/javax.transaction-api/1.3/javax.transaction-api-1.3.jar:/home/robscientist/.m2/repository/javax/xml/bind/jaxb-api/2.3.1/jaxb-api-2.3.1.jar:/home/robscientist/.m2/repository/javax/activation/javax.activation-api/1.2.0/javax.activation-api-1.2.0.jar:/home/robscientist/.m2/repository/org/hibernate/hibernate-core/5.3.7.Final/hibernate-core-5.3.7.Final.jar:/home/robscientist/.m2/repository/javax/persistence/javax.persistence-api/2.2/javax.persistence-api-2.2.jar:/home/robscientist/.m2/repository/org/javassist/javassist/3.23.1-GA/javassist-3.23.1-GA.jar:/home/robscientist/.m2/repository/net/bytebuddy/byte-buddy/1.9.10/byte-buddy-1.9.10.jar:/home/robscientist/.m2/repository/antlr/antlr/2.7.7/antlr-2.7.7.jar:/home/robscientist/.m2/repository/org/jboss/jandex/2.0.5.Final/jandex-2.0.5.Final.jar:/home/robscientist/.m2/repository/org/dom4j/dom4j/2.1.1/dom4j-2.1.1.jar:/home/robscientist/.m2/repository/org/hibernate/common/hibernate-commons-annotations/5.0.4.Final/hibernate-commons-annotations-5.0.4.Final.jar:/home/robscientist/.m2/repository/org/springframework/data/spring-data-jpa/2.1.5.RELEASE/spring-data-jpa-2.1.5.RELEASE.jar:/home/robscientist/.m2/repository/org/springframework/data/spring-data-commons/2.1.5.RELEASE/spring-data-commons-2.1.5.RELEASE.jar:/home/robscientist/.m2/repository/org/springframework/spring-orm/5.1.5.RELEASE/spring-orm-5.1.5.RELEASE.jar:/home/robscientist/.m2/repository/org/springframework/spring-aspects/5.1.5.RELEASE/spring-aspects-5.1.5.RELEASE.jar:/home/robscientist/.m2/repository/com/h2database/h2/1.4.197/h2-1.4.197.jar:/home/robscientist/.m2/repository/org/webjars/stomp-websocket/2.3.3/stomp-websocket-2.3.3.jar:/home/robscientist/.m2/repository/org/webjars/bootstrap/3.3.7/bootstrap-3.3.7.jar:/home/robscientist/.m2/repository/org/webjars/jquery/1.11.1/jquery-1.11.1.jar:/home/robscientist/.m2/repository/org/springframework/kafka/spring-kafka/2.2.4.RELEASE/spring-kafka-2.2.4.RELEASE.jar:/home/robscientist/.m2/repository/org/springframework/spring-context/5.1.5.RELEASE/spring-context-5.1.5.RELEASE.jar:/home/robscientist/.m2/repository/org/springframework/retry/spring-retry/1.2.4.RELEASE/spring-retry-1.2.4.RELEASE.jar:/home/robscientist/.m2/repository/org/apache/kafka/kafka-clients/2.0.1/kafka-clients-2.0.1.jar:/home/robscientist/.m2/repository/org/lz4/lz4-java/1.4.1/lz4-java-1.4.1.jar:/home/robscientist/.m2/repository/org/xerial/snappy/snappy-java/1.1.7.1/snappy-java-1.1.7.1.jar:/home/robscientist/.m2/repository/org/apache/kafka/kafka_2.12/2.2.0/kafka_2.12-2.2.0.jar:/home/robscientist/.m2/repository/com/fasterxml/jackson/core/jackson-databind/2.9.8/jackson-databind-2.9.8.jar:/home/robscientist/.m2/repository/com/fasterxml/jackson/core/jackson-annotations/2.9.0/jackson-annotations-2.9.0.jar:/home/robscientist/.m2/repository/com/fasterxml/jackson/datatype/jackson-datatype-jdk8/2.9.8/jackson-datatype-jdk8-2.9.8.jar:/home/robscientist/.m2/repository/net/sf/jopt-simple/jopt-simple/5.0.4/jopt-simple-5.0.4.jar:/home/robscientist/.m2/repository/com/yammer/metrics/metrics-core/2.2.0/metrics-core-2.2.0.jar:/home/robscientist/.m2/repository/org/scala-lang/scala-library/2.12.8/scala-library-2.12.8.jar:/home/robscientist/.m2/repository/org/scala-lang/scala-reflect/2.12.8/scala-reflect-2.12.8.jar:/home/robscientist/.m2/repository/com/typesafe/scala-logging/scala-logging_2.12/3.9.0/scala-logging_2.12-3.9.0.jar:/home/robscientist/.m2/repository/org/slf4j/slf4j-api/1.7.25/slf4j-api-1.7.25.jar:/home/robscientist/.m2/repository/com/101tec/zkclient/0.11/zkclient-0.11.jar:/home/robscientist/.m2/repository/org/apache/zookeeper/zookeeper/3.4.13/zookeeper-3.4.13.jar:/home/robscientist/.m2/repository/org/slf4j/slf4j-log4j12/1.7.25/slf4j-log4j12-1.7.25.jar:/home/robscientist/.m2/repository/log4j/log4j/1.2.17/log4j-1.2.17.jar:/home/robscientist/.m2/repository/jline/jline/0.9.94/jline-0.9.94.jar:/home/robscientist/.m2/repository/org/apache/yetus/audience-annotations/0.5.0/audience-annotations-0.5.0.jar:/home/robscientist/.m2/repository/io/netty/netty/3.10.6.Final/netty-3.10.6.Final.jar:/home/robscientist/.m2/repository/org/springframework/boot/spring-boot-starter-reactor-netty/2.1.3.RELEASE/spring-boot-starter-reactor-netty-2.1.3.RELEASE.jar:/home/robscientist/.m2/repository/io/projectreactor/netty/reactor-netty/0.8.5.RELEASE/reactor-netty-0.8.5.RELEASE.jar:/home/robscientist/.m2/repository/io/netty/netty-codec-http/4.1.33.Final/netty-codec-http-4.1.33.Final.jar:/home/robscientist/.m2/repository/io/netty/netty-common/4.1.33.Final/netty-common-4.1.33.Final.jar:/home/robscientist/.m2/repository/io/netty/netty-buffer/4.1.33.Final/netty-buffer-4.1.33.Final.jar:/home/robscientist/.m2/repository/io/netty/netty-transport/4.1.33.Final/netty-transport-4.1.33.Final.jar:/home/robscientist/.m2/repository/io/netty/netty-resolver/4.1.33.Final/netty-resolver-4.1.33.Final.jar:/home/robscientist/.m2/repository/io/netty/netty-codec/4.1.33.Final/netty-codec-4.1.33.Final.jar:/home/robscientist/.m2/repository/io/netty/netty-codec-http2/4.1.33.Final/netty-codec-http2-4.1.33.Final.jar:/home/robscientist/.m2/repository/io/netty/netty-handler/4.1.33.Final/netty-handler-4.1.33.Final.jar:/home/robscientist/.m2/repository/io/netty/netty-handler-proxy/4.1.33.Final/netty-handler-proxy-4.1.33.Final.jar:/home/robscientist/.m2/repository/io/netty/netty-codec-socks/4.1.33.Final/netty-codec-socks-4.1.33.Final.jar:/home/robscientist/.m2/repository/io/netty/netty-transport-native-epoll/4.1.33.Final/netty-transport-native-epoll-4.1.33.Final-linux-x86_64.jar:/home/robscientist/.m2/repository/io/netty/netty-transport-native-unix-common/4.1.33.Final/netty-transport-native-unix-common-4.1.33.Final.jar:/home/robscientist/.m2/repository/io/projectreactor/reactor-core/3.2.6.RELEASE/reactor-core-3.2.6.RELEASE.jar:/home/robscientist/.m2/repository/org/reactivestreams/reactive-streams/1.0.2/reactive-streams-1.0.2.jar:/home/robscientist/.m2/repository/org/postgresql/postgresql/42.2.5/postgresql-42.2.5.jar:/home/robscientist/.m2/repository/org/projectlombok/lombok/1.18.6/lombok-1.18.6.jar:/home/robscientist/.m2/repository/org/springframework/boot/spring-boot-devtools/2.0.0.RELEASE/spring-boot-devtools-2.0.0.RELEASE.jar:/home/robscientist/.m2/repository/org/springframework/boot/spring-boot/2.1.3.RELEASE/spring-boot-2.1.3.RELEASE.jar:/home/robscientist/.m2/repository/org/springframework/boot/spring-boot-autoconfigure/2.1.3.RELEASE/spring-boot-autoconfigure-2.1.3.RELEASE.jar
06:50:07.116 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.library.path=/usr/java/packages/lib:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib
06:50:07.116 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.io.tmpdir=/tmp
06:50:07.116 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.compiler=<NA>
06:50:07.116 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.name=Linux
06:50:07.116 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.arch=amd64
06:50:07.117 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.version=4.15.0-50-generic
06:50:07.117 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.name=robscientist
06:50:07.117 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.home=/home/robscientist
06:50:07.117 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.dir=/home/robscientist/STS-WORKSPACE/poc_chat
06:50:07.118 [main] INFO org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=127.0.0.1:2181 sessionTimeout=15000 watcher=kafka.zookeeper.ZooKeeperClient$ZooKeeperClientWatcher$#5bda8e08
06:50:07.120 [main] DEBUG org.apache.zookeeper.ClientCnxn - zookeeper.disableAutoWatchReset is false
06:50:07.129 [main] DEBUG kafka.utils.KafkaScheduler - Initializing task scheduler.
06:50:07.131 [main] INFO kafka.zookeeper.ZooKeeperClient - [ZooKeeperClient] Waiting until connected.
06:50:07.138 [main-SendThread(localhost:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
06:50:07.144 [main-SendThread(localhost:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket connection established to localhost/127.0.0.1:2181, initiating session
06:50:07.145 [main-SendThread(localhost:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Session establishment request sent on localhost/127.0.0.1:2181
06:50:07.403 [main-SendThread(localhost:2181)] INFO org.apache.zookeeper.ClientCnxn - Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x1000016739e0016, negotiated timeout = 15000
06:50:07.409 [main-EventThread] DEBUG kafka.zookeeper.ZooKeeperClient - [ZooKeeperClient] Received event: WatchedEvent state:SyncConnected type:None path:null
06:50:07.415 [main] INFO kafka.zookeeper.ZooKeeperClient - [ZooKeeperClient] Connected.
06:50:07.469 [main-SendThread(localhost:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x1000016739e0016, packet:: clientPath:/brokers/ids serverPath:/brokers/ids finished:false header:: 1,12 replyHeader:: 1,237,0 request:: '/brokers/ids,F response:: v{'0},s{5,5,1559484903664,1559484903664,0,5,0,0,0,1,191}
06:50:07.492 [main-SendThread(localhost:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x1000016739e0016, packet:: clientPath:/brokers/ids/0 serverPath:/brokers/ids/0 finished:false header:: 2,4 replyHeader:: 2,237,0 request:: '/brokers/ids/0,F response:: #7b226c697374656e65725f73656375726974795f70726f746f636f6c5f6d6170223a7b22504c41494e54455854223a22504c41494e54455854227d2c22656e64706f696e7473223a5b22504c41494e544558543a2f2f6b756e74613a39303932225d2c226a6d785f706f7274223a2d312c22686f7374223a226b756e7461222c2274696d657374616d70223a2231353539353332323533363131222c22706f7274223a393039322c2276657273696f6e223a347d,s{191,191,1559532253652,1559532253652,1,0,0,72057690466942976,180,0,191}
06:50:07.690 [main-SendThread(localhost:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x1000016739e0016, packet:: clientPath:/brokers/topics/superTopic serverPath:/brokers/topics/superTopic finished:false header:: 3,3 replyHeader:: 3,237,-101 request:: '/brokers/topics/superTopic,F response::
Exception in thread "main" java.lang.NoSuchFieldError: DEFAULT_SSL_PRINCIPAL_MAPPING_RULES
at kafka.server.Defaults$.<init>(KafkaConfig.scala:224)
at kafka.server.Defaults$.<clinit>(KafkaConfig.scala)
at kafka.log.Defaults$.<init>(LogConfig.scala:36)
at kafka.log.Defaults$.<clinit>(LogConfig.scala)
at kafka.log.LogConfig$.<init>(LogConfig.scala:219)
at kafka.log.LogConfig$.<clinit>(LogConfig.scala)
at kafka.zk.AdminZkClient.validateTopicCreate(AdminZkClient.scala:128)
at kafka.zk.AdminZkClient.createTopicWithAssignment(AdminZkClient.scala:86)
at kafka.zk.AdminZkClient.createTopic(AdminZkClient.scala:56)
at com.predisurge.kafka.KafkaTopicCreator.createTopic(KafkaTopicCreator.java:41)
at com.predisurge.ChatApplication.main(ChatApplication.java:32)
After running the command to display my topics, I don't see the one I want to add.
IDE : STS 4,
Kafka version : 2.2.0,
ZK version : 3.5.5
not sure why you get this exception. Here's a solution that works for me, and just uses the KafkaAdminClient:
#SpringBootApplication
#EnableJpaAuditing
public class TopicCreator {
public static void main(String[] args) throws InterruptedException, ExecutionException {
Properties properties = new Properties();
properties.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
AdminClient kafkaAdminClient = KafkaAdminClient.create(properties);
CreateTopicsResult result = kafkaAdminClient.createTopics(
Stream.of("foo", "bar", "baz").map(
name -> new NewTopic(name, 3, (short) 1)
).collect(Collectors.toList())
);
result.all().get();
}
}
As the above code shows, for newer versions of Kafka you can create topics directly through Kafka. Does this help?
I have a local two instance Solr Cloud setup with a single zookeeper instance. I am trying to connect via SolrJ to execute a query however my code hangs for 2mins or so when executing the query and then fails. I have followed the basic example on the Solr wiki. The logs/code is below
2016-07-24 13:29:01.932 INFO 83666 --- [qtp699221219-28] org.apache.zookeeper.ZooKeeper : Initiating client connection, connectString=localhost:2181 sessionTimeout=10000 watcher=org.apache.solr.common.cloud.SolrZkClient$3#496eab9
2016-07-24 13:29:01.948 INFO 83666 --- [qtp699221219-28] o.a.solr.common.cloud.ConnectionManager : Waiting for client to connect to ZooKeeper
2016-07-24 13:29:01.953 INFO 83666 --- [localhost:2181)] org.apache.zookeeper.ClientCnxn : Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
2016-07-24 13:29:01.955 INFO 83666 --- [localhost:2181)] org.apache.zookeeper.ClientCnxn : Socket connection established to localhost/127.0.0.1:2181, initiating session
2016-07-24 13:29:01.967 INFO 83666 --- [localhost:2181)] org.apache.zookeeper.ClientCnxn : Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x1561cdd875e0004, negotiated timeout = 10000
2016-07-24 13:29:01.972 INFO 83666 --- [back-3-thread-1] o.a.solr.common.cloud.ConnectionManager : Watcher org.apache.solr.common.cloud.ConnectionManager#4bb95d56 name:ZooKeeperConnection Watcher:localhost:2181 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
2016-07-24 13:29:01.972 INFO 83666 --- [qtp699221219-28] o.a.solr.common.cloud.ConnectionManager : Client is connected to ZooKeeper
2016-07-24 13:29:01.973 INFO 83666 --- [qtp699221219-28] o.apache.solr.common.cloud.SolrZkClient : Using default ZkACLProvider
2016-07-24 13:29:01.974 INFO 83666 --- [qtp699221219-28] o.a.solr.common.cloud.ZkStateReader : Updating cluster state from ZooKeeper...
2016-07-24 13:29:01.990 INFO 83666 --- [qtp699221219-28] o.a.solr.common.cloud.ZkStateReader : Loaded empty cluster properties
2016-07-24 13:29:01.995 INFO 83666 --- [qtp699221219-28] o.a.solr.common.cloud.ZkStateReader : Updated live nodes from ZooKeeper... (0) -> (2)
2016-07-24 13:31:24.653 ERROR 83666 --- [qtp699221219-28] o.a.s.client.solrj.impl.CloudSolrClient : Request to collection foo failed due to (0) java.net.ConnectException: Operation timed out, retry? 0
and my code is:
String zkHostString = "localhost:2181";
CloudSolrClient solr = new CloudSolrClient.Builder().withZkHost(zkHostString).build();
solr.setDefaultCollection("foo");
SolrQuery query = new SolrQuery();
query.set("q", "*:*");
QueryResponse response = null;
try {
response = solr.query(query);
} catch (SolrServerException e) {
return null;
}
//Do Something with the results...
Urgh, I'm an idiot, the zookeeper instance and solr instances are inside docker, the code posted above is not. So Zookeeper reported back the solr urls using the docker containers ip...The host needs to connect via localhost and not the docker container ip.
Eg: Zookeeper responds [http://172.17.0.5:8983/solr/foo_shard1_replica2, http://172.17.0.6:8984/solr/foo_shard1_replica1]
but my code needs to call [http://localhost:8983/solr/foo_shard1_replica2, http://localhost:8984/solr/foo_shard1_replica1]
I have this example of MapReduce [1], and I want to print info in the stdout and in a log file[3]. It seems that the logs isn’t print something. How can I make my map class print output?
I also have configured yarn-site.xml to retain log[2]. Although the logs are retained in the /app-logs dir, the userlogs dir that contains the output of the job execution is deleted at the end of the job execution. How can I make MapReduce to not delete files in the userlogs dir?
I am using Yarn.
Thanks,
[1] Wordcount exampla with just the map part.
public class MyWordCount {
public static class MyMap extends Mapper {
Log log = LogFactory.getLog(MyWordCount.class);
private final static IntWritable one = new IntWritable(1);
private Text word = new Text();
public void map(LongWritable key, Text value, OutputCollector<Text, IntWritable> output, Reporter reporter) throws IOException {
StringTokenizer itr = new StringTokenizer(value.toString());
System.out.println("HERRE");
log.info("HERRRRRE");
while (itr.hasMoreTokens()) {
word.set(itr.nextToken());
output.collect(word, one);
}
}
public void run(Context context) throws IOException, InterruptedException {
setup(context);
try {
while (context.nextKeyValue()) {
System.out.println("Key: " + context.getCurrentKey() + " Value: " + context.getCurrentValue());
map(context.getCurrentKey(), context.getCurrentValue(), context);
}
} finally {
cleanup(context);
}
}
public void cleanup(Mapper.Context context) {}
}
[2] yarn-site.xml
<!-- job history -->
<property> <name>yarn.log-aggregation-enable</name> <value>true</value> </property>
<property> <name>yarn.nodemanager.log.retain-seconds</name> <value>900000</value> </property>
<property> <name>yarn.nodemanager.remote-app-log-dir</name> <value>/app-logs</value> </property>
[3] log output
Log Type: stderr
Log Upload Time: 24-Sep-2015 12:45:19
Log Length: 317
Java HotSpot(TM) Client VM warning: You have loaded library /home/xubuntu/Programs/hadoop-2.6.0/lib/native/libhadoop.so which might have disabled stack guard. The VM will try to fix the stack guard now.
It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.
Log Type: stdout
Log Upload Time: 24-Sep-2015 12:45:19
Log Length: 0
Log Type: syslog
Log Upload Time: 24-Sep-2015 12:45:19
Log Length: 2604
2015-09-24 12:45:04,569 WARN [main] org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2015-09-24 12:45:05,139 INFO [main] org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
2015-09-24 12:45:05,412 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
2015-09-24 12:45:05,413 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics system started
2015-09-24 12:45:05,462 INFO [main] org.apache.hadoop.mapred.YarnChild: Executing with tokens:
2015-09-24 12:45:05,463 INFO [main] org.apache.hadoop.mapred.YarnChild: Kind: mapreduce.job, Service: job_1443113036547_0001, Ident: (org.apache.hadoop.mapreduce.security.token.JobTokenIdentifier#1b5a082)
2015-09-24 12:45:05,847 INFO [main] org.apache.hadoop.mapred.YarnChild: Sleeping for 0ms before retrying again. Got null now.
2015-09-24 12:45:06,915 INFO [main] org.apache.hadoop.mapred.YarnChild: mapreduce.cluster.local.dir for child: /tmp/hadoop-temp/nm-local-dir/usercache/xubuntu/appcache/application_1443113036547_0001
2015-09-24 12:45:07,604 INFO [main] org.apache.hadoop.conf.Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id
2015-09-24 12:45:09,402 INFO [main] org.apache.hadoop.mapred.Task: Using ResourceCalculatorProcessTree : [ ]
2015-09-24 12:45:10,187 INFO [main] org.apache.hadoop.mapred.MapTask: Processing split: hdfs://hadoop-coc-1:9000/input1/b.txt:0+21
2015-09-24 12:45:10,812 INFO [main] org.apache.hadoop.mapred.Task: Task:attempt_1443113036547_0001_m_000000_0 is done. And is in the process of committing
2015-09-24 12:45:10,969 INFO [main] org.apache.hadoop.mapred.Task: Task attempt_1443113036547_0001_m_000000_0 is allowed to commit now
2015-09-24 12:45:10,993 INFO [main] org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: Saved output of task 'attempt_1443113036547_0001_m_000000_0' to hdfs://192.168.10.110:9000/output1-1442847968/_temporary/1/task_1443113036547_0001_m_000000
2015-09-24 12:45:11,135 INFO [main] org.apache.hadoop.mapred.Task: Task 'attempt_1443113036547_0001_m_000000_0' done.
2015-09-24 12:45:11,135 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping MapTask metrics system...
2015-09-24 12:45:11,136 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics system stopped.
2015-09-24 12:45:11,136 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics system shutdown complete.
I have found the error. It is a bug in my code.