Slow startup of Selenium-Server-Standalone on OSX - java

Selenium-server-standalone has a 5 seconds delay while starting. If I run selenium with debug option it shows that the delay is caused by New random session seed.
I start Selenium server with java -jar selenium-server-standalone-2.44.0.jar.
If I want to specify chromedriver then I write java -jar selenium-server-standalone-2.44.0.jar -Dwebdriver.chrome.driver=chromedriver_2.11
It happens with:
OSX 10.9.5 and OSX 10.10
Java 1.7.0_51 (update 67 and update 71)
selenium-server-standalone 2.42.0, 2.43.1, 2.44.0
with and without chromedriver (2.9, 2.10, 2.11)
running as sudo and as non-priveleged user
From what I know on other machines with the same configuration there is no such delay.
It may seem that 5 seconds is nothing, but I have another issue that most probably is caused by the same problem. It gives me 1 minute of waiting instead of few seconds.
Here is the log
11:22:50.445 INFO - Launching a standalone server
11:22:50.820 INFO - Java: Oracle Corporation 24.51-b03
11:22:50.820 INFO - OS: Mac OS X 10.10 x86_64
11:22:50.948 INFO - v2.44.0, with Core v2.44.0. Built from revision 76d78cf
11:22:50.948 INFO - Selenium server running in debug mode.
11:22:51.001 DEBUG - add component: SocketListener0#0.0.0.0:4444
11:22:51.022 DEBUG - add component: org.openqa.jetty.http.ResourceCache#50265e47
11:22:51.046 DEBUG - add component: org.openqa.selenium.server.ProxyHandler in HttpContext[/,/]
11:22:51.065 DEBUG - add component: HttpContext[/,/]
11:22:51.070 DEBUG - Added HttpContext[/,/] for host *
11:22:51.071 DEBUG - add component: org.openqa.jetty.http.ResourceCache#36264c17
11:22:51.073 DEBUG - added SC{BASIC,null,user,CONFIDENTIAL} at /org/openqa/selenium/tests/html/basicAuth/*
11:22:51.107 DEBUG - add component: org.openqa.jetty.http.handler.SecurityHandler in HttpContext[/selenium-server,/selenium-server]
11:22:51.111 DEBUG - add component: org.openqa.selenium.server.StaticContentHandler in HttpContext[/selenium-server,/selenium-server]
11:22:51.123 DEBUG - add component: org.openqa.selenium.server.SessionExtensionJsHandler in HttpContext[/selenium-server,/selenium-server]
11:22:51.125 DEBUG - add component: org.openqa.selenium.server.htmlrunner.SingleTestSuiteResourceHandler in HttpContext[/selenium-server,/selenium-server]
11:22:51.126 DEBUG - add component: org.openqa.selenium.server.htmlrunner.SeleniumHTMLRunnerResultsHandler#6d9d3901
11:22:51.132 DEBUG - add component: HttpContext[/selenium-server,/selenium-server]
11:22:51.132 DEBUG - Added HttpContext[/selenium-server,/selenium-server] for host *
11:22:51.242 INFO - Default driver org.openqa.selenium.ie.InternetExplorerDriver registration is skipped: registration capabilities Capabilities [{platform=WINDOWS, ensureCleanSession=true, browserName=internet explorer, version=}] does not match with current platform: MAC
11:22:51.446 DEBUG - add component: org.openqa.jetty.http.ResourceCache#61f3318a
11:22:51.483 DEBUG - add component: org.openqa.selenium.server.SeleniumDriverResourceHandler in HttpContext[/selenium-server,/selenium-server]
11:22:51.483 DEBUG - add component: HttpContext[/selenium-server/driver,/selenium-server/driver]
11:22:51.484 DEBUG - Added HttpContext[/selenium-server/driver,/selenium-server/driver] for host *
11:22:51.484 DEBUG - add component: org.openqa.jetty.http.ResourceCache#c3b626c
11:22:51.528 DEBUG - add component: WebDriver remote server
11:22:51.568 DEBUG - add component: org.openqa.jetty.jetty.servlet.HashSessionManager#5298d146
11:22:51.568 DEBUG - add component: org.openqa.jetty.jetty.servlet.ServletHandler#2ed37507
11:22:51.596 INFO - RemoteWebDriver instances should connect to: http://127.0.0.1:4444/wd/hub
11:22:51.596 DEBUG - add component: HttpContext[/wd,/wd]
11:22:51.597 DEBUG - Added HttpContext[/wd,/wd] for host *
11:22:51.597 DEBUG - Starting org.openqa.jetty.jetty.Server#a5bdce3
11:22:51.598 INFO - Version Jetty/5.1.x
11:22:51.598 DEBUG - LISTENERS: [SocketListener0#0.0.0.0:4444]
11:22:51.598 DEBUG - HANDLER: {null={/selenium-server/driver/*=[HttpContext[/selenium-server/driver,/selenium-server/driver]], /selenium-server/*=[HttpContext[/selenium-server,/selenium-server]], /=[HttpContext[/,/]], /wd/*=[HttpContext[/wd,/wd]]}}
11:22:51.599 DEBUG - Starting HttpContext[/selenium-server/driver,/selenium-server/driver]
11:22:51.612 DEBUG - Init classloader from null, sun.misc.Launcher$AppClassLoader#5eb1404f for HttpContext[/selenium-server/driver,/selenium-server/driver]
11:22:51.612 INFO - Started HttpContext[/selenium-server/driver,/selenium-server/driver]
11:22:51.612 DEBUG - Starting HttpContext[/selenium-server,/selenium-server]
11:22:51.612 DEBUG - Init classloader from null, sun.misc.Launcher$AppClassLoader#5eb1404f for HttpContext[/selenium-server,/selenium-server]
11:22:51.613 DEBUG - Started org.openqa.jetty.http.handler.SecurityHandler in HttpContext[/selenium-server,/selenium-server]
11:22:51.613 DEBUG - Started org.openqa.selenium.server.StaticContentHandler in HttpContext[/selenium-server,/selenium-server]
11:22:51.613 DEBUG - Started org.openqa.selenium.server.SessionExtensionJsHandler in HttpContext[/selenium-server,/selenium-server]
11:22:51.613 DEBUG - Started org.openqa.selenium.server.htmlrunner.SingleTestSuiteResourceHandler in HttpContext[/selenium-server,/selenium-server]
11:22:51.613 DEBUG - Started org.openqa.selenium.server.SeleniumDriverResourceHandler in HttpContext[/selenium-server,/selenium-server]
11:22:51.613 INFO - Started HttpContext[/selenium-server,/selenium-server]
11:22:51.614 DEBUG - Starting HttpContext[/,/]
11:22:51.614 DEBUG - Init classloader from null, sun.misc.Launcher$AppClassLoader#5eb1404f for HttpContext[/,/]
11:22:51.614 DEBUG - Started org.openqa.selenium.server.ProxyHandler in HttpContext[/,/]
11:22:51.614 INFO - Started HttpContext[/,/]
11:22:51.614 DEBUG - Starting HttpContext[/wd,/wd]
11:22:51.614 DEBUG - Init classloader from null, sun.misc.Launcher$AppClassLoader#5eb1404f for HttpContext[/wd,/wd]
11:22:51.614 DEBUG - Starting org.openqa.jetty.jetty.servlet.ServletHandler#2ed37507
11:22:51.614 DEBUG - New random session seed
11:22:56.882 DEBUG - Session scavenger period = 30s
11:22:56.890 DEBUG - Started holder of class org.openqa.selenium.remote.server.DriverServlet
11:22:56.891 INFO - Started org.openqa.jetty.jetty.servlet.ServletHandler#2ed37507
11:22:56.891 INFO - Started HttpContext[/wd,/wd]
11:22:56.928 INFO - Started SocketListener on 0.0.0.0:4444
11:22:56.928 INFO - Started org.openqa.jetty.jetty.Server#a5bdce3

Related

My ChromeBrowser doesn't open, despite my step definitions, feature files and my code looking fine (JavaSeleniumCucumber) testing)

Error says
Testing started at 12:03 ...
12:03:10.624 [main] DEBUG io.github.bonigarcia.wdm.cache.ResolutionCache - Resolution chrome=103 in cache (valid until 12:20:19 30/07/2022 BST)
12:03:10.634 [main] DEBUG io.github.bonigarcia.wdm.cache.ResolutionCache - Resolution chrome103=103.0.5060.134 in cache (valid until 17:22:03 30/07/2022 BST)
12:03:10.635 [main] INFO io.github.bonigarcia.wdm.WebDriverManager - Using chromedriver 103.0.5060.134 (resolved driver for Chrome 103)
12:03:10.678 [main] DEBUG io.github.bonigarcia.wdm.WebDriverManager - Driver chromedriver 103.0.5060.134 found in cache
12:03:10.681 [main] INFO io.github.bonigarcia.wdm.WebDriverManager - Exporting webdriver.chrome.driver as C:\Users\lukem.cache\selenium\chromedriver\win32\103.0.5060.134\chromedriver.exe
0 Scenarios
0 Steps
0m1.300s

Java Hadoop client cannot list files in HDFS

I'm trying to operate HDFS via Java Hadoop client. But when I call FileSystem::listFiles, the returned iterator give me no entry.
Here is my java code:
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.LocatedFileStatus;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.fs.RemoteIterator;
import java.net.URI;
import java.net.URISyntaxException;
import java.io.IOException;
class HadoopTest {
public static void main(String[] args) throws IOException, URISyntaxException {
String url = "hdfs://10.2.206.148";
FileSystem fs = FileSystem.get(new URI(url), new Configuration());
System.out.println("get fs success!");
RemoteIterator<LocatedFileStatus> iterator = fs.listFiles(new Path("/"), false);
while (iterator.hasNext()) {
LocatedFileStatus lfs = iterator.next();
System.out.println(lfs.getPath().toString());
}
System.out.println("iteration finished");
}
}
And here is the outputs:
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/admin/pengduo/hadoop_test/lib/logback-classic-1.2.3.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/admin/pengduo/hadoop_test/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [ch.qos.logback.classic.util.ContextSelectorStaticBinder]
10:49:26.019 [main] DEBUG org.apache.hadoop.util.Shell - setsid exited with exit code 0
10:49:26.064 [main] DEBUG org.apache.hadoop.metrics2.lib.MutableMetricsFactory - field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginSuccess with annotation #org.apache.hadoop.metrics2.annotation.Metric(always=false, sampleName=Ops, valueName=Time, about=, interval=10, type=DEFAULT, value=[Rate of successful kerberos logins and latency (milliseconds)])
10:49:26.069 [main] DEBUG org.apache.hadoop.metrics2.lib.MutableMetricsFactory - field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginFailure with annotation #org.apache.hadoop.metrics2.annotation.Metric(always=false, sampleName=Ops, valueName=Time, about=, interval=10, type=DEFAULT, value=[Rate of failed kerberos logins and latency (milliseconds)])
10:49:26.069 [main] DEBUG org.apache.hadoop.metrics2.lib.MutableMetricsFactory - field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.getGroups with annotation #org.apache.hadoop.metrics2.annotation.Metric(always=false, sampleName=Ops, valueName=Time, about=, interval=10, type=DEFAULT, value=[GetGroups])
10:49:26.070 [main] DEBUG org.apache.hadoop.metrics2.lib.MutableMetricsFactory - field private org.apache.hadoop.metrics2.lib.MutableGaugeLong org.apache.hadoop.security.UserGroupInformation$UgiMetrics.renewalFailuresTotal with annotation #org.apache.hadoop.metrics2.annotation.Metric(always=false, sampleName=Ops, valueName=Time, about=, interval=10, type=DEFAULT, value=[Renewal failures since startup])
10:49:26.070 [main] DEBUG org.apache.hadoop.metrics2.lib.MutableMetricsFactory - field private org.apache.hadoop.metrics2.lib.MutableGaugeInt org.apache.hadoop.security.UserGroupInformation$UgiMetrics.renewalFailures with annotation #org.apache.hadoop.metrics2.annotation.Metric(always=false, sampleName=Ops, valueName=Time, about=, interval=10, type=DEFAULT, value=[Renewal failures since last successful login])
10:49:26.071 [main] DEBUG org.apache.hadoop.metrics2.impl.MetricsSystemImpl - UgiMetrics, User and group related metrics
10:49:26.084 [main] DEBUG org.apache.hadoop.security.SecurityUtil - Setting hadoop.security.token.service.use_ip to true
10:49:26.096 [main] DEBUG org.apache.hadoop.security.Groups - Creating new Groups object
10:49:26.097 [main] DEBUG org.apache.hadoop.util.NativeCodeLoader - Trying to load the custom-built native-hadoop library...
10:49:26.097 [main] DEBUG org.apache.hadoop.util.NativeCodeLoader - Failed to load native-hadoop with error: java.lang.UnsatisfiedLinkError: no hadoop in java.library.path
10:49:26.097 [main] DEBUG org.apache.hadoop.util.NativeCodeLoader - java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
10:49:26.097 [main] WARN org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
10:49:26.098 [main] DEBUG org.apache.hadoop.util.PerformanceAdvisory - Falling back to shell based
10:49:26.098 [main] DEBUG org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback - Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping
10:49:26.153 [main] DEBUG org.apache.hadoop.security.Groups - Group mapping impl=org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback; cacheTimeout=300000; warningDeltaMs=5000
10:49:26.157 [main] DEBUG org.apache.hadoop.security.UserGroupInformation - hadoop login
10:49:26.158 [main] DEBUG org.apache.hadoop.security.UserGroupInformation - hadoop login commit
10:49:26.161 [main] DEBUG org.apache.hadoop.security.UserGroupInformation - using local user:UnixPrincipal: admin
10:49:26.161 [main] DEBUG org.apache.hadoop.security.UserGroupInformation - Using user: "UnixPrincipal: admin" with name admin
10:49:26.161 [main] DEBUG org.apache.hadoop.security.UserGroupInformation - User entry: "admin"
10:49:26.161 [main] DEBUG org.apache.hadoop.security.UserGroupInformation - UGI loginUser:admin (auth:SIMPLE)
log4j:WARN No appenders could be found for logger (org.apache.htrace.core.Tracer).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
10:49:26.201 [main] DEBUG org.apache.hadoop.fs.FileSystem - Loading filesystems
10:49:26.211 [main] DEBUG org.apache.hadoop.fs.FileSystem - file:// = class org.apache.hadoop.fs.LocalFileSystem from /home/admin/pengduo/hadoop_test/lib/hadoop-common-3.2.1.jar
10:49:26.216 [main] DEBUG org.apache.hadoop.fs.FileSystem - viewfs:// = class org.apache.hadoop.fs.viewfs.ViewFileSystem from /home/admin/pengduo/hadoop_test/lib/hadoop-common-3.2.1.jar
10:49:26.218 [main] DEBUG org.apache.hadoop.fs.FileSystem - har:// = class org.apache.hadoop.fs.HarFileSystem from /home/admin/pengduo/hadoop_test/lib/hadoop-common-3.2.1.jar
10:49:26.219 [main] DEBUG org.apache.hadoop.fs.FileSystem - http:// = class org.apache.hadoop.fs.http.HttpFileSystem from /home/admin/pengduo/hadoop_test/lib/hadoop-common-3.2.1.jar
10:49:26.219 [main] DEBUG org.apache.hadoop.fs.FileSystem - https:// = class org.apache.hadoop.fs.http.HttpsFileSystem from /home/admin/pengduo/hadoop_test/lib/hadoop-common-3.2.1.jar
10:49:26.226 [main] DEBUG org.apache.hadoop.fs.FileSystem - hdfs:// = class org.apache.hadoop.hdfs.DistributedFileSystem from /home/admin/pengduo/hadoop_test/lib/hadoop-hdfs-client-3.2.1.jar
10:49:26.233 [main] DEBUG org.apache.hadoop.fs.FileSystem - webhdfs:// = class org.apache.hadoop.hdfs.web.WebHdfsFileSystem from /home/admin/pengduo/hadoop_test/lib/hadoop-hdfs-client-3.2.1.jar
10:49:26.234 [main] DEBUG org.apache.hadoop.fs.FileSystem - swebhdfs:// = class org.apache.hadoop.hdfs.web.SWebHdfsFileSystem from /home/admin/pengduo/hadoop_test/lib/hadoop-hdfs-client-3.2.1.jar
10:49:26.234 [main] DEBUG org.apache.hadoop.fs.FileSystem - Looking for FS supporting hdfs
10:49:26.234 [main] DEBUG org.apache.hadoop.fs.FileSystem - looking for configuration option fs.hdfs.impl
10:49:26.251 [main] DEBUG org.apache.hadoop.fs.FileSystem - Looking in service filesystems for implementation class
10:49:26.251 [main] DEBUG org.apache.hadoop.fs.FileSystem - FS for hdfs is class org.apache.hadoop.hdfs.DistributedFileSystem
10:49:26.282 [main] DEBUG org.apache.hadoop.hdfs.client.impl.DfsClientConf - dfs.client.use.legacy.blockreader.local = false
10:49:26.282 [main] DEBUG org.apache.hadoop.hdfs.client.impl.DfsClientConf - dfs.client.read.shortcircuit = false
10:49:26.282 [main] DEBUG org.apache.hadoop.hdfs.client.impl.DfsClientConf - dfs.client.domain.socket.data.traffic = false
10:49:26.282 [main] DEBUG org.apache.hadoop.hdfs.client.impl.DfsClientConf - dfs.domain.socket.path =
10:49:26.291 [main] DEBUG org.apache.hadoop.hdfs.DFSClient - Sets dfs.client.block.write.replace-datanode-on-failure.min-replication to 0
10:49:26.297 [main] DEBUG org.apache.hadoop.io.retry.RetryUtils - multipleLinearRandomRetry = null
10:49:26.312 [main] DEBUG org.apache.hadoop.ipc.Server - rpcKind=RPC_PROTOCOL_BUFFER, rpcRequestWrapperClass=class org.apache.hadoop.ipc.ProtobufRpcEngine$RpcProtobufRequest, rpcInvoker=org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker#7c729a55
10:49:26.322 [main] DEBUG org.apache.hadoop.ipc.Client - getting client out of cache: org.apache.hadoop.ipc.Client#222545dc
10:49:26.587 [main] DEBUG org.apache.hadoop.util.PerformanceAdvisory - Both short-circuit local reads and UNIX domain socket are disabled.
10:49:26.593 [main] DEBUG org.apache.hadoop.hdfs.protocol.datatransfer.sasl.DataTransferSaslUtil - DataTransferProtocol not using SaslPropertiesResolver, no QOP found in configuration for dfs.data.transfer.protection
get fs success!
10:49:26.629 [main] DEBUG org.apache.hadoop.ipc.Client - The ping interval is 60000 ms.
10:49:26.631 [main] DEBUG org.apache.hadoop.ipc.Client - Connecting to /10.2.206.148:8020
10:49:26.658 [IPC Client (1923598304) connection to /10.2.206.148:8020 from admin] DEBUG org.apache.hadoop.ipc.Client - IPC Client (1923598304) connection to /10.2.206.148:8020 from admin: starting, having connections 1
10:49:26.660 [IPC Parameter Sending Thread #0] DEBUG org.apache.hadoop.ipc.Client - IPC Client (1923598304) connection to /10.2.206.148:8020 from admin sending #0 org.apache.hadoop.hdfs.protocol.ClientProtocol.getListing
10:49:26.666 [IPC Client (1923598304) connection to /10.2.206.148:8020 from admin] DEBUG org.apache.hadoop.ipc.Client - IPC Client (1923598304) connection to /10.2.206.148:8020 from admin got value #0
10:49:26.666 [main] DEBUG org.apache.hadoop.ipc.ProtobufRpcEngine - Call: getListing took 52ms
iteration finished
10:49:26.695 [shutdown-hook-0] DEBUG org.apache.hadoop.ipc.Client - stopping client from cache: org.apache.hadoop.ipc.Client#222545dc
10:49:26.695 [shutdown-hook-0] DEBUG org.apache.hadoop.ipc.Client - removing client from cache: org.apache.hadoop.ipc.Client#222545dc
10:49:26.695 [shutdown-hook-0] DEBUG org.apache.hadoop.ipc.Client - stopping actual client because no more references remain: org.apache.hadoop.ipc.Client#222545dc
10:49:26.695 [shutdown-hook-0] DEBUG org.apache.hadoop.ipc.Client - Stopping client
10:49:26.696 [IPC Client (1923598304) connection to /10.2.206.148:8020 from admin] DEBUG org.apache.hadoop.ipc.Client - IPC Client (1923598304) connection to /10.2.206.148:8020 from admin: closed
10:49:26.696 [IPC Client (1923598304) connection to /10.2.206.148:8020 from admin] DEBUG org.apache.hadoop.ipc.Client - IPC Client (1923598304) connection to /10.2.206.148:8020 from admin: stopped, remaining connections 0
10:49:26.797 [Thread-4] DEBUG org.apache.hadoop.util.ShutdownHookManager - Completed shutdown in 0.102 seconds; Timeouts: 0
10:49:26.808 [Thread-4] DEBUG org.apache.hadoop.util.ShutdownHookManager - ShutdownHookManger completed shutdown.
Note that we can get the file system successfully. And the iteration is executed with no errors.
However, list the same directory using hadoop fs command looks good:
$ $HADOOP_HOME/bin/hadoop fs -ls hdfs://10.2.206.148/
Warning: fs.defaultFs is not set when running "ls" command.
Found 4 items
drwxr-x--x - hadoop hadoop 0 2020-09-21 20:29 hdfs://10.2.206.148/apps
drwxr-x--x - hadoop hadoop 0 2021-07-08 10:44 hdfs://10.2.206.148/spark-history
drwxrwxrwt - root hadoop 0 2021-07-08 10:43 hdfs://10.2.206.148/tmp
drwxr-x--t - hadoop hadoop 0 2020-11-20 11:31 hdfs://10.2.206.148/user
I have set HADOOP_HOME appropriately.
My Hadoop libs versions are 3.2.1:
$ ll hadoop-*
-rw-r--r-- 1 admin admin 60258 Jul 8 10:42 hadoop-annotations-3.2.1.jar
-rw-r--r-- 1 admin admin 139109 Jul 8 10:42 hadoop-auth-3.2.1.jar
-rw-r--r-- 1 admin admin 44163 Jul 8 10:42 hadoop-client-3.2.1.jar
-rw-r--r-- 1 admin admin 4137520 Jul 8 10:42 hadoop-common-3.2.1.jar
-rw-r--r-- 1 admin admin 5959246 Jul 8 10:42 hadoop-hdfs-3.2.1.jar
-rw-r--r-- 1 admin admin 5094412 Jul 8 10:42 hadoop-hdfs-client-3.2.1.jar
-rw-r--r-- 1 admin admin 805845 Jul 8 10:42 hadoop-mapreduce-client-common-3.2.1.jar
-rw-r--r-- 1 admin admin 1657002 Jul 8 10:42 hadoop-mapreduce-client-core-3.2.1.jar
-rw-r--r-- 1 admin admin 85900 Jul 8 10:42 hadoop-mapreduce-client-jobclient-3.2.1.jar
-rw-r--r-- 1 admin admin 3287723 Jul 8 10:42 hadoop-yarn-api-3.2.1.jar
-rw-r--r-- 1 admin admin 322882 Jul 8 10:42 hadoop-yarn-client-3.2.1.jar
-rw-r--r-- 1 admin admin 2919779 Jul 8 10:42 hadoop-yarn-common-3.2.1.jar
I'm confused why Java Hadoop client behaves differently from Hadoop CLI, and how to make my Java program performs correctly. Can anyone help me? Many thanks!

I have figured out this on myself. The problem is that I use FileSystem::listFiles. This method will only list all the files (but not the directories) under the given path. While I have only 4 directories under the given path. To list all entries including files and directories and the given path, I should use FileSystem::listLocatedStatus instead of FileSystem::listFiles.
// this will list only the files but not the directories under "/"
// RemoteIterator<LocatedFileStatus> iterator = fs.listFiles(new Path("/"), false);
// this will list all entries including the files and the directories
RemoteIterator<LocatedFileStatus> iterator = fs.listLocatedStatus(new Path("/"));

starting kairosdb gives error when used with cassandra, new installation

04:39:30.558 [main] INFO [Cluster.java:1614] - Cannot connect with protocol version V4, trying with V3
04:39:30.580 [main] INFO [Cluster.java:1614] - Cannot connect with protocol version V3, trying with V2
04:39:30.671 [main] WARN [Cluster.java:2202] - You listed localhost/0:0:0:0:0:0:0:1:9042 in your contact points, but it wasn't found in the control host's system.peers at startup
04:39:30.847 [main] INFO [KairosRetryPolicy.java:104] - Initializing KairosRetryPolicy: retry count set to 1
04:39:30.848 [main] INFO [Cluster.java:1568] - New Cassandra host localhost/127.0.0.1:9042 added
04:39:31.333 [main] ERROR [CassandraModule.java:136] - Unable to setup cassandra schema
com.datastax.driver.core.exceptions.SyntaxError: line 6:28 no viable alternative at input '>'
at com.datastax.driver.core.exceptions.SyntaxError.copy(SyntaxError.java:58)
at com.datastax.driver.core.exceptions.SyntaxError.copy(SyntaxError.java:24)
at com.datastax.driver.core.DriverThrowables.propagateCause(DriverThrowables.java:37)
at com.datastax.driver.core.DefaultResultSetFuture.getUninterruptibly(DefaultResultSetFuture.java:245)
at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:68)
at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:43)

Hikari CP JDBCConnectionException: Could not open connection

Environment
HikariCP version : 2.4.2
JDK version : 1.7.0_141
Database : MySQL
MySQL Driver version : 5.6.27
Hibernate : 4.3.4
We have successfully migrated Hikari from C3PO and everything was running smooth until recently when we encountered the following exception.
org.hibernate.exception.JDBCConnectionException: Could not open
connection Caused by: java.sql.SQLTransientConnectionException:
HikariPool-0 - Connection is not available, request timed out after
30006ms.
We deploy our webapp in AWS EC2 and database is hosted in RDS.
We have deployed the web app on 4 tomcats. The max connection pool size is configured at 250 per tomcat web app.
A total of 4000 logins happened in about 2 hours.
The use case is as follows
We have scheduled Online Tests which the student users take within a given period of time (say 9:00 am to 12:30 pm). There are simultaneous logins ( say 4000 logins )
We are able to reproduce the issue by using Locust. And we enable log4j for logging. Following is the logs
DEBUG [http-apr-8080-exec-4] - HikariPool-0 - configuration:
DEBUG [http-apr-8080-exec-4] - allowPoolSuspension.............false *
DEBUG [http-apr-8080-exec-4] - autoCommit......................true *
DEBUG [http-apr-8080-exec-4] - catalog.........................
DEBUG [http-apr-8080-exec-4] - connectionInitSql...............
DEBUG [http-apr-8080-exec-4] - connectionTestQuery.............
DEBUG [http-apr-8080-exec-4] - connectionTimeout...............30000 *
DEBUG [http-apr-8080-exec-4] - dataSource......................
DEBUG [http-apr-8080-exec-4] - dataSourceClassName.............
DEBUG [http-apr-8080-exec-4] - dataSourceJNDI..................
DEBUG [http-apr-8080-exec-4] - dataSourceProperties............{useUnicode=true, password=, prepStmtCacheSqlLimit=2048, characterEncoding=utf8, cachePrepStmts=true, useServerPrepStmts=true, prepStmtCacheSize=250}
DEBUG [http-apr-8080-exec-4] - driverClassName.................
DEBUG [http-apr-8080-exec-4] - healthCheckProperties...........{}
DEBUG [http-apr-8080-exec-4] - healthCheckRegistry.............
DEBUG [http-apr-8080-exec-4] - idleTimeout.....................600000 *
DEBUG [http-apr-8080-exec-4] - initializationFailFast..........true *
DEBUG [http-apr-8080-exec-4] - isolateInternalQueries..........false *
DEBUG [http-apr-8080-exec-4] - jdbc4ConnectionTest.............false *
DEBUG [http-apr-8080-exec-4] - jdbcUrl.........................jdbc:mysql://rds url
DEBUG [http-apr-8080-exec-4] - leakDetectionThreshold..........0 *
DEBUG [http-apr-8080-exec-4] - maxLifetime.....................1800000 *
DEBUG [http-apr-8080-exec-4] - maximumPoolSize.................250
DEBUG [http-apr-8080-exec-4] - metricRegistry..................
DEBUG [http-apr-8080-exec-4] - metricsTrackerFactory...........
DEBUG [http-apr-8080-exec-4] - minimumIdle.....................5
DEBUG [http-apr-8080-exec-4] - password........................
DEBUG [http-apr-8080-exec-4] - poolName........................HikariPool-0
DEBUG [http-apr-8080-exec-4] - readOnly........................false *
DEBUG [http-apr-8080-exec-4] - registerMbeans..................false *
DEBUG [http-apr-8080-exec-4] - scheduledExecutorService........
DEBUG [http-apr-8080-exec-4] - threadFactory...................
DEBUG [http-apr-8080-exec-4] - transactionIsolation............
DEBUG [http-apr-8080-exec-4] - username........................username
DEBUG [http-apr-8080-exec-4] - validationTimeout...............5000 *
There are few defaulted value which are marked * whereas others are set explicitly.
HikariConfig config = new HikariConfig();
config.setJdbcUrl(JDBC URL);
config.setUsername(USER_NAME);
config.setPassword(PASSWOrD);
config.setMinimumIdle(5);
config.setMaximumPoolSize(250);
config.addDataSourceProperty("cachePrepStmts", true);
config.addDataSourceProperty("prepStmtCacheSize", 250);
config.addDataSourceProperty("prepStmtCacheSqlLimit", 2048);
config.addDataSourceProperty("useServerPrepStmts", true);
datasource = new HikariDataSource(config)
Within 20 seconds or so, the connection pool is exhausted
DEBUG [http-apr-8080-exec-494] - Timeout failure pool HikariPool-0
stats
(total=100, active=100, idle=0, waiting=194)
waiting count is increasing and finally the fore mentioned exceptions are thrown.
We are not able to pinpoint what causes the issue. We tried variants of the configuration but unable to solve it yet.

Standalone spark on Windows 10

Using this guide to get a Standalone. But whenever I perform the "sbin\start-master.sh" command, a new CMD window pops up and then disappears immediately. No URL is being printed.
I've done the initial setup and set all the environment variables properly but it still doesn't seem to work. How can I fix this?
Edit: Also tried this example using Intelli-J. Its showing the following without any other output:
[Thread-0] INFO org.eclipse.jetty.util.log - Logging initialized #568ms
[Thread-0] INFO spark.webserver.JettySparkServer - == Spark has ignited ...
[Thread-0] INFO spark.webserver.JettySparkServer - >> Listening on 0.0.0.0:4567
[Thread-0] INFO org.eclipse.jetty.server.Server - jetty-9.3.2.v20150730
[Thread-0] INFO org.eclipse.jetty.server.ServerConnector - Started ServerConnector#e92ccd{HTTP/1.1,[http/1.1]}{0.0.0.0:4567}
[Thread-0] INFO org.eclipse.jetty.server.Server - Started #942ms

Categories