I try to run FSCrawler via docker-compose following the steps described in https://fscrawler.readthedocs.io/en/fscrawler-2.9/installation.html#using-docker-compose.
ELASTIC_VERSION = "7.17.8"
FSCRAWLER_VERSION = "2.9"
PWD = ""
I verified that elasticsearch is running sucessfully.
On running docker-compose up fscrawler, I get the following exception.
You need to use the 2.10-SNAPSHOT which is much more stable although the name does not say that ;)
Related
Environment variables with Spring Boot & Kubernetes
In my OS, the following command retrieves a key managed by K8s:
kubectl exec -it nginx-secret-7665cd8fb8-mj26h cat /mnt/secrets/ScAppCryptKey
As mentioned, the result is something like this:
xPeShVmYq3t6w9z#B#E?H#McQfTjWnZr4u7x%D*F-JaNdRgUkXp2s5v8y#B?E%
Now, I have to find a way to get this result (key) and use it in my Spring Boot Application. I think the most proper way to express my problem is:
During runtime (or when starting my app), I need to execute a command in the OS and get its result.
Is there a way to automatically execute commands in the terminal with Spring? Let me try to explain through code:
// If I had an environment variable, this would be like:
String key = System.getenv("myKey");
// But in my case, I need to execute a command, like this:
String keyFromKubernetes = System.getenv(
"kubectl exec -it nginx-secret-7665cd8fb8-mj26h cat /mnt/secrets/ScAppCryptKey"
);
I'm using CentOS and have downloaded Elasticsearch 6.2.1. I created a new user "elastic" and when I run ./bin/elasticsearch I get the error:
Could not find or load main class org.elasticsearch.tools.launchers.JavaVersionChecker
I tried placing this user in an admin group ("wheel"), and the same problem occurs. If I try it with "sudo ./bin/elasticsearch" I get:
[2018-02-15T17:42:39,776][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.RuntimeException: can not run elasticsearch as root
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:125) ~[elasticsearch-6.2.1.jar:6.2.1]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:112) ~[elasticsearch-6.2.1.jar:6.2.1]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.2.1.jar:6.2.1]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.2.1.jar:6.2.1]
at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.2.1.jar:6.2.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-6.2.1.jar:6.2.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:85) ~[elasticsearch-6.2.1.jar:6.2.1]
Caused by: java.lang.RuntimeException: can not run elasticsearch as root
at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:105) ~[elasticsearch-6.2.1.jar:6.2.1]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:172) ~[elasticsearch-6.2.1.jar:6.2.1]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:323) ~[elasticsearch-6.2.1.jar:6.2.1]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:121) ~[elasticsearch-6.2.1.jar:6.2.1]
I searched a bit and saw that it this error can be due to the java version, but it seems to be up to date:
[elastic#sandbox-hdp elasticsearch-6.1.1]$ sudo update-alternatives --config java
There are 3 programs which provide 'java'.
Selection Command
1 /usr/lib/jvm/jre-1.7.0-openjdk.x86_64/bin/java
*+ 2 /usr/lib/jvm/jre-1.8.0-openjdk.x86_64/bin/java
3 /usr/lib/jvm/jre-1.5.0-gcj/bin/java
This also happens if I try Elasticsearch 6.1
Any suggestions would be appreciated.
In case you still have this problem... or just for curiosity... take a look at the file ./bin/elasticsearch-env.sh, around the line 70, look for the command:
"$JAVA" -cp "$ES_CLASSPATH" org.elasticsearch.tools.launchers.JavaVersionChecker
Check if it is correctly "spelled" for your environment, write this command just before it and see what you've got:
echo "$JAVA" -cp "$ES_CLASSPATH" org.elasticsearch.tools.launchers.JavaVersionChecker
in my case... I was running in a Git Bash inside a Windows machine, and the command was pointing to:
/c/Users/Ualter/Developer/ELKStack/elasticsearch-6.2.2/lib/*
Changing to:
/Users/Ualter/Developer/ELKStack/elasticsearch-6.2.2/lib/*
As I was running on a Windows S.O. it get up and running OK the Elasticsearch.
I was trying hard to use elasticsearch 6.7.1 on Windows.
I had this exact error when running bin/elasticsearch.bat .
Long story short, there was probably an issue with my java version and elastic search, it couldn't run the jars (using Java 8 211).
I installed the last ElasticSearch 6.x from here :
https://www.elastic.co/downloads/past-releases#elasticsearch
And unset the existing ES_HOME and ES_CLASSPATH environment variables that I created, and all went well.
I also encounter such strange question, the first time it raises exception for running elastic as root, after I add a new user elastic and run it as elastic , I see this exception "Could not find or load main class org.elasticsearch.tools.launchers.JavaVersionChecker
".
I solved this problem by moving the program dirs to the non /root/ dirs, for example, I put it to /opt/elk/, and it works.
So I guess elastic may not allow to run as root for security , thus we can not put the program in the root user's dir. Hope it can make some tips for you.
I'm installing elasticsearch plugin es-sql on windows.
In my case, run the cmd as administrator role.
Seems like you don't have right permissions. Try to set them to 774 (rwxrwxr--) and check again:
sudo chmod 774 -R elasticsearch-6.3.2/
I've been trying for the last week or so to make integration tests work on travis for a school project. I've debugged a fair bit of the project but now I'm blocked and need external help.
To give a bit of context, so far, I've debugged the java project so that the tests can be launched from eclipse or from maven in command line. I've worked on the travis file so that a database is created, the database scripts run and the java tests launch. However, the tests fail on travis because of a "table missing" in the database.
This is a link to our repo.
This is the travis.yml file's code:
language : java
jdk:
- oraclejdk8
service:
- mysql
before_script:
- mysql -e 'DROP DATABASE IF EXISTS koalatest'
- mysql -e 'CREATE DATABASE IF NOT EXISTS koalatest;'
- mysql -u root --default-character-set=utf8 koalatest < backend/koalacal-backend/koalacal.sql
script: cd backend && cd koalacal-backend && mvn test -X
after_success:
- bash <(curl -s https://codecov.io/bash)
The java project that is being built and run by maven is located under rootfolder -> backend -> koalacal-backend.
Here is a link to the error log maven produces on travis.
This line seems to be the source of the error:
com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Table 'koalatest.Calendar' doesn't exist
I have two hypothesis:
1- The sql script that creates all the tables is not being run properly by travis.
To test this hypothesis, I changed the name of the script called by travis. As expected, I got an error saying that travis can't find the file. So at least, I know that this line of code causes travis to look up at an sql file.
- mysql -u root --default-character-set=utf8 koalatest < backend/koalacal-backend/koalacal.sql
That being said, I have no idea if the file is run properly on the database.
For the sake of putting all relevant informations in this post, here is a link to the database script.
2- The tests can't connect properly to the database.
Here is the config file that contain the info regarding which database to connect to:
TestInstance=true
user=root
password=
serverName=localhost
databaseName=koalacal
portNumber=3306
testUser=root
testPassword=
testServerName=127.0.0.1
testDatabaseName=koalatest
testPortNumber=3306</code>
If the parameter TestInstance is set to true, the tests take the informations testUser, testPassword, testServerName, testDatabaseName and testPortNumber to connect to the relevant database.
I believe the connection informations currently contained in the config file match how the travis documentation says we need to connect to a mysql database. I tried to change the testUser to something invalid (like root3) and got error messages as expected.
Maybe somehow the tests can't connect to the database and don't produce a related error message, but I doubt it.
Can someone look at my problem and see if I've missed something obvious (or not)? I don't know what else to try and I don't want to be blocked one more week on a technical issue.
For anyone who may google travis mysql and has a similar error to the one I had, I solved my problem.
The error was caused by a case sensitivity issue. The java code tried to connect to tables like 'Calendar' and 'Event' while the sql script created the tables 'calendar' and 'event'.
It took a long time to troubleshoot this because the case sensitivity didn't pose any problem on my machine. Maven can run its tests properly without any issue. It's only on the travis servers that case sensitivity of the tables started to matter.
I'm attempting to start up a MiniAccumuloCluster for testing as described in the Accumulo Docs:
TemporaryFolder folder= new TemporaryFolder();
File temp_dir = folder.newFolder("AccumuloTempFolder");
MiniAccumuloCluster accumulo = new MiniAccumuloCluster(tempDirectory, "password");
accumulo.start();
Instance instance = new ZooKeeperInstance(accumulo.getInstanceName(), accumulo.getZooKeepers());
Connector conn = instance.getConnector("root", new PasswordToken("password"));
When calling accumulo.start(), a ZooKeeperBindException is thrown because "Zookeeper did not start within 20 seconds." Documentation and usage notes for the MiniAccumuloCluster seem sparse - can anyone help me understand what's going wrong here? I assumed that all of the Zookeeper configuration was being handled under the covers of the MiniAccumuloCluster so I'm not even sure where to start looking for a solution.
I ended up running a docker container for the mini fluo and mini accumulo, must have been something in my local environment causing the issue. This will print out the results or put a jar in the target directory of your local machine.
docker run -it --rm --name my-maven-project -v "$(pwd)":/path/to/code/fluo-
tour/src/main/java/ft -w /path/to/code/fluo-tour/src/main/java/ft maven:3.6.3-
openjdk-8 mvn -q clean compile exec:java
I have this line in my code:
DistributedFileSystem.get(conf).delete(new Path(new URI(otherArgs[1])), true);
otherArgs[1] has this value: hdfs://master:54310/input/results
I receive this exception:
Exception in thread "main" java.lang.IllegalArgumentException: Wrong FS:hdfs://master:54310/input/results, expected: file:///
at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:354)
at org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:55)
at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:367)
at org.apache.hadoop.fs.ChecksumFileSystem.delete(ChecksumFileSystem.java:430)
at <package>.<classname>.main(Degree.java:137)
Note: I tried to use new Path(otherArgs[1]) without URI but got the exact same error !
Thanks,
-K
It looks like you have not set fs.default.name in core-site.xml.
check this link
If you have already set that, make sure the config files are in classpath
You can also set the fs.default.name property from your driver
conf.set("fs.default.name", "hdfs://yourserver:port");
It turns out that I was running my jar using "hadoop -jar " instead of "hadoop jar ". All conf files are correct and in place.
Problem solved but i still have no idea why using "-jar" made it run as a local (pseudo distributed) !