I have a very basic question regarding environment variables. The Tika python library that I'm trying to work with mentioned the following:
TIKA_SERVER_ENDPOINT: set to the host (local or remote) for the running Tika server jar
I downloaded the server jar from here and ran it with java -jar jarname.jar I'm trying to set the environment variable to the localhost, but I'm not sure how exactly to do this. I tried the following:
TIKA_SERVER_ENDPOINT=http://localhost:9998
But this simply resulted in a 404, with Network Error (dns_unresolved_hostname), Your host cannot be resolved by the DNS,
I also ran ps aux | grep java to check if the jar file was running and saw that it was running on the port 9998..... i.e:
java -cp /tmp/tika-server.jar org.apache.tika.server.TikaServerCLi -- port 9998 --host localhost
What exactly am I doing wrong here with my environment variable? I'm just very new to messing with jar files and environment variables, so any help would be greatly appreciated, thanks!
I had a similar issue and was able to resolve it by simply downloading the latest server jar from the Apache downloads page and then starting the server on my local machine from bash with:
java -jar tika-server -x.x.jar --port xxxx
Which started the tika server. I was then able to access tika in python through:
from tika import parser
parser.from_file('path_to_file')
Maybe java is installed incorrectly. You can check java installation here
Installation (with pip)
pip install tika
Installation (without pip)
python setup.py build
python setup.py install
Environment Variables
These are read once, when tika/tika.py is initially loaded and used throughout after that.
TIKA_VERSION - set to the version string, e.g., 1.12 or default to current Tika version.
TIKA_SERVER_JAR - set to the full URL to the remote Tika server jar to download and cache.
TIKA_SERVER_ENDPOINT - set to the host (local or remote) for the running Tika server jar.
TIKA_CLIENT_ONLY - if set to True, then TIKA_SERVER_JAR is ignored, and relies on the value for TIKA_SERVER_ENDPOINT and treats Tika like a REST client.
TIKA_TRANSLATOR - set to the fully qualified class name (defaults to Lingo24) for the Tika translator implementation.
TIKA_SERVER_CLASSPATH - set to a string (delimited by ':' for each additional path) to prepend to the Tika server jar path.
TIKA_LOG_PATH - set to a directory with write permissions and the tika.log and tika-server.log files will be placed in this directory.
More information you can get here.
Related
I'm trying to install Kafka message queue on Windows for testing purposes (not for production).
I found this article on how to install Apache Kafka 0.8 on Windows: http://janschulte.wordpress.com/2013/10/13/apache-kafka-0-8-on-windows/
It's a good article, but it's outdated, unfortunately.
Does someone know a way to achieve that?
Ok, it's finally not complicated :)
The only steps are:
Edit (2021/10) Latest versions of Kafka end up with an AccessDeniedException upon startup, and even when successfully started, throw the same error during log rotation. Kafka utilizes Java methods designed for Unix filesystem libraries, so installing Kafka in WSL2 is recommended (link to Confluent blog, but steps also applicable to Apache Kafka direct installation).
Old answer
Download Kafka and uncompress it somewhere nice (let's say C:/Kafka)
Install Cygwin
If using Cygwin, edit \bin\kafka-run-class.sh and at the end of the file, change
`exec $JAVA $KAFKA_HEAP_OPTS $KAFKA_JVM_PERFORMANCE_OPTS $KAFKA_GC_LOG_OPTS $KAFKA_JMX_OPTS $KAFKA_LOG4J_OPTS -cp $CLASSPATH $KAFKA_OPTS "$#"`
to
``exec java $KAFKA_HEAP_OPTS $KAFKA_JVM_PERFORMANCE_OPTS $KAFKA_GC_LOG_OPTS $KAFKA_JMX_OPTS $KAFKA_LOG4J_OPTS -cp `cygpath -wp $CLASSPATH` $KAFKA_OPTS "$#"``
In Environment Variables, Add java to your Path System Variable:
That's it.. you can now run ZooKeeper and Kafka servers and start playing with topics and stuff..
These are the steps I followed to run kafka on Windows
Install Zookeeper first (I downloaded v3.3.6) zookeeper-3.3.6.tar.gz
Extract Zookeeper and run this command in powershell/cmd \zookeeper-3.3.6\bin> .\zkServer.cmd Now this should up a Zookeeper instance on localhost:2181
Download Kafka binary version (I downloaded v0.10.0.1)kafka_2.10-0.10.0.1.tgz
Extract Kafka, time to modify some configs
Inside Kafka extraction you can find .\config\server.properties
In .\config\server.properties replace log.dirs=c:/kafka/kafka-logs
Note: Make sure to create those folders in relevant paths
Happy news: Now Kafka ships with windows .bat scripts, You can find these files inside ./bin/windows folder
Start powershell/cmd and run this command to start Kafka broker .\bin\windows\kafka-server-start.bat .\config\server.properties
DONE!, Now you have a running Zookeeper instance and a Kafka broker.
You will probably find that Kafka comes with Windows .bat files under the bin folder to help you run Kafka under windows. However, those bat files are not updated since version 0.8.0 and some of Kafka's logic has changed (regarding topics, etc.).
I have tried two ways to solve this problem. One is to use Cygwin or MinGW to simulate Linux environment and run the Linux shell script, but there would be tons of other problems related to path names. The other solution, which is simpler and come with less trouble, is to use the corrected version of Windows bat files.
Please refer to this blog post.
Download any binary version(I've downloaded kafka_2.11-0.10.2.0) of Kafka from https://kafka.apache.org/downloads
Extract it to any folder(I've extracted to "C:\Kafka")
Open Command Prompt
Go to the folder that you extract Kafka(C:\Kafka\kafka_2.11-0.10.2.0)
Run this command to start Zookeeper
.\bin\windows\zookeeper-server-start.bat .\config\zookeeper.properties
Run this command to start Kafka
.\bin\windows\kafka-server-start.bat .\config\server.properties
Now it works!
In version 0.9.0. there are scripts in "bin" folder for windows. You can use them.
Running From A windows Shell
As of April 2019, downloading Kafka from their website worked on Windows almost right out of the box
Downloading and using the windows version .bat files is described here:
https://kafka.apache.org/quickstart
I ran into two problems when I did this:
1) JAVA_HOME was set to an unsupported JDK which led to this kind of error
Exception in thread "main" java.lang.VerifyError: Uninitialized object
exists on backward branch 209
Replacing with JDK 11 solved the problem.
2) The JAVA_HOME must not contain spaces which caused a 'cannot find the specified file' error. To fix this I used a shortened path like set JAVA_HOME=C:\Progra~1\Java\jdk-11.0.1
Running From Cygwin
If instead of the .bat files, you want to run the .sh files from cygwin, there is actually quite a bit that needs to be done, and even after that there may be problems that come up later. I can't really recommend this option, but I do use it as it's pretty convenient for some purposes.
If your JAVA_HOME path contains a space e.g. "C:\Program Files\Java\Jdk..." you will see something like this:
bin/kafka-run-class.sh: line 305: exec: C:\Program: not found
One solution is to copy the jdk to a path without spaces, and change the Java home accordingly.
If you don't want to change the JDK location, you can change the cygwin env variable as follows:
JAVA_HOME="/cygdrive/c/Program Files/Java/jdk-11.0.1"
and change the line
exec $JAVA $KAFKA_HEAP_OPTS $KAFKA_JVM_PERFORMANCE_OPTS $KAFKA_GC_LOG_OPTS $KAFKA_JMX_OPTS $KAFKA_LOG4J_OPTS -cp $CLASSPATH $KAFKA_OPTS "$#"
to
exec "$JAVA" $KAFKA_HEAP_OPTS $KAFKA_JVM_PERFORMANCE_OPTS $KAFKA_GC_LOG_OPTS $KAFKA_JMX_OPTS $KAFKA_LOG4J_OPTS -cp $CLASSPATH $KAFKA_OPTS "$#"
There was another problem with the log parameters in kafka-run-class.sh and I had to replace the line
KAFKA_GC_LOG_OPTS="-Xlog:gc*:file=$LOG_DIR/$GC_LOG_FILE_NAME:time,tags:filecount=10,filesize=102400"
with
KAFKA_GC_LOG_OPTS="-Xlog:gc*"
And even after all these changes I occasionally run into problems of Kafka shutting down because of an incompatible windows style path in the logs directories as described here:
Kafka 1.0 stops with FATAL SHUTDOWN error. Logs directory failed
In short, you may be better off running the Kafka .bat scripts from the windows directory...
With Chocolatey and Powershell:
chocolatey.exe install kafka -> installs Kafka into C:\ProgramData\chocolatey\lib\kafka\tools\kafka_2.11-1.0.0\ - your version might differ of course
add C:\ProgramData\chocolatey\lib\kafka\tools\kafka_2.11-1.0.0\ to your powershell path using notepad $PROFILE
After restarting powershell kafka-console-consumer should work as a command.
You also need to install ssl.properties with keystore and truststore. Put them e.g. into C:\ProgramData\kafka and put that into ssl.properties (please note the escaping of the backslashes):
security.protocol=SSL
ssl.truststore.location=C:\\ProgramData\\kafka\\kafka-truststore.jks
ssl.truststore.password=PASSWORD
ssl.keystore.location=C:\\ProgramData\\kafka\\kafka-keystore.jks
ssl.keystore.password=PASSWORD
ssl.key.password=PASSWORD
client.id=console-test
Now things like kafka-consumer-groups --bootstrap-server SERVERNAME:9094 --command-config C:\ProgramData\kafka\ssl.properties --list should first pop up a firewall warning which you'd need to accept and then output a list of groups.
Kafka & Zookeeper have inbuilt Windows specific scripts as verified in the latest version. As mentioned in other other answers, the directory for windows scripts will be under kafka_2.12-2.4.0\bin\windows for Kafka. Zookeeper scripts are under /bin only but ending with "*.cmd"
There will be few minor additional steps you will have to do with ZooKeeper & Kafka like
Creating zoo.cfg for ZooKeeper
Configuring windows specific data directory for ZeeKeeper
Kafka comes with default configuration file & it has very good OS specific handling i.e. log.dirs=/tmp/kafka-logs but it will create that directory structure under C:/ directory seamlessly without failing.
It is also easy to quickly setup a Kafka Cluster with multiple Kafka Brokers on Windows with minor configurations. You can also create topics, publish/consume messages, verify topics/partitions etc. all from Windows Command Prompt. Here is a good detailed reference for the windows specific setup Kafka Setup on Windows OS | Basic Installation, Setup, Verification, Cluster Setup, Storage
I had configured Zookeeper and Apache Kafka on my Windows machine.
For Zookeeper, simply configure ZOOKEEPER_HOME and set the PATH also and then rename the zoo.sample.cfg to zoo.cfg and also change the path for dataDir in the zoo.cfg
For Apache Kafka, simply unzip and start the server from bin folder and start the Producer/Consumer from windows folder under the Kafka directory
I'm trying to install Kafka message queue on Windows for testing purposes (not for production).
I found this article on how to install Apache Kafka 0.8 on Windows: http://janschulte.wordpress.com/2013/10/13/apache-kafka-0-8-on-windows/
It's a good article, but it's outdated, unfortunately.
Does someone know a way to achieve that?
Ok, it's finally not complicated :)
The only steps are:
Edit (2021/10) Latest versions of Kafka end up with an AccessDeniedException upon startup, and even when successfully started, throw the same error during log rotation. Kafka utilizes Java methods designed for Unix filesystem libraries, so installing Kafka in WSL2 is recommended (link to Confluent blog, but steps also applicable to Apache Kafka direct installation).
Old answer
Download Kafka and uncompress it somewhere nice (let's say C:/Kafka)
Install Cygwin
If using Cygwin, edit \bin\kafka-run-class.sh and at the end of the file, change
`exec $JAVA $KAFKA_HEAP_OPTS $KAFKA_JVM_PERFORMANCE_OPTS $KAFKA_GC_LOG_OPTS $KAFKA_JMX_OPTS $KAFKA_LOG4J_OPTS -cp $CLASSPATH $KAFKA_OPTS "$#"`
to
``exec java $KAFKA_HEAP_OPTS $KAFKA_JVM_PERFORMANCE_OPTS $KAFKA_GC_LOG_OPTS $KAFKA_JMX_OPTS $KAFKA_LOG4J_OPTS -cp `cygpath -wp $CLASSPATH` $KAFKA_OPTS "$#"``
In Environment Variables, Add java to your Path System Variable:
That's it.. you can now run ZooKeeper and Kafka servers and start playing with topics and stuff..
These are the steps I followed to run kafka on Windows
Install Zookeeper first (I downloaded v3.3.6) zookeeper-3.3.6.tar.gz
Extract Zookeeper and run this command in powershell/cmd \zookeeper-3.3.6\bin> .\zkServer.cmd Now this should up a Zookeeper instance on localhost:2181
Download Kafka binary version (I downloaded v0.10.0.1)kafka_2.10-0.10.0.1.tgz
Extract Kafka, time to modify some configs
Inside Kafka extraction you can find .\config\server.properties
In .\config\server.properties replace log.dirs=c:/kafka/kafka-logs
Note: Make sure to create those folders in relevant paths
Happy news: Now Kafka ships with windows .bat scripts, You can find these files inside ./bin/windows folder
Start powershell/cmd and run this command to start Kafka broker .\bin\windows\kafka-server-start.bat .\config\server.properties
DONE!, Now you have a running Zookeeper instance and a Kafka broker.
You will probably find that Kafka comes with Windows .bat files under the bin folder to help you run Kafka under windows. However, those bat files are not updated since version 0.8.0 and some of Kafka's logic has changed (regarding topics, etc.).
I have tried two ways to solve this problem. One is to use Cygwin or MinGW to simulate Linux environment and run the Linux shell script, but there would be tons of other problems related to path names. The other solution, which is simpler and come with less trouble, is to use the corrected version of Windows bat files.
Please refer to this blog post.
Download any binary version(I've downloaded kafka_2.11-0.10.2.0) of Kafka from https://kafka.apache.org/downloads
Extract it to any folder(I've extracted to "C:\Kafka")
Open Command Prompt
Go to the folder that you extract Kafka(C:\Kafka\kafka_2.11-0.10.2.0)
Run this command to start Zookeeper
.\bin\windows\zookeeper-server-start.bat .\config\zookeeper.properties
Run this command to start Kafka
.\bin\windows\kafka-server-start.bat .\config\server.properties
Now it works!
In version 0.9.0. there are scripts in "bin" folder for windows. You can use them.
Running From A windows Shell
As of April 2019, downloading Kafka from their website worked on Windows almost right out of the box
Downloading and using the windows version .bat files is described here:
https://kafka.apache.org/quickstart
I ran into two problems when I did this:
1) JAVA_HOME was set to an unsupported JDK which led to this kind of error
Exception in thread "main" java.lang.VerifyError: Uninitialized object
exists on backward branch 209
Replacing with JDK 11 solved the problem.
2) The JAVA_HOME must not contain spaces which caused a 'cannot find the specified file' error. To fix this I used a shortened path like set JAVA_HOME=C:\Progra~1\Java\jdk-11.0.1
Running From Cygwin
If instead of the .bat files, you want to run the .sh files from cygwin, there is actually quite a bit that needs to be done, and even after that there may be problems that come up later. I can't really recommend this option, but I do use it as it's pretty convenient for some purposes.
If your JAVA_HOME path contains a space e.g. "C:\Program Files\Java\Jdk..." you will see something like this:
bin/kafka-run-class.sh: line 305: exec: C:\Program: not found
One solution is to copy the jdk to a path without spaces, and change the Java home accordingly.
If you don't want to change the JDK location, you can change the cygwin env variable as follows:
JAVA_HOME="/cygdrive/c/Program Files/Java/jdk-11.0.1"
and change the line
exec $JAVA $KAFKA_HEAP_OPTS $KAFKA_JVM_PERFORMANCE_OPTS $KAFKA_GC_LOG_OPTS $KAFKA_JMX_OPTS $KAFKA_LOG4J_OPTS -cp $CLASSPATH $KAFKA_OPTS "$#"
to
exec "$JAVA" $KAFKA_HEAP_OPTS $KAFKA_JVM_PERFORMANCE_OPTS $KAFKA_GC_LOG_OPTS $KAFKA_JMX_OPTS $KAFKA_LOG4J_OPTS -cp $CLASSPATH $KAFKA_OPTS "$#"
There was another problem with the log parameters in kafka-run-class.sh and I had to replace the line
KAFKA_GC_LOG_OPTS="-Xlog:gc*:file=$LOG_DIR/$GC_LOG_FILE_NAME:time,tags:filecount=10,filesize=102400"
with
KAFKA_GC_LOG_OPTS="-Xlog:gc*"
And even after all these changes I occasionally run into problems of Kafka shutting down because of an incompatible windows style path in the logs directories as described here:
Kafka 1.0 stops with FATAL SHUTDOWN error. Logs directory failed
In short, you may be better off running the Kafka .bat scripts from the windows directory...
With Chocolatey and Powershell:
chocolatey.exe install kafka -> installs Kafka into C:\ProgramData\chocolatey\lib\kafka\tools\kafka_2.11-1.0.0\ - your version might differ of course
add C:\ProgramData\chocolatey\lib\kafka\tools\kafka_2.11-1.0.0\ to your powershell path using notepad $PROFILE
After restarting powershell kafka-console-consumer should work as a command.
You also need to install ssl.properties with keystore and truststore. Put them e.g. into C:\ProgramData\kafka and put that into ssl.properties (please note the escaping of the backslashes):
security.protocol=SSL
ssl.truststore.location=C:\\ProgramData\\kafka\\kafka-truststore.jks
ssl.truststore.password=PASSWORD
ssl.keystore.location=C:\\ProgramData\\kafka\\kafka-keystore.jks
ssl.keystore.password=PASSWORD
ssl.key.password=PASSWORD
client.id=console-test
Now things like kafka-consumer-groups --bootstrap-server SERVERNAME:9094 --command-config C:\ProgramData\kafka\ssl.properties --list should first pop up a firewall warning which you'd need to accept and then output a list of groups.
Kafka & Zookeeper have inbuilt Windows specific scripts as verified in the latest version. As mentioned in other other answers, the directory for windows scripts will be under kafka_2.12-2.4.0\bin\windows for Kafka. Zookeeper scripts are under /bin only but ending with "*.cmd"
There will be few minor additional steps you will have to do with ZooKeeper & Kafka like
Creating zoo.cfg for ZooKeeper
Configuring windows specific data directory for ZeeKeeper
Kafka comes with default configuration file & it has very good OS specific handling i.e. log.dirs=/tmp/kafka-logs but it will create that directory structure under C:/ directory seamlessly without failing.
It is also easy to quickly setup a Kafka Cluster with multiple Kafka Brokers on Windows with minor configurations. You can also create topics, publish/consume messages, verify topics/partitions etc. all from Windows Command Prompt. Here is a good detailed reference for the windows specific setup Kafka Setup on Windows OS | Basic Installation, Setup, Verification, Cluster Setup, Storage
I had configured Zookeeper and Apache Kafka on my Windows machine.
For Zookeeper, simply configure ZOOKEEPER_HOME and set the PATH also and then rename the zoo.sample.cfg to zoo.cfg and also change the path for dataDir in the zoo.cfg
For Apache Kafka, simply unzip and start the server from bin folder and start the Producer/Consumer from windows folder under the Kafka directory
I want to run JMeter test plans from a UNIX server where env variables for java is not set. Its a test server and I dont have access to set that. We have different JDK versions and all are sitting in directories. I need to run my JMeter TestPlans on this server. I went to JDK /bin folder and tired to execute the below command
$ {jmeter-path}/bin/jmeter -nt testplan.jmx -l testresult.jtl
but this says
./bin/java: not found
But if I do simple java -version it shows the version result.Is that something that JMeter needs specifically the java env variable set or it wont run ? I dont have permission to set and I want to run the testplan using the JDK/JRE from its directories. A help would be appreciated. Thanks in advance!
UPDATE:
I think I could work around this by editing the jmeter script file as suggested in the one of the comments. Since my requirement was running the JMeter from a specific server, I could achieve this by editing the JAVA_HOME variable value in the script.
JMeter looks for java executable in system PATH so you have 2 options:
Add bin folder of your JDK or JRE to PATH, something like:
PATH=$PATH:/location/of/your/jbk/bin && export PATH && {jmeter-path}/bin/jmeter -nt testplan.jmx -l testresult.jtl
Or if you have java in PATH just run ApacheJMeter.jar like:
java -jar {jmeter-path}/bin/ApacheJMeter.jar
You might also want to use jmeter.sh wrapper script instead of jmeter, it has some logic regarding java binary location
More information: Get Started With JMeter: Installation & Tests
The official JMeter Getting Started documentation says this:
To install a release build, simply unzip the zip/tar file into the directory where you want JMeter to be installed. Provided that you have a JRE/JDK correctly installed and the JAVA_HOME environment variable set, there is nothing more for you to do.
Based on the symptoms that you reported, I think that you have not set JAVA_HOME correctly. It should be set to an absolute path to your Java installation's top directory.
I have access to a server using SSH. I need to run a stand-alone Java application on it to access a MySQL server installed there. How do I go about in doing this?
Assuming you have the requirement to copy the JAR file on the *nix box and then run it(and not connect a Java process to it remotely)
Create a standalone JAR which contains all the dependencies required to run the application
Make sure you have Java installed on that machine
Assuming it's a *nix box, set the $PATH environment variable to point to $JAVA_HOME/bin
Log on to that box using a SSH client. Any decent SSH client also comes with a FTP plugin which allows you to transfer files between your local box and the server
Copy the JAR to the appropriate directory and run it using the java -jar your.jar command
Assuming it is a linux machine, you have to connect to by using SSH it and use scp command to upload the files and deploy it...
than you have to run the JAR you deployed:
java -jar /path/to/file.jar
Or, provide more details please
I need an urgent help from you people.I got something very odd while trying to integrate php with java. First, my system specifications:
Windows XP
I have installed
XAMPP server:- xampp-win32-1.6.1-installer
This install PHP, Apache, and MySQL on my system. There versions are as follows
Apache Version :- Apache/2.2.4 (Win32)
PHP version :- 4.3.1
Sun Microsystems JDK version :- jdk1.6.0_16
I am achieving this PHP-JAVA extensoion using php-javabridge. I have downloaded javabridge.jar file from following url.
placed the downloaded javabridge.jar file on this path C:\xampp\php\ext\
Settings done in php.ini file for php-java integration are as follows.
; Directory in which the loadable extensions (modules) reside. extension_dir = "C:\xampp\php\ext\"
I also uncomment the java extension.
extension=php_java.dll
I have added following lines in Module Settings section of PHP.ini file.
;;;;;;;;;;;;;;;;;;; ; Module Settings ; ;;;;;;;;;;;;;;;;;;;
[Java] ;This points to the directory where your Java ;classes will be stored. You can designate multiple ;paths, each path separated by a semicolon. ;It must also include the location of php_java.jar java.class.path = "C:\xampp\php\ext\JavaBridge.jar;C:\xampp\php\extensions\php_java.jar;C:\Program Files\Java\jdk1.6.0_16\jre\lib;C:\Program Files\Java\jdk1.6.0_16;C:\prog"
;java.class.path = "C:\xampp\php\extensions\php_java.jar;C:\prog" ; This points to the bin directory of the JDK. java.home = "C:\Program Files\Java\jdk1.6.0_16\bin"
; This must point to the Java Virtual Machine (jvm.dll) file. java.library = "C:\Program Files\Java\jdk1.6.0_16\jre\bin\server\jvm.dll"
; This must point to the location of php_java.dll. java.library.path = "C:\xampp\php\ext;C:\Program Files\Java\jdk1.6.0_16\jre\lib"
java.java = "C:\Program Files\Java\jdk1.6.0_16\bin\javaw.exe"
When I start the apache server service, the program runs ok. But, if I close the browser and open it again, the program no longer runs and give me a "Fatal error: Unable to create Java Virtual Machine in C:\php\java.php ...". If I restart the apache server service, the program works again, but with the same behavior: if I close the browser window and open it again, it does not work. I checked on internet but dont get any solution but found many people facing same problem. And many of them told its a bug in PHP-JAVA bridge. So is there any solution on this problem. I ran out of options and, if anyone could help, I'll appreciate.
Thank You.
I tried the Php-JavaBridge recently, by downloading the latest version of Apache, Tomcat and using JDK8.0 and I think Php 5.4 and got it to work totally fine.
Perhaps try downloading the latest version of everything, start from scratch and install again, it may be you have some version mismatches.
I was using a Windows 64 bit computer, running Windows7.