Added one parameter in flink-conf.yaml :
env.java.opts.taskmanager: "-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=51005"
Then i started a jobmanager and a taskmanager using below commands on localhost :
flink-1.9.1_bin/bin/jobmanager.sh start
flink-1.9.1_bin/bin/taskmanager.sh start
Jobmanager runs and i can see flink UI dashboard on localhost:8081. Taskmanager waits for connection on 51001 port. It runs after i debug my code from IDE(IntelliJ) which has setup for remote debugging on localhost:51001. I can see tasks slots been added when i start debugging from IDE. After this i ran below command:
flink-1.9.1_bin/bin/flink run -c myapp.Main myapp.jar
I am expecting debug point to come to my local code in IDE, but it's not coming. My code goes in running state directly that i can in flink UI dashboard.
I am able do remote debugging for normal java projects but not for flink jobs.
I am trying to run a basic robotframework test with RemoteSwingLibrary, but I can't seem to get it to work.
I was able to run the following test just fine with SwingLibrary:
*** Settings ***
Library SwingLibrary
*** Test Cases ***
Foobar Test
Start Application fully.qualified.name.of.my.class
I am running it from within Eclipse via the robot framework standalone jar. I put robotframework-3.0.2.jar and swinglibrary-1.9.7.jar on my class path, created a new run configuration using my projects default classpath with the main class being org.robotframework.RobotFramework. In the arguments, I specify the name of my test file.
So, that seems to work fine. Then I tried to do almost exactly the same thing with RemoteSwingLibrary:
*** Settings ***
Library RemoteSwingLibrary
*** Test Cases ***
Foobar Test
Start Application my_app java fully.qualified.name.of.my.class
I replaced the swinglibrary jar on my classpath with remoteswinglibrary-2.2.1.jar and I ran the same test. The output says:
console: Failed to install '': java.nio.charset.UnsupportedCharsetException: cp0.
==============================================================================
Remotetest
==============================================================================
Foobar Test | FAIL |
RemoteSwingLibraryTimeoutError: Agent port not received before timeout
------------------------------------------------------------------------------
Remotetest | FAIL |
1 critical test, 0 passed, 1 failed
1 test total, 0 passed, 1 failed
==============================================================================
Output: C:\<path>\output.xml
Log: C:\<path>\log.html
Report: C:\<path>\report.html
(note that the "console: Failed to install..." part is normal and I don't think I need to be concerned about it)
When I dig into the report for the Start Application Keyword, it says
13:48:40.070 INFO Link to stdout
13:48:40.070 INFO Link to stderr
13:48:40.070 INFO -javaagent:"C:<pathToProject>\__pyclasspath__"=127.0.0.1:63110
13:48:40.071 INFO Starting process:
java fully.qualified.name.of.my.class
13:49:40.145 INFO Failed to start application: Traceback (most recent call last):
File "__pyclasspath__/RemoteSwingLibrary.py", line 307, in start_application
File "__pyclasspath__/RemoteSwingLibrary.py", line 353, in _application_started
File "__pyclasspath__/RemoteSwingLibrary.py", line 373, in _get_agent_address
RemoteSwingLibraryTimeoutError: Agent port not received before timeout
13:49:40.155 INFO Waiting for process to complete.
13:49:40.165 INFO Process completed.
13:49:40.165 INFO STDOUT: Error occurred during initialization of VM
agent library failed to init: instrument
13:49:40.175 INFO STDERR: Error opening zip file or JAR manifest missing : C:<pathToProject>\__pyclasspath__
Picked up JAVA_TOOL_OPTIONS: -javaagent:"C:<pathToProject>\__pyclasspath__"=127.0.0.1:63110
Picked up _JAVA_OPTIONS: -Djava.security.policy="C:\<UserFolder>\
AppData\Local\Temp\2\grant_all_f_vfit.policy"
13:49:40.195 FAIL RemoteSwingLibraryTimeoutError: Agent port not received before timeout
I'm not sure if this hunch is really founded on anything, but my suspicion is that there's a classpath issue. I would think that when I call the Start Application keyword with "java fully.qualified.name.of.my.class" I would probably have to pass along my classpath. I was hoping that maybe I wouldn't since the non-remote version seems to handle that just fine (probably because its running my app in the same JVM as I launched robot framework in, and that already included everything I needed in my classpath).
So if that's the issue, I can probably fix it by using the -cp option in my java command. However, the classpath for this particular project is actually quite long (my project is dependent on a lot of other projects) and I would rather not have to list it all out. Also, when I add a new dependency down the road, I would rather not have to update the classpath in my test file as well.
Is there some magic way that I can pass the current classpath along to the remote app? Or perhaps some other solution to this problem?
Update
I tried again with -cp and my giant classpath and it gave me the same errors, so its possible that my classpath has nothing to do with whatever is going on.
I also tried specifying the port number explicitly and it gives me slightly different output (though it still doesn't work). I also discovered that there's a debug setting that gives a little more detail. Here's me new setup:
*** Settings ***
Library RemoteSwingLibrary debug=True port=8242
*** Test Cases ***
Foobar Test
Start Application my_app java -cp <classpath> <fully.qualified.name.of.my.class> remote_port=8242
And here is my new output:
13:48:40.070 INFO Link to stdout
13:48:40.070 INFO Link to stderr
13:48:40.070 DEBUG Picked old JAVA_TOOL_OPTIONS=''
13:48:40.070 DEBUG Picked old _JAVA_OPTIONS=''
13:48:40.070 INFO -javaagent:"C:<pathToProject>\__pyclasspath__"=127.0.0.1:51509:DEBUG
13:48:40.070 DEBUG Set JAVA_TOOL_OPTIONS='-javaagent:"C:<pathToProject>\__pyclasspath__"=127.0.0.1:8242:APPORT=8242:DEBUG'
13:48:40.070 DEBUG SET _JAVA_OPTIONS='-Djava.security.policy="C:\<UserFolder>\AppData\Local\Temp\4\grant_all_haud6t.policy"'
13:48:40.071 INFO Starting process:
java -cp <classpath> <fully.qualified.name.of.my.class>
13:48:40.071 DEBUG Process configuration:
cwd: C:<PathToProject>
shell: True
stdout: C:<PathToProject>\<someBigPath>.txt
stderr: C:<PathToProject>\<someBigPath>.txt
alias: my_app
env: None
13:48:40.100 DEBUG Returned old JAVA_TOOL_OPTIONS=''
13:48:40.100 DEBUG Returned old _JAVA_OPTIONS=''
13:48:40.102 INFO connection to started application at 127.0.0.1:8242
13:48:40.120 DEBUG remote swinglibrary instantiated
13:48:40.121 DEBUG remote services instantiated
13:48:40.121 INFO waiting for api at 127.0.0.1:8242
13:49:40.145 INFO Failed to start application: Traceback (most recent call last):
File "__pyclasspath__/RemoteSwingLibrary.py", line 307, in start_application
File "__pyclasspath__/RemoteSwingLibrary.py", line 357, in _application_started
File "__pyclasspath__/RemoteSwingLibrary.py", line 344, in _wait_for_api
RuntimeError: Connecting to api at 127.00.1:8242 has failed: ''
13:49:40.155 INFO Waiting for process to complete.
13:49:40.165 INFO Process completed.
13:49:40.165 INFO STDOUT: Error occurred during initialization of VM
agent library failed to init: instrument
13:49:40.175 INFO STDERR: Error opening zip file or JAR manifest missing : C:<pathToProject>\__pyclasspath__
Picked up JAVA_TOOL_OPTIONS: -javaagent:"C:<pathToProject>\__pyclasspath__"=127.0.0.1:8242:APPORT=8242:DEBUG
Picked up _JAVA_OPTIONS: -Djava.security.policy="C:\<UserFolder>\
AppData\Local\Temp\2\grant_all_f_vfit.policy"
13:49:40.195 FAIL Connecting to api at 127.0.0.1:8242 has failed: ''
13:49:40.195 DEBUG Traceback (most recent call last):
File "__pyclasspath__/RemoteSwingLibrary.py", line 498, in run_keyword
File "__pyclasspath__/RemoteSwingLibrary.py", line 307, in start_application
File "__pyclasspath__/RemoteSwingLibrary.py", line 357, in _application_started
File "__pyclasspath__/RemoteSwingLibrary.py", line 344, in _wait_for_api
Yet another update
I think that this may have something to do with the "__pyclasspath__" that shows up in the error message. I put my java command in a .bat file preceded by a "set" command to list my environment variables. I can run the .bat file from the command line just fine (it launches the java app).
Then I modified my .robot file to do:
Foobar Test
Start Application my_app cd robotframework/tests && Run.bat remote_port=8242
I get essentially the same error output that I got before:
Error opening zip file or JAR manifest missing : C:<pathToProject>\__pyclasspath__
Picked up JAVA_TOOL_OPTIONS: -javaagent:"__pyclasspath__"=127.0.0.1:8242:APPORT=8242:DEBUG
Picked up _JAVA_OPTIONS: -Djava.security.policy="C:\\AppData\Local\Temp\1\grant_all_lgirs7.policy"
The environment that my batch script prints out appears the same except for JAVA_TOOL_OPTIONS and _JAVA_OPTIONS (which match with what the error output says). There is no environment variable called __pyclasspath__ so I'm wondering if that's the issue.
I set JAVA_TOOL_OPTIONS in my cmd environment to match the output that I saw from my robot run and now I get the same error message when I try to run my .bat script from the command line, so that environment variable seems to be the issue.
Well, I came up with a fix, though its definitely not an ideal fix.
I found out that when the framework kicks of my remote app, it sets:
JAVA_TOOL_OPTIONS=-javaagent:"C:\<pathToProject>\__pyclasspath__
But __pyclasspath__ is not actually an environment variable. I figured out that the javaagent has to point to the remote framework library, so I decided to just set it explicitly.
So now I changed my robot file to something like this:
*** Settings ***
Library RemoteSwingLibrary
*** Test Cases ***
Foobar Test
Start Application my_app run.bat remote_port=1234
And then made a file called run.bat that does:
set JAVA_TOOL_OPTIONS=-javaagent:"C:\<pathToProject\lib\remoteswinglibrary-2.2.1.jar"=127.0.0.1:1234:APPORT=1234
java -cp <classPath> fully.qualified.name.of.class
This is obviously sort of a hacky solution and it will take a little work to make it portable (I'll need to get rid of the hardcoded "pathToProject" and also make it support Linux environments as well). It also requires hardcoding in a port, whereas it would be nice if the robotframework could just select a port for me.
So, I would definitely like to see a better solution if anyone can find one, but for now, this will at least work.
I wrote a few Spark job in Java then submitted the jars with submit script.
bin/spark-submit --class "com.company.spark.jobName.SparkMain" --master local[*] /tmp/spark-job-1.0.jar
There will be a service and will run in same server. The service should stop the job when receive the stop command.
I have these information about job in service:
SparkHome
AppName
AppResource
Master uri
app-id
status
Is there any way to stop running spark job in java code.
Have you reviewed the REST server and the ability to use /submissions/kill/[submissionId]? That seems like it would work for your need.
Using win7-64, jdk8, sparks1.6.2.
I have spark running, winutils, HADOOP_HOME, etc
Per documentation Note: The launch scripts do not currently support Windows. To run a Spark cluster on Windows, start the master and workers by hand. But does not say how?
How do I launch spark master on windows?
Tried running sh start-master.sh thru git bash : failed to launch org.apache.spark.deploy.master.Master: Even though it prints out Master --ip Sam-Toshiba --port 7077 --webui-port 8080 - So I don't know what all this means.
But when I try spark-submit --class " " --master spark://Sam-Toshiba:7077 target/ .jar -
I get errors:
WARN AbstractLifeCycle: FAILED SelectChannelConnector#0.0.0.0:
4040: java.net.BindException: Address already in use: bind
java.net.BindException: Address already in use
WARN Utils: Service 'SparkUI' could not bind on port 4040. Attempting port 4041.
17/01/12 14:44:29 WARN AppClient$ClientEndpoint: Failed to connect to master Sam-Toshiba:7077
java.io.IOException: Failed to connect to Sam-Toshiba/192.168.137.1:7077
at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:216)
Also tried spark://localhost:7077 - same errors
On Windows you can launch Master using below command. Open command prompt and go to Spark bin folder and execute
spark-class.cmd org.apache.spark.deploy.master.Master
Above command will print like Master: Starting Spark master at spark://192.168.99.1:7077 in console as per IP of your machine. You can check the UI at http://192.168.99.1:8080/
If you want to launch worker once your master is up you can use below command. This will use all the available cores of your machine.
spark-class.cmd org.apache.spark.deploy.worker.Worker spark://192.168.99.1:7077
If you want to utilize 2 cores of your 4 cores of machine then use
spark-class.cmd org.apache.spark.deploy.worker.Worker -c 2 spark://192.168.99.1:7077
I can't define a valid upstart conf script to run a java service using upstart with the following requirements:
I have to specify classpath using folders because I have many jars in multiple folders
I have to listen to the shutdown signal fired by service myservicename stop
Based on that answer, I implemented a shutdown hook listener so I need upstart to send me the termination signal and wait for my application to terminate.
Here is my buggy upstart script:
description "masa"
author "Muhammad Gelbana <m.glba#gmail.com>"
start on runlevel [2345]
stop on shutdown
kill timeout 120
script
LOGS_DIR=/home/mgelbana/services/RealServices/logs
IPK_DB=/home/mgelbana/services/RealServices/config/db-ipk.properties
PRO_DB=/home/mgelbana/services/RealServices/config/db-reporting-engine.properties
MAIN_CLASS=com.sger.masaTA
mkdir -p $LOGS_DIR
CLASSPATH="/home/mgelbana/services/RealServices/masa-RealService-TA.jar"
for i in /home/mgelbana/services/commons/*.jar; do
CLASSPATH="$CLASSPATH:$i"
done
for i in /home/mgelbana/services/RealServices/lib/*.jar; do
CLASSPATH="$CLASSPATH:$i"
done
echo '\n\n\n====================================================='
echo 'Service startup:\t'`date`
echo 'Main class:\t\t'`echo $MAIN_CLASS`
echo 'Logs directory:\t\t'`echo $LOGS_DIR`
echo 'masa database configuration:\t'`echo $IPK_DB`
echo 'Pro configuration file:\t'`echo $PRO_DB`
echo 'Starting engine...'
java -Dta.id=2 -DIPK_DB=$IPK_DB -DPRO_DB=$PRO_DB -cp $CLASSPATH $MAIN_CLASS
end script
The following error is shown in the /var/log/upstart/myservicename.log log:
/proc/self/fd/9: 9: /proc/self/fd/9: Syntax error: word unexpected (expecting "do")
Thank you.