I've got problem forwarding jmx from kubernetes pod to localhost.
Everything works fine when port is forwarded to the same port:
kubectl port-forward ... 9010:9010 OR
kubectl port-forward ... 9010
However, when I try to forward to a different local port,
kubectl port-forward ... 9011:9010
neither jconsole or jms can connect.
It is blocking me from profiling multiple applications at once.
JMS Error:
com.oracle.jmc.rjmx.ConnectionException caused by java.rmi.NoSuchObjectException: no such object in table
at com.oracle.jmc.rjmx.internal.RJMXConnection.connect(RJMXConnection.java:406)
at com.oracle.jmc.rjmx.internal.ServerHandle.doConnect(ServerHandle.java:88)
at com.oracle.jmc.rjmx.internal.ServerHandle.connect(ServerHandle.java:78)
at com.oracle.jmc.console.ui.editor.internal.ConsoleEditor$ConnectJob.run(ConsoleEditor.java:73)
at org.eclipse.core.internal.jobs.Worker.run(Worker.java:55)
Caused by: java.rmi.NoSuchObjectException: no such object in table
at java.rmi/sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:283)
at java.rmi/sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:260)
at java.rmi/sun.rmi.server.UnicastRef.invoke(UnicastRef.java:161)
at java.management.rmi/javax.management.remote.rmi.RMIServerImpl_Stub.newClient(Unknown Source)
at java.management.rmi/javax.management.remote.rmi.RMIConnector.getConnection(RMIConnector.java:2105)
at java.management.rmi/javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:321)
at com.oracle.jmc.rjmx.internal.RJMXConnection.connectJmxConnector(RJMXConnection.java:451)
at com.oracle.jmc.rjmx.internal.RJMXConnection.establishConnection(RJMXConnection.java:427)
at com.oracle.jmc.rjmx.internal.RJMXConnection.connect(RJMXConnection.java:399)
... 4 more
jconsole error:
Connection to localhost:9011 did no succeed.
This is an RMI issue. The problem is that the RMI stub delivered to the JMX client is created for 9010 and so it fails when attempting to connect at 9011. There's a couple of decent solutions outlined here. Another option is to switch to JMXMP which is a pure socket JMX protocol so port forwarding works without any additional workarounds.
Related
I'm attempting to create a Docker container using quay.io/keycloak/keycloak:20.0.1 image the that will connect to a Postgres Database listening on IPv6. Working with a downloaded zip of Keycloak, the application is able to make a connection if I export JAVA_OPTS as follows:
$ export JAVA_OPTS=-Djava.net.preferIPv6Addresses=true
If I attempt to do the same within Docker, however, I receive java.net.UnknownHostException. I've tried both the hostname which should resolve, as well as the IP that is behind the hostname.
If I try to use the IP within the Docker container, I receive:
Caused by: java.net.SocketException: Protocol family unavailable
I'm not sure how to proceed from here, thanks!
UPDATE: This seems to extend beyond just Keycloak, I’ve attempted connecting to the same DB via container using both Spring Boot and Quarkus and receive the same error.
I want to run distributed load testing by jmeter, where servers(linux) are in the cloud and I have ability to start test from my local pc.
Is it even possible or client and servers should be in same subnet?
And in case its possible, may you provide me step by step guide how to achive this goal?
I have tired
https://jmeter.apache.org/usermanual/remote-test.html
https://cloud.google.com/community/tutorials/ssh-port-forwarding-set-up-load-testing-on-compute-engine?hl=ja
but I got error after error.
Before I disabled ssl verification, next error appeared:
error during JRMP connection establishment; nested exception is:
javax.net.ssl.SSLHandshakeException: Remote host terminated the handshake
With disabled ssl verification I have next error:
Error in rconfigure() method java.rmi.MarshalException: error marshalling argume
nts; nested exception is:
java.net.SocketException: Connection reset by peer: socket write error
I have feeligs, like I making smth wrong with setting ip, port or firewall, but cant understand what
It is definitely possible to starts load agents (jmeter-server) from your local system. The only condition is that the traffic should not be blocked by firewall.
Based on your error, it looks like jmeter process is not having permission to open a port, port is already open (although there is a specific exception for this)
You can also try allowing all incoming traffic to your slaves (cloud linux boxes) and try running the test again.
According to the manual you mentioned all you need to do is just to take the following steps:
On your local machine set up the SSH tunnel:
ssh -L 24000:127.0.0.1:24000 -R 25000:127.0.0.1:25000 -L 26000:127.0.0.1:26000 username#hostname_of_cloud_machine
On the cloud machine launch the JMeter slave
./jmeter -s -Jserver.rmi.ssl.disable=true -Jserver_port=24000 -Jserver.rmi.localhostname=127.0.0.1 -Jserver.rmi.localport=26000 -j jmeter.log
Again on the local machine launch the JMeter master:
./jmeter -Jserver.rmi.ssl.disable=true -Jremote_hosts=127.0.0.1:24000 -Jclient.rmi.localport=25000 -Jmode=Statistical -n -t test.jmx -l result.jtl
You don't need to configure any firewalls as all the communication is happening over the SSH tunnel
More information:
Remote hosts and RMI configuration
Apache JMeter Properties Customization Guide
I am running siple java jar service on openshift which tried to connect with DatagramSocket and Getting this Exception:
java.net.BindException: Permission denied
at java.net.PlainDatagramSocketImpl.bind0(Native Method)
at java.net.AbstractPlainDatagramSocketImpl.bind(AbstractPlainDatagramSocketImpl.java:96)
at java.net.DatagramSocket.bind(DatagramSocket.java:397)
at java.net.DatagramSocket.<init>(DatagramSocket.java:251)
at java.net.DatagramSocket.<init>(DatagramSocket.java:304)
at java.net.DatagramSocket.<init>(DatagramSocket.java:276)
at com.gasmps.service.TerminationListener.run(EmailService.java:542)
at java.lang.Thread.run(Thread.java:745)
I know this is port permission problems but i tried with port like 8000,8443,15005,16005 but getting same how could i resolve this? How i can know all open port that can i use in my application.
Edit: My jboss server is already running in the same application and I don't want to create separate application for these jar service bcos I have to run this jar from my application from remote location.
Iv looked into this before after having the same issue and discovered the answer on stack-overflow before:
In Openshift, you may only bind to the port that the server allocates
for you. Typically that is the value of the environment variable
$OPENSHIFT_JBOSS_PORT (it varies between cartridges) and the value
differs between gears. Binding to any other port will be reported as
an SELinux policy violation and denied.
link
The port is usually 8080
you can bind your app to 8080 (not to 8000 or 8443), all traffic >received on 8000 and 8443 will be routed by the proxy to your app >listening on 8080.
This worked for me, but let me know how it goes and if it still doesn't work i'll try and help.
I am trying to profile a Java Spring application with the Oracle Java Mission control.
I have the JVM Connection configured as: localhost:7091
and I started the application with java -Dcom.sun.management.jmxremote.rmi.port=7091 -jar app.jar.
When I attempt to start the flight recorder I get an error message.
Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: localhost; nested exception is:
java.net.ConnectException: Connection refused: connect]
Could not connect to a JVM at service:jmx:rmi:///jndi/rmi://localhost:7091/jmxrmi. Make sure one is running and that you are using the correct protocol in the Service URL.
Does anyone know what could be the reason for that error?
You could try to use auto discovery. Then the connection will show up automatically in JMC
-Dcom.sun.management.jmxremote.autodiscovery=true
You can also name the connection so you can identify it easily.
-Dcom.sun.management.jdp.name=App
and disable security
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
I had this issue. I had installed JDK 10 on Windows Server 2016. With JMC and the JVM I wished to monitor on the same server.
When going live, we switched the IP from a temporary IP to a live IP. However, JMC resolved localhost to the original IP. Also when specifying the server's current IP, JMC resolved it to the original IP.
Adding -Djava.rmi.server.hostname=localhost to the Java process I wanted to monitor, resolved the issue.
However, I will would still be interested to know why JMC (or RMI) still resolves incorrectly.
I have a java rmi application i simply do:
Client:
Registry registry = LocateRegistry.getRegistry("localhost");
costApi = (CostApi) registry.lookup("server.CostApi");
Everything works fine when I host the server at localhost. When I start the same program at another machine withing the local network, at 192.168.x.x and change to:
Client:
Registry registry = LocateRegistry.getRegistry("192.168.x.x");
costApi = (CostApi) registry.lookup("server.CostApi");
it does not work anymore and it fails with a very strange error:
java.rmi.ConnectException: Connection refused to host: 127.0.1.1; nested exception is:
java.net.ConnectException: Connection refused
at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:619)
at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:216)
at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202)
at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:129)
at java.rmi.server.RemoteObjectInvocationHandler.invokeRemoteMethod(RemoteObjectInvocationHandler.java:194)
at java.rmi.server.RemoteObjectInvocationHandler.invoke(RemoteObjectInvocationHandler.java:148)
at com.sun.proxy.$Proxy0.dataCost(Unknown Source)
at billing.data.DataBiller.performBilling(DataBiller.java:57)
at billing.data.DataBiller.consumeMessage(DataBiller.java:46)
at general.templates.RabbitWorker.run(RabbitWorker.java:124)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
I'm not even trying to connect to 127.0.1.1 but to 192.168.x.x, how do I solve this? I prefer to use java code only and not modify my machine with config files. I'm using linux
This is usually caused by a misconfiguration. Check your /etc/hosts file to ensure that:
localhost maps to 127.0.0.1
your real hostname maps to your real host address
Some Linux distributions are known to have this back to front.
If the problem persists, try setting java.rmi.server.hostname at the server to the IP address the client should use when executing remote method calls. It needs to be set before you export any remote objects, including the Registry.
The problem is caused by the IP address embedded in the stub, which ultimately comes from something like InetAddress.getLocalAddress(), which is fallible as above. It is overridden by java.rmi.server.hostname.
This is item A.1 in the FMI FAQ, but note that the item is mistitled. It doesn't happen during lookup(), it happens when you call a remote method on the resulting stub.
I just ran into the same issue. I'm doing something very similar to what you are doing. What I noticed was that the first time I ran the client program, and it failed ( by design of the firewall test ) - that it failed with an error message showing the actual ip address of the host that I originally specified ( the 192.168.x.x address ), but all subsequent failures show that it is failing to make a connection to 127.0.0.1. Currently I'm suspecting some kind of caching on the client - has the JVM marked that ip address as never accessible again and it's refusing to ever try to connect to it again?
UPDATE: In my case, the JVM on the RMI Server side was not able to properly set the java.rmi.server.hostname property at JVM startup. This property was being left as null. When clients connect to a specific RMI Registry and ask for a stub to a particular named object, they receive a stub containing the ip address of the remote machine where the actual object can be found. The RMI server copies the contents of the java.rmi.server.hostname property into the stubs it returns to clients, so if the java.rmi.server.hostname property is "" and it copies that to each stub it creates, each stub contains a reference to remote server with an IP address of "". By default the client jvm reacts to this by attempting to connect to the server object on the localhost, 127.0.0.1. To work around the problem, try this line of code before exporting any remote objects on the server side:
System.setProperty( "java.rmi.server.hostname", "192.168.RMIServer.IP" ) ;
This property will be automatically copied to all remote stubs exported on that server, and clients who receive that stub should then be able to reach the remote server ( assuming any firewalls are configured correctly ).