I've got problem forwarding jmx from kubernetes pod to localhost.
Everything works fine when port is forwarded to the same port:
kubectl port-forward ... 9010:9010 OR
kubectl port-forward ... 9010
However, when I try to forward to a different local port,
kubectl port-forward ... 9011:9010
neither jconsole or jms can connect.
It is blocking me from profiling multiple applications at once.
JMS Error:
com.oracle.jmc.rjmx.ConnectionException caused by java.rmi.NoSuchObjectException: no such object in table
at com.oracle.jmc.rjmx.internal.RJMXConnection.connect(RJMXConnection.java:406)
at com.oracle.jmc.rjmx.internal.ServerHandle.doConnect(ServerHandle.java:88)
at com.oracle.jmc.rjmx.internal.ServerHandle.connect(ServerHandle.java:78)
at com.oracle.jmc.console.ui.editor.internal.ConsoleEditor$ConnectJob.run(ConsoleEditor.java:73)
at org.eclipse.core.internal.jobs.Worker.run(Worker.java:55)
Caused by: java.rmi.NoSuchObjectException: no such object in table
at java.rmi/sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:283)
at java.rmi/sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:260)
at java.rmi/sun.rmi.server.UnicastRef.invoke(UnicastRef.java:161)
at java.management.rmi/javax.management.remote.rmi.RMIServerImpl_Stub.newClient(Unknown Source)
at java.management.rmi/javax.management.remote.rmi.RMIConnector.getConnection(RMIConnector.java:2105)
at java.management.rmi/javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:321)
at com.oracle.jmc.rjmx.internal.RJMXConnection.connectJmxConnector(RJMXConnection.java:451)
at com.oracle.jmc.rjmx.internal.RJMXConnection.establishConnection(RJMXConnection.java:427)
at com.oracle.jmc.rjmx.internal.RJMXConnection.connect(RJMXConnection.java:399)
... 4 more
jconsole error:
Connection to localhost:9011 did no succeed.
This is an RMI issue. The problem is that the RMI stub delivered to the JMX client is created for 9010 and so it fails when attempting to connect at 9011. There's a couple of decent solutions outlined here. Another option is to switch to JMXMP which is a pure socket JMX protocol so port forwarding works without any additional workarounds.
This question already has answers here:
JDBC connection failed, error: TCP/IP connection to host failed
(9 answers)
Closed 5 years ago.
com.microsoft.sqlserver.jdbc.SQLServerException: The TCP/IP connection to the host localhost, port 1433 has failed. Error: "Connection refused: connect. Verify the connection properties. Make sure that an instance of SQL Server is running on the host and accepting TCP/IP connections at the port. Make sure that TCP connections to the port are not blocked by a firewall.".
at com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDriverError(SQLServerException.java:191)
at com.microsoft.sqlserver.jdbc.SQLServerException.ConvertConnectExceptionToSQLServerException(SQLServerException.java:242)
at com.microsoft.sqlserver.jdbc.SocketFinder.findSocket(IOBuffer.java:2369)
at com.microsoft.sqlserver.jdbc.TDSChannel.open(IOBuffer.java:551)
at com.microsoft.sqlserver.jdbc.SQLServerConnection.connectHelper(SQLServerConnection.java:1963)
at com.microsoft.sqlserver.jdbc.SQLServerConnection.login(SQLServerConnection.java:1628)
at com.microsoft.sqlserver.jdbc.SQLServerConnection.connectInternal(SQLServerConnection.java:1459)
I'm facing this exception after I tried to connect with my driver through the new database profile.
The information I have was correct and the TCP port was enabled and the firewall was turned off.
I'm using Microsoft SQL server 2008 JDBC driver and I've added the jars.
A "connection refused" is rarely caused by the connecting client, i.e. the JDBC client within your Eclipse setup. A good check is to open a shell (e.g. cmd.exe on Windows) and do a
telnet [server name used in JDBC configuration] 1433
On newer Windows installations you need to install the telnet client because it's not part of the standard installation anymore (Settings -> Software -> Add/Remove Features).
If that leads to a similar error message you can focus your trouble shooting to the network part of your system, since the JDBC side is completely uninvolved. If you can connect, make sure that the server name resolves to the same IP when using the command prompt as it's resolved within Java (I once had this effect which took quite a time to find that out). Especially with localhost as server name one might resolve to 127.0.0.1 while the other resolvs to ::1.
java.rmi.ConnectException: Connection refused to host: 127.0.1.1; nested exception is:
java.net.ConnectException: Connection refused
at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:619)
at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:216)
at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202)
at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:128)
at java.rmi.server.RemoteObjectInvocationHandler.invokeRemoteMethod(RemoteObjectInvocationHandler.java:194)
at java.rmi.server.RemoteObjectInvocationHandler.invoke(RemoteObjectInvocationHandler.java:148)
at com.sun.proxy.$Proxy0.notifyMe(Unknown Source)
at CallbackServerImpl.doCallback(CallbackServerImpl.java:149)
at CallbackServerImpl.registerForCallback(CallbackServerImpl.java:70)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
at sun.rmi.transport.Transport$1.run(Transport.java:177)
at sun.rmi.transport.Transport$1.run(Transport.java:174)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)
Caused by: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:391)
at java.net.Socket.connect(Socket.java:579)
at java.net.Socket.connect(Socket.java:528)
at java.net.Socket.<init>(Socket.java:425)
at java.net.Socket.<init>(Socket.java:208)
at sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirectSocketFactory.java:40)
at sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(RMIMasterSocketFactory.java:146)
at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:613)
... 23 more
I get this exception when I try to connect a remote client to my server. In both, server and client the hostName for the registryUrl of rmi is the public IP address of the server. I also tried to put localhost in server but the error doesn't change.
My java.policy is set to grant all connections to all ports and I have no firewalls enabled in the server or the client.
Any suggestions what could be?
This is item A.1 in the RMI FAQ. You need to either fix your /etc/hosts file or set the java.rmi.server.hostname property at the server.
PROBLEM SOLVED
I had exactly the same error. When the remote object got binded to the rmiregistry it was attached with the loopback IP Address which will obviously fail if you try to invoke a method from a remote address. In order to fix this we need to set the java.rmi.server.hostname property to the IP address where other devices can reach your rmiregistry over the network. It doesn't work when you try to set the parameter through the JVM. It worked for me just by adding the following line to my code just before binding the object to the rmiregistry:
System.setProperty("java.rmi.server.hostname","192.168.1.2");
In this case the IP address on the local network of the PC binding the remote object on the RMI Registry is 192.168.1.2.
you can use LocateRegistry such as:
Registry rgsty = LocateRegistry.createRegistry(1888);
rgsty.rebind("hello", hello);
I found many of the Q&A on this topic, not nothing was helping me - that's because my issue was more basic ( what can I say I am not a networking guru :) ). My ip address in /etc/hosts was incorrect. What I had tried included the following for CATALINA_OPTS:
CATALINA_OPTS="$CATALINA_OPTS -Djava.awt.headless=true -Xmx128M -server
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=7091
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
-Djava.rmi.server.hostname=A.B.C.D" #howeverI put the wrong ip here!
export CATALINA_OPTS
My problem was that I had changed my ip address many months ago, but never updated my /etc/hosts file. it seems that by default the jconsole uses the hostname -i ip address in some fashion even though I was viewing local processes. The best solution was to simply change the /etc/hosts file.
The other solution which can work is to get your correct ip address from /sbin/ifconfig and use that ip address when specifying the ip address in, for example, a catalina.sh script:
-Djava.rmi.server.hostname=A.B.C.D
If you're running in a Linux environment, open the file /etc/hosts.allow
add the following line
ALL
Wildcards
Also check the /etc/hostname and /etc/host to see if there might be something wrong there.
I had to change my / etc / host from
127.0.0.1 localhost
127.0.1.1 AMK
to
127.0.0.1 localhost
127.0.0.1 AMK
also wrote in ALL in the file /etc/hosts.allow which was previously completely empty
Now everything works
do not know how safe it is. you have to read more about possible options for /etc/hosts.allow to do something that requires a touch of security.
Maybe your rmiregistry not be created before client trying connect to your server and it would lead to this exception.In Linux, you can use "netstat" to check your rmiregistry be bond on the right port you assigned in java code.
If you've tried modifying etc/hosts and adding java.rmi.server.hostname property as well but still registry is being bind to 127.0.0.1
the issue for me was resolved after explicitly setting System property through code though the same property wasn't picked from jvm args
It works for me after getting rid of "::1" in /etc/hosts.
I had the same exact problem and my issue was that I had 2 IP addresses from 2 different networks configured in the etc/hosts as below.
10.xxx.x.xxx localhost
192.xxx.x.xxx localhost
This should be because there was a conflict as to which IP to be used for the other devices to reach the rmiregistry over the network.
Once I removed the extra-record that is not required, I was able to solve the issue.
So my etc/hosts file had only the following record.
10.xxx.x.xxx localhost
In my case I was unable to edit the hosts file because using a pc from the university.
I fixed the problem running rmiregistry in another port (instead of 1099) with:
rmiregistry <port>
and then running the server on that port.
It was basically an error caused by occupied port.
when you want to connect to remote server with RMI you must add a system property same as:
System.setProperty("java.rmi.server.hostname","Ip or DNS of the server");
or add environment variable.
For me I got Connection Refused and solve it by adding this line of code in server side:
java -jar -Djava.rmi.server.hostname="ip or dns of the server" packageName.jar
Thank to other guy for guide me to solve it.
On Windows make sure your Windows firewall is correctly configure / disabled. I had to disable the Windows firewall (because I didn't bother with configuring it) to get things to work even when I was testing with localhost.
When I got the same error on my machine ("connection is refused"), the reason was that I had defined the following on the server side:
Naming.rebind("rmi://localhost:8080/AddService"
,addService);
Thus the server binds both the IP = 127.0.0.1 and the port 8080.
But on the client side I had used:
AddServerInterface st = (AddServerInterface)Naming.lookup("rmi://localhost"
+"/AddService");
Thus I forgot to add the port number after the localhost, so I rewrote the above command and added the port number 8080 as follows:
AddServerInterface st = (AddServerInterface)Naming.lookup("rmi://localhost:8080"
+"/AddService");
and everything worked fine.
You can simply use:
on server side:
Registry <objectName1> = LocateRegistry.createRegistry(1099);
Registry <objectName2> = LocateRegistry.getRegistry();
on Client Side:
Registry <object name you want> = LocateRegistry.getRegistry();
I am trying to workout a JMX Java client for Zookeeper instance for a custom monitoring web app. As provided in document, Zookeeper provides various statistics through JMX MBeans.
For the excercise, I am running Zookeeper intance locally in standalone mode on Windows 7 Enterprise using following arguments:-
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.local.only=false
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.port=10010
-Dzookeeper.jmx.log4j.disable=false
After running my zookeeper intance, I am able to connect to JMX beans using JConsole that correctly shows all the statistics :-
PROBLEM
While trying to connect using my own code I am getting java.net.ConnectException: Connection refused: connect error. Code that I am trying :-
public static void main(String[] args) throws Exception {
// service:jmx:rmi:///jndi/rmi://#{host}:#{port}/jmxrmi
JMXServiceURL url = new JMXServiceURL("service:jmx:rmi:///jndi/rmi://localhost:10010/jmxrmi");
// This throws java.net.ConnectException !!!
JMXConnector jmxConnector = JMXConnectorFactory.connect(url);
MBeanServerConnection mbeanServerConnection = jmxConnector.getMBeanServerConnection();
ObjectName mbeanName = new ObjectName("org.apache.ZooKeeperService:name0=StandaloneServer_port2181");
ZooKeeperServerMXBean newProxyInstance = MBeanServerInvocationHandler.newProxyInstance(mbeanServerConnection,
mbeanName, ZooKeeperServerMXBean.class, true);
System.out.println("Created zoo mbean proxy");
System.out.println(newProxyInstance.getAvgRequestLatency());
}
Facing same problem while trying to connect using Java Visual VM.
What is the correct way to connect to Zookeeper MBean using Java code ?
UPDATE 1
There is 4 years old unresolved JIRA ticket that seems to be saying that there are two kind of ports that comes into play - jmx port & rmi port. The rmi port is generated randomly & I guess that is what needed while creating connection.
But then how JConsole is able to connect ?
UPDATE 2
This blog says that talking to remote JMX server over RMI protocol might be problem and suggests using JMXMP (JMX-Messaging Protocol) instead.
Now how do I exactly do I do that ?
I am unable to connect VisualVM to a remote JVM. I have started the remote JVM with the following parameters:
java -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.port=9000 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Djava.rmi.server.hostname=[server ip] -Dlog4j.configurationFile=file:///home/ubuntu/servicesLog4j2.xml -jar /home/ubuntu/Program.jar &
Note the -Dcom.sun.management.jmxremote.ssl=false
I then start VisualVM and add a new JMX connection to the server. I specifically check the option:
Do not require SLL connection
When the connection fails, I can see the following error in the VisualVM log. I am unsure why SSL is involved if I have disabled SSL with a flag on the remote VM and specified not to require SSL in VisualVM. But it would appear this error is preventing the connection. What am I missing?
java.io.EOFException: SSL peer shut down incorrectly
at sun.security.ssl.InputRecord.read(InputRecord.java:505)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:954)
Caused: javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973)
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1343)
at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:728)
at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:123)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
at java.io.DataOutputStream.flush(DataOutputStream.java:123)
at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:229)
Caused: java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is:
javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake
at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:304)
at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202)
at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:342)
at sun.rmi.registry.RegistryImpl_Stub.lookup(Unknown Source)
at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:118)
Caused: javax.naming.CommunicationException [Root exception is java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is:
javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake]
at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:122)
at com.sun.jndi.toolkit.url.GenericURLContext.lookup(GenericURLContext.java:205)
at javax.naming.InitialContext.lookup(InitialContext.java:417)
at javax.management.remote.rmi.RMIConnector.findRMIServerJNDI(RMIConnector.java:1929)
at javax.management.remote.rmi.RMIConnector.findRMIServer(RMIConnector.java:1896)
at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:287)
Caused: java.io.IOException: Failed to retrieve RMIServer stub
at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:369)
at com.sun.tools.visualvm.jmx.impl.JmxModelImpl$ProxyClient.tryConnect(JmxModelImpl.java:569)
[catch] at com.sun.tools.visualvm.jmx.impl.JmxModelImpl$ProxyClient.connect(JmxModelImpl.java:506)
at com.sun.tools.visualvm.jmx.impl.JmxModelImpl.connect(JmxModelImpl.java:234)
at com.sun.tools.visualvm.jmx.impl.JmxModelImpl.<init>(JmxModelImpl.java:223)
at com.sun.tools.visualvm.jmx.impl.JmxModelProvider.createModelFor(JmxModelProvider.java:60)
at com.sun.tools.visualvm.jmx.impl.JmxModelProvider.createModelFor(JmxModelProvider.java:41)
at com.sun.tools.visualvm.core.model.ModelFactory.getModel(ModelFactory.java:111)
at com.sun.tools.visualvm.tools.jmx.JmxModelFactory.getJmxModelFor(JmxModelFactory.java:69)
at com.sun.tools.visualvm.jmx.impl.JmxApplicationProvider.addJmxApplication(JmxApplicationProvider.java:295)
at com.sun.tools.visualvm.jmx.impl.JmxApplicationProvider.createJmxApplication(JmxApplicationProvider.java:200)
at com.sun.tools.visualvm.jmx.JmxApplicationsSupport.createJmxApplicationImpl(JmxApplicationsSupport.java:319)
at com.sun.tools.visualvm.jmx.JmxApplicationsSupport.createJmxApplicationInteractive(JmxApplicationsSupport.java:296)
at com.sun.tools.visualvm.jmx.impl.AddJMXConnectionAction$1.run(AddJMXConnectionAction.java:80)
at org.openide.util.RequestProcessor$Task.run(RequestProcessor.java:1423)
at org.openide.util.RequestProcessor$Processor.run(RequestProcessor.java:2033)
How I managed to solve this after some headaches:
If you have a firewall at the remote machine, you might have issues with the second random port (where remote objects are exported). Try to disable the firewall and if it works then you might have to do something like this. Tomcat have a listener to solve it: link
Make sure that the value under -Djava.rmi.server.hostname resolves to the host from your client location (that was my case)
And finally jconsole (at the JDK bin folder, the same than VisualVM) provides very useful information. Execute it from a console with the -debug option. With the debug option, Jconsole will pop up stacktraces explaining the real reason why your client can't connect. When you try to connect to nonSSL servers JConsole will initially fail and then it will ask you to connect "insecure", next you'll see the real reason, in my case it was looking for the object export port at the loopback IP 127.0.0.1 (because I wasn't using the -Djava.rmi.server.hostname option).
Hope it helps!