Spring RmiExporter service fails after successful startup - java

I'm experiencing a situation where an RMI Service configured using Spring's RmiServiceExporter is starting up just fine, and is usable for a while - then after some unknown amount of time, the service is unavailable even though the java process for that Spring context is still running.
At the moment the work around is to reboot the java process, but this is hardly acceptable in a production environment. I cannot figure out, or even begin to guess, why this might be happening, or what to look for that might be going wrong. No S/O or Google search has been useful, because all I find are examples of RmiServiceExporter failing to start at all, not starting ok and then failing later on. Any clue at all would be useful.
Output of lsof | head -1;lsof | grep 1197, before the service fails:
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
java 6882 ubuntu 176u IPv6 54677985 0t0 TCP *:1197 (LISTEN)
Service-side Spring config:
<bean class="org.springframework.remoting.rmi.RmiServiceExporter">
<property name="serviceName" value="MyRmiService" />
<property name="service" ref="myService" />
<property name="serviceInterface" value="package.MyService" />
<property name="registryPort" value="${my.rmi.port}" />
</bean>
Client-side Spring config:
<bean id="remoteService" class="org.springframework.remoting.rmi.RmiProxyFactoryBean">
<property name="serviceUrl" value="rmi://${my.rmi.host}:${my.rmi.port}/MyRmiService"/>
<property name="serviceInterface" value="package.MyService"/>
<property name="lookupStubOnStartup" value="false"/>
<property name="refreshStubOnConnectFailure" value="true"/>
</bean>
Client-side stacktrace, after the service fails (lsof reports nothing):
Exception in thread "main" org.springframework.remoting.RemoteLookupFailureException: Lookup of RMI stub failed; nested exception is java.rmi.ConnectException: Connection refused to host: rmiservice.host; nested exception is:
java.net.ConnectException: Connection refused
at org.springframework.remoting.rmi.RmiClientInterceptor.lookupStub(RmiClientInterceptor.java:215)
at org.springframework.remoting.rmi.RmiClientInterceptor.getStub(RmiClientInterceptor.java:237)
at org.springframework.remoting.rmi.RmiClientInterceptor.invoke(RmiClientInterceptor.java:257)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
at $Proxy61.serviceMethod(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.springframework.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:191)
at package.Client.main(Client.java:47)
Caused by: java.rmi.ConnectException: Connection refused to host: rmiservice.host; nested exception is:
java.net.ConnectException: Connection refused
at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:601)
at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:198)
at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:184)
at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:322)
at sun.rmi.registry.RegistryImpl_Stub.lookup(Unknown Source)
at java.rmi.Naming.lookup(Naming.java:84)
at org.springframework.remoting.rmi.RmiClientInterceptor.lookupStub(RmiClientInterceptor.java:200)
... 11 more
Caused by: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)
at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
at java.net.Socket.connect(Socket.java:529)
at java.net.Socket.connect(Socket.java:478)
at java.net.Socket.<init>(Socket.java:375)
at java.net.Socket.<init>(Socket.java:189)
at sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirectSocketFactory.java:22)
at sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(RMIMasterSocketFactory.java:128)
at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:595)
... 17 more

Which web server you are using ?
If it is Apache Tomcat, then add to CATALINA_OPTS variable
-Djava.rmi.server.hostname=10.0.34.11
in catalina.sh file and then export CATALINA_OPTS variable.

Related

MyBatis hits Cannot Assign Requested Address Socket Exception at high volume with Pooled connection

I am using MyBatis to pull data from database into a cache as needed. At high volumes of message traffic, I start to see the following exception:
### Error querying database. Cause: org.postgresql.util.PSQLException: Connection to <my-database>:<my-port> refused. Check that the hostname and port are correct and that the postmaster is
accepting TCP/IP connections.
### The error may exist in mappers/<my-mapper>.xml
### The error may involve <my-mapper>.getById
### The error occurred while executing a query
### Cause: org.postgresql.util.PSQLException: Connection to <my-database>:<my-port> refused. Check that
the hostname and port are correct and that the postmaster is accepting TCP/IP connections.
at org.apache.ibatis.exceptions.ExceptionFactory.wrapException(ExceptionFactory.java:30)
at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:122)
at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:113)
at org.apache.ibatis.session.defaults.DefaultSqlSession.selectOne(DefaultSqlSession.java:73)
<truncated>
Caused by: org.postgresql.util.PSQLException: Connection to <my-database>:<my-port> refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.
at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:262)
at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:52)
at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:216)
at org.postgresql.Driver.makeConnection(Driver.java:404)
at org.postgresql.Driver.connect(Driver.java:272)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:208)
at org.apache.ibatis.datasource.unpooled.UnpooledDataSource.doGetConnection(UnpooledDataSource.java:201)
at org.apache.ibatis.datasource.unpooled.UnpooledDataSource.doGetConnection(UnpooledDataSource.java:196)
at org.apache.ibatis.datasource.unpooled.UnpooledDataSource.getConnection(UnpooledDataSource.java:93)
at org.apache.ibatis.datasource.pooled.PooledDataSource.popConnection(PooledDataSource.java:385)
at org.apache.ibatis.datasource.pooled.PooledDataSource.getConnection(PooledDataSource.java:89)
at org.apache.ibatis.transaction.jdbc.JdbcTransaction.openConnection(JdbcTransaction.java:140)
at org.apache.ibatis.transaction.jdbc.JdbcTransaction.getConnection(JdbcTransaction.java:62)
at org.apache.ibatis.executor.BaseExecutor.getConnection(BaseExecutor.java:315)
at org.apache.ibatis.executor.SimpleExecutor.prepareStatement(SimpleExecutor.java:75)
at org.apache.ibatis.executor.SimpleExecutor.doQuery(SimpleExecutor.java:61)
at org.apache.ibatis.executor.BaseExecutor.queryFromDatabase(BaseExecutor.java:303)
at org.apache.ibatis.executor.BaseExecutor.query(BaseExecutor.java:154)
at org.apache.ibatis.executor.CachingExecutor.query(CachingExecutor.java:102)
at org.apache.ibatis.executor.CachingExecutor.query(CachingExecutor.java:82)
at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:120)
... 16 more
Caused by: java.net.ConnectException: Cannot assign requested address (connect failed)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.postgresql.core.PGStream.<init>(PGStream.java:61)
at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:144)
... 37 more
I do not see these exceptions with small amounts of traffic, only at scale, so it doesn't seem to be any sort of functional problem or syntax issue with the mapper. In my config file for mybatis, this is how my source is configured:
<environments default = "ingestion_environment">
<environment id = "ingestion_environment">
<transactionManager type = "JDBC"/>
<dataSource type = "POOLED">
<property name = "driver" value = "org.postgresql.Driver"/>
<property name="url" value="jdbc:postgresql://<my-database>:<my-port>/<my-user>" />
<property name="username" value="<username>" />
<property name="password" value="<password>" />
<property name="poolMaximumCheckoutTime" value="50000"/>
<property name="poolMaximumActiveConnections" value="10"/>
</dataSource>
</environment>
</environments>
I'm trying to figure out what about high traffic could lead to these exceptions--with a pooled connection, it seems like the number of connections really shouldn't be changing even when there are more messages to handle. What could cause this?

Not sending scheduled reports by email with JasperReports Server

I'm dealing with an issue with JasperReports Server, i have it running in CentOS, i'm trying to schedule and send reports by email but still no success.
I just configured the js.quartz.properties file as follows:
report.scheduler.mail.sender.host=smtp.gmail.com
report.scheduler.mail.sender.username=myemailaddress#gmail.com
report.scheduler.mail.sender.password=Mypassword
report.scheduler.mail.sender.from=myemailaddress#gmail.com
report.scheduler.mail.sender.protocol=smtps
report.scheduler.mail.sender.port=465
report.scheduler.web.deployment.uri=http://etcetcetc.com:8080/jasperserver
also my applicationContext-report-scheduling.xml file:
<bean id="reportSchedulerMailSender" class="org.springframework.mail.javamail.JavaMailSenderImpl">
<property name="host" value="${report.scheduler.mail.sender.host}"/>
<property name="username" value="${report.scheduler.mail.sender.username}"/>
<property name="password" value="${report.scheduler.mail.sender.password}"/>
<property name="protocol" value="${report.scheduler.mail.sender.protocol}"/>
<property name="port" value="${report.scheduler.mail.sender.port}"/>
<property name="javaMailProperties">
<props>
<prop key="mail.smtps.auth">true</prop>
<prop key="mail.smtps.starttls.enable">true</prop>
</props>
</property>
</bean>
I tried with ports 25, 465 and 587, disabling firewall, using smtp protocol instead of smtps and still i'm not able to accomplish this task, below you can see the error i'm gettins, i'll appreciate your help here.
No se completó la notificación de error de trabajo. Se produjo un error al enviarlo.
org.springframework.mail.MailSendException: Mail server connection failed; nested exception is javax.mail.MessagingException: Could not connect to SMTP host: smtp.gmail.com, port: 465;
nested exception is:
java.net.ConnectException: Connection refused. Failed messages: javax.mail.MessagingException: Could not connect to SMTP host: smtp.gmail.com, port: 465;
nested exception is:
java.net.ConnectException: Connection refused; message exception details (1) are:
Failed message 1:
javax.mail.MessagingException: Could not connect to SMTP host: smtp.gmail.com, port: 465;
nested exception is:
java.net.ConnectException: Connection refused
at com.sun.mail.smtp.SMTPTransport.openServer(SMTPTransport.java:1282)
at com.sun.mail.smtp.SMTPTransport.protocolConnect(SMTPTransport.java:370)
at javax.mail.Service.connect(Service.java:275)
at org.springframework.mail.javamail.JavaMailSenderImpl.doSend(JavaMailSenderImpl.java:389)
at org.springframework.mail.javamail.JavaMailSenderImpl.send(JavaMailSenderImpl.java:340)
at org.springframework.mail.javamail.JavaMailSenderImpl.send(JavaMailSenderImpl.java:336)
at com.jaspersoft.jasperserver.api.engine.scheduling.quartz.ReportExecutionJobMailNotificationImpl.sendMailNotification(ReportExecutionJobMailNotificationImpl.java:123)
at com.jaspersoft.jasperserver.api.engine.scheduling.quartz.ReportExecutionJob.sendMailNotification(ReportExecutionJob.java:942)
at com.jaspersoft.jasperserver.api.engine.scheduling.quartz.ReportExecutionJob.executeAndSendReport(ReportExecutionJob.java:538)
at com.jaspersoft.jasperserver.api.engine.scheduling.quartz.ReportExecutionJob.execute(ReportExecutionJob.java:219)
at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557)
Caused by: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
at java.net.Socket.connect(Socket.java:525)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:550)
at com.sun.net.ssl.internal.ssl.BaseSSLSocketImpl.connect(BaseSSLSocketImpl.java:141)
at com.sun.mail.util.SocketFetcher.createSocket(SocketFetcher.java:232)
at com.sun.mail.util.SocketFetcher.getSocket(SocketFetcher.java:189)
at com.sun.mail.smtp.SMTPTransport.openServer(SMTPTransport.java:1250)

Spring and Tomcat: Access to DialectResolutionInfo cannot be null when 'hibernate.dialect' not set

I have an application that uses the Spring Repository framework and a MySQL database. I currently have it deployed on Tomcat 7.
Once I deploy and attempt to log into the application (the first call to the database) I receive the following exception:
SEVERE: Exception sending context initialized event to listener instance of class org.springframework.web.context.ContextLoaderListener
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'entityManagerFactory' defined in URL [file:/opt/apache-tomcat-7.0.62/webapps/waypoint/WEB-INF/classes/repository-context.xml]: Invocation of init method failed; nested exception is org.hibernate.HibernateException: Access to DialectResolutionInfo cannot be null when 'hibernate.dialect' not set
I have enabled query logging on the database and can confirm that a query is being made to the database for a user object. However, on the login page itself, I see the following message:
Your login attempt was not successful, try again.
Reason: could not extract ResultSet; SQL [n/a]; nested exception is org.hibernate.exception.SQLGrammarException: could not extract ResultSet
To make things a bit more confusing, this only happens when I try deploying on a remote server. When I start a local instance of MySQL and run an instance of Tomcat on localhost, everything works as expected.
Below is my repository-context.xml:
<beans:bean id="dataSource" class="org.apache.commons.dbcp.BasicDataSource">
<beans:property name="driverClassName" value="com.mysql.jdbc.Driver" />
<beans:property name="url"
value="jdbc:mysql://localhost:3306/waypoint_dev" />
<beans:property name="username" value="root" />
</beans:bean>
<beans:bean id="entityManagerFactory"
class="org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean">
<beans:property name="dataSource" ref="dataSource" />
<beans:property name="jpaVendorAdapter">
<beans:bean
class="org.springframework.orm.jpa.vendor.HibernateJpaVendorAdapter" />
</beans:property>
<!-- spring based scanning for entity classes> -->
<beans:property name="packagesToScan"
value="com.cigna.waypoint.repository, com.cigna.waypoint.core" />
</beans:bean>
<beans:bean id="transactionManager"
class="org.springframework.orm.jpa.JpaTransactionManager">
<beans:property name="entityManagerFactory" ref="entityManagerFactory" />
</beans:bean>
Does anyone have any ideas what could be causing the problem? Is there a configuration that is needed with Tomcat that I am not aware of, or have forgotten?
Thanks for your help.
=== UPDATE 1 ====
After specifying the dialect, I am receiving the following stack trace:
Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
The last packet successfully received from the server was 161,752 milliseconds ago. The last packet sent successfully to the server was 0 milliseconds ago.
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[na:1.8.0_45]
at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source) ~[na:1.8.0_45]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source) ~[na:1.8.0_45]
at java.lang.reflect.Constructor.newInstance(Unknown Source) ~[na:1.8.0_45]
at com.mysql.jdbc.Util.handleNewInstance(Util.java:377) ~[mysql-connector-java-5.1.33.jar:5.1.33]
at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1036) ~[mysql-connector-java-5.1.33.jar:5.1.33]
at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3427) ~[mysql-connector-java-5.1.33.jar:5.1.33]
at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3327) ~[mysql-connector-java-5.1.33.jar:5.1.33]
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3814) ~[mysql-connector-java-5.1.33.jar:5.1.33]
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2435) ~[mysql-connector-java-5.1.33.jar:5.1.33]
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2582) ~[mysql-connector-java-5.1.33.jar:5.1.33]
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2530) ~[mysql-connector-java-5.1.33.jar:5.1.33]
at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1907) ~[mysql-connector-java-5.1.33.jar:5.1.33]
at com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:2030) ~[mysql-connector-java-5.1.33.jar:5.1.33]
at org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:96) ~[commons-dbcp-1.4.jar:1.4]
at org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:96) ~[commons-dbcp-1.4.jar:1.4]
at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.extract(ResultSetReturnImpl.java:82) ~[hibernate-core-4.3.6.Final.jar:4.3.6.Final]
... 80 common frames omitted
Caused by: java.io.EOFException: Can not read response from server. Expected to read 4 bytes, read 0 bytes before connection was unexpectedly lost.
at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:2914) ~[mysql-connector-java-5.1.33.jar:5.1.33]
at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3337) ~[mysql-connector-java-5.1.33.jar:5.1.33]
... 90 common frames omitted
20:48:32.455 [http-bio-8080-exec-34] DEBUG o.s.s.w.a.UsernamePasswordAuthenticationFilter - Authentication request failed: org.springframework.security.authentication.InternalAuthenticationServiceException: could not extract ResultSet; nested exception is org.hibernate.exception.JDBCConnectionException: could not extract ResultSet
Hibernate needs to know the dialect so that it can connect to the underlying database. You need to set the dialect as org.hibernate.dialect.MySQL5Dialect for MySQL along with the Datasource definition.
Here is a link which says why you need dialect.
I was able to resolve this issue by working through the steps listed in #Soheil 's answer found here:
Solving a "communications link failure" with JDBC and MySQL
It turns out that all I needed was add the hibernate dialect, as #Dhanush Gopinath suggested, and add the following to my my.cnf file:
interactive_timeout=180
wait_timeout=180
connect_timeout=180
max_connect_errors=9999
skip-name-resolve

Spring: Cannot connect to a JMX Server using RMI from behind a firewall

My Spring application is running on a machine that is behind a NAT firewall (pfSense). The machine's internal IP is a.b.c.d, and the NAT IP is w.x.y.z
The Spring configuration's serviceUrl is set to my internal IP (a.b.c.d) on port 1100, and when I start the application, I provide the following switches:
-Dcom.sun.management.jmxremote -Djava.rmi.server.hostname=w.x.y.z -Dcom.sun.management.jmxremote.port=1099 -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false
As shown above, I set -Djava.rmi.server.hostname=w.x.y.z in order to make it possible to connect to the application through the NAT. I have also opened up the relevant ports on the machine's firewall, and set up a port forward from w.x.y.z:1100 to a.b.c.d:1100
When I then try to connect to the app from outside the network using jconsole on w.x.y.z:1100, I get a java.io.IOException: jmxrmi
java.io.IOException: jmxrmi
at sun.tools.jconsole.ProxyClient.checkSslConfig(ProxyClient.java:236)
at sun.tools.jconsole.ProxyClient.<init>(ProxyClient.java:127)
at sun.tools.jconsole.ProxyClient.getProxyClient(ProxyClient.java:483)
at sun.tools.jconsole.JConsole$3.run(JConsole.java:524)
Caused by: java.rmi.NotBoundException: jmxrmi
at sun.rmi.registry.RegistryImpl.lookup(RegistryImpl.java:136)
at sun.rmi.registry.RegistryImpl_Skel.dispatch(Unknown Source)
at sun.rmi.server.UnicastServerRef.oldDispatch(UnicastServerRef.java:409)
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:267)
at sun.rmi.transport.Transport$1.run(Transport.java:177)
at sun.rmi.transport.Transport$1.run(Transport.java:174)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:556)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:811)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:670)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
at sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:275)
at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:252)
at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:378)
at sun.rmi.registry.RegistryImpl_Stub.lookup(Unknown Source)
at sun.tools.jconsole.ProxyClient.checkSslConfig(ProxyClient.java:234)
If I try to connect using jconsole on w.x.y.z:1099, I get a java.rmi.ConnectException (Connection refused). How can I expose my JMX MBeans outside my NATted firewall?
java.rmi.ConnectException: Connection refused to host: w.x.y.z; nested exception is:
java.net.ConnectException: Operation timed out
at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:619)
at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:216)
at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202)
at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:129)
at javax.management.remote.rmi.RMIServerImpl_Stub.newClient(Unknown Source)
at javax.management.remote.rmi.RMIConnector.getConnection(RMIConnector.java:2373)
at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:297)
at sun.tools.jconsole.ProxyClient.tryConnect(ProxyClient.java:355)
at sun.tools.jconsole.ProxyClient.connect(ProxyClient.java:313)
at sun.tools.jconsole.VMPanel$2.run(VMPanel.java:292)
Caused by: java.net.ConnectException: Operation timed out
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at java.net.Socket.connect(Socket.java:528)
at java.net.Socket.<init>(Socket.java:425)
at java.net.Socket.<init>(Socket.java:208)
at sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirectSocketFactory.java:40)
at sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(RMIMasterSocketFactory.java:147)
at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:613)
... 9 more
Interestingly enough, I can connect to both ports (1099 and 1100) on w.x.y.z using telnet.
The relevant portion of my Spring configuration is:
<bean id="registry" class="org.springframework.remoting.rmi.RmiRegistryFactoryBean">
<property name="port" value="1100" />
</bean>
<bean id="serverConnector" class="org.springframework.jmx.support.ConnectorServerFactoryBean" depends-on="registry">
<property name="server" ref="mbeanServer" />
<property name="objectName" value="connector:name=rmi" />
<property name="serviceUrl" value="service:jmx:rmi://a.b.c.d:1100/jndi/rmi://a.b.c.d:1100/server" />
<property name="environment">
<props>
<prop key="jmx.remote.jndi.rebind">true</prop>
</props>
</property>
</bean>
<bean id="mbeanServer" class="org.springframework.jmx.support.MBeanServerFactoryBean">
<!-- indicate to first look for a server -->
<property name="locateExistingServerIfPossible" value="true"/>
</bean>
Note: I am not using any servlet containers.
EDIT: The first answer provided me with the basis for the solution to my specific problem, which can be seen in the second answer.
The initial port you define with com.sun.management.jmxremote.port is called a registry port and is only used to start negotiation and determine next port(s) to use for "real" communication. Java RMI mechanism uses dynamically allocated ports and in general is not compatible with firewalls.
What port is used by Java RMI connection?
That said, for JMX it is possible to work around it
a) Use system properties to lock both ports (requires Java 7)
com.sun.management.jmxremote.port
com.sun.management.jmxremote.rmi.port
b) Use custom code to request a specific port. See
JConsole over ssh local port forwarding
See also:
Why Java opens 3 ports when JMX is configured?
http://realjenius.com/2012/11/21/java7-jmx-tunneling-freedom/
Elaborating on the solution I identified using the links and pointers anttix gave me in his answer.
So, as mentioned in the answer above, using the system properties com.sun.management.jmxremote.port and com.sun.management.jmxremote.rmi.port is the basis for the solution.
I changed my Spring config to remove the registry and serverConnector beans, as this was not allowing me to register port 1099 (PermissionExceptions resulted in my being unable to bind to port 1099), and used the platform mbean server.
The Spring config I used was just this:
<bean id="mbeanServer" class="java.lang.management.ManagementFactory" factory-method="getPlatformMBeanServer"/>
Then, I started the application using the following switches:
-Dcom.sun.management.jmxremote -Djava.rmi.server.hostname=w.x.y.z -Dcom.sun.management.jmxremote.port=1099 -Dcom.sun.management.jmxremote.rmi.port=1099 -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false
Setting the java.rmi.server.hostname to the NAT VIP w.x.y.z is critical as we need to be able to execute remote operations on locally created remote objects from outwith the network.
Note: This solution will only work on Java 7u4 or higher.

How to stop tomcat from making failed attempts to connect to local memcached servers?

I have deployed my web app inside a tomcat container but due to a possible connection leak , the web app is constantly making failed attempts to connect to the local memcached server listening at port 11211 and 11212. I am using the spy-memcached client.
I have a ContextListener defined which basically shuts down all active memcached client connections.
However when I un-deploy my web app , it appears to me that tomcat is still trying to continue with failed attempts to connect to the memcached server, which it should not.I have checked the active tcp connections on memcached server using netstat but I could not find any entry.
I have also restarted the tomcat server but to no avail as such.
How should I restrict tomcat from making these connections?
2011-11-13 21:21:34.575 INFO net.spy.memcached.MemcachedConnection: Reconnecting due to failure to connect to {QA sa=localhost/127.0.0.1:11212, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0}
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
at net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:407)
at net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:275)
at net.spy.memcached.MemcachedClient.run(MemcachedClient.java:2030)
2011-11-13 21:21:34.576 WARN net.spy.memcached.MemcachedConnection: Closing, and reopening {QA sa=localhost/127.0.0.1:11212, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0}, attempt 32.
I was facing the same problem.Setting daemon true works for me. I am using spymecached-2.8.4 I get the Memcached Client through net.spy.memcached.spring.MemcachedClientFactoryBean though Spring (spring - 3.1.1), here is my spring configuration that I use in my web applicaton :
<bean id="memcachedClient" class="net.spy.memcached.spring.MemcachedClientFactoryBean">
<property name="servers" value="localhost:11211"/>
<property name="protocol" value="BINARY"/>
<property name="transcoder">
<bean class="net.spy.memcached.transcoders.SerializingTranscoder">
<property name="compressionThreshold" value="1024"/>
</bean>
</property>
<property name="opTimeout" value="1000"/>
<property name="timeoutExceptionThreshold" value="1998"/>
<property name="hashAlg">
<value type="net.spy.memcached.DefaultHashAlgorithm">KETAMA_HASH</value>
</property>
<property name="locatorType" value="CONSISTENT"/>
<property name="failureMode" value="Redistribute"/>
<property name="useNagleAlgorithm" value="false"/>
<property name="daemon" value="true"/>
</bean>

Categories