ServiceLocatorImpl shutdown issue in Jersey 2.34 - java

We've been getting errors occasionally from our test and production environments where ServiceLocatorImpl has shut down which causes it to fail to create stateless EJBs from any #Schedule methods.
javax.ejb.EJBException: javax.ejb.EJBException: javax.ejb.CreateException: Could not create stateless EJB
at com.sun.ejb.containers.StatelessSessionContainer._getContext(StatelessSessionContainer.java:399)
at com.sun.ejb.containers.BaseContainer.getContext(BaseContainer.java:2607)
at com.sun.ejb.containers.BaseContainer.preInvoke(BaseContainer.java:2024)
at com.sun.ejb.containers.EJBLocalObjectInvocationHandler.invoke(EJBLocalObjectInvocationHandler.java:210)
at com.sun.ejb.containers.EJBLocalObjectInvocationHandlerDelegate.invoke(EJBLocalObjectInvocationHandlerDelegate.java:90)
...
Caused by: java.lang.IllegalStateException: ServiceLocatorImpl(__HK2_Generated_9,12,566324949) has been shut down
at org.jvnet.hk2.internal.ServiceLocatorImpl.checkState(ServiceLocatorImpl.java:2384)
at org.jvnet.hk2.internal.ServiceLocatorImpl.inject(ServiceLocatorImpl.java:997)
at org.glassfish.jersey.inject.hk2.AbstractHk2InjectionManager.inject(AbstractHk2InjectionManager.java:207)
at org.glassfish.jersey.inject.hk2.ImmediateHk2InjectionManager.inject(ImmediateHk2InjectionManager.java:30)
at org.glassfish.jersey.ext.cdi1x.internal.CdiComponentProvider$InjectionManagerInjectedCdiTarget.inject(CdiComponentProvider.java:668)
at org.glassfish.weld.services.JCDIServiceImpl$JCDIInjectionContextImpl.inject(JCDIServiceImpl.java:581)
at org.glassfish.weld.services.JCDIServiceImpl.injectEJBInstance(JCDIServiceImpl.java:328)
at com.sun.ejb.containers.BaseContainer.injectEjbInstance(BaseContainer.java:1818)
at com.sun.ejb.containers.StatelessSessionContainer.createStatelessEJB(StatelessSessionContainer.java:437)
... 88 more
Most of the application continues to work fine and and users can continue using it but background processes that are run from #Schedule all fail until we restart Payara.
We originally ran into this in our test environment after it had been running for multiple days under a fairly high load while we were on Payara 5.2021.4 (uses Jersey 2.30). We found servicelocator impl shutdown issue in jersey 2.22 talking about this issue and it seemed to indicate that Jersey 2.26 fixed the issue.
Since we were already on a later version we dug into the release notes and found that in hk2 2.6.1 there was a fix "Fix for child service locator shutdown" (https://github.com/eclipse-ee4j/glassfish-hk2/blob/master/CHANGELOG) and that Jersey 2.29.1 had updated the hk2 version from 5.1.0 to 6.2.1 (https://github.com/eclipse-ee4j/jersey/pull/4234) and that we had been listing Jersey 2.29 as a dependency. So we updated our versions to Payara 5.2021.8, Jersey 2.34 (hk2 2.6.1) and made our environment restart payara every night. After that we didn't seen the error for a while but since then we've still gotten it a couple times in production.
We haven't been able to reproduce the issue on demand or pin down any other errors that could be causing it. Has anyone else run into this issue on the current Payara and come up with a way to fix it, work around it or mitigate it short of restarting payara whenever it happens?

Related

Wildfly datasource Enable is not working at runtime when configured through CLI

We are migrating from jboss Eap 6.4 to Wildfly. During migration, as part of our application installation, we will configure different xa-datasources in domain.xml. This configuration will be done through jboss-cli.sh.
After configuring the data sources, when deploying the ears, (particularly at *-ejb.jar) we are getting an error saying that the 'persistent unit' is not available and the deployment is rolled back.
Loggers are showing the below:
WFLYCTL0180: Services with missing/unavailable dependencies" => ["jboss.persistenceunit.\"xxyy.ear/xxyy-ejb.jar#xxyyDBUnit\" is missing [jboss.naming.context.java.jboss.datasources.xxyyDS]"]
Post restart of the jboss service, the same deployment works fine. So, we believe that the datasources configured in the domain.xml is not activated or enabled (although we have set the attribute 'enabled=true' for the xa-datasource. We could see the following warning while adding the xa-datasource in jboss.
2021-09-21 16:57:48,308 INFO [org.jboss.as.connector] (Host Controller Service Threads - 8) WFLYJCA0093: The 'enable' operation is deprecated. Use of the 'add' or 'remove' operations is preferred, or if required the 'write-attribute' operation can used to set the deprecated 'enabled' attribute
With this warning, we understood that the enable/disable operation has been removed in the jboss wildfly. Is there anyway we can make it enabled on the runtime without restart of the jboss service.
Below are the commands which we use to create the xa-datasource in wildfly.
/profile=full-ha/subsystem=datasources/xa-data-source=xxyyDS:add(use-java-context=true,use-ccm=false,driver-name=com.mysql,transaction-isolation=TRANSACTION_READ_COMMITTED,min-pool-size=15,max-pool-size=150,pool-prefill=true,flush-strategy=IdleConnections,valid-connection-checker-class-name="org.jboss.jca.adapters.jdbc.extensions.mysql.MySQLValidConnectionChecker",stale-connection-checker-class-name="org.jboss.jca.adapters.jdbc.extensions.mysql.MySQLValidConnectionChecker",exception-sorter-class-name="org.jboss.jca.adapters.jdbc.extensions.mysql.MySQLExceptionSorter",set-tx-query-timeout=true,blocking-timeout-wait-millis=5000,idle-timeout-minutes=1,query-timeout=300,prepared-statements-cache-size=100,share-prepared-statements=true,jndi-name="java:jboss/datasources/xxyyDS",user-name=jboss,password="${VAULT::ds_MySqlDS::password::1}")
/profile=full-ha/subsystem=datasources/xa-data-source=xxyyDS/xa-datasource-properties=DatabaseName:add(value=xxyy_db)
/profile=full-ha/subsystem=datasources/xa-data-source=xxyyDS/xa-datasource-properties=ServerName:add(value=xx-cluster)
/profile=full-ha/subsystem=datasources/xa-data-source=xxyyDS/xa-datasource-properties=AutoReconnectForPools:add(value=true)
/profile=full-ha/subsystem=datasources/xa-data-source=xxyyDS/xa-datasource-properties=AutoReconnect:add(value=true)
/profile=full-ha/subsystem=datasources/xa-data-source=xxyyDS/xa-datasource-properties=useSSL:add(value=false)
/profile=full-ha/subsystem=datasources/xa-data-source=xxyyDS:enable
Any inputs on how to enable the xa-datasource on runtime would help us.
You can enable the datasource via the CLI:
/subsystem=datasources/xa-data-source=xxyyDS:write-attribute(name=enabled,value=true)
It might be required to reload:
:reload
After running through many trial and error method, we found that when adding the datasources by default it will be in enabled state. So no need to do explicitly enable when adding a datasource.
So we removed the below line from the list of CLIs.
/profile=full-ha/subsystem=datasources/xa-data-source=xxyyDS:enable
Post removing the line, the configuration of datasources through CLI and the deployment of ears went fine without issues. Posting this solution as it will be helpful for others who are facing similar kind of issue.
Finally our datasource CLIs will be like the ones below.
/profile=full-ha/subsystem=datasources/xa-data-source=xxyyDS:add(use-java-context=true,use-ccm=false,driver-name=com.mysql,transaction-isolation=TRANSACTION_READ_COMMITTED,min-pool-size=15,max-pool-size=150,pool-prefill=true,flush-strategy=IdleConnections,valid-connection-checker-class-name="org.jboss.jca.adapters.jdbc.extensions.mysql.MySQLValidConnectionChecker",stale-connection-checker-class-name="org.jboss.jca.adapters.jdbc.extensions.mysql.MySQLValidConnectionChecker",exception-sorter-class-name="org.jboss.jca.adapters.jdbc.extensions.mysql.MySQLExceptionSorter",set-tx-query-timeout=true,blocking-timeout-wait-millis=5000,idle-timeout-minutes=1,query-timeout=300,prepared-statements-cache-size=100,share-prepared-statements=true,jndi-name="java:jboss/datasources/xxyyDS",user-name=jboss,password="${VAULT::ds_MySqlDS::password::1}")
/profile=full-ha/subsystem=datasources/xa-data-source=xxyyDS/xa-datasource-properties=DatabaseName:add(value=xxyy_db)
/profile=full-ha/subsystem=datasources/xa-data-source=xxyyDS/xa-datasource-properties=ServerName:add(value=xx-cluster)
/profile=full-ha/subsystem=datasources/xa-data-source=xxyyDS/xa-datasource-properties=AutoReconnectForPools:add(value=true)
/profile=full-ha/subsystem=datasources/xa-data-source=xxyyDS/xa-datasource-properties=AutoReconnect:add(value=true)
/profile=full-ha/subsystem=datasources/xa-data-source=xxyyDS/xa-datasource-properties=useSSL:add(value=false)

Spring Tomcat Startup Issue setCatalinaBase(Ljava/io/File;)

I'm fairly new to spring and spring boot so am not 100 percent sure what i'm doing... Up until a few days ago everything was working fine with my springboot application, but now it throws this error upon starting:
Exception encountered during context initialization - cancelling refresh attempt: org.springframework.context.ApplicationContextException: Unable to start embedded container; nested exception is java.lang.NoSuchMethodError: org.apache.catalina.Server.setCatalinaBase(Ljava/io/File;)V
After some reading around people are saying that the version of tomcat might be outdated, but I'm not really sure where to start
You need to provide us information which version of tomcat and spring boot you are using. I guess you are having problem that your spring boot version is in collision with tomcat version.(One version of spring boot cannot work with all versions of tomcat, I guess that if your application worked before then it must be that you changed, added or deleted some dependency and that is the reason why you app don't work now)

Spring Boot 1.5.1 and "The dependencies of some of the beans in the application context form a cycle"

A couple of weeks ago, I upgraded the version of Spring from to 1.4.x.RELEASE to 1.5.1.RELEASE.
Suddenly it has become a fight to start my Spring Boot service due this error:
"The dependencies of some of the beans in the application context form a cycle"
Same code, different version
It's quite funny, since on my local env (OSX) the same code is usually starting fine, but on Heroku is failing randomly on different deploys (looks like a different order on classpath Class resolution, due OS or due different JDK implementations)
I have even refactored the code to use field injection instead of constructor one everywhere without success.
Any suggestion? why this Spring version has suddenly started to produce these errors?
Thanks!
Find out which bean(s) is the reason for cycle from logs and use #Lazy annotation with #Autowired for this bean(s) in your code. For most cases this should do the trick.

EclipseLink fix not available in Eclipse - Jersey POST/PUT not working

I am using Eclipse Luna (versions 4.4.2) and Glassfish 4 to build a REST web-app using JAX-RS.
All POST and PUT operations, which were previously working fine, are now throwing this error on the webserver:
Warning: StandardWrapperValve[Jersey Web Application]: Servlet.service() for servlet Jersey Web Application threw exception
java.lang.ClassNotFoundException: javax.xml.parsers.ParserConfigurationException not found by org.eclipse.persistence.moxy
I have found out (in the comments here: https://java.net/jira/browse/JERSEY-2888) that it has been fixed in EclipseLink 2.6.1 and Jersey 2.19.
I have updated to Jersey 2.22.1.
Eclipselink 2.6.1 has been released on Oct 15th 2015, as you can see here: http://www.eclipse.org/eclipselink/releases/
However, as you can see here, it doesn't seem to have been incorporated into eclipse for "help -> update software": http://download.eclipse.org/rt/eclipselink/updates/
Does anybody know how to get 2.6.1 working with Eclipse? At the moment, it is preventing me from completing my application, and this problem I've had for 4 weeks.
Or is there another way to fix it?
All help much appreciated...
File a bug report at https://bugs.eclipse.org/bugs/enter_bug.cgi?product=EclipseLink indicating that it's neither in the main repository nor downloadable as a P2 repository on its own.

Glassfish 3.1.2.2 starts my web application twice

I have a strange behavior of Glassfish 3.1.2.2 (build 5) running on OpenIndiana 151.a3
Using oracle jdk 1.7.0_5.
When i deploy my web application which has a spring web dispatcher and stripes servlet
I get them initialized twice in a row e.g.
PWC1412: WebModule[null] ServletContext.log():Initializing Spring root WebApplicationContext|#]
INFO StripesFilter - Stripes Initialization Complete. Version: 1.5.6, Build: 1.5.6
PWC1412: WebModule[null] ServletContext.log():Initializing Spring root WebApplicationContext|#]
INFO StripesFilter - Stripes Initialization Complete. Version: 1.5.6,
Build: 1.5.6
WEB0671: Loading application [my-web] at [/my-web]|#]
That wouldn't be a much problem if not for an blazeds servlet that i initialize from spring which fails completely on second initialization. I tried disabling blazeds servlet but that doesn't solve double initialization problem because i still get messages of initialization twice in a log. And my application wouldn't work without blazeds so i'm in a pinch here.
UPDATE: I've confirmed it only happens on OpenIndiana host, same configuration on linux works without issues
We had a similar problem. We also tried to deploy to a particular virtual server, which did not work. To fix the issue I ended up having to manually delete the extra virtual-server from the domain.xml:
<virtual-server id="admin-server" network-listeners="http-listener-2"></virtual-server>
If you have more than one virtual server:
"If you deploy a web application and don't specify any assigned virtual servers, the web application is assigned to all currently defined virtual servers"
http://docs.oracle.com/cd/E18930_01/html/821-2416/ggncs.html
But this did not solve our problem.
We had to delete the extra virtual server for it to work. This also happens on 3.1.2 and it is not just a windows problem. We replicated the issue on Linux too.

Categories