We Keep Coding

Service 'hdfs' check failed: java.net.UnknownHostException - Hortonworks

hi I m learing in Hortonworks, I m facing a problem when ever I
open ambari -> Files View, or Hive, Pig.
how can i solve this,
Issues detected
Service 'hdfs' check failed: java.net.UnknownHostException: sandbox.hortonworks.com
Collapse Stack Trace
Service 'hdfs' check failed:
java.lang.IllegalArgumentException: java.net.UnknownHostException: sandbox.hortonworks.com
at org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:411)
at org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:429)
at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.initialize(WebHdfsFileSystem.java:207)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2761)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:99)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2795)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2777)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:386)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:179)
at org.apache.ambari.view.utils.hdfs.HdfsApi$1.run(HdfsApi.java:70)
at org.apache.ambari.view.utils.hdfs.HdfsApi$1.run(HdfsApi.java:68)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
at org.apache.ambari.view.utils.hdfs.HdfsApi.execute(HdfsApi.java:397)
at org.apache.ambari.view.utils.hdfs.HdfsApi.<init>(HdfsApi.java:68)
at org.apache.ambari.view.utils.hdfs.HdfsUtil.connectToHDFSApi(HdfsUtil.java:127)
at org.apache.ambari.view.commons.hdfs.HdfsService.hdfsSmokeTest(HdfsService.java:136)
at org.apache.ambari.view.filebrowser.HelpService.hdfsStatus(HelpService.java:86)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205)
at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)
at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept(SubLocatorRule.java:137)
at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept(SubLocatorRule.java:137)
at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept(SubLocatorRule.java:137)
at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept(SubLocatorRule.java:137)
at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept(SubLocatorRule.java:137)
at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542)
at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473)
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409)
at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409)
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558)
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1507)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:118)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:84)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.apache.ambari.server.security.authorization.AmbariAuthorizationFilter.doFilter(AmbariAuthorizationFilter.java:257)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:103)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:54)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:45)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.apache.ambari.server.security.authorization.jwt.JwtAuthenticationFilter.doFilter(JwtAuthenticationFilter.java:96)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:150)
at org.apache.ambari.server.security.authentication.AmbariAuthenticationFilter.doFilter(AmbariAuthenticationFilter.java:88)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.apache.ambari.server.security.authorization.AmbariUserAuthorizationFilter.doFilter(AmbariUserAuthorizationFilter.java:91)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:192)
at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:160)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1478)
at org.apache.ambari.server.api.MethodOverrideFilter.doFilter(MethodOverrideFilter.java:72)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1478)
at org.apache.ambari.server.api.AmbariPersistFilter.doFilter(AmbariPersistFilter.java:47)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1478)
at org.apache.ambari.server.view.AmbariViewsMDCLoggingFilter.doFilter(AmbariViewsMDCLoggingFilter.java:54)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1478)
at org.apache.ambari.server.view.ViewThrottleFilter.doFilter(ViewThrottleFilter.java:161)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1478)
at org.apache.ambari.server.security.AbstractSecurityHeaderFilter.doFilter(AbstractSecurityHeaderFilter.java:109)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1478)
at org.apache.ambari.server.security.AbstractSecurityHeaderFilter.doFilter(AbstractSecurityHeaderFilter.java:109)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1478)
at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:82)
at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:294)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1478)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:499)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:427)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
at org.apache.ambari.server.controller.AmbariHandlerList.processHandlers(AmbariHandlerList.java:212)
at org.apache.ambari.server.controller.AmbariHandlerList.processHandlers(AmbariHandlerList.java:201)
at org.apache.ambari.server.controller.AmbariHandlerList.handle(AmbariHandlerList.java:150)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
at org.eclipse.jetty.server.Server.handle(Server.java:370)
at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494)
at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:973)
at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1035)
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:641)
at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:231)
at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:696)
at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:53)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.UnknownHostException: sandbox.hortonworks.com
... 121 more
I m newbie. please kindly guide me. i m using Hortonworks of docker in windows .
Update:
As per PRE-REQUISITES I have added like this in
c/Windows/System32/Drivers/etc/hosts
localhost name resolution is handled within DNS itself.
127.0.0.1 sandbox.hortonworks.com
192.168.XX.XX sandbox.hortonworks.com
Still i have getting the error . Can anyone help me here.

This error is coming from inside of the sandbox virtual machine from hadoop itself. Updating your Windows c/Windows/System32/Drivers/etc/hosts will only help you to resolve that hostname to access that web ui from a browser using the hostname. The Hadoop Sandbox is a virtual machine, you have to update it inside of the Virtual Machines operating system, which is most likely Linux.
You will need to open a terminal session into the Virtual Machine using a client such as PuTTy which you can find free out there for download. Open Putty the input the ip address of the sandbox to open an ssh connection to it. In the hostname field of putty enter 127.0.0.1 then for the port enter 2222 then press okay. You will maybe be prompted to accept a host key, press okay. You should be prompted for a username, enter root as the user name. It shouldn't ask for a password. Then you should have a terminal prompt in linux, run the following to echo the hostname to /etc/hosts file.
cat /etc/hosts;
echo "127.0.0.1 sandbox.hortonworks.com sandbox localhost" >> /etc/hosts
To check updates;
cat /etc/hosts;
ping sandbox.hortonworks.com
Exit your terminal session then try configuration again.
enter image description here

Take a look at the prerequisites for running hortonworks hadoop in docker. What you're probably missing is an entry in your hosts file. Here's the excerpt from the link:
If on windows 7, to add sandbox.hortonworks.com to your list of hosts, open git bash, enter the following command, replace {Host-Name} with the appropriate host for your sandbox:
echo '{Host-Name} sandbox.hortonworks.com' | tee -a /c/Windows/System32/Drivers/etc/hosts

SSL/TLS Connection reset - how to force Java client to use SNI?

I had BOTH: a STS server and spring web application on localhost. And it worked. I've tested OAuth2 Authorization Code Grant Flow - fully working.
I moved STS to other machine and there is a problem from web application in acquiring an access token - full trace (stop/error after redirection from STS; just in the beginning of acquiring token from: https://sts-machine/authz/oauth2-resource/oauth/token endpoint; stop at visiting url: https://localwebapp:7443/swd/webLogin?code=306b88903d394f0b8145b2e8035fd306&state=PtTrt8):
SEVERE: Servlet.service() for servlet [swdServlet] in context with path [/swd] threw exception
error="access_denied", error_description="Error requesting access token."
at org.springframework.security.oauth2.client.token.OAuth2AccessTokenSupport.retrieveToken(OAuth2AccessTokenSupport.java:144)
at org.springframework.security.oauth2.client.token.grant.code.AuthorizationCodeAccessTokenProvider.obtainAccessToken(AuthorizationCodeAccessTokenProvider.java:198)
at org.springframework.security.oauth2.client.token.AccessTokenProviderChain.obtainNewAccessTokenInternal(AccessTokenProviderChain.java:142)
at org.springframework.security.oauth2.client.token.AccessTokenProviderChain.obtainAccessToken(AccessTokenProviderChain.java:118)
at org.springframework.security.oauth2.client.OAuth2RestTemplate.acquireAccessToken(OAuth2RestTemplate.java:221)
at org.springframework.security.oauth2.client.OAuth2RestTemplate.getAccessToken(OAuth2RestTemplate.java:173)
at org.springframework.security.oauth2.client.filter.OAuth2ClientAuthenticationProcessingFilter.attemptAuthentication(OAuth2ClientAuthenticationProcessingFilter.java:90)
at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:211)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.springframework.security.oauth2.client.filter.OAuth2ClientContextFilter.doFilter(OAuth2ClientContextFilter.java:57)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:103)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:154)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:45)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:50)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:108)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.springframework.security.web.access.channel.ChannelProcessingFilter.doFilter(ChannelProcessingFilter.java:144)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:192)
at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:160)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:344)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:261)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:219)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:106)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:503)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:136)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)
at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:610)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:526)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1078)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:655)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:146)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:279)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.springframework.web.client.ResourceAccessException: I/O error on POST request for "https://sts-machine/authz/oauth2-resource/oauth/token":Connection reset; nested exception is java.net.SocketException: Connection reset
at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:557)
at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:510)
at org.springframework.security.oauth2.client.token.OAuth2AccessTokenSupport.retrieveToken(OAuth2AccessTokenSupport.java:136)
... 47 more
Caused by: java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:196)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at sun.security.ssl.InputRecord.readFully(InputRecord.java:442)
at sun.security.ssl.InputRecord.read(InputRecord.java:480)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:927)
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1312)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1339)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1323)
at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:563)
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(HttpsURLConnectionImpl.java:153)
at org.springframework.http.client.SimpleBufferingClientHttpRequest.executeInternal(SimpleBufferingClientHttpRequest.java:78)
at org.springframework.http.client.AbstractBufferingClientHttpRequest.executeInternal(AbstractBufferingClientHttpRequest.java:46)
at org.springframework.http.client.AbstractClientHttpRequest.execute(AbstractClientHttpRequest.java:52)
at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:541)
... 49 more
So there is connection reset when webapp tries to exchange "code" from access token from: https://sts-machine/authz/oauth2-resource/oauth/token.
Fun fact: Making request manually works well! (Manually exchanging code for access token) Tested with Chrome application "Advanced Rest Client Application" and with "Fiddler" request composer.
I sniffed both machines using wiresharks. Same result on both - screenshot: http://i.imgur.com/2AYstN0.png (where: 10.1.0.121 is localwebapp machine, and: 10.1.0.80 is sts-machine)
Why there is connection reset?
Details about config:
localwebapp: runs on tc Pivotal Server (kinda Tomcat), Win 8.1
sts-machine: is Thinktecture Authorization Server on IIS, Win Server 2012
UPDATE 1
What I found out that:
openssl s_client -connect sts-machine:443 does not work, but:
openssl s_client -connect sts-machine:443 -servername sts-machine does work! This indicates issue with the SNI on Java client. Tried to solve problem with setting system property: jsse.enableSNIExtension to true, but without any success... How to enforce JVM to use SNI?

Can you specify which Spring version you are using?
We fought our SNI bug today and the solution was:
* Upgrade to commons/httpclient 4.3.2+ (we use 4.3.5)
* Important! Replace DefaultHttpClient with HttpClientBuilder as DefaultHttpClient is still shipped with httpclient 4.3.x but does NOT support SNI

I had the same problem and also had an older version of HttpClient. I upgraded Java from 1.8.0_92 to 1.8.0_241 and the problem was solved.

Does Spring actually start a new transaction with REQUIRES_NEW?

My spring (4.1.1) application is deployed on a JBoss-6.10-final instance, so it uses the container-based transaction manager and data sources. For messaging, I use TIBCO EMS 8.1 with an XA queue connection factory set up. The Java version is 1.8.0_20. All of this is running on my Ubuntu 14.04 laptop.
I need to send a request via JMS and then wait for the reply. The bean I’m calling from has transaction propagation set to Propagation.REQUIRED, so I need to send the request in a new transaction and then wait for the reply. This means the request is sent in a separate bean with transaction propagation set to Propagation.REQUIRES_NEW.
It works, but I’m getting a worrisome warning from JBoss:
14-10-02 12:06:12,902 WARN
[org.jboss.tm.usertx.UserTransactionRegistry] (http-0.0.0.0-8080-1)
Error notifying listener
org.jboss.resource.connectionmanager.CachedConnectionManager#1917b4de
of userTransactionStarted: java.lang.IllegalStateException: Trying to
change transaction TransactionImple < ac, BasicAction:
0:ffff7f000101:126a:542d2010:d8 status: ActionStatus.RUNNING > in
enlist! at
org.jboss.resource.connectionmanager.TxConnectionManager$TxConnectionEventListener.enlist(TxConnectionManager.java:690)
at
org.jboss.resource.connectionmanager.TxConnectionManager.transactionStarted(TxConnectionManager.java:427)
at
org.jboss.resource.connectionmanager.CachedConnectionManager.userTransactionStarted(CachedConnectionManager.java:350)
at
org.jboss.tm.usertx.UserTransactionRegistry.userTransactionStarted(UserTransactionRegistry.java:119)
at
org.jboss.tm.usertx.client.ServerVMClientUserTransaction.begin(ServerVMClientUserTransaction.java:141)
at
org.springframework.transaction.jta.JtaTransactionManager.doJtaBegin(JtaTransactionManager.java:875)
at
org.springframework.transaction.jta.JtaTransactionManager.doBegin(JtaTransactionManager.java:832)
at
org.springframework.transaction.support.AbstractPlatformTransactionManager.handleExistingTransaction(AbstractPlatformTransactionManager.java:425)
at
org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:349)
at
org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:438)
at
org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:261)
at
org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:95)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:207)
at com.sun.proxy.$Proxy234.request(Unknown Source) at
com.izazi.ioriginate.framework.spring.jms.AbstractRequestReply.request(AbstractRequestReply.java:58)
at
com.izazi.ioriginate.service.addressvalidation.AddressValidationServiceImpl.validate(AddressValidationServiceImpl.java:34)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483) at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at
org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:98)
at
org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:266)
at
org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:95)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:207)
at com.sun.proxy.$Proxy235.validate(Unknown Source) at
com.izazi.ioriginate.services.dwr.AddressValidation.validate(AddressValidation.java:40)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483) at
org.directwebremoting.impl.ExecuteAjaxFilter.doFilter(ExecuteAjaxFilter.java:34)
at
org.directwebremoting.impl.DefaultRemoter$1.doFilter(DefaultRemoter.java:428)
at
org.directwebremoting.impl.DefaultRemoter.execute(DefaultRemoter.java:431)
at
org.directwebremoting.impl.DefaultRemoter.execute(DefaultRemoter.java:283)
at
org.directwebremoting.servlet.PlainCallHandler.handle(PlainCallHandler.java:52)
at
org.directwebremoting.servlet.UrlProcessor.handle(UrlProcessor.java:101)
at
org.directwebremoting.servlet.DwrServlet.doPost(DwrServlet.java:146)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:754) at
javax.servlet.http.HttpServlet.service(HttpServlet.java:847) at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:324)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:242)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
at
org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:118)
at
org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:84)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at
org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at
org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:103)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at
org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at
org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:154)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at
org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:45)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at
org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:201)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at
org.springframework.security.web.authentication.ui.DefaultLoginPageGeneratingFilter.doFilter(DefaultLoginPageGeneratingFilter.java:155)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at
org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:199)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at
org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:110)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at
org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:50)
at
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at
org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at
org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:192)
at
org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:160)
at
org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:344)
at
org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:261)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:274)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:242)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:275)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:161)
at
org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:181)
at
org.jboss.modcluster.catalina.CatalinaContext$RequestListenerValve.event(CatalinaContext.java:285)
at
org.jboss.modcluster.catalina.CatalinaContext$RequestListenerValve.invoke(CatalinaContext.java:261)
at
org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:88)
at
org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.invoke(SecurityContextEstablishmentValve.java:100)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:159)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at
org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:158)
at
org.apache.catalina.valves.RequestDumperValve.invoke(RequestDumperValve.java:151)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at
org.jboss.web.tomcat.service.request.ActiveRequestResponseCacheValve.invoke(ActiveRequestResponseCacheValve.java:53)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:362)
at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:877)
at
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:654)
at
org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:951)
at java.lang.Thread.run(Thread.java:745)
...and on the EMS side, I see some XA errors:
johan#my_machine:~/opt/jboss-6.1.0.Final/bin$ 2014-10-02 10:43:15.801
ERROR: Transaction for non-existent consumer: 15 connID=16 sessID=20
{formatID=131076 gtrid_length=29 bqual_length=28
data=%00%00%00%00%00%00%00%00%00%00%FF%FF%7F%00%01%01%00%00%12jT-%0D*%00%00%00%AF1%00%00%00%00%00%00%00%00%00%00%FF%FF%7F%00%01%01%00%00%12jT-%0D*%00%00%00%B1}
2014-10-02 10:43:15.832 ERROR: Error processing xa end - transaction
marked ROLLBACKONLY, Exception. connID=16 sessID=20 {formatID=131076
gtrid_length=29 bqual_length=28
data=%00%00%00%00%00%00%00%00%00%00%FF%FF%7F%00%01%01%00%00%12jT-%0D*%00%00%00%AF1%00%00%00%00%00%00%00%00%00%00%FF%FF%7F%00%01%01%00%00%12jT-%0D*%00%00%00%B1}
After looking at the stack trace, I opened the source for Spring’s AbstractPlatformTransactionManager and came across the following code for handing of REQUIRES_NEW (starting at line 415):
if (definition.getPropagationBehavior() == TransactionDefinition.PROPAGATION_REQUIRES_NEW) {
if (debugEnabled) {
logger.debug("Suspending current transaction, creating new transaction with name [" +
definition.getName() + "]");
}
SuspendedResourcesHolder suspendedResources = suspend(transaction);
try {
boolean newSynchronization = (getTransactionSynchronization() != SYNCHRONIZATION_NEVER);
DefaultTransactionStatus status = newTransactionStatus(
definition, transaction, true, newSynchronization, debugEnabled, suspendedResources);
doBegin(transaction, definition);
prepareSynchronization(status, definition);
return status;
}
catch (RuntimeException beginEx) {
resumeAfterBeginException(transaction, suspendedResources, beginEx);
throw beginEx;
}
catch (Error beginErr) {
resumeAfterBeginException(transaction, suspendedResources, beginErr);
throw beginErr;
}
}
My question is: where is a new transaction being started?
On the surface it looks like the existing transaction is being used instead of a new one being started - see how 'transaction' is being passed to doBegin(...). I looked in doBegin as well, and there is no indication of a new transaction being requested or created. This view appears to be supported by the stack trace and warning I get from JBoss...

Happy to see that I'am not alone stuck in this ligthless hole...
As far as I got it, the deep cause of this warning is described here (end of the thread)
when the outer transaction is suspended and the new inner transaction
started, the managed connection that Jboss connection pool retrieves
for the inner transaction is the same as that of the outer
transaction, which is causing the IllegalStateException to get thrown!
and is due to a specific behavior of the jboss JCA contract implementation (Lazy JCA enlistment).
A defect has been opened on spring side, flagged as "won't fix" but they provide a workaround configuration :
The typical solution is to use Spring's
TransactionAwareDataSourceProxy and switch the
"reobtainTransactionalConnections" flag to "true" there
Have fun !

Weblogic and Oracle: Stuck ExecuteThread

We are running SpringMVC with Hibernate on a clustered Weblogic environment. Our application worked well, until we run into database space problems. Due to that, our DB provider extended the database storage.
After that change the Weblogic system status is complaining about the thread pool "Thread pool has stucked threads".
The stack-trace in the log file looks as follows:
<Error> <WebLogicServer> <BEA-000337> <[STUCK] ExecuteThread: '12' for queue: 'weblogic.ke
rnel.Default (self-tuning)' has been busy for "641" seconds working on the request "weblogic.servlet.internal.ServletReq
uestImpl#17782ff3[
GET /default/default/applicationPreloader/load.do?_=1331887527968 HTTP/1.1
User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:10.0.2) Gecko/20100101 Firefox/10.0.2
Accept: */*
Accept-Language: en-gb,en;q=0.5
Accept-Encoding: gzip, deflate
x-requested-with: XMLHttpRequest
Referer: http://novintegweb01.eu.acme.net:9136/default/default/applicationPreloader.do
Cookie: JSESSIONID1=NhBdPv9Hx8Wj5lnr5vnGhLSbR2GGM1DPhqT2YGzWCGFvhN3WJWVH!700169023!1391220923
Connection: Keep-Alive
Proxy-Client-IP: 192.168.1.1
X-Forwarded-For: 192.168.1.1
X-WebLogic-KeepAliveSecs: 30
X-WebLogic-Request-ClusterInfo: true
x-weblogic-cluster-hash: G4kGT1kfuDBbtj5w//SYNt6hlVo
]", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
oracle.net.ns.Packet.receive(Packet.java:293)
oracle.net.ns.DataPacket.receive(DataPacket.java:104)
oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:315)
oracle.net.ns.NetInputStream.read(NetInputStream.java:260)
oracle.net.ns.NetInputStream.read(NetInputStream.java:185)
oracle.net.ns.NetInputStream.read(NetInputStream.java:102)
oracle.jdbc.driver.T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:124)
oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:80)
oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1136)
oracle.jdbc.driver.T4CMAREngine.unmarshalSB1(T4CMAREngine.java:1113)
oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:288)
oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:191)
oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:523)
oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:207)
oracle.jdbc.driver.T4CPreparedStatement.executeForRows(T4CPreparedStatement.java:1009)
oracle.jdbc.driver.OraclePreparedStatement.executeForRowsWithTimeout(OraclePreparedStatement.java:10117)
oracle.jdbc.driver.OraclePreparedStatement.executeBatch(OraclePreparedStatement.java:10221)
oracle.jdbc.driver.OracleStatementWrapper.executeBatch(OracleStatementWrapper.java:216)
weblogic.jdbc.wrapper.PreparedStatement.executeBatch(PreparedStatement.java:210)
org.hibernate.jdbc.BatchingBatcher.doExecuteBatch(BatchingBatcher.java:48)
org.hibernate.jdbc.AbstractBatcher.executeBatch(AbstractBatcher.java:247)
org.hibernate.jdbc.AbstractBatcher.prepareStatement(AbstractBatcher.java:93)
org.hibernate.jdbc.AbstractBatcher.prepareStatement(AbstractBatcher.java:88)
org.hibernate.jdbc.AbstractBatcher.prepareBatchStatement(AbstractBatcher.java:223)
org.hibernate.persister.collection.AbstractCollectionPersister.recreate(AbstractCollectionPersister.java:1119)
org.hibernate.action.CollectionRecreateAction.execute(CollectionRecreateAction.java:39)
org.hibernate.engine.ActionQueue.execute(ActionQueue.java:279)
org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:263)
org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:171)
org.hibernate.event.def.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:298)
org.hibernate.event.def.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:27)
org.hibernate.impl.SessionImpl.flush(SessionImpl.java:1001)
org.hibernate.impl.SessionImpl.managedFlush(SessionImpl.java:339)
org.hibernate.transaction.JDBCTransaction.commit(JDBCTransaction.java:106)
org.springframework.orm.hibernate3.HibernateTransactionManager.doCommit(HibernateTransactionManager.java:656)
org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:754)
org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:723)
org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:393)
org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:120)
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:89)
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:621)
com.acme.superproject.dao.CertificationDaoImpl$$EnhancerByCGLIB$$bf3135a5.getById(<generated>)
com.acme.superproject.integration.UserProfileSabaServiceImpl.determineTargetDateFromWs(UserProfileSabaServiceImpl.java:436)
com.acme.superproject.integration.UserProfileSabaServiceImpl.determineTargetDate(UserProfileSabaServiceImpl.java:401)
com.acme.superproject.integration.UserProfileSabaServiceImpl.createCertificationRelationObject(UserProfileSabaServiceImpl.java:373)
com.acme.superproject.integration.UserProfileSabaServiceImpl.getRequiredLearning(UserProfileSabaServiceImpl.java:334)
com.acme.superproject.integration.UserProfileSabaServiceImpl.extractUserProfile(UserProfileSabaServiceImpl.java:203)
com.acme.superproject.integration.UserProfileSabaServiceImpl.getUserProfile(UserProfileSabaServiceImpl.java:73)
com.acme.superproject.integration.UserProfileSabaServiceImpl.getUserProfile(UserProfileSabaServiceImpl.java:114)
com.acme.superproject.integration.UserProfileSabaServiceImpl$$FastClassByCGLIB$$e48c61c9.invoke(<generated>)
net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:149)
org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation.invokeJoinpoint(Cglib2AopProxy.java:688)
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:80)
com.acme.superproject.techarch.TracingAspect.trace(TracingAspect.java:31)
sun.reflect.GeneratedMethodAccessor5382.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621)
org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610)
org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:65)
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:89)
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:621)
com.acme.superproject.integration.UserProfileSabaServiceImpl$$EnhancerByCGLIB$$5d7b85df.getUserProfile(<generated>)
com.acme.superproject.service.UserProfileServiceImpl.getUserProfile(UserProfileServiceImpl.java:26)
com.acme.superproject.service.UserProfileServiceImpl$$FastClassByCGLIB$$70cfe815.invoke(<generated>)
net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:149)
org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation.invokeJoinpoint(Cglib2AopProxy.java:688)
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:80)
com.acme.superproject.techarch.TracingAspect.trace(TracingAspect.java:31)
sun.reflect.GeneratedMethodAccessor5382.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621)
org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610)
org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:65)
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:89)
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:621)
com.acme.superproject.service.UserProfileServiceImpl$$EnhancerByCGLIB$$5bcb212b.getUserProfile(<generated>)
com.acme.superproject.web.session.SessionDataProvider.fetchUserProfile(SessionDataProvider.java:97)
com.acme.superproject.web.session.SessionDataProvider.getUserProfile(SessionDataProvider.java:78)
com.acme.superproject.web.session.SessionDataProvider.fetchUserRegistrationCollection(SessionDataProvider.java:155)
com.acme.superproject.web.session.SessionDataProvider.getUserRegistrationCollection(SessionDataProvider.java:141)
com.acme.superproject.web.ApplicationPreloaderController.loadAction(ApplicationPreloaderController.java:84)
sun.reflect.GeneratedMethodAccessor6506.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
org.springframework.web.bind.annotation.support.HandlerMethodInvoker.invokeHandlerMethod(HandlerMethodInvoker.java:176)
org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.invokeHandlerMethod(AnnotationMethodHandlerAdapter.java:426)
org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.handle(AnnotationMethodHandlerAdapter.java:414)
org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:790)
org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:719)
org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:644)
org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:549)
javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227)
weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125)
weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:300)
weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:368)
org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:109)
org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:83)
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:380)
org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:97)
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:380)
org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:105)
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:380)
org.springframework.security.web.authentication.preauth.AbstractPreAuthenticatedProcessingFilter.doFilter(AbstractPreAuthenticatedProcessingFilter.java:89)
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:380)
org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:79)
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:380)
org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:169)
org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237)
org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:88)
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3715)
weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3681)
weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
weblogic.security.service.SecurityManager.runAs(SecurityManager.java:120)
weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2277)
weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2183)
weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1454)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:207)
weblogic.work.ExecuteThread.run(ExecuteThread.java:176)
>
We already have restarted Weblogic, the database server and made a new deployment. The problem is always the same: When we try to access the login page of the application it hangs up - as far as I can see while trying to perform a DB transaction.
In addition to that, when I try to connect to our database with "Oracle SQL Developer" we also get the error "ORA-00257: archiver error. Connect internal only, until freed."
Unfortunately we have no SYSDBA rights (because managed by an external provider).
My question are:
Has anyone of you had a similar problem before?
Do you think the root cause of such problems is more on application side, or more on DB side?
How can I at least get rid of the stucked threads?

Chances are that the database storage is full, or that the archiver cannot move or create new archive logs. This is almost definitely an issue at the database level and not the app server level. Since you are getting this message that means you already have a connection to the database, so it is not sitting and waiting for a connection.
Stuck Threads in WebLogic don't always need to be 'fixed' and are not always a problem. The purpose of the stuck thread mechanism is to notify you when a thread is taking an abnormally long time to do something (like access the database). The stuck thread notification is just that - a notification that something might not be right, and in this case it is serving that purpose.
If you have threads that routinely take 20 minutes to do something but want to be notified if it takes longer than 25 minutes, then have to increase the Stuck Thread Limit to 25 minutes and get the notification at that point then you must put this limit from 600 seconds to 1500 seconds (25*60).
You can edit stuck thread limit in weblogic admin base_domain > Environment > Servers > Admin Server > Configuration/Tuning.
refer below link :
https://docs.oracle.com/cd/E19316-01/820-3745/ghudn/index.html

This seems to be an issue with the database. The threads are stuck on the DB. The performance of your seems to have degraded. I would suggest to refer the case to your DBA for futher investigation. Also, try recreating your indexes on the tables being accessed.

We have faced this issue when the Oracle DB Server was on a VirtualBox, in a developer environment. Pointing out the issue was bit difficult. We noticed this when the ThreadPool did not have any more threads to handle the requests and the requests were rejected.

tomcat is giving error while deployement

I am working on tomcat 7.1, after 4,5 deployment I am getting following error:
SEVERE: Catalina.stop: java.net.SocketException: No buffer space available (maximum connections reached?): connect
Then I need to restart my system, because my internet also stopped working when this error happens. How is this problem caused and how can I solve it?
I just found the issue again, pasting the log. please help me.
INFO [http-apr-8080-exec-14] (OrderDAO_MySqlImpl.java:287) - INSIDE THE ORDER DAO IMPL: SUBCAT IS: Cpmuter Services
com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
Last packet sent to the server was 0 ms ago.
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:353)
at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1074)
at com.mysql.jdbc.Connection.createNewIO(Connection.java:2001)
at com.mysql.jdbc.Connection.<init>(Connection.java:716)
at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:46)
at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:353)
at com.mysql.jdbc.Connection.getInstance(Connection.java:295)
at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:266)
at java.sql.DriverManager.getConnection(DriverManager.java:582)
at java.sql.DriverManager.getConnection(DriverManager.java:185)
at bseller.Order.dao.mysql.OrderDAO_MySqlImpl.getConnection(OrderDAO_MySqlImpl.java:264)
at bseller.Order.dao.mysql.OrderDAO_MySqlImpl.getAllOrder(OrderDAO_MySqlImpl.java:294)
at bseller.postadd.utils.OrderControlService.getAllOrders(OrderControlService.java:33)
at bseller.postadd.utils.OrderManager.PopulateOrderDetail(OrderManager.java:50)
at org.apache.jsp.SubCategoryDeails_jsp._jspService(SubCategoryDeails_jsp.java:182)
at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:722)
at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:433)
at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:389)
at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:333)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:722)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:304)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:684)
at org.apache.catalina.core.ApplicationDispatcher.doInclude(ApplicationDispatcher.java:593)
at org.apache.catalina.core.ApplicationDispatcher.include(ApplicationDispatcher.java:530)
at org.apache.jasper.runtime.JspRuntimeLibrary.include(JspRuntimeLibrary.java:927)
at org.apache.jasper.runtime.PageContextImpl.include(PageContextImpl.java:652)
at org.apache.struts.tiles.TilesUtilImpl.doInclude(TilesUtilImpl.java:137)
at org.apache.struts.tiles.TilesUtil.doInclude(TilesUtil.java:177)
at org.apache.struts.taglib.tiles.InsertTag.doInclude(InsertTag.java:756)
at org.apache.struts.taglib.tiles.InsertTag$InsertHandler.doEndTag(InsertTag.java:881)
at org.apache.struts.taglib.tiles.InsertTag.doEndTag(InsertTag.java:473)
at org.apache.jsp.Layout_jsp._jspx_meth_tiles_005finsert_005f1(Layout_jsp.java:244)
at org.apache.jsp.Layout_jsp._jspService(Layout_jsp.java:137)
at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:722)
at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:433)
at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:389)
at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:333)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:722)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:304)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:684)
at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:471)
at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:402)
at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:329)
at org.apache.struts.action.RequestProcessor.doForward(RequestProcessor.java:1069)
at org.apache.struts.tiles.TilesRequestProcessor.doForward(TilesRequestProcessor.java:274)
at org.apache.struts.tiles.TilesRequestProcessor.processTilesDefinition(TilesRequestProcessor.java:254)
at org.apache.struts.tiles.TilesRequestProcessor.processForwardConfig(TilesRequestProcessor.java:309)
at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:279)
at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1482)
at org.apache.struts.action.ActionServlet.doGet(ActionServlet.java:507)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:621)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:722)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:304)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:224)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:462)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:164)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:100)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:851)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:405)
at org.apache.coyote.http11.Http11AprProcessor.process(Http11AprProcessor.java:257)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:515)
at org.apache.tomcat.util.net.AprEndpoint$SocketProcessor.run(AprEndpoint.java:1764)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.net.SocketException: No buffer space available (maximum connections reached?): connect
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
at java.net.Socket.connect(Socket.java:519)
at java.net.Socket.connect(Socket.java:469)
at java.net.Socket.<init>(Socket.java:366)
at java.net.Socket.<init>(Socket.java:209)
at com.mysql.jdbc.StandardSocketFactory.connect(StandardSocketFactory.java:168)
at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:262)
at com.mysql.jdbc.Connection.createNewIO(Connection.java:1926)
... 72 more
Sep 11, 2011 2:46:24 PM org.apache.catalina.core.ApplicationDispatcher invoke
SEVERE: Servlet.service() for servlet jsp threw exception

java.net.SocketException: No buffer space available (maximum connections reached?): connect
In plain english, this is the operating system telling you it can't create an additional TCP socket due to resource congestion. Most operating systems have a hard limit on how many connections can be open concurrently. It is possible you have hit that limit.
To avoid issues like that, database connections are usually pooled, i.e. rather than dispose of a no-longer needed connection, the connection is kept around, to be reused for future requests to the database. There are a lot of connection pooling libraries. Pooling capability is also provided by most application servers, if you obtain connections from a DataSource mapped in JNDI.
Another possible cause is that you are not closing all connections after you have used them.

Does your application use directly socket connections or databases? Maybe your application creates many connections and does not close them.
You may try to enter in cmd
netstat -a
and analyze what foreign addres occurs most times and do you know it - check is it database address or sth else familiar to you.