We Keep Coding

How to rewrite a RewriteValve for Undertow / JBoss 7.2 EAP?

I'm migrating from JBoss 6.4.3 to JBoss 7.2 and saw a Valves are no longer supported warning during deployment. This came from a jboss-web.xml file with:
<valve>
<class-name>org.jboss.web.rewrite.RewriteValve</class-name>
</valve>
...and a corresponding rewrite.properties file:
RewriteCond %{HTTP:X-Forwarded-Proto} http
RewriteRule .* https://%{HTTP_HOST}%{REQUEST_URI} [R,L]
Could anyone advise how to rewrite this (no pun intended) for Undertow?

You can create a rewrite filter in the Undertow subsystem, and then reference it in the server host within the configuration file (standalone.xml or domain.xml - depending on the mode in which you start the application server).
I can think of two options, which might help you:
Using the JBoss Application Server Client (should be placed in path/to/jboss-7.2/bin/)
Creating the rewrite filter with the custom name redirect-http-to-https:
./jboss-cli.sh --connect --command="/subsystem=undertow/configuration=filter/rewrite=redirect-http-to-https:add(redirect=\"true\",target=\"https://%{LOCAL_SERVER_NAME}%{REQUEST_URL}\")"
Using/referencing the filter redirect-http-to-https:
./jboss-cli.sh --connect --command="/subsystem=undertow/server=default-server/host=default-host/filter-ref=redirect-http-to-https:add(predicate=\"equals(%p,80)\")"
Manually editing the respective config file (e.g. standalone.xml)
<subsystem xmlns="urn:jboss:domain:undertow:11.0" default-server="default-server" [...]>
<buffer-cache name="default"/>
<server name="default-server">
[...]
<host name="default-host" alias="localhost">
<filter-ref name="redirect-http-to-https" predicate="equals(%p,80)"/>
</host>
</server>
[...]
<filters>
<rewrite name="redirect-http-to-https" target="https://%{LOCAL_SERVER_NAME}%{REQUEST_URL}" redirect="true"/>
</filters>
</subsystem>
Note: For the Undertow exchange attributes (e.g. LOCAL_SERVER_NAME) refer to Undertow documentation. Further, the part predicate=\"equals(%p,80)\" in the filter-refchecks the requested port (%p -> just another Undertow exchange attribute) and if it equals to 80, then it triggers our redirect filter redirect-http-to-https - you can change the port 80 as per your need.

tomcat starts but its not listening on any port

I have a tomcat service configured to run on port 8380. This site was running fine until recently. I started to see this in the mod_jk.log on apache:
[3409:140197396281344] [info] jk_open_socket::jk_connect.c (627): connect to 127.0.0.1:8309 failed (errno=111)
[3409:140197396281344] [info] ajp_connect_to_endpoint::jk_ajp_common.c (995): Failed opening socket to (127.0.0.1:8309) (errno=111)
[3409:140197396281344] [error] ajp_send_request::jk_ajp_common.c (1630): (worker1) connecting to backend failed. Tomcat is probably not started or is listening on the wrong port (errno=111)
[3409:140197396281344] [info] ajp_service::jk_ajp_common.c (2623): (worker1) sending request to tomcat failed (recoverable), because of error during request sending (attempt=1)
[3409:140197396281344] [info] jk_open_socket::jk_connect.c (627): connect to 127.0.0.1:8309 failed (errno=111)
[3409:140197396281344] [info] ajp_connect_to_endpoint::jk_ajp_common.c (995): Failed opening socket to (127.0.0.1:8309) (errno=111)
[3409:140197396281344] [error] ajp_send_request::jk_ajp_common.c (1630): (worker1) connecting to backend failed. Tomcat is probably not started or is listening on the wrong port (errno=111)
[3409:140197396281344] [info] ajp_service::jk_ajp_common.c (2623): (worker1) sending request to tomcat failed (recoverable), because of error during request sending (attempt=2)
[3409:140197396281344] [error] ajp_service::jk_ajp_common.c (2643): (worker1) connecting to tomcat failed.
[3409:140197396281344] [info] jk_handler::mod_jk.c (2788): Service error=-3 for worker=worker1
I'm able to start up the tomcat service but the port it's supposed to be listening on is not accepting connections. nmap and netstat don't show the ports as available. I've also tried telnetting to to the port just in case but as i expected there's no response. I've stopped the firewall in case it's a iptables rule issue but that too had no effect. I've googled similar issues resolved by what i tried above but none of them apply to my issues. I'm not sure what else to try at this point and so I'm turning to you all!
For this particular host the CATALINA_BASE is from a mounted nfs export. I have 2 tomcat sites hosted on this server. For the sake of testing I've disabled one from starting up since they both have same issue. i figured if i can get 1 running the other will be work as well.
server.xml:
<Server port="8105" shutdown="SHUTDOWN">
<Listener className="org.apache.catalina.core.AprLifecycleListener" SSLEngine="on" />
<Listener className="org.apache.catalina.core.JasperListener" />
<Listener className="org.apache.catalina.core.JreMemoryLeakPreventionListener" />
<Listener className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener" />
<Listener className="org.apache.catalina.core.ThreadLocalLeakPreventionListener" />
<GlobalNamingResources>
<Resource name="UserDatabase" auth="Container"
type="org.apache.catalina.UserDatabase"
description="User database that can be updated and saved"
factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
pathname="conf/tomcat-users.xml" />
</GlobalNamingResources>
<Service name="Catalina">
<Connector port="8380" protocol="HTTP/1.1"
connectionTimeout="20000"
URIEncoding="UTF-8"
redirectPort="8443" />
<Connector port="8309" protocol="AJP/1.3" redirectPort="8443" />
<Engine name="Catalina" defaultHost="localhost">
<Realm className="org.apache.catalina.realm.LockOutRealm">
<Realm className="org.apache.catalina.realm.UserDatabaseRealm"
resourceName="UserDatabase"/>
</Realm>
<Host name="localhost" appBase="webapps"
unpackWARs="true" autoDeploy="true">
<Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"
prefix="localhost_access_log." suffix=".txt"
pattern="%h %l %u %t "%r" %s %b" />
</Host>
<Host name="mytomcat.site.org" appBase="webapps"
unpackWARs="true" autoDeploy="true">
</Host>
</Engine>
</Service>
</Server>
My worker.properties in apache. I have 2 workers 1 for each tomcat site.
workers.tomcat_home=/opt/tomcat
workers.java_home=/usr/local/jdk/jre
ps=/
# Define workers
worker.list=worker1
worker.list=worker2
# Set properties for worker1 - localhost
worker.worker1.type=ajp13
worker.worker1.host=127.0.0.1
worker.worker1.port=8309
worker.worker1.socket_keepalive=True
worker.worker1.socket_timeout=300
# Set properties for worker2 - localhost
worker.worker2.type=ajp13
worker.worker2.host=127.0.0.1
worker.worker2.port=8409
worker.worker2.socket_keepalive=True
worker.worker2.socket_timeout=300
Apache starts up ok and is serving static pages ok. tomcat starts ok but like i said it's not listening on any port.
Any help would be appreciated. TIA!
MORE INFO:
I ran the init script with -debug in the JAVA_OPTS var. and I see the following towards the bottom of the output that scrolls out to stdout:
...
user changed to 'daemon'
User 'daemon' validated
Attempting to locate Java Home in /usr/local/jdk
Attempting to locate VM configuration file /usr/local/jdk/jre/lib/jvm.cfg
Attempting to locate VM configuration file /usr/local/jdk/lib/jvm.cfg
Attempting to locate VM configuration file /usr/local/jdk/jre/lib/amd64/jvm.cfg
Found VM configuration file at /usr/local/jdk/jre/lib/amd64/jvm.cfg
Found VM server definition in configuration
Checking library /usr/local/jdk/jre/lib/amd64/server/libjvm.so
Found VM client definition in configuration
Checking library /usr/local/jdk/jre/lib/amd64/client/libjvm.so
Checking library /usr/local/jdk/lib/amd64/client/libjvm.so
Cannot locate library for VM client (skipping)
Found VM hotspot definition in configuration
Checking library /usr/local/jdk/jre/lib/amd64/hotspot/libjvm.so
Checking library /usr/local/jdk/lib/amd64/hotspot/libjvm.so
Cannot locate library for VM hotspot (skipping)
Found VM classic definition in configuration
Checking library /usr/local/jdk/jre/lib/amd64/classic/libjvm.so
Checking library /usr/local/jdk/lib/amd64/classic/libjvm.so
Cannot locate library for VM classic (skipping)
Found VM native definition in configuration
Checking library /usr/local/jdk/jre/lib/amd64/native/libjvm.so
Checking library /usr/localjdk/lib/amd64/native/libjvm.so
Cannot locate library for VM native (skipping)
Found VM green definition in configuration
Checking library /usr/local/jdk/jre/lib/amd64/green/libjvm.so
Checking library /usr/local/jdk/lib/amd64/green/libjvm.so
Cannot locate library for VM green (skipping)
Java Home located in /usr/local/jdk
+-- DUMPING JAVA HOME STRUCTURE ------------------------
| Java Home: "/usr/local/jdk"
| Java VM Config.: "/usr/local/jdk/jre/lib/amd64/jvm.cfg"
| Found JVMs: 1
| JVM Name: "server"
| "/usr/local/jdk/jre/lib/amd64/server/libjvm.so"
+-------------------------------------------------------
Running w/ LD_LIBRARY_PATH=/usr/local/jdk/jre/lib/amd64/server:/usr/local/jdk/jre/lib/amd64
wait_child 12035
redirecting stdout to /opt/www/mytomcat/logs/catalina-daemon.out and stderr to &1
Then it goes back to the shell prompt. If it completed successfully it would have additional lines scrolling out:
get_pidf: 5 in /opt/www/mytomcat/logs/catalina-daemon.pid
get_pidf: pid 97253
check_tmp_file: /tmp/97253.jsvc_up
get_pidf: 5 in /opt/www/mytomcat/logs/catalina-daemon.pid
get_pidf: pid 97253
check_tmp_file: /tmp/97253.jsvc_up
get_pidf: 5 in /opt/www/mytomcat/logs/catalina-daemon.pid
get_pidf: pid 97253
check_tmp_file: /tmp/97253.jsvc_up
get_pidf: 5 in /opt/www/mytomcat/logs/catalina-daemon.pid
get_pidf: pid 97253
check_tmp_file: /tmp/97253.jsvc_up
get_pidf: 5 in /opt/www/mytomcat/logs/catalina-daemon.pid
get_pidf: pid 97253
check_tmp_file: /tmp/97253.jsvc_up
get_pidf: 5 in /opt/www/mytomcat/logs/catalina-daemon.pid
get_pidf: pid 97253
check_tmp_file: /tmp/97253.jsvc_up
I also noticed both tomcat process are running as root when i expect the child process to be running as daemon. Not sure why this is happening but it may be related to the issue.

Based on the debug output I started to try a bunch of different things specifically around catalina-daemon.out and catalina-daemon.pid files. As it turns out because these files are on the nfs location it causes the odd issue of the service starting but not listening. Once I moved these to the local filesystem the service started up AND it was listening on the correct port again. I also noticed the child tomcat process switched to the unprivileged user daemon. The page is reachable again.
Not entirely sure why it was working before but all of a sudden its not when i made no changes to the server. Something must have changed in the nfs export from the zfs machine that was serving up the CATALINA_BASE. Somehow this may have exposed a bug that causes this issue on the tomcat server.

Wildfly10 - EJB-Remote Client - no response

I'm currently migrating our code from Jboss7 to Wildfly10.
The Server itself starts up totaly fine.
When trying to connect our client with the working new wildfly10 server for ejb-remote calls it just won't work.
The only thing I get to work with is the following error:
org.jboss.ejb.client.remoting.ConfigBasedEJBClientContextSelector setupEJBReceivers
WARN: Could not register a EJB receiver for connection to remote-ip:8080
java.lang.RuntimeException: Operation failed with status WAITING at
org.jboss.ejb.client.remoting.IoFutureHelper.get(IoFutureHelper.java:94)
at
org.jboss.ejb.client.remoting.ConnectionPool.getConnection(ConnectionPool.java:80)
at
org.jboss.ejb.client.remoting.RemotingConnectionManager.getConnection(RemotingConnectionManager.java:51)
at
org.jboss.ejb.client.remoting.ConfigBasedEJBClientContextSelector.setupEJBReceivers(ConfigBasedEJBClientContextSelector.java:161)
at
org.jboss.ejb.client.remoting.ConfigBasedEJBClientContextSelector.getCurrent(ConfigBasedEJBClientContextSelector.java:118)
at
org.jboss.ejb.client.remoting.ConfigBasedEJBClientContextSelector.getCurrent(ConfigBasedEJBClientContextSelector.java:47)
at
org.jboss.ejb.client.EJBClientContext.getCurrent(EJBClientContext.java:281)
at
org.jboss.ejb.client.EJBClientContext.requireCurrent(EJBClientContext.java:291)
at
org.jboss.ejb.client.EJBInvocationHandler.doInvoke(EJBInvocationHandler.java:178)
at
org.jboss.ejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:146)
at com.sun.proxy.$Proxy2.connect(Unknown Source) at
de.cinovo.rcp.test.RemoteEJBClient.invokeStatelessBean(RemoteEJBClient.java:39)
at de.cinovo.rcp.test.RemoteEJBClient.main(RemoteEJBClient.java:25)
Exception in thread "main" java.lang.IllegalStateException:
EJBCLIENT000025: No EJB receiver available for handling
[appName:de.cinovo.tcc.server-ear,
moduleName:de-cinovo-tcc-server-ejb-6.0-SNAPSHOT, distinctName:]
combination for invocation context
org.jboss.ejb.client.EJBClientInvocationContext#180542f at
org.jboss.ejb.client.EJBClientContext.requireEJBReceiver(EJBClientContext.java:798)
at
org.jboss.ejb.client.ReceiverInterceptor.handleInvocation(ReceiverInterceptor.java:128)
at
org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:186)
at
org.jboss.ejb.client.EJBInvocationHandler.sendRequestWithPossibleRetries(EJBInvocationHandler.java:255)
at
org.jboss.ejb.client.EJBInvocationHandler.doInvoke(EJBInvocationHandler.java:200)
at
org.jboss.ejb.client.EJBInvocationHandler.doInvoke(EJBInvocationHandler.java:183)
at
org.jboss.ejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:146)
at com.sun.proxy.$Proxy2.connect(Unknown Source) at
de.cinovo.rcp.test.RemoteEJBClient.invokeStatelessBean(RemoteEJBClient.java:39)
at de.cinovo.rcp.test.RemoteEJBClient.main(RemoteEJBClient.java:25)
There is no error, warning, info or anything showing up in the server log, while trying to connect.
There is some action on the port via tcp while watching during a call attempt.
The realy funny part is:
If I use the same wildfly setup on my local machine, the exact same connection method works, but only while using localhost as the ip address within the jboss-ejb-client.properties.
As soon as I change the ip into 127.0.0.1 or my current ip address, it will fail with the same error as above.
Relevant information:
Wildfly will respond to a telnet on port 8080
Wildfly is the only service listening on 8080
My /etc/hosts is correctly configured
Changing the port doesn't fix the problem
Wildfly Version 10.1.0.Final
Relevant parts from my standalone.xml
<subsystem xmlns="urn:jboss:domain:remoting:3.0">
<endpoint/>
<http-connector name="http-remoting-connector" connector-ref="default" security-realm="ApplicationRealm"/>
</subsystem>
[...]
<subsystem xmlns="urn:jboss:domain:undertow:3.1">
<buffer-cache name="default"/>
<server name="default-server">
<http-listener name="default" socket-binding="http" redirect-socket="https" enable-http2="true"/>
[...]
</subsystem>
[...]
<interfaces>
<interface name="public">
<any-address/>
</interface>
</interfaces>
[...]
<socket-binding-group name="standard-sockets" default-interface="public" port-offset="${jboss.socket.binding.port-offset:0}">
<socket-binding name="http" interface="public" port="${jboss.http.port:8080}"/>
<socket-binding name="https" port="${jboss.https.port:8443}"/>
[...]
</socket-binding-group>
My jboss-ejb-client.properties
endpoint.name=client-endpoint
remote.connectionprovider.create.options.org.xnio.Options.SSL_ENABLED=false
remote.connections=default
remote.connection.default.host=<host-ip>
remote.connection.default.port=8080
remote.connection.default.connect.options.org.xnio.Options.SASL_POLICY_NOANONYMOUS=false
remote.connection.default.username=<usernmae>
remote.connection.default.password=<pswd>
Client-Code
final Hashtable jndiProperties = new Hashtable();
jndiProperties.put(Context.URL_PKG_PREFIXES, "org.jboss.ejb.client.naming");
final Context context = new InitialContext(jndiProperties);
[...]
return context.lookup("ejb:" + appName + "/" + moduleName + "/" + distinctName + "/" + beanName + "!" + viewClassName);
EJB-Client-Maven-Dependency :
<dependency>
<groupId>org.wildfly</groupId>
<artifactId>wildfly-ejb-client-bom</artifactId>
<version>10.1.0.Final</version>
<type>pom</type>
</dependency>
Anyone out there who had the same problem and knows what I'm doing wrong?

Looks like there is a missing definition in the standalone.xml in the socket.binding-group:
<outbound-socket-binding name="remote-ejb">
<local-destination socket-binding-ref="http"/>
</outbound-socket-binding>

So for everybody interested, here is the solution to my problem:
based on the comment by Steve C and the help of a friend, we figured out that the problem is not server based.
It appears that there are some antivirus programs, which do something with your HTTP messages, as soon as the HTTP-upgrade negotiations with the Wildfly/server are done. They seem to manipulate the sent/received packages, which leads to the problem in the client, as it is no longer able to understand the answers.
Therefore it never gets to react, since the package appears to have been lost - hence the IoFuture exception and EJB receiver not found.
Long story short: removing the antivirus program from our systems (in our case Bitdefender) leads to everything working as intended...

Tomcat memory leak warning - AS400 job threads not getting closed on shutdown/redeploy

I have a spring application running on a Tomcat 8.0 server the following error when either Stopping the server or Redeploying the app on the server.
INFO: Closing Spring root WebApplicationContext
Jul 25, 2016 10:18:40 AM org.apache.catalina.loader.WebappClassLoaderBase clearReferencesThreads
WARNING: The web application [APP_NAME] appears to have started a thread named [AS400 Read Daemon [system:*connectionurl*;job:768339/QUSER/QZDASOINIT]] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.socketRead(Unknown Source)
java.net.SocketInputStream.read(Unknown Source)
java.net.SocketInputStream.read(Unknown Source)
com.ibm.as400.access.DataStream.readFromStream(DataStream.java:52)
com.ibm.as400.access.ClientAccessDataStream.construct(ClientAccessDataStream.java:58)
com.ibm.as400.access.AS400ThreadedServer.run(AS400ThreadedServer.java:357)
java.lang.Thread.run(Unknown Source)
I am connecting to an AS400 db with the JT400 maven dependency (version 9.0, though I've tried others), so the AS400JDBCDriver. Here is how my connection is set up in my app's context.xml
<Resource name="jdbc/test_name"
auth="Container"
driverClassName="com.ibm.as400.access.AS400JDBCDriver"
factory="org.apache.tomcat.jdbc.pool.DataSourceFactory"
initialSize="3"
logAbandoned="true"
maxActive="100"
maxIdle="40"
maxWait="30000"
minEvictableIdleTimeMillis="60000"
minIdle="10"
password="*password*"
removeAbandoned="true"
removeAbandonedTimeout="60"
testOnBorrow="true"
testOnReturn="false"
testWhileIdle="true"
timeBetweenEvictionRunsMillis="30000"
type="javax.sql.DataSource"
url="jdbc:as400:*connection url*;date format=iso;time format=iso;"
username="*username*"
validationInterval="30000"
validationQuery="SELECT current date FROM sysibm.sysdummy1"
/>
The datasource is then declared in a spring configuration file:
<bean id="dataSourceTester" class="org.springframework.jndi.JndiObjectFactoryBean">
<property name="jndiName" value="java:comp/env/jdbc/test_name"/>
</bean>
And then I have dozens of beans using that datasource.
<bean id="testTesterDAO" class="*class url*" scope="request">
<property name="dataSource" ref="dataSourceTester" />
So the problem is, when I stop my Tomcat or redeploy, I am greeted with a number of these messages about threads that have failed to stop. I know that it specifically comes from this connection source because when I change the "initialSize" attribute (like from 3 to 5), I get that many messages when the server stops. So it's clear that all of these connections being opened by the connection pool are not getting terminated.
When the server shuts down, the threads do dissappear (obviously, the server doesn't exist anymore), but when its just a redeploy, these threads definitely stay open. I know this because I can go on the as400 machine and find the open job, and they dont really seem to time out.
I've tried a bunch of things, I have a ServletContextListener which deregisters the driver (which got rid of that warning, but not these), I have tried moving the jt400.jar to the tomcat/lib folder (but I can't really tell if its working off of there or not when I run it), and I've tried tweaking the settings and trying different versions of the jt400.jar. No luck.
Any advice or help would be greatly appreciated.

JBoss EAP 6.2 any-address and Netty for remote JMS MDBs

I am running JBoss EAP 6.2 (embedded HornetQ 2.3.12.Final) as a standalone Messaging server that will host JMS Queues and Topics.
There will be other JBoss servers that remotely connect to the Messaging Server's Queues and Topics.
Using this Stackoverflow answer: Binding JBoss AS 7 to all interfaces
I made the following change to my standalone-full.xml configuration:
<interfaces>
...
<interface name="public">
<!--<inet-address value="${jboss.bind.address:127.0.0.1}"/>-->
<any-address/>
</interface>
</interfaces>
After making this change, I can successfully connect from other JBoss servers via port 4447 (JBoss Remoting) and writing JMS Client code (ConnectionFactory, Connection, MessageProducer, etc).
JBoss log has the following, which is reassuring:
[org.jboss.as.remoting] (MSC service thread 1-3) JBAS017100:
Listening on 0.0.0.0:4447
However, I cannot connect via a remote MDB. :-( I need to figure this out because we do MDBs and not JMS Client code.
I looked at my JBoss log, and see the following error:
HQ121005: Invalid "host" value "0.0.0.0" detected for "netty" connector.
Switching to "ptenn-fc20.arbfile.org". If this new address is incorrect please
manually configure the connector to use the proper one.
I looked through the <subsystem xmlns="urn:jboss:domain:messaging:1.4"><hornetq-server> ... section and did not see anything about how to specify a different address for the "netty" connector.
It definitely seems that Netty is using the public interface that I specified, I'm puzzled as to why JBoss Remoting (on port 4447) is fine listening on 0.0.0.0, but Netty (on port 5445) is not.
If anyone has any ideas, insight, or suggestions, I would sincerely appreciate it. :-)
Thanks,
Philip

Philip,
The host is specified via the interfaces element of standalone-full-ha.xml.
You then refer to that interface in the socket-bindings element of same config file.
e.g. The sample below binds JMS to a specific IP address and no longer 0.0.0.0...
<interfaces>
<interface name="management">
<!--<inet-address value="${jboss.bind.address.management:127.0.0.1}"/>-->
<inet-address value="0.0.0.0"/>
</interface>
<interface name="public">
<inet-address value="0.0.0.0"/>
<!--just an FYI that we can use the following syntax too-->
<!--<any-address/>-->
</interface>
<interface name="jms"> <-- HERE
<inet-address value="${jboss.bind.address}"/>
</interface>
<!-- TODO - only show this if the jacorb subsystem is added -->
<interface name="unsecure">
<!--
~ Used for IIOP sockets in the standard configuration.
~ To secure JacORB you need to setup SSL
-->
<inet-address value="${jboss.bind.address.unsecure:127.0.0.1}"/>
</interface>
</interfaces>
<socket-binding-group name="standard-sockets" default-interface="public" port-offset="${jboss.socket.binding.port-offset:0}">
.
.
.
<!--JMS-->
<socket-binding name="messaging" port="5445" interface="jms"/> <-- HERE
.
.
.
</socket-binding-group>
Rob