I'm migrating from JBoss 6.4.3 to JBoss 7.2 and saw a Valves are no longer supported warning during deployment. This came from a jboss-web.xml file with:
<valve>
<class-name>org.jboss.web.rewrite.RewriteValve</class-name>
</valve>
...and a corresponding rewrite.properties file:
RewriteCond %{HTTP:X-Forwarded-Proto} http
RewriteRule .* https://%{HTTP_HOST}%{REQUEST_URI} [R,L]
Could anyone advise how to rewrite this (no pun intended) for Undertow?
You can create a rewrite filter in the Undertow subsystem, and then reference it in the server host within the configuration file (standalone.xml or domain.xml - depending on the mode in which you start the application server).
I can think of two options, which might help you:
Using the JBoss Application Server Client (should be placed in path/to/jboss-7.2/bin/)
Creating the rewrite filter with the custom name redirect-http-to-https:
./jboss-cli.sh --connect --command="/subsystem=undertow/configuration=filter/rewrite=redirect-http-to-https:add(redirect=\"true\",target=\"https://%{LOCAL_SERVER_NAME}%{REQUEST_URL}\")"
Using/referencing the filter redirect-http-to-https:
./jboss-cli.sh --connect --command="/subsystem=undertow/server=default-server/host=default-host/filter-ref=redirect-http-to-https:add(predicate=\"equals(%p,80)\")"
Manually editing the respective config file (e.g. standalone.xml)
<subsystem xmlns="urn:jboss:domain:undertow:11.0" default-server="default-server" [...]>
<buffer-cache name="default"/>
<server name="default-server">
[...]
<host name="default-host" alias="localhost">
<filter-ref name="redirect-http-to-https" predicate="equals(%p,80)"/>
</host>
</server>
[...]
<filters>
<rewrite name="redirect-http-to-https" target="https://%{LOCAL_SERVER_NAME}%{REQUEST_URL}" redirect="true"/>
</filters>
</subsystem>
Note: For the Undertow exchange attributes (e.g. LOCAL_SERVER_NAME) refer to Undertow documentation. Further, the part predicate=\"equals(%p,80)\" in the filter-refchecks the requested port (%p -> just another Undertow exchange attribute) and if it equals to 80, then it triggers our redirect filter redirect-http-to-https - you can change the port 80 as per your need.
Related
We are migrating Jboss EAP 6.4 to Jboss EAP 7.2. Jboss 6.4 has system variables supported for keystore and password. Question here is how to use the SSL keystore/password from Jboss EAP 7.2 configuration file (standalone.xml) in Java application since there is no system variables supported for these two variables? Instead of these two variables, Jboss 7.2 uses Elytron framework for TLS.
I'm getting below error since I'm not able to pass these two system variables.
javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure.
Any help/pointers will be greatly appreciated. Thanks.
Jboss 7.2 Standalone.xml file:
<tls>
<key-stores>
<key-store name="default-trust-store">
<credential-reference store="keystore" alias="truststore_pw"/>
<implementation type="JKS"/>
<file path="../certs/truststore.jks"/>
</key-store>
<key-store name="httpsKS" alias-filter="">
<credential-reference store="keystore" alias="keystore_pw"/>
<implementation type="JKS"/>
<file path="/.../keystore.jks"/>
</key-store>
</key-stores>
<trust-managers>
<trust-manager name="default-trust-manager" key-store="default-trust-store"/>
</trust-managers>
<server-ssl-contexts>
<server-ssl-context name="httpsSSC" cipher-suite-filter="[ciphers here]" protocols="TLSv1.2" key-manager="httpsKM"/>
</server-ssl-contexts>
</tls>
I am load testing an application deployed on Wildfly 20 using JMeter. The observation is that under load of 300 requests/sec and more Wildfly server doesn't respond to few requests and those requests time out. On analyzing server logs (both system and application) and thread dumps nothing related to this is found and memory utilization is also normal.
On tuning some parameters such as Open File Descriptors, Wildfly IO threads/Task Max Threads, Max Connections and Apache configuration [details below] the number of requests getting timed out have significantly reduced but still a lot of requests time out.
Using Wildfly Runtime for monitoring during load testing one thing drew my attention that even on setting the Undertow's Http Max Connections parameter to a value lets say 1000 the Connection Count value (circled in screenshot below) never exceeds 256. On playing with this parameter a bit I saw that Http Max Connections parameter sets to any value less than 256 but never more. And it looks like once this limit is reached no new request is served as gets timed out until some connections are freed.
Is there is some other configuration (Apache or OS) which puts this limit or any other Wildfly configuration responsible for this? Am I missing anything?
[Used Wildfly Management Console for all configurations]
[from standalone.xml]
<subsystem xmlns="urn:jboss:domain:io:3.0">
<worker name="default" io-threads="300" task-max-threads="300"/>
<buffer-pool name="default"/>
</subsystem>
...
<subsystem xmlns="urn:jboss:domain:undertow:11.0" default-server="default-server" default-virtual-host="default-host" default-servlet-container="default" default-security-domain="other" statistics-enabled="true">
<buffer-cache name="default"/>
<server name="default-server">
<http-listener name="default" max-connections="500" socket-binding="http" max-cookies="100" record-request-start-time="true" redirect-socket="https" enable-http2="true"/>
<https-listener name="https" socket-binding="https" max-cookies="100" record-request-start-time="true" security-realm="ApplicationRealm" enable-http2="true"/>
<host name="default-host">
<location name="/" handler="welcome-content"/>
<filter-ref name="limit-connections"/>
<http-invoker security-realm="ApplicationRealm"/>
</host>
</server>
<servlet-container name="default">
<jsp-config/>
<websockets/>
</servlet-container>
<handlers>
<file name="welcome-content" path="${jboss.home.dir}/welcome-content"/>
</handlers>
<filters>
<request-limit name="limit-connections" max-concurrent-requests="10000" queue-size="10000"/>
</filters>
</subsystem>
[Apache configuration]
<IfModule mpm_worker_module>
ServerLimit 40
StartServers 2
MaxClients 1000
MinSpareThreads 25
MaxSpareThreads 75
ThreadsPerChild 25
MaxRequestsPerChild 0
</IfModule>
[Wildfly Runtime monitoring console during testing]
Maybe your server is not tuned to cater the maximum number of processes? Add the following to your /etc/security/limits.conf file:
jboss soft nproc 8000
jboss hard nproc 12000
jboss soft nofile 9999
jboss hard nofile 9999
I have a tomcat service configured to run on port 8380. This site was running fine until recently. I started to see this in the mod_jk.log on apache:
[3409:140197396281344] [info] jk_open_socket::jk_connect.c (627): connect to 127.0.0.1:8309 failed (errno=111)
[3409:140197396281344] [info] ajp_connect_to_endpoint::jk_ajp_common.c (995): Failed opening socket to (127.0.0.1:8309) (errno=111)
[3409:140197396281344] [error] ajp_send_request::jk_ajp_common.c (1630): (worker1) connecting to backend failed. Tomcat is probably not started or is listening on the wrong port (errno=111)
[3409:140197396281344] [info] ajp_service::jk_ajp_common.c (2623): (worker1) sending request to tomcat failed (recoverable), because of error during request sending (attempt=1)
[3409:140197396281344] [info] jk_open_socket::jk_connect.c (627): connect to 127.0.0.1:8309 failed (errno=111)
[3409:140197396281344] [info] ajp_connect_to_endpoint::jk_ajp_common.c (995): Failed opening socket to (127.0.0.1:8309) (errno=111)
[3409:140197396281344] [error] ajp_send_request::jk_ajp_common.c (1630): (worker1) connecting to backend failed. Tomcat is probably not started or is listening on the wrong port (errno=111)
[3409:140197396281344] [info] ajp_service::jk_ajp_common.c (2623): (worker1) sending request to tomcat failed (recoverable), because of error during request sending (attempt=2)
[3409:140197396281344] [error] ajp_service::jk_ajp_common.c (2643): (worker1) connecting to tomcat failed.
[3409:140197396281344] [info] jk_handler::mod_jk.c (2788): Service error=-3 for worker=worker1
I'm able to start up the tomcat service but the port it's supposed to be listening on is not accepting connections. nmap and netstat don't show the ports as available. I've also tried telnetting to to the port just in case but as i expected there's no response. I've stopped the firewall in case it's a iptables rule issue but that too had no effect. I've googled similar issues resolved by what i tried above but none of them apply to my issues. I'm not sure what else to try at this point and so I'm turning to you all!
For this particular host the CATALINA_BASE is from a mounted nfs export. I have 2 tomcat sites hosted on this server. For the sake of testing I've disabled one from starting up since they both have same issue. i figured if i can get 1 running the other will be work as well.
server.xml:
<Server port="8105" shutdown="SHUTDOWN">
<Listener className="org.apache.catalina.core.AprLifecycleListener" SSLEngine="on" />
<Listener className="org.apache.catalina.core.JasperListener" />
<Listener className="org.apache.catalina.core.JreMemoryLeakPreventionListener" />
<Listener className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener" />
<Listener className="org.apache.catalina.core.ThreadLocalLeakPreventionListener" />
<GlobalNamingResources>
<Resource name="UserDatabase" auth="Container"
type="org.apache.catalina.UserDatabase"
description="User database that can be updated and saved"
factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
pathname="conf/tomcat-users.xml" />
</GlobalNamingResources>
<Service name="Catalina">
<Connector port="8380" protocol="HTTP/1.1"
connectionTimeout="20000"
URIEncoding="UTF-8"
redirectPort="8443" />
<Connector port="8309" protocol="AJP/1.3" redirectPort="8443" />
<Engine name="Catalina" defaultHost="localhost">
<Realm className="org.apache.catalina.realm.LockOutRealm">
<Realm className="org.apache.catalina.realm.UserDatabaseRealm"
resourceName="UserDatabase"/>
</Realm>
<Host name="localhost" appBase="webapps"
unpackWARs="true" autoDeploy="true">
<Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"
prefix="localhost_access_log." suffix=".txt"
pattern="%h %l %u %t "%r" %s %b" />
</Host>
<Host name="mytomcat.site.org" appBase="webapps"
unpackWARs="true" autoDeploy="true">
</Host>
</Engine>
</Service>
</Server>
My worker.properties in apache. I have 2 workers 1 for each tomcat site.
workers.tomcat_home=/opt/tomcat
workers.java_home=/usr/local/jdk/jre
ps=/
# Define workers
worker.list=worker1
worker.list=worker2
# Set properties for worker1 - localhost
worker.worker1.type=ajp13
worker.worker1.host=127.0.0.1
worker.worker1.port=8309
worker.worker1.socket_keepalive=True
worker.worker1.socket_timeout=300
# Set properties for worker2 - localhost
worker.worker2.type=ajp13
worker.worker2.host=127.0.0.1
worker.worker2.port=8409
worker.worker2.socket_keepalive=True
worker.worker2.socket_timeout=300
Apache starts up ok and is serving static pages ok. tomcat starts ok but like i said it's not listening on any port.
Any help would be appreciated. TIA!
MORE INFO:
I ran the init script with -debug in the JAVA_OPTS var. and I see the following towards the bottom of the output that scrolls out to stdout:
...
user changed to 'daemon'
User 'daemon' validated
Attempting to locate Java Home in /usr/local/jdk
Attempting to locate VM configuration file /usr/local/jdk/jre/lib/jvm.cfg
Attempting to locate VM configuration file /usr/local/jdk/lib/jvm.cfg
Attempting to locate VM configuration file /usr/local/jdk/jre/lib/amd64/jvm.cfg
Found VM configuration file at /usr/local/jdk/jre/lib/amd64/jvm.cfg
Found VM server definition in configuration
Checking library /usr/local/jdk/jre/lib/amd64/server/libjvm.so
Found VM client definition in configuration
Checking library /usr/local/jdk/jre/lib/amd64/client/libjvm.so
Checking library /usr/local/jdk/lib/amd64/client/libjvm.so
Cannot locate library for VM client (skipping)
Found VM hotspot definition in configuration
Checking library /usr/local/jdk/jre/lib/amd64/hotspot/libjvm.so
Checking library /usr/local/jdk/lib/amd64/hotspot/libjvm.so
Cannot locate library for VM hotspot (skipping)
Found VM classic definition in configuration
Checking library /usr/local/jdk/jre/lib/amd64/classic/libjvm.so
Checking library /usr/local/jdk/lib/amd64/classic/libjvm.so
Cannot locate library for VM classic (skipping)
Found VM native definition in configuration
Checking library /usr/local/jdk/jre/lib/amd64/native/libjvm.so
Checking library /usr/localjdk/lib/amd64/native/libjvm.so
Cannot locate library for VM native (skipping)
Found VM green definition in configuration
Checking library /usr/local/jdk/jre/lib/amd64/green/libjvm.so
Checking library /usr/local/jdk/lib/amd64/green/libjvm.so
Cannot locate library for VM green (skipping)
Java Home located in /usr/local/jdk
+-- DUMPING JAVA HOME STRUCTURE ------------------------
| Java Home: "/usr/local/jdk"
| Java VM Config.: "/usr/local/jdk/jre/lib/amd64/jvm.cfg"
| Found JVMs: 1
| JVM Name: "server"
| "/usr/local/jdk/jre/lib/amd64/server/libjvm.so"
+-------------------------------------------------------
Running w/ LD_LIBRARY_PATH=/usr/local/jdk/jre/lib/amd64/server:/usr/local/jdk/jre/lib/amd64
wait_child 12035
redirecting stdout to /opt/www/mytomcat/logs/catalina-daemon.out and stderr to &1
Then it goes back to the shell prompt. If it completed successfully it would have additional lines scrolling out:
get_pidf: 5 in /opt/www/mytomcat/logs/catalina-daemon.pid
get_pidf: pid 97253
check_tmp_file: /tmp/97253.jsvc_up
get_pidf: 5 in /opt/www/mytomcat/logs/catalina-daemon.pid
get_pidf: pid 97253
check_tmp_file: /tmp/97253.jsvc_up
get_pidf: 5 in /opt/www/mytomcat/logs/catalina-daemon.pid
get_pidf: pid 97253
check_tmp_file: /tmp/97253.jsvc_up
get_pidf: 5 in /opt/www/mytomcat/logs/catalina-daemon.pid
get_pidf: pid 97253
check_tmp_file: /tmp/97253.jsvc_up
get_pidf: 5 in /opt/www/mytomcat/logs/catalina-daemon.pid
get_pidf: pid 97253
check_tmp_file: /tmp/97253.jsvc_up
get_pidf: 5 in /opt/www/mytomcat/logs/catalina-daemon.pid
get_pidf: pid 97253
check_tmp_file: /tmp/97253.jsvc_up
I also noticed both tomcat process are running as root when i expect the child process to be running as daemon. Not sure why this is happening but it may be related to the issue.
Based on the debug output I started to try a bunch of different things specifically around catalina-daemon.out and catalina-daemon.pid files. As it turns out because these files are on the nfs location it causes the odd issue of the service starting but not listening. Once I moved these to the local filesystem the service started up AND it was listening on the correct port again. I also noticed the child tomcat process switched to the unprivileged user daemon. The page is reachable again.
Not entirely sure why it was working before but all of a sudden its not when i made no changes to the server. Something must have changed in the nfs export from the zfs machine that was serving up the CATALINA_BASE. Somehow this may have exposed a bug that causes this issue on the tomcat server.
How previously my web-application setup with keycloak?
Installed keycloak in server machine
Deployed the web-application in Jboss server
Added the keycloak certificate into the Java Keytool in the machine where web-app running.
Made following changes in Standalone.xml file of JBOSS ES6.4
Added Extension
<extension module="org.keycloak.keycloak-adapter-subsystem"/>
Added security-domain
<security-domain name="keycloak">
<authentication>
<login-module code="org.keycloak.adapters.jboss.KeycloakLoginModule" flag="required"/>
</authentication>
</security-domain>
Added Connector
<connector name="https" protocol="HTTP/1.1" scheme="https" socket-binding="https" enable-lookups="false" secure="true">
<ssl name="ssl" key-alias="initcert" password="keycloak" certificate-key-file="C:\Code_Base\keycloak_certificates\keycloak_server\keycloak.jks" protocol="TLSv1,SSLv3,SSLv2" verify-client="false"/>
</connector>
Add SubSystem
<subsystem xmlns="urn:jboss:domain:keycloak:1.1">
<secure-deployment name="MyApp.war">
<realm>MYRealm</realm>
<resource>MyApp</resource>
<use-resource-role-mappings>true</use-resource-role-mappings>
<auth-server-url>https://<Server IP>/auth/</auth-server-url>
<ssl-required>NONE</ssl-required>
<credential name="secret">5c9f6ea3-3448-4569-b6c4-a9e62de4db52</credential>
</secure-deployment>
</subsystem>
So above changes working fine and my web application showing keycloak login page and after successful login its showing my web app Home page.
Now my requirement to run Keycloak in HA mode with load balancer NGINX So NGINX will be in front end and NGINX will decide which keycloak to point when hitting https://<NGINX IP ADDRESS>/auth and below things already done .
What already done?
Installed HA Keycloak in Node1 and Running.
Installed HA Keycloak in Node2 and Running.
Installed common Database(Used by above keycloak) in Node3 and Running.
Installed NGINX in Node4 and Running.
Configure Both keycloak(Step1 and Step2) with NGINX
Now NGINX url (https://<IP ADDRESS>/auth) showing login page of Keycloak and login also working.
Now can someone please tell me what changes i have to do in Jboss so it will work with new setup.
You basically need to do the same configuration, but in standalone-ha.xml instead of standalone.xml...
I'm currently migrating our code from Jboss7 to Wildfly10.
The Server itself starts up totaly fine.
When trying to connect our client with the working new wildfly10 server for ejb-remote calls it just won't work.
The only thing I get to work with is the following error:
org.jboss.ejb.client.remoting.ConfigBasedEJBClientContextSelector setupEJBReceivers
WARN: Could not register a EJB receiver for connection to remote-ip:8080
java.lang.RuntimeException: Operation failed with status WAITING at
org.jboss.ejb.client.remoting.IoFutureHelper.get(IoFutureHelper.java:94)
at
org.jboss.ejb.client.remoting.ConnectionPool.getConnection(ConnectionPool.java:80)
at
org.jboss.ejb.client.remoting.RemotingConnectionManager.getConnection(RemotingConnectionManager.java:51)
at
org.jboss.ejb.client.remoting.ConfigBasedEJBClientContextSelector.setupEJBReceivers(ConfigBasedEJBClientContextSelector.java:161)
at
org.jboss.ejb.client.remoting.ConfigBasedEJBClientContextSelector.getCurrent(ConfigBasedEJBClientContextSelector.java:118)
at
org.jboss.ejb.client.remoting.ConfigBasedEJBClientContextSelector.getCurrent(ConfigBasedEJBClientContextSelector.java:47)
at
org.jboss.ejb.client.EJBClientContext.getCurrent(EJBClientContext.java:281)
at
org.jboss.ejb.client.EJBClientContext.requireCurrent(EJBClientContext.java:291)
at
org.jboss.ejb.client.EJBInvocationHandler.doInvoke(EJBInvocationHandler.java:178)
at
org.jboss.ejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:146)
at com.sun.proxy.$Proxy2.connect(Unknown Source) at
de.cinovo.rcp.test.RemoteEJBClient.invokeStatelessBean(RemoteEJBClient.java:39)
at de.cinovo.rcp.test.RemoteEJBClient.main(RemoteEJBClient.java:25)
Exception in thread "main" java.lang.IllegalStateException:
EJBCLIENT000025: No EJB receiver available for handling
[appName:de.cinovo.tcc.server-ear,
moduleName:de-cinovo-tcc-server-ejb-6.0-SNAPSHOT, distinctName:]
combination for invocation context
org.jboss.ejb.client.EJBClientInvocationContext#180542f at
org.jboss.ejb.client.EJBClientContext.requireEJBReceiver(EJBClientContext.java:798)
at
org.jboss.ejb.client.ReceiverInterceptor.handleInvocation(ReceiverInterceptor.java:128)
at
org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:186)
at
org.jboss.ejb.client.EJBInvocationHandler.sendRequestWithPossibleRetries(EJBInvocationHandler.java:255)
at
org.jboss.ejb.client.EJBInvocationHandler.doInvoke(EJBInvocationHandler.java:200)
at
org.jboss.ejb.client.EJBInvocationHandler.doInvoke(EJBInvocationHandler.java:183)
at
org.jboss.ejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:146)
at com.sun.proxy.$Proxy2.connect(Unknown Source) at
de.cinovo.rcp.test.RemoteEJBClient.invokeStatelessBean(RemoteEJBClient.java:39)
at de.cinovo.rcp.test.RemoteEJBClient.main(RemoteEJBClient.java:25)
There is no error, warning, info or anything showing up in the server log, while trying to connect.
There is some action on the port via tcp while watching during a call attempt.
The realy funny part is:
If I use the same wildfly setup on my local machine, the exact same connection method works, but only while using localhost as the ip address within the jboss-ejb-client.properties.
As soon as I change the ip into 127.0.0.1 or my current ip address, it will fail with the same error as above.
Relevant information:
Wildfly will respond to a telnet on port 8080
Wildfly is the only service listening on 8080
My /etc/hosts is correctly configured
Changing the port doesn't fix the problem
Wildfly Version 10.1.0.Final
Relevant parts from my standalone.xml
<subsystem xmlns="urn:jboss:domain:remoting:3.0">
<endpoint/>
<http-connector name="http-remoting-connector" connector-ref="default" security-realm="ApplicationRealm"/>
</subsystem>
[...]
<subsystem xmlns="urn:jboss:domain:undertow:3.1">
<buffer-cache name="default"/>
<server name="default-server">
<http-listener name="default" socket-binding="http" redirect-socket="https" enable-http2="true"/>
[...]
</subsystem>
[...]
<interfaces>
<interface name="public">
<any-address/>
</interface>
</interfaces>
[...]
<socket-binding-group name="standard-sockets" default-interface="public" port-offset="${jboss.socket.binding.port-offset:0}">
<socket-binding name="http" interface="public" port="${jboss.http.port:8080}"/>
<socket-binding name="https" port="${jboss.https.port:8443}"/>
[...]
</socket-binding-group>
My jboss-ejb-client.properties
endpoint.name=client-endpoint
remote.connectionprovider.create.options.org.xnio.Options.SSL_ENABLED=false
remote.connections=default
remote.connection.default.host=<host-ip>
remote.connection.default.port=8080
remote.connection.default.connect.options.org.xnio.Options.SASL_POLICY_NOANONYMOUS=false
remote.connection.default.username=<usernmae>
remote.connection.default.password=<pswd>
Client-Code
final Hashtable jndiProperties = new Hashtable();
jndiProperties.put(Context.URL_PKG_PREFIXES, "org.jboss.ejb.client.naming");
final Context context = new InitialContext(jndiProperties);
[...]
return context.lookup("ejb:" + appName + "/" + moduleName + "/" + distinctName + "/" + beanName + "!" + viewClassName);
EJB-Client-Maven-Dependency :
<dependency>
<groupId>org.wildfly</groupId>
<artifactId>wildfly-ejb-client-bom</artifactId>
<version>10.1.0.Final</version>
<type>pom</type>
</dependency>
Anyone out there who had the same problem and knows what I'm doing wrong?
Looks like there is a missing definition in the standalone.xml in the socket.binding-group:
<outbound-socket-binding name="remote-ejb">
<local-destination socket-binding-ref="http"/>
</outbound-socket-binding>
So for everybody interested, here is the solution to my problem:
based on the comment by Steve C and the help of a friend, we figured out that the problem is not server based.
It appears that there are some antivirus programs, which do something with your HTTP messages, as soon as the HTTP-upgrade negotiations with the Wildfly/server are done. They seem to manipulate the sent/received packages, which leads to the problem in the client, as it is no longer able to understand the answers.
Therefore it never gets to react, since the package appears to have been lost - hence the IoFuture exception and EJB receiver not found.
Long story short: removing the antivirus program from our systems (in our case Bitdefender) leads to everything working as intended...