We Keep Coding

`ERR_HTTP2_PROTOCOL_ERROR` when pipe character (|) in GET request query parameter in Wildfly 21

We've recently migrated a Spring REST application from Wildfly 15.0.1.Final to Wildfly 21.0.0.Final which apparently introduced an issue with GET requests: whenever we have a | (pipe) character in the query parameter string of the GET request, the request returns no response and we get ERR_HTTP2_PROTOCOL_ERROR.
I know that '|' (pipe) character is unsafe according to the RFC1738 specification of HTTP, while RFC3986 allows for the encoding of Unicode characters.
I would like this to keep working though, as we have external clients sending requests with | character in the query parameter, and currently if we would move to the current Wildfly 21 config, those requests would fail.
The same configuration was working fine on Wildfly 15.0.1.Final.
I have these in standalone.xml with no avail:
<system-properties>
<property name="org.apache.catalina.connector.URI_ENCODING" value="UTF-8"/>
<property name="org.apache.catalina.connector.USE_BODY_ENCODING_FOR_QUERY_STRING" value="true"/>
</system-properties>
<http-listener name="default" socket-binding="http" allow-unescaped-characters-in-url="true" redirect-socket="https" enable-http2="true" url-charset="UTF-8" />
<https-listener name="https" socket-binding="https" max-post-size="1048576000" allow-unescaped-characters-in-url="true" ssl-context="LocalhostSslContext" enable-http2="true" url-charset="UTF-8" />
...and this in standalone.conf.bat:
set "JAVA_OPTS=%JAVA_OPTS% -Dorg.apache.catalina.connector.URI_ENCODING=UTF-8"
The very same code on the very same VM, with (migrated) config works fine on Wildfly 15.0.1.Final but throws the ERR_HTTP2_PROTOCOL_ERROR in Wildfly 21.0.0.Final whenever I have a | in the request. In these cases it looks like the request is not even hitting my breakpoints.
I can programmatically do a dirty fix by URL encoding all | in our $.ajaxSetup, but this only fixes requests originating from the server itself, and not requests that are coming externally with | in their GET request query params.
The dirty (and insufficient) fix:
$.ajaxSetup({
beforeSend: function (jqXHR, settings) {
settings.url = settings.url.replace(/\|\|/g, "%7C%7C");
}
});
Has anyone encountered this issue?
Full standalone.xml (with sensitivre info masked) here.
EDIT: In the meantime I noticed that this issue only happens when I hit endpoints defined in Windows hosts file. When I go through our company's load balancer, it works fine.
So e.g. http://localhost.myproduct.com is not working from SERVER1 if 127.0.0.1 localhost.myproduct.com is in hosts file, but https://server1.myproduct.com that hits the very same server works fine, if the endpoint is routed through the load balancer.

I saw a few related postings around this time, all of which seem to have gone unanswered.
I've also encountered a similar issue with Wildfly 23.0.0.Final, which was a problem with http/2 handling - there is a fix for that: UndertowOptions.ALLOW_UNESCAPED_CHARACTERS_IN_URL has no effect for HTTP/2, but as of this reply AFAIK is not yet released in a Wildfly build.
Setting enable-http2="false" on the listeners - while not ideal - worked around the problem for me.
It could be that your load balancer is doing http/1.1 on the backend which would be why you don't encounter the problem when routing through it.

http.nonProxyHosts ignored on localhost only

I run a proxy on localhost:2080 for debugging purposes. I can see trafic going through my proxy using either:
curl http://localhost:8888/stuff --proxy localhost:2080
curl http://some.server.com:8888/stuff --proxy localhost:2080
Using a Spring application, or a bare bones HttpURLConnectionExample Java application, I see requests go through the proxy when I call some.server.com, but never on localhost, even though I set http.nonProxyHosts to a random value (to avoid the default of localhost).
I made sure my JVM arguments were taken into accounts by adding the following code:
System.out.println("java.version="+props.getProperty("java.version"));
System.out.println("http.proxyHost="+props.getProperty("http.proxyHost"));
System.out.println("http.proxyPort="+props.getProperty("http.proxyPort"));
System.out.println("http.nonProxyHosts="+props.getProperty("http.nonProxyHosts"));
That produces the following output:
java.version=1.8.0_131
http.proxyHost=localhost
http.proxyPort=2080
http.nonProxyHosts=dummy.snafu.com
https.proxyHost=localhost
https.proxyPort=2080
https.nonProxyHost=dummy.snafu.com
Testing 1 - Send Http GET request
Sending 'GET' request to URL : http://localhost:8888/stuff
Response Code : 200
hello
Why are HTTP requests to localhost not going through my proxy when run with the following JVM arguments?
-Dhttp.proxyHost=localhost
-Dhttp.proxyPort=2080
-Dhttp.nonProxyHosts=dummy.snafu.com

This is a limitation of DefaultProxySelector, It always appends local Uris(localhost, 127.*, etc) to whatever you configure as nonProxyHosts, essentially bypassing proxy for all local Uris(localhost|127. *|[::1]|0.0.0.0|[::0]). Fortunately, there is a workaround by creating your own proxy selector and registering it as default.

Https web service call failure

We have implemented webservice call using JAX-WS RI 2.1.6 in JDK 6 now problem comes when we enable https webservice call stops reaching server and java reports following error,
javax.xml.ws.WebServiceException: java.io.IOException: Async IO
operation failed (3), reason: RC: 55 The specified network resource
or device is no longer available.
Now I have tested this within SoapUI and response from the service is received there.
Looked into various solution where it tells us to provide timeout settings but nothing seems work.
#WebEndpoint(name = "RulesSoap")
public RulesSoap getRulesSoap() {
((BindingProvider)super.getPort(new QName("urn:decision:Rules", "RulesSoap"), RulesSoap.class)).getRequestContext().put("com.sun.xml.internal.ws.connect.timeout", 1000);
((BindingProvider)super.getPort(new QName("urn:decision:Rules", "RulesSoap"), RulesSoap.class)).getRequestContext().put("com.sun.xml.internal.ws.request.timeout", 1000);
return super.getPort(new QName("urn:decision:Rules", "RulesSoap"), RulesSoap.class);
}
And just for information JAX-WS implementation is following few simple lines,
of course we submit all necessary data into respective stubs and all but I am not submitting here because our http calls are getting through,
Rules rules = new Rules(new URL(url), new QName("urn:decision:Rules", "Rules"));
RulesSoap rulesSoap = rules.getRulesSoap();
CorticonResponse response = rulesSoap.processRequest(request);
Note : Our application server WebSphere Application Server and Version 7.0.0.19
Thanks in Advance.

After lots of efforts we resolved this. I will provide steps if anything related to this happens how to find root cause,
Step 1 :
First of all we enabled soap tracing in WebSphere Application Server by following setting,
Admin Console > Servers > Server Types > WebSphere Application Servers >
{your server} > Troubleshooting > Change Log Detail Levels > Runtime
In run time please put this , *=info: com.ibm.ws.websvcs.*=all: org.apache.axis2.jaxws.*=all
This step will create trace.log file in your logs folder.
Now any web service request which goes out of your server will add logs to this file and necessary props like endpoint, request, response etc.
Step 2 :
Reading this trace.log file we found following endpoint,
PropertyValid 1 org.apache.axis2.jaxws.client.PropertyValidator validate validate property=(javax.xml.ws.service.endpoint.address) with value=(http://uxm.solutions.lnet.com:9445/axis/dswsdl/Rules/1/0)
HTTPConnectio 3 resetConnection : http://uxm.solutions.lnet.com:9445/axis/dswsdl/Rules/1/0 Persistent : true
Now if you notice here that our soap has endpoint address javax.xml.ws.service.endpoint.address where protocol is still using http which causes to fail ssl handshake.
Step 3 :
Solution for this is to override endpoint inside your soap stubs which can be implemented by adding following line,
RulesSoap rulesSoap = rules.getRulesSoap();
((BindingProvider)rulesSoap).getRequestContext().put(BindingProvider.ENDPOINT_ADDRESS_PROPERTY, "https://uxm.solutions.lnet.com:9445/axis/dswsdl/Rules/1/0");
Conclusion :
So here is what i think even we pass https url while we are creating objects but still does not take this https url on runtime, to me this looks like stubs creation issue with JAX-WS.
Thanks.

What protocol /ciphers are you using? You have mentioned there is connection to webservice on WAS7 with JDK6 and Java 6 does not support TLS1.2 (and TLS1.1 only from certain fixpack).
See this:
How to use TLS 1.2 in Java 6

GlassFish Security Realm, Active Directory and Referral

I've setup up a Security Realm in Glassfish to authenticate against an Active Directory server. The configuration of the realm is as follows:
Class Name: com.sun.enterprise.security.auth.realm.ldap.LDAPRealm
JAAS context: ldapRealm
Directory: ldap://172.16.76.10:389/
Base DN: dc=smallbusiness,dc=local
search-filter: (&(objectClass=user)(sAMAccountName=%s))
group-search-filter: (&(objectClass=group)(member=%d))
search-bind-dn: cN=Administrator,CN=Users,dc=smallbusiness,dc=local
search-bind-password: abcd1234!
The realm is functional and I can log-in, but when ever I log in I get the following error in the log:
SEC1106: Error during LDAP search with filter [(&(objectClass=group)(member=CN=Administrator,CN=Users,dc=smallbusiness,dc=local))].
SEC1000: Caught exception.
javax.naming.PartialResultException: Unprocessed Continuation Reference(s); remaining name 'dc=smallbusiness,dc=local'
at com.sun.jndi.ldap.LdapCtx.processReturnCode(LdapCtx.java:2820)
....
....
ldaplm.searcherror
While searching for a solution I found that it was recommended to add java.naming.referral=follow to the properties of the realm. However, after I add this it takes 20 minutes for GlassFish to authenticate against Active Directory. I suspect it is a DNS problem on the Active Directory server. The Active Directory server is a vanilla Windows Server 2003 setup in a Virtual Machine.
Any help/recommendation is highly appreciated!

This is the configuration I use in my domain.xml file, it might be of some interrest to you :
<auth-realm classname="com.sun.enterprise.security.auth.realm.ldap.LDAPRealm" name="ldapRealm">
<property name="search-bind-password" value="Demodemo01"/>
<property name="search-bind-dn" value="Administrator"/>
<property name="search-filter" value="(&(objectClass=user)(sAMAccountName=%s)(!(userAccountControl:1.2.840.113556.1.4.803:=2)))"/>
<property name="group-search-filter" value="(&(objectClass=group)(member=%d))"/>
<property name="jaas-context" value="ldapRealm"/>
<property name="base-dn" value="CN=Users,DC=saierp,DC=net"/>
<property name="directory" value="ldap://192.168.1.38:389"/>
</auth-realm>
Specially, make sure to add the userAccountControl to your filter, otherwise, disabled accounts in AD will be allowed to connect.

This was maddening trying to solve this... Glassfish 3.0.1 trying to connect with Windows, and getting the above error.
I'm not a Windows or LDAP whiz at all... but finally found this:
http://forum.springsource.org/showthread.php?t=87673
And the very last line is the key: use the "Global Catalog Port" - instead of 389, it is 3268 by default. And the exception disappears.
Why?
Who cares?
(well, ok, I'm going to read about it now.)

Hopefully, you've resolved this, but just in case:
I used 'objectCategory' in place of 'objectClass' as I read the former are indexed, hence faster.
I had to add this property:
property name="assign-groups" value="Domain Users"
where "Domain Users" is the group in AD that all of our users are placed into. This must match the value in sun-web.xml for security-role-mapping.
Later, I was able to create a specific group for this application and make the appropriate changes.

Jolokia - Origin null is not allowed to call this agent

{"stacktrace":"java.lang.Exception: Origin null is not allowed to call
this agent\n\tat
org.jolokia.http.HttpRequestHandler.handleThrowable(HttpRequestHandler.java:242)\n\tat
org.jolokia.jvmagent.handler.JolokiaHttpHandler.doHandle(JolokiaHttpHandler.java:243)\n\tat
org.jolokia.jvmagent.handler.JolokiaHttpHandler.handle(JolokiaHttpHandler.java:178)\n\tat
com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:79)\n\tat
sun.net.httpserver.AuthFilter.doFilter(AuthFilter.java:83)\n\tat
com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:82)\n\tat
sun.net.httpserver.ServerImpl$Exchange$LinkHandler.handle(ServerImpl.java:675)\n\tat
com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:79)\n\tat
sun.net.httpserver.ServerImpl$Exchange.run(ServerImpl.java:647)\n\tat
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\n\tat
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\n\tat
java.lang.Thread.run(Thread.java:748)\n","error_type":"java.lang.Exception","error":"java.lang.Exception
: Origin null is not allowed to call this agent","status":403}
I get this error when I try to query my "jolokia" agent.
curl http://localhost:8778/jolokia/list
I've launched my java application (kibana) with the jolokia agent, as explained in the Manual https://jolokia.org/reference/html/agents.html#agents-jvm.
java -javaagent:agent.jar=port=8778,host=localhost
I can see (ps -aux) that the process launched with the jolokia agent in between the Java Arguments.
I have also tried to deploy the jolokia war in my Tomcat /webapps. I've edited the user.xml file to add the User Jolokia, but I get the same result
The only result I get by googling the error seems to be the Jolokia code, at line 287, which seems to imply the host or address are wrong, but i'm doing everything from localhost, which should be allowed.
https://github.com/rhuss/jolokia/blob/master/agent/core/src/main/java/org/jolokia/http/HttpRequestHandler.java
Am I missing some settings? which is the best way to test it? I have zero experience with Java applications and Jolokia.

Jolokia default setting is set to block all CORS, in apache-activemq/webapps/api/WEB-INF/classes/jolokia-access.xml file.
<cors>
<strict-checking/>
</cors>
You can disable it by removing above config or adding allowed origins before ** strict-checking**.
<cors>
<!-- Allow cross origin access from www.jolokia.org ... -->
<allow-origin>http://www.jolokia.org</allow-origin>
<!-- ... and all servers from jmx4perl.org with any protocol ->
<allow-origin>*://*.jmx4perl.org</allow-origin>
<!-- Check for the proper origin on the server side, too -->
<strict-checking/>
</cors>
For more details please refer 4.1.5. Cross-Origin Resource Sharing (CORS) restrictions

Depending on your current configuration need to be changed.
Based on your exact error line, you need to add "Origin" header in your http request.
For example - curl http://localhost:8778/jolokia/list -H "Origin:http://localhost"
Remember, if you have default CORS is enabled, add exception with <allow-origin> under <cors> tag.