I am developing a Spring Boot Application that uses HTTPS Only. I am using AWS services and the corresponding AWS Java SDK. How can I view the HTTP(S) request that the java sdk methods call on the backend of my application? I want to make sure when doing uploads to S3, etc, that everything is done over HTTPS only as the security of this application is important. A little confused on how to see when the backend of the application is interacting with the AWS services. Thanks in advance.
AWS uses HTTPS by default for all communications, and gives you options (such as VPC endpoints) that prevent traffic from leaving the AWS VPC.
Unfortunately, I couldn't find a reference in the Java SDK documentation that says it follows this practice. You can find guarantees for individual services (for example, S3). And it's implied that the SDK uses TLS by the page that describes how to enforce using TLS 1.2.
However, if you really want to be sure, you need to enable logging at the wire level.
Update in response to comment:
I ran the following program with debugging on:
public static void main(String[] argv)
throws Exception
{
S3Client client = S3Client.builder().build();
for (Bucket bucket : client.listBuckets().buckets())
{
System.out.println(bucket.name());
}
}
Looking at the logs, it's definitely making an HTTPS connection:
2021-11-17 17:51:24,802 [main] DEBUG request - Sending Request: DefaultSdkHttpFullRequest(httpMethod=GET, protocol=https, host=s3.amazonaws.com, port=443, encodedPath=/, headers=[amz-sdk-invocation-id, User-Agent], queryParameters=[])
...
2021-11-17 17:51:24,832 [main] DEBUG PoolingHttpClientConnectionManager - Connection request: [route: {s}->https://s3.amazonaws.com:443][total kept alive: 0; route allocated: 0 of 50; total allocated: 0 of 50]
2021-11-17 17:51:24,839 [main] DEBUG PoolingHttpClientConnectionManager - Connection leased: [id: 0][route: {s}->https://s3.amazonaws.com:443][total kept alive: 0; route allocated: 1 of 50; total allocated: 1 of 50]
2021-11-17 17:51:24,840 [main] DEBUG MainClientExec - Opening connection {s}->https://s3.amazonaws.com:443
2021-11-17 17:51:24,871 [main] DEBUG DefaultHttpClientConnectionOperator - Connecting to s3.amazonaws.com/52.216.251.102:443
2021-11-17 17:51:24,871 [main] DEBUG SdkTlsSocketFactory - Connecting socket to s3.amazonaws.com/52.216.251.102:443 with timeout 2000
2021-11-17 17:51:24,902 [main] DEBUG SdkTlsSocketFactory - Enabled protocols: [TLSv1.2]
That's followed by the TLS negotiation, and then the actual request. One thing that may be confusing, if you're not familiar with the HTTP protocol, is this:
2021-11-17 17:51:25,022 [main] DEBUG MainClientExec - Executing request GET / HTTP/1.1
That's the HTTP request line, and the "HTTP/1.1" indicates the protocol version. This information is sent over the secure connection.
Related
Has somebody succeeded in setting up OIDC connection to Keycloack in a Quarkus app deployed in a Kubernetes cluster ?
Could you clarify how does the connection-delay (and other related parameters) work ?
(Here is the documentation I tried to follow)
In our env (Quarkus 1.13.3.Final, Keycloak 12.0.4) we have such config:
quarkus.oidc.connection-delay: 6M
quarkus.oidc.connection-timeout: 30S
quarkus.oidc.tenant-id: testTenant-01
And these msgs appear in pod's log when it's being started:
2021-07-26 14:44:22,523 INFO [main] [OidcRecorder.java:264] -
Connecting to IDP for up to 180 times every 2 seconds
2021-07-26
14:44:24,142 DEBUG [vert.x-eventloop-thread-1]
[OidcRecorder.java:115] - 'testTenant-01' tenant initialization has failed:
'OpenId Connect Provider configuration metadata is not configured and
can not be discovered'. Access to resources protected by this tenant
will fail with HTTP 401.
(... following log comes later as the pod is running ...)
2021-07-27 06:11:54,261 DEBUG [vert.x-eventloop-thread-0]
[DefaultTenantConfigResolver.java:112] - Tenant 'null' is not
initialized
2021-07-27 06:11:54,262 ERROR
[vert.x-eventloop-thread-0] [QuarkusErrorHandler.java:101] - HTTP
Request to /q/health/live failed, error id:
89f83d1d-894c-4fed-9995-0d42d60cec17-2: io.quarkus.oidc.OIDCException:
Tenant configuration has not been resolved at
io.quarkus.oidc.runtime.OidcAuthenticationMechanism.resolve(OidcAuthenticationMechanism.java:61)
at
io.quarkus.oidc.runtime.OidcAuthenticationMechanism.authenticate(OidcAuthenticationMechanism.java:40)
at
io.quarkus.oidc.runtime.OidcAuthenticationMechanism_ClientProxy.authenticate(OidcAuthenticationMechanism_ClientProxy.zig:189)
at
io.quarkus.vertx.http.runtime.security.HttpAuthenticator.attemptAuthentication(HttpAuthenticator.java:100)
at
io.quarkus.vertx.http.runtime.security.HttpAuthenticator_ClientProxy.attemptAuthentication(HttpAuthenticator_ClientProxy.zig:157)
at
io.quarkus.vertx.http.runtime.security.HttpSecurityRecorder$2.handle(HttpSecurityRecorder.java:101)
at
io.quarkus.vertx.http.runtime.security.HttpSecurityRecorder$2.handle(HttpSecurityRecorder.java:51)
at
io.vertx.ext.web.impl.RouteState.handleContext(RouteState.java:1038)
Questions:
Any way how to find out what metadata are missing ?
Can I somehow change the 2s period between connection attempts ?
Any relation between connection-delay and connection-timeout ?
It failed after cca 2s - does it mean that it fails immediately in the 1st attempt, or has it finished 180 attempts so fast ?
Does DefaultTenantConfigResolver get tenant from different resource than OidcRecorder in initialization, i.e. should tenant be configured at multiple places ?
Finally made it work. Caused by incorrect auth-server-url which is not clear at all from the log messages.
quarkus.oidc.client-id: my-app
quarkus.oidc.enabled: true
quarkus.oidc.connection-delay: 6M
quarkus.oidc.connection-timeout: 30S
quarkus.oidc.tenant-id: testTenant-01
quarkus.oidc.auth-server-url: ${keycloak.url}/auth/realms/${quarkus.oidc.tenant-id}
The URL format is emphasized in Quarkus doc: Note if you work with Keycloak OIDC server, make sure the base URL is in the following format: https://host:port/auth/realms/{realm} where {realm} has to be replaced by the name of the Keycloak realm
In my team we are using ActiveMQ 5.15.11 in production and our messages consumers are connected through HTTP protocol. Since we have network issues on the consumer side, we tried to disable the InactivityMonitor in the activemq.xml (see below)
<transportConnectors>
<transportConnector name="openwire" uri="tcp://host:61616?maximumConnections=1000&wireFormat.maxFrameSize=104857600"/>
<transportConnector name="http" uri="http://host:61617?wireFormat.maxInactivityDuration=0"/>
</transportConnectors>
It seems like the "wireFormat.maxInactivityDuration=0"is not applied since we have logs from InactivityMonitor in the consumer :
2020-03-18 00:00:03,426 [DEBUG] - - - WriteChecker: 10000ms elapsed since last write check.
2020-03-18 00:00:03,426 [DEBUG] - - - Running WriteCheck[https://host:443/activemq]
2020-03-18 00:00:03,509 [DEBUG] - - - WriteChecker: 10000ms elapsed since last write check.
2020-03-18 00:00:03,509 [DEBUG] - - - Running WriteCheck[https://host:443/activemq]
2020-03-18 00:00:03,514 [DEBUG] - - - WriteChecker: 10000ms elapsed since last write check.
2020-03-18 00:00:03,514 [DEBUG] - - - Running WriteCheck[https://host:443/activemq]
2020-03-18 00:00:03,642 [DEBUG] - - - WriteChecker: 10000ms elapsed since last write check.
2020-03-18 00:00:03,642 [DEBUG] - - - Running WriteCheck[https://host:443/activemq]
2020-03-18 00:00:03,706 [DEBUG] - - - WriteChecker: 10000ms elapsed since last write check.
2020-03-18 00:00:03,706 [DEBUG] - - - Running WriteCheck[https://host:443/activemq]
2020-03-18 00:00:03,738 [DEBUG] - - - WriteChecker: 10000ms elapsed since last write check.
2020-03-18 00:00:03,738 [DEBUG] - - - Running WriteCheck[https://host:443/activemq]
Furthermore, as soon as we get long network delay, we also get log :
2020-03-01 20:43:17,578 [WARN ] - - - Transport (https://host:443/activemq) failed , attempting to automatically reconnect: {}
org.apache.activemq.transport.InactivityIOException: Channel was inactive for too (>30000) long: https://host:443/activemq
at org.apache.activemq.transport.AbstractInactivityMonitor$5.run(AbstractInactivityMonitor.java:246)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Notice 1 : ActiveMQ clients are JMS Consumers.
Notice 2 : we tried to disable InactivityMonitor on TCP transport and it works well with this protocol.
Thanks for your help in advance.
For the HTTP transport I don't think that the standard OpenWire protocol negotiation step is performed which likely means that the server side configuration of max-inactivity-duration is visible at the client side. This results in the client using the defaults for the inactivity checks which I recall as being 30 seconds or so which it then splits into smaller chunks for read and write checks.
You would likely need to configure the client end to not do the checking using the same wireformat options. I don't think any of that is extensively tested though so there could likely be issues with altering defaults over the HTTP transport wrappers. If it continues to misbehave then likely you'd need to open a JIRA with the ActiveMQ project to get something implemented to let you control it better.
On the client side you should be able to disable the inactivity monitor with transport.useInactivityMonitor=false or maybe just useInactivityMonitor=false on the URI.
SSL debug logs are noisy and verbose; is there a way of enabling SSL debug for just a single Socket in Java ?
(Rather than the usual way of enabling JVM-wide debugging using "-Djavax.net.debug=all" etc).
I want to test pulling data from Apache HBase with a Java application. The application will use SQL-like queries via a JDBC to Apache Phoenix.
I've set up my Hadoop "cluster" on one machine using Ambari and the HortonWorks HDP 2.5 platform. I've also Kerberized the environment using Ambari's wizard, where my KDC is a seperate machine running Windows Active Directory.
Ambari shows no errors, and I am able to use sqlline.py to successfully make SQL-like calls to HBase through Phoenix. I set up some example tables this way (cf. HortonWorks Phoenix & ODBC tutorial, although I had to kinit etc. first).
However, I am having problems creating a JDBC datasource to be used by the Java application. In my case, I am planning to host the webapp on WildFly 10.1 and I am developing with Eclipse JEE with the JBoss Tools plugin.
These are the steps I used to create the datasource:
Datasource Explorer > Database Connections > New...
Connection Profile: Generic JDBC
URL: jdbc:phoenix:hdfs.eaa.local:2181/hbase-secure:HTTP/hbase.eaa.local#EAA.LOCAL:jboss.server.temp.dir/spnego.service.keytab
Username: hbase -I'm unsure what to put here-
Driver: I've created a new driver of the type "Generic JDBC Driver" and I had to add JAR files for all of the dependencies of phoenix-core-[version].jar. The Driver Class is org.apache.phoenix.jbdc.PhoenixDriver.
I got the connection string from an extant post in the HortonWorks community, which is why it includes the Kerberos principal and keytab used for the connection.
When I try to test the datasource connection, it churns for about 5 minutes before spitting out an error message (after something like 35 attempts). The client returns Java exceptions that the sockets are in a "closing state", and the Zookeeper logs are less helpful:
INFO [SyncThread:0:ZooKeeperServer#617] - Established session 0x157aef451560217 with negotiated timeout 40000 for client /192.168.40.3:52674
INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory#197] - Accepted socket connection from /192.168.40.41:43860
INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn#827] - Processing ruok command from /192.168.40.41:43860
INFO [Thread-1448:NIOServerCnxn#1007] - Closed socket connection for client /192.168.40.41:43860 (no session established for client)
INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory#197] - Accepted socket connection from /192.168.40.41:43922
INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer#868] - Client attempting to establish new session at /192.168.40.41:43922
INFO [SyncThread:0:ZooKeeperServer#617] - Established session 0x157aef451560218 with negotiated timeout 40000 for client /192.168.40.41:43922
INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:SaslServerCallbackHandler#118] - Successfully authenticated client: authenticationID=hbase/hdfs.eaa.local#EAA.LOCAL; authorizationID=hbase/hdfs.eaa.local#EAA.LOCAL.
INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:SaslServerCallbackHandler#134] - Setting authorizedID: hbase
INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer#964] - adding SASL authorization for authorizationID: hbase
INFO [ProcessThread(sid:0 cport:-1)::PrepRequestProcessor#494] - Processed session termination for sessionid: 0x157aef451560218
INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn#1007] - Closed socket connection for client /192.168.40.41:43922 which had sessionid 0x157aef451560218
INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory#197] - Accepted socket connection from /192.168.40.41:44008
INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn#827] - Processing ruok command from /192.168.40.41:44008
INFO [Thread-1449:NIOServerCnxn#1007] - Closed socket connection for client /192.168.40.41:44008 (no session established for client)
NB. 192.168.40.3 is the VPN server, which my host machine is using to tunnel into the environment with the Hadoop cluster. 192.168.40.41 is the machine running the cluster, hdfs.eaa.local.
There are plenty of accepted socket connections which are then immediately closed. Occasionally the client authenticates successfully (so I'm confident in my Kerberos settings) but then there is a session termination immediately afterward.
I've also tried to deploy the Datasource directly in WildFly with jboss-cli and standalone.xml and module.xml modifications. But I get lots of problems with missing dependencies that I'm not sure how to resolve without creating a new module for each required JAR (and there are a lot) for phoenix-core-[version].jar. I followed this guide.
What can I do to fix the issue or diagnose further? I've been pulling my hair out for a couple of days now.
You need to add hbase-site.xml and core-site.xml to your classpath.
See How to connect to a Kerberos-secured Apache Phoenix data source with WildFly? for more information.
I am trying to get tweets via hbc-twitter4j-v3 . Example code is : https://github.com/twitter/hbc/blob/master/hbc-example/src/main/java/com/twitter/hbc/example/Twitter4jSampleStreamExample.java
For enabling authentication on proxy, I have also set system properties for host,port and authentication. But it is showing following error-
[main] INFO com.twitter.hbc.httpclient.BasicClient - New connection executed: hosebird-client-0, endpoint: /1.1/statuses/sample.json?delimited=length&stall_warnings=true
[hosebird-client-io-thread-0] INFO com.twitter.hbc.httpclient.ClientBase - hosebird-client-0 Establishing a connection
[main] INFO com.twitter.hbc.httpclient.BasicClient - Stopping the client: hosebird-client-0, endpoint: /1.1/statuses/sample.json?delimited=length&stall_warnings=true
[main] INFO com.twitter.hbc.httpclient.ClientBase - hosebird-client-0 exit event - Stopped by user: waiting for 5000 ms
[main] WARN com.twitter.hbc.httpclient.ClientBase - hosebird-client-0 Client thread failed to finish in 5000 millis
[main] INFO com.twitter.hbc.httpclient.BasicClient - Successfully stopped the client: hosebird-client-0, endpoint: /1.1/statuses/sample.json?delimited=length&stall_warnings=true
[hosebird-client-io-thread-0] WARN com.twitter.hbc.httpclient.ClientBase - hosebird-client-0 Unknown host - stream.twitter.com
[hosebird-client-io-thread-0] WARN com.twitter.hbc.httpclient.ClientBase - hosebird-client-0 failed to establish connection properly
[hosebird-client-io-thread-0] INFO com.twitter.hbc.httpclient.ClientBase - hosebird-client-0 Done processing, preparing to close connection
[hosebird-client-io-thread-0] INFO com.twitter.hbc.httpclient.ClientBase - hosebird-client-0 Shutting down httpclient connection manager
Any help??
Thanks in advance
Hopefully I haven't overlooked something but this is how it appears to me...
If by setting properties you mean the http.proxy* ones, I don't think it will work as hosebird-client uses Apache's HTTP client under the hood which doesn't seem to use them.
From a cursory glance at the code, specifically around the ClientBuilder, it doesn't look like hbc supports proxy configuration - perhaps they have a good reason not to or just don't need the feature themselves, maybe try requesting it?
It looks like one of the ways you can get HttpClient to use a proxy is by adding it to the HttpParams object, e.g.:
HttpParams params = ...
HttpHost proxy = new HttpHost(hostname, port);
params.setParameter(ConnRoutePNames.DEFAULT_PROXY, proxy);
Whilst the HttpParams object isn't exposed anywhere you could potentially extend the ClientBuilder in order to supply your proxy configuration. If you look at the ClientBuilder#build() method, you can see where the HttpParams object is being set up. Good luck!
EDIT: Additionally, this issue indicates there are no plans to add proxy support directly in hbc.