We Keep Coding

Java `https.proxyHost` and `https.proxyPort` succeed then fail when using google-cloud-storage

I have some code that unit tests fine using https.proxyHost and https.proxyPort to access Google Cloud Storage Buckets on on a corporate network which needs to go via a proxy:
log.info("resolving service....");
Storage storage = StorageOptions.newBuilder()
.setCredentials(
ServiceAccountCredentials.fromStream(
new FileInputStream(fullPath)))
.build()
.getService();
log.info("resolving bucket....");
bucket = storage.get(bucketName);
Yet when I run it in a larger app that starts a lot of other internal services (e.g., RMI) the proxy settings stop working.
Running as:
java -Dhttps.proxyHost=googleapis-dev.gcp.cloud.corporate -Dhttps.proxyPort=3128 ...
When it tries to resolve a bucket with the unit tested code it hangs for ages then throws:
com.google.cloud.storage.StorageException: Error getting access token for service account: oauth2.googleapis.com
at com.google.cloud.storage.spi.v1.HttpStorageRpc.translate(HttpStorageRpc.java:231) ~[htu-gcs-plugin.jar:?]
...
Caused by: java.io.IOException: Error getting access token for service account: oauth2.googleapis.com
at com.google.auth.oauth2.ServiceAccountCredentials.refreshAccessToken(ServiceAccountCredentials.java:444) ~[?:?]
at com.google.auth.oauth2.OAuth2Credentials.refresh(OAuth2Credentials.java:157) ~[?:?]
...
Caused by: java.net.UnknownHostException: oauth2.googleapis.com
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:184) ~[?:1.8.0_231]
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:172) ~[?:1.8.0_231]
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_231]
at java.net.Socket.connect(Socket.java:606) ~[?:1.8.0_231]
at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:666) ~[?:1.8.0_231]
I can get exactly the same error without proxy settings by adding this to the top of the main method:
String hostname = "https://oauth2.googleapis.com";
HttpURLConnection con = (HttpURLConnection) new URL(hostname).openConnection();
int code = con.getResponseCode();
System.out.println("> https://oauth2.googleapis.com returned: "+code);
Yet if the proxy settings are pass that runs fine then later throws the java.net.UnknownHostException as through the proxy settings are cleared.
To make things a bit more complex a custom URLClassLoader is used to load the code in question. Yet I have made a standalone app that uses the classloader and runs the code fine with the proxy settings passed as normal.
So it appears that something in the larger app is messing with the proxy system settings. Searching the codebase I can see no trace of that.
I have looked at https://googleapis.github.io/google-http-java-client/http-transport.html to see if there is way to plugin in a transport that has a proxy but cannot find a clear example.
Is there a way to coerce the use of a proxy when using google-cloud-storage?

To explicitly force a proxy to not rely upon the standard java System properties add the client libraries:
<!-- https://mvnrepository.com/artifact/com.google.http-client/google-http-client-apache-v2 -->
<dependency>
<groupId>com.google.http-client</groupId>
<artifactId>google-http-client-apache-v2</artifactId>
<version>1.37.0</version>
</dependency>
Then you can create a custom HttpTransportFactory with something like:
public class ProxyAwareTransportFactory implements HttpTransportFactory {
public static SSLContext trustAllSSLContext() throws Exception {
SSLContext sslContext = SSLContext.getInstance("TLSv1.2");
sslContext.init(null, null, null);
return sslContext;
}
#Override
public HttpTransport create() {
InetSocketAddress socketAddress = new InetSocketAddress(this.host,this.port);
Proxy proxy = new Proxy(Proxy.Type.HTTP, socketAddress);
try {
return new NetHttpTransport.Builder()
.setProxy(proxy)
.setConnectionFactory(new DefaultConnectionFactory(proxy) )
.setSslSocketFactory(trustAllSSLContext().getSocketFactory())
.setHostnameVerifier(new DefaultHostnameVerifier())
.build();
} catch (Exception e) {
final String msg = "Could not build HttpTransport due to; " + e.getMessage();
log.error(msg, e);
throw new RuntimeException(e);
}
}
}
You can then use it using something like:
StorageOptions.Builder builder = StorageOptions.newBuilder();
if( useProxy ) {
HttpTransportFactory httpTransportFactory = new ProxyAwareTransportFactory(proxyHost, proxyPort);
HttpTransportOptions options = HttpTransportOptions.newBuilder().setHttpTransportFactory(httpTransportFactory).build();
builder.setTransportOptions(options);
}
Storage storage = builder
.setCredentials(
ServiceAccountCredentials.fromStream(
new FileInputStream(fullPath)))
.build()
.getService();

Flutter https with self signed certificate

I am using flutter to connect with java java server implementation over https. I first tested it to be working using just http.
I then switched to https on the server side and pointed it at my self signed certificate I created using keytool.
Then I tried to connect to it using the http dart package. The resulted in the following exception...
Unhandled Exception: HandshakeException: Handshake error in client (OS
Error: E/flutter ( 7370): CERTIFICATE_VERIFY_FAILED: self signed
certificate(handshake.cc:354))
I am assuming I need to set my client to trust my servers self signed certificate. I have looked at the APi reference and could not figure out how to get this to happen...
My dart code in my flutter app is as follows...
void testMessage() {
var url = 'https://192.168.100.105:8443';
var response = await http.post(url, body: "{\"message_name\": \"TestMessage\", \"contents\": { \"field1\":\"blah\", \"field2\":\"blah\" }}");
print('Response status: ${response.statusCode}');
print('Response body: ${response.body}');
}

While Pascal's answer works, it only applies to the dart:io HttpClient.
To apply the badCertificateCallback to the http package's Client instances, do the following:
Create a class that overrides HttpOverrides in the following way:
class DevHttpOverrides extends HttpOverrides {
#override
HttpClient createHttpClient(SecurityContext context) {
return super.createHttpClient(context)
..badCertificateCallback = (X509Certificate cert, String host, int port) => true;
}
}
Then in your main, instantiate your class as the global HttpOverride:
HttpOverrides.global = new DevHttpOverrides();
This should make all Client ignore bad certificates and is therefore onl;y recommended in development.
Credit goes to this issue: https://github.com/dart-lang/http/issues/458

While developing you can use the badCertificateCallback callback of HttpClient and just return true. This will accept all bad certificates.
HttpClient client = HttpClient()
..badCertificateCallback = ((X509Certificate cert, String host, int port) => true);
To accept a specific bad certificate you may experiment with this code from here: https://github.com/dart-lang/http/issues/14#issuecomment-311184690
import 'dart:io';
import 'package:http/http.dart' as http;
bool _certificateCheck(X509Certificate cert, String host, int port) =>
host == 'local.domain.ext'; // <- change
HttpClient client = new HttpClient()
..badCertificateCallback = (_certificateCheck);

Amazing #Wecherowski, I think more safe way to do this is to check the other details and return true.
Something like:
HttpClient createHttpClient(SecurityContext? context)
{
return super.createHttpClient(context)
..badCertificateCallback = (X509Certificate cert, String host, int port)
{
if (host.isNotEmpty && host == 'xyz.example.com')
{
return true;
}
else
{ return false; }
};

If you use dio library https://pub.dev/packages/dio you can make requests from http to https

Cognito SRP Authentication JAVA SDK

Iam trying to authenticate a Java app with Cognito.
I have used for python the warrant library that worked very good. But i want to do the same in java now.
My Python function i used for authentication with the warrant library
def SRPauthentication(organizationAdmin,
password,
pool_id,
client_id,
client):
aws = AWSSRP(username=organizationAdmin,
password=password,
pool_id=pool_id,
client_id=client_id,
client=client)
tokens = aws.authenticate_user()
authorization_token= tokens['AuthenticationResult']['IdToken']
return authorization_token
with this i could easily acces some secured APIs.
Now i want to do the same with Java but i have problems.
This is my solution so far is this method:
public static void GetCreds()
{
AWSCognitoIdentityProvider identityProvider = AWSCognitoIdentityProviderClientBuilder.defaultClient();
AdminInitiateAuthRequest adminInitiateAuthRequest = new AdminInitiateAuthRequest().
withAuthFlow(AuthFlowType.USER_SRP_AUTH).
withClientId("234234234234").withUserPoolId("eu-central-1_sdfsdfdsf")
.addAuthParametersEntry("USERNAME", "UserK").
addAuthParametersEntry("PASSWORD","#######);
adminInitiateAuthRequest.getAuthFlow();
AdminInitiateAuthResult adminInitiateAuth = identityProvider.adminInitiateAuth(adminInitiateAuthRequest);
System.out.println(adminInitiateAuth.getAuthenticationResult().getIdToken());
}
When i run this i get an Exception:
Exception in thread "main" `com.amazonaws.services.cognitoidp.model.AWSCognitoIdentityProviderException: User: arn:aws:iam::XXXXXXXXXXXXXXXXX:user/khan is not authorized to perform: cognito-idp:AdminInitiateAuth on resource: arn:aws:cognito-idp:eu-central-1:XXXXXXXX:userpool/eu-central-1_XXXXXXX with an explicit deny (Service: AWSCognitoIdentityProvider; Status Code: 400; Error Code: AccessDeniedException; Request ID: 21be0b8e-adec-11e8-ad45-234234234)`
It says iam not authorized to perform this kind of instruction. So i guess iam doing something generally wrong. Because its working with my python code and in Java it recognizes my username from the credentials. The Cognito call should actually be independent from my aws credentials/useraccount right?
How to authenticate with Cognito using Java to get an Token to access secured aws services?
EDIT:
AWSCognitoIdentityProvider identityProvider = AWSCognitoIdentityProviderClientBuilder.standard()
.build();
InitiateAuthRequest adminInitiateAuthRequest = new InitiateAuthRequest()
.withAuthFlow(AuthFlowType.USER_SRP_AUTH)
.withClientId("XXXXXXXXXXXXXXXXX")
.addAuthParametersEntry("USERNAME", "user").
addAuthParametersEntry("PASSWORD","za$Lwn")
.addAuthParametersEntry("SRP_A",new AuthenticationHelper("eu-central-1XXXXXXXXX").getA().toString(16));
adminInitiateAuthRequest.getAuthFlow();
InitiateAuthResult adminInitiateAuth = identityProvider.initiateAuth(adminInitiateAuthRequest);
System.out.println(adminInitiateAuth);
I changed the AdminInitateAuthRequest to InitateAuthRequest. After that i had the Error missing SRP_A parameter that i somehow fixed with a similiar question here
And now i recive this :
{ChallengeName: PASSWORD_VERIFIER,ChallengeParameters: {SALT=877734234324234ed68300f39bc5b, SECRET_BLOCK=lrkwejrlewrjlewkjrewlrkjwerlewkjrewlrkjewrlkewjrlewkrjZ+Q==, USER_ID_FOR_SRP=user, USERNAME=user, SRP_B=43ecc1lwkerjwelrkjewlrjewrlkewjrpoipweoriwe9r873jr34h9r834hr3455f7d079d71e5012f1623ed54dd10b832792dafa3438cca3f59c0f462cbaee255d5b7c2werwerwerkjweorkjwerwerewrf5020e4f8b5452f3b89caef4a797456743602b80b5259261f90e52374adc06b456521a9026cce9c1cbe8b9ffd6040e8c1589d35546861422110ac7e38c1c93389b802a03e3e2e4a50e75d088275195f836f66e25f1a431dd56bb2},}
I have shorten the result with all the keys, but what to do next ?

Finally i could solve it with this code class.
There are multiple challenges involved in SRP authentication. The InitiateAuthRequest is one first request that is necessary.
This similiar question helped me :
stackoverflow
stackoverfow
String PerformSRPAuthentication(String username, String password) {
String authresult = null;
InitiateAuthRequest initiateAuthRequest = initiateUserSrpAuthRequest(username);
try {
AnonymousAWSCredentials awsCreds = new AnonymousAWSCredentials();
AWSCognitoIdentityProvider cognitoIdentityProvider = AWSCognitoIdentityProviderClientBuilder
.standard()
.withCredentials(new AWSStaticCredentialsProvider(awsCreds))
.withRegion(Regions.fromName(this.region))
.build();
InitiateAuthResult initiateAuthResult = cognitoIdentityProvider.initiateAuth(initiateAuthRequest);
if (ChallengeNameType.PASSWORD_VERIFIER.toString().equals(initiateAuthResult.getChallengeName())) {
RespondToAuthChallengeRequest challengeRequest = userSrpAuthRequest(initiateAuthResult, password);
RespondToAuthChallengeResult result = cognitoIdentityProvider.respondToAuthChallenge(challengeRequest);
//System.out.println(result);
System.out.println(CognitoJWTParser.getPayload(result.getAuthenticationResult().getIdToken()));
authresult = result.getAuthenticationResult().getIdToken();
}
} catch (final Exception ex) {
System.out.println("Exception" + ex);
}
return authresult;
}
private InitiateAuthRequest initiateUserSrpAuthRequest(String username) {
InitiateAuthRequest initiateAuthRequest = new InitiateAuthRequest();
initiateAuthRequest.setAuthFlow(AuthFlowType.USER_SRP_AUTH);
initiateAuthRequest.setClientId(this.clientId);
//Only to be used if the pool contains the secret key.
//initiateAuthRequest.addAuthParametersEntry("SECRET_HASH", this.calculateSecretHash(this.clientId,this.secretKey,username));
initiateAuthRequest.addAuthParametersEntry("USERNAME", username);
initiateAuthRequest.addAuthParametersEntry("SRP_A", this.getA().toString(16));
return initiateAuthRequest;
}

Redirect all network of Android App (which is using Libraries) through a proxy network (if available)

I am working on an Android application and I am using various libraries like Picaso, Retrofit, Crashlytics, etc. and my college WIFI have proxy on it.
I know how to use a proxy when sending a HTTP requests to a server, but all the libraries that I am using have there own HTTP requests, so overriding all their HTTP classes would be a headache, and I am not even sure how to do that.
So, is there a way to route all the traffic of the App through a proxy (when available), like any library or some HTTP request overriding all the outgoing requests.

I implemented this as below.
(I forgot the original source).
import java.net.Proxy;
import java.net.ProxySelector;
//setting Application-Wide proxy Values
private void setProxy() {
//Device proxy settings
ProxySelector defaultProxySelector = ProxySelector.getDefault();
Proxy proxy = null;
List<Proxy> proxyList = defaultProxySelector.select(URI.create("http://www.google.in"));
if (proxyList.size() > 0) {
proxy = proxyList.get(0);
Log.d("proxy", String.valueOf(proxy));
try {
String proxyType = String.valueOf(proxy.type());
//setting HTTP Proxy
if (proxyType.equals("HTTP")) {
String proxyAddress = String.valueOf(proxy.address());
String[] proxyDetails = proxyAddress.split(":");
String proxyHost = proxyDetails[0];
String proxyPort = proxyDetails[1];
Log.d("proxy", proxyType + " " + proxyHost + " " + proxyPort);
System.setProperty("http.proxyHost", proxyHost);
System.setProperty("http.proxyPort", proxyPort);
}
} catch (Exception e) {
e.printStackTrace();
}
}
}
Put this method in your base activity's onCreate method so it will get executed everytime user opens the application.

You can redirect specific network traffic (on a per application basis) as per the following link.
Network traffic via Proxy

InvalidAMIID.NotFound, AWS Error Message: The AMI ID 'ami-c1aaabb5' does not exist

I'm trying to launch a Ubuntu 12.04 LTS Server 64 bit, with EBS, programmatically. I've written the following code:
Placement placement = new Placement("eu-west-1b");
RunInstancesRequest runInstancesRequest = new RunInstancesRequest()
.withPlacement(placement).withInstanceType("t1.micro")
.withImageId("ami-c1aaabb5").withMinCount(1).withMaxCount(1)
.withSecurityGroupIds("testGroup").withKeyName("testKey")
.withUserData(Base64.encodeBase64String(startupScript.getBytes()));
RunInstancesResult runInstances = amazonEC2.runInstances(runInstancesRequest);
List<Instance> instances = runInstances.getReservation().getInstances();
However, I get this error:
AWS Error Code: InvalidAMIID.NotFound, AWS Error Message:
The AMI ID 'ami-c1aaabb5' does not exist
I've found that instance Id in Amazon AWS console and also at this URL: http://cloud-images.ubuntu.com/releases/precise/release-20121001/
What am I doing wrong? How can I do this right?

I've figured out, by default requests are sent to US Virginia data centre, setting the correct end point solved the problem.
public AmazonEC2 createAmazonEC2() {
AmazonEC2 ec2 = new AmazonEC2Client(createAwsCredentials());
ec2.setEndpoint("https://eu-west-1.ec2.amazonaws.com");
return ec2;
}
private AWSCredentials createAwsCredentials() {
AWSCredentials credentials = new BasicAWSCredentials(awsAccessKey, awsSecretKey);
return credentials;
}