I want to port the open source project android-market-api from java to python. But now I've stucked in the https problem.
When I use HTTPSConnection to request https://android.clients.google.com/market/api/ApiRequest, it always return 403. And after some debug, I think may be there's something difference between java HttpsURLCollection and python HTTPSConnection.
Python port:
headers = {
'Cookie': 'ANDROIDSECURE=' + auth_key,
'User-Agent': 'Android-Market/2 (sapphire PLAT-RC33); gzip',
'Content-Type': 'application/x-www-form-urlencoded',
'Accept-Charset': 'ISO-8859-1,utf-8;q=0.7,*;q=0.7',
'Accept': 'text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2',
'Connection': 'keep-alive'
}
conn = httplib.HTTPSConnection('android.clients.google.com')
conn.request('POST', '/market/api/ApiRequest', 'version=2&request=' + urlsafe_b64encode(data), headers)
Origin java code:
TrustManager[] trustAllCerts = new TrustManager[]{
new X509TrustManager() {
public java.security.cert.X509Certificate[] getAcceptedIssuers() {
return null;
}
public void checkClientTrusted(
java.security.cert.X509Certificate[] certs, String authType) {
}
public void checkServerTrusted(
java.security.cert.X509Certificate[] certs, String authType) {
}
}
};
SSLContext sc = SSLContext.getInstance("SSL");
sc.init(null, trustAllCerts, new java.security.SecureRandom());
HttpsURLConnection.setDefaultSSLSocketFactory(sc.getSocketFactory());
HttpsURLConnection.setDefaultHostnameVerifier(new HostnameVerifier()
{
public boolean verify(String arg0, SSLSession arg1) {
return true;
}
});
URL url = new URL("https://android.clients.google.com/market/api/ApiRequest");
HttpsURLConnection cnx = (HttpsURLConnection)url.openConnection();
cnx.setDoOutput(true);
cnx.setRequestMethod("POST");
cnx.setRequestProperty("Cookie","ANDROIDSECURE=" + this.getAuthSubToken());
cnx.setRequestProperty("User-Agent", "Android-Market/2 (sapphire PLAT-RC33); gzip");
cnx.setRequestProperty("Content-Type", "application/x-www-form-urlencoded");
cnx.setRequestProperty("Accept-Charset","ISO-8859-1,utf-8;q=0.7,*;q=0.7");
String request64 = Base64.encodeBytes(request,Base64.URL_SAFE);
String requestData = "version="+PROTOCOL_VERSION+"&request="+request64;
cnx.setFixedLengthStreamingMode(requestData.getBytes("UTF-8").length);
OutputStream os = cnx.getOutputStream();
os.write(requestData.getBytes());
os.close();
403 means that the request is successful i.e., "the provided credentials were successfully authenticated but that the credentials still do not grant the client permission to access the resource" e.g., without correct headers, body I get 400 not 403.
Here's a more complete example on how to make https request using httplib.
Related
I have a Spring Boot API that runs locally, with a self-signed certificate, using the HTTPS protocol.
Obviously, when I send GET Requests from the browser, I receive the io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown error on the server side, which is normal, because the self-signed is not trusted by the browser. Postman works just fine for GET and POST.
However, I want to send GET requests from an Android client to the Spring API but, even I've used a function to allow all SSL traffic (yes, I know it's not recommended), I still can't send requests to the API, receiving the following output:
I/STATUS: 405
I/MSG: Method Not Allowed
I thought my allowAllSSL() function (HttpsTrustManager class) would solve the issue, because if I remove the function call, I receive the following error, which seems to match the one on the server side:
javax.net.ssl.SSLHandshakeException: java.security.cert.CertPathValidatorException: Trust anchor for certification path not found.
at com.android.org.conscrypt.ConscryptFileDescriptorSocket.startHandshake(ConscryptFileDescriptorSocket.java:239)
Now, you may think that the GET request is not implemented correctly in Spring, but it's not true, since the same GET request works just fine from Postman. I believe that the problem is still linked to the certificate, but I can't figure out what do I need to change. Here is my code:
Spring BOOT Rest Controller
#RestController
#RequestMapping("/post")
public class PostRequest {
#GetMapping("")
public String string(#RequestBody ImageRequest newEmployee){
....
The ImageRequest class contains just three private String members.
HttpsTrustManager class (to allow all SSL)
package com.example.androidclient;
import java.security.KeyManagementException;
import java.security.NoSuchAlgorithmException;
import java.security.SecureRandom;
import java.security.cert.X509Certificate;
import javax.net.ssl.HostnameVerifier;
import javax.net.ssl.HttpsURLConnection;
import javax.net.ssl.SSLContext;
import javax.net.ssl.SSLSession;
import javax.net.ssl.TrustManager;
import javax.net.ssl.X509TrustManager;
public class HttpsTrustManager implements X509TrustManager {
private static TrustManager[] trustManagers;
private static final X509Certificate[] _AcceptedIssuers = new X509Certificate[]{};
#Override
public void checkClientTrusted(
X509Certificate[] x509Certificates, String s)
throws java.security.cert.CertificateException {
}
#Override
public void checkServerTrusted(
X509Certificate[] x509Certificates, String s)
throws java.security.cert.CertificateException {
}
public boolean isClientTrusted(X509Certificate[] chain) {
return true;
}
public boolean isServerTrusted(X509Certificate[] chain) {
return true;
}
#Override
public X509Certificate[] getAcceptedIssuers() {
return _AcceptedIssuers;
}
public static void allowAllSSL() {
HttpsURLConnection.setDefaultHostnameVerifier(new HostnameVerifier() {
#Override
public boolean verify(String arg0, SSLSession arg1) {
return true;
}
});
SSLContext context = null;
if (trustManagers == null) {
trustManagers = new TrustManager[]{new HttpsTrustManager()};
}
try {
context = SSLContext.getInstance("TLS");
context.init(null, trustManagers, new SecureRandom());
} catch (NoSuchAlgorithmException | KeyManagementException e) {
e.printStackTrace();
}
HttpsURLConnection.setDefaultSSLSocketFactory(context != null ? context.getSocketFactory() : null);
}
}
Android Request
HttpsTrustManager.allowAllSSL();
URL url = new URL("https://192.168.1.106:8443/post");
HttpsURLConnection conn = (HttpsURLConnection) url.openConnection();
conn.setRequestMethod("GET");
conn.setRequestProperty("Content-Type", "application/json;charset=UTF-8");
conn.setRequestProperty("Accept", "application/json");
conn.setDoOutput(true);
conn.setDoInput(true);
JSONObject jsonParam = new JSONObject();
jsonParam.put("location", "Somewhere");
jsonParam.put("date", "22.05.2020");
jsonParam.put("imageBytes", strings[0]);
Log.i("JSON", jsonParam.toString());
DataOutputStream os = new DataOutputStream(conn.getOutputStream());
//os.writeBytes(URLEncoder.encode(jsonParam.toString(), "UTF-8"));
os.writeBytes(jsonParam.toString());
os.flush();
os.close();
Log.i("STATUS", String.valueOf(conn.getResponseCode()));
Log.i("MSG", conn.getResponseMessage());
conn.disconnect();
} catch (Exception e) {
e.printStackTrace();
}
return "ok";
}
Use this function in your android application.
Please note this will allow all ssl certificates without verification. I would encourage you to follow the recommended approach when dealing with self-signed certificates outlined here: https://developer.android.com/training/articles/security-ssl#java
// Create a trust manager that does not validate certificate chains
final TrustManager[] trustAllCerts = new TrustManager[] {
new X509TrustManager() {
#Override
public void checkClientTrusted(X509Certificate[] chain, String authType) throws CertificateException {}
#Override
public void checkServerTrusted(X509Certificate[] chain, String authType) throws CertificateException {}
#Override
public X509Certificate[] getAcceptedIssuers() {
return new X509Certificate[0];
}
}
};
// Install the all-trusting trust manager
final SSLContext sslContext = SSLContext.getInstance("SSL");
sslContext.init(null, trustAllCerts, new java.security.SecureRandom());
// Create an ssl socket factory with our all-trusting manager
final SSLSocketFactory sslSocketFactory = sslContext.getSocketFactory();
I have found the solution on my own.
Apparently, the Connection.setDoOutput(true) method is working just for POST and PUT requests, but not for GET.
Thus, I have changed my RequestMapping to work on POST, like, this:
#RequestMapping(
value = "/post",
produces = "application/json",
method = RequestMethod.POST)
Now I receive 200 OK.
I have a Spring Boot application that tries to open a javax.net.ssl.HttpsURLConnection to a server but the response received is: java.io.IOException: Server returned HTTP response code: 403 for URL: https://serverIP:8443/path
When the keyStore, trustStore and their passwords are set as system properties the request works correctly and the expected JSON response is received:
System.setProperty("javax.net.ssl.keyStore", "src/main/resources/myKeyStore.p12");
System.setProperty("javax.net.ssl.trustStore", "src/main/resources/myTrustStore.truststore");
System.setProperty("javax.net.ssl.keyStorePassword", "myPassword");
System.setProperty("javax.net.ssl.trustStorePassword", "myPassword");
But the 403 response code is received when trying to set the information in SSLContext, instead of setting the system properties, by using this method that returns an SSLContext object:
public static SSLContext getSslContext(String trustStoreFile, String keystoreFile, String password)
throws GeneralSecurityException, IOException {
final KeyStore keystore = KeyStore.getInstance("pkcs12"); // also tried with JKS
try (final InputStream inKeystore = new FileInputStream(keystoreFile)) {
keystore.load(inKeystore, password.toCharArray());
}
try (final InputStream inTruststore = new FileInputStream(trustStoreFile)) {
keystore.load(inTruststore, password.toCharArray());
}
final KeyManagerFactory keyManagerFactory = KeyManagerFactory.getInstance("PKIX"); // also tried with .getDefaultAlgorithm()
keyManagerFactory.init(keystore, password.toCharArray());
final TrustManagerFactory trustManagerFactory = TrustManagerFactory.getInstance(TrustManagerFactory.getDefaultAlgorithm());
trustManagerFactory.init(keystore);
X509TrustManager x509Tm = null;
for (final TrustManager trustManager : trustManagerFactory.getTrustManagers()) {
if (trustManager instanceof X509TrustManager) {
x509Tm = (X509TrustManager) trustManager;
break;
}
}
final X509TrustManager finalTm = x509Tm;
final X509ExtendedTrustManager customTm = new X509ExtendedTrustManager() {
#Override
public java.security.cert.X509Certificate[] getAcceptedIssuers() {
return finalTm.getAcceptedIssuers();
}
#Override
public void checkClientTrusted(java.security.cert.X509Certificate[] certs, String authType) {
}
#Override
public void checkServerTrusted(java.security.cert.X509Certificate[] certs, String authType) {
}
#Override
public void checkClientTrusted(java.security.cert.X509Certificate[] xcs, String string, Socket socket) throws CertificateException {
}
#Override
public void checkServerTrusted(java.security.cert.X509Certificate[] xcs, String string, Socket socket) throws CertificateException {
}
#Override
public void checkClientTrusted(java.security.cert.X509Certificate[] xcs, String string, SSLEngine ssle) throws CertificateException {
}
#Override
public void checkServerTrusted(java.security.cert.X509Certificate[] xcs, String string, SSLEngine ssle) throws CertificateException {
}
};
final SSLContext sslContext = SSLContext.getInstance("TLS"); // also tried with SSL
sslContext.init(
keyManagerFactory.getKeyManagers(),
new TrustManager[]{customTm},
new SecureRandom());
final HostnameVerifier allHostsValid = new HostnameVerifier() {
#Override
public boolean verify(String hostname, SSLSession session) {
return true;
}
};
HttpsURLConnection.setDefaultHostnameVerifier(allHostsValid);
return sslContext;
}
OBS: The trustStore and keyStore have the same password, that's why the method has only one password parameter and used for both key and trust manager factories.
The way the getSslContext method is called and used is:
final SSLContext sslContext = SSLContextHelper.getSslContext("src/main/resources/myTrustStore.truststore",
"src/main/resources/myKeyStore.p12",
"myPassword");
final SSLSocketFactory sslSocketFactory = sslContext.getSocketFactory();
final URL url = new URL("https://serverIP:8443/path");
final HttpsURLConnection urlConnection = (HttpsURLConnection) url.openConnection();
urlConnection.setSSLSocketFactory(sslSocketFactory);
// tried adding some headers to the request
urlConnection.addRequestProperty("Content-Type", "application/json");
urlConnection.addRequestProperty("Accept", "application/json");
urlConnection.addRequestProperty("User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:25.0) Gecko/20100101 Firefox/25.0");
urlConnection.connect();
final InputStream inputstream = urlConnection.getInputStream();
The error is thrown at the last line when trying to get the inputStream of the URL connection.
Also, I tried using the following classes from org.apache.http: SSLConnectionSocketFactory, HttpClient, HttpGet, HttpResponse but response code is still 403.
I can only think that there is something missing from the SSL configuration because the system properties work. Any suggestions on what I miss setting in the SSLContext/SSLSocketFactory or how can I solve/better debug the problem are welcome! Thanks!
I managed to open the HTTPS connections only by using Spring's RestTemplate (org.springframework.web.client.RestTemplate) that uses the org.apache.http.client.HttpClient.
The method for getting the RestTemplate that has in its SSLContext the keyStore, trustStore and their passwords is the following:
public RestTemplate getRestTemplate(final String keyStoreFile, final String trustStoreFile,
final String password) throws Exception {
final SSLContext sslContext = SSLContextBuilder.create()
.loadKeyMaterial(ResourceUtils.getFile(keyStoreFile), password.toCharArray(), password.toCharArray())
.loadTrustMaterial(ResourceUtils.getFile(trustStoreFile), password.toCharArray())
.build();
final HttpClient client = HttpClients.custom()
.setSSLContext(sslContext)
.build();
final HttpComponentsClientHttpRequestFactory httpComponentsClientHttpRequestFactory = new HttpComponentsClientHttpRequestFactory();
httpComponentsClientHttpRequestFactory.setHttpClient(client);
return new RestTemplate(httpComponentsClientHttpRequestFactory);
}
The way that the RestTemplate is used for the HTTPS call is:
final String keyStoreFile = "src/main/resources/myKeyStore.p12";
final String trustStoreFile = "src/main/resources/myTrustStore.truststore";
final String password = "myPassword"; // same password for keyStore and trustStore
final String response = getRestTemplate(keyStoreFile, trustStoreFile, password).getForObject("https://serverIP:8443/path", String.class);
LOGGER.info("Response received: " + response);
Hope this helps anyone, had a lot of struggle with the HTTPS connections :)
I am trying to access the REST API through rest assured library in java, this API is secured with SSL certificate and key.
Is there a way to pass the certificate and key and access the REST API?
I tried with RestAssured.Keystore() method by importing the certificate and key to keystore but it does not work.
Any help on this is much appreciated..!!
You may need to set both the keystore and trustStore (this would be your cacert file).
RestAssured.keystore("path_to_jks_file", "keystore_passsword");
RestAssured.trustStore("path_to_cacert", "trustStore_password");
The default password for cacert is changeit.
There was a bug on this in the earlier versions of RestAssured, but it's been fixed since v3.0.2. So be sure to use the latest.
Create a SpringBoot or java application like this:
HttpHeaders headers = new HttpHeaders();
headers.set(HttpHeaders.USER_AGENT, "YOURS/1.0.0");
headers.set("X-App-Username", "YOURS");
headers.set("App-Username", "YOURS");
headers.set(HttpHeaders.CONTENT_TYPE, "application/json");
headers.set(HttpHeaders.CONNECTION, "Keep-Alive");
headers.set(HttpHeaders.HOST, "IP");
headers.set(HttpHeaders.ACCEPT_LANGUAGE, "pt-BR");
headers.add("Accept","application/json;charset=UTF-8");
headers.setContentType(MediaType.APPLICATION_JSON_UTF8);
request = new HttpEntity<HttpHeaders>(headers);
System.setProperty("javax.net.ssl.trustStore", "YOUR_PATH/clientcert.jks");
System.setProperty("javax.net.ssl.trustStorePassword", "pwd123");
System.setProperty("javax.net.ssl.trustStoreType", "JKS");
System.setProperty("javax.net.ssl.keyStore", "YOUR_PATH/trustStore.jks");
System.setProperty("javax.net.ssl.keyStorePassword", "pwd123");
System.setProperty("javax.net.ssl.keyStoreType", "JKS");
System.setProperty("javax.net.ssl.keyAlias", "localhost");
System.setProperty("javax.net.ssl.enabled", "true");
System.setProperty("javax.net.ssl.defaul-type", "JKS");
System.setProperty("javax.net.ssl.client-auth", "need");
System.setProperty("javax.net.ssl.protocols", "TLSv1.2");
TrustManager[] trustAllCerts = new TrustManager[] { new X509TrustManager() {
public java.security.cert.X509Certificate[] getAcceptedIssuers() {
return null;
}
public void checkClientTrusted(X509Certificate[] certs, String authType) { }
public void checkServerTrusted(X509Certificate[] certs, String authType) { }
} };
SSLContext sc = null;
try {
sc = SSLContext.getInstance("SSL");
sc.init(null, trustAllCerts, new java.security.SecureRandom());
} catch (KeyManagementException e) {
e.printStackTrace();
} catch (NoSuchAlgorithmException e) {
e.printStackTrace();
}
HttpsURLConnection.setDefaultSSLSocketFactory(sc.getSocketFactory());
HostnameVerifier allHostsValid = new HostnameVerifier() {
public boolean verify(String hostname, SSLSession session) {
return true;
}
};
HttpsURLConnection.setDefaultHostnameVerifier(allHostsValid);
requestEntity = new HttpEntity<LinkedMultiValueMap<String, Object>>(null, headers);
responseEntity = null;
mapper = new ObjectMapper();
mapper.configure(DeserializationFeature.USE_JAVA_ARRAY_FOR_JSON_ARRAY, true);
You can use org.springframework.web.client.RestTemplate if your are using Spring.
Using spring, you can set this in a config file (application.yml):
server:
port: 9090
address: 0.0.0.0
contextPath: /your-url
ssl:
key-store: classpath:clientcert.jks
key-store-password: pwd123
key-alias: localhost
#enabled: true
trust-store: classpath:trustStore.jks
trust-store-password: pwd123
defaul-type: JKS
client-auth: need
protocols: TLSv1.2
I have following CURL command which works fine using -k options, which means: insecure, Allow connections to SSL sites without certs.
I am using WSO2 API Manager tool version 1.9.1.
curl -k -d "grant_type=password&username=test&password=test" -H
"Authorization: Basic XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,
Content-Type: application/x-www-form-urlencoded"
https://XXX.XXX.XXX.XXX:8243/token
Now I wanted to achieve the same using the RestTemplate in Spring, so I developed the following code so far, and it's giving me a Connection refused error:
My Code
#Before
public void testBefore(){
disableSslVerification();
}
private static void disableSslVerification() {
try {
TrustManager[] trustAllCerts = new TrustManager[]{
new X509TrustManager() {
#Override
public X509Certificate[] getAcceptedIssuers() {
return null;
}
#Override
public void checkServerTrusted(X509Certificate[] chain, String authType) throws CertificateException {
}
#Override
public void checkClientTrusted(X509Certificate[] chain, String authType) throws CertificateException {
}
}
};
SSLContext sc = SSLContext.getInstance("SSL");
sc.init(null, trustAllCerts, new java.security.SecureRandom());
HttpsURLConnection.setDefaultSSLSocketFactory(sc.getSocketFactory());
HostnameVerifier allHostsValid = new HostnameVerifier() {
public boolean verify(String hostname, SSLSession session) {
return true;
}
};
HttpsURLConnection.setDefaultHostnameVerifier(allHostsValid);
} catch (NoSuchAlgorithmException e) {
e.printStackTrace();
} catch (KeyManagementException e) {
e.printStackTrace();
}
}
#Test
public void testGetTokenFromWSo2(){
String url = "https://XXXXXXXXXXX:8243/token";
MultiValueMap<String, Object> map = new LinkedMultiValueMap<String, Object>();
map.add("grant_type", "password");
map.add("username", "test");
map.add("password", "test");
HttpHeaders headers =new HttpHeaders();
headers.add("Authorization", "Basic XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX");
headers.add("Content-Type", "application/x-www-form-urlencoded");
RestTemplate restTemplate = new RestTemplate();
HttpEntity<String> entity = new HttpEntity<String>(headers);
HttpEntity<String> response = restTemplate.exchange(url, HttpMethod.GET, entity, String.class, map);
System.out.println("RESPONSE : "+response.getBody());
}
The error I see
org.springframework.web.client.ResourceAccessException: I/O error on GET request for "https://XXXXXXXXXXXXXX:8243/token":Connection refused: connect; nested exception is java.net.ConnectException: Connection refused: connect
at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:567)
at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:520)
at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:463)
I was able to resolved this error. We can connect to SSL protected site by insecure way i.e, connecting to SSL sites without verifying cert. You can find the solution here and it works fine java.security.cert.CertificateException: No subject alternative names present;.
I am writing a Java JSP code that authenthicates against a WSO2 IS server using openid. I have taken the example code from de developer so I get the following:
<%
ConsumerManager manager = new ConsumerManager ();
String _returnURL = "https://192.168.15.48:9443/ficlient/secret.jsp";
List discoveries = manager.discover("https://myserverIP/openid");
DiscoveryInformation discovered = manager.associate(discoveries);
session.setAttribute("discovered", discovered);
AuthRequest authReq = manager.authenticate(discovered, _returnURL);
nextlink = authReq.getDestinationUrl(true);
%>
Secret data
On the 3rd line (List discoveries...) I get an exception:
org.openid4java.discovery.yadis.YadisException: 0x704: I/O transport error: peer not authenticated
I have understood that this is due to a non valid ssl certificate issued for the https comunication and have tried including the following (as found on Internet), to avoid the verification:
<%
// Create a trust manager that does not validate certificate chains
TrustManager[] trustAllCerts = new TrustManager[]{
new X509TrustManager() {
public java.security.cert.X509Certificate[] getAcceptedIssuers() {
return null;
}
public void checkClientTrusted( java.security.cert.X509Certificate[] certs, String authType) {}
public void checkServerTrusted( java.security.cert.X509Certificate[] certs, String authType) {}
}
};
HostnameVerifier allHostsValid = new HostnameVerifier() {
public boolean verify(String hostname, SSLSession session) {
return true;
}
};
try {
SSLContext sc = SSLContext.getInstance("SSL");
sc.init(null, trustAllCerts, new java.security.SecureRandom());
HttpsURLConnection.setDefaultSSLSocketFactory(sc.getSocketFactory());
HttpsURLConnection.setDefaultHostnameVerifier(allHostsValid);
} catch (Exception e) {}
%>
But it is still not working. What have I missed?
I finally solved it by using Oltu libraries for authenthication.