What are my options if I want to make a call to Amazon AWS Rest API from Java.
When implementing my own request, generating the AWS4-HMAC-SHA256 Authorization header would be the hardest.
Essentially, this is the header I need to generate:
Authorization: AWS4-HMAC-SHA256 Credential=AKIAJTOUYS27JPVRDUYQ/20200602/us-east-1/route53/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=ba85affa19fa4a8735ce952e50d41c8c93406a11d22b88cc98b109b529bcc15e
Not saying that this is a complete list, but I would consider using established libraries like:
Official AWS SDK v1, or v2 - current and comprehensive but depends on netty.io and many other jars.
Apache JClouds - depends on JAXB which is not longer a part of JDK but now available at maven central separately.
But sometimes, all you want is to make a simple call, and you don't want to bring many dependencies into your application. You may want to implement the rest call yourself. Generating the right AWS Authorization header is the hardest bit to implement.
Here is the code to do that in pure java OpenJDK with no external dependencies.
It implements Amazon AWS API Signature Version 4 signing process.
AmazonRequestSignatureV4Utils.java
package com.frusal.amazonsig4;
import java.nio.charset.StandardCharsets;
import java.security.MessageDigest;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import java.util.Map.Entry;
import java.util.stream.Collectors;
import javax.crypto.Mac;
import javax.crypto.spec.SecretKeySpec;
public class AmazonRequestSignatureV4Utils {
/**
* Generates signing headers for HTTP request in accordance with Amazon AWS API Signature version 4 process.
* <p>
* Following steps outlined here: docs.aws.amazon.com
* <p>
* Simple usage example is here: {#link AmazonRequestSignatureV4Example}
* <p>
* This method takes many arguments as read-only, but adds necessary headers to #{code headers} argument, which is a map.
* The caller should make sure those parameters are copied to the actual request object.
* <p>
* The ISO8601 date parameter can be created by making a call to:<br>
* - {#code java.time.format.DateTimeFormatter.ofPattern("yyyyMMdd'T'HHmmss'Z'").format(ZonedDateTime.now(ZoneOffset.UTC))}<br>
* or, if you prefer joda:<br>
* - {#code org.joda.time.format.ISODateTimeFormat.basicDateTimeNoMillis().print(DateTime.now().withZone(DateTimeZone.UTC))}
*
* #param method - HTTP request method, (GET|POST|DELETE|PUT|...), e.g., {#link java.net.HttpURLConnection#getRequestMethod()}
* #param host - URL host, e.g., {#link java.net.URL#getHost()}.
* #param path - URL path, e.g., {#link java.net.URL#getPath()}.
* #param query - URL query, (parameters in sorted order, see the AWS spec) e.g., {#link java.net.URL#getQuery()}.
* #param headers - HTTP request header map. This map is going to have entries added to it by this method. Initially populated with
* headers to be included in the signature. Like often compulsory 'Host' header. e.g., {#link java.net.HttpURLConnection#getRequestProperties()}.
* #param body - The binary request body, for requests like POST.
* #param isoDateTime - The time and date of the request in ISO8601 basic format, see comment above.
* #param awsIdentity - AWS Identity, e.g., "AKIAJTOUYS27JPVRDUYQ"
* #param awsSecret - AWS Secret Key, e.g., "I8Q2hY819e+7KzBnkXj66n1GI9piV+0p3dHglAzQ"
* #param awsRegion - AWS Region, e.g., "us-east-1"
* #param awsService - AWS Service, e.g., "route53"
*/
public static void calculateAuthorizationHeaders(
String method, String host, String path, String query, Map<String, String> headers,
byte[] body,
String isoDateTime,
String awsIdentity, String awsSecret, String awsRegion, String awsService
) {
try {
String bodySha256 = hex(sha256(body));
String isoJustDate = isoDateTime.substring(0, 8); // Cut the date portion of a string like '20150830T123600Z';
headers.put("Host", host);
headers.put("X-Amz-Content-Sha256", bodySha256);
headers.put("X-Amz-Date", isoDateTime);
// (1) https://docs.aws.amazon.com/general/latest/gr/sigv4-create-canonical-request.html
List<String> canonicalRequestLines = new ArrayList<>();
canonicalRequestLines.add(method);
canonicalRequestLines.add(path);
canonicalRequestLines.add(query);
List<String> hashedHeaders = new ArrayList<>();
for (Entry<String, String> e : headers.entrySet()) {
hashedHeaders.add(e.getKey().toLowerCase());
canonicalRequestLines.add(e.getKey().toLowerCase() + ":" + normalizeSpaces(e.getValue().toString()));
}
canonicalRequestLines.add(null); // new line required after headers
String signedHeaders = hashedHeaders.stream().collect(Collectors.joining(";"));
canonicalRequestLines.add(signedHeaders);
canonicalRequestLines.add(bodySha256);
String canonicalRequestBody = canonicalRequestLines.stream().map(line -> line == null ? "" : line).collect(Collectors.joining("\n"));
String canonicalRequestHash = hex(sha256(canonicalRequestBody.getBytes(StandardCharsets.UTF_8)));
// (2) https://docs.aws.amazon.com/general/latest/gr/sigv4-create-string-to-sign.html
List<String> strignToSignLines = new ArrayList<>();
strignToSignLines.add("AWS4-HMAC-SHA256");
strignToSignLines.add(isoDateTime);
String credentialScope = isoJustDate + "/" + awsRegion + "/" + awsService + "/aws4_request";
strignToSignLines.add(credentialScope);
strignToSignLines.add(canonicalRequestHash);
String stringToSign = strignToSignLines.stream().collect(Collectors.joining("\n"));
// (3) https://docs.aws.amazon.com/general/latest/gr/sigv4-calculate-signature.html
byte[] kDate = hmac(("AWS4" + awsSecret).getBytes(StandardCharsets.UTF_8), isoJustDate);
byte[] kRegion = hmac(kDate, awsRegion);
byte[] kService = hmac(kRegion, awsService);
byte[] kSigning = hmac(kService, "aws4_request");
String signature = hex(hmac(kSigning, stringToSign));
String authParameter = "AWS4-HMAC-SHA256 Credential=" + awsIdentity + "/" + credentialScope + ", SignedHeaders=" + signedHeaders + ", Signature=" + signature;
headers.put("Authorization", authParameter);
} catch (Exception e) {
if (e instanceof RuntimeException) {
throw (RuntimeException) e;
} else {
throw new IllegalStateException(e);
}
}
}
private static String normalizeSpaces(String value) {
return value.replaceAll("\\s+", " ").trim();
}
public static String hex(byte[] a) {
StringBuilder sb = new StringBuilder(a.length * 2);
for(byte b: a) {
sb.append(String.format("%02x", b));
}
return sb.toString();
}
private static byte[] sha256(byte[] bytes) throws Exception {
MessageDigest digest = MessageDigest.getInstance("SHA-256");
digest.update(bytes);
return digest.digest();
}
public static byte[] hmac(byte[] key, String msg) throws Exception {
Mac mac = Mac.getInstance("HmacSHA256");
mac.init(new SecretKeySpec(key, "HmacSHA256"));
return mac.doFinal(msg.getBytes(StandardCharsets.UTF_8));
}
}
And the usage example:
AmazonRequestSignatureV4Utils.java
package com.frusal.amazonsig4;
import java.net.HttpURLConnection;
import java.net.URL;
import java.nio.charset.StandardCharsets;
import java.time.ZoneOffset;
import java.time.ZonedDateTime;
import java.time.format.DateTimeFormatter;
import java.util.LinkedHashMap;
import java.util.Map;
import java.util.stream.Collectors;
public class AmazonRequestSignatureV4Example {
public static void main(String[] args) throws Exception {
String route53HostedZoneId = "Z08118721NNU878C4PBNA";
String awsIdentity = "AKIAJTOUYS27JPVRDUYQ";
String awsSecret = "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";
String awsRegion = "us-east-1";
String awsService = "route53";
URL url = new URL("https://route53.amazonaws.com/2013-04-01/hostedzone/" + route53HostedZoneId + "/rrset");
HttpURLConnection connection = (HttpURLConnection) url.openConnection();
connection.setRequestMethod("POST");
System.out.println(connection.getRequestMethod() + " " + url);
String body = "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n" +
"<ChangeResourceRecordSetsRequest xmlns=\"https://route53.amazonaws.com/doc/2013-04-01/\">\n" +
"<ChangeBatch>\n" +
// " <Comment>optional comment about the changes in this change batch request</Comment>\n" +
" <Changes>\n" +
" <Change>\n" +
" <Action>UPSERT</Action>\n" +
" <ResourceRecordSet>\n" +
" <Name>c001cxxx.frusal.com.</Name>\n" +
" <Type>A</Type>\n" +
" <TTL>300</TTL>\n" +
" <ResourceRecords>\n" +
" <ResourceRecord>\n" +
" <Value>157.245.232.185</Value>\n" +
" </ResourceRecord>\n" +
" </ResourceRecords>\n" +
// " <HealthCheckId>optional ID of a Route 53 health check</HealthCheckId>\n" +
" </ResourceRecordSet>\n" +
" </Change>\n" +
" </Changes>\n" +
"</ChangeBatch>\n" +
"</ChangeResourceRecordSetsRequest>";
byte[] bodyBytes = body.getBytes(StandardCharsets.UTF_8);
Map<String, String> headers = new LinkedHashMap<>();
String isoDate = DateTimeFormatter.ofPattern("yyyyMMdd'T'HHmmss'Z'").format(ZonedDateTime.now(ZoneOffset.UTC));
AmazonRequestSignatureV4Utils.calculateAuthorizationHeaders(
connection.getRequestMethod(),
connection.getURL().getHost(),
connection.getURL().getPath(),
connection.getURL().getQuery(),
headers,
bodyBytes,
isoDate,
awsIdentity,
awsSecret,
awsRegion,
awsService);
// Unsigned headers
headers.put("Content-Type", "text/xml; charset=utf-8"); // I guess it get modified somewhere on the way... Let's just leave it out of the signature.
// Log headers and body
System.out.println(headers.entrySet().stream().map(e -> e.getKey() + ": " + e.getValue()).collect(Collectors.joining("\n")));
System.out.println(body);
// Send
headers.forEach((key, val) -> connection.setRequestProperty(key, val));
connection.setDoOutput(true);
connection.getOutputStream().write(bodyBytes);
connection.getOutputStream().flush();
int responseCode = connection.getResponseCode();
System.out.println("connection.getResponseCode()=" + responseCode);
String responseContentType = connection.getHeaderField("Content-Type");
System.out.println("responseContentType=" + responseContentType);
System.out.println("Response BODY:");
if (connection.getErrorStream() != null) {
System.out.println(new String(connection.getErrorStream().readAllBytes(), StandardCharsets.UTF_8));
} else {
System.out.println(new String(connection.getInputStream().readAllBytes(), StandardCharsets.UTF_8));
}
}
}
And the trace it would generate:
POST https://route53.amazonaws.com/2013-04-01/hostedzone/Z08118721NNU878C4PBNA/rrset
Host: route53.amazonaws.com
X-Amz-Content-Sha256: 46c7521da55bcf9e99fa6e12ec83997fab53128b5df0fb12018a6b76fb2bf891
X-Amz-Date: 20200602T035618Z
Authorization: AWS4-HMAC-SHA256 Credential=AKIAJTOUYS27JPVRDUYQ/20200602/us-east-1/route53/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=6a59090f837cf71fa228d2650e9b82e9769e0ec13e9864e40bd2f81c682ef8cb
Content-Type: text/xml; charset=utf-8
<?xml version="1.0" encoding="UTF-8"?>
<ChangeResourceRecordSetsRequest xmlns="https://route53.amazonaws.com/doc/2013-04-01/">
<ChangeBatch>
<Changes>
<Change>
<Action>UPSERT</Action>
<ResourceRecordSet>
<Name>c001cxxx.frusal.com.</Name>
<Type>A</Type>
<TTL>300</TTL>
<ResourceRecords>
<ResourceRecord>
<Value>157.245.232.185</Value>
</ResourceRecord>
</ResourceRecords>
</ResourceRecordSet>
</Change>
</Changes>
</ChangeBatch>
</ChangeResourceRecordSetsRequest>
connection.getResponseCode()=200
responseContentType=text/xml
Response BODY:
<?xml version="1.0"?>
<ChangeResourceRecordSetsResponse xmlns="https://route53.amazonaws.com/doc/2013-04-01/"><ChangeInfo><Id>/change/C011827119UYGF04GVIP6</Id><Status>PENDING</Status><SubmittedAt>2020-06-02T03:56:25.822Z</SubmittedAt></ChangeInfo></ChangeResourceRecordSetsResponse>
For the latest version of this code, please see java-amazon-request-signature-v4 repository at GitHub.
Related
I want to expand my software, written in JavaFX, with Amazon Chime API to consume its messaging. I know there's JS SDK that allows establish messaging websocket session with no problems. But in java SDK there're no related classes. So I want to use STOMP library to consuming the websocket endpoint.
At the time I am struggling with making correct request, namely with signing AWS request (calculating X-AMZ-Signature)
According to the post I'm trying to calculate correct X-AMZ-Signature request parameter. Here's the class:
#Slf4j
#Service
public class Aws4Signer {
private final static String REQUEST_CONTENT_TYPE = "application/json";
private final static String AUTH_ALGORITHM = "AWS4-HMAC-SHA256";
private final static String REQUEST_METHOD = "GET";
#Data
class AuthenticationData {
#NonNull
String timestamp;
#NonNull
String date;
#NonNull
String authorizationHeader;
}
private AppConfig appConfig = new AppConfig();
/**
* Gets the timestamp in YYYYMMDD'T'HHMMSS'Z' format, which is the required
* format for AWS4 signing request headers and credential string
*
* #param dateTime
* an OffsetDateTime object representing the UTC time of current
* signing request
* #return the formatted timestamp string
*
* #see <a href=
* "https://docs.aws.amazon.com/general/latest/gr/sigv4-signed-request-examples.html">
* Examples of the Complete Version 4 Signing Process (Python)</a>
*/
public String getTimeStamp(OffsetDateTime dateTime) {
DateTimeFormatter formatter = DateTimeFormatter.ofPattern("yyyyMMdd'T'HHmmss'Z'");
String formatDateTime = dateTime.format(formatter);
return formatDateTime;
}
/**
* Gets the date string in yyyyMMdd format, which is required to build the
* credential scope string
*
* #return the formatted date string
*/
public String getDate(OffsetDateTime dateTime) {
DateTimeFormatter formatter = DateTimeFormatter.ofPattern("yyyyMMdd");
String formatDateTime = dateTime.format(formatter);
return formatDateTime;
}
public byte[] generateAws4SigningKey(String timestamp) {
String secretKey = appConfig.getAwsAuthConfig().getSecretKey();
String regionName = appConfig.getAwsAuthConfig().getServiceRegion();
String serviceName = appConfig.getAwsAuthConfig().getServiceName();
byte[] signatureKey = null;
try {
signatureKey = Aws4SignatureKeyGenerator.generateSignatureKey(secretKey, timestamp, regionName,
serviceName);
} catch (Exception e) {
log.error("An error has ocurred when generate signature key: " + e, e);
}
return signatureKey;
}
/**
* Builds an {#link AuthenticationData} object containing the timestamp, date,
* payload hash and the AWS4 signature
* <p>
*
* The signing logic was translated from the Python implementation, see this
* link for more details: <a href=
* "https://docs.aws.amazon.com/general/latest/gr/sigv4-signed-request-examples.html">Examples
* of the Complete Version 4 Signing Process (Python)</a>
*
* #param target
* #param requestBody
*
* #return
* #throws NoSuchAlgorithmException
* #throws UnsupportedEncodingException
* #throws InvalidKeyException
* #throws SignatureException
* #throws IllegalStateException
*
*/
public AuthenticationData buildAuthorizationData() throws NoSuchAlgorithmException,
UnsupportedEncodingException, InvalidKeyException, SignatureException, IllegalStateException {
log.info("predict - start");
// Starting building the lengthy signing data
AwsAuthConfig awsAuthConfig = appConfig.getAwsAuthConfig();
String payloadHash = Hmac.getSha256Hash(requestBody);
OffsetDateTime now = OffsetDateTime.now(ZoneOffset.UTC);
String timestamp = getTimeStamp(now);
String date = getDate(now);
// Step 1 is to define the verb (GET, POST, etc.) -- already done by defining
// constant REQUEST_METHOD
// Step 2: Create canonical URI--the part of the URI from domain to query
// string (use '/' if no path)
String canonical_uri = "/connect";
// Step 3: Create the canonical query string. In this example, request
// parameters are passed in the body of the request and the query string
// is blank.
String canonical_querystring = buildCanonicalQueryString();
// Step 4: Create the canonical headers. Header names must be trimmed
// and lowercase, and sorted in code point order from low to high.
// Note that there is a trailing \n.
String canonical_headers = "content-type:" + REQUEST_CONTENT_TYPE + "\n"
+ "host:" + awsAuthConfig.getServiceHost() + "\n"
+ "x-amz-date:" + timestamp + "\n";
String signed_headers = "content-type;host;x-amz-date";
log.debug("canonical_headers : {}", canonical_headers);
String canonical_request = REQUEST_METHOD + "\n" + canonical_uri + "\n" + canonical_querystring + "\n"
+ canonical_headers + "\n" + signed_headers;
log.debug("canonical_request : {}", canonical_request);
String credential_scope = date + "/" + awsAuthConfig.getServiceRegion() + "/" + awsAuthConfig.getServiceName()
+ "/" + "aws4_request";
String canonical_request_hash = Hmac.getSha256Hash(canonical_request);
log.debug("canonical_request_hash : {}", canonical_request_hash);
String string_to_sign = AUTH_ALGORITHM + "\n" + timestamp + "\n" + credential_scope + "\n"
+ canonical_request_hash;
log.debug("string_to_sign : {}", string_to_sign);
byte[] sigKey = generateAws4SigningKey(date);
String signature = Hmac.calculateHMAC(string_to_sign, sigKey, Hmac.HMAC_SHA256);
String authorization_header = AUTH_ALGORITHM + " " + "Credential=" + awsAuthConfig.getAccessKey() + "/"
+ credential_scope + ", " + "SignedHeaders=" + signed_headers + ", " + "Signature=" + signature;
log.debug("authorization_header : {}", authorization_header);
return new AuthenticationData(timestamp, date, authorization_header);
}
private String buildCanonicalQueryString() {
String canonicalRequest = REQUEST_METHOD + "\n" +
"/connect" + "\n" +
"X-Amz-Algorithm=AWS4-HMAC-SHA256\n" +
"&X-Amz-Credential=MYACCESKEY%2F"+ getDate(OffsetDateTime.now()) + "%2Fus-east-1%2Fchime%2Faws4_request\n" +
"&X-Amz-Date=" + getTimeStamp(OffsetDateTime.now()) +"\n" +
"&X-Amz-Expires=10\n" +
"&X-Amz-SignedHeaders=host\n" +
"&sessionId=" + UUID.randomUUID() +"\n" +
"&userArn=" + "MYUSERARN";
return canonicalRequest;
}
}
Provided information
host: node001.ue1.ws-messaging.chime.aws
service name: chime
region: us-east-1
It makes the signature, and I'm trying use it via postman, but postman can't connect to the endpoint node001.ue1.ws-messaging.chime.aws/connect, just saying 'connect ETIMEDOUT 54.162.103.101:80'.
I'm new to Amazon, so it's kinda hard to me. Can you say where am I wrong?
Any help appreciated!
Wrote fully working code for signing URL for connecting to chime websocket. Hope this will helps somebody!
This is my request code. aa means my consumer key and bb is userAcessTokenSecret. I changed the values for the sake of security. I don't know is it becasue of cursor parameter or the ways of encoding and hashing the values and keys.
public static void getLocationAndNameOfFollowers(String userAcessToken, String userAcessTokenSecret) throws IOException, InvalidKeyException, NoSuchAlgorithmException {
userAcessTokenSecret = "bb";
String signingKey = GenerateSignature.oAuthSigningKey("aa", userAcessTokenSecret);
long ts = new Timestamp(System.currentTimeMillis()).getTime() / 1000;
String timeStamp = String.valueOf(ts);
String nonce = GenerateSignature.generateNonce("aa", timeStamp);
String baseString = GenerateSignature.oAuthBaseString("GET", "https://api.twitter.com/1.1/followers/list.json?",
"cursor-1", "fHkdJVy3x1fKE1Yop9qraJyCp", userAcessToken, timeStamp, nonce);
String oauth_signature = GenerateSignature.oAuthSignature(baseString, signingKey);
JSONObject response = Unirest.get("https://api.twitter.com/1.1/followers/list.json?cursor=-1")
.header("Content-Type", "application/x-www-form-urlencoded")
.header("Authorization", "OAuth oauth_consumer_key=\"consumer_key\"," +
"oauth_token=" + "\"" + userAcessToken +"\""+ "," +
"oauth_signature_method=" + "\"HMAC-SHA1\"," +
"oauth_timestamp=" + "\""+timeStamp + "\"" + "," +
"oauth_nonce=" + "\""+nonce + "\"" + "," +
"oauth_version=\"1.0\"," +
"oauth_signature=" + "\"" +oauth_signature + "\"")
.asJson().getBody().getObject();
The code in below contains my helper functions.
private static String percentEncoding(String originalString) {
String encodedString = Base64.getUrlEncoder().encodeToString(originalString.getBytes());
return encodedString;
}
public static String oAuthBaseString(String method, String url, String parameters, String key, String token, String timestamp, String nonce) {
System.out.println("generated sorted parameter string -> "+generateSortedParameterString(parameters, key, token, timeStamp , nonce));
String res = method + "&" + percentEncoding(url)
+ "&" + generateSortedParameterString(parameters, key, token, timeStamp , nonce);
System.out.println("oauth base string -> \n\n\n" + res);
return res;
}
public static String oAuthSignature(String baseString, String tokenSecret) throws NoSuchAlgorithmException, InvalidKeyException {
byte[] bytes = baseString.getBytes(StandardCharsets.UTF_8);
Mac mac = Mac.getInstance("HmacSHA1");
mac.init(new SecretKeySpec(bytes, "HmacSHA1"));
byte[] res = mac.doFinal(tokenSecret.getBytes(StandardCharsets.UTF_8));
return percentEncoding(Base64.getEncoder().toString());
}
public static String oAuthSigningKey(String consumerSecret, String accessTokenSecret) {
return consumerSecret + "&" + accessTokenSecret;
}
public static String generateNonce(String consumerKey, String timeStamp) {
String nonce = Base64.getEncoder().encodeToString((consumerKey + ":" + timeStamp).getBytes());
return nonce;
}
public static String generateSortedParameterString(String parameter, String key, String token, String timeStamp, String nonce) {
Map<String, String> parameters = new LinkedHashMap<>();
parameters.put("oauth_consumer_key", key);
parameters.put("oauth_nonce", nonce);
parameters.put("oauth_signature_method", "HMAC-SHA1");
parameters.put("oauth_timestamp", timeStamp);
parameters.put("oauth_token", token);
parameters.put("oauth_version", "1.0");
System.out.println("parameter map is here -> "+parameters);
String parameterString = parameters.entrySet().stream()
.sorted(Map.Entry.comparingByKey())
.map(e -> percentEncoding(e.getKey()) + "=" + percentEncoding(e.getValue()))
.collect(Collectors.joining("&"));
return parameterString;
}
I am getting this response
{"errors":[{"code":32,"message":"Could not authenticate you."}]}
I see your code and looks like you may not have finished the steps needed before you make a call to get followers.
Since Twitter follows OAuth1 here is what you need to do for this. If you confirm that you have done these then I can help you with the signing process.
The variables that you are working with
consumerKey = Copied from twitter during registration
accesstokenSecret = Copied from twitter during registration
oauth_token = Received as response from step 1 below
oauth_token_secret = Received as response from step 2 below
oauth_verifier = Received as response from step 3 below
accesstoken = Received as response from step 4 below. Finally to be used while signing all API calls
Steps for OAuth 1.0 that twitter follows
GET oauth_token making signed call to https://api.twitter.com/oauth/request_token. Twitter will return oauth_token and oauth_token_secret
Redirect user to https://api.twitter.com/oauth/authorize?oauth_token={{oauth_token}}
User Authenticates and twitter will send a code to your redirect url
Then send a signed request to https://api.twitter.com/oauth/access_token to recieve the access_token
The process for signing are the same for each step with changes to the baseString and signing key. If you have achieved the signing logic for step1. All the others should work.
If you have not done the above steps and are struggling with signing at step1 then I'll help you with the basic structure of signing. Do confirm?
I work at Pathfix and we built it out as a middleware to solve the exact problem without having you to download SDK. If you are dealing with multiple Providers, you will eventually notice the lack of reusability and bulk of unneccessary code. It might save you hours and $$. All of what you are trying to achieve can take you not more than 10 minutes :)
Trying to generate an SAS Token to access certain files in a Storage Account. I'm using the methods listed here:
https://learn.microsoft.com/en-us/rest/api/eventhub/generate-sas-token
Now, the problem I have is I cannot, for the life of me, make the sasToken string work. If I generate the token via the Portal (Shared Access Signature in the Storage Account), I can access those files via a URL with the provided Token.
However I have yet to be able to generate an SAS token programmatically via Java using the methods I linked above. I think my problem is the StringToSign that is being encrypted. I've been following this example when constructing the string to encrypt:
https://learn.microsoft.com/en-us/rest/api/storageservices/constructing-an-account-sas
All my efforts have resulted in either:
<AuthenticationErrorDetail>Signature fields not well formed.</AuthenticationErrorDetail>
or
<AuthenticationErrorDetail>Signature did not match. String to sign used was <insert string details here>
Looking at the Portal generated sasToken that works for me:
?sv=2017-11-09&ss=f&srt=o&sp=r&se=2018-12-06T22:15:20Z&st=2018-12-06T14:15:20Z&spr=https&sig=%2Bi1TWv5D80U%2BoaIeoBh1wjaO1p4xVFx4nRZt%2FzwiszY%3D
It seems I need a String like so:
String stringToSign = accountName + "\n" +
"r\n" +
"f\n" +
"o\n" +
URLEncoder.encode(start, "UTF-8") + "\n" +
URLEncoder.encode(expiry, "UTF-8") + "\n" +
"\n" +
"https\n" +
azureApiVersion;
Where accountName is the storage account name from Azure, and start/expiry are the start and expiry strings (ie- 2018-12-06T22:15:20Z) and azureApiVersion is "2017-11-09".
I then try to return the token after constructing the string like so:
String signature = getHMAC256(key, stringToSign);
sasToken = "sv=" + azureApiVersion +
"&ss=f" +
"&srt=o" +
"&sp=r" +
"&se=" +URLEncoder.encode(expiry, "UTF-8") +
"&st=" + URLEncoder.encode(start, "UTF-8") +
"&spr=https" +
"&sig=" + URLEncoder.encode(signature, "UTF-8");
I've tried URL encoding and not URL encoding the the start/expiry dates as well, just in case that was messing things up. What am I missing?
Three points to fix
getHMAC256 method problem as mentioned by #Gaurav
Don't encode expiry and start in stringToSign or the signature won't match. Because the encoded part in url will be decoded by Azure Storage Service to calculate the expected signature.
In stringToSign, miss one \n after azureApiVersion.
Here's the complete sample.
public static void GetFileSAS(){
String accountName = "accountName";
String key = "accountKey";
String resourceUrl = "https://"+accountName+".file.core.windows.net/fileShare/fileName";
String start = "startTime";
String expiry = "expiry";
String azureApiVersion = "2017-11-09";
String stringToSign = accountName + "\n" +
"r\n" +
"f\n" +
"o\n" +
start + "\n" +
expiry + "\n" +
"\n" +
"https\n" +
azureApiVersion+"\n";
String signature = getHMAC256(key, stringToSign);
try{
String sasToken = "sv=" + azureApiVersion +
"&ss=f" +
"&srt=o" +
"&sp=r" +
"&se=" +URLEncoder.encode(expiry, "UTF-8") +
"&st=" + URLEncoder.encode(start, "UTF-8") +
"&spr=https" +
"&sig=" + URLEncoder.encode(signature, "UTF-8");
System.out.println(resourceUrl+"?"+sasToken);
} catch (UnsupportedEncodingException e) {
e.printStackTrace();
}
}
private static String getHMAC256(String accountKey, String signStr) {
String signature = null;
try {
SecretKeySpec secretKey = new SecretKeySpec(Base64.getDecoder().decode(accountKey), "HmacSHA256");
Mac sha256HMAC = Mac.getInstance("HmacSHA256");
sha256HMAC.init(secretKey);
signature = Base64.getEncoder().encodeToString(sha256HMAC.doFinal(signStr.getBytes("UTF-8")));
} catch (Exception e) {
e.printStackTrace();
}
return signature;
}
I got a simpler method
SharedAccessAccountPolicy sharedAccessAccountPolicy = new SharedAccessAccountPolicy();
sharedAccessAccountPolicy.setPermissionsFromString("racwdlup");
long date = new Date().getTime();
long expiryDate = new Date(date + 8640000).getTime();
sharedAccessAccountPolicy.setSharedAccessStartTime(new Date(date));
sharedAccessAccountPolicy.setSharedAccessExpiryTime(new Date(expiryDate));
sharedAccessAccountPolicy.setResourceTypeFromString("sco");
sharedAccessAccountPolicy.setServiceFromString("bfqt");
String sasToken = "?" + storageAccount.generateSharedAccessSignature(sharedAccessAccountPolicy);
You can get the storage account like this:
private String storageConnectionString = "DefaultEndpointsProtocol=https;AccountName=<storage name>;AccountKey=<your key>;EndpointSuffix=core.windows.net";
storageAccount = CloudStorageAccount.parse(storageConnectionString);
Hey guys!
I'm currently trying to create an app that uses the Twitter API to get timelines of users. I'm currently stuck at a specific point! My user has already logged in and I've already received the access token and the token secret. I'm now trying to send a get request to the Twitter server.
My problem is that I'm always getting a 400 bad request error code WITHOUT any kind of message.
I'm using Volley to send the requests - Heres the code
StringRequest stringRequest = new StringRequest(Request.Method.GET, url,
new Response.Listener<String>() {
#Override
public void onResponse(String response) {
}
}, new Response.ErrorListener() {
#Override
public void onErrorResponse(VolleyError error) {
Log.e(TAG, "Response Time: " + error.getNetworkTimeMs() + " ms");
Log.e(TAG, "Code: " + error.networkResponse.statusCode);
Log.e(TAG, "Data: " + new String(error.networkResponse.data));
}
}) {
#Override
public Map<String, String> getHeaders() throws AuthFailureError {
Map<String, String> params = new HashMap<String, String>();
Long tsLong = System.currentTimeMillis() / 1000;
//I receive all the difference parts
String consumerKey = context.getString(R.string.twitter_consumer_key);
String nonce = GenerationHelper.generateNonce();
String signature_method = "HMAC-SHA1";
String timestamp = tsLong.toString();
String token = sTwitterToken;
String version = "1.0";
// I use this list to pass the parameters to the function
// generating the signature
List<String> param= new ArrayList<>();
param.add("screen_name=" + username);
param.add("count=" + count);
param.add("oauth_token" + sTwitterToken);
param.add("oauth_consumer_key=" + consumerKey);
param.add("oauth_nonce=" + nonce);
param.add("oauth_signature_method=" + signature_method);
param.add("oauth_timestamp=" + timestamp);
param.add("oauth_version=" + version);
String signature = GenerationHelper.generateSignature(context, param, "POST", "https://api.twitter.com/1.1/statuses/user_timeline.json");
// I create the header String
StringBuilder paramBuilder = new StringBuilder();
paramBuilder.append("oauth_consumer_key=\"" + consumerKey + "\", ");
paramBuilder.append("oauth_nonce=\"" + nonce + "\", ");
paramBuilder.append("oauth_signature=\"" + signature + "\", ");
paramBuilder.append("oauth_signature_method=\"" + "HMAC-SHA1" + "\", ");
paramBuilder.append("oauth_timestamp=\"" + timestamp + "\", ");
paramBuilder.append("oauth_token=\"" + sTwitterToken + "\", ");
paramBuilder.append("oauth_version=\"" + "1.0" + "\"");
String credentialString = paramBuilder.toString();
Log.d(TAG, credentialString);
params.put("Authorization", "OAuth " + credentialString);
return params;
}
};
My current response is
Code: 400
Data:
If I remove the line adding the authorization data I get the response
Code: 400
Data: {"errors":[{"code":215,"message":"Bad Authentication data."}]}
I'm pretty sure that I don't get rate limited because I'm just sending about 10 requests per 15 minutes.
Does anybody have any idea why I'm having this problem?
This question is unlikely to help any future visitors; it is only relevant to a small geographic area, a specific moment in time, or an extraordinarily narrow situation that is not generally applicable to the worldwide audience of the internet. For help making this question more broadly applicable, visit the help center.
Closed 9 years ago.
I'm trying to use the Basic crawler example in crawler4j. I took the code from the crawler4j website here.
package edu.crawler;
import edu.uci.ics.crawler4j.crawler.Page;
import edu.uci.ics.crawler4j.crawler.WebCrawler;
import edu.uci.ics.crawler4j.parser.HtmlParseData;
import edu.uci.ics.crawler4j.url.WebURL;
import java.util.List;
import java.util.regex.Pattern;
import org.apache.http.Header;
public class MyCrawler extends WebCrawler {
private final static Pattern FILTERS = Pattern.compile(".*(\\.(css|js|bmp|gif|jpe?g" + "|png|tiff?|mid|mp2|mp3|mp4"
+ "|wav|avi|mov|mpeg|ram|m4v|pdf" + "|rm|smil|wmv|swf|wma|zip|rar|gz))$");
/**
* You should implement this function to specify whether the given url
* should be crawled or not (based on your crawling logic).
*/
#Override
public boolean shouldVisit(WebURL url) {
String href = url.getURL().toLowerCase();
return !FILTERS.matcher(href).matches() && href.startsWith("http://www.ics.uci.edu/");
}
/**
* This function is called when a page is fetched and ready to be processed
* by your program.
*/
#Override
public void visit(Page page) {
int docid = page.getWebURL().getDocid();
String url = page.getWebURL().getURL();
String domain = page.getWebURL().getDomain();
String path = page.getWebURL().getPath();
String subDomain = page.getWebURL().getSubDomain();
String parentUrl = page.getWebURL().getParentUrl();
String anchor = page.getWebURL().getAnchor();
System.out.println("Docid: " + docid);
System.out.println("URL: " + url);
System.out.println("Domain: '" + domain + "'");
System.out.println("Sub-domain: '" + subDomain + "'");
System.out.println("Path: '" + path + "'");
System.out.println("Parent page: " + parentUrl);
System.out.println("Anchor text: " + anchor);
if (page.getParseData() instanceof HtmlParseData) {
HtmlParseData htmlParseData = (HtmlParseData) page.getParseData();
String text = htmlParseData.getText();
String html = htmlParseData.getHtml();
List<WebURL> links = htmlParseData.getOutgoingUrls();
System.out.println("Text length: " + text.length());
System.out.println("Html length: " + html.length());
System.out.println("Number of outgoing links: " + links.size());
}
Header[] responseHeaders = page.getFetchResponseHeaders();
if (responseHeaders != null) {
System.out.println("Response headers:");
for (Header header : responseHeaders) {
System.out.println("\t" + header.getName() + ": " + header.getValue());
}
}
System.out.println("=============");
}
}
Above is the code for the crawler class from the example.
public class Controller {
public static void main(String[] args) throws Exception {
String crawlStorageFolder = "../data/";
int numberOfCrawlers = 7;
CrawlConfig config = new CrawlConfig();
config.setCrawlStorageFolder(crawlStorageFolder);
/*
* Instantiate the controller for this crawl.
*/
PageFetcher pageFetcher = new PageFetcher(config);
RobotstxtConfig robotstxtConfig = new RobotstxtConfig();
RobotstxtServer robotstxtServer = new RobotstxtServer(robotstxtConfig, pageFetcher);
CrawlController controller = new CrawlController(config, pageFetcher, robotstxtServer);
/*
* For each crawl, you need to add some seed urls. These are the first
* URLs that are fetched and then the crawler starts following links
* which are found in these pages
*/
controller.addSeed("http://www.ics.uci.edu/~welling/");
controller.addSeed("http://www.ics.uci.edu/~lopes/");
controller.addSeed("http://www.ics.uci.edu/");
/*
* Start the crawl. This is a blocking operation, meaning that your code
* will reach the line after this only when crawling is finished.
*/
controller.start(MyCrawler.class, numberOfCrawlers);
}
}
Above is the class for the controller class for the web crawler.
When I try to run the Controller class from my IDE (Intellij) I get the following error:
Exception in thread "main" java.lang.UnsupportedClassVersionError: edu/uci/ics/crawler4j/crawler/CrawlConfig : Unsupported major.minor version 51.0
Is there something about the maven config that is found here that I should know? Do I have to use a different version or something?
The problem wasn't with crawler4j. The problem was that the version of Java that I was using was different from the latest version of Java that is used in crawler4j. I switched the version right before they updated to Java 7 and everything worked fine. I'm guessing that upgrading my version of Java to 7 would have the same effect.