We Keep Coding

SignatureDoesNotMatch For PresignedURL in java

public S3PresignedURLServiceImpl() {
amazonS3Client = AmazonS3ClientBuilder
.standard()
.withCredentials(new DefaultAWSCredentialsProviderChain())
.withRegion(S3PresignedURLConstants.DEFAULT_REGION)
.build();
}
[In local it is working but when lambda is deployed on the console getting
SignatureDoesNotMatch The request signature we calculated does not match the signature you provided. Check your key and signing method.
try {
// Set the pre-signed URL to expire after specified time.
java.util.Date expiration = new java.util.Date();
long expTimeMillis = expiration.getTime();
if(data.getExpiryTime() > 0) {
expTimeMillis += 1000 * 60 * data.getExpiryTime();
} else {
expTimeMillis += 100 * 60 * 60 * 6;
}
expiration.setTime(expTimeMillis);
HttpMethod httpMethod = data.isUpload()?HttpMethod.PUT:HttpMethod.GET;
Logger.logInfo("Generating pre-signed URL.",REPORTER);
GeneratePresignedUrlRequest generatePresignedUrlRequest = new GeneratePresignedUrlRequest(data.getBucketName(), data.getFilePath())
.withMethod(httpMethod)
.withExpiration(expiration);
if(!data.isUpload())
generatePresignedUrlRequest.withVersionId(data.getVersionId());
else generatePresignedUrlRequest.withContentType(data.getContentType());
url = amazonS3Client.generatePresignedUrl(generatePresignedUrlRequest);
responseData.setPreSignedUrl(url.toString());
}
catch(Exception e) {
throw new S3PresignedURLException(e.getMessage(), e);
}

Please check version of library org.apache.httpcomponents:httpclient, if it's 4.5.7 or 4.5.8, try to downgrade to 4.5.6, as there exist issues for AWS SDK S3. For more details, please take a look at Amazon S3 Signature Does Not Match - AWS SDK Java and org.apache.httpcomponents:httpclient:4.5.7 breaks fetching S3 objects.

The solution we received when we reported a ticket with AWS because all the approaches failed. The scenario is we have our custom AWS KMS encryption enabled for S3 bucket, but we were trying to send "kms key" along with our request when using GeneratePresignedUrlRequest api. AWS said, we don't have to send KMS key, instead send without encrypting from client. When I say unencrypted, it is not exactly that, it is already coming in encrypted form and when we were using "AWSS3V4SinerType" to sign, it was signing an already encrypted file. Hope this makes sense.

AWS Amazon S3 Java SDK - Refresh credentials / token when expired while uploading large file

I'm trying to upload a large file to a server which uses a token and the token expires after 10 minutes, so if I upload a small file it will work therefore if the file is big than I will get some problems and will be trying to upload for ever while the access is denied
So I need refresh the token in the BasicAWSCredentials which is than used for the AWSStaticCredentialsProvider therefore I'm not sure how can i do it, please help =)
Worth to mention that we use a local server (not amazon cloud) with provides the token and for convenience we use amazon's code.
here is my code:
public void uploadMultipart(File file) throws Exception {
//this method will give you a initial token for a given user,
//than calculates when a new token is needed and will refresh it just when necessary
String token = getUsetToken();
String existingBucketName = myTenant.toLowerCase() + ".package.upload";
String endPoint = urlAPI + "s3/buckets/";
String strSize = FileUtils.byteCountToDisplaySize(FileUtils.sizeOf(file));
System.out.println("File size: " + strSize);
AwsClientBuilder.EndpointConfiguration endpointConfiguration = new AwsClientBuilder.EndpointConfiguration(endPoint, null);//note: Region has to be null
//AWSCredentialsProvider
BasicAWSCredentials sessionCredentials = new BasicAWSCredentials(token, "NOT_USED");//secretKey should be set to NOT_USED
AmazonS3 s3 = AmazonS3ClientBuilder
.standard()
.withCredentials(new AWSStaticCredentialsProvider(sessionCredentials))
.withEndpointConfiguration(endpointConfiguration)
.enablePathStyleAccess()
.build();
int maxUploadThreads = 5;
TransferManager tm = TransferManagerBuilder
.standard()
.withS3Client(s3)
.withMultipartUploadThreshold((long) (5 * 1024 * 1024))
.withExecutorFactory(() -> Executors.newFixedThreadPool(maxUploadThreads))
.build();
PutObjectRequest request = new PutObjectRequest(existingBucketName, file.getName(), file);
//request.putCustomRequestHeader("Access-Token", token);
ProgressListener progressListener = progressEvent -> System.out.println("Transferred bytes: " + progressEvent.getBytesTransferred());
request.setGeneralProgressListener(progressListener);
Upload upload = tm.upload(request);
LocalDateTime uploadStartedAt = LocalDateTime.now();
log.info("Starting upload at: " + uploadStartedAt);
try {
upload.waitForCompletion();
//upload.waitForUploadResult();
log.info("Upload completed. " + strSize);
} catch (Exception e) {//AmazonClientException
log.error("Error occurred while uploading file - " + strSize);
e.printStackTrace();
}
}

Solution found !
I found a way to get this working and for to be honest I quite happy about the result, I've done so many tests with big files (50gd.zip) and in every scenario worked very well
My solution is, remove the line: BasicAWSCredentials sessionCredentials = new BasicAWSCredentials(token, "NOT_USED");
AWSCredentials is a interface so we can override it with something dynamic, the the logic of when the token is expired and needs a new fresh token is held inside the getToken() method meaning you can call every time with no harm
AWSCredentials sessionCredentials = new AWSCredentials() {
#Override
public String getAWSAccessKeyId() {
try {
return getToken(); //getToken() method return a string
} catch (Exception e) {
return null;
}
}
#Override
public String getAWSSecretKey() {
return "NOT_USED";
}
};

When uploading a file (or parts of a multi-part file), the credentials that you use must last long enough for the upload to complete. You CANNOT refresh the credentials as there is no method to update AWS S3 that you are using new credentials for an already signed request.
You could break the upload into smaller files that upload quicker. Then only upload X parts. Refresh your credentials and upload Y parts. Repeat until all parts are uploaded. Then you will need to finish by combining the parts (which is a separate command). This is not a perfect solution as transfer speeds cannot be accurately controlled AND this means that you will have to write your own upload code (which is not hard).

AWS Creating new files from an s3 object using JAVA getting error

I have a shape file and i need to read the shape file from my java code. I used below code for reading shape file.
public class App {
public static void main(String[] args) {
File file = new File("C:\\Test\\sample.shp");
Map<String, Object> map = new HashMap<>();//
try {
map.put("url", URLs.fileToUrl(file));
DataStore dataStore = DataStoreFinder.getDataStore(map);
String typeName = dataStore.getTypeNames()[0];
SimpleFeatureSource source = dataStore.getFeatureSource(typeName);
SimpleFeatureCollection collection = source.getFeatures();
try (FeatureIterator<SimpleFeature> features = collection.features()) {
while (features.hasNext()) {
SimpleFeature feature = features.next();
SimpleFeatureType schema = feature.getFeatureType();
Class<?> geomType = schema.getGeometryDescriptor().getType().getBinding();
String type = "";
if (Polygon.class.isAssignableFrom(geomType) || MultiPolygon.class.isAssignableFrom(geomType)) {
MultiPolygon geom = (MultiPolygon) feature.getDefaultGeometry();
type = "Polygon";
if (geom.getNumGeometries() > 1) {
type = "MultiPolygon";
}
} else if (LineString.class.isAssignableFrom(geomType)
|| MultiLineString.class.isAssignableFrom(geomType)) {
} else {
}
System.out.println(feature.getDefaultGeometryProperty().getValue().toString());
}
}
} catch (Exception e) {
// TODO: handle exception
}
}
}
I got the desired output. But my requirement is write an aws lambda function to read shape file. For this
1. I created a Lambda java project of s3 event. I wrote the same code inside the handleRequest. I uploaded the java lambda project as a lanbda function and added one trigger. When I am uploading a .shp file to as s3 bucket lmbda function will automatically invoked. But I am getting an error like below
java.lang.RuntimeException: java.io.FileNotFoundException: /sample.shp (No such file or directory)
I have sample.shp file inside my s3 bucket. I go through below link.
How to write an S3 object to a file?
I am getting the same error. I tried to change my code like below
S3Object object = s3.getObject(new GetObjectRequest(bucket, key));
InputStream objectData = object.getObjectContent();
map.put("url", objectData );
instead of
File file = new File("C:\\Test\\sample.shp");
map.put("url", URLs.fileToUrl(file));
:-( Now i am getting an error like below
java.lang.NullPointerException
Also I tried the below code
DataStore dataStore = DataStoreFinder.getDataStore(objectData);
instead of
DataStore dataStore = DataStoreFinder.getDataStore(map);
the error was like below
java.lang.ClassCastException:
com.amazonaws.services.s3.model.S3ObjectInputStream cannot be cast to
java.util.Map
Also I tried to add key directly to the map and also as DataStore object. Everything went wrong..:-(
Is there anyone who can help me?
It will be very helpful if someone can do it for me...

The DataStoreFinder.getDataStore method in geotools requires you to provide a map containing a key/value pair with key "url". The value associated with that "url" key needs to be a file URL like "file://host/path/my.shp".
You're trying to insert a Java input stream into the map. That won't work, because it's not a file URL.
The geotools library does not accept http/https URLs (see the geotools code here and here), so you need a file:// URL. That means you will need to download the file from S3 to the local Lambda filesystem and then provide a file:// URL pointing to that local file. To do that, here's Java code that should work:
// get the shape file from S3 to local filesystem
File localshp = new File("/tmp/download.shp");
s3.getObject(new GetObjectRequest(bucket, key), localshp);
// now store file:// URL in the map
map.put("url", localshp.getURI().getURL().toString());
If the geotools library had accepted real URLs (not just file:// URLs) then you could have avoided the download and simply created a time-limited, pre-signed URL for the S3 object and put that URL into the map.
Here's an example of how to do that:
// get current time and add one hour
java.util.Date expiration = new java.util.Date();
long msec = expiration.getTime();
msec += 1000 * 60 * 60;
expiration.setTime(msec);
// request pre-signed URL that will allow bearer to GET the object
GeneratePresignedUrlRequest gpur = new GeneratePresignedUrlRequest(bucket, key);
gpur.setMethod(HttpMethod.GET);
gpur.setExpiration(expiration);
// get URL that will expire in one hour
URL url = s3.generatePresignedUrl(gpur);

Access GoogleCloudStorage using GoogleCloudEndpoints

I'm working on this project in which I'm using a Google-App-Engine backend connected to an Android app via Google-Cloud-Endpoints. For Google-Cloud-Datastore access I'm using Objectify and everything works fine.
Now I decided to add the functionality to upload images to Google-Cloud-Storage but I couldn't find a clear explanation on how to do this using the Google-Cloud-Endpoints setup.
I found the following explanation how to use Google-Cloud-Storage with Google-App-Engine:
https://cloud.google.com/appengine/docs/java/googlecloudstorageclient/app-engine-cloud-storage-sample
but instead of adding it to the Endpoints Api the article writes an additional servlet.
Furthermore I found this example of upload/download for Android:
github.com /thorrism/GoogleCloudExample
Sadly this is using the Google Cloud Storage API for direct access to the Google-Cloud-Storage and you need to add a P12-file to the asset folder, which seems unsecure.
My Google-App-Engine code looks like that:
#Api(
name = "example",
version = "v1",
scopes = { Constants.EMAIL_SCOPE },
clientIds = { Constants.WEB_CLIENT_ID, Constants.ANDROID_CLIENT_ID, Constants.API_EXPLORER_CLIENT_ID },
audiences = {Constants.ANDROID_AUDIENCE},
description = "API for the Example Backend application."
)
public class ExampleApi{
#ApiMethod(name = "doSomething", path = "dosomething", httpMethod = HttpMethod.POST)
public String doSomething(#Named("text") String text){
TestEntity test = new TestEntity(text);
ofy().save().entity(test).now();
return test;
}
After I uploaded it I generated the Endpoints Client Library and imported it into my android project.
Then I'm calling Endpoints from Android like explained here:
https://cloud.google.com/appengine/docs/java/endpoints/calling-from-android#creating_the_service_object
public static com.appspot.******.example.Example buildServiceHandler(Context context, String email) {
GoogleAccountCredential credential = GoogleAccountCredential.usingAudience(
context, AppConstants.AUDIENCE);
credential.setSelectedAccountName(email);
com.appspot.******.example.Example.Builder builder = new com.appspot.******.example.Example.Builder(
AppConstants.HTTP_TRANSPORT,
AppConstants.JSON_FACTORY, null);
builder.setApplicationName("example-server");
return builder.build();
}
sApiServiceHandler = buildServiceHandlerWithAuth(context,email);
And each Api-Method I call like this:
com.appspot.******.example.Example.DoSomething doSomething = sApiServiceHandler.doSomething(someString);
doSomething.execute();
All of this works fine, but only for storing/receiving Datastore Entities. How would I go about uploading/downloading files to Google Cloud Storage using the Google Cloud Endpoints setup?
Is it somehow possible to send a POST with my image data via Endpoints to the UploadServlet using the already build ServiceHandler ?
Is it possible to call a servlet from an Endpoints Method? How am I supposed to send the Post to the Servlet and how would I go about the authentication?
Any help or advice would be greatly appreciated!

There are different ways to do this, but the most recommended way is to use Signed URLs, so that your Android app can upload the file securely to Google Cloud Storage directly, without going through your Endpoints backend. The basic process is:
1) Create an Endpoints method that creates a new signed URL and returns it to the Android client. Signing the URL on the server still requires a P12 key but is stored on App Engine, not on the client, so is secure. Try to use a short expiration for the URL, for example no more than 5 minutes.
2) Have the Android client upload the file directly to the signed URL, as you would doing a normal HTTP PUT to the Cloud Storage XML API to upload a file (resumable uploads with the JSON API are also supported, but not covered here).
Your Endpoints method might look like this:
#ApiMethod(name = "getUploadUrl", path = "getuploadurl", httpMethod = HttpMethod.GET)
public MyApiResponse getUploadUrl(#Named("fileName") String fileName
#Named("contentType" String contentType)
{
String stringToSign
= "PUT\n" + contentType
+ "\n" + EXPIRATION_TIMESTAMP_EPOCH_SECONDS + "\n"
+ YOUR_GCS_BUCKET + "/" + fileName;
// Load P12 key
FileInputStream fileInputStream = new FileInputStream(PATH_TO_P12_KEY);
KeyStore keyStore = KeyStore.getInstance("PKCS12");
keyStore.load(fileInputStream, password);
PrivateKey key = keyStore.getKey(privatekey", YOUR_P12_KEY_PASSWORD);
// Get signature
Signature signer = Signature.getInstance("SHA256withRSA");
signer.initSign(key);
signer.update(stringToSign.getBytes("UTF-8"));
byte[] rawSignature = signer.sign();
String signature = new String(Base64.encodeBase64(rawSignature, false), "UTF-8");
// Construct signed url
String url
= "http://storage.googleapis.com/" + YOUR_GCS_BUCKET + fileName
+ "?GoogleAccessId=" + P12_KEY_SERVICE_ACCOUNT_CLIENT_ID
+ "&Expires=" + EXPIRATION_TIMESTAMP_EPOCH_SECONDS
+ "&Signature=" + URLEncoder.encode(signature, "UTF-8");
// Endpoints doesn't let you return 'String' directly
MyApiResponse response = new MyApiResponse();
response.setString(url);
return response;
}
On the Android side, you might use the method like this:
// Get the upload URL from the API
getUploadUrl = sApiServiceHandler.getUploadUrl(fileName, contentType);
MyApiResponse response = getUploadUrl.execute();
String uploadUrl = response.getString();
// Open connection to GCS
URL url = new URL(uploadUrl);
HttpURLConnection httpConnection = (HttpURLConnection) url.openConnection();
httpConnection.setDoOutput(true);
httpConnection.setRequestMethod("PUT");
httpConnection.setRequestProperty("Content-Type", contentType);
// Write file data
OutputStreamWriter out = new OutputStreamWriter(httpConnection.getOutputStream());
out.write(fileData);
out.flush();
// Get response, check status code etc.
InputStreamReader in = new InputStreamReader(httpConnection.getInputStream());
// ...
(Disclaimer: I'm just typing code freely into a text editor but not actually testing it, but it should be enough to give you a general idea.)

Want to upload file to s3 bucket using presigned url

I am having trouble to upload image using presigned url . I am following amazon java code but it is not working.
My requirement is as follows
I have created bucket on Amazon XYZBucket and my bucket is empty.
I am acting as a server which gives presigned url to user and user will use this url to upload image.
Code to generate presigned url
AmazonS3 s3client = new AmazonS3Client(new ProfileCredentialsProvider());
URL url = null;
try {
java.util.Date expiration = new java.util.Date();
long milliSeconds = expiration.getTime();
milliSeconds += 1000 * 60 * 60 * 12; // Add 1 hour.
expiration.setTime(milliSeconds);
GeneratePresignedUrlRequest generatePresignedUrlRequest =
new GeneratePresignedUrlRequest(bucketName, objectKey);
generatePresignedUrlRequest.setMethod(HttpMethod.GET);
generatePresignedUrlRequest.setExpiration(expiration);
url = s3client.generatePresignedUrl(generatePresignedUrlRequest);
} catch (AmazonServiceException exception) {
} catch (AmazonClientException ace) {
}
return url.toString();
I have also use put method
AmazonS3 s3Client = new AmazonS3Client(new ProfileCredentialsProvider());
java.util.Date expiration = new java.util.Date();
long msec = expiration.getTime();
msec += 1000 * 60 * 60; // Add 1 hour.
expiration.setTime(msec);
GeneratePresignedUrlRequest generatePresignedUrlRequest = new GeneratePresignedUrlRequest(bucketName, objectKey);
generatePresignedUrlRequest.setMethod(HttpMethod.PUT);
generatePresignedUrlRequest.setExpiration(expiration);
URL url = s3Client.generatePresignedUrl(generatePresignedUrlRequest);
return url.toString()
My bucketName and objectkey is
XYZBucket and file1
When I hit the url in browser it gives me
SignatureDoesNotMatch
error.
Can anyone help me to upload file using presigned url to s3 bucket?

According to the AWS documentation, you should use the "PUT" method to create an "upload" URL. Then the user will make a "PUT" request on this URL to upload its files.
Hitting this URL within the browser will make a "GET" request, but the signature contains "PUT" so it throws a SignatureDoesNotMatch error.

According to the AWS S3 documentation Signing and Authenticating REST request, S3 is now using SignatureVersion4 by default.
But the AWS-SDK is using SignatureVersion2 by default.
So we have to explicitly specify SignatureVersion4 in request header