We Keep Coding

Minio presigned url not working in browser

I am generating minio presigned url using code and then trying to open the url in browser but when i am opening that url in browser it gives me below error:
<Code>SignatureDoesNotMatch</Code>
<Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message>
<Key>PAEHXVONAHDVJ/A</Key>
<BucketName>test</BucketName>
<Resource>/test/PAEHXVONAHDVJ/A</Resource>
<RequestId>173708A8FD5721F0</RequestId>
<HostId>fcc0c2f9-167f-4502-981c-61a3fedb3487</HostId>
</Error>
Below is the code i am using to create pre signed url :
Map<String, String> reqParams = new HashMap<>();
reqParams.put("response-content-type", "application/json");
String minioUploadUrl = "";
try {
minioUploadUrl = minioPrimaryClient.getPresignedObjectUrl(
GetPresignedObjectUrlArgs.builder()
.method(Method.GET)
.bucket(org) //org is variable here
.object(tId + "/" + task) //task is variable
.expiry(60 * 60)
.extraQueryParams(reqParams)
.build());
} catch (Exception e) {
throw new RuntimeException(e);
}
return minioUploadUrl;
Basically i am creating url which will have two folders inside the mentioned bucket and i would like ti upload multiple files inside that.
Same url which is getting generated when i am using through postman it's working fine but in minio server it gets uploaded without name.

How to log every Retry operation in S3 in java

I am uploading certain media files to S3 using an existing code. Due to some connectivity issue I am getting exceptions and the upload requests are getting cancelled.
I added a retry policy so in case the connectivity issue is intermittent, the retry policy will take care of the connectivity issue and continue with the upload request.
Here is my retry policy and the client config:
private static RetryPolicy getS3BaseRetryPolicy() {
return new RetryPolicy( new PredefinedRetryPolicies.SDKDefaultRetryCondition(),
new PredefinedBackoffStrategies.ExponentialBackoffStrategy(2000, 300), PredefinedRetryPolicies.DEFAULT_MAX_ERROR_RETRY,true);
}
AWSCredentials credentials = new BasicAWSCredentials(EncryptDecryptUtils.decrypt(s3AK), EncryptDecryptUtils.decrypt(s3SK));
ClientConfiguration clientConfiguration = new ClientConfiguration().withProtocol(Protocol.HTTPS).withTcpKeepAlive(true).withRetryPolicy(getS3BaseRetryPolicy());
s3Client = new AmazonS3Client(credentials, clientConfiguration);
s3Client.setS3ClientOptions(S3ClientOptions.builder().setPathStyleAccess(true).build());
s3Client.setEndpoint(s3Url);
Here is my uploading logic:
ObjectMetadata objectMetadata = new ObjectMetadata();
objectMetadata.setHeader(Constants.VIVA_REGION_HEADER, region);
if(s3client.getUrl(s3BucketName, key).toString().contains("amazon")) {
objectMetadata.setSSEAlgorithm(ObjectMetadata.AES_256_SERVER_SIDE_ENCRYPTION);
}
PutObjectRequest putRequest = new PutObjectRequest(s3BucketName, key, inputStream, objectMetadata);
PutObjectResult response = s3client.putObject(putRequest);
LOGGER.error("ObjectStore>uploadAudioInStream ==>> Uploaded object[" + key + "] with encryption Algorithm : [" + response.getSSEAlgorithm() + "]");
return URLDecoder.decode(s3client.getUrl(s3BucketName, key).toString(), "UTF-8");
Now the issue is, if the connection issue comes up, I am not sure if the retry mechanism is working because in the log I only see one single failure entry. It does not show whether retry has happened 3 times (my default max retry is set to 3) and failed 3 times.
Is there a way I can log each of the retry mechanism and its output so as to ensure that the retry mechanism is working.
Thanks

AWS Amazon S3 Java SDK - Refresh credentials / token when expired while uploading large file

I'm trying to upload a large file to a server which uses a token and the token expires after 10 minutes, so if I upload a small file it will work therefore if the file is big than I will get some problems and will be trying to upload for ever while the access is denied
So I need refresh the token in the BasicAWSCredentials which is than used for the AWSStaticCredentialsProvider therefore I'm not sure how can i do it, please help =)
Worth to mention that we use a local server (not amazon cloud) with provides the token and for convenience we use amazon's code.
here is my code:
public void uploadMultipart(File file) throws Exception {
//this method will give you a initial token for a given user,
//than calculates when a new token is needed and will refresh it just when necessary
String token = getUsetToken();
String existingBucketName = myTenant.toLowerCase() + ".package.upload";
String endPoint = urlAPI + "s3/buckets/";
String strSize = FileUtils.byteCountToDisplaySize(FileUtils.sizeOf(file));
System.out.println("File size: " + strSize);
AwsClientBuilder.EndpointConfiguration endpointConfiguration = new AwsClientBuilder.EndpointConfiguration(endPoint, null);//note: Region has to be null
//AWSCredentialsProvider
BasicAWSCredentials sessionCredentials = new BasicAWSCredentials(token, "NOT_USED");//secretKey should be set to NOT_USED
AmazonS3 s3 = AmazonS3ClientBuilder
.standard()
.withCredentials(new AWSStaticCredentialsProvider(sessionCredentials))
.withEndpointConfiguration(endpointConfiguration)
.enablePathStyleAccess()
.build();
int maxUploadThreads = 5;
TransferManager tm = TransferManagerBuilder
.standard()
.withS3Client(s3)
.withMultipartUploadThreshold((long) (5 * 1024 * 1024))
.withExecutorFactory(() -> Executors.newFixedThreadPool(maxUploadThreads))
.build();
PutObjectRequest request = new PutObjectRequest(existingBucketName, file.getName(), file);
//request.putCustomRequestHeader("Access-Token", token);
ProgressListener progressListener = progressEvent -> System.out.println("Transferred bytes: " + progressEvent.getBytesTransferred());
request.setGeneralProgressListener(progressListener);
Upload upload = tm.upload(request);
LocalDateTime uploadStartedAt = LocalDateTime.now();
log.info("Starting upload at: " + uploadStartedAt);
try {
upload.waitForCompletion();
//upload.waitForUploadResult();
log.info("Upload completed. " + strSize);
} catch (Exception e) {//AmazonClientException
log.error("Error occurred while uploading file - " + strSize);
e.printStackTrace();
}
}

Solution found !
I found a way to get this working and for to be honest I quite happy about the result, I've done so many tests with big files (50gd.zip) and in every scenario worked very well
My solution is, remove the line: BasicAWSCredentials sessionCredentials = new BasicAWSCredentials(token, "NOT_USED");
AWSCredentials is a interface so we can override it with something dynamic, the the logic of when the token is expired and needs a new fresh token is held inside the getToken() method meaning you can call every time with no harm
AWSCredentials sessionCredentials = new AWSCredentials() {
#Override
public String getAWSAccessKeyId() {
try {
return getToken(); //getToken() method return a string
} catch (Exception e) {
return null;
}
}
#Override
public String getAWSSecretKey() {
return "NOT_USED";
}
};

When uploading a file (or parts of a multi-part file), the credentials that you use must last long enough for the upload to complete. You CANNOT refresh the credentials as there is no method to update AWS S3 that you are using new credentials for an already signed request.
You could break the upload into smaller files that upload quicker. Then only upload X parts. Refresh your credentials and upload Y parts. Repeat until all parts are uploaded. Then you will need to finish by combining the parts (which is a separate command). This is not a perfect solution as transfer speeds cannot be accurately controlled AND this means that you will have to write your own upload code (which is not hard).

Access GoogleCloudStorage using GoogleCloudEndpoints

I'm working on this project in which I'm using a Google-App-Engine backend connected to an Android app via Google-Cloud-Endpoints. For Google-Cloud-Datastore access I'm using Objectify and everything works fine.
Now I decided to add the functionality to upload images to Google-Cloud-Storage but I couldn't find a clear explanation on how to do this using the Google-Cloud-Endpoints setup.
I found the following explanation how to use Google-Cloud-Storage with Google-App-Engine:
https://cloud.google.com/appengine/docs/java/googlecloudstorageclient/app-engine-cloud-storage-sample
but instead of adding it to the Endpoints Api the article writes an additional servlet.
Furthermore I found this example of upload/download for Android:
github.com /thorrism/GoogleCloudExample
Sadly this is using the Google Cloud Storage API for direct access to the Google-Cloud-Storage and you need to add a P12-file to the asset folder, which seems unsecure.
My Google-App-Engine code looks like that:
#Api(
name = "example",
version = "v1",
scopes = { Constants.EMAIL_SCOPE },
clientIds = { Constants.WEB_CLIENT_ID, Constants.ANDROID_CLIENT_ID, Constants.API_EXPLORER_CLIENT_ID },
audiences = {Constants.ANDROID_AUDIENCE},
description = "API for the Example Backend application."
)
public class ExampleApi{
#ApiMethod(name = "doSomething", path = "dosomething", httpMethod = HttpMethod.POST)
public String doSomething(#Named("text") String text){
TestEntity test = new TestEntity(text);
ofy().save().entity(test).now();
return test;
}
After I uploaded it I generated the Endpoints Client Library and imported it into my android project.
Then I'm calling Endpoints from Android like explained here:
https://cloud.google.com/appengine/docs/java/endpoints/calling-from-android#creating_the_service_object
public static com.appspot.******.example.Example buildServiceHandler(Context context, String email) {
GoogleAccountCredential credential = GoogleAccountCredential.usingAudience(
context, AppConstants.AUDIENCE);
credential.setSelectedAccountName(email);
com.appspot.******.example.Example.Builder builder = new com.appspot.******.example.Example.Builder(
AppConstants.HTTP_TRANSPORT,
AppConstants.JSON_FACTORY, null);
builder.setApplicationName("example-server");
return builder.build();
}
sApiServiceHandler = buildServiceHandlerWithAuth(context,email);
And each Api-Method I call like this:
com.appspot.******.example.Example.DoSomething doSomething = sApiServiceHandler.doSomething(someString);
doSomething.execute();
All of this works fine, but only for storing/receiving Datastore Entities. How would I go about uploading/downloading files to Google Cloud Storage using the Google Cloud Endpoints setup?
Is it somehow possible to send a POST with my image data via Endpoints to the UploadServlet using the already build ServiceHandler ?
Is it possible to call a servlet from an Endpoints Method? How am I supposed to send the Post to the Servlet and how would I go about the authentication?
Any help or advice would be greatly appreciated!

There are different ways to do this, but the most recommended way is to use Signed URLs, so that your Android app can upload the file securely to Google Cloud Storage directly, without going through your Endpoints backend. The basic process is:
1) Create an Endpoints method that creates a new signed URL and returns it to the Android client. Signing the URL on the server still requires a P12 key but is stored on App Engine, not on the client, so is secure. Try to use a short expiration for the URL, for example no more than 5 minutes.
2) Have the Android client upload the file directly to the signed URL, as you would doing a normal HTTP PUT to the Cloud Storage XML API to upload a file (resumable uploads with the JSON API are also supported, but not covered here).
Your Endpoints method might look like this:
#ApiMethod(name = "getUploadUrl", path = "getuploadurl", httpMethod = HttpMethod.GET)
public MyApiResponse getUploadUrl(#Named("fileName") String fileName
#Named("contentType" String contentType)
{
String stringToSign
= "PUT\n" + contentType
+ "\n" + EXPIRATION_TIMESTAMP_EPOCH_SECONDS + "\n"
+ YOUR_GCS_BUCKET + "/" + fileName;
// Load P12 key
FileInputStream fileInputStream = new FileInputStream(PATH_TO_P12_KEY);
KeyStore keyStore = KeyStore.getInstance("PKCS12");
keyStore.load(fileInputStream, password);
PrivateKey key = keyStore.getKey(privatekey", YOUR_P12_KEY_PASSWORD);
// Get signature
Signature signer = Signature.getInstance("SHA256withRSA");
signer.initSign(key);
signer.update(stringToSign.getBytes("UTF-8"));
byte[] rawSignature = signer.sign();
String signature = new String(Base64.encodeBase64(rawSignature, false), "UTF-8");
// Construct signed url
String url
= "http://storage.googleapis.com/" + YOUR_GCS_BUCKET + fileName
+ "?GoogleAccessId=" + P12_KEY_SERVICE_ACCOUNT_CLIENT_ID
+ "&Expires=" + EXPIRATION_TIMESTAMP_EPOCH_SECONDS
+ "&Signature=" + URLEncoder.encode(signature, "UTF-8");
// Endpoints doesn't let you return 'String' directly
MyApiResponse response = new MyApiResponse();
response.setString(url);
return response;
}
On the Android side, you might use the method like this:
// Get the upload URL from the API
getUploadUrl = sApiServiceHandler.getUploadUrl(fileName, contentType);
MyApiResponse response = getUploadUrl.execute();
String uploadUrl = response.getString();
// Open connection to GCS
URL url = new URL(uploadUrl);
HttpURLConnection httpConnection = (HttpURLConnection) url.openConnection();
httpConnection.setDoOutput(true);
httpConnection.setRequestMethod("PUT");
httpConnection.setRequestProperty("Content-Type", contentType);
// Write file data
OutputStreamWriter out = new OutputStreamWriter(httpConnection.getOutputStream());
out.write(fileData);
out.flush();
// Get response, check status code etc.
InputStreamReader in = new InputStreamReader(httpConnection.getInputStream());
// ...
(Disclaimer: I'm just typing code freely into a text editor but not actually testing it, but it should be enough to give you a general idea.)

Want to upload file to s3 bucket using presigned url

I am having trouble to upload image using presigned url . I am following amazon java code but it is not working.
My requirement is as follows
I have created bucket on Amazon XYZBucket and my bucket is empty.
I am acting as a server which gives presigned url to user and user will use this url to upload image.
Code to generate presigned url
AmazonS3 s3client = new AmazonS3Client(new ProfileCredentialsProvider());
URL url = null;
try {
java.util.Date expiration = new java.util.Date();
long milliSeconds = expiration.getTime();
milliSeconds += 1000 * 60 * 60 * 12; // Add 1 hour.
expiration.setTime(milliSeconds);
GeneratePresignedUrlRequest generatePresignedUrlRequest =
new GeneratePresignedUrlRequest(bucketName, objectKey);
generatePresignedUrlRequest.setMethod(HttpMethod.GET);
generatePresignedUrlRequest.setExpiration(expiration);
url = s3client.generatePresignedUrl(generatePresignedUrlRequest);
} catch (AmazonServiceException exception) {
} catch (AmazonClientException ace) {
}
return url.toString();
I have also use put method
AmazonS3 s3Client = new AmazonS3Client(new ProfileCredentialsProvider());
java.util.Date expiration = new java.util.Date();
long msec = expiration.getTime();
msec += 1000 * 60 * 60; // Add 1 hour.
expiration.setTime(msec);
GeneratePresignedUrlRequest generatePresignedUrlRequest = new GeneratePresignedUrlRequest(bucketName, objectKey);
generatePresignedUrlRequest.setMethod(HttpMethod.PUT);
generatePresignedUrlRequest.setExpiration(expiration);
URL url = s3Client.generatePresignedUrl(generatePresignedUrlRequest);
return url.toString()
My bucketName and objectkey is
XYZBucket and file1
When I hit the url in browser it gives me
SignatureDoesNotMatch
error.
Can anyone help me to upload file using presigned url to s3 bucket?

According to the AWS documentation, you should use the "PUT" method to create an "upload" URL. Then the user will make a "PUT" request on this URL to upload its files.
Hitting this URL within the browser will make a "GET" request, but the signature contains "PUT" so it throws a SignatureDoesNotMatch error.

According to the AWS S3 documentation Signing and Authenticating REST request, S3 is now using SignatureVersion4 by default.
But the AWS-SDK is using SignatureVersion2 by default.
So we have to explicitly specify SignatureVersion4 in request header