We Keep Coding

Minio presigned url not working in browser

I am generating minio presigned url using code and then trying to open the url in browser but when i am opening that url in browser it gives me below error:
<Code>SignatureDoesNotMatch</Code>
<Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message>
<Key>PAEHXVONAHDVJ/A</Key>
<BucketName>test</BucketName>
<Resource>/test/PAEHXVONAHDVJ/A</Resource>
<RequestId>173708A8FD5721F0</RequestId>
<HostId>fcc0c2f9-167f-4502-981c-61a3fedb3487</HostId>
</Error>
Below is the code i am using to create pre signed url :
Map<String, String> reqParams = new HashMap<>();
reqParams.put("response-content-type", "application/json");
String minioUploadUrl = "";
try {
minioUploadUrl = minioPrimaryClient.getPresignedObjectUrl(
GetPresignedObjectUrlArgs.builder()
.method(Method.GET)
.bucket(org) //org is variable here
.object(tId + "/" + task) //task is variable
.expiry(60 * 60)
.extraQueryParams(reqParams)
.build());
} catch (Exception e) {
throw new RuntimeException(e);
}
return minioUploadUrl;
Basically i am creating url which will have two folders inside the mentioned bucket and i would like ti upload multiple files inside that.
Same url which is getting generated when i am using through postman it's working fine but in minio server it gets uploaded without name.

SignatureDoesNotMatch For PresignedURL in java

public S3PresignedURLServiceImpl() {
amazonS3Client = AmazonS3ClientBuilder
.standard()
.withCredentials(new DefaultAWSCredentialsProviderChain())
.withRegion(S3PresignedURLConstants.DEFAULT_REGION)
.build();
}
[In local it is working but when lambda is deployed on the console getting
SignatureDoesNotMatch The request signature we calculated does not match the signature you provided. Check your key and signing method.
try {
// Set the pre-signed URL to expire after specified time.
java.util.Date expiration = new java.util.Date();
long expTimeMillis = expiration.getTime();
if(data.getExpiryTime() > 0) {
expTimeMillis += 1000 * 60 * data.getExpiryTime();
} else {
expTimeMillis += 100 * 60 * 60 * 6;
}
expiration.setTime(expTimeMillis);
HttpMethod httpMethod = data.isUpload()?HttpMethod.PUT:HttpMethod.GET;
Logger.logInfo("Generating pre-signed URL.",REPORTER);
GeneratePresignedUrlRequest generatePresignedUrlRequest = new GeneratePresignedUrlRequest(data.getBucketName(), data.getFilePath())
.withMethod(httpMethod)
.withExpiration(expiration);
if(!data.isUpload())
generatePresignedUrlRequest.withVersionId(data.getVersionId());
else generatePresignedUrlRequest.withContentType(data.getContentType());
url = amazonS3Client.generatePresignedUrl(generatePresignedUrlRequest);
responseData.setPreSignedUrl(url.toString());
}
catch(Exception e) {
throw new S3PresignedURLException(e.getMessage(), e);
}

Please check version of library org.apache.httpcomponents:httpclient, if it's 4.5.7 or 4.5.8, try to downgrade to 4.5.6, as there exist issues for AWS SDK S3. For more details, please take a look at Amazon S3 Signature Does Not Match - AWS SDK Java and org.apache.httpcomponents:httpclient:4.5.7 breaks fetching S3 objects.

The solution we received when we reported a ticket with AWS because all the approaches failed. The scenario is we have our custom AWS KMS encryption enabled for S3 bucket, but we were trying to send "kms key" along with our request when using GeneratePresignedUrlRequest api. AWS said, we don't have to send KMS key, instead send without encrypting from client. When I say unencrypted, it is not exactly that, it is already coming in encrypted form and when we were using "AWSS3V4SinerType" to sign, it was signing an already encrypted file. Hope this makes sense.

AWS Amazon S3 Java SDK - Refresh credentials / token when expired while uploading large file

I'm trying to upload a large file to a server which uses a token and the token expires after 10 minutes, so if I upload a small file it will work therefore if the file is big than I will get some problems and will be trying to upload for ever while the access is denied
So I need refresh the token in the BasicAWSCredentials which is than used for the AWSStaticCredentialsProvider therefore I'm not sure how can i do it, please help =)
Worth to mention that we use a local server (not amazon cloud) with provides the token and for convenience we use amazon's code.
here is my code:
public void uploadMultipart(File file) throws Exception {
//this method will give you a initial token for a given user,
//than calculates when a new token is needed and will refresh it just when necessary
String token = getUsetToken();
String existingBucketName = myTenant.toLowerCase() + ".package.upload";
String endPoint = urlAPI + "s3/buckets/";
String strSize = FileUtils.byteCountToDisplaySize(FileUtils.sizeOf(file));
System.out.println("File size: " + strSize);
AwsClientBuilder.EndpointConfiguration endpointConfiguration = new AwsClientBuilder.EndpointConfiguration(endPoint, null);//note: Region has to be null
//AWSCredentialsProvider
BasicAWSCredentials sessionCredentials = new BasicAWSCredentials(token, "NOT_USED");//secretKey should be set to NOT_USED
AmazonS3 s3 = AmazonS3ClientBuilder
.standard()
.withCredentials(new AWSStaticCredentialsProvider(sessionCredentials))
.withEndpointConfiguration(endpointConfiguration)
.enablePathStyleAccess()
.build();
int maxUploadThreads = 5;
TransferManager tm = TransferManagerBuilder
.standard()
.withS3Client(s3)
.withMultipartUploadThreshold((long) (5 * 1024 * 1024))
.withExecutorFactory(() -> Executors.newFixedThreadPool(maxUploadThreads))
.build();
PutObjectRequest request = new PutObjectRequest(existingBucketName, file.getName(), file);
//request.putCustomRequestHeader("Access-Token", token);
ProgressListener progressListener = progressEvent -> System.out.println("Transferred bytes: " + progressEvent.getBytesTransferred());
request.setGeneralProgressListener(progressListener);
Upload upload = tm.upload(request);
LocalDateTime uploadStartedAt = LocalDateTime.now();
log.info("Starting upload at: " + uploadStartedAt);
try {
upload.waitForCompletion();
//upload.waitForUploadResult();
log.info("Upload completed. " + strSize);
} catch (Exception e) {//AmazonClientException
log.error("Error occurred while uploading file - " + strSize);
e.printStackTrace();
}
}

Solution found !
I found a way to get this working and for to be honest I quite happy about the result, I've done so many tests with big files (50gd.zip) and in every scenario worked very well
My solution is, remove the line: BasicAWSCredentials sessionCredentials = new BasicAWSCredentials(token, "NOT_USED");
AWSCredentials is a interface so we can override it with something dynamic, the the logic of when the token is expired and needs a new fresh token is held inside the getToken() method meaning you can call every time with no harm
AWSCredentials sessionCredentials = new AWSCredentials() {
#Override
public String getAWSAccessKeyId() {
try {
return getToken(); //getToken() method return a string
} catch (Exception e) {
return null;
}
}
#Override
public String getAWSSecretKey() {
return "NOT_USED";
}
};

When uploading a file (or parts of a multi-part file), the credentials that you use must last long enough for the upload to complete. You CANNOT refresh the credentials as there is no method to update AWS S3 that you are using new credentials for an already signed request.
You could break the upload into smaller files that upload quicker. Then only upload X parts. Refresh your credentials and upload Y parts. Repeat until all parts are uploaded. Then you will need to finish by combining the parts (which is a separate command). This is not a perfect solution as transfer speeds cannot be accurately controlled AND this means that you will have to write your own upload code (which is not hard).

Java amazonS3.generatePresignedUrl - How to configure https://s3.amazonaws.com/mycompany instead of https://mycompany.s3.amazonaws.com/com.mycompany

We are using presigned s3 urls to provide web access to images stored in s3.
The java code we are using to generate the presigned urls is similar to below
String accessKey = ...;
String secretKey = ...;
String region = ...;
com.amazonaws.HttpMethod awsHttpMethod = ...;
String bucketName = ...;
String objectKey = ...;
Date expirationDate = ...;
BasicAWSCredentials creds = new BasicAWSCredentials(accessKey, secretKey);
AmazonS3 s3Client = AmazonS3ClientBuilder.standard().withCredentials(new AWSStaticCredentialsProvider(creds)).withRegion(region).build();
GeneratePresignedUrlRequest generatePresignedUrlRequest = new GeneratePresignedUrlRequest(bucketName, objectKey);
generatePresignedUrlRequest.setMethod(awsHttpMethod);
generatePresignedUrlRequest.setExpiration(expirationDate);
URL url = s3Client.generatePresignedUrl(generatePresignedUrlRequest);
The url that is generated by the code looks similar to
https://com.mycompany.personalpictures.s3.amazonaws.com/picture123.png?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20170623T150540Z&X-Amz-SignedHeaders=host&X-Amz-Expires=59&X-Amz-Credential=AKIAIVLB4ANK6B45G3IA%2F20170623%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Signature=d25d407ee8efa76f339388ec93579a19be8eaead9663d6d378cf2ec6d9d9cac2
However since our bucket naming standard contains dots, a call to above URL results in a SSL: no alternative certificate subject name matches target host name 'com.mycompany.personalpictures.s3.amazonaws.com' error
I read in this post that the root cause is the dots in the bucket name and that using https://s3.amazonaws.com/com.mycompany.personalpictures/picture123.png should circumvent the problem.
How can I generate presigned urls using the url format https://s3.amazonaws.com/mybucket/myfile?

Figured it out...
Needed to use .enablePathStyleAccess() when creating the s3 client. With that the code line now is
AmazonS3 s3Client = AmazonS3ClientBuilder.standard().withCredentials(new AWSStaticCredentialsProvider(creds)).withRegion(region).enablePathStyleAccess().build();

Not able to upload image using Amazon S3 SDK in Android application. 400 error

I am trying to implement Amazon S3 SDK to upload image from Android application but always getting this 400 error, malformed xml bad request. I've taken source code from this reference link
I've correct access_key, secret_key and bucket_key. There is no error related to this.
If anybody have a working sample in order to access this S3 API, please share. I am not able to find SDK error and stuck with the same.

I think this would help you to upload image
String EXPIRY_DATE = "Mon, Jan 1 2030 11:11:11 GMT";
BasicAWSCredentials awsCreds = new BasicAWSCredentials(ACCESS_KEY_ID, SECRET_ACCESS_KEY);
AmazonS3Client s3Client = new AmazonS3Client(awsCreds);
ResponseHeaderOverrides override = new ResponseHeaderOverrides();
override.setContentType("image/jpeg");
override.setExpires(EXPIRY_DATE);
File imageFile = new File(path);
PutObjectRequest pros = new PutObjectRequest(BUCKET_NAME, BUCKET_PATH, imageFile);
ObjectMetadata meta = new ObjectMetadata();
meta.addUserMetadata("expires", EXPIRY_DATE);
meta.setHeader("expires", EXPIRY_DATE);
pros.setMetadata(meta);
s3Client.putObject(pros);
EDIT
Use this for generating URL
String generatedURL=null;
try {
String key = BUCKET_PATH + ImageName;
GeneratePresignedUrlRequest urlRequest = new GeneratePresignedUrlRequest(BUCKET_NAME, key);
Date date1 = new Date(EXPIRY_DATE);
urlRequest.setExpiration(date1);
urlRequest.setResponseHeaders(override);
URL url = s3Client.generatePresignedUrl(urlRequest);
generatedURL = url.toString();
} catch (Exception e) {
logger.error("Error at ", e);
}