I am generating minio presigned url using code and then trying to open the url in browser but when i am opening that url in browser it gives me below error:
<Code>SignatureDoesNotMatch</Code>
<Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message>
<Key>PAEHXVONAHDVJ/A</Key>
<BucketName>test</BucketName>
<Resource>/test/PAEHXVONAHDVJ/A</Resource>
<RequestId>173708A8FD5721F0</RequestId>
<HostId>fcc0c2f9-167f-4502-981c-61a3fedb3487</HostId>
</Error>
Below is the code i am using to create pre signed url :
Map<String, String> reqParams = new HashMap<>();
reqParams.put("response-content-type", "application/json");
String minioUploadUrl = "";
try {
minioUploadUrl = minioPrimaryClient.getPresignedObjectUrl(
GetPresignedObjectUrlArgs.builder()
.method(Method.GET)
.bucket(org) //org is variable here
.object(tId + "/" + task) //task is variable
.expiry(60 * 60)
.extraQueryParams(reqParams)
.build());
} catch (Exception e) {
throw new RuntimeException(e);
}
return minioUploadUrl;
Basically i am creating url which will have two folders inside the mentioned bucket and i would like ti upload multiple files inside that.
Same url which is getting generated when i am using through postman it's working fine but in minio server it gets uploaded without name.
The following is the source code generated url from cloud storage that I made
public String generateImageUrl(String fileName, Integer duration, String folderName) throws IOException {
Credentials credentials = GoogleCredentials.fromStream(new ClassPathResource(STORAGE_FILE_NAME).getInputStream());
Storage storage = StorageOptions.newBuilder().setCredentials(credentials).setProjectId(PROJECT_ID).build().getService();
Bucket bucket = storage.get(BUCKET_NAME, Storage.BucketGetOption.fields(Storage.BucketField.values()));
logger.info("Bucket name : " + bucket.getName());
String fullImagePath = folderName + "/" + fileName;
BlobId imgId = BlobId.of(BUCKET_NAME, fullImagePath);
if(null!=imgId) {
Blob blob = storage.get(imgId);
if(null!=blob && blob.exists()) {
URL signedUrl = storage.signUrl(blob, duration, TimeUnit.MINUTES);
String imageUrl = signedUrl.toExternalForm();
logger.info("Generated image url : " + imageUrl);
return imageUrl;
}
}
return null;
}
But the generated url has an access duration
how to implement it so that no access duration is given ??
It's not possible to create a signed url with no access duration.
Cloud Storage Signed Urls have a maximum expiration delay of 7 days.
You specify an expiration time when you create the signed URL. Anyone
who knows the URL can access the resource until the expiration time
for the URL is reached or the key used to sign the URL is rotated.
Excerpt from Java client library docs
Note that V4 Signed URLs can't have an expiration longer than 7 days.
You can also check more details here.
I'm trying to upload a large file to a server which uses a token and the token expires after 10 minutes, so if I upload a small file it will work therefore if the file is big than I will get some problems and will be trying to upload for ever while the access is denied
So I need refresh the token in the BasicAWSCredentials which is than used for the AWSStaticCredentialsProvider therefore I'm not sure how can i do it, please help =)
Worth to mention that we use a local server (not amazon cloud) with provides the token and for convenience we use amazon's code.
here is my code:
public void uploadMultipart(File file) throws Exception {
//this method will give you a initial token for a given user,
//than calculates when a new token is needed and will refresh it just when necessary
String token = getUsetToken();
String existingBucketName = myTenant.toLowerCase() + ".package.upload";
String endPoint = urlAPI + "s3/buckets/";
String strSize = FileUtils.byteCountToDisplaySize(FileUtils.sizeOf(file));
System.out.println("File size: " + strSize);
AwsClientBuilder.EndpointConfiguration endpointConfiguration = new AwsClientBuilder.EndpointConfiguration(endPoint, null);//note: Region has to be null
//AWSCredentialsProvider
BasicAWSCredentials sessionCredentials = new BasicAWSCredentials(token, "NOT_USED");//secretKey should be set to NOT_USED
AmazonS3 s3 = AmazonS3ClientBuilder
.standard()
.withCredentials(new AWSStaticCredentialsProvider(sessionCredentials))
.withEndpointConfiguration(endpointConfiguration)
.enablePathStyleAccess()
.build();
int maxUploadThreads = 5;
TransferManager tm = TransferManagerBuilder
.standard()
.withS3Client(s3)
.withMultipartUploadThreshold((long) (5 * 1024 * 1024))
.withExecutorFactory(() -> Executors.newFixedThreadPool(maxUploadThreads))
.build();
PutObjectRequest request = new PutObjectRequest(existingBucketName, file.getName(), file);
//request.putCustomRequestHeader("Access-Token", token);
ProgressListener progressListener = progressEvent -> System.out.println("Transferred bytes: " + progressEvent.getBytesTransferred());
request.setGeneralProgressListener(progressListener);
Upload upload = tm.upload(request);
LocalDateTime uploadStartedAt = LocalDateTime.now();
log.info("Starting upload at: " + uploadStartedAt);
try {
upload.waitForCompletion();
//upload.waitForUploadResult();
log.info("Upload completed. " + strSize);
} catch (Exception e) {//AmazonClientException
log.error("Error occurred while uploading file - " + strSize);
e.printStackTrace();
}
}
Solution found !
I found a way to get this working and for to be honest I quite happy about the result, I've done so many tests with big files (50gd.zip) and in every scenario worked very well
My solution is, remove the line: BasicAWSCredentials sessionCredentials = new BasicAWSCredentials(token, "NOT_USED");
AWSCredentials is a interface so we can override it with something dynamic, the the logic of when the token is expired and needs a new fresh token is held inside the getToken() method meaning you can call every time with no harm
AWSCredentials sessionCredentials = new AWSCredentials() {
#Override
public String getAWSAccessKeyId() {
try {
return getToken(); //getToken() method return a string
} catch (Exception e) {
return null;
}
}
#Override
public String getAWSSecretKey() {
return "NOT_USED";
}
};
When uploading a file (or parts of a multi-part file), the credentials that you use must last long enough for the upload to complete. You CANNOT refresh the credentials as there is no method to update AWS S3 that you are using new credentials for an already signed request.
You could break the upload into smaller files that upload quicker. Then only upload X parts. Refresh your credentials and upload Y parts. Repeat until all parts are uploaded. Then you will need to finish by combining the parts (which is a separate command). This is not a perfect solution as transfer speeds cannot be accurately controlled AND this means that you will have to write your own upload code (which is not hard).
I have a shape file and i need to read the shape file from my java code. I used below code for reading shape file.
public class App {
public static void main(String[] args) {
File file = new File("C:\\Test\\sample.shp");
Map<String, Object> map = new HashMap<>();//
try {
map.put("url", URLs.fileToUrl(file));
DataStore dataStore = DataStoreFinder.getDataStore(map);
String typeName = dataStore.getTypeNames()[0];
SimpleFeatureSource source = dataStore.getFeatureSource(typeName);
SimpleFeatureCollection collection = source.getFeatures();
try (FeatureIterator<SimpleFeature> features = collection.features()) {
while (features.hasNext()) {
SimpleFeature feature = features.next();
SimpleFeatureType schema = feature.getFeatureType();
Class<?> geomType = schema.getGeometryDescriptor().getType().getBinding();
String type = "";
if (Polygon.class.isAssignableFrom(geomType) || MultiPolygon.class.isAssignableFrom(geomType)) {
MultiPolygon geom = (MultiPolygon) feature.getDefaultGeometry();
type = "Polygon";
if (geom.getNumGeometries() > 1) {
type = "MultiPolygon";
}
} else if (LineString.class.isAssignableFrom(geomType)
|| MultiLineString.class.isAssignableFrom(geomType)) {
} else {
}
System.out.println(feature.getDefaultGeometryProperty().getValue().toString());
}
}
} catch (Exception e) {
// TODO: handle exception
}
}
}
I got the desired output. But my requirement is write an aws lambda function to read shape file. For this
1. I created a Lambda java project of s3 event. I wrote the same code inside the handleRequest. I uploaded the java lambda project as a lanbda function and added one trigger. When I am uploading a .shp file to as s3 bucket lmbda function will automatically invoked. But I am getting an error like below
java.lang.RuntimeException: java.io.FileNotFoundException: /sample.shp (No such file or directory)
I have sample.shp file inside my s3 bucket. I go through below link.
How to write an S3 object to a file?
I am getting the same error. I tried to change my code like below
S3Object object = s3.getObject(new GetObjectRequest(bucket, key));
InputStream objectData = object.getObjectContent();
map.put("url", objectData );
instead of
File file = new File("C:\\Test\\sample.shp");
map.put("url", URLs.fileToUrl(file));
:-( Now i am getting an error like below
java.lang.NullPointerException
Also I tried the below code
DataStore dataStore = DataStoreFinder.getDataStore(objectData);
instead of
DataStore dataStore = DataStoreFinder.getDataStore(map);
the error was like below
java.lang.ClassCastException:
com.amazonaws.services.s3.model.S3ObjectInputStream cannot be cast to
java.util.Map
Also I tried to add key directly to the map and also as DataStore object. Everything went wrong..:-(
Is there anyone who can help me?
It will be very helpful if someone can do it for me...
The DataStoreFinder.getDataStore method in geotools requires you to provide a map containing a key/value pair with key "url". The value associated with that "url" key needs to be a file URL like "file://host/path/my.shp".
You're trying to insert a Java input stream into the map. That won't work, because it's not a file URL.
The geotools library does not accept http/https URLs (see the geotools code here and here), so you need a file:// URL. That means you will need to download the file from S3 to the local Lambda filesystem and then provide a file:// URL pointing to that local file. To do that, here's Java code that should work:
// get the shape file from S3 to local filesystem
File localshp = new File("/tmp/download.shp");
s3.getObject(new GetObjectRequest(bucket, key), localshp);
// now store file:// URL in the map
map.put("url", localshp.getURI().getURL().toString());
If the geotools library had accepted real URLs (not just file:// URLs) then you could have avoided the download and simply created a time-limited, pre-signed URL for the S3 object and put that URL into the map.
Here's an example of how to do that:
// get current time and add one hour
java.util.Date expiration = new java.util.Date();
long msec = expiration.getTime();
msec += 1000 * 60 * 60;
expiration.setTime(msec);
// request pre-signed URL that will allow bearer to GET the object
GeneratePresignedUrlRequest gpur = new GeneratePresignedUrlRequest(bucket, key);
gpur.setMethod(HttpMethod.GET);
gpur.setExpiration(expiration);
// get URL that will expire in one hour
URL url = s3.generatePresignedUrl(gpur);
When I invoke API endpoints from REST client, I got error by concerning with Signature.
Request:
Host: https://xxx.execute-api.ap-southeast-1.amazonaws.com/latest/api/name
Authorization: AWS4-HMAC-SHA256 Credential={AWSKEY}/20160314/ap-southeast-1/execute-api/aws4_request,SignedHeaders=host;range;x-amz-date,Signature={signature}
X-Amz-Date: 20160314T102915Z
Response:
{
"message": "The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details. The Canonical String for this request should have been 'xxx' "
}
From Java code, I followed AWS reference of how to generate Signature.
String secretKey = "{mysecretkey}";
String dateStamp = "20160314";
String regionName = "ap-southeast-1";
String serviceName = "execute-api";
byte[] signature = getSignatureKey(secretKey, dateStamp, regionName, serviceName);
System.out.println("Signature : " + Hex.encodeHexString(signature));
static byte[] HmacSHA256(String data, byte[] key) throws Exception {
String algorithm="HmacSHA256";
Mac mac = Mac.getInstance(algorithm);
mac.init(new SecretKeySpec(key, algorithm));
return mac.doFinal(data.getBytes("UTF8"));
}
static byte[] getSignatureKey(String key, String dateStamp, String regionName, String serviceName) throws Exception {
byte[] kSecret = ("AWS4" + key).getBytes("UTF8");
byte[] kDate = HmacSHA256(dateStamp, kSecret);
byte[] kRegion = HmacSHA256(regionName, kDate);
byte[] kService = HmacSHA256(serviceName, kRegion);
byte[] kSigning = HmacSHA256("aws4_request", kService);
return kSigning;
}
May I know what I was wrong while generating Signature?
Reference how to generate Signature : http://docs.aws.amazon.com/general/latest/gr/signature-v4-examples.html#signature-v4-examples-java
You can use classes from aws-java-sdk-core: https://github.com/aws/aws-sdk-java/tree/master/aws-java-sdk-core
More specifically, Request, Aws4Signer and a few other ones:
//Instantiate the request
Request<Void> request = new DefaultRequest<Void>("es"); //Request to ElasticSearch
request.setHttpMethod(HttpMethodName.GET);
request.setEndpoint(URI.create("http://..."));
//Sign it...
AWS4Signer signer = new AWS4Signer();
signer.setRegionName("...");
signer.setServiceName(request.getServiceName());
signer.sign(request, new AwsCredentialsFromSystem());
//Execute it and get the response...
Response<String> rsp = new AmazonHttpClient(new ClientConfiguration())
.requestExecutionBuilder()
.executionContext(new ExecutionContext(true))
.request(request)
.errorResponseHandler(new SimpleAwsErrorHandler())
.execute(new SimpleResponseHandler<String>());
If you want a cleaner design, you can use the Decorator pattern to compose some elegant classes and hide the above mess. An example for that here: http://www.amihaiemil.com/2017/02/18/decorators-with-tunnels.html
From the code example above it looks like you are not creating a canonical request and including it in the string that gets signed as per http://docs.aws.amazon.com/general/latest/gr/sigv4-create-canonical-request.html
Instead of implementing this yourself have you looked at using a third-party library.
aws-v4-signer-java is a lightweight, zero-dependency library that makes it easy to generate AWS V4 signatures.
String contentSha256 = "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855";
HttpRequest request = new HttpRequest("GET", new URI("https://examplebucket.s3.amazonaws.com?max-keys=2&prefix=J"));
String signature = Signer.builder()
.awsCredentials(new AwsCredentials(ACCESS_KEY, SECRET_KEY))
.header("Host", "examplebucket.s3.amazonaws.com")
.header("x-amz-date", "20130524T000000Z")
.header("x-amz-content-sha256", contentSha256)
.buildS3(request, contentSha256)
.getSignature();
Disclaimer: I'm the libraries author.
This is possible using 100% java libraries without additional dependencies, just use the query parameters generated here:
import java.security.InvalidKeyException;
import java.security.NoSuchAlgorithmException;
import java.security.SignatureException;
import java.util.Formatter;
import javax.crypto.Mac;
import javax.crypto.spec.SecretKeySpec;
import java.util.Base64;
...
private static final String ACCESS_KEY = "...";
private static final String SECRET_KEY = "...";
private static final int expiresTime = 1 * 24 * 60 * 60;
private static final String HMAC_SHA1_ALGORITHM = "HmacSHA1";
public void sign(String protocol, String bucketName, String contentPath) throws Exception {
Calendar cal = Calendar.getInstance();
cal.add(Calendar.HOUR_OF_DAY, 24);
String host = bucketName + ".s3-us-west-2.amazonaws.com";
long expireTime = cal.getTimeInMillis() / 1000;
String signString = "GET\n" +
"\n" +
"\n" +
expireTime + "\n" +
"/" + bucketName + contentPath;
SecretKeySpec signingKey = new SecretKeySpec(SECRET_KEY.getBytes(), HMAC_SHA1_ALGORITHM);
Mac mac = Mac.getInstance(HMAC_SHA1_ALGORITHM);
mac.init(signingKey);
String signature = URLEncoder.encode(new String(Base64.getEncoder().encode(mac.doFinal(signString.getBytes()))));
System.out.println(signature);
String fullPayload = "?AWSAccessKeyId=" + ACCESS_KEY +
"&Expires=" + expireTime +
"&Signature=" + signature;
System.out.println(protocol + "://" + host + contentPath + fullPayload);
}
...
The signing process is lengthy and error-prone, here are some tips
Make sure your access key and secret is correct, try to use Postman to test the request at first, it's easy and fast, see Use Postman to Call a REST API
Make sure you use UTC time
The signing process uses both timestamp(YYYYMMDD'T'HHMMSS'Z') and datetime(YYYYMMDD), so double check your implementation for that
Use any online hash tool to verify your hash algorithm behaves as expected
Read the python implementation carefully, see Examples of the Complete Version 4 Signing Process (Python)
See my fully working java implementation on Github - A Java(SpringBoot) template for Java and AWS SageMaker DeepAR model endpoint invocation integration
You may investigate code samples that is shared by AWS web site. I used some of the util classes and a few java class I need. So you don't have to use all classes and other stuff. I left the link below.
AWS Java Samples in doc of Amazon
For me, in Java, the following code worked to generate a signed request to sent to web socket client via api gateway -
Request<Void> request = new DefaultRequest<Void>("execute-api"); //Request to API gateway
request.setHttpMethod(HttpMethodName.POST);
request.setEndpoint(URI.create(url));
String bodyContnt= "test data";
InputStream targetStream = new ByteArrayInputStream(bodyContnt.getBytes());
request.setContent(targetStream);
//Sign it...
AWS4Signer signer = new AWS4Signer();
signer.setRegionName("ap-south-1");
signer.setServiceName(request.getServiceName());
signer.sign(request, new Creds());
signer.setOverrideDate(new Date()); // needed as current ts is required
//Execute it and get the response...
Response<String> rsp = new AmazonHttpClient(new ClientConfiguration())
.requestExecutionBuilder()
.executionContext(new ExecutionContext(true))
.request(request)
.errorResponseHandler(new SimpleAwsErrorHandler(true))
.execute(new SimpleResponseHandler());