hey anyone worked on Amazon S3 bucket...can anyone tell me how to get the content type of a particular object uploaded into the bucket?
I tried converting the object again into an file and getting the extension but it did not work.
To get the content type of a particular object uploaded into an Amazon S3 bucket, you can use the AWS SDK for Java v2. You can use the AWS SDK for Java v2 in a Spring BOOT project.
To perform this use case, you can use the following code example. When you run this and the object is a .jpeg, the output is:
The object content type is image/jpeg
Java code:
public class GetObjectContentType {
public static void main(String[] args) {
final String usage = "\n" +
"Usage:\n" +
" <bucketName> <keyName>>\n\n" +
"Where:\n" +
" bucketName - The Amazon S3 bucket name. \n\n"+
" keyName - The key name. \n\n";
if (args.length != 2) {
System.out.println(usage);
System.exit(1);
}
String bucketName = args[0];
String keyName = args[1];
ProfileCredentialsProvider credentialsProvider = ProfileCredentialsProvider.create();
Region region = Region.US_EAST_1;
S3Client s3 = S3Client.builder()
.region(region)
.credentialsProvider(credentialsProvider)
.build();
getContentType(s3,bucketName,keyName);
s3.close();
}
// snippet-start:[s3.java2.getobjectcontenttype.main]
public static void getContentType (S3Client s3, String bucketName, String keyName) {
try {
HeadObjectRequest objectRequest = HeadObjectRequest.builder()
.key(keyName)
.bucket(bucketName)
.build();
HeadObjectResponse objectHead = s3.headObject(objectRequest);
String type = objectHead.contentType();
System.out.println("The object content type is "+type);
} catch (S3Exception e) {
System.err.println(e.awsErrorDetails().errorMessage());
System.exit(1);
}
}
// snippet-end:[s3.java2.getobjectcontenttype.main]
}
This code example can be found in the AWS Code Library here:
Determine the existence and content type of an object in an Amazon S3 bucket using an AWS SDK
I am using the following code to create a signed resumable url (link here). It works fine except for one thing: the ACLs are not being set on the object after I upload it with the created signed url.
public static void generateV4PutObjectSignedUrl(String projectId, String bucketName, String objectName) throws StorageException {
// String projectId = "my-project-id";
// String bucketName = "my-bucket";
// String objectName = "my-object";
Storage storage = StorageOptions.newBuilder().setProjectId(projectId).build().getService();
// Define Resource
BlobInfo blobInfo = BlobInfo
.newBuilder(BlobId.of(bucketName, objectName))
.setAcl(new ArrayList<>(Arrays.asList(Acl.of(Acl.User.ofAllUsers(), Acl.Role.READER),
Acl.of(new Acl.Project(Acl.Project.ProjectRole.OWNERS, "project-id"), Acl.Role.OWNER)))).build();
// Generate Signed URL
Map<String, String> extensionHeaders = new HashMap<>();
extensionHeaders.put("Content-Type", "application/octet-stream");
URL url =
storage.signUrl(
blobInfo,
15,
TimeUnit.MINUTES,
Storage.SignUrlOption.httpMethod(HttpMethod.PUT),
Storage.SignUrlOption.withExtHeaders(extensionHeaders),
Storage.SignUrlOption.withV4Signature());
System.out.println("Generated PUT signed URL:");
System.out.println(url);
System.out.println("You can use this URL with any user agent, for example:");
System.out.println(
"curl -X PUT -H 'Content-Type: application/octet-stream' --upload-file my-file '"
+ url
+ "'");
}
}
How can I set the ACLs I want of a signed resumable url using the above code?
Note: in non-signed url situations, the above way of setting the ACL works.
The following is the source code generated url from cloud storage that I made
public String generateImageUrl(String fileName, Integer duration, String folderName) throws IOException {
Credentials credentials = GoogleCredentials.fromStream(new ClassPathResource(STORAGE_FILE_NAME).getInputStream());
Storage storage = StorageOptions.newBuilder().setCredentials(credentials).setProjectId(PROJECT_ID).build().getService();
Bucket bucket = storage.get(BUCKET_NAME, Storage.BucketGetOption.fields(Storage.BucketField.values()));
logger.info("Bucket name : " + bucket.getName());
String fullImagePath = folderName + "/" + fileName;
BlobId imgId = BlobId.of(BUCKET_NAME, fullImagePath);
if(null!=imgId) {
Blob blob = storage.get(imgId);
if(null!=blob && blob.exists()) {
URL signedUrl = storage.signUrl(blob, duration, TimeUnit.MINUTES);
String imageUrl = signedUrl.toExternalForm();
logger.info("Generated image url : " + imageUrl);
return imageUrl;
}
}
return null;
}
But the generated url has an access duration
how to implement it so that no access duration is given ??
It's not possible to create a signed url with no access duration.
Cloud Storage Signed Urls have a maximum expiration delay of 7 days.
You specify an expiration time when you create the signed URL. Anyone
who knows the URL can access the resource until the expiration time
for the URL is reached or the key used to sign the URL is rotated.
Excerpt from Java client library docs
Note that V4 Signed URLs can't have an expiration longer than 7 days.
You can also check more details here.
When I invoke API endpoints from REST client, I got error by concerning with Signature.
Request:
Host: https://xxx.execute-api.ap-southeast-1.amazonaws.com/latest/api/name
Authorization: AWS4-HMAC-SHA256 Credential={AWSKEY}/20160314/ap-southeast-1/execute-api/aws4_request,SignedHeaders=host;range;x-amz-date,Signature={signature}
X-Amz-Date: 20160314T102915Z
Response:
{
"message": "The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details. The Canonical String for this request should have been 'xxx' "
}
From Java code, I followed AWS reference of how to generate Signature.
String secretKey = "{mysecretkey}";
String dateStamp = "20160314";
String regionName = "ap-southeast-1";
String serviceName = "execute-api";
byte[] signature = getSignatureKey(secretKey, dateStamp, regionName, serviceName);
System.out.println("Signature : " + Hex.encodeHexString(signature));
static byte[] HmacSHA256(String data, byte[] key) throws Exception {
String algorithm="HmacSHA256";
Mac mac = Mac.getInstance(algorithm);
mac.init(new SecretKeySpec(key, algorithm));
return mac.doFinal(data.getBytes("UTF8"));
}
static byte[] getSignatureKey(String key, String dateStamp, String regionName, String serviceName) throws Exception {
byte[] kSecret = ("AWS4" + key).getBytes("UTF8");
byte[] kDate = HmacSHA256(dateStamp, kSecret);
byte[] kRegion = HmacSHA256(regionName, kDate);
byte[] kService = HmacSHA256(serviceName, kRegion);
byte[] kSigning = HmacSHA256("aws4_request", kService);
return kSigning;
}
May I know what I was wrong while generating Signature?
Reference how to generate Signature : http://docs.aws.amazon.com/general/latest/gr/signature-v4-examples.html#signature-v4-examples-java
You can use classes from aws-java-sdk-core: https://github.com/aws/aws-sdk-java/tree/master/aws-java-sdk-core
More specifically, Request, Aws4Signer and a few other ones:
//Instantiate the request
Request<Void> request = new DefaultRequest<Void>("es"); //Request to ElasticSearch
request.setHttpMethod(HttpMethodName.GET);
request.setEndpoint(URI.create("http://..."));
//Sign it...
AWS4Signer signer = new AWS4Signer();
signer.setRegionName("...");
signer.setServiceName(request.getServiceName());
signer.sign(request, new AwsCredentialsFromSystem());
//Execute it and get the response...
Response<String> rsp = new AmazonHttpClient(new ClientConfiguration())
.requestExecutionBuilder()
.executionContext(new ExecutionContext(true))
.request(request)
.errorResponseHandler(new SimpleAwsErrorHandler())
.execute(new SimpleResponseHandler<String>());
If you want a cleaner design, you can use the Decorator pattern to compose some elegant classes and hide the above mess. An example for that here: http://www.amihaiemil.com/2017/02/18/decorators-with-tunnels.html
From the code example above it looks like you are not creating a canonical request and including it in the string that gets signed as per http://docs.aws.amazon.com/general/latest/gr/sigv4-create-canonical-request.html
Instead of implementing this yourself have you looked at using a third-party library.
aws-v4-signer-java is a lightweight, zero-dependency library that makes it easy to generate AWS V4 signatures.
String contentSha256 = "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855";
HttpRequest request = new HttpRequest("GET", new URI("https://examplebucket.s3.amazonaws.com?max-keys=2&prefix=J"));
String signature = Signer.builder()
.awsCredentials(new AwsCredentials(ACCESS_KEY, SECRET_KEY))
.header("Host", "examplebucket.s3.amazonaws.com")
.header("x-amz-date", "20130524T000000Z")
.header("x-amz-content-sha256", contentSha256)
.buildS3(request, contentSha256)
.getSignature();
Disclaimer: I'm the libraries author.
This is possible using 100% java libraries without additional dependencies, just use the query parameters generated here:
import java.security.InvalidKeyException;
import java.security.NoSuchAlgorithmException;
import java.security.SignatureException;
import java.util.Formatter;
import javax.crypto.Mac;
import javax.crypto.spec.SecretKeySpec;
import java.util.Base64;
...
private static final String ACCESS_KEY = "...";
private static final String SECRET_KEY = "...";
private static final int expiresTime = 1 * 24 * 60 * 60;
private static final String HMAC_SHA1_ALGORITHM = "HmacSHA1";
public void sign(String protocol, String bucketName, String contentPath) throws Exception {
Calendar cal = Calendar.getInstance();
cal.add(Calendar.HOUR_OF_DAY, 24);
String host = bucketName + ".s3-us-west-2.amazonaws.com";
long expireTime = cal.getTimeInMillis() / 1000;
String signString = "GET\n" +
"\n" +
"\n" +
expireTime + "\n" +
"/" + bucketName + contentPath;
SecretKeySpec signingKey = new SecretKeySpec(SECRET_KEY.getBytes(), HMAC_SHA1_ALGORITHM);
Mac mac = Mac.getInstance(HMAC_SHA1_ALGORITHM);
mac.init(signingKey);
String signature = URLEncoder.encode(new String(Base64.getEncoder().encode(mac.doFinal(signString.getBytes()))));
System.out.println(signature);
String fullPayload = "?AWSAccessKeyId=" + ACCESS_KEY +
"&Expires=" + expireTime +
"&Signature=" + signature;
System.out.println(protocol + "://" + host + contentPath + fullPayload);
}
...
The signing process is lengthy and error-prone, here are some tips
Make sure your access key and secret is correct, try to use Postman to test the request at first, it's easy and fast, see Use Postman to Call a REST API
Make sure you use UTC time
The signing process uses both timestamp(YYYYMMDD'T'HHMMSS'Z') and datetime(YYYYMMDD), so double check your implementation for that
Use any online hash tool to verify your hash algorithm behaves as expected
Read the python implementation carefully, see Examples of the Complete Version 4 Signing Process (Python)
See my fully working java implementation on Github - A Java(SpringBoot) template for Java and AWS SageMaker DeepAR model endpoint invocation integration
You may investigate code samples that is shared by AWS web site. I used some of the util classes and a few java class I need. So you don't have to use all classes and other stuff. I left the link below.
AWS Java Samples in doc of Amazon
For me, in Java, the following code worked to generate a signed request to sent to web socket client via api gateway -
Request<Void> request = new DefaultRequest<Void>("execute-api"); //Request to API gateway
request.setHttpMethod(HttpMethodName.POST);
request.setEndpoint(URI.create(url));
String bodyContnt= "test data";
InputStream targetStream = new ByteArrayInputStream(bodyContnt.getBytes());
request.setContent(targetStream);
//Sign it...
AWS4Signer signer = new AWS4Signer();
signer.setRegionName("ap-south-1");
signer.setServiceName(request.getServiceName());
signer.sign(request, new Creds());
signer.setOverrideDate(new Date()); // needed as current ts is required
//Execute it and get the response...
Response<String> rsp = new AmazonHttpClient(new ClientConfiguration())
.requestExecutionBuilder()
.executionContext(new ExecutionContext(true))
.request(request)
.errorResponseHandler(new SimpleAwsErrorHandler(true))
.execute(new SimpleResponseHandler());
I have some URLs' in string form, and from these URLs' I want to generate a URI using java.net.URI.
These URLs' are actually hyperlinks in an Android Webview:
clc://C# or clc://C++
final URI u = new URI(newURL);
final String sScheme = u.getScheme();
final String sHost = u.getHost();
final String sPath = u.getPath();
But in the above code, if a URL has # or + then getHost() returns null.
I tried to encode the URL as follows, but it doesn't work:
String encodedUrl = URLEncoder.encode(url, "UTF-8");
I also tried putting %23 for #, then too it doesn`t work.
Please help me to resolve this.....
URLEncoder doesn't always provide the correct output, especially when URIs' are involved.
Try the following approach instead:
Uri u = Uri.parse(newURL)
.buildUpon()
.appendQueryParameter("param", param)
.build();
String url = u.toString();
where param is a web service parameter (if using any). This will encode the URL in UTF-8 format correctly. Then,
final String sScheme = u.getScheme( );
final String sHost = u.getHost( );
final String sPath = u.getPath( );
It will work as expected.