Uploading a file to Amazon S3 using presigned URL which has Signature, Expiration and AccessKey, using the following code I'm able to upload the file using normal java code but same code in Android gives me 403 error. Presigned URL is generate using Amazon SDK
I have read http://developer.android.com/reference/java/net/HttpURLConnection.html
and http://android-developers.blogspot.com/2011/09/androids-http-clients.html but not able to figure out what header param should i use, I guess in android it is setting headers in request which server rejects
HttpURLConnection connection=(HttpURLConnection) url.openConnection();
connection.setDoOutput(true);
connection.setRequestMethod("PUT");
OutputStreamWriter out = new OutputStreamWriter(connection.getOutputStream());
out.write("This text uploaded as object.");
out.close();
int responseCode = connection.getResponseCode();
Exception: 403; Signature don't match :-o
Has anyone come across this issue?
Or more details into which Header-parameters are added behind the scenes from android library?
please set your content type like this:
connection.setRequestProperty("Content-Type"," ");
because the default for HttpsUrlConnection Automatically generates content type as:
"Content-Type: application/x-www-form-urlencoded"
This will cause signature mismatch.
So after some trial/error/search found out the issue:
While creating a presigned URL is it important to specify Content-Type (based on your data) or else you will keep getting 403 Signature don't match, the contentType that you specify here should be mentioned in HttpURLConnection connection object
string s3url = s3Client.GetPreSignedURL(new GetPreSignedUrlRequest()
.WithBucketName(bucketName)
.WithKey(keyName)
.WithContentType("application/octet-stream") // IMPORTANT
.WithVerb(HttpVerb.PUT)
.WithExpires(<YOUR EXPIRATION TIME>);
Inside your connection..add this to the code in question after ("PUT")
connection.setFixedLengthStreamingMode(< YOUR DATA.LENGTH ?>);
connection.setRequestProperty("Content-Type","application/octet-stream");
Check this bug, latest SDK should let you set Content-Type
The way to generate the URL:
private static URL generateRUL(String objectKey, String ACCESS_KEY, String SECRET_KEY, String BUCKET_NAME) {
AmazonS3 s3Client = new AmazonS3Client(new BasicAWSCredentials(ACCESS_KEY, SECRET_KEY));
URL url = null;
try {
GeneratePresignedUrlRequest request = new GeneratePresignedUrlRequest(BUCKET_NAME, objectKey);
request.setMethod(com.amazonaws.HttpMethod.PUT);
request.setExpiration(new Date( System.currentTimeMillis() + (60 * 60 * 1000)));
// Very important ! It won't work without adding this!
// And request.addRequestParameter("Content-Type", "application/octet-stream") won't work neither
request.setContentType("application/octet-stream");
url = s3Client.generatePresignedUrl(request );
} catch (AmazonServiceException exception) {
} catch (AmazonClientException ace) { }
return url;
}
The way to upload the file:
public int upload(byte[] fileBytes, URL url) {
HttpURLConnection connection = (HttpURLConnection) url.openConnection();
connection.setDoOutput(true);
connection.setRequestMethod("PUT");
connection.setRequestProperty("Content-Type", "application/octet-stream"); // Very important ! It won't work without adding this!
OutputStream output = connection.getOutputStream();
InputStream input = new ByteArrayInputStream(fileBytes);
byte[] buffer = new byte[4096];
int length;
while ((length = input.read(buffer)) > 0) {
output.write(buffer, 0, length);
}
output.flush();
return connection.getResponseCode();
}
Are you using the AWS SDK for Android? That has the correct APIs and I suspect it will be easier to upload to S3 using that, and more secure for that matter. I believe they have tutorials, code samples, and demos there as well.
There is also an API for S3 and other classes you may need. A PutObjectRequest is what will help you upload the file from the phone client and can be useful in your case.
I had this same problem. Here's the reason why I faced it:
A pre-signed URL gives you access to the object identified in the URL,
provided that the creator of the pre-signed URL has permissions to
access that object. That is, if you receive a pre-signed URL to upload
an object, you can upload the object only if the creator of the
pre-signed URL has the necessary permissions to upload that object.
http://docs.aws.amazon.com/AmazonS3/latest/dev/PresignedUrlUploadObject.html
Once I gave my lambda function PutObject permissions on that s3 bucket then it worked!
Related
Am making webservice calls to HTTPS server from an android application. Below is the code snippet, with which am able to make web service calls successfully and getting response.
My Question is, do we need to perform any additional step to encrypt data before making call to HTTPS server?
Because, from android profiler am able to see all my Web Requests in plain text format. My understanding is that request will gets encrypted before making HTTPS call.
public static WebServiceResp makeWebServiceCall(String XML, String urlPath) throws IOException{
//Code to make a web service HTTP request
String responseString = "";
String outputString = "";
String wsURL = urlPath;
URL url = new URL(wsURL);
URLConnection connection = url.openConnection();
HttpsURLConnection httpConn = (HttpsURLConnection)connection;
ByteArrayOutputStream bout = new ByteArrayOutputStream();
//System.out.println(XML);
byte[] buffer = new byte[XML.length()];
buffer = XML.getBytes();
bout.write(buffer);
byte[] b = bout.toByteArray();
// Set the appropriate HTTP parameters.
httpConn.setRequestProperty("Content-Length",
String.valueOf(b.length));
httpConn.setRequestProperty("Content-Type", "text/xml; charset=utf-8");
httpConn.setRequestMethod("POST");
httpConn.setRequestProperty("Cache-Control", "no-cache");
httpConn.setDoOutput(true);
httpConn.setDoInput(true);
OutputStream out = httpConn.getOutputStream();
//Write the content of the request to the outputstream of the HTTP Connection.
out.write(b);
out.close();
//Ready with sending the request.
//Check the status
int status = httpConn.getResponseCode();
Log.d(TAG, "makeWebServiceCall: "+"Processing Status: "+status);
BufferedReader in;
if (status <= 200) {
//Read the response.
Log.d(TAG, "makeWebServiceCall: Getting Input Stream");
InputStreamReader isr =
new InputStreamReader(httpConn.getInputStream());
in = new BufferedReader(isr);
}else{
//Read the response.
Log.d(TAG, "makeWebServiceCall: Getting Error Stream");
InputStreamReader isr =
new InputStreamReader(httpConn.getErrorStream());
in = new BufferedReader(isr);
}
//Write the SOAP message response to a String.
while ((responseString = in.readLine()) != null) {
outputString = outputString + responseString;
}
Log.d(TAG, "makeWebServiceCall: WebServiceResponse " + outputString);
//Parse the String output to a org.w3c.dom.Document and be able to reach every node with the org.w3c.dom API.
Document document = Utils.parseXmlFile(outputString);
//NodeList nodeLst = document.getElementsByTagName("GetWeatherResult");
// String weatherResult = nodeLst.item(0).getTextContent();
//System.out.println("Weather: " + weatherResult);
//Write the SOAP message formatted to the console.
WebServiceResp webServiceResp = new WebServiceResp();
webServiceResp.setDocument(document);
webServiceResp.setStatus(status);
return webServiceResp;
}
No. If you're sending it to an https website, the encryption is done as part of the protocol. You don't need to do any additional work.
No. The encryption that you see is on the network layer. The client which initiates the https call see what was sent and what was received. That is how https works.
When you look at chrome browser's network tab, you see what was sent and what was received. Now this is not a security problem, https is more about you doing things which make its difficult for anyone between the network to eavesdrop your data.
Now if you still want an additional level of security you can use certificate pinning
https://www.owasp.org/index.php/Certificate_and_Public_Key_Pinning
https://medium.com/#appmattus/android-security-ssl-pinning-1db8acb6621e
How can you add to network_security_config from MainActivity
So in this technique you basically say that the certificate hash that you expected is to have this content. And then if someone uses a trusted proxy with trusted CAs on the system, even after generating a valid certificate for the given domain the connections will not be established.
HTTPS is transparent to your application, all of the magic happens between Transport Layer(so it calls 'Transport Layer Security'), you may imagine encrypted telegrams in the old days, generals tell telegrapher messages in plain text, and telegrapher send them in encrypted form(maybe use some kind of codebook), anyone who didn't have the same codebook can't decrypt the message easily, and anyone who uses telegrams didn't care about the codebook(or even known about it, except those telegraphers on both side of the 'Transport Layer').
The encryption/decryption is done by built-in network client module provided by OS. So you needn't worry about it.
You can view plain texts with some client tools as they know exactly what they are sending/receiving. E.g. chrome developer tool. (Actually they don't care about encryption/decryption either).
I have a service deployed in Heroku that produces a pdf file output. When I hit the URL in the browser, I am able to download the pdf file (I am not prompting to save (as per my requirement), it auto save to defined path in the code). So service is up and available. But when I am accessing it using HttpURLConnection I am getting 404 error. . Could anyone help me out on this?
Following is the link I am accessing:
http://quiet-savannah-7144.herokuapp.com/services/time/temp
Here is the service code, deployed in Heroku server:
#jawax.ws.rs.core.Context
ServletContext context;
#GET
#path("/temp")
#Produces("application/pdf")
public Response getPdf() throws IOException{
InputStream is = context.getResourceAsStream("/static/temp.pdf");
ResponseBuilder res = Response.ok(is);
res.header("Content-Disposition","attachment; filename=temp.pdf");
return res.build();
}
Note: I have my file in the location webapp/static/temp.pdf
Client code is as follows:
try {
URL url = new URL("http://quiet-savannah-7144.herokuapp.com/sevices/time/temp");
HttpURLConnection conn = (HttpURLConnection) url.openConnection();
conn.setDoOutput(true);
conn.setRequestMethod("GET");
conn.setRequestProperty("Content-Type", "application/json");
int code = conn.getResponseCode();
System.out.println("code>>"+code+"<<");
if (conn.getResponseCode() == 200) {
System.out.println("*************************done****************************");
InputStream inputStream = conn.getInputStream();
OutputStream output = new FileOutputStream("D:/copyOfTest.pdf");
byte[] buffer = new byte[1024];
int bytesRead;
while ((bytesRead = inputStream.read(buffer)) != -1) {
output.write(buffer, 0, bytesRead);
}
output.close();
} else {
Scanner scanner = new Scanner(conn.getErrorStream());
while(scanner.hasNext())
System.out.println(scanner.next());
scanner.close();
}
conn.disconnect();
} catch (MalformedURLException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
I tried content type with pdf and x-pdf as shown below, nothing is working
conn.setRequestProperty("Content-Type", "application/pdf");
conn.setRequestProperty("Content-Type", "application/x-pdf");
When I deploy the service locally in Tomcat server in my machine, then things are absolutely fine. I am struggling from the past 6 hours to resolve this, but no clue. Actually I have to fetch it in the android AsyncTask. If I am able to do it in java, then I could achieve the same in android. Could someone help me out on this.
Thanks in advance
I see a few problems.
First, if you do a GET you should not write conn.setDoOutput(true); cause you're not outputting from your application to the server.
Second, the Content-Type header is the content-type of what YOU send to the server, not the opposite, so since you're not sending anything but just doing a get, you should not set it.
Instead, maybe, if you want, you can set the Accept header.
Content-Type is a server header. You should send an Accept header, maybe you could try something generic like Accept: *.
Though this thread is old, may be useful for some:
Came up with similar problem and solved by adding the "Accept-Encoding: gzip, deflate, br" header
I have a separate services that is managing files and s3 authentication. It produces presigned URLs, which I am able to use in other services to upload (and download) files.
I would like to take advantage of the Multipart upload sdk- currently the 'uploadToUrl' method seems to spend most of its time on getResponseCode, so it's difficult to provide user feedback. Also, the multipart upload seems much faster in my testing.
Ideally, I'd like to be able to create some AWSCredentials using a presigned URL instead of a secret key / access key for temporary use. Is that just a pipe dream?
//s3 service
public URL getUrl(String bucketName, String objectKey, Date expiration, AmazonS3 s3Client, HttpMethod method, String contentType) {
GeneratePresignedUrlRequest generatePresignedUrlRequest;
generatePresignedUrlRequest = new GeneratePresignedUrlRequest(bucketName, objectKey);
generatePresignedUrlRequest.setMethod(method);
generatePresignedUrlRequest.setExpiration(expiration);
generatePresignedUrlRequest.setContentType(contentType);
URL s = s3Client.generatePresignedUrl(generatePresignedUrlRequest);
System.out.println(String.format("Generated Presigned URL: %n %S", s.toString()));
return s;
}
//Upload service
Override
public void uploadToUrl(URL url, File file) {
HttpURLConnection connection;
try {
InputStream inputStream = new FileInputStream(file);
connection = (HttpURLConnection) url.openConnection();
connection.setDoOutput(true);
connection.setRequestMethod("PUT");
OutputStream out =
connection.getOutputStream();
byte[] buf = new byte[1024];
int count;
int total = 0;
long fileSize = file.length();
while ((count =inputStream.read(buf)) != -1)
{
if (Thread.interrupted())
{
throw new InterruptedException();
}
out.write(buf, 0, count);
total += count;
int pctComplete = new Double(new Double(total) / new Double(fileSize) * 100).intValue();
System.out.print("\r");
System.out.print(String.format("PCT Complete: %d", pctComplete));
}
System.out.println();
out.close();
inputStream.close();
System.out.println("Finishing...");
int responseCode = connection.getResponseCode();
if (responseCode == 200) {
System.out.printf("Successfully uploaded.");
}
} catch (IOException e) {
e.printStackTrace();
} catch (InterruptedException e) {
e.printStackTrace();
}
}
A few years later, but digging around in the AWS Java SDK reveals that adding the following to GeneratePresignedUrlRequest works pretty well:
AmazonS3Client amazonS3Client = /* ... */;
GeneratePresignedUrlRequest request = /* ... */;
// the following are required to trigger the multipart upload API
request.addRequestParameter("uploadId", uploadIdentifier);
request.addRequestParameter("partNumber", Integer.toString(partNumber));
// the following may be optional but are recommended to validate data integrity during upload
request.putCustomRequestHeader(Headers.CONTENT_MD5, md5Hash);
request.putCustomRequestHeader(Headers.CONTENT_LENGTH, Long.toString(contentLength));
URL presignedURL = amazonS3Client.generatePresignedUrl(request);
(I haven't dug deeply enough to determine whether CONTENT_MD5 or CONTENT_LENGTH are required.)
with PHP, you can
$command = $this->s3client->getCommand ('CreateMultipartUpload', array (
'Bucket' => $this->rootBucket,
'Key' => $objectName
));
$signedUrl = $command->createPresignedUrl ('+5 minutes');
But I found no way so far how to acheive this with Java.
For a single PUT (or GET) operation, one can use generatePresignedUrl, but I wouldn't know how to apply this to multipart upload like with the PHP getCommand ('CreateMultipartUpload'/'UploadPart'/'CompleteMultipartUpload') methods.
For now I am exploring returning temporary credentials returned by my trusted code instead of a signed url.
http://docs.aws.amazon.com/AmazonS3/latest/dev/AuthUsingTempSessionTokenJava.html
When I call this in java :
URL url = new URL("http://www.google.com/finance/getprices?q=MSFT");
URLConnection goog = url.openConnection();
BufferedReader in = new BufferedReader(new InputStreamReader(goog.getInputStream()));
I get this as exception :
java.io.IOException: Server returned HTTP response code: 503 for URL: http://www.google.com/sorry/?continue=http://www.google.com/finance/getprices%3Fq%3MSFTO%26
I don't have converted URL in my function because its generated automatically when my URL is called, My original URL is string after "continue=", how can I get it back from this URL?
EDIT :
Because I am calling this page again and again it generates this URL http://www.google.com/sorry/?continue=http://www.google.com/finance/getprices%3Fq%3MSFTO%26 and it says :
Our systems have detected unusual traffic from your computer network. This page checks to see if it's really you sending the requests, and not a robot.
If I copy paste URL after continue= it gives me actual content of page.
503 is a response code which tells service is down or unavailabe
wiki link
According to wikipedia
503 Service Unavailable
The server is currently unavailable (because it is overloaded or down for maintenance).[2] Generally, this is a temporary state.
I have tried your code and this is woriking fine (giving no exception)
URL url = new URL("http://www.google.com/finance/getprices?q=MSFT");
URLConnection goog = url.openConnection();
BufferedReader in = new BufferedReader(new InputStreamReader(goog.getInputStream()));
String temp;
while((temp = in.readLine())!= null){
System.out.println(temp);
}
}catch(Exception ex){
ex.printStackTrace();
}
and its return me
EXCHANGE%3DNASDAQ
MARKET_OPEN_MINUTE=570
MARKET_CLOSE_MINUTE=960
INTERVAL=86400
COLUMNS=DATE,CLOSE,HIGH,LOW,OPEN,VOLUME
DATA=
TIMEZONE_OFFSET=-240
a1370289600,35.59,35.63,34.83,34.92,51256272
1,34.99,35.74,34.771,35.62,65538438
2,34.78,34.89,34.43,34.6,46032657
3,34.96,35.11,34.49,34.84,37627133
4,35.67,35.78,35.06,35.25,40762249
7,35.47,35.65,35.14,35.51,35995223
8,34.84,35.18,34.68,35.05,39350316
9,35,35.27,34.85,35.14,37373032
10,34.715,35.02,34.59,34.99,45654803
11,34.4,34.6901,34.25,34.55,53116371
14,35,35.16,34.63,34.69,49672492
15,34.98,35.17,34.895,34.97,28622929
16,34.59,35.09,34.59,34.96,30820208
17,33.49,34.33,33.37,34.26,54496758
18,33.265,33.73,33.05,33.66,85338395
21,33.715,34.2,32.57,32.94,56113708
22,33.67,34.38,33.46,34.08,44073348
23,34.35,34.48,33.8875,34.12,48667834
24,34.62,34.78,34.5,34.52,28993542
25,34.545,34.79,34.34,34.38,65548196
28,34.36,34.99,34.33,34.75,31064000
So probably problem is that service is not available temporarily in your area.
Any Java API in client side can check its modified date?
You can use HttpURLConnection to check the Last-Modified value on a page, assuming the server returns one.
This request uses the HTTP HEAD method to return only the headers for the resource:
URL url = new URL(
"http://en.wikipedia.org/wiki/Main_Page");
HttpURLConnection httpConnection = (HttpURLConnection) url
.openConnection();
httpConnection.setRequestMethod("HEAD");
httpConnection.connect();
long lastModified = httpConnection.getLastModified();
if (lastModified != 0) {
System.out.println(new Date(lastModified));
} else {
System.out.println("Last-Modified not returned");
}
httpConnection.disconnect();
// TODO: error handling
HttpURLConnection is adequate for some things, but if you want a more rounded API, have a look at Apache HttpComponents.
You can use the lastModified method in java.io.File to find out the last time a file was modified.