I want to check for a version changed/get metadata of a text-file with a shared link on dropbox. I will not be using dropbox api as it makes users use their own accounts. I want them to link to my account and I cannot do that manually since I might change my password later.
so: no auth token, just get metadata from shared link of dropbox so that I can check for version changes and if the version has changed download the contents of the new file.
also: I'm open to other suggestions to make this work as well. Please explain in a little detail your solution.
Updated E-Tag Issue:
public void getFromOnlineTxtDatabase(){
try{
URL url = new URL("url-here");
HttpURLConnection.setFollowRedirects(true);
HttpURLConnection con = (HttpURLConnection) url.openConnection();
con.setDoOutput(false);
con.setReadTimeout(20000);
con.setRequestProperty("Connection", "keep-alive");
//get etag for update check
String etag = con.getHeaderField("etag");
//String etag= "";
con.setRequestProperty("User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:16.0) Gecko/20100101 Firefox/16.0");
((HttpURLConnection) con).setRequestMethod("GET");
//System.out.println(con.getContentLength()) ;
con.setConnectTimeout(5000);
BufferedInputStream in = new BufferedInputStream(con.getInputStream());
int responseCode = con.getResponseCode();
if (responseCode == HttpURLConnection.HTTP_OK) {
System.out.println(responseCode);
}
StringBuffer buffer = new StringBuffer();
int chars_read;
//int total = 0;
while ((chars_read = in.read()) != -1)
{
char g = (char) chars_read;
buffer.append(g);
}
final String page = buffer.toString();
//create password_ems.txt to internal
if (fileExistance("data.txt")){
File dir = getFilesDir();
File file = new File(dir, "data.txt");
boolean deleted = file.delete();
stringToTxt(page, "data.txt");
}else{
stringToTxt(page, "data.txt");
}
if (fileExistance("data_etag.txt")){
File dir = getFilesDir();
File file = new File(dir, "etag.txt");
boolean deleted = file.delete();
stringToTxt(etag, "etag.txt");
}else{
//create etag_file
stringToTxt(etag, "data_etag.txt");
}
// Log.i("Page", page);
}catch(Exception e){
showDialog("Database Fetch Failure","Unable to Fetch Password Database, check your internet" +
" connection and try again later.",0);
Log.i("Page", "Error");
}
}
If you do an HTTP HEAD request against a public or shared Dropbox URL, you'll get, among other things, an etag header. I don't know that this behavior is guaranteed, since I don't think it's documented anywhere, but at least for now the etag header can be used to determine when a file has changed. (If the etag is different, the file has changed.)
EDIT
In general when using ETags, the most efficient thing to do is issue a GET request with a header of If-None-Match: <old etag>. If the content hasn't changed, this will respond with a 304, but if the content has changed, this will download the new content as per a normal GET request (and the response will be 200).
Related
I have been trying to upload large files by generating a signed URL. Here is the documentation I was following to generate the signed URL: https://cloud.google.com/storage/docs/access-control/signing-urls-with-helpers#code-samples
It was working fine for files in 100's of MB's but as soon as the file size went up to 1 GB, the curl command started timing out even after increasing the expiration time. I tried looking at the answer here: https://stackoverflow.com/a/63789297/7466551, but I am still unable to get the URL working to upload the URL.
I am using this command to upload the file:
curl -X POST -H 'x-goog-resumable: start' --upload-file file-name 'pre_signed_google_url'. I am adding the 'x-goog-resumable: start' header as I am having "x-goog-resumable", "start" header as a part of my code to generate the URL.
Can someone please let me know if I need to do any additional thing to generate the URL to upload large files?
Answering my own question as I had to use two separate sources to reach a solution.
On top of the java code here: https://stackoverflow.com/a/63789297/7466551, I referred to the medium article here: https://medium.com/google-cloud/google-cloud-storage-signedurl-resumable-upload-with-curl-74f99e41f0a2
So you need the following additional lines of code on top of the StackOverflow answer to get a signed URL for resumable uploads:
// Open a HTTP connection and add header
URL obj = new URL(url.toString());
HttpURLConnection connection = (HttpURLConnection) obj.openConnection();
connection.setRequestMethod("POST");
connection.setRequestProperty("x-goog-resumable", "start");
connection.setDoOutput(true);
// Connect to the URL and post headers
DataOutputStream writer = new DataOutputStream(
connection.getOutputStream());
writer.writeBytes("");
writer.flush();
writer.close();
// Checking the responseCode to
if (connection.getResponseCode() == connection.HTTP_CREATED) {
connection.disconnect();
System.out.println("Location: " + connection.getHeaderField("Location"));
}
else {
// Throw error
BufferedReader reader = new BufferedReader(new InputStreamReader(connection.getInputStream()));
String inputLine;
StringBuffer response = new StringBuffer();
while ((inputLine = reader.readLine()) != null) {
response.append(inputLine);
}
reader.close();
String errorMessage = response.toString();
connection.disconnect();
throw new IOException(errorMessage);
}
I am trying to upload (POST) a file to an endpoint using java.net.HttpURLConnection but I keep getting http code 400 (bad request).
I refered to Send File And Parameters To Server With HttpURLConnection in android API 23
but problem is that I need to send this file as request body param (file=).
Note: The files will be of small size only (4-5mb) so I am reading it entirely in memory.
Corresponding curl request is:
curl -X POST "API" -H "Content-Type: multipart/form-data" -F "file="
Excerpts of Code that I am using:
Proxy webproxy = new Proxy(Proxy.Type.HTTP, new InetSocketAddress("
<proxy host>", <proxy_port>));
HttpURLConnection http_conn = (HttpURLConnection)
url.openConnection(webproxy);
String authorization = getAuthorization(access_token);
http_conn.setRequestMethod("POST");
http_conn.setRequestProperty("Accept-Charset", "UTF-8");
http_conn.setRequestProperty("Authorization", authorization);
http_conn.setRequestProperty("Connection", "Keep-Alive");
http_conn.setRequestProperty("Content-Type", "multipart/form-data);
http_conn.setDoOutput(true);
http_conn.setDoInput(true);
DataOutputStream outputStream;
outputStream = new DataOutputStream(http_conn.getOutputStream());
File file_obj = new File(this.file);
byte[] allBytes = new byte[(int) file_obj.length()];
FileInputStream fileInputStream = new FileInputStream(file_obj);
outputStream.write("file=".getBytes("UTF-8")); <---Trying to add file param here
fileInputStream.read(allBytes);
outputStream.write(allBytes);
Post that I just read response using below piece of code (works fine for different GET requests):
InputStream inputStream = http_conn.getInputStream();
BufferedReader bufferedReader = new BufferedReader(new
InputStreamReader(inputStream));
String line = "";
while ((line = bufferedReader.readLine()) != null) {
data = data + line;
}
Note: I use java rarely an am not very familiar with it so please be descriptive in your response.
When looking at your curl command line, it shows that the file needs to be send as a multipart/form-data request. This is actually a complex way of formatting your data when it is requires.
An example of the format you need to send is:
Headers:
Content-Type: multipart/form-data; boundary=AaB03x
Body:
--AaB03x
Content-Disposition: form-data; name="files"; filename="file1.txt"
Content-Type: text/plain
... contents of file1.txt ...
--AaB03x--
At the moment, your code is sending the file as a POST/GET formatted request, and this doesn't work as the backend isn't expecting that.
To solve this problem, we need to format the source files into the format required by the backend, and once you know that the "boundary" header option is just a randomly generated value, it becomes more easy to send the request.
String boundary = "MY_AWESOME_BOUNDARY"
http_conn.setRequestProperty("Content-Type", "multipart/form-data; boundary=" + boundary);
try(DataOutputStream outputStream = new DataOutputStream(http_conn.getOutputStream())) {
File file_obj = new File(this.file);
// Write form-data header
outputStream.write(("--" + boundary + "\r\n").getBytes("UTF-8"));
outputStream.write(("Content-Disposition: form-data; name=\"file\"; filename=\"file1.txt\"\r\n").getBytes("UTF-8"));
outputStream.write(("Content-Type: text/plain\r\n").getBytes("UTF-8"));
outputStream.write(("\r\n").getBytes("UTF-8"));
// Write form-data body
Files.copy(file_obj.toPath(), outputStream)
// Write form-data "end"
outputStream.write(("--" + boundary + "--\r\n").getBytes("UTF-8"));
}
// Read backend response here
try(InputStream inputStream = http_conn.getInputStream()) {
BufferedReader bufferedReader = new BufferedReader(new
InputStreamReader(inputStream));
StringBuilder lines = new StringBuilder(); // StringBuilder is faster for concatination than appending strings
while ((line = bufferedReader.readLine()) != null) {
lines.append(line);
}
System.out.println(lines);
}
Note that I used "try-with-resource" blocks, these blocks make sure that any external resources are closed and disposed when you are done using them, generally the open resource limit of the OS is very low, compared to the amount of memory your program has, so what happens is that your program could give weird errors that only happens after some time of running or when the user executes certain actions inside your application
The above didnt worked for me so I switched to different package (okhttp3), here is what worked for me:
File file_obj = new File(this.file);
String authorization = "my authorization string";
Proxy webproxy = new Proxy(Proxy.Type.HTTP, new
InetSocketAddress("proxy", <port>));
OkHttpClient client = new OkHttpClient.Builder().proxy(webproxy).build();
RequestBody requestBody = new MultipartBody.Builder().setType(MultipartBody.FORM).addFormDataPart("file", "filename",
RequestBody.create(MediaType.parse("application/octet-stream"), file_obj)).build();
Request request = new Request.Builder().header("Authorization", authorization).url(this.url).post(requestBody).build();
try (Response response = client.newCall(request).execute()){
if(!response.isSuccessful()) return "NA";
return (response.body().string());
}
Uploading a file to Amazon S3 using presigned URL which has Signature, Expiration and AccessKey, using the following code I'm able to upload the file using normal java code but same code in Android gives me 403 error. Presigned URL is generate using Amazon SDK
I have read http://developer.android.com/reference/java/net/HttpURLConnection.html
and http://android-developers.blogspot.com/2011/09/androids-http-clients.html but not able to figure out what header param should i use, I guess in android it is setting headers in request which server rejects
HttpURLConnection connection=(HttpURLConnection) url.openConnection();
connection.setDoOutput(true);
connection.setRequestMethod("PUT");
OutputStreamWriter out = new OutputStreamWriter(connection.getOutputStream());
out.write("This text uploaded as object.");
out.close();
int responseCode = connection.getResponseCode();
Exception: 403; Signature don't match :-o
Has anyone come across this issue?
Or more details into which Header-parameters are added behind the scenes from android library?
please set your content type like this:
connection.setRequestProperty("Content-Type"," ");
because the default for HttpsUrlConnection Automatically generates content type as:
"Content-Type: application/x-www-form-urlencoded"
This will cause signature mismatch.
So after some trial/error/search found out the issue:
While creating a presigned URL is it important to specify Content-Type (based on your data) or else you will keep getting 403 Signature don't match, the contentType that you specify here should be mentioned in HttpURLConnection connection object
string s3url = s3Client.GetPreSignedURL(new GetPreSignedUrlRequest()
.WithBucketName(bucketName)
.WithKey(keyName)
.WithContentType("application/octet-stream") // IMPORTANT
.WithVerb(HttpVerb.PUT)
.WithExpires(<YOUR EXPIRATION TIME>);
Inside your connection..add this to the code in question after ("PUT")
connection.setFixedLengthStreamingMode(< YOUR DATA.LENGTH ?>);
connection.setRequestProperty("Content-Type","application/octet-stream");
Check this bug, latest SDK should let you set Content-Type
The way to generate the URL:
private static URL generateRUL(String objectKey, String ACCESS_KEY, String SECRET_KEY, String BUCKET_NAME) {
AmazonS3 s3Client = new AmazonS3Client(new BasicAWSCredentials(ACCESS_KEY, SECRET_KEY));
URL url = null;
try {
GeneratePresignedUrlRequest request = new GeneratePresignedUrlRequest(BUCKET_NAME, objectKey);
request.setMethod(com.amazonaws.HttpMethod.PUT);
request.setExpiration(new Date( System.currentTimeMillis() + (60 * 60 * 1000)));
// Very important ! It won't work without adding this!
// And request.addRequestParameter("Content-Type", "application/octet-stream") won't work neither
request.setContentType("application/octet-stream");
url = s3Client.generatePresignedUrl(request );
} catch (AmazonServiceException exception) {
} catch (AmazonClientException ace) { }
return url;
}
The way to upload the file:
public int upload(byte[] fileBytes, URL url) {
HttpURLConnection connection = (HttpURLConnection) url.openConnection();
connection.setDoOutput(true);
connection.setRequestMethod("PUT");
connection.setRequestProperty("Content-Type", "application/octet-stream"); // Very important ! It won't work without adding this!
OutputStream output = connection.getOutputStream();
InputStream input = new ByteArrayInputStream(fileBytes);
byte[] buffer = new byte[4096];
int length;
while ((length = input.read(buffer)) > 0) {
output.write(buffer, 0, length);
}
output.flush();
return connection.getResponseCode();
}
Are you using the AWS SDK for Android? That has the correct APIs and I suspect it will be easier to upload to S3 using that, and more secure for that matter. I believe they have tutorials, code samples, and demos there as well.
There is also an API for S3 and other classes you may need. A PutObjectRequest is what will help you upload the file from the phone client and can be useful in your case.
I had this same problem. Here's the reason why I faced it:
A pre-signed URL gives you access to the object identified in the URL,
provided that the creator of the pre-signed URL has permissions to
access that object. That is, if you receive a pre-signed URL to upload
an object, you can upload the object only if the creator of the
pre-signed URL has the necessary permissions to upload that object.
http://docs.aws.amazon.com/AmazonS3/latest/dev/PresignedUrlUploadObject.html
Once I gave my lambda function PutObject permissions on that s3 bucket then it worked!
I need to download a CSV file from Google insights programatically. Since it requires authentication, I used the clientLogin to get the session id.
How do I download the file by passing the session id as a cookie?
I tried using a new URLConnection object and set the cookie in setRequestParameter method hoping it would authenticate my login then, however it doesn't seem to be working. I have a feeling I shouldn't use two separate connections, is that true?
If so then how do I pass session id as parameter when i download the file? I also tried using the same connection this didn't work either. Please help.
try {
URL url1 = new URL("https://www.google.com/accounts/ClientLogin?accountType=GOOGLE&Email=*******.com&Passwd=*****&service=trendspro&source=test-test-v1");
URL url2 = new URL("http://www.google.com/insights/search/overviewReport?cat=0-7&geo=BR&cmpt=geo&content=1&export=1");
URLConnection conn = url1.openConnection();
// fake request coming from browser
conn.setRequestProperty("User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.56 Safari/535.11");
BufferedReader in = new BufferedReader(new InputStreamReader(conn.getInputStream(), "UTF-8"));
String f = in.readLine();
// obtaining the sid.
String sid=f.substring(4);
System.out.println(sid);
URLConnection conn2 = url2.openConnection();
conn2.setRequestProperty("Cookie", sid);
BufferedInputStream i= new BufferedInputStream(conn2.getInputStream());
FileOutputStream fos = new FileOutputStream("f:/testplans.csv");
BufferedOutputStream bout = new BufferedOutputStream(fos,1024);
byte data[] = new byte[1024];
while(i.read(data,0,1024)>=0) {
bout.write(data);
}
bout.close();
in.close();
}
Try the following: link. Check the top answer: they don't use the SID, but the Auth.
If it's working for Google Reader, it will probably work for Google Insights as well.
My httpURLConnection fails with 404 - Resource not Found error. The same URL works on my firefox poster and returns 200/ok. Both the applications(webapp and the URL-app) are located on the same machine. I found from another thread that I need to add below 2 lines to make it work when the same URL works from browser-
urlConnection.setRequestProperty("User-Agent","Mozilla/5.0 ( compatible ) ");
urlConnection.setRequestProperty("Accept","/");
Unfortunately adding above lines didn't fix the problem. Am I doing anything wrong in the code below ? Any useful suggestions will be much appreciated.
{
String computerName = InetAddress.getLocalHost().getHostName();
url = new URL("http://"+computerName+":8080"+"/cerprovider/devices");
urlConnection = (HttpURLConnection)url.openConnection();
urlConnection.setRequestMethod("POST");
urlConnection.setDefaultUseCaches(false);
urlConnection.setUseCaches(false);
urlConnection.setDoOutput(true);
urlConnection.setDoInput(true);
urlConnection.setRequestProperty("Connection", "Keep-Alive");
urlConnection.setRequestProperty("Content-Length", Integer.toString(Integer.MAX_VALUE));
urlConnection.setRequestProperty("User-Agent","Mozilla/5.0 ( compatible ) ");
urlConnection.setRequestProperty("Accept","*/*");
urlConnection.setConnectTimeout(500);
// Create I/O streams
outStream = new DataOutputStream(urlConnection.getOutputStream());
//inStream = new DataInputStream(urlConnection.getInputStream());
// Send request
outStream.writeBytes(<blah>.toString());
outStream.flush();
outStream.close();
BufferedReader in = null;
String responseErrorString = "";
_tracer.info("Response code --"+urlConnection.getResponseCode());
InputStream error = urlConnection.getErrorStream();
_tracer.info("Error from the connection --"+error.toString());
StringBuilder buf = new StringBuilder();
for(int c = -1; (c = error.read()) != -1; )
buf.append((char)c);
responseErrorString = new String (buf);
_tracer.info("responseErrorString--"+responseErrorString);
}
try to send request again. In my project I am using Apache HttpClient to send request & many times the 404 error resolve by sending request again. (but not all the time)
Also use httpfox plugin on firefox to check what exactly happening when u send request.
404 error is supposed to b for 'resource temporarily not available'.