I want to get a list of objects and folders that are in a Bucket in Amazon S3 but I can't, I should not use Amazon S3 SDK.
It's important not to use SDK and only with Rest and Java should I sent a Request and then receive a Response.
I have a method like this :
public String BucketSubList(String strPath) throws Exception {
String answer = null;
// S3 timestamp pattern.
String fmt = "EEE, dd MMM yyyy HH:mm:ss ";
SimpleDateFormat df = new SimpleDateFormat(fmt, Locale.US);
df.setTimeZone(TimeZone.getTimeZone("GMT"));
// Data needed for signature
String method = "GET";
String contentMD5 = "";
String contentType = "";
String date = df.format(new Date()) + "GMT";
String bucket = "/" + strPath + "/";
// Generate signature
StringBuffer buf = new StringBuffer();
buf.append(method).append("\n");
buf.append(contentMD5).append("\n");
buf.append(contentType).append("\n");
buf.append(date).append("\n");
buf.append(bucket);
// try {
String signature = sign(buf.toString());
// Connection to s3.amazonaws.com
URL url = new URL("http", "s3.amazonaws.com", 80, bucket);
HttpURLConnection httpConn = null;
httpConn = (HttpURLConnection) url.openConnection();
httpConn.setDoInput(true);
httpConn.setDoOutput(true);
httpConn.setUseCaches(false);
httpConn.setDefaultUseCaches(false);
httpConn.setAllowUserInteraction(true);
httpConn.setRequestMethod(method);
httpConn.setRequestProperty("Date", date);
// httpConn.setRequestProperty("Content-Type", "text/plain");
String AWSAuth = "AWS " + keyId + ":" + signature;
httpConn.setRequestProperty("Authorization", AWSAuth);
// Send the HTTP PUT request.
int statusCode = httpConn.getResponseCode();
System.out.println(statusCode);
if ((statusCode / 100) != 2) {
// Deal with S3 error stream.
InputStream in = httpConn.getErrorStream();
String errorStr = getS3ErrorCode(in);
System.out.println("Error: " + errorStr);
} else {
answer = "";
// System.out.println("Bucket listed successfully");
InputStream inst = httpConn.getInputStream();
BufferedReader in = new BufferedReader(new InputStreamReader(inst));
String decodedString;
while ((decodedString = in.readLine()) != null) {
answer += decodedString;
System.out.println(answer);
}
in.close();
}
return answer;
}
Without knowing what your problem is, i just can give you the link to the AWS S3 Rest API.
This method does what you want.
I hope this might help you.
Otherwise please give us some more information about your problem.
Related
Getting the below error while making a call to Create Container.
Response Code : 411 Response Message : Length Required
String stringToSign = "PUT\n\n\n\n0\n\n\n\n\n\n\n\nx-ms-date:" + date + "\nx-ms-version:" + "2014-02-14\n" + "/" + storageAccount + "/"+ "container-create-test"+"\nrestype:container"+"\ntimeout:60";
Java code snippet.
HttpURLConnection connection = (HttpURLConnection)new URL(url).openConnection();
connection.setRequestMethod(vMethod);
connection.addRequestProperty("Authorization", authHeader);
connection.addRequestProperty("x-ms-date", date);
connection.addRequestProperty("x-ms-version", "2014-02-14");
connection.addRequestProperty("Content-Length", "0");
Nothing wrong with the format of StringToSign.
411 Response Message : Length Required
This error means you don't add Content-Length:0 header in your http request.
Update
As you work with HttpURLConnection in Java, Content-Length header can't be set manually by default, see this thread.
In case of other trouble, here's the complete sample for you to refer.
public static void putContainer() throws Exception {
// Account info
String accountName = "accountName";
String accountKey = "accountKey";
// Request Uri and Method
String containerName = "containerName";
String requestUri = "https://"+accountName+".blob.core.windows.net/"+containerName+"?restype=container&timeout=60";
HttpURLConnection connection = (HttpURLConnection) (new URL(requestUri)).openConnection();
connection.setRequestMethod("PUT");
// Request Headers
// 1. x-ms-version, recommend to use the latest version if possible
String serviceVersion = "2018-03-28";
// 2. x-ms-date
SimpleDateFormat fmt = new SimpleDateFormat("EEE, dd MMM yyyy HH:mm:ss");
fmt.setTimeZone(TimeZone.getTimeZone("GMT"));
String date = fmt.format(Calendar.getInstance().getTime()) + " GMT";
// 3. Authorization
String authKeyFormat = "SharedKey";
String caHeader = "x-ms-date:"+date+"\nx-ms-version:"+serviceVersion+"\n";
String caResource = "/"+accountName+"/"+containerName+"\nrestype:container\ntimeout:60";
String signStr = "PUT\n\n\n\n\n\n\n\n\n\n\n\n"+caHeader+caResource;
String authorization = getAuthorization(accountName, authKeyFormat, signStr, accountKey);
// Send request
connection.setRequestProperty("x-ms-version", serviceVersion);
connection.setRequestProperty("x-ms-date",date);
connection.setRequestProperty("Authorization", authorization);
// Send 0 byte, code sets Content-Length:0 automatically
connection.setDoOutput(true);
connection.setFixedLengthStreamingMode(0);
System.out.println("Response message : " + connection.getResponseMessage());
System.out.println("Response code : " + connection.getResponseCode());
}
private static String getAuthorization(String accountName, String authKeyFormat, String signStr, String accountKey) throws NoSuchAlgorithmException, UnsupportedEncodingException, InvalidKeyException {
SecretKeySpec secretKey = new SecretKeySpec(Base64.getDecoder().decode(accountKey), "HmacSHA256");
Mac sha256HMAC = Mac.getInstance("HmacSHA256");
sha256HMAC.init(secretKey);
String signature = Base64.getEncoder().encodeToString(sha256HMAC.doFinal(signStr.getBytes("UTF-8")));
return authKeyFormat+" "+accountName+":"+signature;
}
I need to write a small tool in JAVA which will translate text from English to French using the Google translate API. Everything works but I have an apostrophe decoding problem.
Original text:
Inherit Tax Rate
Text translated with Google translate API:
Taux d' imposition hérité
How it should be:
Taux d'imposition hérité
This is my translate method(sorry for the long method):
private String translate(String text, String from, String to) {
StringBuilder result = new StringBuilder();
try {
String encodedText = URLEncoder.encode(text, "UTF-8");
String urlStr = "https://www.googleapis.com/language/translate/v2?key=" + sKey + "&q=" + encodedText + "&target=" + to + "&source=" + from;
URL url = new URL(urlStr);
HttpsURLConnection conn = (HttpsURLConnection) url.openConnection();
InputStream googleStream;
if (conn.getResponseCode() == 200) {
googleStream = conn.getInputStream(); //success
} else
googleStream = conn.getErrorStream();
BufferedReader reader = new BufferedReader(new InputStreamReader(googleStream));
String line;
while ((line = reader.readLine()) != null) {
result.append(line);
}
JsonParser parser = new JsonParser();
JsonElement element = parser.parse(result.toString());
if (element.isJsonObject()) {
JsonObject obj = element.getAsJsonObject();
if (obj.get("error") == null) {
String translatedText = obj.get("data").getAsJsonObject().
get("translations").getAsJsonArray().
get(0).getAsJsonObject().
get("translatedText").getAsString();
return translatedText;
}
}
if (conn.getResponseCode() != 200) {
System.err.println(result);
}
} catch (IOException | JsonSyntaxException ex) {
System.err.println(ex.getMessage());
}
return null;
}
I'm using an XML writer to write the text and first I though that this has a problem, but I observed that the text is returned like this in the stream so I introduced the encoding parameter when I initialise the InputStreamReader:
BufferedReader reader = new BufferedReader(new InputStreamReader(googleStream, "UTF-8"));
But I receive the string with the same problem. Any ideas about what I can do?
I think this problem is solved by using the format parameter (docs). It defaults to html, but you can change it to text to receive unencoded data. Your request should look like this:
String urlStr = "https://www.googleapis.com/language/translate/v2?key=" + sKey + "&q=" + encodedText + "&target=" + to + "&source=" + from + "&format=text";
I am trying to use azure storage rest api to push block blob type but issue is about Content Length to be known before hand in order to upload.
Do we have a workaround for cases wherein inputStream need to be relayed without having this length info available.
String accesskey = "accesskey";
String storageAccount = "storageAccount";
String containerName = "containerName";
String workgroupId = UUID.randomUUID().toString();
String objectId = "1." + UUID.randomUUID().toString();
String blobName = getAzureAccessKey(containerName, workgroupId, objectId);
String version = "2018-03-28";
String putData = "testData";
SimpleDateFormat fmt = new SimpleDateFormat("EEE, dd MMM yyyy HH:mm:ss");
fmt.setTimeZone(TimeZone.getTimeZone("GMT"));
String currentDate = fmt.format(Calendar.getInstance().getTime()) + " GMT";
String urlResource = "/"+ Paths.get(storageAccount, containerName, blobName).toString();
String headerResource = "x-ms-blob-type:BlockBlob\nx-ms-date:" + currentDate + "\nx-ms-version:" + version;
String putUrl = "https://" + storageAccount + ".blob.core.windows.net/" + containerName + "/" + blobName;
System.out.println(putUrl);
String newline = "\n";
List listToSign = Lists.newArrayList();
listToSign.add("PUT");
listToSign.add("");
listToSign.add("");
listToSign.add("");
listToSign.add("");
listToSign.add("application/octet-stream");
listToSign.add("");
listToSign.add("");
listToSign.add("");
listToSign.add("");
listToSign.add("");
listToSign.add("");
listToSign.add(headerResource);
listToSign.add(urlResource);
String stringToSign = String.join(newline, listToSign);
Base64 base64 = new Base64();
System.out.println(stringToSign);
Mac mac = Mac.getInstance("HmacSHA256");
mac.init(new SecretKeySpec(base64.decode(accesskey), "HmacSHA256"));
String authKey = new String(base64.encode(mac.doFinal(stringToSign.getBytes("UTF-8"))));
String authHeader = "SharedKey " + storageAccount + ":"+ authKey;
System.out.println(authHeader);
InputStreamEntity entity = new InputStreamEntity(
new ByteArrayInputStream(putData.getBytes(StandardCharsets.UTF_8)), -1,
ContentType.APPLICATION_OCTET_STREAM);
// set chunked transfer encoding ie. no Content-length
entity.setChunked(true);
HttpPut httpPut = null;
try {
DefaultHttpClient httpClient = new DefaultHttpClient();
httpClient.removeRequestInterceptorByClass(org.apache.http.protocol.RequestContent.class);
httpPut = new HttpPut(putUrl);
httpPut.setHeader("Host", storageAccount + ".blob.core.windows.net");
httpPut.setHeader("Transfer-Encoding","chunked");
//httpPut.setHeader("Content-Length","0");
httpPut.setHeader("Content-Type","application/octet-stream");
httpPut.addHeader("x-ms-blob-type", "BlockBlob");
httpPut.addHeader("x-ms-date", currentDate);
httpPut.addHeader("x-ms-version", version);
httpPut.addHeader("Authorization",authHeader);
httpPut.setEntity(entity);
System.out.println("Request Headers");
for (Header header : httpPut.getAllHeaders()) {
System.out.println(header.getName() + ":" + header.getValue());
}
HttpResponse response = httpClient.execute(httpPut);
System.out.println(response.getStatusLine());
for (Header header: response.getAllHeaders()) {
System.out.println(header.getName()+":"+ header.getValue());
}
// Read the contents of an entity and return it as a String.
String content = EntityUtils.toString(response.getEntity());
System.out.println(content);
} finally {
if(httpPut != null ){
httpPut.releaseConnection();
}
}
Response from server
HTTP Error 400. There is an invalid content length or chunk length in the request.
Above code works if i set content length as part of signature and header info.
We can use ByteArrayEntity to get the content length instead of using InputStreamEntity
Here is a simple demo for your reference:
FileInputStream fileInputStream=null;
ByteArrayOutputStream bos = null ;
try {
fileInputStream=new FileInputStream("D:/Test/Test.txt");
bos = new ByteArrayOutputStream();
byte[] bytes=new byte[102400];
int x=0;
while ((x=fileInputStream.read(bytes))!= -1){
bos.write(bytes,0,x);
}
byte[] data = bos.toByteArray();
org.apache.http.entity.ByteArrayEntity byteArrayEntity=new ByteArrayEntity(data);
int contentLength=data.length;
} catch (Exception e) {
e.printStackTrace();
}
As Zhaoxing Lu said above, we can use Java SDK to achieve this, here is the demo that use Java SDK for your reference:
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.net.URISyntaxException;
import java.security.InvalidKeyException;
import com.microsoft.azure.storage.CloudStorageAccount;
import com.microsoft.azure.storage.blob.CloudBlobClient;
import com.microsoft.azure.storage.blob.CloudBlobContainer;
import com.microsoft.azure.storage.blob.CloudBlockBlob;
import com.microsoft.azure.storage.blob.ListBlobItem;
public class Main {
public static final String ConnString="DefaultEndpointsProtocol=https;AccountName=xxxxxxxxb;AccountKey=O7xxxx8e86XQSy2vkvSi/x/e9l9FhLqxxxxjkly1DsQPYY5dF2JrAVxxxxo29ZrrGJA==;EndpointSuffix=core.windows.net";
public static void main(String[] args) {
// TODO Auto-generated method stub
uploadBlob("mycontainer","TechTalk.pptx","E:\\Test\\TechTalk.pptx");
System.out.println("Success");
}
public static void uploadBlob(String containerName, String blobName,String filePath) {
CloudStorageAccount account = null;
CloudBlobContainer container = null;
try {
account = CloudStorageAccount.parse(ConnString);
CloudBlobClient client = account.createCloudBlobClient();
container = client.getContainerReference(containerName);
container.createIfNotExists();
CloudBlockBlob cloudBlockBlob = container.getBlockBlobReference(blobName);
FileInputStream fileinputStream=new FileInputStream(filePath);
cloudBlockBlob.upload(fileinputStream, fileinputStream.available());
}catch(Exception ex) {
ex.printStackTrace();
}
}
}
We can get Java SDK at: Java SDK
How do you add parameters or extra line code to rest calls in java.
GET /ObjectName HTTP/1.1
Host: BucketName.s3.amazonaws.com
Date: date Authorization: authorization string (see Authenticating Requests (AWS Signature Version
4)) Range:bytes=byte_range
Putting /Objectname after GET on the set method causes an error.
The code i am using is;
public void getObject() throws Exception {
String fmt = "EEE, dd MMM yyyy HH:mm:ss ";
SimpleDateFormat df = new SimpleDateFormat(fmt, Locale.US);
df.setTimeZone(TimeZone.getTimeZone("GMT"));
String ob2 = "/bucket/test.txt";
String bucket = "/bucket";
String method = "GET";
String contentMD5 = "";
String contentType = "";
String date = df.format(new Date()) + "GMT";
// Generate signature
StringBuffer buf = new StringBuffer();
buf.append(method).append("\n");`enter code here`
buf.append(contentMD5).append("\n");
buf.append(contentType).append("\n");
buf.append(date).append("\n");
buf.append(ob2);
String signature = sign(buf.toString());
HttpURLConnection httpConn = null;
URL url = new URL("https”,”s3demo.s3demosite.com",443,bucket);
httpConn = (HttpURLConnection) url.openConnection();
httpConn.setDoInput(true);
httpConn.setDoOutput(true);
httpConn.setUseCaches(false);
httpConn.setDefaultUseCaches(false);
httpConn.setAllowUserInteraction(true);
httpConn.setRequestMethod(method);
httpConn.setRequestProperty("Date", date);
httpConn.setRequestProperty("Content-Length", "0");
String AWSAuth = "AWS " + keyId + ":" + signature;
httpConn.setRequestProperty("Authorization", AWSAuth);
// Send the HTTP PUT request.
int statusCode = httpConn.getResponseCode();
InputStream err = httpConn.getErrorStream();
InputStream is = null;
is = httpConn.getInputStream();
int ch;
StringBuffer sb = new StringBuffer();
while ((ch = err.read()) != -1) {
sb.append((char) ch);
}
if ((statusCode/100) != 2)
{
// Deal with S3 error stream.
InputStream in = httpConn.getErrorStream();
System.out.println("Error: "+errorStr);
}
else {
System.out.println("download worked”);
}
}
In REST Services you can pass a parameter in two ways.
as path variables
As a query argument. Example: GET /students/tom or GET /students?name=tom
I'm developing a service to upload videos to YouTube and I got it working on V2, but I'm migrating to V3 but I can't find REST endpoints to hit.
The Resumable Upload is well docummented here but the Direct Upload is only documented as a Python script reference.
My process can't support Resumable Uploads and I can't use the Java Client Library because I don't have the files stored locally. For V2 I used a different approach that allowed me to do piping.
Is there a way to do this on V3???
String lineEnd = "\r\n";
String twoHyphens = "--";
String access_token = "A1S2D3F4G5H6J7K8L9";
String gameClipID = "gameClipID";
String sourceServerURL = "sourceServerURL";
String boundary = "Boundary" + gameClipID;
String closingBoundary = lineEnd + twoHyphens + boundary + twoHyphens;
URL youTubeURL = new URL("http://uploads.gdata.youtube.com/feeds/api/users/default/uploads");
HttpURLConnection youTubeConnection = (HttpURLConnection)youTubeURL.openConnection();
youTubeConnection.setRequestMethod("POST");
youTubeConnection.setDoOutput(true);
youTubeConnection.setDoInput(true);
// Set Headers
youTubeConnection.setRequestProperty("Authorization" , "GoogleLogin auth=" + auth);
youTubeConnection.setRequestProperty("GData-Version", "2");
youTubeConnection.setRequestProperty("X-GData-Key", "key=" + developerKey);
youTubeConnection.setRequestProperty("Slug", gameClipID);
youTubeConnection.setRequestProperty("Accept", "*/*");
youTubeConnection.setRequestProperty("Accept-Charset", "UTF-8");
youTubeConnection.setRequestProperty("Content-Type", "multipart/related;boundary=" + boundary);
String content = prepareContent(boundary, "Title", "Description", keywords);
Long totalLength = (long)content.length() + (long)fileSize + (long)closingBoundary.length();
youTubeConnection.setRequestProperty("Content-Length", totalLength.toString());
youTubeConnection.setRequestProperty("Connection", "close");
URL sourceURL = new URL(sourceServerURL);
HttpURLConnection sourceConnection = (HttpURLConnection) sourceURL.openConnection();
sourceConnection.setAllowUserInteraction(false);
sourceConnection.setRequestProperty("Authorization", access_token);
sourceConnection.connect();
BufferedInputStream bis = new BufferedInputStream(sourceConnection.getInputStream());
BufferedOutputStream request = new BufferedOutputStream(youTubeConnection.getOutputStream());
request.write(content.getBytes("UTF-8"));
boolean eof = false;
byte[] input = new byte[4096];
while (!eof) {
int length = bis.read(input);
if (length == -1) {
eof = true;
} else {
request.write(input, 0, length);
}
}
request.write(closingBoundary.getBytes("UTF-8"));
request.flush();
request.close();
bis.close();
sourceConnection.disconnect();
InputStream responseStream = new BufferedInputStream(youTubeConnection.getInputStream());
BufferedReader responseStreamReader = new BufferedReader(new InputStreamReader(responseStream));
String line;
StringBuilder stringBuilder = new StringBuilder();
while ((line = responseStreamReader.readLine()) != null) {
stringBuilder.append(line);
}
responseStreamReader.close();
String response = stringBuilder.toString();
responseStream.close();
youTubeConnection.disconnect();
return YouTubeClient.parseXMLResponse(response);
Direct upload equivalent of the documentation, would be just setting uploadType=direct.
You can set directUploadEnabled to true in this example and see the Post request to make sure.
https://github.com/youtube/api-samples/blob/master/java/src/main/java/com/google/api/services/samples/youtube/cmdline/data/UploadVideo.java#L136
Also here's the source code of the directUpload so you can build this manually.
https://code.google.com/p/google-api-java-client/source/browse/google-api-client/src/main/java/com/google/api/client/googleapis/media/MediaHttpUploader.java#345