I'm working on the application where user will upload ZIP file to my server, on the server that ZIP file will be expanded and then I need to upload it to the server. Now my questions is: how to upload directory with multiple files and sub-folders using Java to S3 bucket? Is there any examples for that? Currently i'm using JetS3t to manage all my communications with S3.
HI This is the simple way to upload the Directory into S3 bucket.
BasicAWSCredentials awsCreds = new BasicAWSCredentials(access_key_id,
secret_access_key);
AmazonS3 s3Client = new AmazonS3Client(awsCreds);
TransferManager tm = TransferManagerBuilder.standard().withS3Client(s3Client).build();
MultipleFileUpload upload = tm.uploadDirectory(existingBucketName,
"BuildNumber#1", "FilePathYouWant", true);
I built something very similar. After expanding the zip on the server call FileUtils.listFiles() which will recursively list files in a folder. Just iterate the list and create s3objects and upload the files to s3. Make use of the threadedstorage service so that multiple files can be uploaded at the same time. Also ensure you process the upload events. If some files couldn't be uploaded the jets3t library will tell you.
I could post the code I wrote once get into the office.
EDIT: CODE:
Here's the code:
private static ProviderCredentials credentials;
private static S3Service s3service;
private static ThreadedS3Service storageService;
private static S3Bucket bucket;
private List<S3Object> s3Objs=new ArrayList<S3Object>();
private Set<String> s3ObjsCompleted=new HashSet<String>();
private boolean isErrorOccured=true;
private final ByteFormatter byteFormatter = new ByteFormatter();
private final TimeFormatter timeFormatter = new TimeFormatter();
private void initialise() throws ServiceException, S3ServiceException {
credentials=<create your credentials>;
s3service = new RestS3Service(credentials);
bucket = new S3Bucket(<bucket details>);
storageService=new ThreadedS3Service(s3service, this);
}
}
private void uploadFolder(File folder) throws NoSuchAlgorithmException, IOException {
readFolderContents(folder);
uploadFilesInList(folder);
}
private void readFolderContents(File folder) throws NoSuchAlgorithmException, IOException {
Iterator<File> filesinFolder=FileUtils.iterateFiles(folder,null,null);
while(filesinFolder.hasNext()) {
File file=filesinFolder.next();
String key = <create your key from the filename or something>;
S3Object s3Obj=new S3Object(bucket, file);
s3Obj.setKey(key);
s3Obj.setContentType(Mimetypes.getInstance().getMimetype(s3Obj.getKey()));
s3Objs.add(s3Obj);
}
}
private void uploadFilesInList(File folder) {
logger.debug("Uploading files in folder "+folder.getAbsolutePath());
isErrorOccured=false;
s3ObjsCompleted.clear();
storageService.putObjects(bucket.getName(), s3Objs.toArray(new S3Object[s3Objs.size()]));
if(isErrorOccured || s3Objs.size()!=s3ObjsCompleted.size()) {
logger.debug("Have to try uploading a few objects again for folder "+folder.getAbsolutePath()+" - Completed = "+s3ObjsCompleted.size()+" and Total ="+s3Objs.size());
List<S3Object> s3ObjsRemaining=new ArrayList<S3Object>();
for(S3Object s3Obj : s3Objs) {
if(!s3ObjsCompleted.contains(s3Obj.getKey())) {
s3ObjsRemaining.add(s3Obj);
}
}
s3Objs=s3ObjsRemaining;
uploadFilesInList(folder);
}
}
#Override
public void event(CreateObjectsEvent event) {
super.event(event);
if (ServiceEvent.EVENT_IGNORED_ERRORS == event.getEventCode()) {
Throwable[] throwables = event.getIgnoredErrors();
for (int i = 0; i < throwables.length; i++) {
logger.error("Ignoring error: " + throwables[i].getMessage());
}
}else if(ServiceEvent.EVENT_STARTED == event.getEventCode()) {
logger.debug("**********************************Upload Event Started***********************************");
}else if(event.getEventCode()==ServiceEvent.EVENT_ERROR) {
isErrorOccured=true;
}else if(event.getEventCode()==ServiceEvent.EVENT_IN_PROGRESS) {
StorageObject[] storeObjs=event.getCreatedObjects();
for(StorageObject storeObj : storeObjs) {
s3ObjsCompleted.add(storeObj.getKey());
}
ThreadWatcher watcher = event.getThreadWatcher();
if (watcher.getBytesTransferred() >= watcher.getBytesTotal()) {
logger.debug("Upload Completed.. Verifying");
}else {
int percentage = (int) (((double) watcher.getBytesTransferred() / watcher.getBytesTotal()) * 100);
long bytesPerSecond = watcher.getBytesPerSecond();
StringBuilder transferDetailsText=new StringBuilder("Uploading.... ");
transferDetailsText.append("Speed: " + byteFormatter.formatByteSize(bytesPerSecond) + "/s");
if (watcher.isTimeRemainingAvailable()) {
long secondsRemaining = watcher.getTimeRemaining();
if (transferDetailsText.length() > 0) {
transferDetailsText.append(" - ");
}
transferDetailsText.append("Time remaining: " + timeFormatter.formatTime(secondsRemaining));
}
logger.debug(transferDetailsText.toString()+" "+percentage);
}
}else if(ServiceEvent.EVENT_COMPLETED==event.getEventCode()) {
logger.debug("**********************************Upload Event Completed***********************************");
if(isErrorOccured) {
logger.debug("**********************But with errors, have to retry failed uploads**************************");
}
}
}
Here is how I did it in December of 2021 since BasicAWSCredentials is deprecated now.
AWSCredentials = new BasicAWSCredentials(env.getProperty("AWS_ACCESS_KEY_ID"),
env.getProperty("AWS_SECRET_ACCESS_KEY"));
AmazonS3 s3Client = AmazonS3ClientBuilder.standard()
.withRegion(Regions.US_EAST_1).withCredentials(new AWSStaticCredentialsProvider(AWSCredentials))
.build();
TransferManager tm = TransferManagerBuilder.standard().withS3Client(s3Client).build();
MultipleFileUpload upload = tm.uploadDirectory(existingBucketName,
"BuildNumber#1", "FilePathYouWant", true);
Related
I am testing android managment API using localhost as call back url. I followed each and every step following this url Android Management API Sample.
Now i m stuck on place.. according to this guide, i download the json file from service account. Now i copy that json file and save in app folder of my project.
This is my enterprise.json file
Screenshot of json file in android studio
and i just give folder location as enterprise.json in location string
This is my code
private static final String PROJECT_ID = "enterprise-271814";
private static final String SERVICE_ACCOUNT_CREDENTIAL_FILE =
"enterprise.json";
private static final String POLICY_ID = "samplePolicy";
/** The package name of the COSU app. */
private static final String COSU_APP_PACKAGE_NAME =
"com.ariaware.devicepoliceycontroller";
/** The OAuth scope for the Android Management API. */
private static final String OAUTH_SCOPE =
"https://www.googleapis.com/auth/androidmanagement";
private static final String APP_NAME = "Device Policey Controller";
private final AndroidManagement androidManagementClient;
public Sample(AndroidManagement androidManagementClient) {
this.androidManagementClient = androidManagementClient;
}
public void run() throws IOException {
// Create an enterprise. If you've already created an enterprise, the
// createEnterprise call can be commented out and replaced with your
// enterprise name.
String enterpriseName = createEnterprise();
System.out.println("Enterprise created with name: " + enterpriseName);
// Set the policy to be used by the device.
setPolicy(enterpriseName, POLICY_ID, getCosuPolicy());
// Create an enrollment token to enroll the device.
String token = createEnrollmentToken(enterpriseName, POLICY_ID);
System.out.println("Enrollment token (to be typed on device): " + token);
// List some of the devices for the enterprise. There will be no devices for
// a newly created enterprise, but you can run the app again with an
// existing enterprise after enrolling a device.
List<Device> devices = listDevices(enterpriseName);
for (Device device : devices) {
System.out.println("Found device with name: " + device.getName());
}
// If there are any devices, reboot one.
if (devices.isEmpty()) {
System.out.println("No devices found.");
} else {
rebootDevice(devices.get(0));
}
}
public static AndroidManagement getAndroidManagementClient()
throws IOException, GeneralSecurityException {
try (FileInputStream input =
new FileInputStream(SERVICE_ACCOUNT_CREDENTIAL_FILE)) {
GoogleCredential credential =
GoogleCredential.fromStream(input)
.createScoped(Collections.singleton(OAUTH_SCOPE));
return new AndroidManagement.Builder(
GoogleNetHttpTransport.newTrustedTransport(),
JacksonFactory.getDefaultInstance(),
credential)
.setApplicationName(APP_NAME)
.build();
}
}
private String createEnterprise() throws IOException {
// Initiate signup process.
System.out.println("Creating signup URL...");
SignupUrl signupUrl =
androidManagementClient
.signupUrls()
.create()
.setProjectId(PROJECT_ID)
.setCallbackUrl("https://localhost:9999")
.execute();
System.out.print(
"To sign up for a new enterprise, open this URL in your browser: ");
System.out.println(signupUrl.getUrl());
System.out.println(
"After signup, you will see an error page in the browser.");
System.out.print(
"Paste the enterpriseToken value from the error page URL here: ");
String enterpriseToken =
new BufferedReader(new InputStreamReader(System.in)).readLine();
// Create the enterprise.
System.out.println("Creating enterprise...");
return androidManagementClient
.enterprises()
.create(new Enterprise())
.setProjectId(PROJECT_ID)
.setSignupUrlName(signupUrl.getName())
.setEnterpriseToken(enterpriseToken)
.execute()
.getName();
}
private Policy getCosuPolicy() {
List<String> categories = new ArrayList<>();
categories.add("android.intent.category.HOME");
categories.add("android.intent.category.DEFAULT");
return new Policy()
.setApplications(
Collections.singletonList(
new ApplicationPolicy()
.setPackageName(COSU_APP_PACKAGE_NAME)
.setInstallType("FORCE_INSTALLED")
.setDefaultPermissionPolicy("GRANT")
.setLockTaskAllowed(true)))
.setPersistentPreferredActivities(
Collections.singletonList(
new PersistentPreferredActivity()
.setReceiverActivity(COSU_APP_PACKAGE_NAME)
.setActions(
Collections.singletonList("android.intent.action.MAIN"))
.setCategories(categories)))
.setKeyguardDisabled(true)
.setStatusBarDisabled(true);
}
private void setPolicy(String enterpriseName, String policyId, Policy policy)
throws IOException {
System.out.println("Setting policy...");
String name = enterpriseName + "/policies/" + policyId;
androidManagementClient
.enterprises()
.policies()
.patch(name, policy)
.execute();
}
private String createEnrollmentToken(String enterpriseName, String policyId)
throws IOException {
System.out.println("Creating enrollment token...");
EnrollmentToken token =
new EnrollmentToken().setPolicyName(policyId).setDuration("86400s");
return androidManagementClient
.enterprises()
.enrollmentTokens()
.create(enterpriseName, token)
.execute()
.getValue();
}
private List<Device> listDevices(String enterpriseName) throws IOException {
System.out.println("Listing devices...");
ListDevicesResponse response =
androidManagementClient
.enterprises()
.devices()
.list(enterpriseName)
.execute();
return response.getDevices() ==null
? new ArrayList<Device>() : response.getDevices();
}
private void rebootDevice(Device device) throws IOException {
System.out.println(
"Sending reboot command to " + device.getName() + "...");
Command command = new Command().setType("REBOOT");
androidManagementClient
.enterprises()
.devices()
.issueCommand(device.getName(), command)
.execute();
}
Moreover i m using android management api for the first time and i dont know its proper implementation. Anyone who has experience on this kinllt guide me a little bit. I found a lot about this but i didn't found any userful tutorial
For Android, you have to store the service account file either in the assets folder or raw folder.
This thread provides code on a number of ways to load the json data into an InputStream depending on the location you selected.
I am trying to get a youtube video comments through youtube API on my dynamic web-app project.
My code:
private static int counter = 0;
private static YouTube youtube;
public static void getYoutubeOauth() throws Exception {
List<String> scopes = Lists.newArrayList("https://www.googleapis.com/auth/youtube.force-ssl");
Credential credential = Auth.authorize(scopes, "commentthreads");
youtube = new YouTube.Builder(Auth.HTTP_TRANSPORT, Auth.JSON_FACTORY, credential).build();
String videoId = "KIgxmV9xXBQ";
// Get video comments threads
CommentThreadListResponse commentsPage = prepareListRequest(videoId).execute();
while (true) {
handleCommentsThreads(commentsPage.getItems());
String nextPageToken = commentsPage.getNextPageToken();
if (nextPageToken == null)
break;
// Get next page of video comments threads
commentsPage = prepareListRequest(videoId).setPageToken(nextPageToken).execute();
}
System.out.println("Total: " + counter);
}
private static YouTube.CommentThreads.List prepareListRequest(String videoId) throws Exception {
return youtube.commentThreads()
.list("snippet,replies")
.setVideoId(videoId)
.setMaxResults(100L)
.setModerationStatus("published")
.setTextFormat("plainText");
}
private static void handleCommentsThreads(List<CommentThread> commentThreads) {
for (CommentThread commentThread : commentThreads) {
List<Comment> comments = Lists.newArrayList();
comments.add(commentThread.getSnippet().getTopLevelComment());
CommentThreadReplies replies = commentThread.getReplies();
if (replies != null)
comments.addAll(replies.getComments());
System.out.println("Found " + comments.size() + " comments.");
// Do your comments logic here
counter += comments.size();
}
}
while executing above code I'm getting a java.lang.NullPointerException at line Credential credential = Auth.authorize(scopes, "commentthreads");
FYI: I do Know what NullPointerException is. i.e string n = null; if(n=="h"); causes for NullPointerException
I would expect the cause to be countered by client_secrets.json file, because I don't know where to place and how to call it in my code.
My current client_secrets.json is placed under /home/UserName/workspace/RemoteSystemsTempFiles/Duck/WebContent/WEB-INF/client_secrets.json
You can load the client secrets json file as follows:
// Load client secrets.
Reader clientSecretReader = new InputStreamReader(Auth.class.getResourceAsStream("/client_secrets.json"));
GoogleClientSecrets clientSecrets = GoogleClientSecrets.load(JSON_FACTORY, clientSecretReader);
For more information on how to use Auth with google API take a look here
I'm trying to read the files available on Amazon S3, as the question explains the problem. I couldn't find an alternative call for the deprecated constructor.
Here's the code:
private String AccessKeyID = "xxxxxxxxxxxxxxxxxxxx";
private String SecretAccessKey = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxx";
private static String bucketName = "documentcontainer";
private static String keyName = "test";
//private static String uploadFileName = "/PATH TO FILE WHICH WANT TO UPLOAD/abc.txt";
AWSCredentials credentials = new BasicAWSCredentials(AccessKeyID, SecretAccessKey);
void downloadfile() throws IOException
{
// Problem lies here - AmazonS3Client is deprecated
AmazonS3 s3client = new AmazonS3Client(credentials);
try {
System.out.println("Downloading an object...");
S3Object s3object = s3client.getObject(new GetObjectRequest(
bucketName, keyName));
System.out.println("Content-Type: " +
s3object.getObjectMetadata().getContentType());
InputStream input = s3object.getObjectContent();
BufferedReader reader = new BufferedReader(new InputStreamReader(input));
while (true) {
String line = reader.readLine();
if (line == null) break;
System.out.println(" " + line);
}
System.out.println();
} catch (AmazonServiceException ase) {
//do something
} catch (AmazonClientException ace) {
// do something
}
}
Any help? If more explanation is needed please mention it.
I have checked on the sample code provided in .zip file of SDK, and it's the same.
You can either use AmazonS3ClientBuilder or
AwsClientBuilder as alternatives.
For S3, simplest would be with AmazonS3ClientBuilder.
BasicAWSCredentials creds = new BasicAWSCredentials("access_key", "secret_key");
AmazonS3 s3Client = AmazonS3ClientBuilder
.standard()
.withCredentials(new AWSStaticCredentialsProvider(creds))
.build();
Use the code listed below to create an S3 client without credentials:
AmazonS3 s3Client = AmazonS3ClientBuilder.standard().build();
An usage example would be a lambda function calling S3.
You need to pass the region information through the
com.amazonaws.regions.Region object.
Use AmazonS3Client(credentials, Region.getRegion(Regions.REPLACE_WITH_YOUR_REGION))
You can create S3 default client as follows(with aws-java-sdk-s3-1.11.232):
AmazonS3ClientBuilder.defaultClient();
Deprecated with only creentials in constructor, you can use something like this:
val awsConfiguration = AWSConfiguration(context)
val awsCreds = CognitoCachingCredentialsProvider(context, awsConfiguration)
val s3Client = AmazonS3Client(awsCreds, Region.getRegion(Regions.EU_CENTRAL_1))
Using the AWS SDK for Java 2.x, one can also build its own credentialProvider like so:
// Credential provider
package com.myproxylib.aws;
import software.amazon.awssdk.auth.credentials.AwsCredentials;
import software.amazon.awssdk.auth.credentials.AwsCredentialsProvider;
public class CustomCredentialsProvider implements AwsCredentialsProvider {
private final String accessKeyId;
private final String secretAccessKey;
public CustomCredentialsProvider(String accessKeyId, String secretAccessKey) {
this.secretAccessKey = secretAccessKey;
this.accessKeyId = accessKeyId;
}
#Override
public AwsCredentials resolveCredentials() {
return new CustomAwsCredentialsResolver(accessKeyId, secretAccessKey);
}
}
// Crenditals resolver
package com.myproxylib.aws;
import software.amazon.awssdk.auth.credentials.AwsCredentials;
public class CustomAwsCredentialsResolver implements AwsCredentials {
private final String accessKeyId;
private final String secretAccessKey;
CustomAwsCredentialsResolver(String accessKeyId, String secretAccessKey) {
this.secretAccessKey = secretAccessKey;
this.accessKeyId = accessKeyId;
}
#Override
public String accessKeyId() {
return accessKeyId;
}
#Override
public String secretAccessKey() {
return secretAccessKey;
}
}
// Usage of the provider
package com.myproxylib.aws.s3;
public class S3Storage implements IS3StorageCapable {
private final S3Client s3Client;
public S3Storage(String accessKeyId, String secretAccessKey, String region) {
this.s3Client = S3Client.builder().credentialsProvider(new CustomCredentialsProvider(accessKeyId, secretAccessKey)).region(of(region)).build();
}
NOTE:
of course, the library user can get the credentials from wherever he wants, parse it into a java Properties before calling the S3 constructor.
When possible, favour the other methods mentionned in other answers and doc. My use case was necessary for this.
implementation 'com.amazonaws:aws-android-sdk-s3:2.16.12'
val key = "XXX"
val secret = "XXX"
val credentials = BasicAWSCredentials(key, secret)
val s3 = AmazonS3Client(
credentials, com.amazonaws.regions.Region.getRegion(
Regions.US_EAST_2
)
)
val expires = Date(Date().time + 1000 * 60 * 60)
val keyFile = "13/thumbnail_800x600_13_photo.jpeg"
val generatePresignedUrlRequest = GeneratePresignedUrlRequest(
"bucket_name",
keyFile
)
generatePresignedUrlRequest.expiration = expires
val url: URL = s3.generatePresignedUrl(generatePresignedUrlRequest)
GlideApp.with(this)
.load(url.toString())
.apply(RequestOptions.centerCropTransform())
.into(image)
When I run Microsoft Azure Media Services code written using Java in local it is working but when I deploy the same code in dev environment , I am unable to access the Azure and its throwing java.net.HostNotFoundException.
What is the best approach to use network proxy to connect to Azure
Below is the code I am using via java and using azure-java-sdk
import java.io.*;
import java.security.NoSuchAlgorithmException;
import java.util.EnumSet;
import com.microsoft.windowsazure.Configuration;
import com.microsoft.windowsazure.exception.ServiceException;
import com.microsoft.windowsazure.services.media.MediaConfiguration;
import com.microsoft.windowsazure.services.media.MediaContract;
import com.microsoft.windowsazure.services.media.MediaService;
import com.microsoft.windowsazure.services.media.WritableBlobContainerContract;
import com.microsoft.windowsazure.services.media.models.AccessPolicy;
import com.microsoft.windowsazure.services.media.models.AccessPolicyInfo;
import com.microsoft.windowsazure.services.media.models.AccessPolicyPermission;
import com.microsoft.windowsazure.services.media.models.Asset;
import com.microsoft.windowsazure.services.media.models.AssetFile;
import com.microsoft.windowsazure.services.media.models.AssetFileInfo;
import com.microsoft.windowsazure.services.media.models.AssetInfo;
import com.microsoft.windowsazure.services.media.models.Job;
import com.microsoft.windowsazure.services.media.models.JobInfo;
import com.microsoft.windowsazure.services.media.models.JobState;
import com.microsoft.windowsazure.services.media.models.ListResult;
import com.microsoft.windowsazure.services.media.models.Locator;
import com.microsoft.windowsazure.services.media.models.LocatorInfo;
import com.microsoft.windowsazure.services.media.models.LocatorType;
import com.microsoft.windowsazure.services.media.models.MediaProcessor;
import com.microsoft.windowsazure.services.media.models.MediaProcessorInfo;
import com.microsoft.windowsazure.services.media.models.Task;
public class HelloMediaServices
{
// Media Services account credentials configuration
private static String mediaServiceUri = "https://media.windows.net/API/";
private static String oAuthUri = "https://wamsprodglobal001acs.accesscontrol.windows.net/v2/OAuth2-13";
private static String clientId = "account name";
private static String clientSecret = "account key";
private static String scope = "urn:WindowsAzureMediaServices";
private static MediaContract mediaService;
// Encoder configuration
private static String preferedEncoder = "Media Encoder Standard";
private static String encodingPreset = "H264 Multiple Bitrate 720p";
public static void main(String[] args)
{
try {
// Set up the MediaContract object to call into the Media Services account
Configuration configuration = MediaConfiguration.configureWithOAuthAuthentication(
mediaServiceUri, oAuthUri, clientId, clientSecret, scope);
mediaService = MediaService.create(configuration);
// Upload a local file to an Asset
AssetInfo uploadAsset = uploadFileAndCreateAsset("BigBuckBunny.mp4");
System.out.println("Uploaded Asset Id: " + uploadAsset.getId());
// Transform the Asset
AssetInfo encodedAsset = encode(uploadAsset);
System.out.println("Encoded Asset Id: " + encodedAsset.getId());
// Create the Streaming Origin Locator
String url = getStreamingOriginLocator(encodedAsset);
System.out.println("Origin Locator URL: " + url);
System.out.println("Sample completed!");
} catch (ServiceException se) {
System.out.println("ServiceException encountered.");
System.out.println(se.toString());
} catch (Exception e) {
System.out.println("Exception encountered.");
System.out.println(e.toString());
}
}
private static AssetInfo uploadFileAndCreateAsset(String fileName)
throws ServiceException, FileNotFoundException, NoSuchAlgorithmException {
WritableBlobContainerContract uploader;
AssetInfo resultAsset;
AccessPolicyInfo uploadAccessPolicy;
LocatorInfo uploadLocator = null;
// Create an Asset
resultAsset = mediaService.create(Asset.create().setName(fileName).setAlternateId("altId"));
System.out.println("Created Asset " + fileName);
// Create an AccessPolicy that provides Write access for 15 minutes
uploadAccessPolicy = mediaService
.create(AccessPolicy.create("uploadAccessPolicy", 15.0, EnumSet.of(AccessPolicyPermission.WRITE)));
// Create a Locator using the AccessPolicy and Asset
uploadLocator = mediaService
.create(Locator.create(uploadAccessPolicy.getId(), resultAsset.getId(), LocatorType.SAS));
// Create the Blob Writer using the Locator
uploader = mediaService.createBlobWriter(uploadLocator);
File file = new File("BigBuckBunny.mp4");
// The local file that will be uploaded to your Media Services account
InputStream input = new FileInputStream(file);
System.out.println("Uploading " + fileName);
// Upload the local file to the asset
uploader.createBlockBlob(fileName, input);
// Inform Media Services about the uploaded files
mediaService.action(AssetFile.createFileInfos(resultAsset.getId()));
System.out.println("Uploaded Asset File " + fileName);
mediaService.delete(Locator.delete(uploadLocator.getId()));
mediaService.delete(AccessPolicy.delete(uploadAccessPolicy.getId()));
return resultAsset;
}
// Create a Job that contains a Task to transform the Asset
private static AssetInfo encode(AssetInfo assetToEncode)
throws ServiceException, InterruptedException {
// Retrieve the list of Media Processors that match the name
ListResult<MediaProcessorInfo> mediaProcessors = mediaService
.list(MediaProcessor.list().set("$filter", String.format("Name eq '%s'", preferedEncoder)));
// Use the latest version of the Media Processor
MediaProcessorInfo mediaProcessor = null;
for (MediaProcessorInfo info : mediaProcessors) {
if (null == mediaProcessor || info.getVersion().compareTo(mediaProcessor.getVersion()) > 0) {
mediaProcessor = info;
}
}
System.out.println("Using Media Processor: " + mediaProcessor.getName() + " " + mediaProcessor.getVersion());
// Create a task with the specified Media Processor
String outputAssetName = String.format("%s as %s", assetToEncode.getName(), encodingPreset);
String taskXml = "<taskBody><inputAsset>JobInputAsset(0)</inputAsset>"
+ "<outputAsset assetCreationOptions=\"0\"" // AssetCreationOptions.None
+ " assetName=\"" + outputAssetName + "\">JobOutputAsset(0)</outputAsset></taskBody>";
Task.CreateBatchOperation task = Task.create(mediaProcessor.getId(), taskXml)
.setConfiguration(encodingPreset).setName("Encoding");
// Create the Job; this automatically schedules and runs it.
Job.Creator jobCreator = Job.create()
.setName(String.format("Encoding %s to %s", assetToEncode.getName(), encodingPreset))
.addInputMediaAsset(assetToEncode.getId()).setPriority(2).addTaskCreator(task);
JobInfo job = mediaService.create(jobCreator);
String jobId = job.getId();
System.out.println("Created Job with Id: " + jobId);
// Check to see if the Job has completed
checkJobStatus(jobId);
// Done with the Job
// Retrieve the output Asset
ListResult<AssetInfo> outputAssets = mediaService.list(Asset.list(job.getOutputAssetsLink()));
return outputAssets.get(0);
}
public static String getStreamingOriginLocator(AssetInfo asset) throws ServiceException {
// Get the .ISM AssetFile
ListResult<AssetFileInfo> assetFiles = mediaService.list(AssetFile.list(asset.getAssetFilesLink()));
AssetFileInfo streamingAssetFile = null;
for (AssetFileInfo file : assetFiles) {
if (file.getName().toLowerCase().endsWith(".ism")) {
streamingAssetFile = file;
break;
}
}
AccessPolicyInfo originAccessPolicy;
LocatorInfo originLocator = null;
// Create a 30-day readonly AccessPolicy
double durationInMinutes = 60 * 24 * 30;
originAccessPolicy = mediaService.create(
AccessPolicy.create("Streaming policy", durationInMinutes, EnumSet.of(AccessPolicyPermission.READ)));
// Create a Locator using the AccessPolicy and Asset
originLocator = mediaService
.create(Locator.create(originAccessPolicy.getId(), asset.getId(), LocatorType.OnDemandOrigin));
// Create a Smooth Streaming base URL
return originLocator.getPath() + streamingAssetFile.getName() + "/manifest";
}
private static void checkJobStatus(String jobId) throws InterruptedException, ServiceException {
boolean done = false;
JobState jobState = null;
while (!done) {
// Sleep for 5 seconds
Thread.sleep(5000);
// Query the updated Job state
jobState = mediaService.get(Job.get(jobId)).getState();
System.out.println("Job state: " + jobState);
if (jobState == JobState.Finished || jobState == JobState.Canceled || jobState == JobState.Error) {
done = true;
}
}
}
}
I verified following code below which is working through fiddler proxy. Thanks to how to Capture https with fiddler, in java post which gave me hints:
System.setProperty("http.proxyHost", "127.0.0.1");
System.setProperty("https.proxyHost", "127.0.0.1");
System.setProperty("http.proxyPort", "8888");
System.setProperty("https.proxyPort", "8888");
System.setProperty("javax.net.ssl.trustStore", "C:\\Program Files\\Java\\jdk1.8.0_102\\bin\\FiddlerKeyStore");
System.setProperty("javax.net.ssl.trustStorePassword", "mypassword");
For others who face issue like me we can connect to azure mediaservices using network proxy by using below code
// Set up the MediaContract object to call into the Media Services account
Configuration configuration = MediaConfiguration.configureWithOAuthAuthentication(
mediaServiceUri, oAuthUri, clientId, clientSecret, scope);
configuration.getProperties().put(Configuration.PROPERTY_HTTP_PROXY_HOST, "Hostvalue");
configuration.getProperties().put(Configuration.PROPERTY_HTTP_PROXY_PORT, "Portvalue");
configuration.getProperties().put(Configuration.PROPERTY_HTTP_PROXY_SCHEME, "http");
MediaContract mediaService = MediaService.create(configuration);
Now use the mediaService to perform other operations.
When i am trying create container for my storage i get StorageException.
1.I created azure account.
2.I created azure storage for blob
3.I wrotten simple test(below)
4.I made this code on local machine and got exception. What is wrong?
public class Test {
public static final String storageConnectionString =
"DefaultEndpointsProtocol=https;" +
"AccountName=my_account;" +
"AccountKey=my_account_key";
public static void main(String[] args) throws StorageException, InvalidKeyException, URISyntaxException {
pushControll();
}
public static void pushControll() throws URISyntaxException, StorageException, InvalidKeyException {
CloudStorageAccount storageAccount = CloudStorageAccount.parse(storageConnectionString);
CloudBlobClient blobClient = storageAccount.createCloudBlobClient();
CloudBlobContainer container = blobClient.getContainerReference("observer");
container.create();
}
}
I get StorageException - >:
Exception in thread "main" com.microsoft.azure.storage.StorageException: Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
at com.microsoft.azure.storage.StorageException.translateException(StorageException.java:89)
at com.microsoft.azure.storage.core.StorageRequest.materializeException(StorageRequest.java:307)
at com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:182)
at com.microsoft.azure.storage.blob.CloudBlobContainer.create(CloudBlobContainer.java:279)
at com.microsoft.azure.storage.blob.CloudBlobContainer.create(CloudBlobContainer.java:252)
at ru.marketirs.model.Test.pushControll(Test.java:40)
at ru.marketirs.model.Test.main(Test.java:25)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:120)
Process finished with exit code 1
What have i done wrong?
Your code looks ok to me. Please check for 2 things: 1) Make sure that the account name/key is proper and 2) Check the clock on your computer and see if it is running slow. These 2 things could cause the error you're getting.
To by pass the proxy then please use like below, it works as expected and same has been tested.
public class AzureUpload {
// Define the connection-string with your values
/*public static final String storageConnectionString =
"DefaultEndpointsProtocol=http;" +
"AccountName=your_storage_account;" +
"AccountKey=your_storage_account_key";*/
public static final String storageConnectionString =
"DefaultEndpointsProtocol=http;" +
"AccountName=test2rdrhgf62;" +
"AccountKey=1gy3lpE7Du1j5ljKiupjhgjghjcbfgTGhbntjnRfr9Yi6GUQqVMQqGxd7/YThisv/OVVLfIOv9kQ==";
// Define the path to a local file.
static final String filePath = "D:\\Project\\Supporting Files\\Jar's\\Azure\\azure-storage-1.2.0.jar";
static final String file_Path = "D:\\Project\\Healthcare\\Azcopy_To_Azure\\data";
public static void main(String[] args) {
try
{
// Retrieve storage account from connection-string.
//String storageConnectionString = RoleEnvironment.getConfigurationSettings().get("StorageConnectionString");
//Proxy httpProxy = new Proxy(Proxy.Type.HTTP,new InetSocketAddress("132.186.192.234",8080));
System.setProperty("http.proxyHost", "102.122.15.234");
System.setProperty("http.proxyPort", "80");
System.setProperty("https.proxyUser", "ad001\\empid001");
System.setProperty("https.proxyPassword", "pass!1");
// Retrieve storage account from connection-string.
CloudStorageAccount storageAccount = CloudStorageAccount.parse(storageConnectionString);
// Create the blob client.
CloudBlobClient blobClient = storageAccount.createCloudBlobClient();
// Get a reference to a container.
// The container name must be lower case
CloudBlobContainer container = blobClient.getContainerReference("rpmsdatafromhospital");
// Create the container if it does not exist.
container.createIfNotExists();
// Create a permissions object.
BlobContainerPermissions containerPermissions = new BlobContainerPermissions();
// Include public access in the permissions object.
containerPermissions.setPublicAccess(BlobContainerPublicAccessType.CONTAINER);
// Set the permissions on the container.
container.uploadPermissions(containerPermissions);
// Create or overwrite the new file to blob with contents from a local file.
/*CloudBlockBlob blob = container.getBlockBlobReference("azure-storage-1.2.0.jar");
File source = new File(filePath);
blob.upload(new FileInputStream(source), source.length());*/
String envFilePath = System.getenv("AZURE_FILE_PATH");
//upload list of files/directory to blob storage
File folder = new File(envFilePath);
File[] listOfFiles = folder.listFiles();
for (int i = 0; i < listOfFiles.length; i++) {
if (listOfFiles[i].isFile()) {
System.out.println("File " + listOfFiles[i].getName());
CloudBlockBlob blob = container.getBlockBlobReference(listOfFiles[i].getName());
File source = new File(envFilePath+"\\"+listOfFiles[i].getName());
blob.upload(new FileInputStream(source), source.length());
System.out.println("File " + listOfFiles[i].getName()+ " upload successful");
}
//directory upload
/*else if (listOfFiles[i].isDirectory()) {
System.out.println("Directory " + listOfFiles[i].getName());
CloudBlockBlob blob = container.getBlockBlobReference(listOfFiles[i].getName());
File source = new File(file_Path+"\\"+listOfFiles[i].getName());
blob.upload(new FileInputStream(source), source.length());
}*/
}
}catch (Exception e)
{
// Output the stack trace.
e.printStackTrace();
}
}
}
.Net or C# then please add below code to "App.config"
<?xml version="1.0" encoding="utf-8" ?>
<configuration>
<startup>
<supportedRuntime version="v4.0" sku=".NETFramework,Version=v4.5.2" />
</startup>
<system.net>
<defaultProxy enabled="true" useDefaultCredentials="true">
<proxy usesystemdefault="true" />
</defaultProxy>
</system.net>
</configuration>