StorageException for azure blob with java - java

When i am trying create container for my storage i get StorageException.
1.I created azure account.
2.I created azure storage for blob
3.I wrotten simple test(below)
4.I made this code on local machine and got exception. What is wrong?
public class Test {
public static final String storageConnectionString =
"DefaultEndpointsProtocol=https;" +
"AccountName=my_account;" +
"AccountKey=my_account_key";
public static void main(String[] args) throws StorageException, InvalidKeyException, URISyntaxException {
pushControll();
}
public static void pushControll() throws URISyntaxException, StorageException, InvalidKeyException {
CloudStorageAccount storageAccount = CloudStorageAccount.parse(storageConnectionString);
CloudBlobClient blobClient = storageAccount.createCloudBlobClient();
CloudBlobContainer container = blobClient.getContainerReference("observer");
container.create();
}
}
I get StorageException - >:
Exception in thread "main" com.microsoft.azure.storage.StorageException: Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
at com.microsoft.azure.storage.StorageException.translateException(StorageException.java:89)
at com.microsoft.azure.storage.core.StorageRequest.materializeException(StorageRequest.java:307)
at com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:182)
at com.microsoft.azure.storage.blob.CloudBlobContainer.create(CloudBlobContainer.java:279)
at com.microsoft.azure.storage.blob.CloudBlobContainer.create(CloudBlobContainer.java:252)
at ru.marketirs.model.Test.pushControll(Test.java:40)
at ru.marketirs.model.Test.main(Test.java:25)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:120)
Process finished with exit code 1
What have i done wrong?

Your code looks ok to me. Please check for 2 things: 1) Make sure that the account name/key is proper and 2) Check the clock on your computer and see if it is running slow. These 2 things could cause the error you're getting.

To by pass the proxy then please use like below, it works as expected and same has been tested.
public class AzureUpload {
// Define the connection-string with your values
/*public static final String storageConnectionString =
"DefaultEndpointsProtocol=http;" +
"AccountName=your_storage_account;" +
"AccountKey=your_storage_account_key";*/
public static final String storageConnectionString =
"DefaultEndpointsProtocol=http;" +
"AccountName=test2rdrhgf62;" +
"AccountKey=1gy3lpE7Du1j5ljKiupjhgjghjcbfgTGhbntjnRfr9Yi6GUQqVMQqGxd7/YThisv/OVVLfIOv9kQ==";
// Define the path to a local file.
static final String filePath = "D:\\Project\\Supporting Files\\Jar's\\Azure\\azure-storage-1.2.0.jar";
static final String file_Path = "D:\\Project\\Healthcare\\Azcopy_To_Azure\\data";
public static void main(String[] args) {
try
{
// Retrieve storage account from connection-string.
//String storageConnectionString = RoleEnvironment.getConfigurationSettings().get("StorageConnectionString");
//Proxy httpProxy = new Proxy(Proxy.Type.HTTP,new InetSocketAddress("132.186.192.234",8080));
System.setProperty("http.proxyHost", "102.122.15.234");
System.setProperty("http.proxyPort", "80");
System.setProperty("https.proxyUser", "ad001\\empid001");
System.setProperty("https.proxyPassword", "pass!1");
// Retrieve storage account from connection-string.
CloudStorageAccount storageAccount = CloudStorageAccount.parse(storageConnectionString);
// Create the blob client.
CloudBlobClient blobClient = storageAccount.createCloudBlobClient();
// Get a reference to a container.
// The container name must be lower case
CloudBlobContainer container = blobClient.getContainerReference("rpmsdatafromhospital");
// Create the container if it does not exist.
container.createIfNotExists();
// Create a permissions object.
BlobContainerPermissions containerPermissions = new BlobContainerPermissions();
// Include public access in the permissions object.
containerPermissions.setPublicAccess(BlobContainerPublicAccessType.CONTAINER);
// Set the permissions on the container.
container.uploadPermissions(containerPermissions);
// Create or overwrite the new file to blob with contents from a local file.
/*CloudBlockBlob blob = container.getBlockBlobReference("azure-storage-1.2.0.jar");
File source = new File(filePath);
blob.upload(new FileInputStream(source), source.length());*/
String envFilePath = System.getenv("AZURE_FILE_PATH");
//upload list of files/directory to blob storage
File folder = new File(envFilePath);
File[] listOfFiles = folder.listFiles();
for (int i = 0; i < listOfFiles.length; i++) {
if (listOfFiles[i].isFile()) {
System.out.println("File " + listOfFiles[i].getName());
CloudBlockBlob blob = container.getBlockBlobReference(listOfFiles[i].getName());
File source = new File(envFilePath+"\\"+listOfFiles[i].getName());
blob.upload(new FileInputStream(source), source.length());
System.out.println("File " + listOfFiles[i].getName()+ " upload successful");
}
//directory upload
/*else if (listOfFiles[i].isDirectory()) {
System.out.println("Directory " + listOfFiles[i].getName());
CloudBlockBlob blob = container.getBlockBlobReference(listOfFiles[i].getName());
File source = new File(file_Path+"\\"+listOfFiles[i].getName());
blob.upload(new FileInputStream(source), source.length());
}*/
}
}catch (Exception e)
{
// Output the stack trace.
e.printStackTrace();
}
}
}
.Net or C# then please add below code to "App.config"
<?xml version="1.0" encoding="utf-8" ?>
<configuration>
<startup>
<supportedRuntime version="v4.0" sku=".NETFramework,Version=v4.5.2" />
</startup>
<system.net>
<defaultProxy enabled="true" useDefaultCredentials="true">
<proxy usesystemdefault="true" />
</defaultProxy>
</system.net>
</configuration>

Related

Android management API service file allocation

I am testing android managment API using localhost as call back url. I followed each and every step following this url Android Management API Sample.
Now i m stuck on place.. according to this guide, i download the json file from service account. Now i copy that json file and save in app folder of my project.
This is my enterprise.json file
Screenshot of json file in android studio
and i just give folder location as enterprise.json in location string
This is my code
private static final String PROJECT_ID = "enterprise-271814";
private static final String SERVICE_ACCOUNT_CREDENTIAL_FILE =
"enterprise.json";
private static final String POLICY_ID = "samplePolicy";
/** The package name of the COSU app. */
private static final String COSU_APP_PACKAGE_NAME =
"com.ariaware.devicepoliceycontroller";
/** The OAuth scope for the Android Management API. */
private static final String OAUTH_SCOPE =
"https://www.googleapis.com/auth/androidmanagement";
private static final String APP_NAME = "Device Policey Controller";
private final AndroidManagement androidManagementClient;
public Sample(AndroidManagement androidManagementClient) {
this.androidManagementClient = androidManagementClient;
}
public void run() throws IOException {
// Create an enterprise. If you've already created an enterprise, the
// createEnterprise call can be commented out and replaced with your
// enterprise name.
String enterpriseName = createEnterprise();
System.out.println("Enterprise created with name: " + enterpriseName);
// Set the policy to be used by the device.
setPolicy(enterpriseName, POLICY_ID, getCosuPolicy());
// Create an enrollment token to enroll the device.
String token = createEnrollmentToken(enterpriseName, POLICY_ID);
System.out.println("Enrollment token (to be typed on device): " + token);
// List some of the devices for the enterprise. There will be no devices for
// a newly created enterprise, but you can run the app again with an
// existing enterprise after enrolling a device.
List<Device> devices = listDevices(enterpriseName);
for (Device device : devices) {
System.out.println("Found device with name: " + device.getName());
}
// If there are any devices, reboot one.
if (devices.isEmpty()) {
System.out.println("No devices found.");
} else {
rebootDevice(devices.get(0));
}
}
public static AndroidManagement getAndroidManagementClient()
throws IOException, GeneralSecurityException {
try (FileInputStream input =
new FileInputStream(SERVICE_ACCOUNT_CREDENTIAL_FILE)) {
GoogleCredential credential =
GoogleCredential.fromStream(input)
.createScoped(Collections.singleton(OAUTH_SCOPE));
return new AndroidManagement.Builder(
GoogleNetHttpTransport.newTrustedTransport(),
JacksonFactory.getDefaultInstance(),
credential)
.setApplicationName(APP_NAME)
.build();
}
}
private String createEnterprise() throws IOException {
// Initiate signup process.
System.out.println("Creating signup URL...");
SignupUrl signupUrl =
androidManagementClient
.signupUrls()
.create()
.setProjectId(PROJECT_ID)
.setCallbackUrl("https://localhost:9999")
.execute();
System.out.print(
"To sign up for a new enterprise, open this URL in your browser: ");
System.out.println(signupUrl.getUrl());
System.out.println(
"After signup, you will see an error page in the browser.");
System.out.print(
"Paste the enterpriseToken value from the error page URL here: ");
String enterpriseToken =
new BufferedReader(new InputStreamReader(System.in)).readLine();
// Create the enterprise.
System.out.println("Creating enterprise...");
return androidManagementClient
.enterprises()
.create(new Enterprise())
.setProjectId(PROJECT_ID)
.setSignupUrlName(signupUrl.getName())
.setEnterpriseToken(enterpriseToken)
.execute()
.getName();
}
private Policy getCosuPolicy() {
List<String> categories = new ArrayList<>();
categories.add("android.intent.category.HOME");
categories.add("android.intent.category.DEFAULT");
return new Policy()
.setApplications(
Collections.singletonList(
new ApplicationPolicy()
.setPackageName(COSU_APP_PACKAGE_NAME)
.setInstallType("FORCE_INSTALLED")
.setDefaultPermissionPolicy("GRANT")
.setLockTaskAllowed(true)))
.setPersistentPreferredActivities(
Collections.singletonList(
new PersistentPreferredActivity()
.setReceiverActivity(COSU_APP_PACKAGE_NAME)
.setActions(
Collections.singletonList("android.intent.action.MAIN"))
.setCategories(categories)))
.setKeyguardDisabled(true)
.setStatusBarDisabled(true);
}
private void setPolicy(String enterpriseName, String policyId, Policy policy)
throws IOException {
System.out.println("Setting policy...");
String name = enterpriseName + "/policies/" + policyId;
androidManagementClient
.enterprises()
.policies()
.patch(name, policy)
.execute();
}
private String createEnrollmentToken(String enterpriseName, String policyId)
throws IOException {
System.out.println("Creating enrollment token...");
EnrollmentToken token =
new EnrollmentToken().setPolicyName(policyId).setDuration("86400s");
return androidManagementClient
.enterprises()
.enrollmentTokens()
.create(enterpriseName, token)
.execute()
.getValue();
}
private List<Device> listDevices(String enterpriseName) throws IOException {
System.out.println("Listing devices...");
ListDevicesResponse response =
androidManagementClient
.enterprises()
.devices()
.list(enterpriseName)
.execute();
return response.getDevices() ==null
? new ArrayList<Device>() : response.getDevices();
}
private void rebootDevice(Device device) throws IOException {
System.out.println(
"Sending reboot command to " + device.getName() + "...");
Command command = new Command().setType("REBOOT");
androidManagementClient
.enterprises()
.devices()
.issueCommand(device.getName(), command)
.execute();
}
Moreover i m using android management api for the first time and i dont know its proper implementation. Anyone who has experience on this kinllt guide me a little bit. I found a lot about this but i didn't found any userful tutorial
For Android, you have to store the service account file either in the assets folder or raw folder.
This thread provides code on a number of ways to load the json data into an InputStream depending on the location you selected.

How to add SAS for file services of azure in java?

I got code in c# or code for blob storage. I need some reference code in java to have SAS token for file storage in azure. The SAS may be applicable for account or services.
Code 1 :
static string GetAccountSASToken()
{
// To create the account SAS, you need to use your shared key credentials. Modify for your account.
const string ConnectionString = "DefaultEndpointsProtocol=https;AccountName=account-name;AccountKey=account-key";
CloudStorageAccount storageAccount = CloudStorageAccount.Parse(ConnectionString);
// Create a new access policy for the account.
SharedAccessAccountPolicy policy = new SharedAccessAccountPolicy()
{
Permissions = SharedAccessAccountPermissions.Read | SharedAccessAccountPermissions.Write | SharedAccessAccountPermissions.List,
Services = SharedAccessAccountServices.Blob | SharedAccessAccountServices.File,
ResourceTypes = SharedAccessAccountResourceTypes.Service,
SharedAccessExpiryTime = DateTime.UtcNow.AddHours(24),
Protocols = SharedAccessProtocol.HttpsOnly
};
// Return the SAS token.
return storageAccount.GetSharedAccessSignature(policy);
}
This code is about creating SAS token for account verification and expiry time.I need the same in java. I am not getting few things like, in first code how I can write the 'Permission' in java? I mean multiple in one line. Please provide equivalent java code for this.
Code 2 :
#Test
public String testFileSAS(CloudFileShare share, CloudFile file) throws InvalidKeyException,
IllegalArgumentException, StorageException, URISyntaxException, InterruptedException {
SharedAccessFilePolicy policy = createSharedAccessPolicy(EnumSet.of(SharedAccessFilePermissions.READ,
SharedAccessFilePermissions.LIST, SharedAccessFilePermissions.WRITE), 24);
FileSharePermissions perms = new FileSharePermissions();
// SharedAccessProtocols protocol = SharedAccessProtocols.HTTPS_ONLY;
perms.getSharedAccessPolicies().put("readperm", policy);
share.uploadPermissions(perms);
// Thread.sleep(30000);
CloudFile sasFile = new CloudFile(
new URI(file.getUri().toString() + "?" + file.generateSharedAccessSignature(null, "readperm")));
sasFile.download(new ByteArrayOutputStream());
// do not give the client and check that the new file's client has the
// correct permissions
CloudFile fileFromUri = new CloudFile(
PathUtility.addToQuery(file.getStorageUri(), file.generateSharedAccessSignature(null, "readperm")));
assertEquals(StorageCredentialsSharedAccessSignature.class.toString(),
fileFromUri.getServiceClient().getCredentials().getClass().toString());
// create credentials from sas
StorageCredentials creds = new StorageCredentialsSharedAccessSignature(
file.generateSharedAccessSignature(policy, null, null));
System.out.println("Generated SAS token is : " + file.generateSharedAccessSignature(policy, null, null));
String token = file.generateSharedAccessSignature(policy, null, null);
CloudFileClient client = new CloudFileClient(sasFile.getServiceClient().getStorageUri(), creds);
CloudFile fileFromClient = client.getShareReference(file.getShare().getName()).getRootDirectoryReference()
.getFileReference(file.getName());
assertEquals(StorageCredentialsSharedAccessSignature.class.toString(),
fileFromClient.getServiceClient().getCredentials().getClass().toString());
assertEquals(client, fileFromClient.getServiceClient());
// self written
// String sharedUri =
// file.generateSharedAccessSignature(policy,null,null);
// System.out.println("token created is : "+sharedUri);
return token;
}
private final static SharedAccessFilePolicy createSharedAccessPolicy(EnumSet<SharedAccessFilePermissions> sap,
int expireTimeInSeconds) {
Calendar calendar = new GregorianCalendar(TimeZone.getTimeZone("UTC"));
calendar.setTime(new Date());
calendar.add(Calendar.HOUR, expireTimeInSeconds);
SharedAccessFilePolicy policy = new SharedAccessFilePolicy();
policy.setPermissions(sap);
policy.setSharedAccessExpiryTime(calendar.getTime());
return policy;
}
This code is a jUnit test. I don' want to import jUnit library. Want to do it in pure java.How I can convert the code? What I can use instead of assertEqauls?
Please consider the following code snippet in Java.
public static final String storageConnectionString = "DefaultEndpointsProtocol=https;AccountName=account-name;AccountKey=account-key";
public String getAccountSASToken() throws InvalidKeyException, URISyntaxException, StorageException {
CloudStorageAccount account = CloudStorageAccount.parse(storageConnectionString);
SharedAccessAccountPolicy policy = new SharedAccessAccountPolicy();
policy.setPermissions(EnumSet.of(SharedAccessAccountPermissions.READ, SharedAccessAccountPermissions.WRITE, SharedAccessAccountPermissions.LIST));
policy.setServices(EnumSet.of(SharedAccessAccountService.BLOB, SharedAccessAccountService.FILE) );
policy.setResourceTypes(EnumSet.of(SharedAccessAccountResourceType.SERVICE));
policy.setSharedAccessExpiryTime(Date.from(ZonedDateTime.now(ZoneOffset.UTC).plusHours(24L).toInstant()));
policy.setProtocols(SharedAccessProtocols.HTTPS_ONLY);
return account.generateSharedAccessSignature(policy);
}

How to use network proxy when connecting to Microsoft Azure Media Services

When I run Microsoft Azure Media Services code written using Java in local it is working but when I deploy the same code in dev environment , I am unable to access the Azure and its throwing java.net.HostNotFoundException.
What is the best approach to use network proxy to connect to Azure
Below is the code I am using via java and using azure-java-sdk
import java.io.*;
import java.security.NoSuchAlgorithmException;
import java.util.EnumSet;
import com.microsoft.windowsazure.Configuration;
import com.microsoft.windowsazure.exception.ServiceException;
import com.microsoft.windowsazure.services.media.MediaConfiguration;
import com.microsoft.windowsazure.services.media.MediaContract;
import com.microsoft.windowsazure.services.media.MediaService;
import com.microsoft.windowsazure.services.media.WritableBlobContainerContract;
import com.microsoft.windowsazure.services.media.models.AccessPolicy;
import com.microsoft.windowsazure.services.media.models.AccessPolicyInfo;
import com.microsoft.windowsazure.services.media.models.AccessPolicyPermission;
import com.microsoft.windowsazure.services.media.models.Asset;
import com.microsoft.windowsazure.services.media.models.AssetFile;
import com.microsoft.windowsazure.services.media.models.AssetFileInfo;
import com.microsoft.windowsazure.services.media.models.AssetInfo;
import com.microsoft.windowsazure.services.media.models.Job;
import com.microsoft.windowsazure.services.media.models.JobInfo;
import com.microsoft.windowsazure.services.media.models.JobState;
import com.microsoft.windowsazure.services.media.models.ListResult;
import com.microsoft.windowsazure.services.media.models.Locator;
import com.microsoft.windowsazure.services.media.models.LocatorInfo;
import com.microsoft.windowsazure.services.media.models.LocatorType;
import com.microsoft.windowsazure.services.media.models.MediaProcessor;
import com.microsoft.windowsazure.services.media.models.MediaProcessorInfo;
import com.microsoft.windowsazure.services.media.models.Task;
public class HelloMediaServices
{
// Media Services account credentials configuration
private static String mediaServiceUri = "https://media.windows.net/API/";
private static String oAuthUri = "https://wamsprodglobal001acs.accesscontrol.windows.net/v2/OAuth2-13";
private static String clientId = "account name";
private static String clientSecret = "account key";
private static String scope = "urn:WindowsAzureMediaServices";
private static MediaContract mediaService;
// Encoder configuration
private static String preferedEncoder = "Media Encoder Standard";
private static String encodingPreset = "H264 Multiple Bitrate 720p";
public static void main(String[] args)
{
try {
// Set up the MediaContract object to call into the Media Services account
Configuration configuration = MediaConfiguration.configureWithOAuthAuthentication(
mediaServiceUri, oAuthUri, clientId, clientSecret, scope);
mediaService = MediaService.create(configuration);
// Upload a local file to an Asset
AssetInfo uploadAsset = uploadFileAndCreateAsset("BigBuckBunny.mp4");
System.out.println("Uploaded Asset Id: " + uploadAsset.getId());
// Transform the Asset
AssetInfo encodedAsset = encode(uploadAsset);
System.out.println("Encoded Asset Id: " + encodedAsset.getId());
// Create the Streaming Origin Locator
String url = getStreamingOriginLocator(encodedAsset);
System.out.println("Origin Locator URL: " + url);
System.out.println("Sample completed!");
} catch (ServiceException se) {
System.out.println("ServiceException encountered.");
System.out.println(se.toString());
} catch (Exception e) {
System.out.println("Exception encountered.");
System.out.println(e.toString());
}
}
private static AssetInfo uploadFileAndCreateAsset(String fileName)
throws ServiceException, FileNotFoundException, NoSuchAlgorithmException {
WritableBlobContainerContract uploader;
AssetInfo resultAsset;
AccessPolicyInfo uploadAccessPolicy;
LocatorInfo uploadLocator = null;
// Create an Asset
resultAsset = mediaService.create(Asset.create().setName(fileName).setAlternateId("altId"));
System.out.println("Created Asset " + fileName);
// Create an AccessPolicy that provides Write access for 15 minutes
uploadAccessPolicy = mediaService
.create(AccessPolicy.create("uploadAccessPolicy", 15.0, EnumSet.of(AccessPolicyPermission.WRITE)));
// Create a Locator using the AccessPolicy and Asset
uploadLocator = mediaService
.create(Locator.create(uploadAccessPolicy.getId(), resultAsset.getId(), LocatorType.SAS));
// Create the Blob Writer using the Locator
uploader = mediaService.createBlobWriter(uploadLocator);
File file = new File("BigBuckBunny.mp4");
// The local file that will be uploaded to your Media Services account
InputStream input = new FileInputStream(file);
System.out.println("Uploading " + fileName);
// Upload the local file to the asset
uploader.createBlockBlob(fileName, input);
// Inform Media Services about the uploaded files
mediaService.action(AssetFile.createFileInfos(resultAsset.getId()));
System.out.println("Uploaded Asset File " + fileName);
mediaService.delete(Locator.delete(uploadLocator.getId()));
mediaService.delete(AccessPolicy.delete(uploadAccessPolicy.getId()));
return resultAsset;
}
// Create a Job that contains a Task to transform the Asset
private static AssetInfo encode(AssetInfo assetToEncode)
throws ServiceException, InterruptedException {
// Retrieve the list of Media Processors that match the name
ListResult<MediaProcessorInfo> mediaProcessors = mediaService
.list(MediaProcessor.list().set("$filter", String.format("Name eq '%s'", preferedEncoder)));
// Use the latest version of the Media Processor
MediaProcessorInfo mediaProcessor = null;
for (MediaProcessorInfo info : mediaProcessors) {
if (null == mediaProcessor || info.getVersion().compareTo(mediaProcessor.getVersion()) > 0) {
mediaProcessor = info;
}
}
System.out.println("Using Media Processor: " + mediaProcessor.getName() + " " + mediaProcessor.getVersion());
// Create a task with the specified Media Processor
String outputAssetName = String.format("%s as %s", assetToEncode.getName(), encodingPreset);
String taskXml = "<taskBody><inputAsset>JobInputAsset(0)</inputAsset>"
+ "<outputAsset assetCreationOptions=\"0\"" // AssetCreationOptions.None
+ " assetName=\"" + outputAssetName + "\">JobOutputAsset(0)</outputAsset></taskBody>";
Task.CreateBatchOperation task = Task.create(mediaProcessor.getId(), taskXml)
.setConfiguration(encodingPreset).setName("Encoding");
// Create the Job; this automatically schedules and runs it.
Job.Creator jobCreator = Job.create()
.setName(String.format("Encoding %s to %s", assetToEncode.getName(), encodingPreset))
.addInputMediaAsset(assetToEncode.getId()).setPriority(2).addTaskCreator(task);
JobInfo job = mediaService.create(jobCreator);
String jobId = job.getId();
System.out.println("Created Job with Id: " + jobId);
// Check to see if the Job has completed
checkJobStatus(jobId);
// Done with the Job
// Retrieve the output Asset
ListResult<AssetInfo> outputAssets = mediaService.list(Asset.list(job.getOutputAssetsLink()));
return outputAssets.get(0);
}
public static String getStreamingOriginLocator(AssetInfo asset) throws ServiceException {
// Get the .ISM AssetFile
ListResult<AssetFileInfo> assetFiles = mediaService.list(AssetFile.list(asset.getAssetFilesLink()));
AssetFileInfo streamingAssetFile = null;
for (AssetFileInfo file : assetFiles) {
if (file.getName().toLowerCase().endsWith(".ism")) {
streamingAssetFile = file;
break;
}
}
AccessPolicyInfo originAccessPolicy;
LocatorInfo originLocator = null;
// Create a 30-day readonly AccessPolicy
double durationInMinutes = 60 * 24 * 30;
originAccessPolicy = mediaService.create(
AccessPolicy.create("Streaming policy", durationInMinutes, EnumSet.of(AccessPolicyPermission.READ)));
// Create a Locator using the AccessPolicy and Asset
originLocator = mediaService
.create(Locator.create(originAccessPolicy.getId(), asset.getId(), LocatorType.OnDemandOrigin));
// Create a Smooth Streaming base URL
return originLocator.getPath() + streamingAssetFile.getName() + "/manifest";
}
private static void checkJobStatus(String jobId) throws InterruptedException, ServiceException {
boolean done = false;
JobState jobState = null;
while (!done) {
// Sleep for 5 seconds
Thread.sleep(5000);
// Query the updated Job state
jobState = mediaService.get(Job.get(jobId)).getState();
System.out.println("Job state: " + jobState);
if (jobState == JobState.Finished || jobState == JobState.Canceled || jobState == JobState.Error) {
done = true;
}
}
}
}
I verified following code below which is working through fiddler proxy. Thanks to how to Capture https with fiddler, in java post which gave me hints:
System.setProperty("http.proxyHost", "127.0.0.1");
System.setProperty("https.proxyHost", "127.0.0.1");
System.setProperty("http.proxyPort", "8888");
System.setProperty("https.proxyPort", "8888");
System.setProperty("javax.net.ssl.trustStore", "C:\\Program Files\\Java\\jdk1.8.0_102\\bin\\FiddlerKeyStore");
System.setProperty("javax.net.ssl.trustStorePassword", "mypassword");
For others who face issue like me we can connect to azure mediaservices using network proxy by using below code
// Set up the MediaContract object to call into the Media Services account
Configuration configuration = MediaConfiguration.configureWithOAuthAuthentication(
mediaServiceUri, oAuthUri, clientId, clientSecret, scope);
configuration.getProperties().put(Configuration.PROPERTY_HTTP_PROXY_HOST, "Hostvalue");
configuration.getProperties().put(Configuration.PROPERTY_HTTP_PROXY_PORT, "Portvalue");
configuration.getProperties().put(Configuration.PROPERTY_HTTP_PROXY_SCHEME, "http");
MediaContract mediaService = MediaService.create(configuration);
Now use the mediaService to perform other operations.

Java client to set SAS, Policy and CORS on Azure

I'm trying to set CORS properties on Azure using Java client. After executing code, I run HTML5 code to upload a file and facing following errors in chrome javascript console:
max block size = 47276
total blocks = 1
https:myacc.blob.core.windows.net/mycon/ch1.jpg?sr=c&sv=2015-04-05&sig=djbVxIBlyVy18bV0SkqNSLql1n9efAVcYnGy3VsGKis%3D&si=champ
current file pointer = 0 bytes read = 47276
block id = block-000000
https:myacc.blob.core.windows.net/mycon/ch1.jpg?sr=c&sv=2015-0…kqNSLql1n9efAVcYnGy3VsGKis%3D&si=champ&comp=block&blockid=YmxvY2stMDAwMDAw
Failed to load resource: the server responded with a status of 403 (CORS not enabled or no matching rule found for this request.)
XMLHttpRequest cannot load
https:myacc.blob.core.windows.net/mycon/ch1.jpg?sr=c&sv=2015-0…kqNSLql1n9efAVcYnGy3VsGKis%3D&si=heath&comp=block&blockid=YmxvY2stMDAwMDAw.
Response to preflight request doesn't pass access control check: No 'Access-Control-Allow-Origin' header is present on the requested
resource. Origin 'file://' is therefore not allowed access. The
response had HTTP status code 403.
What I'm wondering is why it didn't set CORS if Java client was executed successfully ? Also, how do I verify if rule Policy "champ" is configured properly, if my generated SAS is correct and CORS properties are created or not
Here is Java Client code:
public class CORS_and_SAS {
public static void main(String[] args) {
// Define the connection-string with your values
final String storageConnectionString ="DefaultEndpointsProtocol=http;" + "AccountName=myacc;" + "AccountKey=B2q4AGp6YoRsTREXIkOv3e/Sxf46YzqzfnM9F8U+o7VA5Y3EiKc+CuritnvuyZxGXKNOQ5nJy2KfkniF970on1dQ==";
try {
// Retrieve storage account from connection-string.
CloudStorageAccount storageAccount = CloudStorageAccount.parse(storageConnectionString);
// Create the blob client.
CloudBlobClient blobClient = storageAccount.createCloudBlobClient();
// Get a reference to a container.
// The container name must be lower case
CloudBlobContainer container = blobClient.getContainerReference("mycon");
// Create the container if it does not exist.
//container.createIfNotExists();
// Set CORS support
//ServiceProperties blobServiceProperties = blobClient.GetServiceProperties();
ServiceProperties propers = getCORS();
blobClient.uploadServiceProperties(propers);
SharedAccessBlobPolicy policy = new SharedAccessBlobPolicy();
GregorianCalendar calendar =
new GregorianCalendar(TimeZone.getTimeZone("UTC"));
calendar.setTime(new Date());
policy.setSharedAccessStartTime(calendar.getTime()); //Immediately applicable
calendar.add(Calendar.HOUR, 3000); //Applicable time-span is 3000 hours
policy.setSharedAccessExpiryTime(calendar.getTime());
policy.setPermissions(EnumSet.of(SharedAccessBlobPermissions.READ,
SharedAccessBlobPermissions.WRITE, SharedAccessBlobPermissions.DELETE,
SharedAccessBlobPermissions.LIST));
BlobContainerPermissions containerPermissions = new BlobContainerPermissions();
//Private container with no access for anonymous users
containerPermissions.setPublicAccess(BlobContainerPublicAccessType.OFF);
//Name the shared access policy: heath
containerPermissions.getSharedAccessPolicies().put("champ", policy);
container.uploadPermissions(containerPermissions);
//Generate the policy SAS string for heath access
String sas = container.generateSharedAccessSignature(
new SharedAccessBlobPolicy(),"champ");
System.out.println("The stored access policy signature:");
System.out.println(sas);
} catch (Exception e) {
// Output the stack trace.
e.printStackTrace();
}
}
private static ServiceProperties getCORS() {
// TODO Auto-generated method stub
ServiceProperties propers = new ServiceProperties();
CorsProperties corsprop = propers.getCors();
CorsRule cr = new CorsRule();
List<String> allowedHeaders = new ArrayList<String>();
allowedHeaders.add("x-ms-*");
List<String> exposedHeaders = new ArrayList<String>();
exposedHeaders.add("x-ms-*");
cr.setAllowedHeaders(allowedHeaders);
cr.setExposedHeaders(exposedHeaders);
EnumSet<CorsHttpMethods> allowedMethod = EnumSet.of(CorsHttpMethods.PUT,CorsHttpMethods.GET,CorsHttpMethods.POST,CorsHttpMethods.HEAD,CorsHttpMethods.DELETE);
//EnumSet<CorsHttpMethods> allowedMethod1 = EnumSet.of(CorsHttpMethods.GET);
cr.setAllowedMethods(allowedMethod);
List<String> allowedOrigin = new ArrayList<String>();
allowedOrigin.add("*");
cr.setAllowedOrigins(allowedOrigin);
cr.setMaxAgeInSeconds(600);
corsprop.getCorsRules().add(cr);
//corsprop.getCorsRules().add(cr);
propers.setCors(corsprop);
return propers;
}
}
I tried to reproduce the issue, and checked carefully the Java Client code & erros in JS console. I found that the issue was caused by using blob container Shared Access Signature for the uploading file url.
Here is the Java code modified by yours.
private static final String accountName = "<account-name>";
private static final String accountKey = "<account-key>";
private static final String connectionStringTemplate = "DefaultEndpointsProtocol=http;AccountName=%s;AccountKey=%s";
private static final String containerName = "<block-blob-container-name>";
private static final String blobFileName = "<blob-file-name>";
public static void main(String[] args) throws InvalidKeyException, URISyntaxException, StorageException {
String connectionString = String.format(connectionStringTemplate, accountName, accountKey);
CloudStorageAccount account = CloudStorageAccount.parse(connectionString);
CloudBlobClient blobClient = account.createCloudBlobClient();
/*
* Enable CORS
*/
// CORS should be enabled once at service startup
// Given a BlobClient, download the current Service Properties
ServiceProperties blobServiceProperties = blobClient.downloadServiceProperties();
// Enable and Configure CORS
CorsProperties cors = new CorsProperties();
CorsRule corsRule = new CorsRule();
List<String> allowedHeaders = new ArrayList<String>();
allowedHeaders.add("*");
EnumSet<CorsHttpMethods> allowedMethods = EnumSet.of(CorsHttpMethods.PUT, CorsHttpMethods.GET, CorsHttpMethods.HEAD, CorsHttpMethods.POST);
System.out.println(Arrays.toString(allowedMethods.toArray()));
List<String> allowedOrigins = new ArrayList<String>();
allowedOrigins.add("*");
List<String> exposedHeaders = new ArrayList<String>();
exposedHeaders.add("*");
int maxAgeInSeconds = 1800;
corsRule.setAllowedHeaders(allowedHeaders);
corsRule.setAllowedMethods(allowedMethods);
corsRule.setAllowedOrigins(allowedOrigins);
corsRule.setExposedHeaders(exposedHeaders);
corsRule.setMaxAgeInSeconds(maxAgeInSeconds);
cors.getCorsRules().add(corsRule);
blobServiceProperties.setCors(cors);
// Commit the CORS changes into the Service Properties
blobClient.uploadServiceProperties(blobServiceProperties);
/*
* Generate the SAS for the uploading url
*/
CloudBlobContainer container = blobClient.getContainerReference(containerName);
CloudBlockBlob blockBlob = container.getBlockBlobReference(blobFileName);
SharedAccessBlobPolicy sharedAccessBlobPolicy = new SharedAccessBlobPolicy();
GregorianCalendar calendar = new GregorianCalendar(TimeZone.getTimeZone("UTC"));
calendar.setTime(new Date());
sharedAccessBlobPolicy.setSharedAccessStartTime(calendar.getTime());
calendar.add(Calendar.HOUR, 1);
sharedAccessBlobPolicy.setSharedAccessExpiryTime(calendar.getTime());
sharedAccessBlobPolicy.setPermissions(EnumSet.of(SharedAccessBlobPermissions.WRITE));
String sas = blockBlob.generateSharedAccessSignature(sharedAccessBlobPolicy, null);
System.out.println(sas);
String blobUploadSASURL = String.format("https://%s.blob.core.windows.net/%s/%s?%s", accountName, containerName, blobFileName, sas);
System.out.println(blobUploadSASURL);
}
Run the code and get the uploading blob SAS Url as the form https://<account-name>.blob.core.windows.net/<container>/<blob-file-name>?sig=<SIG>&st=2015-12-01T11%3A51%3A20Z&se=2015-12-01T12%3A51%3A20Z&sv=2015-04-05&sp=r&sr=b
Using PUT method with header x-ms-blob-type: BlockBlob for the URL to upload a file successfully.
Further details and sample ajax code to do this is available, please refer to the blog from the Azure Storage team http://blogs.msdn.com/b/windowsazurestorage/archive/2014/02/03/windows-azure-storage-introducing-cors.aspx.

Upload Directory with files to S3 using Java

I'm working on the application where user will upload ZIP file to my server, on the server that ZIP file will be expanded and then I need to upload it to the server. Now my questions is: how to upload directory with multiple files and sub-folders using Java to S3 bucket? Is there any examples for that? Currently i'm using JetS3t to manage all my communications with S3.
HI This is the simple way to upload the Directory into S3 bucket.
BasicAWSCredentials awsCreds = new BasicAWSCredentials(access_key_id,
secret_access_key);
AmazonS3 s3Client = new AmazonS3Client(awsCreds);
TransferManager tm = TransferManagerBuilder.standard().withS3Client(s3Client).build();
MultipleFileUpload upload = tm.uploadDirectory(existingBucketName,
"BuildNumber#1", "FilePathYouWant", true);
I built something very similar. After expanding the zip on the server call FileUtils.listFiles() which will recursively list files in a folder. Just iterate the list and create s3objects and upload the files to s3. Make use of the threadedstorage service so that multiple files can be uploaded at the same time. Also ensure you process the upload events. If some files couldn't be uploaded the jets3t library will tell you.
I could post the code I wrote once get into the office.
EDIT: CODE:
Here's the code:
private static ProviderCredentials credentials;
private static S3Service s3service;
private static ThreadedS3Service storageService;
private static S3Bucket bucket;
private List<S3Object> s3Objs=new ArrayList<S3Object>();
private Set<String> s3ObjsCompleted=new HashSet<String>();
private boolean isErrorOccured=true;
private final ByteFormatter byteFormatter = new ByteFormatter();
private final TimeFormatter timeFormatter = new TimeFormatter();
private void initialise() throws ServiceException, S3ServiceException {
credentials=<create your credentials>;
s3service = new RestS3Service(credentials);
bucket = new S3Bucket(<bucket details>);
storageService=new ThreadedS3Service(s3service, this);
}
}
private void uploadFolder(File folder) throws NoSuchAlgorithmException, IOException {
readFolderContents(folder);
uploadFilesInList(folder);
}
private void readFolderContents(File folder) throws NoSuchAlgorithmException, IOException {
Iterator<File> filesinFolder=FileUtils.iterateFiles(folder,null,null);
while(filesinFolder.hasNext()) {
File file=filesinFolder.next();
String key = <create your key from the filename or something>;
S3Object s3Obj=new S3Object(bucket, file);
s3Obj.setKey(key);
s3Obj.setContentType(Mimetypes.getInstance().getMimetype(s3Obj.getKey()));
s3Objs.add(s3Obj);
}
}
private void uploadFilesInList(File folder) {
logger.debug("Uploading files in folder "+folder.getAbsolutePath());
isErrorOccured=false;
s3ObjsCompleted.clear();
storageService.putObjects(bucket.getName(), s3Objs.toArray(new S3Object[s3Objs.size()]));
if(isErrorOccured || s3Objs.size()!=s3ObjsCompleted.size()) {
logger.debug("Have to try uploading a few objects again for folder "+folder.getAbsolutePath()+" - Completed = "+s3ObjsCompleted.size()+" and Total ="+s3Objs.size());
List<S3Object> s3ObjsRemaining=new ArrayList<S3Object>();
for(S3Object s3Obj : s3Objs) {
if(!s3ObjsCompleted.contains(s3Obj.getKey())) {
s3ObjsRemaining.add(s3Obj);
}
}
s3Objs=s3ObjsRemaining;
uploadFilesInList(folder);
}
}
#Override
public void event(CreateObjectsEvent event) {
super.event(event);
if (ServiceEvent.EVENT_IGNORED_ERRORS == event.getEventCode()) {
Throwable[] throwables = event.getIgnoredErrors();
for (int i = 0; i < throwables.length; i++) {
logger.error("Ignoring error: " + throwables[i].getMessage());
}
}else if(ServiceEvent.EVENT_STARTED == event.getEventCode()) {
logger.debug("**********************************Upload Event Started***********************************");
}else if(event.getEventCode()==ServiceEvent.EVENT_ERROR) {
isErrorOccured=true;
}else if(event.getEventCode()==ServiceEvent.EVENT_IN_PROGRESS) {
StorageObject[] storeObjs=event.getCreatedObjects();
for(StorageObject storeObj : storeObjs) {
s3ObjsCompleted.add(storeObj.getKey());
}
ThreadWatcher watcher = event.getThreadWatcher();
if (watcher.getBytesTransferred() >= watcher.getBytesTotal()) {
logger.debug("Upload Completed.. Verifying");
}else {
int percentage = (int) (((double) watcher.getBytesTransferred() / watcher.getBytesTotal()) * 100);
long bytesPerSecond = watcher.getBytesPerSecond();
StringBuilder transferDetailsText=new StringBuilder("Uploading.... ");
transferDetailsText.append("Speed: " + byteFormatter.formatByteSize(bytesPerSecond) + "/s");
if (watcher.isTimeRemainingAvailable()) {
long secondsRemaining = watcher.getTimeRemaining();
if (transferDetailsText.length() > 0) {
transferDetailsText.append(" - ");
}
transferDetailsText.append("Time remaining: " + timeFormatter.formatTime(secondsRemaining));
}
logger.debug(transferDetailsText.toString()+" "+percentage);
}
}else if(ServiceEvent.EVENT_COMPLETED==event.getEventCode()) {
logger.debug("**********************************Upload Event Completed***********************************");
if(isErrorOccured) {
logger.debug("**********************But with errors, have to retry failed uploads**************************");
}
}
}
Here is how I did it in December of 2021 since BasicAWSCredentials is deprecated now.
AWSCredentials = new BasicAWSCredentials(env.getProperty("AWS_ACCESS_KEY_ID"),
env.getProperty("AWS_SECRET_ACCESS_KEY"));
AmazonS3 s3Client = AmazonS3ClientBuilder.standard()
.withRegion(Regions.US_EAST_1).withCredentials(new AWSStaticCredentialsProvider(AWSCredentials))
.build();
TransferManager tm = TransferManagerBuilder.standard().withS3Client(s3Client).build();
MultipleFileUpload upload = tm.uploadDirectory(existingBucketName,
"BuildNumber#1", "FilePathYouWant", true);

Categories