how to call client_secrets.json in youtube api? - java

I am trying to get a youtube video comments through youtube API on my dynamic web-app project.
My code:
private static int counter = 0;
private static YouTube youtube;
public static void getYoutubeOauth() throws Exception {
List<String> scopes = Lists.newArrayList("https://www.googleapis.com/auth/youtube.force-ssl");
Credential credential = Auth.authorize(scopes, "commentthreads");
youtube = new YouTube.Builder(Auth.HTTP_TRANSPORT, Auth.JSON_FACTORY, credential).build();
String videoId = "KIgxmV9xXBQ";
// Get video comments threads
CommentThreadListResponse commentsPage = prepareListRequest(videoId).execute();
while (true) {
handleCommentsThreads(commentsPage.getItems());
String nextPageToken = commentsPage.getNextPageToken();
if (nextPageToken == null)
break;
// Get next page of video comments threads
commentsPage = prepareListRequest(videoId).setPageToken(nextPageToken).execute();
}
System.out.println("Total: " + counter);
}
private static YouTube.CommentThreads.List prepareListRequest(String videoId) throws Exception {
return youtube.commentThreads()
.list("snippet,replies")
.setVideoId(videoId)
.setMaxResults(100L)
.setModerationStatus("published")
.setTextFormat("plainText");
}
private static void handleCommentsThreads(List<CommentThread> commentThreads) {
for (CommentThread commentThread : commentThreads) {
List<Comment> comments = Lists.newArrayList();
comments.add(commentThread.getSnippet().getTopLevelComment());
CommentThreadReplies replies = commentThread.getReplies();
if (replies != null)
comments.addAll(replies.getComments());
System.out.println("Found " + comments.size() + " comments.");
// Do your comments logic here
counter += comments.size();
}
}
while executing above code I'm getting a java.lang.NullPointerException at line Credential credential = Auth.authorize(scopes, "commentthreads");
FYI: I do Know what NullPointerException is. i.e string n = null; if(n=="h"); causes for NullPointerException
I would expect the cause to be countered by client_secrets.json file, because I don't know where to place and how to call it in my code.
My current client_secrets.json is placed under /home/UserName/workspace/RemoteSystemsTempFiles/Duck/WebContent/WEB-INF/client_secrets.json

You can load the client secrets json file as follows:
// Load client secrets.
Reader clientSecretReader = new InputStreamReader(Auth.class.getResourceAsStream("/client_secrets.json"));
GoogleClientSecrets clientSecrets = GoogleClientSecrets.load(JSON_FACTORY, clientSecretReader);
For more information on how to use Auth with google API take a look here

Related

Android management API service file allocation

I am testing android managment API using localhost as call back url. I followed each and every step following this url Android Management API Sample.
Now i m stuck on place.. according to this guide, i download the json file from service account. Now i copy that json file and save in app folder of my project.
This is my enterprise.json file
Screenshot of json file in android studio
and i just give folder location as enterprise.json in location string
This is my code
private static final String PROJECT_ID = "enterprise-271814";
private static final String SERVICE_ACCOUNT_CREDENTIAL_FILE =
"enterprise.json";
private static final String POLICY_ID = "samplePolicy";
/** The package name of the COSU app. */
private static final String COSU_APP_PACKAGE_NAME =
"com.ariaware.devicepoliceycontroller";
/** The OAuth scope for the Android Management API. */
private static final String OAUTH_SCOPE =
"https://www.googleapis.com/auth/androidmanagement";
private static final String APP_NAME = "Device Policey Controller";
private final AndroidManagement androidManagementClient;
public Sample(AndroidManagement androidManagementClient) {
this.androidManagementClient = androidManagementClient;
}
public void run() throws IOException {
// Create an enterprise. If you've already created an enterprise, the
// createEnterprise call can be commented out and replaced with your
// enterprise name.
String enterpriseName = createEnterprise();
System.out.println("Enterprise created with name: " + enterpriseName);
// Set the policy to be used by the device.
setPolicy(enterpriseName, POLICY_ID, getCosuPolicy());
// Create an enrollment token to enroll the device.
String token = createEnrollmentToken(enterpriseName, POLICY_ID);
System.out.println("Enrollment token (to be typed on device): " + token);
// List some of the devices for the enterprise. There will be no devices for
// a newly created enterprise, but you can run the app again with an
// existing enterprise after enrolling a device.
List<Device> devices = listDevices(enterpriseName);
for (Device device : devices) {
System.out.println("Found device with name: " + device.getName());
}
// If there are any devices, reboot one.
if (devices.isEmpty()) {
System.out.println("No devices found.");
} else {
rebootDevice(devices.get(0));
}
}
public static AndroidManagement getAndroidManagementClient()
throws IOException, GeneralSecurityException {
try (FileInputStream input =
new FileInputStream(SERVICE_ACCOUNT_CREDENTIAL_FILE)) {
GoogleCredential credential =
GoogleCredential.fromStream(input)
.createScoped(Collections.singleton(OAUTH_SCOPE));
return new AndroidManagement.Builder(
GoogleNetHttpTransport.newTrustedTransport(),
JacksonFactory.getDefaultInstance(),
credential)
.setApplicationName(APP_NAME)
.build();
}
}
private String createEnterprise() throws IOException {
// Initiate signup process.
System.out.println("Creating signup URL...");
SignupUrl signupUrl =
androidManagementClient
.signupUrls()
.create()
.setProjectId(PROJECT_ID)
.setCallbackUrl("https://localhost:9999")
.execute();
System.out.print(
"To sign up for a new enterprise, open this URL in your browser: ");
System.out.println(signupUrl.getUrl());
System.out.println(
"After signup, you will see an error page in the browser.");
System.out.print(
"Paste the enterpriseToken value from the error page URL here: ");
String enterpriseToken =
new BufferedReader(new InputStreamReader(System.in)).readLine();
// Create the enterprise.
System.out.println("Creating enterprise...");
return androidManagementClient
.enterprises()
.create(new Enterprise())
.setProjectId(PROJECT_ID)
.setSignupUrlName(signupUrl.getName())
.setEnterpriseToken(enterpriseToken)
.execute()
.getName();
}
private Policy getCosuPolicy() {
List<String> categories = new ArrayList<>();
categories.add("android.intent.category.HOME");
categories.add("android.intent.category.DEFAULT");
return new Policy()
.setApplications(
Collections.singletonList(
new ApplicationPolicy()
.setPackageName(COSU_APP_PACKAGE_NAME)
.setInstallType("FORCE_INSTALLED")
.setDefaultPermissionPolicy("GRANT")
.setLockTaskAllowed(true)))
.setPersistentPreferredActivities(
Collections.singletonList(
new PersistentPreferredActivity()
.setReceiverActivity(COSU_APP_PACKAGE_NAME)
.setActions(
Collections.singletonList("android.intent.action.MAIN"))
.setCategories(categories)))
.setKeyguardDisabled(true)
.setStatusBarDisabled(true);
}
private void setPolicy(String enterpriseName, String policyId, Policy policy)
throws IOException {
System.out.println("Setting policy...");
String name = enterpriseName + "/policies/" + policyId;
androidManagementClient
.enterprises()
.policies()
.patch(name, policy)
.execute();
}
private String createEnrollmentToken(String enterpriseName, String policyId)
throws IOException {
System.out.println("Creating enrollment token...");
EnrollmentToken token =
new EnrollmentToken().setPolicyName(policyId).setDuration("86400s");
return androidManagementClient
.enterprises()
.enrollmentTokens()
.create(enterpriseName, token)
.execute()
.getValue();
}
private List<Device> listDevices(String enterpriseName) throws IOException {
System.out.println("Listing devices...");
ListDevicesResponse response =
androidManagementClient
.enterprises()
.devices()
.list(enterpriseName)
.execute();
return response.getDevices() ==null
? new ArrayList<Device>() : response.getDevices();
}
private void rebootDevice(Device device) throws IOException {
System.out.println(
"Sending reboot command to " + device.getName() + "...");
Command command = new Command().setType("REBOOT");
androidManagementClient
.enterprises()
.devices()
.issueCommand(device.getName(), command)
.execute();
}
Moreover i m using android management api for the first time and i dont know its proper implementation. Anyone who has experience on this kinllt guide me a little bit. I found a lot about this but i didn't found any userful tutorial
For Android, you have to store the service account file either in the assets folder or raw folder.
This thread provides code on a number of ways to load the json data into an InputStream depending on the location you selected.

Hello Analytics API: Java quickstart errors

I'm trying to access data from the Google analytics reporting API using Java.
I was following the "Hello Analytics API: Java quickstart for installed applications" tutorial, and i did everything it tells you, and i get following issues:
com.google.api.client.util.store.FileDataStoreFactory setPermissionsToOwnerOnly
WARNING: unable to change permissions for everybody: C:\Users\<user>\.store\hello_analytics
com.google.api.client.util.store.FileDataStoreFactory setPermissionsToOwnerOnly
WARNING: unable to change permissions for owner: C:\Users\timst\.store\hello_analytics
java.lang.NullPointerException
at java.io.Reader.<init>(Reader.java:78)
at java.io.InputStreamReader.<init>(InputStreamReader.java:72)
at com.example.demo.HelloAnalytics.initializeAnalytics(HelloAnalytics.java:60)
at com.example.demo.HelloAnalytics.main(HelloAnalytics.java:44)
I tried using the full path for the client_secret.json.
tried using different methods i found online, but none seem to work.
After getting frustrated by this error i tried the "Hello Analytics API: Java quickstart for service accounts" tutorial.
But here i have the issue that i can't add users to the account, property or view for the accounts i can access.
I have access to other peoples analytics accounts and I can only remove myself from the accounts.
All code I'm using is from the tutorials, using Intellij and gradle.
tl;dr; All I want to do is access the analytics data for all my
accounts, using the reporting API so i can put all this data in my own
database and use this database for my other projects.
the Tutorials google provides doesn't work for me. (the data is mostly Google Adwords data.)
So the Warning is not the problem, it's a known issue with it not working properly on windows.
The java.lang.NullPointerException is because the profile I call to has no rows of data for the given metric. so the return value of the call doesn't have a .getRows() methode because there isn't a row value.
you should check for the row's first,
GaData results;
if (null != results) {
if(results.get("rows") != null){
if (!results.getRows().isEmpty()){
//do something with the rows exp.
for (List<String> row : results.getRows()) {
for (int i=0; i<results.getColumnHeaders().size();i++) {
List<GaData.ColumnHeaders> headers = results.getColumnHeaders();
System.out.println( headers.get(i).getName()+": " + row.get(i));
}
}
}
}
}
In the example I also use the ColumnHeaders, wich you should also check first.
It was also easier to check every single account i had access to and every webProperty and Profile and not just the first value of each of those.
Also, the query explorer is really useful. you should use it to check out which metrics you can use and which dimensions.
Here is my full HelloAnalytics class i just print everything that might be usefull to the console i also use multiple metrics and a dimension from Google AdWords in the getResults methode:
import com.google.api.client.auth.oauth2.Credential;
import com.google.api.client.extensions.java6.auth.oauth2.AuthorizationCodeInstalledApp;
import com.google.api.client.extensions.jetty.auth.oauth2.LocalServerReceiver;
import com.google.api.client.googleapis.auth.oauth2.GoogleAuthorizationCodeFlow;
import com.google.api.client.googleapis.auth.oauth2.GoogleClientSecrets;
import com.google.api.client.googleapis.javanet.GoogleNetHttpTransport;
import com.google.api.client.http.javanet.NetHttpTransport;
import com.google.api.client.json.JsonFactory;
import com.google.api.client.json.gson.GsonFactory;
import com.google.api.client.util.store.FileDataStoreFactory;
import com.google.api.services.analytics.Analytics;
import com.google.api.services.analytics.AnalyticsScopes;
import com.google.api.services.analytics.model.*;
import java.io.*;
import java.util.ArrayList;
import java.util.List;
/**
* A simple example of how to access the Google Analytics API.
*/
public class HelloAnalytics {
// Path to client_secrets.json file downloaded from the Developer's Console.
// The path is relative to HelloAnalytics.java.
private static final String CLIENT_SECRET_JSON_RESOURCE = "/client_secret.json";
// The directory where the user's credentials will be stored.
private static final File DATA_STORE_DIR = new File("out/DataStore/hello_analytics");
private static final File OUTPUT_FILE = new File("out/DataStore/output.text");
private static final String APPLICATION_NAME = "Online Marketing Buddy";
private static final JsonFactory JSON_FACTORY = GsonFactory.getDefaultInstance();
private static NetHttpTransport httpTransport;
private static FileDataStoreFactory dataStoreFactory;
public static void main(String[] args) {
try {
Analytics analytics = initializeAnalytics();
getProfileIds(analytics);
} catch (Exception e) {
e.printStackTrace();
}
}
private static Analytics initializeAnalytics() throws Exception {
httpTransport = GoogleNetHttpTransport.newTrustedTransport();
dataStoreFactory = new FileDataStoreFactory(DATA_STORE_DIR);
// Load client secrets.
InputStream in =
HelloAnalytics.class.getResourceAsStream(CLIENT_SECRET_JSON_RESOURCE);
GoogleClientSecrets clientSecrets =
GoogleClientSecrets.load(JSON_FACTORY, new InputStreamReader(in));
// Set up authorization code flow for all auth scopes.
GoogleAuthorizationCodeFlow flow = new GoogleAuthorizationCodeFlow
.Builder(httpTransport, JSON_FACTORY, clientSecrets,AnalyticsScopes.all())
.setDataStoreFactory(dataStoreFactory)
.build();
// Authorize.
Credential credential = new AuthorizationCodeInstalledApp(flow, new LocalServerReceiver())
.authorize("user");
// Construct the Analytics service object.
Analytics response = new Analytics
.Builder(httpTransport, JSON_FACTORY, credential)
.setApplicationName(APPLICATION_NAME).build();
return response;
}
private static void getProfileIds(Analytics analytics) throws IOException {
// Get the all view (profile) IDs for the authorized user.
List<String> profileIds = new ArrayList<>();
// Query for the list of all accounts associated with the service account.
Accounts accounts = analytics.management().accounts().list().execute();
if (accounts.getItems().isEmpty()) {
System.err.println("No accounts found");
} else {
for (Account account : accounts.getItems()) {
System.out.println("account: " + account.getName());
String accountId = account.getId();
// Query for the list of properties associated with the each account.
Webproperties properties = analytics.management().webproperties()
.list(accountId).execute();
if (properties.getItems().isEmpty()) {
System.err.println("No properties found for accountId: " + accountId);
} else {
for (Webproperty webproperty : properties.getItems()) {
System.out.println("\nwebproperty: " + webproperty.getName());
String webpropertyId = webproperty.getId();
// Query for the list views (profiles) associated with the property.
Profiles profiles = analytics.management().profiles()
.list(accountId, webpropertyId).execute();
if (profiles.getItems().isEmpty()) {
System.err.println("No views (profiles) found for accoundId: " + accountId + "and webpropertyId: " + webpropertyId);
} else {
// Return the first (view) profile associated with the property.
for (Profile profile : profiles.getItems()) {
System.out.println("\nprofileId added for profile: " + profile.getName());
profileIds.add(profile.getId());
printResults(getResults(analytics,profile.getId()), profile.getId());
}
}
System.out.println("---------- ---------- end webproperty: " + webproperty.getName() + "---------- ----------");
}
}
System.out.println("---------- ---------- end account: " + account.getName() + "---------- ----------");
}
}
}
private static GaData getResults(Analytics analytics, String profileId) throws IOException {
// Query the Core Reporting API for the number of sessions
// in the past 30 days.
GaData data = analytics.data().ga()
.get("ga:" + profileId, "30daysAgo", "yesterday", "ga:adClicks, ga:adCost, ga:transactions, ga:transactionRevenue, ga:users, ga:sessions")
.setDimensions("ga:adwordsCampaignID")
.execute();
return data;
}
private static void printResults(GaData results, String profile) {
// Parse the response from the Core Reporting API for
// the profile name and number of sessions.
if (null != results) {
System.out.println("View (Profile: " + profile + ") Name: "
+ results.getProfileInfo().getProfileName() + "\n");
if (results.get("rows") != null && results.get("columnHeaders") != null) {
if (!results.getRows().isEmpty() && !results.getColumnHeaders().isEmpty()) {
for (List<String> row : results.getRows()) {
for (int i=0; i<results.getColumnHeaders().size();i++) {
List<GaData.ColumnHeaders> headers = results.getColumnHeaders();
System.out.println( headers.get(i).getName()+": " + row.get(i) + "\n");
}
System.out.println("---------- ---------- ----------\n");
}
} else {
System.out.println("No rows or columHeaders empty\n");
}
} else {
System.out.println("No rows or columHeaders\n");
}
}
}
}

Using AWS Java's SDKs, how can I terminate the CloudFormation stack of the current instance?

Uses on-line decomentation I come up with the following code to terminate the current EC2 Instance:
public class Ec2Utility {
static private final String LOCAL_META_DATA_ENDPOINT = "http://169.254.169.254/latest/meta-data/";
static private final String LOCAL_INSTANCE_ID_SERVICE = "instance-id";
static public void terminateMe() throws Exception {
TerminateInstancesRequest terminateRequest = new TerminateInstancesRequest().withInstanceIds(getInstanceId());
AmazonEC2 ec2 = new AmazonEC2Client();
ec2.terminateInstances(terminateRequest);
}
static public String getInstanceId() throws Exception {
//SimpleRestClient, is an internal wrapper on http client.
SimpleRestClient client = new SimpleRestClient(LOCAL_META_DATA_ENDPOINT);
HttpResponse response = client.makeRequest(METHOD.GET, LOCAL_INSTANCE_ID_SERVICE);
return IOUtils.toString(response.getEntity().getContent(), "UTF-8");
}
}
My issue is that my EC2 instance is under an AutoScalingGroup which is under a CloudFormationStack, that is because of my organisation deployment standards though this single EC2 is all there is there for this feature.
So, I want to terminate the entire CloudFormationStack from the JavaSDK, keep in mind, I don't have the CloudFormation Stack Name in advance as I didn't have the EC2 Instance Id so I will have to get it from the code using the API calls.
How can I do that, if I can do it?
you should be able to use the deleteStack method from cloud formation sdk
DeleteStackRequest request = new DeleteStackRequest();
request.setStackName(<stack_name_to_be_deleted>);
AmazonCloudFormationClient client = new AmazonCloudFormationClient (<credentials>);
client.deleteStack(request);
If you don't have the stack name, you should be able to retrieve from the Tag of your instance
DescribeInstancesRequest request =new DescribeInstancesRequest();
request.setInstanceIds(instancesList);
DescribeInstancesResult disresult = ec2.describeInstances(request);
List <Reservation> list = disresult.getReservations();
for (Reservation res:list){
List <Instance> instancelist = res.getInstances();
for (Instance instance:instancelist){
List <Tag> tags = instance.getTags();
for (Tag tag:tags){
if (tag.getKey().equals("aws:cloudformation:stack-name")) {
tag.getValue(); // name of the stack
}
}
At the end I've achieved the desired behaviour using the set of the following util functions I wrote:
/**
* Delete the CloudFormationStack with the given name.
*
* #param stackName
* #throws Exception
*/
static public void deleteCloudFormationStack(String stackName) throws Exception {
AmazonCloudFormationClient client = new AmazonCloudFormationClient();
DeleteStackRequest deleteStackRequest = new DeleteStackRequest().withStackName("");
client.deleteStack(deleteStackRequest);
}
static public String getCloudFormationStackName() throws Exception {
AmazonEC2 ec2 = new AmazonEC2Client();
String instanceId = getInstanceId();
List<Tag> tags = getEc2Tags(ec2, instanceId);
for (Tag t : tags) {
if (t.getKey().equalsIgnoreCase(TAG_KEY_STACK_NAME)) {
return t.getValue();
}
}
throw new Exception("Couldn't find stack name for instanceId:" + instanceId);
}
static private List<Tag> getEc2Tags(AmazonEC2 ec2, String instanceId) throws Exception {
DescribeInstancesRequest describeInstancesRequest = new DescribeInstancesRequest().withInstanceIds(instanceId);
DescribeInstancesResult describeInstances = ec2.describeInstances(describeInstancesRequest);
List<Reservation> reservations = describeInstances.getReservations();
if (reservations.isEmpty()) {
throw new Exception("DescribeInstances didn't returned reservation for instanceId:" + instanceId);
}
List<Instance> instances = reservations.get(0).getInstances();
if (instances.isEmpty()) {
throw new Exception("DescribeInstances didn't returned instance for instanceId:" + instanceId);
}
return instances.get(0).getTags();
}
// XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
// Example of usage from the code:
deleteCloudFormationStack(getCloudFormationStackName());
// XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

Atlassian Confluence : how do I update page using REST API

I am trying to update a Confluence page using this code:
https://bitbucket.org/jaysee00/confluence-rest-api-example/src/master/src/main/java/com/atlassian/api/examples/Main.java
Code is:
public class Confluence {
/**
* Demonstrates how to update a page using the Conflunence 5.5 REST API.
*/
private static final Logger LOGGER = Logger.getLogger(Confluence.class);;
private static final String BASE_URL = "http://confluence:8080";
private static final String USERNAME = "admin";
private static final String PASSWORD = "admin";
private static final String ENCODING = "utf-8";
private String getContentRestUrl(Long contentId, String[] expansions)
throws UnsupportedEncodingException {
String expand = URLEncoder.encode(StringUtils.join(expansions, ","),
ENCODING);
return String
.format("%s/rest/api/content/%s?expand=%s&os_authType=basic&os_username=%s&os_password=%s",
BASE_URL, contentId, expand,
URLEncoder.encode(USERNAME, ENCODING),
URLEncoder.encode(PASSWORD, ENCODING));
}
public void publish() throws ClientProtocolException, IOException, Exception {
final long pageId = 36307446;
HttpClient client = new DefaultHttpClient();
// Get current page version
String pageObj = null;
HttpEntity pageEntity = null;
try {
String restUrl = getContentRestUrl(pageId,
new String[] { "body.storage", "version", "ancestors" });
HttpGet getPageRequest = new HttpGet(restUrl);
HttpResponse getPageResponse = client.execute(getPageRequest);
pageEntity = getPageResponse.getEntity();
pageObj = IOUtils.toString(pageEntity.getContent());
LOGGER.info("Get Page Request returned "
+ getPageResponse.getStatusLine().toString());
LOGGER.info(pageObj);
LOGGER.info((int)pageObj.trim().charAt(0));
} finally {
if (pageEntity != null) {
EntityUtils.consume(pageEntity);
}
}
// Parse response into JSON
JSONObject page = new JSONObject(pageObj.trim());
// Update page
// The updated value must be Confluence Storage Format
// NOT HTML.
page.getJSONObject("body").getJSONObject("storage")
.put("value", "hello, world");
int currentVersion = page.getJSONObject("version").getInt("number");
page.getJSONObject("version").put("number", currentVersion + 1);
// Send update request
HttpEntity putPageEntity = null;
try {
HttpPut putPageRequest = new HttpPut(getContentRestUrl(pageId,
new String[] {}));
StringEntity entity = new StringEntity(page.toString());
entity.setContentType("application/json");
putPageRequest.setEntity(entity);
HttpResponse putPageResponse = client.execute(putPageRequest);
putPageEntity = putPageResponse.getEntity();
System.out.println("Put Page Request returned "
+ putPageResponse.getStatusLine().toString());
System.out.println("");
System.out.println(IOUtils.toString(putPageEntity.getContent()));
} finally {
EntityUtils.consume(putPageEntity);
}
}
}
The response is alway 'HTTP 404 - Page not found'. I have changed the page id to one I know exists in Confluence.
An exception follows when it tries to parse the response into a JSON object:
avvvaorg.json.JSONException: A JSONObject text must begin with '{' at character 1
at org.json.JSONTokener.syntaxError(JSONTokener.java:496)
at org.json.JSONObject.<init>(JSONObject.java:180)
at org.json.JSONObject.<init>(JSONObject.java:403)
at com.openet.report.publish.Confluence.publish(Confluence.java:74)
at com.openet.report.miner.ReportMiner.generateSummary(ReportMiner.java:268)
at com.openet.report.miner.ReportMiner.runReport(ReportMiner.java:251)
at com.openet.report.miner.ReportMiner.main(ReportMiner.java:138)
Updating confluence pages using REST is not supported by Confluence 4.3.1. The API is much more limited:
https://docs.atlassian.com/atlassian-confluence/REST/4.3.1/
You can however update confluence using XML RPC:
public void publish() throws IOException {
DateFormat df = new SimpleDateFormat("MM/dd/yyyy HH:mm:ss");
Date today = Calendar.getInstance().getTime();
XWikiXmlRpcClient rpc = new XWikiXmlRpcClient(CONFLUENCE_URI);
try {
rpc.login(USER_NAME, PASSWORD);
//The info macro would get rendered an info box in the Page
Page page = new Page();
page.setSpace("Some space");
page.setTitle("Testing XML RPC calls in confluence_" + df.format(today));
//page.setContent(
String s = String.format("||Heading 1||Heading 2||Heading 3||%s|col A1|col A2|col A3|", "\r\n");
page.setContent(s);
page.setParentId(PAGEID);
rpc.storePage(page);
} catch (XmlRpcException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
// TODO Auto-generated catch block
}
This requires the following libraries:
import org.apache.xmlrpc.XmlRpcException;
import org.codehaus.swizzle.confluence.Page;
import org.w3c.dom.Document;
import org.xwiki.xmlrpc.XWikiXmlRpcClient;
Note that these libraries are not in the standard maven repository. You will have to update your repository manager (artifactory in my case) to sync with the XWiki maven repo. You will also need the service rocket plugin (https://community.servicerocket.com/servicerocket/topics/the-license-could-not-be-verified-there-is-no-license-certificate-installed-for-customware-scaffolding-plugin-for-confluence) configured correctly on Confluence.

Upload Directory with files to S3 using Java

I'm working on the application where user will upload ZIP file to my server, on the server that ZIP file will be expanded and then I need to upload it to the server. Now my questions is: how to upload directory with multiple files and sub-folders using Java to S3 bucket? Is there any examples for that? Currently i'm using JetS3t to manage all my communications with S3.
HI This is the simple way to upload the Directory into S3 bucket.
BasicAWSCredentials awsCreds = new BasicAWSCredentials(access_key_id,
secret_access_key);
AmazonS3 s3Client = new AmazonS3Client(awsCreds);
TransferManager tm = TransferManagerBuilder.standard().withS3Client(s3Client).build();
MultipleFileUpload upload = tm.uploadDirectory(existingBucketName,
"BuildNumber#1", "FilePathYouWant", true);
I built something very similar. After expanding the zip on the server call FileUtils.listFiles() which will recursively list files in a folder. Just iterate the list and create s3objects and upload the files to s3. Make use of the threadedstorage service so that multiple files can be uploaded at the same time. Also ensure you process the upload events. If some files couldn't be uploaded the jets3t library will tell you.
I could post the code I wrote once get into the office.
EDIT: CODE:
Here's the code:
private static ProviderCredentials credentials;
private static S3Service s3service;
private static ThreadedS3Service storageService;
private static S3Bucket bucket;
private List<S3Object> s3Objs=new ArrayList<S3Object>();
private Set<String> s3ObjsCompleted=new HashSet<String>();
private boolean isErrorOccured=true;
private final ByteFormatter byteFormatter = new ByteFormatter();
private final TimeFormatter timeFormatter = new TimeFormatter();
private void initialise() throws ServiceException, S3ServiceException {
credentials=<create your credentials>;
s3service = new RestS3Service(credentials);
bucket = new S3Bucket(<bucket details>);
storageService=new ThreadedS3Service(s3service, this);
}
}
private void uploadFolder(File folder) throws NoSuchAlgorithmException, IOException {
readFolderContents(folder);
uploadFilesInList(folder);
}
private void readFolderContents(File folder) throws NoSuchAlgorithmException, IOException {
Iterator<File> filesinFolder=FileUtils.iterateFiles(folder,null,null);
while(filesinFolder.hasNext()) {
File file=filesinFolder.next();
String key = <create your key from the filename or something>;
S3Object s3Obj=new S3Object(bucket, file);
s3Obj.setKey(key);
s3Obj.setContentType(Mimetypes.getInstance().getMimetype(s3Obj.getKey()));
s3Objs.add(s3Obj);
}
}
private void uploadFilesInList(File folder) {
logger.debug("Uploading files in folder "+folder.getAbsolutePath());
isErrorOccured=false;
s3ObjsCompleted.clear();
storageService.putObjects(bucket.getName(), s3Objs.toArray(new S3Object[s3Objs.size()]));
if(isErrorOccured || s3Objs.size()!=s3ObjsCompleted.size()) {
logger.debug("Have to try uploading a few objects again for folder "+folder.getAbsolutePath()+" - Completed = "+s3ObjsCompleted.size()+" and Total ="+s3Objs.size());
List<S3Object> s3ObjsRemaining=new ArrayList<S3Object>();
for(S3Object s3Obj : s3Objs) {
if(!s3ObjsCompleted.contains(s3Obj.getKey())) {
s3ObjsRemaining.add(s3Obj);
}
}
s3Objs=s3ObjsRemaining;
uploadFilesInList(folder);
}
}
#Override
public void event(CreateObjectsEvent event) {
super.event(event);
if (ServiceEvent.EVENT_IGNORED_ERRORS == event.getEventCode()) {
Throwable[] throwables = event.getIgnoredErrors();
for (int i = 0; i < throwables.length; i++) {
logger.error("Ignoring error: " + throwables[i].getMessage());
}
}else if(ServiceEvent.EVENT_STARTED == event.getEventCode()) {
logger.debug("**********************************Upload Event Started***********************************");
}else if(event.getEventCode()==ServiceEvent.EVENT_ERROR) {
isErrorOccured=true;
}else if(event.getEventCode()==ServiceEvent.EVENT_IN_PROGRESS) {
StorageObject[] storeObjs=event.getCreatedObjects();
for(StorageObject storeObj : storeObjs) {
s3ObjsCompleted.add(storeObj.getKey());
}
ThreadWatcher watcher = event.getThreadWatcher();
if (watcher.getBytesTransferred() >= watcher.getBytesTotal()) {
logger.debug("Upload Completed.. Verifying");
}else {
int percentage = (int) (((double) watcher.getBytesTransferred() / watcher.getBytesTotal()) * 100);
long bytesPerSecond = watcher.getBytesPerSecond();
StringBuilder transferDetailsText=new StringBuilder("Uploading.... ");
transferDetailsText.append("Speed: " + byteFormatter.formatByteSize(bytesPerSecond) + "/s");
if (watcher.isTimeRemainingAvailable()) {
long secondsRemaining = watcher.getTimeRemaining();
if (transferDetailsText.length() > 0) {
transferDetailsText.append(" - ");
}
transferDetailsText.append("Time remaining: " + timeFormatter.formatTime(secondsRemaining));
}
logger.debug(transferDetailsText.toString()+" "+percentage);
}
}else if(ServiceEvent.EVENT_COMPLETED==event.getEventCode()) {
logger.debug("**********************************Upload Event Completed***********************************");
if(isErrorOccured) {
logger.debug("**********************But with errors, have to retry failed uploads**************************");
}
}
}
Here is how I did it in December of 2021 since BasicAWSCredentials is deprecated now.
AWSCredentials = new BasicAWSCredentials(env.getProperty("AWS_ACCESS_KEY_ID"),
env.getProperty("AWS_SECRET_ACCESS_KEY"));
AmazonS3 s3Client = AmazonS3ClientBuilder.standard()
.withRegion(Regions.US_EAST_1).withCredentials(new AWSStaticCredentialsProvider(AWSCredentials))
.build();
TransferManager tm = TransferManagerBuilder.standard().withS3Client(s3Client).build();
MultipleFileUpload upload = tm.uploadDirectory(existingBucketName,
"BuildNumber#1", "FilePathYouWant", true);

Categories