Does anyone know how to get a service ticket from the Key Distribution Center (KDC) using the Java GSS-API?
I have a thick-client-application that first authenticates via JAAS using the Krb5LoginModule to fetch the TGT from the ticket cache (background: Windows e.g. uses a kerberos implementation and stores the ticket granting ticket in a secure memory area). From the LoginManager I get the Subject object which contains the TGT. Now I hoped when I create a specific GSSCredential object for my service, the service ticket will be put into the Subject's private credentials as well (I've read so somewhere in the web). So I have tried the following:
// Exception handling ommitted
LoginContext lc = new LoginContext("HelloEjbClient", new DialogCallbackHandler());
lc.login()
Subject.doAs(lc.getSubject(), new PrivilegedAction() {
public Object run() {
GSSManager manager = GSSManager.getInstance();
GSSName clientName = manager.createName("clientUser", GSSName.NT_USER_NAME);
GSSCredential clientCreds = manager.createCredential(clientName, 8 * 3600, createKerberosOid(), GSSCredential.INITIATE_ONLY);
GSSName serverName = manager.createName("myService#localhost", GSSName.NT_HOSTBASED_SERVICE);
manager.createCredential(serverName, GSSCredential.INDEFINITE_LIFETIME, createKerberosOid(), GSSCredential.INITIATE_ONLY);
return null;
}
private Oid createKerberosOid() {
return new Oid("1.2.840.113554.1.2.2");
}
});
Unfortunately I get a GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt).
My understanding of getting the service ticket was wrong. I do not need to get the credentials from the service - this is not possible on the client, because the client really doesn't have a TGT for the server and therefore doesn't have the rights to get the service credentials.
What's just missing here is to create a new GSSContext and to initialize it. The return value from this method contains the service ticket, if I have understood that correctly. Here is a working code example. It must be run in a PrivilegedAction on behalf of a logged in subject:
GSSManager manager = GSSManager.getInstance();
GSSName clientName = manager.createName("clientUser", GSSName.NT_USER_NAME);
GSSCredential clientCred = manager.createCredential(clientName,
8 * 3600,
createKerberosOid(),
GSSCredential.INITIATE_ONLY);
GSSName serverName = manager.createName("http#server", GSSName.NT_HOSTBASED_SERVICE);
GSSContext context = manager.createContext(serverName,
createKerberosOid(),
clientCred,
GSSContext.DEFAULT_LIFETIME);
context.requestMutualAuth(true);
context.requestConf(false);
context.requestInteg(true);
byte[] outToken = context.initSecContext(new byte[0], 0, 0);
System.out.println(new BASE64Encoder().encode(outToken));
context.dispose();
The outToken contains then contains the Service Ticket. However this is not the way the GSS-API was meant to be used. Its goal was to hide those details to the code, so it is better to establish a GSSContext using the GSS-API on both sides. Otherwise you really should know what you are doing because of potential security holes.
For more information read the Sun SSO tutorial with kerberos more carefully than I did.
EDIT:
Just forgot that I am using Windows XP with SP2. There is a new "feature" in this version of Windows that disallows using the TGT in the Windows RAM. You have to edit the registry to allow this. For more information have a look at the JGSS Troubleshooting page topic in case you experience a "KrbException: KDC has no support for encryption type (14)" like I did.
I had a lot of problems to use this code, but I have at least a solution. I post it here, perhaps it will help some of you...
/**
* Tool to retrieve a kerberos ticket. This one will not be stored in the windows ticket cache.
*/
public final class KerberosTicketRetriever
{
private final static Oid KERB_V5_OID;
private final static Oid KRB5_PRINCIPAL_NAME_OID;
static {
try
{
KERB_V5_OID = new Oid("1.2.840.113554.1.2.2");
KRB5_PRINCIPAL_NAME_OID = new Oid("1.2.840.113554.1.2.2.1");
} catch (final GSSException ex)
{
throw new Error(ex);
}
}
/**
* Not to be instanciated
*/
private KerberosTicketRetriever() {};
/**
*
*/
private static class TicketCreatorAction implements PrivilegedAction
{
final String userPrincipal;
final String applicationPrincipal;
private StringBuffer outputBuffer;
/**
*
* #param userPrincipal p.ex. <tt>MuelleHA#MYFIRM.COM</tt>
* #param applicationPrincipal p.ex. <tt>HTTP/webserver.myfirm.com</tt>
*/
private TicketCreatorAction(final String userPrincipal, final String applicationPrincipal)
{
this.userPrincipal = userPrincipal;
this.applicationPrincipal = applicationPrincipal;
}
private void setOutputBuffer(final StringBuffer newOutputBuffer)
{
outputBuffer = newOutputBuffer;
}
/**
* Only calls {#link #createTicket()}
* #return <tt>null</tt>
*/
public Object run()
{
try
{
createTicket();
}
catch (final GSSException ex)
{
throw new Error(ex);
}
return null;
}
/**
*
* #throws GSSException
*/
private void createTicket () throws GSSException
{
final GSSManager manager = GSSManager.getInstance();
final GSSName clientName = manager.createName(userPrincipal, KRB5_PRINCIPAL_NAME_OID);
final GSSCredential clientCred = manager.createCredential(clientName,
8 * 3600,
KERB_V5_OID,
GSSCredential.INITIATE_ONLY);
final GSSName serverName = manager.createName(applicationPrincipal, KRB5_PRINCIPAL_NAME_OID);
final GSSContext context = manager.createContext(serverName,
KERB_V5_OID,
clientCred,
GSSContext.DEFAULT_LIFETIME);
context.requestMutualAuth(true);
context.requestConf(false);
context.requestInteg(true);
final byte[] outToken = context.initSecContext(new byte[0], 0, 0);
if (outputBuffer !=null)
{
outputBuffer.append(String.format("Src Name: %s\n", context.getSrcName()));
outputBuffer.append(String.format("Target : %s\n", context.getTargName()));
outputBuffer.append(new BASE64Encoder().encode(outToken));
outputBuffer.append("\n");
}
context.dispose();
}
}
/**
*
* #param realm p.ex. <tt>MYFIRM.COM</tt>
* #param kdc p.ex. <tt>kerbserver.myfirm.com</tt>
* #param applicationPrincipal cf. {#link #TicketCreatorAction(String, String)}
* #throws GSSException
* #throws LoginException
*/
static public String retrieveTicket(
final String realm,
final String kdc,
final String applicationPrincipal)
throws GSSException, LoginException
{
// create the jass-config-file
final File jaasConfFile;
try
{
jaasConfFile = File.createTempFile("jaas.conf", null);
final PrintStream bos = new PrintStream(new FileOutputStream(jaasConfFile));
bos.print(String.format(
"Krb5LoginContext { com.sun.security.auth.module.Krb5LoginModule required refreshKrb5Config=true useTicketCache=true debug=true ; };"
));
bos.close();
jaasConfFile.deleteOnExit();
}
catch (final IOException ex)
{
throw new IOError(ex);
}
// set the properties
System.setProperty("java.security.krb5.realm", realm);
System.setProperty("java.security.krb5.kdc", kdc);
System.setProperty("java.security.auth.login.config",jaasConfFile.getAbsolutePath());
// get the Subject(), i.e. the current user under Windows
final Subject subject = new Subject();
final LoginContext lc = new LoginContext("Krb5LoginContext", subject, new DialogCallbackHandler());
lc.login();
// extract our principal
final Set<Principal> principalSet = subject.getPrincipals();
if (principalSet.size() != 1)
throw new AssertionError("No or several principals: " + principalSet);
final Principal userPrincipal = principalSet.iterator().next();
// now try to execute the SampleAction as the authenticated Subject
// action.run() without doAsPrivileged leads to
// No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
final TicketCreatorAction action = new TicketCreatorAction(userPrincipal.getName(), applicationPrincipal);
final StringBuffer outputBuffer = new StringBuffer();
action.setOutputBuffer(outputBuffer);
Subject.doAsPrivileged(lc.getSubject(), action, null);
return outputBuffer.toString();
}
public static void main (final String args[]) throws Throwable
{
final String ticket = retrieveTicket("MYFIRM.COM", "kerbserver", "HTTP/webserver.myfirm.com");
System.out.println(ticket);
}
}
Related
I am testing android managment API using localhost as call back url. I followed each and every step following this url Android Management API Sample.
Now i m stuck on place.. according to this guide, i download the json file from service account. Now i copy that json file and save in app folder of my project.
This is my enterprise.json file
Screenshot of json file in android studio
and i just give folder location as enterprise.json in location string
This is my code
private static final String PROJECT_ID = "enterprise-271814";
private static final String SERVICE_ACCOUNT_CREDENTIAL_FILE =
"enterprise.json";
private static final String POLICY_ID = "samplePolicy";
/** The package name of the COSU app. */
private static final String COSU_APP_PACKAGE_NAME =
"com.ariaware.devicepoliceycontroller";
/** The OAuth scope for the Android Management API. */
private static final String OAUTH_SCOPE =
"https://www.googleapis.com/auth/androidmanagement";
private static final String APP_NAME = "Device Policey Controller";
private final AndroidManagement androidManagementClient;
public Sample(AndroidManagement androidManagementClient) {
this.androidManagementClient = androidManagementClient;
}
public void run() throws IOException {
// Create an enterprise. If you've already created an enterprise, the
// createEnterprise call can be commented out and replaced with your
// enterprise name.
String enterpriseName = createEnterprise();
System.out.println("Enterprise created with name: " + enterpriseName);
// Set the policy to be used by the device.
setPolicy(enterpriseName, POLICY_ID, getCosuPolicy());
// Create an enrollment token to enroll the device.
String token = createEnrollmentToken(enterpriseName, POLICY_ID);
System.out.println("Enrollment token (to be typed on device): " + token);
// List some of the devices for the enterprise. There will be no devices for
// a newly created enterprise, but you can run the app again with an
// existing enterprise after enrolling a device.
List<Device> devices = listDevices(enterpriseName);
for (Device device : devices) {
System.out.println("Found device with name: " + device.getName());
}
// If there are any devices, reboot one.
if (devices.isEmpty()) {
System.out.println("No devices found.");
} else {
rebootDevice(devices.get(0));
}
}
public static AndroidManagement getAndroidManagementClient()
throws IOException, GeneralSecurityException {
try (FileInputStream input =
new FileInputStream(SERVICE_ACCOUNT_CREDENTIAL_FILE)) {
GoogleCredential credential =
GoogleCredential.fromStream(input)
.createScoped(Collections.singleton(OAUTH_SCOPE));
return new AndroidManagement.Builder(
GoogleNetHttpTransport.newTrustedTransport(),
JacksonFactory.getDefaultInstance(),
credential)
.setApplicationName(APP_NAME)
.build();
}
}
private String createEnterprise() throws IOException {
// Initiate signup process.
System.out.println("Creating signup URL...");
SignupUrl signupUrl =
androidManagementClient
.signupUrls()
.create()
.setProjectId(PROJECT_ID)
.setCallbackUrl("https://localhost:9999")
.execute();
System.out.print(
"To sign up for a new enterprise, open this URL in your browser: ");
System.out.println(signupUrl.getUrl());
System.out.println(
"After signup, you will see an error page in the browser.");
System.out.print(
"Paste the enterpriseToken value from the error page URL here: ");
String enterpriseToken =
new BufferedReader(new InputStreamReader(System.in)).readLine();
// Create the enterprise.
System.out.println("Creating enterprise...");
return androidManagementClient
.enterprises()
.create(new Enterprise())
.setProjectId(PROJECT_ID)
.setSignupUrlName(signupUrl.getName())
.setEnterpriseToken(enterpriseToken)
.execute()
.getName();
}
private Policy getCosuPolicy() {
List<String> categories = new ArrayList<>();
categories.add("android.intent.category.HOME");
categories.add("android.intent.category.DEFAULT");
return new Policy()
.setApplications(
Collections.singletonList(
new ApplicationPolicy()
.setPackageName(COSU_APP_PACKAGE_NAME)
.setInstallType("FORCE_INSTALLED")
.setDefaultPermissionPolicy("GRANT")
.setLockTaskAllowed(true)))
.setPersistentPreferredActivities(
Collections.singletonList(
new PersistentPreferredActivity()
.setReceiverActivity(COSU_APP_PACKAGE_NAME)
.setActions(
Collections.singletonList("android.intent.action.MAIN"))
.setCategories(categories)))
.setKeyguardDisabled(true)
.setStatusBarDisabled(true);
}
private void setPolicy(String enterpriseName, String policyId, Policy policy)
throws IOException {
System.out.println("Setting policy...");
String name = enterpriseName + "/policies/" + policyId;
androidManagementClient
.enterprises()
.policies()
.patch(name, policy)
.execute();
}
private String createEnrollmentToken(String enterpriseName, String policyId)
throws IOException {
System.out.println("Creating enrollment token...");
EnrollmentToken token =
new EnrollmentToken().setPolicyName(policyId).setDuration("86400s");
return androidManagementClient
.enterprises()
.enrollmentTokens()
.create(enterpriseName, token)
.execute()
.getValue();
}
private List<Device> listDevices(String enterpriseName) throws IOException {
System.out.println("Listing devices...");
ListDevicesResponse response =
androidManagementClient
.enterprises()
.devices()
.list(enterpriseName)
.execute();
return response.getDevices() ==null
? new ArrayList<Device>() : response.getDevices();
}
private void rebootDevice(Device device) throws IOException {
System.out.println(
"Sending reboot command to " + device.getName() + "...");
Command command = new Command().setType("REBOOT");
androidManagementClient
.enterprises()
.devices()
.issueCommand(device.getName(), command)
.execute();
}
Moreover i m using android management api for the first time and i dont know its proper implementation. Anyone who has experience on this kinllt guide me a little bit. I found a lot about this but i didn't found any userful tutorial
For Android, you have to store the service account file either in the assets folder or raw folder.
This thread provides code on a number of ways to load the json data into an InputStream depending on the location you selected.
am trying to read user profile/image form Microsoft graphic and am using adal4j-1.5.0.jar to generate the azure token so that based on token i can make a call to graphic API/Microsoft delve.
i am facing issue in below code. it was simple moving to finally block after below line without generating token or any exception.
"Future future = context.acquireToken(resourceUri, credential, null);"
String clientId = "clientid";
String clientSecret = "cleintsecret";
String resourceUri = "https://graph.microsoft.com/v1.0/me";
String redirectUri = "http://localhost:9082/contextroot";
String authorityUri ="https://login.microsoftonline.com/{tenent id}/oauth2/authorize";
AuthenticationContext context = null;
AuthenticationResult result = null;
ExecutorService service = null;
try {
service = Executors.newFixedThreadPool(1);
context = new AuthenticationContext(authorityUri, false, service);
ClientCredential credential = new ClientCredential(clientId,clientSecret);
Future<AuthenticationResult> future = context.acquireToken(resourceUri, credential, null);
result = future.get();
}
finally {
service.shutdown();
}
It seems that some dependencies of adal4j-1.5.0 are not download as well, please check the jar package files from your project. Based on my test if I use the adal4j 1.5.0.
I found that are some dependencies of adal4j-1.5.0 are missing from the project. Then I can't get the access token.
But If I use the adal4j 1.0.0, it works correctly for me. If the version 1.0.0 is acceptable, you could use that as a workaround or add the dependencies manually.
Test demo code:
private static final String APP_ID = "clientId";
private static final String APP_SECRET = "secret key";
private static final String TENATID = "xxxxx";
public static void main(String[] args) throws Exception {
String authority = "https://login.microsoftonline.com/"+TENATID;
String resourceUrl = "https://graph.microsoft.com"; //Microsoft graph. AD graph: https://graph.windows.net
ExecutorService service = Executors.newFixedThreadPool(1);
AuthenticationContext context = new AuthenticationContext(authority, true, service);
// Acquire Token
Future<AuthenticationResult> result = context.acquireToken(
resourceUrl,
new ClientCredential(APP_ID, APP_SECRET),
null
);
String token = result.get().getAccessToken();
System.out.println(token);
}
Uses on-line decomentation I come up with the following code to terminate the current EC2 Instance:
public class Ec2Utility {
static private final String LOCAL_META_DATA_ENDPOINT = "http://169.254.169.254/latest/meta-data/";
static private final String LOCAL_INSTANCE_ID_SERVICE = "instance-id";
static public void terminateMe() throws Exception {
TerminateInstancesRequest terminateRequest = new TerminateInstancesRequest().withInstanceIds(getInstanceId());
AmazonEC2 ec2 = new AmazonEC2Client();
ec2.terminateInstances(terminateRequest);
}
static public String getInstanceId() throws Exception {
//SimpleRestClient, is an internal wrapper on http client.
SimpleRestClient client = new SimpleRestClient(LOCAL_META_DATA_ENDPOINT);
HttpResponse response = client.makeRequest(METHOD.GET, LOCAL_INSTANCE_ID_SERVICE);
return IOUtils.toString(response.getEntity().getContent(), "UTF-8");
}
}
My issue is that my EC2 instance is under an AutoScalingGroup which is under a CloudFormationStack, that is because of my organisation deployment standards though this single EC2 is all there is there for this feature.
So, I want to terminate the entire CloudFormationStack from the JavaSDK, keep in mind, I don't have the CloudFormation Stack Name in advance as I didn't have the EC2 Instance Id so I will have to get it from the code using the API calls.
How can I do that, if I can do it?
you should be able to use the deleteStack method from cloud formation sdk
DeleteStackRequest request = new DeleteStackRequest();
request.setStackName(<stack_name_to_be_deleted>);
AmazonCloudFormationClient client = new AmazonCloudFormationClient (<credentials>);
client.deleteStack(request);
If you don't have the stack name, you should be able to retrieve from the Tag of your instance
DescribeInstancesRequest request =new DescribeInstancesRequest();
request.setInstanceIds(instancesList);
DescribeInstancesResult disresult = ec2.describeInstances(request);
List <Reservation> list = disresult.getReservations();
for (Reservation res:list){
List <Instance> instancelist = res.getInstances();
for (Instance instance:instancelist){
List <Tag> tags = instance.getTags();
for (Tag tag:tags){
if (tag.getKey().equals("aws:cloudformation:stack-name")) {
tag.getValue(); // name of the stack
}
}
At the end I've achieved the desired behaviour using the set of the following util functions I wrote:
/**
* Delete the CloudFormationStack with the given name.
*
* #param stackName
* #throws Exception
*/
static public void deleteCloudFormationStack(String stackName) throws Exception {
AmazonCloudFormationClient client = new AmazonCloudFormationClient();
DeleteStackRequest deleteStackRequest = new DeleteStackRequest().withStackName("");
client.deleteStack(deleteStackRequest);
}
static public String getCloudFormationStackName() throws Exception {
AmazonEC2 ec2 = new AmazonEC2Client();
String instanceId = getInstanceId();
List<Tag> tags = getEc2Tags(ec2, instanceId);
for (Tag t : tags) {
if (t.getKey().equalsIgnoreCase(TAG_KEY_STACK_NAME)) {
return t.getValue();
}
}
throw new Exception("Couldn't find stack name for instanceId:" + instanceId);
}
static private List<Tag> getEc2Tags(AmazonEC2 ec2, String instanceId) throws Exception {
DescribeInstancesRequest describeInstancesRequest = new DescribeInstancesRequest().withInstanceIds(instanceId);
DescribeInstancesResult describeInstances = ec2.describeInstances(describeInstancesRequest);
List<Reservation> reservations = describeInstances.getReservations();
if (reservations.isEmpty()) {
throw new Exception("DescribeInstances didn't returned reservation for instanceId:" + instanceId);
}
List<Instance> instances = reservations.get(0).getInstances();
if (instances.isEmpty()) {
throw new Exception("DescribeInstances didn't returned instance for instanceId:" + instanceId);
}
return instances.get(0).getTags();
}
// XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
// Example of usage from the code:
deleteCloudFormationStack(getCloudFormationStackName());
// XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
I take example from GitHub.
It works fine, but I need to upload obb file with my apk.
Part of my code:
AndroidPublisher.Edits.Insert editRequest = edits.insert(bundleID, null);
AppEdit edit = editRequest.execute();
final String editId = edit.getId();
final AbstractInputStreamContent apkFile = new FileContent("application/vnd.android.package-archive", new File(apkPath));
AndroidPublisher.Edits.Apks.Upload uploadRequest = edits.apks().upload(bundleID, editId, apkFile);
Apk apk = uploadRequest.execute();
List<Integer> apkVersionCodes = new ArrayList<>();
apkVersionCodes.add(apk.getVersionCode());
AndroidPublisher.Edits.Tracks.Update updateTrackRequest = edits.tracks().update(bundleID, editId, TRACK_ALPHA, new Track().setVersionCodes(apkVersionCodes));
updateTrackRequest.execute();
// Commit changes for edit.
AndroidPublisher.Edits.Commit commitRequest = edits.commit(bundleID, editId);
commitRequest.execute();
How can I do it?
To upload a new expansion file and associate it with an APK:
String expansionFileType = "main"; // either "main" or "patch"
FileContent file = new FileContent("application/octet-stream", new File(path));
edits.expansionfiles().upload(appId, editId, versionCode, expansionFileType, file).execute();
If you want to associate an existing expansion file with an APK:
String expansionFileType = "main";
int previousVersionCode = // find the version code here
ExpansionFile obbFile = new ExpansionFile().setReferencesVersion(previousVersionCode);
edits.expansionfiles().update(appId, editId, versionCode, expansionFileType, obbFile).execute();
thanks to Zynger and the origin guy whoever developed this https://github.com/googlesamples/android-play-publisher-api/tree/master/v2/java
package com.google.play.developerapi.samples;
/**
* Contains global application configuration, which is required by all samples.
*/
public final class ApplicationConfig {
private ApplicationConfig() {
// no instance
}
/**
* Specify the name of your application. If the application name is
* {#code null} or blank, the application will log a warning. Suggested
* format is "MyCompany-Application/1.0".
*/
static final String APPLICATION_NAME = "Test 1234";
/**
* Specify the package name of the app.
*/
static final String PACKAGE_NAME = "my.app";
/**
* Authentication.
* <p>
* Installed application: Leave this string empty and copy or
* edit resources/client_secrets.json.
* </p>
* <p>
* Service accounts: Enter the service
* account email and add your key.p12 file to the resources directory.
* </p>
*/
static final String SERVICE_ACCOUNT_EMAIL = "newaccount#YOUR_SERIVE_ACCOUNT.gserviceaccount.com";
/**
* Specify the apk file path of the apk to upload, i.e. /resources/your_apk.apk
* <p>
* This needs to be set for running {#link BasicUploadApk} and {#link UploadApkWithListing}
* samples.
* </p>
*/
public static final String APK_FILE_PATH = "/resources/x7.apk";
public static final String OBB_FILE_PATH = "/resources/x7.main.obb";
}
public static void main(String[] args) {
try {
Preconditions.checkArgument(!Strings.isNullOrEmpty(ApplicationConfig.PACKAGE_NAME),
"ApplicationConfig.PACKAGE_NAME cannot be null or empty!");
// Create the API service.
AndroidPublisher service = AndroidPublisherHelper.init(ApplicationConfig.APPLICATION_NAME, ApplicationConfig.SERVICE_ACCOUNT_EMAIL);
final Edits edits = service.edits();
// Create a new edit to make changes to your listing.
Insert editRequest = edits.insert(ApplicationConfig.PACKAGE_NAME, null /** no content */);
AppEdit edit = editRequest.execute();
final String editId = edit.getId();
log.info(String.format("Created edit with id: %s", editId));
File f = new File(ApplicationConfig.APK_FILE_PATH);
if(f.exists() && !f.isDirectory()) {
System.out.println("Exist ");
}
// Upload new apk to developer console
final String apkPath = BasicUploadApk.class.getResource(ApplicationConfig.APK_FILE_PATH).toURI().getPath();
final String obbPath = BasicUploadApk.class.getResource(ApplicationConfig.OBB_FILE_PATH).toURI().getPath();
final AbstractInputStreamContent apkFile = new FileContent(AndroidPublisherHelper.MIME_TYPE_APK, new File(apkPath));
Upload uploadRequest = edits.apks().upload(ApplicationConfig.PACKAGE_NAME,editId,apkFile);
final AbstractInputStreamContent obbFile = new FileContent(AndroidPublisherHelper.MIME_TYPE_OBB, new File(obbPath));
Apk apk = uploadRequest.execute();
log.info(String.format("Version code %d has been uploaded",apk.getVersionCode()));
List<Integer> apkVersionCodes = new ArrayList<>();
apkVersionCodes.add(apk.getVersionCode());
Update updateTrackRequest = edits.tracks().update(ApplicationConfig.PACKAGE_NAME,editId,TRACK_ALPHA, new Track().setVersionCodes(apkVersionCodes));
String expansionFileType = "main"; // either "main" or "patch"
FileContent file = new FileContent("application/octet-stream", new File(ApplicationConfig.OBB_FILE_PATH));
edits.expansionfiles().upload(ApplicationConfig.PACKAGE_NAME, editId, apk.getVersionCode(), expansionFileType, obbFile).execute();
Track updatedTrack = updateTrackRequest.execute();
log.info(String.format("Track %s has been updated.", updatedTrack.getTrack()));
// Commit changes for edit.
Commit commitRequest = edits.commit(ApplicationConfig.PACKAGE_NAME, editId);
AppEdit appEdit = commitRequest.execute();
log.info(String.format("App edit with id %s has been comitted", appEdit.getId()));
} catch (IOException | URISyntaxException | GeneralSecurityException ex) {
log.error("Excpetion was thrown while uploading apk to rollout track", ex);
}
}
I want to upload data from Google Cloud Storage to table in Big Query. There is my code to create job:
public class LoadStorageToBigQuery {
// ///////////////////////
// USER GENERATED VALUES: you must fill in values specific to your
// application.
//
// Visit the Google API Console to create a Project and generate an
// OAuth 2.0 Client ID and Secret (http://code.google.com/apis/console).
// Then, add the Project ID below, and update the clientsecrets.json file
// with your client_id and client_secret
//
// ///////////////////////
private static final String PROJECT_ID = "gavd.com:compute";
private static final String CLIENTSECRETS_LOCATION = "/client_secrets.json";
private static final String RESOURCE_PATH =
("E:/Work On/ads/Cloud/Dev/Source/BigQueryDemo02" + CLIENTSECRETS_LOCATION).replace(
'/', File.separatorChar);
static GoogleClientSecrets clientSecrets = loadClientSecrets();
// Static variables for API scope, callback URI, and HTTP/JSON functions
private static final List<String> SCOPES = Arrays
.asList("https://www.googleapis.com/auth/bigquery");
private static final String REDIRECT_URI = "urn:ietf:wg:oauth:2.0:oob";
private static final HttpTransport TRANSPORT = new NetHttpTransport();
private static final JsonFactory JSON_FACTORY = new JacksonFactory();
private static GoogleAuthorizationCodeFlow flow = null;
/**
* #param args
* #throws IOException
* #throws InterruptedException
*/
public static void main(String[] args) throws IOException,
InterruptedException {
System.out.println(CLIENTSECRETS_LOCATION);
// Create a new BigQuery client authorized via OAuth 2.0 protocol
Bigquery bigquery = createAuthorizedClient();
// Print out available datasets to the console
listDatasets(bigquery, "publicdata");
JobReference jobId = startQuery(bigquery, PROJECT_ID);
System.out.println("Job ID = " + jobId);
JobReference jobRef = startQuery(bigquery, PROJECT_ID);
checkQueryResults(bigquery, PROJECT_ID, jobRef);
}
/**
* Creates an authorized BigQuery client service using the OAuth 2.0
* protocol
*
* This method first creates a BigQuery authorization URL, then prompts the
* user to visit this URL in a web browser to authorize access. The
* application will wait for the user to paste the resulting authorization
* code at the command line prompt.
*
* #return an authorized BigQuery client
* #throws IOException
*/
public static Bigquery createAuthorizedClient() throws IOException {
String authorizeUrl = new GoogleAuthorizationCodeRequestUrl(
clientSecrets, REDIRECT_URI, SCOPES).setState("").build();
System.out
.println("Paste this URL into a web browser to authorize BigQuery Access:\n"
+ authorizeUrl);
System.out.println("... and type the code you received here: ");
BufferedReader in = new BufferedReader(new InputStreamReader(System.in));
String authorizationCode = in.readLine();
// Exchange the auth code for an access token and refesh token
Credential credential = exchangeCode(authorizationCode);
return Bigquery.builder(TRANSPORT, JSON_FACTORY)
.setHttpRequestInitializer(credential)
.setApplicationName("Your User Agent Here").build();
}
/**
* Display all BigQuery Datasets associated with a Project
*
* #param bigquery
* an authorized BigQuery client
* #param projectId
* a string containing the current project ID
* #throws IOException
*/
public static void listDatasets(Bigquery bigquery, String projectId)
throws IOException {
Datasets.List datasetRequest = bigquery.datasets().list(projectId);
DatasetList datasetList = datasetRequest.execute();
if (datasetList.getDatasets() != null) {
List<DatasetList.Datasets> datasets = datasetList.getDatasets();
System.out.println("Available datasets\n----------------");
System.out.println( " B = " + datasets.toString());
for (DatasetList.Datasets dataset : datasets) {
System.out.format("%s\n", dataset.getDatasetReference()
.getDatasetId());
}
}
}
public static JobReference startQuery(Bigquery bigquery, String projectId) throws IOException {
Job job = new Job();
JobConfiguration config = new JobConfiguration();
JobConfigurationLoad loadConfig = new JobConfigurationLoad();
config.setLoad(loadConfig);
job.setConfiguration(config);
// Set where you are importing from (i.e. the Google Cloud Storage paths).
List<String> sources = new ArrayList<String>();
sources.add("gs://gms_cloud_project/bigquery_data/06_13_2014/namesbystate.csv");
loadConfig.setSourceUris(sources);
//state:STRING,sex:STRING,year:INTEGER,name:STRING,occurrence:INTEGER
// Describe the resulting table you are importing to:
TableReference tableRef = new TableReference();
tableRef.setDatasetId("gimasys_database");
tableRef.setTableId("table_test");
tableRef.setProjectId(projectId);
loadConfig.setDestinationTable(tableRef);
List<TableFieldSchema> fields = new ArrayList<TableFieldSchema>();
TableFieldSchema fieldState = new TableFieldSchema();
fieldState.setName("state");
fieldState.setType("STRING");
TableFieldSchema fieldSex = new TableFieldSchema();
fieldSex.setName("sex");
fieldSex.setType("STRING");
TableFieldSchema fieldName = new TableFieldSchema();
fieldName.setName("name");
fieldName.setType("STRING");
TableFieldSchema fieldYear = new TableFieldSchema();
fieldYear.setName("year");
fieldYear.setType("INTEGER");
TableFieldSchema fieldOccur = new TableFieldSchema();
fieldOccur.setName("occurrence");
fieldOccur.setType("INTEGER");
fields.add(fieldState);
fields.add(fieldSex);
fields.add(fieldName);
fields.add(fieldYear);
fields.add(fieldOccur);
TableSchema schema = new TableSchema();
schema.setFields(fields);
loadConfig.setSchema(schema);
// Also set custom delimiter or header rows to skip here....
// [not shown].
Insert insert = bigquery.jobs().insert(projectId, job);
insert.setProjectId(projectId);
JobReference jobRef = insert.execute().getJobReference();
// ... see rest of codelab for waiting for job to complete.
return jobRef;
//return jobId;
}
/**
* Polls the status of a BigQuery job, returns Job reference if "Done"
*
* #param bigquery
* an authorized BigQuery client
* #param projectId
* a string containing the current project ID
* #param jobId
* a reference to an inserted query Job
* #return a reference to the completed Job
* #throws IOException
* #throws InterruptedException
*/
private static Job checkQueryResults(Bigquery bigquery, String projectId,
JobReference jobId) throws IOException, InterruptedException {
// Variables to keep track of total query time
long startTime = System.currentTimeMillis();
long elapsedTime;
while (true) {
Job pollJob = bigquery.jobs().get(projectId, jobId.getJobId())
.execute();
elapsedTime = System.currentTimeMillis() - startTime;
System.out.format("Job status (%dms) %s: %s\n", elapsedTime,
jobId.getJobId(), pollJob.getStatus().getState());
if (pollJob.getStatus().getState().equals("DONE")) {
return pollJob;
}
// Pause execution for one second before polling job status again,
// to
// reduce unnecessary calls to the BigQUery API and lower overall
// application bandwidth.
Thread.sleep(1000);
}
}
/**
* Helper to load client ID/Secret from file.
*
* #return a GoogleClientSecrets object based on a clientsecrets.json
*/
private static GoogleClientSecrets loadClientSecrets() {
try {
System.out.println("A");
System.out.println(CLIENTSECRETS_LOCATION);
GoogleClientSecrets clientSecrets = GoogleClientSecrets.load(new JacksonFactory(),
new FileInputStream(new File(
RESOURCE_PATH)));
return clientSecrets;
} catch (Exception e) {
System.out.println("Could not load file Client_Screts");
e.printStackTrace();
}
return clientSecrets;
}
/**
* Exchange the authorization code for OAuth 2.0 credentials.
*
* #return an authorized Google Auth flow
*/
static Credential exchangeCode(String authorizationCode) throws IOException {
GoogleAuthorizationCodeFlow flow = getFlow();
GoogleTokenResponse response = flow.newTokenRequest(authorizationCode)
.setRedirectUri(REDIRECT_URI).execute();
return flow.createAndStoreCredential(response, null);
}
/**
* Build an authorization flow and store it as a static class attribute.
*
* #return a Google Auth flow object
*/
static GoogleAuthorizationCodeFlow getFlow() {
if (flow == null) {
HttpTransport httpTransport = new NetHttpTransport();
JacksonFactory jsonFactory = new JacksonFactory();
flow = new GoogleAuthorizationCodeFlow.Builder(httpTransport,
jsonFactory, clientSecrets, SCOPES)
.setAccessType("offline").setApprovalPrompt("force")
.build();
}
return flow;
}
}
Status informed "DONE" in creating table but when checking my project in console.gooogle.com
Currently, we also don't see any error or issue or exception but it's not upload any data to my table (Table size 0B). I tried to create table without any data and than to upload data to this table but it isn't.
Job ID = {"jobId":"job_MqfuhuAU1Ms0GIOSbiePFGlc6TE","projectId":"ads.com:compute"}
Job status (451ms) job_fOtciwR1pfytbkeMaQ9RvvH18qc: PENDING
Job status (2561ms) job_fOtciwR1pfytbkeMaQ9RvvH18qc: PENDING
Job status (6812ms) job_fOtciwR1pfytbkeMaQ9RvvH18qc: PENDING
Job status (8273ms) job_fOtciwR1pfytbkeMaQ9RvvH18qc: PENDING
Job status (9695ms) job_fOtciwR1pfytbkeMaQ9RvvH18qc: PENDING
Job status (11146ms) job_fOtciwR1pfytbkeMaQ9RvvH18qc: PENDING
Job status (12466ms) job_fOtciwR1pfytbkeMaQ9RvvH18qc: PENDING
Job status (13948ms) job_fOtciwR1pfytbkeMaQ9RvvH18qc: PENDING
Job status (15392ms) job_fOtciwR1pfytbkeMaQ9RvvH18qc: PENDING
Job status (16796ms) job_fOtciwR1pfytbkeMaQ9RvvH18qc: PENDING
Job status (18296ms) job_fOtciwR1pfytbkeMaQ9RvvH18qc: RUNNING
Job status (19755ms) job_fOtciwR1pfytbkeMaQ9RvvH18qc: RUNNING
Job status (21587ms) job_fOtciwR1pfytbkeMaQ9RvvH18qc: DONE
I would appreciate any help,
Thanks
pollJob.getStatus().getState().equals("DONE") would tell you when the job is done but it does not give you the exit code.
You should be checking errorresult explicitly pollJob.getStatus().getErrorResult();
while (true) {
Job pollJob = getBigQuery().jobs().get(projectId, jobId.getJobId()).execute();
elapsedTime = System.currentTimeMillis() - startTime;
if (pollJob.getStatus().getErrorResult() != null) {
// The job ended with an error.
System.out.format("Job %s ended with error %s", jobId.getJobId(),pollJob.getStatus().getErrorResult().getMessage(), projectId);
throw new RuntimeException(String.format("Job %s ended with error %s", jobId.getJobId(),
pollJob.getStatus().getErrorResult().getMessage()));
}
System.out.format("Job status (%dms) %s: %s\n", elapsedTime,
jobId.getJobId(), pollJob.getStatus().getState());
if (pollJob.getStatus().getState().equals("DONE")) {
break;
}
Thread.sleep(5000);
}
So the problem seems to be the code asks for "string,string,string,integer,integer", but the provided data comes as "string, string,integer,string,integer" - hence BigQuery can't transform the string in column 4 to an integer.
To get the exact error message run bq show -j job_yourjobid.