We Keep Coding

How to get query result in Json form using Java API of Google Bigquery

I am using Google Bigquery V2 Java API. I am not able to find a way to get query results in JSON format.
In Bigquery Web UI we can see this JSON and Table form of results. see scrrenshot.
Is there any way to get the GetQueryResultsResponse as JSON, using Java API.

One option is to apply the TO_JSON_STRING function to the results of your query. For example,
#standardSQL
SELECT TO_JSON_STRING(t)
FROM (
SELECT x, y
FROM YourTable
WHERE z = 10
) AS t;
If you want all of the table's columns as JSON, you can use a simpler form:
#standardSQL
SELECT TO_JSON_STRING(t)
FROM YourTable AS t
WHERE z = 10;

I'm using a service account to access the BigQuery REST API to get the response in JSON format.
In order to use a service account, you will have to go to credentials (https://console.cloud.google.com/apis/credentials) and choose a project.
You will get a drop down like this:
Create a Service account for your project and download the secret file in the JSON format. Keep the JSON file in your file system and set the path to it. Check below image to set the file path:
So, now all you have to do in is use JAVA client api to consume the Big Query REST API.
Here's is a simple solution that I've been using for my project.
package com.example.bigquery;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.util.Arrays;
import org.apache.log4j.Logger;
import com.google.api.client.googleapis.auth.oauth2.GoogleCredential;
import com.google.api.client.http.GenericUrl;
import com.google.api.client.http.HttpContent;
import com.google.api.client.http.HttpHeaders;
import com.google.api.client.http.HttpRequest;
import com.google.api.client.http.HttpRequestFactory;
import com.google.api.client.http.HttpResponse;
import com.google.api.client.http.HttpTransport;
import com.google.api.client.http.javanet.NetHttpTransport;
import com.google.api.client.http.json.JsonHttpContent;
import com.google.api.client.json.JsonFactory;
import com.google.api.client.json.jackson2.JacksonFactory;
import com.google.common.io.CharStreams;
public class BigQueryDemo {
private static final String QUERY_URL_FORMAT = "https://www.googleapis.com/bigquery/v2/projects/%s/queries" + "?access_token=%s";
private static final String QUERY = "query";
private static final String QUERY_HACKER_NEWS_COMMENTS = "SELECT * FROM [bigquery-public-data:hacker_news.comments] LIMIT 1000";
private static final Logger logger = Logger.getLogger(BigQueryDemo.class);
static GoogleCredential credential = null;
static final HttpTransport HTTP_TRANSPORT = new NetHttpTransport();
static final JsonFactory JSON_FACTORY = new JacksonFactory();
static {
// Authenticate requests using Google Application Default credentials.
try {
credential = GoogleCredential.getApplicationDefault();
credential = credential.createScoped(Arrays.asList("https://www.googleapis.com/auth/bigquery"));
credential.refreshToken();
} catch (IOException e) {
e.printStackTrace();
}
}
public static void implicit() {
String projectId = credential.getServiceAccountProjectId();
String accessToken = generateAccessToken();
// Set the content of the request.
Dataset dataset = new Dataset().addLabel(QUERY, QUERY_HACKER_NEWS_COMMENTS);
HttpContent content = new JsonHttpContent(JSON_FACTORY, dataset.getLabels());
// Send the request to the BigQuery API.
GenericUrl url = new GenericUrl(String.format(QUERY_URL_FORMAT, projectId, accessToken));
logger.debug("URL: " + url.toString());
String responseJson = getQueryResult(content, url);
logger.debug(responseJson);
}
private static String getQueryResult(HttpContent content, GenericUrl url) {
String responseContent = null;
HttpRequestFactory requestFactory = HTTP_TRANSPORT.createRequestFactory();
HttpRequest request = null;
try {
request = requestFactory.buildPostRequest(url, content);
request.setParser(JSON_FACTORY.createJsonObjectParser());
request.setHeaders(
new HttpHeaders().set("X-HTTP-Method-Override", "POST").setContentType("application/json"));
HttpResponse response = request.execute();
InputStream is = response.getContent();
responseContent = CharStreams.toString(new InputStreamReader(is));
} catch (IOException e) {
logger.error(e);
}
return responseContent;
}
private static String generateAccessToken() {
String accessToken = null;
if ((System.currentTimeMillis() > credential.getExpirationTimeMilliseconds())) {
accessToken = credential.getRefreshToken();
} else {
accessToken = credential.getAccessToken();
}
System.out.println(accessToken);
return accessToken;
}
}
Following is the Github link to the code: https://github.com/vslala/BigQueryRestSample
It is just a demo project to fetch JSON data from the BQ REST API. Do not use it in your project directly.
Let me know if you have any questions.

ContentModel cannot be resolved to a variable

I'm having this error :
Exception in thread "main" java.lang.Error: Unresolved compilation problem:
ContentModel cannot be resolved to a variable
at test2CMIS.Test.main(Test.java:39)"
And I dont understand from where it comes, here is my code :
public class Test {
public static void main(String[] args){
Test atest = new Test();
Session session = atest.iniSession();
AuthenticationService authenticationService=null;
PersonService personService = null;
if (authenticationService.authenticationExists("test") == false)
{
authenticationService.createAuthentication("test", "changeMe".toCharArray());
PropertyMap ppOne = new PropertyMap(4);
ppOne.put(ContentModel.PROP_USERNAME, "test");
ppOne.put(ContentModel.PROP_FIRSTNAME, "firstName");
ppOne.put(ContentModel.PROP_LASTNAME, "lastName");
ppOne.put(ContentModel.PROP_EMAIL, "test"+"#example.com");
personService.createPerson(ppOne);
}
}
I did import the : import org.alfresco.model.ContentModel; and a lot of others librarys for my code.
Thx for help.
The code I'm using and I left some things that I tried too in comments so you can see what things I have done:
import java.io.File;
import java.io.Serializable;
import java.util.HashMap;
import java.util.Map;
import java.util.Scanner;
import org.alfresco.service.cmr.security.*;
import org.alfresco.error.AlfrescoRuntimeException;
import org.alfresco.model.ContentModel;
import java.util.Iterator;
import org.alfresco.repo.jscript.People;
import org.alfresco.repo.security.authentication.AuthenticationException;
import org.alfresco.service.cmr.security.AuthenticationService;
import org.alfresco.service.cmr.security.PersonService;
import org.alfresco.service.namespace.QName;
import org.alfresco.util.PropertyMap;
import org.apache.chemistry.opencmis.client.api.CmisObject;
import org.apache.chemistry.opencmis.client.api.Document;
import org.apache.chemistry.opencmis.client.api.Folder;
import org.apache.chemistry.opencmis.client.api.Session;
import org.apache.chemistry.opencmis.commons.PropertyIds;
import org.apache.chemistry.opencmis.commons.SessionParameter;
import org.apache.chemistry.opencmis.commons.enums.BindingType;
import org.apache.chemistry.opencmis.commons.enums.VersioningState;
import org.apache.chemistry.opencmis.commons.exceptions.CmisContentAlreadyExistsException;
import org.apache.chemistry.opencmis.commons.exceptions.CmisUnauthorizedException;
import org.apache.chemistry.opencmis.client.util.FileUtils;
import org.apache.chemistry.opencmis.client.runtime.SessionFactoryImpl;
public class Test {
public static void main(String[] args){
Test atest = new Test();
Session session = atest.iniSession();
AuthenticationService authenticationService=new AuthenticationServiceImpl();
PersonService personService = new PersonServiceImpl();
HashMap<QName, Serializable> properties = new HashMap<QName, Serializable>();
properties.put(ContentModel.PROP_USERNAME, "test");
properties.put(ContentModel.PROP_FIRSTNAME, "test");
properties.put(ContentModel.PROP_LASTNAME, "qsdqsd");
properties.put(ContentModel.PROP_EMAIL, "wshAlors#gmail.com");
properties.put(ContentModel.PROP_ENABLED, Boolean.valueOf(true));
properties.put(ContentModel.PROP_ACCOUNT_LOCKED, Boolean.valueOf(false));
personService.createPerson(properties);
authenticationService.createAuthentication("test", "changeme".toCharArray());
authenticationService.setAuthenticationEnabled("test", true);
authenticationService.getAuthenticationEnabled("Admin");
//String testAuthen = authenticationService.getCurrentTicket();
//System.out.println(testAuthen);
//QName username = QName.createQName("test");
//Map<QName,Serializable> propertiesUser = new HashMap<QName,Serializable>();
//propertiesUser.put(ContentModel.PROP_USERNAME,username);
//propertiesUser.put(ContentModel.PROP_FIRSTNAME,"test");
//propertiesUser.put(ContentModel.PROP_LASTNAME,"test");
//propertiesUser.put(ContentModel.PROP_EMAIL, "test#example.com");
//propertiesUser.put(ContentModel.PROP_PASSWORD,"0000");
//personService.createPerson(propertiesUser);
//if (authenticationService.authenticationExists("test") == false)
//{
// authenticationService.createAuthentication("test", "changeMe".toCharArray());
// PropertyMap ppOne = new PropertyMap(4);
// ppOne.put(ContentModel.PROP_USERNAME, "test");
// ppOne.put(ContentModel.PROP_FIRSTNAME, "test");
// ppOne.put(ContentModel.PROP_LASTNAME, "test");
// ppOne.put(ContentModel.PROP_EMAIL, "test#example.com");
//ppOne.put(ContentModel.PROP_JOBTITLE, "jobTitle");
// personService.createPerson(ppOne);
//}
}
public Session iniSession() {
Session session;
SessionFactoryImpl sf = SessionFactoryImpl.newInstance();
Map<String, String> parameters = new HashMap<String, String>();
Scanner reader = new Scanner(System.in);
System.out.println("Enter your logging : ");
String log = reader.nextLine();
System.out.println("Enter your password : ");
String pass = reader.nextLine();
parameters.put(SessionParameter.USER, log);
parameters.put(SessionParameter.PASSWORD, pass);
parameters.put(SessionParameter.BROWSER_URL, "http://127.0.0.1:8080/alfresco/api/-default-/public/cmis/versions/1.1/browser");
parameters.put(SessionParameter.BINDING_TYPE, BindingType.BROWSER.value());
parameters.put(SessionParameter.REPOSITORY_ID, "-default-");
try{
session = sf.createSession(parameters);
}catch(CmisUnauthorizedException cue){
session = null;
System.out.println("Wrong logging OR password !");
}
return session;
}

You are writing a runnable class which is not running the same process as Alfresco. In that sense, your class is running "remotely".
Because your class is running remotely to Alfresco, you are correct in using CMIS. But CMIS will only allow you to perform Create, Read, Update, and Delete (CRUD) functions against documents and folders in Alfresco. CMIS does not know how to create users or groups.
Your class will not be able to instantiate the AuthenticationService or PersonService. Those are part of the Alfresco Foundation API which only works when you are running in the same process as Alfresco, such as in an Action, a Behavior, or a Java-backed web script. In those cases, you will use Spring Dependency Injection to inject those services into your Java class. You would then put your class in a JAR that gets deployed into the Alfresco web application and loaded by the same classloader as Alfresco's.
If you want to create users remotely you should consider using the Alfresco REST API. Your runnable class can then use an HTTP client to invoke REST calls to create people and groups.

Thanks you for everything ! Thanks to you and researches I found out how to do it ! For the others who wonder how to do I'll post how I did and what site I used to understand it !
So you just need to manipulate JSON with Java because your alfresco people page (127.0.0.1:8080/alfresco/service/api/people) returns a JSON object, and you'll be able to create, delete, search... users! Thx again !
Sites :
https://api-explorer.alfresco.com/api-explorer/#/people
http://crunchify.com/json-manipulation-in-java-examples/
The code :
This is for creating an user :
public User createUser(String firstN, String lastN, String email, String pass, String authTicket) throws Exception{
try{
String url = "http://127.0.0.1:8080/alfresco/service/api/people?alf_ticket="+authTicket;
HttpClient httpclient = new HttpClient();
PostMethod mPost = new PostMethod(url);
//JSONObject obj = new JSONObject();
//JSONArray people = obj.getJSONArray("people");
JSONObject newUser = new JSONObject();
newUser.put("userName", firstN.toLowerCase().charAt(0)+lastN.toLowerCase());
newUser.put("enabled",true);
newUser.put("firstName",firstN);
newUser.put("lastName", lastN);
newUser.put("email", email);
newUser.put("quota",-1);
newUser.put("emailFreedDisable",false);
newUser.put("isDeleted",false);
newUser.put("isAdminAuthority",false);
newUser.put("password", pass);
//people.put(newUser);
//Response response = PostRequest(newUser.toString()));
StringRequestEntity requestEntity = new StringRequestEntity(
newUser.toString(),
"application/json",
"UTF-8");
mPost.setRequestEntity(requestEntity);
int statusCode2 = httpclient.executeMethod(mPost);
mPost.releaseConnection();
}catch(Exception e){
System.err.println("[ERROR] "+e);
}
return new User(firstN, lastN);
}
And if you want to get all the users you have on alfresco :
public ArrayList<User> getAllUsers(String authTicket)
{
ArrayList<User> allUsers = new ArrayList<>();
String lastName, firstName;
try{
String url = "http://127.0.0.1:8080/alfresco/service/api/people?alf_ticket="+authTicket;
HttpClient httpclient = new HttpClient();
GetMethod mPost = new GetMethod(url);
int statusCode1 = httpclient.executeMethod(mPost);
System.out.println("statusLine >>> "+statusCode1+"....."
+"\n status line \n"
+mPost.getStatusLine()+"\nbody \n"+mPost.getResponseBodyAsString());
JSONObject obj = new JSONObject(mPost.getResponseBodyAsString());
JSONArray people = obj.getJSONArray("people");
int n = people.length();
for(int i =0 ; i < n ; i++)
{
JSONObject peoples = people.getJSONObject(i);
User u = new User(peoples.getString("firstName"), peoples.getString("lastName"));
if (!allUsers.contains(u)){
allUsers.add(u);
}
}
}catch(Exception e){
System.err.println("[ERROR] "+e);
}
return(allUsers);
}

Custom Google search api parameters

I'm developing an application that uses Google search for specific keywords on specific domains. However, scraping Google result page and parsing results from HTML is not very "clean" approach, so I switched to Custom Google Search API. I managed to fetch results for a single keyword so far, but but I'm having trouble getting results for multiple keywords and multiple domains. In the settings of Custom Search I set it to do a whole web search, just to mention. So, my question is, how can search for multiple keywords on specific domains without setting it in developers console?
Thanks in advance.
P.S. This is current code:
package org.wonderland.dev.java;
import com.google.api.client.http.HttpTransport;
import com.google.api.client.http.javanet.NetHttpTransport;
import com.google.api.client.json.JsonFactory;
import com.google.api.client.json.jackson2.JacksonFactory;
import com.google.api.services.customsearch.Customsearch;
import com.google.api.services.customsearch.model.Result;
import com.google.api.services.customsearch.model.Search;
public class GoogleSearchClient {
final private String API_KEY = "AIzaSyDxxxxxxxxxxxxxxxxxx";
final private String SEARCH_ENGINE_ID = "00844xxxxxxxxxx:xxxxxxxx";
public void printResults() {
String searchKeyWord = "app";
List<Result> resultList = this.getSearchResult(searchKeyWord);
if(resultList != null && resultList.size() > 0){
for(Result result: resultList){
System.out.println(result.getTitle());
}
}
}
public List<Result> getSearchResult(String keyword){
HttpTransport httpTransport = new NetHttpTransport();
JsonFactory jsonFactory = new JacksonFactory();
Customsearch customsearch = new Customsearch(httpTransport, jsonFactory,null);
List<Result> resultList = null;
try {
Customsearch.Cse.List list = customsearch.cse().list(keyword);
list.setKey(API_KEY);
list.setCx(SEARCH_ENGINE_ID);
list.setNum(10L);
Search results = list.execute();
resultList = results.getItems();
} catch (Exception e) {
e.printStackTrace();
}
return resultList;
}
}

Webmaster Tools Api, get more then 1000 crawling errors

Im using the new webmaster tools api to get all my site's crawling errors (+ details). Unfort. it only gives me 1000 but i have like 10000. Is there a way to get all of them ?
This is the code i use:
package main;
import com.google.api.client.googleapis.auth.oauth2.GoogleAuthorizationCodeFlow;
import com.google.api.client.googleapis.auth.oauth2.GoogleCredential;
import com.google.api.client.googleapis.auth.oauth2.GoogleTokenResponse;
import com.google.api.client.http.HttpTransport;
import com.google.api.client.http.javanet.NetHttpTransport;
import com.google.api.client.json.JsonFactory;
import com.google.api.client.json.jackson2.JacksonFactory;
import com.google.api.services.webmasters.Webmasters;
import com.google.api.services.webmasters.Webmasters.Urlcrawlerrorssamples;
import com.google.api.services.webmasters.model.SitesListResponse;
import com.google.api.services.webmasters.model.UrlCrawlErrorsSample;
import com.google.api.services.webmasters.model.UrlCrawlErrorsSamplesListResponse;
import com.google.api.services.webmasters.model.WmxSite;
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.net.HttpURLConnection;
import java.net.URL;
import java.util.Arrays;
import java.util.ArrayList;
import java.util.Collection;
import java.util.List;
public class WebmastersCommandLine {
private static String CLIENT_ID = "...";
private static String CLIENT_SECRET = "...";
private static String REDIRECT_URI = "urn:ietf:wg:oauth:2.0:oob";
private static String OAUTH_SCOPE = "https://www.googleapis.com/auth/webmasters.readonly";
private static String PAGE_URL = "...";
public static void main(String[] args) throws IOException {
HttpTransport httpTransport = new NetHttpTransport();
JsonFactory jsonFactory = new JacksonFactory();
GoogleAuthorizationCodeFlow flow = new GoogleAuthorizationCodeFlow.Builder(
httpTransport, jsonFactory, CLIENT_ID, CLIENT_SECRET, Arrays.asList(OAUTH_SCOPE))
.setAccessType("online")
.setApprovalPrompt("auto").build();
String url = flow.newAuthorizationUrl().setRedirectUri(REDIRECT_URI).build();
System.out.println("open URL:");
System.out.println(" " + url);
System.out.println("code:");
BufferedReader br = new BufferedReader(new InputStreamReader(System.in));
String code = br.readLine();
GoogleTokenResponse response = flow.newTokenRequest(code).setRedirectUri(REDIRECT_URI).execute();
GoogleCredential credential = new GoogleCredential().setFromTokenResponse(response);
// Create a new authorized API client
Webmasters service = new Webmasters.Builder(httpTransport, jsonFactory, credential)
.setApplicationName("WebmastersCommandLine")
.build();
Webmasters.Urlcrawlerrorssamples.List req2 = service.urlcrawlerrorssamples().list(PAGE_URL, "notFound", "web");
try
{
UrlCrawlErrorsSamplesListResponse urlList = req2.execute();
System.out.println("start");
for(UrlCrawlErrorsSample sample : urlList.getUrlCrawlErrorSample())
{
Webmasters.Urlcrawlerrorssamples.Get req3 = service.urlcrawlerrorssamples().get(PAGE_URL, sample.getPageUrl(), "notFound", "web");
UrlCrawlErrorsSample details = req3.execute();
System.out.println(sample.getPageUrl() + "," + details.getUrlDetails().getLinkedFromUrls());
}
}
catch(IOException e)
{
System.out.println("An error occurred: " + e);
}
System.out.println("done");
}
}
This however only gives me a list of 1000 errors, but i need all 10000 of them. Does anybody know a way to do that ?

The Webmaster Tools API URL Crawl Errors Sample method returns a sample of 1000 crawl errors. It's not meant to return a complete list (you could compile that from your server logs). If you want more samples through the API, one thing you can do is to mark these errors as fixed and check back in a day. It will then generate a set of samples from the remaining crawl errors.
The order of the samples is the same as in the UI, so the more important ones will be the first ones you see. This means that there are diminishing returns as you move on, with later crawl errors being either similar to the previous ones, or at least seen as being less critical. The original blog post has more on the prioritization:
We determine this based on a multitude of factors, including whether
or not you included the URL in a Sitemap, how many places it’s linked
from (and if any of those are also on your site), and whether the URL
has gotten any traffic recently from search.

Google+ API Exception in thread "main" java.lang.NoClassDefFoundError: com/google/common/base/Preconditions

I am running the YouTubeSample given on the google developers website. I have no errors in the code and my imports appear to be fine. But when I run the project I get the aforementioned error.
I have done some searches but to be honest I have been unable to work out what the problem is. I have already tried importing an external jar guava but it didn't help.
Any help is appreciated. Here is the full class
package com.pengilleys.googlesamples;
import java.io.IOException;
import java.util.List;
import com.google.api.client.googleapis.GoogleHeaders;
import com.google.api.client.googleapis.json.JsonCParser;
import com.google.api.client.http.GenericUrl;
import com.google.api.client.http.HttpRequest;
import com.google.api.client.http.HttpRequestFactory;
import com.google.api.client.http.HttpRequestInitializer;
import com.google.api.client.http.HttpTransport;
import com.google.api.client.http.javanet.NetHttpTransport;
import com.google.api.client.json.JsonFactory;
import com.google.api.client.json.jackson.JacksonFactory;
import com.google.api.client.util.Key;
public class YouTubeSample {
public static class VideoFeed {
#Key List<Video> items;
}
public static class Video {
#Key String title;
#Key String description;
#Key Player player;
}
public static class Player {
#Key("default") String defaultUrl;
}
public static class YouTubeUrl extends GenericUrl {
#Key final String alt = "jsonc";
#Key String author;
#Key("max-results") Integer maxResults;
YouTubeUrl(String url) {
super(url);
}
}
public static void main(String[] args) throws IOException {
// set up the HTTP request factory
HttpTransport transport = new NetHttpTransport();
final JsonFactory jsonFactory = new JacksonFactory();
HttpRequestFactory factory = transport.createRequestFactory(new HttpRequestInitializer() {
#Override
public void initialize(HttpRequest request) {
// set the parser
JsonCParser parser = new JsonCParser();
parser.jsonFactory = jsonFactory;
request.addParser(parser);
// set up the Google headers
GoogleHeaders headers = new GoogleHeaders();
headers.setApplicationName("Google-YouTubeSample/1.0");
headers.gdataVersion = "2";
request.headers = headers;
}
});
// build the YouTube URL
YouTubeUrl url = new YouTubeUrl("https://gdata.youtube.com/feeds/api/videos");
url.author = "searchstories";
url.maxResults = 2;
// build the HTTP GET request
HttpRequest request = factory.buildGetRequest(url);
// execute the request and the parse video feed
VideoFeed feed = request.execute().parseAs(VideoFeed.class);
for (Video video : feed.items) {
System.out.println();
System.out.println("Video title: " + video.title);
System.out.println("Description: " + video.description);
System.out.println("Play URL: " + video.player.defaultUrl);
}
}
}

The setup documentation gives a list of dependencies:
Depending on the application you are building, you may also need these dependencies:
Apache HTTP Client version 4.0.3
Google Guava version r09
Jackson version 1.6.7
Google GSON version 1.6
In this case, it looks like it's Guava which is missing. I don't know what you mean about "exporting" Guava, but if you include the Guava r09 jar file in the classpath when you're running the code, it should be fine.

what's the extra ); for above the // build the YouTube URL and did you mean to close main on that line?