We Keep Coding

How to properly iterate Bigquery TableResult in Java

I am trying to iterate the rows from TableResult using getValues() as below.
if I use getValues(), it's retrieving only the first page rows. I want to iterate all the rows using getValues() and NOT using iterateAll().
In the below code, the problem is its going infinite time. not ending. while(results.hasNextPage()) is not ending. what is the problem in the below code?
{
query = "select from aa.bb.cc";
QueryJobConfiguration queryConfig =
QueryJobConfiguration.newBuilder(query)
.setPriority(QueryJobConfiguration.Priority.BATCH)
.build();
TableResult results = bigquery.query(queryConfig);
int i = 0;
int j=0;
while(results.hasNextPage()) {
j++;
System.out.println("page " + j);
System.out.println("Data Extracted::" + i + " records");
for (FieldValueList row : results.getNextPage().getValues()) {
i++;
}
}
System.out.println("Total Count::" + results.getTotalRows());
System.out.println("Data Extracted::" + i + " records");
}
I have only 200,000 records in the source table. below is the out put and I forcefully stopped the process.
page 1
Data Extracted::0 records
page 2
Data Extracted::85242 records
page 3
Data Extracted::170484 records
page 4
Data Extracted::255726 records
page 5
Data Extracted::340968 records
page 6
Data Extracted::426210 records
page 7
Data Extracted::511452 records
page 8
Data Extracted::596694 records
.......
.......
.......
.......

In short, you need to update TableResults variable with your getNextPage() variable. If you don't update it you will always be looping the same results over and over. Thats why you are getting tons of records in your output.
If you check the following samples: Bigquery Pagination and Using Java Client Library. There are ways that we can deal with pagination results. Although not specific for single run queries.
As show on the code below, which is partially based on pagination sample, you need to use the output of getNextPage() to update results variable and proceed to perform the next iteration inside the while up until it iterates all pages but the last.
QueryRun.Java
package com.projects;
// [START bigquery_query]
import com.google.cloud.bigquery.BigQuery;
import com.google.cloud.bigquery.BigQueryException;
import com.google.cloud.bigquery.BigQueryOptions;
import com.google.cloud.bigquery.QueryJobConfiguration;
import com.google.cloud.bigquery.TableResult;
import com.google.cloud.bigquery.Job;
import com.google.cloud.bigquery.JobId;
import com.google.cloud.bigquery.FieldValueList;
import com.google.cloud.bigquery.JobInfo;
import com.google.cloud.bigquery.BigQuery.QueryResultsOption;
import java.util.UUID;
import sun.jvm.hotspot.debugger.Page;
public class QueryRun {
public static void main(String[] args) {
String projectId = "bigquery-public-data";
String datasetName = "covid19_ecdc_eu";
String tableName = "covid_19_geographic_distribution_worldwide";
String query =
"SELECT * "
+ " FROM `"
+ projectId
+ "."
+ datasetName
+ "."
+ tableName
+ "`"
+ " LIMIT 100";
System.out.println(query);
query(query);
}
public static void query(String query) {
try {
BigQuery bigquery = BigQueryOptions.getDefaultInstance().getService();
QueryJobConfiguration queryConfig = QueryJobConfiguration.newBuilder(query).build();
// Create a job ID so that we can safely retry.
JobId jobId = JobId.of(UUID.randomUUID().toString());
Job queryJob = bigquery.create(JobInfo.newBuilder(queryConfig).setJobId(jobId).build());
TableResult results = queryJob.getQueryResults(QueryResultsOption.pageSize(10));
int i = 0;
int j =0;
// get all paged data except last line
while(results.hasNextPage()) {
j++;
for (FieldValueList row : results.getValues()) {
i++;
}
results = results.getNextPage();
print_msg(i,j);
}
// last line run
j++;
for (FieldValueList row : results.getValues()) {
i++;
}
print_msg(i,j);
System.out.println("Query performed successfully.");
} catch (BigQueryException | InterruptedException e) {
System.out.println("Query not performed \n" + e.toString());
}
}
public static void print_msg(int i,int j)
{
System.out.println("page " + j);
System.out.println("Data Extracted::" + i + " records");
}
}
// [END bigquery_query]
output:
SELECT * FROM `bigquery-public-data.covid19_ecdc_eu.covid_19_geographic_distribution_worldwide` LIMIT 100
page 1
Data Extracted::10 records
page 2
Data Extracted::20 records
page 3
Data Extracted::30 records
page 4
Data Extracted::40 records
page 5
Data Extracted::50 records
page 6
Data Extracted::60 records
page 7
Data Extracted::70 records
page 8
Data Extracted::80 records
page 9
Data Extracted::90 records
page 10
Data Extracted::100 records
Query performed successfully.
As a final note, there are not official sample about pagination for queries so I'm not totally sure of the recommended way to handle pagination with java. Its not quite clear on the BigQuery for Java documentation page. If you can update your question with your approach to pagination I would appreciate.
If you have issues running the attached sample please see Using the BigQuery Java client sample, its github page and its pom.xml file inside of it and check if you are in compliance with it.

Probably I Am late in the response.
But reading on the java client guide (https://cloud.google.com/bigquery/docs/quickstarts/quickstart-client-libraries#complete_source_code)
it says that:
Iterate over the QueryResponse to get all the rows in the results. The iterator automatically handles pagination. Each FieldList exposes the columns by numeric index or column name.
Sai that, it should be easier to simply use the iterateAll() method.
Let me know if I Am wrong.

VMWare java SDK: When doesn't an available PerfMetricID report Data?

I'm trying to use vmware sdk for java to collect the perfomance data of each entity (cluster/datastore/Host/VM) in the vmware environment.
The idea is to get the available PerfMetricIds for the target entity with queryAvailablePerfMetric, query those and report the details of the counter, the timestamp and the value.
However when I get the PerfMetricIds for an entity, not every detected (returned) PerfMetricId is reporting data. For example for each Datastore I get at least 4 ids which do not return data when queried, these IDs represent the counters associated with the average number of read and write operations, and for a cluster I'm missing the cpu usage, and so on ...
so I was wondering when does this happen? Shouldn't every metric returned by queryAvailablePerfMetric report data? what am I missing here?
Minimal code snippet:
// VMWare credentials
String vmwareUrl = args[0];
String vmwareUsername = args[1];
String vmwarePassword = args[2];
// connect to vCenter
ServiceInstance si = new ServiceInstance(new URL(vmwareUrl), vmwareUsername, vmwarePassword, true);
// get performance manager
PerformanceManager perfMgr = si.getPerformanceManager();
// define the time window (the last one hour)
Calendar calTo = Calendar.getInstance();
Calendar calFrom = Calendar.getInstance();
calFrom.setTime(calTo.getTime());
calFrom.add(Calendar.HOUR, -1);
// get any datastore for testing purposes
Folder rootFolder = si.getRootFolder();
ManagedEntity[] datastores = new InventoryNavigator(rootFolder).searchManagedEntities("Datastore");
ManagedEntity me = datastores[1];
// query all available metrics for the entity
PerfMetricId[] availablePmis = perfMgr.queryAvailablePerfMetric(me, calFrom, calTo, perfMgr.getHistoricalInterval()[0].getSamplingPeriod());
// create PerfQuerySpec
PerfQuerySpec qSpec = new PerfQuerySpec();
qSpec.setEntity(me.getMOR());
qSpec.setMetricId(availablePmis);
qSpec.setFormat("csv");
qSpec.setStartTime(calFrom);
qSpec.setEndTime(calTo);
// query perf
PerfEntityMetricBase[] perfValues = perfMgr.queryPerf(new PerfQuerySpec[]{qSpec});
// Printing
System.out.println("Found pmis (CounterIDs only): ");
for (PerfMetricId pmi : availablePmis){
System.out.print(pmi.getCounterId() + ", ");
}
System.out.print("\nPmis with values:");
int pmisCount=0;
for (PerfEntityMetricBase value : perfValues) {
PerfMetricSeriesCSV[] csvValues = ((PerfEntityMetricCSV) value).getValue();
pmisCount += csvValues.length;;
for (PerfMetricSeriesCSV csv : csvValues) {
System.out.println("Counter ID: " + csv.getId().getCounterId() + " ---- Metric instance: " + csv.getId().getInstance());
System.out.println("\tInfo: " + ((PerfEntityMetricCSV) value).getSampleInfoCSV());
System.out.println("\tValues: " + csv.getValue());
}
}
System.out.println("---------------");
System.out.println("Detected PMIs: " + availablePmis.length);
System.out.println("PMIs with values: " + pmisCount);
Any help (or discussions) would be appreciated

How to promptly get workitem history with tfs java sdk?( with out workitem.open())

for reasons, i must use tfs java sdk, so i always need workitem.open() to load revisions for each workitem from workitemclient.Query(),like below code:
WorkItemCollection workItemCollection = workItemClient.query(wiql);
for(int i = 0; i < workItemCollection.size(); i++)
{
WorkItem workItem = workItems.get(i);
workItem.open();
workItem.getRevisions()
.forEach(r -> {
//..."System.History").getValue());
//..."System.History").getOriginalValue());
});
}
if don't use open(), Revisions must be empty;
that open() method cost 55 millseconds for each workitem,that's unacceptable。
so any suggestion for better efficiency ？ Thanks a lot。
/////////////////////////////////////////////////////////////////////make it/////////////////////////////////////////////////////////////////////
i make it that cut the time.
use SQL like blow :
String sql = "select [System.Id],[System.ChangedBy],[System.ChangedDate],[words]"
+ " from Tfs_DefaultCollection.dbo.WorkItemsEverable"
+ " left join Tfs_DefaultCollection.dbo.WorkItemLongTexts on [System.Id] = [ID] and [System.Rev] = [Rev]"
+ " where FldID=54" // behalf History field
+ " and"+conditionSql;
conditionSql : " id in ('12','1324','1')"
now use this method, average cost mills for retrieve history(changedby,changedDate,History) just 2~4 mills（test case is 30 ~ 1000 workitems）
/////////////////////////////////////////////////////////////////////still/////////////////////////////////////////////////////////////////////
any advise for better code/efficiency would be appreciate。

The revision history can only be retrieved for work item fields available in the collection WorkItem.Fields, so if you used the below code to get the workitem revision history you will NOT see the history but end up reloading the current workitem object again and again.
foreach (Revision revision in wi.Revisions)
{
Debug.Write(revision.WorkItem);
}
You may try to get value of fields in the work item revision:
// Returns a work item object
var wi = GetWorkItemDetails(id);
// Get All work item revisions
foreach (Revision revision in wi.Revisions)
{
// Get value of fields in the work item revision
foreach (Field field in wi.Fields)
{
Debug.Write(revision.Fields[field.Name].Value);
}
}
Helpful blog: http://geekswithblogs.net/TarunArora/archive/2011/08/21/tfs-sdk-work-item-history-visualizer-using-tfs-api.aspx

How to get client list using SOAP Web Services in Netsuite ERP?

I am really new to SOAP web services and to Netsuite ERP and I am trying to generate a report in my company where I need to obtain all the Clients and their Invoices using the data available in Netsuite ERP. I followed the Java and Axis tutorial they offer with their sample app for the ERP and I successfully created a Java project in Eclipse that consumes the WSDL for netsuite 2015-2 and compiles the needed classes to run the sample app. So, I followed an example found in their CRM exapmle app to obtain a Client's information but the only problem is that their example method needs you to introduce the Client's ID. Here is the sample code:
public int getCustomerList() throws RemoteException,
ExceededUsageLimitFault, UnexpectedErrorFault, InvalidSessionFault,
ExceededRecordCountFault, UnsupportedEncodingException {
// This operation requires a valid session
this.login(true);
// Prompt for list of internalIds and put in an array
_console
.write("\ninternalIds for records to retrieved (separated by commas): ");
String reqKeys = _console.readLn();
String[] internalIds = reqKeys.split(",");
return getCustomerList(internalIds, false);
}
private int getCustomerList(String[] internalIds, boolean isExternal)
throws RemoteException, ExceededUsageLimitFault,
UnexpectedErrorFault, InvalidSessionFault, ExceededRecordCountFault {
// Build an array of RecordRef objects and invoke the getList()
// operation to retrieve these records
RecordRef[] recordRefs = new RecordRef[internalIds.length];
for (int i = 0; i < internalIds.length; i++) {
RecordRef recordRef = new RecordRef();
recordRef.setInternalId(internalIds[i]);
recordRefs[i] = recordRef;
recordRefs[i].setType(RecordType.customer);
}
// Invoke getList() operation
ReadResponseList getResponseList = _port.getList(recordRefs);
// Process response from get() operation
if (!isExternal)
_console.info("\nRecords returned from getList() operation: \n");
int numRecords = 0;
ReadResponse[] getResponses = getResponseList.getReadResponse();
for (int i = 0; i < getResponses.length; i++) {
_console.info("\n Record[" + i + "]: ");
if (!getResponses[i].getStatus().isIsSuccess()) {
_console.errorForRecord(getStatusDetails(getResponses[i]
.getStatus()));
} else {
numRecords++;
Customer customer = (Customer) getResponses[i].getRecord();
_console.info(" internalId="
+ customer.getInternalId()
+ "\n entityId="
+ customer.getEntityId()
+ (customer.getCompanyName() == null ? ""
: ("\n companyName=" + customer
.getCompanyName()))
+ (customer.getEntityStatus() == null ? ""
: ("\n status=" + customer.getEntityStatus().getName()))
+ (customer.getEmail() == null ? ""
: ("\n email=" + customer.getEmail()))
+ (customer.getPhone() == null ? ""
: ("\n phone=" + customer.getPhone()))
+ "\n isInactive="
+ customer.getIsInactive()
+ (customer.getDateCreated() != null ? ""
: ("\n dateCreated=" + customer
.getDateCreated().toString())));
}
}
return numRecords;
}
So as you can see, this method needs the internal ID of each Customer which I find not useful as I have a many Customers and I don't want to pass each Customer's ID. I read their API docs (which I find hard to navigate and kind of useless) and I found a web service called getAll() that gives all the records given a getAllRecord object which requires a getAllRecordType object. However, the getAllRecordType object does not support Customer entities, so I can't obtain all the customers on the ERP this way.
Is there an easy way to obtain all the Customers in my Netsuite ERP (maybe using other thing rather than the SOAP Web Services they offer? I am desperate about this situation as understanding how Netsuite's Web Services API has been really troublesome.
Thanks!

You would normally use a search to select a list of customers. On a large account you would not normally get all customers on any regular basis. If you are trying to get the invoices you might just find it more practical to get those with a search.
You wrote "in your company". Are you trying to write an application of some sort? If this is an internal project (and even if it's not) you'll probably find using SuiteScripts much more efficient in terms of your time and frustration level.

I made it using the following code on my getCustomerList method:
CustomerSearch customerSrch = new CustomerSearch();
SearchResult searchResult = _port.search(customerSrch);
System.out.println(searchResult.getTotalRecords());
RecordList rl = searchResult.getRecordList();
for (int i = 0; i <searchResult.getTotalRecords()-1; i++) {
Record r = rl.getRecord(i);
System.out.println("Customer # " + i);
Customer testcust = (Customer)r;
System.out.println("First Name: " + testcust.getFirstName());
}

How to Insert time in mysql using java

I'm doing a project to catering item hiring service. They want to save the date and time customer hire their items also save the date and time customer return them. Of course any of those not gona be actual at the movement date or time. So first of all I'm trying to find to save any date and time. Other activities will concern later. So I came up with little table. like given below.
ID | Name | Value | Date | Time
(int) | (Varchar) |( Double) | (DateTime) | (DateTime)
And I used this cord to save data.
private void jButton2ActionPerformed(java.awt.event.ActionEvent evt)
{
String date=new SimpleDateFormat("yyyy-MM-dd").format(dat.getDate());
String time = String.valueOf(dateSpinner.getValue()).split(" ")[3];
try {
new JDBC().putData("INSERT INTO work (name, balance, date) VALUES ('"+txtName.getText()+"',"+txtValue.getText()+"', '"+date+"', '"+time+"') ");
JOptionPane.showMessageDialog(null, "value saved");
} catch (Exception e) {
JOptionPane.showMessageDialog(null, this.getClass().getName() +" "+e);
}
}
When I'm entering data I'm having
Exception in thread "AWT-EventQueue-0" java.lang.ArrayIndexOutOfBoundsException: 3" .
Help me to find the error of this cord and save any time. Actually I can save any date using String Date variable. I saved some demo data before I added time field to table. But I'm having this problem when I'm trying to save time. I am using date Spinner to save time. But I can't format it as HH-mm--ss. If you know another time saving feature please let me know. Please help me.

This line:
String time = String.valueOf(dateSpinner.getValue()).split(" ")[3];
Is the same as this, just done on a single line
String[] splitArr = String.valueOf(dateSpinner.getValue()).split(" ");
String time = splitArr[3];
The error you are getting is saying that there is not 4 elements in the split array. Put in the below piece of code to debug what you are getting so you know what is happening in that single line
String[] splitArr = String.valueOf(dateSpinner.getValue()).split(" ");
for(int i = 0; i < splitArr.length; i++)
{
System.out.println("Element number " + i + " is \"" + splitArr[i] + "\"");
}
Then you can update your insert appropriately. Sorry I cant be of more help but we do not know what datespinner is or what datespinner.getValue() returns so do the above to help debug your code.