We Keep Coding

Best way to get BigQuery temp table created by Job to read large data faster

I am trying to execute a query over a table in BigQuery using its Java client libraries. I create a Job and then get the result of Job using job.getQueryResults().iterateAll() method.
This way is working but for large data like 600k it takes time around 80-120 seconds. I see BigQuery gets data in 40-45k batches which takes around 5-7 sec each.
I want to get the results faster and I found over internet that if we can get the temporary table created by BigQuery from the Job and the read the data in avro or some other format from that table if will be really fast, but in BigQuery API(using version: 1.124.7) I don't see that way.
Does anyone know how to do that in Java, or how to get data faster in case of large number of records.
Any help is appreciated.
Code to Read Table(Takes 20 sec)
Table table = bigQueryHelper.getBigQueryClient().getTable(TableId.of("project","dataset","table"));
String format = "CSV";
String gcsUrl = "gs://name/test.csv";
Job job = table.extract(format, gcsUrl);
// Wait for the job to complete
try {
Job completedJob = job.waitFor(RetryOption.initialRetryDelay(Duration.ofSeconds(1)),
RetryOption.totalTimeout(Duration.ofMinutes(3)));
if (completedJob != null && completedJob.getStatus().getError() == null) {
log.info("job done");
// Job completed successfully
} else {
log.info("job has error");
// Handle error case
}
} catch (InterruptedException e) {
// Handle interrupted wait
}
Code to read same table using Query(Takes 90 Sec)
Job job = bigQueryHelper.getBigQueryClient().getJob(JobId.of(jobId));
for (FieldValueList row : job.getQueryResults().iterateAll()) {
System.out.println(row);
}

I tried certain ways and based on that found the best way of doing it, just thought to post here to help some one in future.
1: If we use job.getQueryResults().iterateAll() on job or directly on table, it takes same time. So if we don't give batch size BigQuery will use batch size of around 35-45k and fetch the data. So for 600k rows (180Mb) it takes 70-100 sec.
2: We can use the temp table details from created job and use extract job feature of table to write the result in GCS, this will be faster and takes around 30-35 sec. This approach would not download on local for that we again need to use ..iterateAll() on temp table and it will be take same time as 1.
Example pseudo code:
try {
Job job = getBigQueryClient().getJob(JobId.of(jobId));
long start = System.currentTimeMillis();
// FieldList list = getFields(job);
Job completedJob =
job.waitFor(
RetryOption.initialRetryDelay(Duration.ofSeconds(1)),
RetryOption.totalTimeout(Duration.ofMinutes(3)));
if (completedJob != null && completedJob.getStatus().getError() == null) {
log.info("job done");
String gcsUrl = "gs://bucketname/test";
//getting the temp table information of the Job
TableId destinationTableInfo =
((QueryJobConfiguration) job.getConfiguration()).getDestinationTable();
log.info("Total time taken in getting schema ::{}", (System.currentTimeMillis() - start));
Table table = bigQueryHelper.getBigQueryClient().getTable(destinationTableInfo);
//Using extract job to write the data in GCS
Job newJob1 =
table.extract(
CsvOptions.newBuilder().setFieldDelimiter("\t").build().toString(), gcsUrl);
System.out.println("DestinationInfo::" + destinationTableInfo);
Job completedJob1 =
newJob1.waitFor(
RetryOption.initialRetryDelay(Duration.ofSeconds(1)),
RetryOption.totalTimeout(Duration.ofMinutes(3)));
if (completedJob1 != null && completedJob1.getStatus().getError() == null) {
log.info("job done");
} else {
log.info("job has error");
}
} else {
log.info("job has error");
}
} catch (InterruptedException e) {
e.printStackTrace();
}
3: This is the best way which I wanted. It downloads/writes the result faster in local file. It downloads data in around 20 sec. This is the new way BigQuery provides and can be checked using below links:
https://cloud.google.com/bigquery/docs/reference/storage#background
List item
https://cloud.google.com/bigquery/docs/reference/storage/libraries#client-libraries-install-java

How do I (only) update the timestamp of an OPC UA Node using an Eclipse Milo OpcUa Server?

Thanks for reading this post. I am gratefull for every help!
Goal:
I am trying to write a ValueNode with the same value, but different timestamp on an Eclipse Milo OpcUa Server. I'm doing all this with Java 11 and Milo 0.3.7.
I tested it among others with the example in the github project. I modified the function addScalarNodes in the class ExampleNamespace to write the same value with an updated timestamp once a second. Checking the node with UaExpert, the timestamp remains on the same value. Only if I update the value too, the timestamp updates.
private void addScalarNodes(UaFolderNode rootNode) {
UaFolderNode scalarTypesFolder = new UaFolderNode(
getNodeContext(),
newNodeId("HelloWorld/ScalarTypes"),
newQualifiedName("ScalarTypes"),
LocalizedText.english("ScalarTypes")
);
getNodeManager().addNode(scalarTypesFolder);
rootNode.addOrganizes(scalarTypesFolder);
for (Object[] os : STATIC_SCALAR_NODES) {
String name = (String) os[0];
NodeId typeId = (NodeId) os[1];
Variant variant = (Variant) os[2];
UaVariableNode node = new UaVariableNode.UaVariableNodeBuilder(getNodeContext())
.setNodeId(newNodeId("HelloWorld/ScalarTypes/" + name))
.setAccessLevel(ubyte(AccessLevel.getMask(AccessLevel.READ_WRITE)))
.setUserAccessLevel(ubyte(AccessLevel.getMask(AccessLevel.READ_WRITE)))
.setBrowseName(newQualifiedName(name))
.setDisplayName(LocalizedText.english(name))
.setDataType(typeId)
.setTypeDefinition(Identifiers.BaseDataVariableType)
.build();
node.setValue(new DataValue(variant));
node.setAttributeDelegate(new ValueLoggingDelegate());
getNodeManager().addNode(node);
scalarTypesFolder.addOrganizes(node);
if (name.equals("Boolean")) {
Thread t = new Thread(() -> {
while (true) {
try {
Thread.sleep(1000L);
node.setValue(new DataValue(new Variant(Boolean.FALSE)));
System.out.println(node.getValue().getSourceTime());
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
}
}
});
t.start();
}
}
}
My Question(s):
1) Is updating the timestamp only allowed in OPC UA specs.
2) How can I achive this with Milo?
Thanks in advance!
EDIT
I check the timestamp by active reading:
Server-side writing:
private void handleValueUpdate(Object value, DateTime dateTime) {
node.setValue(new DataValue(
new Variant(value),
StatusCode.GOOD,
dateTime));
}
Client-side reading:
VariableNode variableNode = getOpcClient().getAddressSpace().getVariableNode(new NodeId(namespaceIndex, nodeIdentifier)).get();
return variableNode.readValue().get().getSourceTime();

Are you checking by reading (refreshing attributes pane) or with a subscription and monitored item?
If you're using MonitoredItems the default MonitoringFilter when not otherwise specified uses DataChangeTrigger.StatusValue, which includes changes to value or quality, but not timestamp, so most clients will not ask for or receive timestamp-only changes by default.

How to transcribe bandwidth call recordings in java

My application uses https://app.bandwidth.com/ for receiving incoming calls. I have an api to handle the incoming calls which record the calls when the call is not answered(This recording is treated as a voice mail).
if (eventType.equalsIgnoreCase(EventType.ANSWER.toString())) {
Timestamp callStartTime = new Timestamp(TimeUtil.now().getTime());
incomingCall.setCallTime(callStartTime);
callStatus = transferCall(callId, incomingCall.getVoiceForwardNumber(), 1);
}
else if (eventType.equalsIgnoreCase(EventType.TIMEOUT.toString())) {
voiceMailIntro(callId);
}
else if (eventType.equalsIgnoreCase(EventType.SPEAK.toString()) && PLAYBACK_STOP.equalsIgnoreCase(callState)) {
recordVoiceMail(callId);
}
else if (eventType.equalsIgnoreCase(EventType.RECORDING.toString()) &&
state.equalsIgnoreCase(BandwidthCallStatus.COMPLETE.toString())) {
createTranscription(recordingId);
}
else if (eventType.equalsIgnoreCase(EventType.TRANSCRIPTION.toString()) && status.equalsIgnoreCase(BandwidthCallStatus.COMPLETED.toString())) {
incomingCall.setVoiceMail(text);
}
This is the code for recording call
private void recordVoiceMail(String callId) {
BandwidthClient client = BandwidthClient.getInstance();
client.setCredentials(bandwidthUserId, bandwidthApiToken, bandwidthApiSecret);
try {
Call call = Call.get(client, callId);
call.recordingOn();
} catch (Exception e) {
log.error("An exception occurred while recording voice mail : " +
e.getMessage(), e);
}
}
Now i need to transcribe these vocie mails.
From documentation i got methods in python, js, c#, ruby etc. to transcribe the recordings using the recordings.
For example in js,
client.Recording.createTranscription(recordingId, function(err, transcription){});
I searched every where, but i couldn't find any method in java for that.
Can any one help me if you know ?

Anyway, as I see, you need that link for java doc.
And here you can follow to java sdk located on Github.
And, also, you can find some more information about transcriptions API here which you are looking for.
First of all, why do you need that? Perhaps, you do not need that.
As I find, you can't do transcribe with POJO, but you can do something like that.
If you want to do that, you can make it with
public void transcribeOn() throws Exception {
final List<Recording> list = Recording.list(0, 5);
if (!list.isEmpty()) {
final Recording recording = Recording.get(list.get(0).getId());
System.out.println("\nRecording by Id");
System.out.println(recording);
final String recordingUri = mockClient.getUserResourceUri(BandwidthConstants.RECORDINGS_URI_PATH);
client.post(recordingUri + "/" + list.get(0).getId() + "/transcriptions", null);
final JSONObject jsonObject = call.toJSONObject(client.get(recordingUri, null));
call.updateProperties(jsonObject);
}
}
I'm not sure it works correctly, but I hope it put you on correct way

always getting 6a82 and 6d00 for reading visa emv card

i am reading data from visa card but always getting response 6a82 and 6d00 for PSE.
I am using smartcardio and following are the commands
Select PSE:
00A404000E315041592E5359532E444446303100
Processing code:
80A8000002830000
Below code works fine for paypak (a Pakistan payment card claiming EMV compatibility) but for visa its not working.
startCardConnection("0");
String commandVisa = "00A4040007A000000003101000";
String command_PSE = "00A404000E315041592E5359532E444446303100";
String command_getProcessingOptionsVISA = "80A8000002830000";
String response;
response = executeCardCommand(command_PSE);
response = executeCardCommand(commandVisa);
readCardRecords(2);
response = executeCardCommand(command_getProcessingOptionsVISA);
response = executeCardCommand("80AE8000210000000000000000000000000586000000000005861802020000E44E4B11040001");
public static String executeCardCommand(String command) {
if (transmissionTrace)
System.out.println("SYS: Executing card command:" + command);
capdu = makeCommandAPDU(command);
TLV tagsList;
try {
if (card == null) {
System.out.println("SYS: ERR: Card not present/not responding!");
return null;
}
responsedAPDU = card.getBasicChannel().transmit(capdu);
showRes(responsedAPDU.getBytes());
tagsList = new TLV(responsedAPDU.getBytes());
allTagsTLV.getChildren().add(tagsList);
System.out.println(">>>>>>>>>>>>" + responsedAPDU.toString());
} catch (CardException e1) {
// TODO Auto-generated catch block
e1.printStackTrace();
} catch (TLVException e1) {
// TODO Auto-generated catch block
System.out.println("SYS: NO tags response. May be correct if not expecting tags.");
}
return allTagsTLV.getJson();
}

Support for PSE for contact is optional - both for cards as well as for terminals. From the terminal/kernel perspective only LoA (List of AIDs) method is obligatory. As described in detail by EMV Book 1 chapter 12.3, when PSE is missing (status word 6A82), terminal should build candidate list using List of AIDs method basing on its configuration. I don't know when you are getting 6D00, but you don't perform application selection correctly, so I guess you are firing the commands like GPO and GenAC with no application selected. The code you are quoting is very wrong - it does not handle any errors, does not select application correctly, it does not check for PDOL presence, does not read records with CDOL1, does not build DOLs. Quite honestly it may work by coincidence only.

Jsoup behaves different from my test PC and the server

I'm testing a web crawler with JSoup. The issue comes when I test the crawler on a regular PC, and works as expected, then I export this web crawler as a jar to work in a server in a cron job. This where the things go wrong.
The code is the same, no changes. The data I'm trying to extract is different comments from the users of how they rate a service, the problem is that the web crawler behaves differently when it's executed in the server, for example: the comments are duplicated, something that doesn't happened when I'm testing the program locally.
Also the web crawler differentiates what language the comments are written (I take that info from the URL, .de for German, .es for Spanish, etc). This info get mixed for example, a comment in Spanish is classified as Portuguese one.
Again I repeat the logic behind the crawler is correct, I tested many times with different input.
What could be the problem behind these issues?
Additional notes:
No exceptions/crashes.
I'm using jsoup 1.9.2.
This is how I get the data from the website:
Document doc = Jsoup.connect(link).userAgent(FakeAgentBooking.getAgent()).timeout(60 * 4000).get();
I already tried to use a proxy just in case the server was banned.
System.getProperties().put("https.proxyHost", "PROXY");
System.getProperties().put("https.proxyPort", "PORT");
System.getProperties().put("https.proxyUser", "USER");
System.getProperties().put("https.proxyPassword", "PASSWORD");
This is the code of the cron job:
#Crawler(name = "Booking comments", nameType = "BOOKING_COMMENTS", sentimetal = true, cron = "${cron.booking.comments}")
public class BookingCommentsJob extends MotherCrawler {
private static final org.slf4j.Logger logger = LoggerFactory.getLogger(BookingCommentsJob.class);
#Value("${full.booking.comments}")
private String full;
#Autowired
private ComentariosCDMXRepository comentariosCDMXRepository;
#Override
#PostConstruct
public void init() {
setInfo(this.getClass().getAnnotation(Crawler.class));
}
#Override
public void exec(int num) {
// <DEBUG>
String startTime = time.format(new Date());
// </DEBUG>
Set<CrawQuery> li = queryManager.getMeQueries(type, num, threadnum);
Gson gson = new GsonBuilder().create();
for (CrawQuery s : li) {
String query = s.getQuery().get("query");
try {
//the crawling begins here-->
String result = BookingComentarios.crawlBookingComentarios(query, Boolean.parseBoolean(full));
//get the result from json to a standarized class
ComentarioCDMX[] myComments = gson.fromJson(result, ComentarioCDMX[].class);
for (ComentarioCDMX myComment : myComments) {
//evaluates if the comment is positive, neutral or negative.
Integer sentiment = sentimentAnalysis.classifyVector(myComment.getComment());
myComment.setSentiment(sentiment);
myComment.setQuery(query);
/* <Analisis de sentimiento /> */
comentariosCDMXRepository.save(myComment);
}
s.setStatus(true);
} catch (Exception e) {
logger.error(query, e);
s.setStatus(false);
mailSend.add(e);
} finally {
s.setLastUse(new Date());
//Saves data to Solr
crawQueryDao.save(s);
}
}
update();
// <DEBUG>
String endTime = time.format(new Date());
logger.info(name + " " + num + " > Inicio: " + startTime + ", Fin: " + endTime);
// </DEBUG>
}
#Scheduled(cron = "${cron.booking.comments}")
public void curro0() throws InterruptedException {
exec(0);
}
}
and this is when the code should be executed:
cron.booking.comments=00 30 02 * * *
Additional notes:
The test PC OS is Windows 7 and the server OS is linux Debian 3.16.7. and tghe java version in the test PC is 1.7 oracle JDK and on the server is 1.8.0 JRE.