I need to periodically clear the data in multiple sheets and re-populate them with data (via the Google Sheets API v4). To do this, I'm executing 2 separate requests for each sheet (1 clear & 1 update). This is kind of a slow process when the user is sitting there waiting for it. It seems to me that each new request significantly adds to the completion time. If I could wrap all these into a single batch-command request, it might help a lot.
I'm currently doing this for each sheet...
service.spreadsheets()
.values()
.clear(idSpreadsheet, sheetTitle + "!$A$1:$Y", new ClearValuesRequest())
.execute();
service.spreadsheets()
.values()
.update(idSpreadsheet, range, new ValueRange().setValues(values))
.setValueInputOption("USER_ENTERED")
.execute();
I don't see a way to just wrap a bunch of generic commands into a single batch request. I see that DeleteDimensionRequest and AppendCellsRequest can be wrapped into a batch, but I can't really find a good AppendCellsRequest example (and it seems that people recommend my current values().update() method anyway).
Can anyone recommend a good way to streamline this? Or am I already doing it the best way?
Still don't know if I'm doing it the BEST way, but I was able to accomplish my goal of clearing and populating multiple sheets with data in a single batch request. The trick was to not use the clear() method but instead, overwrite the the sheet with blank data using RepeatCellRequest. Also, now using AppendCellsRequest instead of update(). These two requests can be wrapped in a batch request.
My early tests with 3 sheets, shows about a 25% performance improvement. Not spectacular, but it helps.
List<Request> requests = new ArrayList<Request>();
for (SheetData mySheet : sheetDatas)
{
List<List<Object>> values = mySheet.getValues();
Request clearSheetRequest = new Request()
.setRepeatCell(new RepeatCellRequest()
.setRange(new GridRange()
.setSheetId(mySheet.getSheetId())
)
.setFields("*")
.setCell(new CellData())
);
List<RowData> preppedRows = new ArrayList<RowData>();
for (List<Object> row : values)
{
RowData preppedRow = new RowData();
List<CellData> cells = new ArrayList<CellData>();
for (Object value : row)
{
CellData cell = new CellData();
ExtendedValue userEnteredValue = new ExtendedValue();
if (value instanceof String)
{
userEnteredValue.setStringValue((String) value);
}
else if (value instanceof Double)
{
userEnteredValue.setNumberValue((Double) value);
}
else if (value instanceof Integer)
{
userEnteredValue.setNumberValue(Double.valueOf((Integer) value).doubleValue());
}
else if (value instanceof Boolean)
{
userEnteredValue.setBoolValue((Boolean) value);
}
cell.setUserEnteredValue(userEnteredValue);
cells.add(cell);
}
preppedRow.setValues(cells);
preppedRows.add(preppedRow);
}
Request appendCellsRequest = new Request().setAppendCells(
new AppendCellsRequest()
.setSheetId(mySheet.getSheetId())
.setRows(preppedRows)
.setFields("*")
);
requests.add(clearSheetRequest);
requests.add(appendCellsRequest);
}
BatchUpdateSpreadsheetRequest batch = new BatchUpdateSpreadsheetRequest().setRequests(requests);
BatchUpdateSpreadsheetResponse batchResponse = service.spreadsheets().batchUpdate(idSpreadsheet, batch).execute();
Related
We have an application that loads all contacts stored in an account using the Microsoft Graph API. The initial call we issue is https://graph.microsoft.com/v1.0/users/{userPrincipalName}/contacts$count=true&$orderBy=displayName%20ASC&$top=100, but we use the Java JDK to do that. Then we iterate over all pages and store all loaded contacts in a Set (local cache).
We do this every 5 minutes using an account with over 3000 contacts and sometimes, the count of contacts we received due to using $count does not match the number of contacts we loaded and stored in the local cache.
Verifying the numbers manually we can say, that the count was always correct, but there are contacts missing.
We use the following code to achieve this.
public List<Contact> loadContacts() {
Set<Contact> contacts = new TreeSet<>((contact1, contact2) -> StringUtils.compare(contact1.id, contact2.id));
List<QueryOption> requestOptions = List.of(
new QueryOption("$count", true),
new QueryOption("$orderBy", "displayName ASC"),
new QueryOption("$top", 100)
);
ContactCollectionRequestBuilder pageRequestBuilder = null;
ContactCollectionRequest pageRequest;
boolean hasNextPage = true;
while (hasNextPage) {
// initialize page request
if (pageRequestBuilder == null) {
pageRequestBuilder = graphClient.users(userId).contacts();
pageRequest = pageRequestBuilder.buildRequest(requestOptions);
} else {
pageRequest = pageRequestBuilder.buildRequest();
}
// load
ContactCollectionPage contactsPage = pageRequest.get();
if (contactsPage == null) {
throw new IllegalStateException("request returned a null page");
} else {
contacts.addAll(contactsPage.getCurrentPage());
}
// handle next page
hasNextPage = contactsPage.getNextPage() != null;
if (hasNextPage) {
pageRequestBuilder = contactsPage.getNextPage();
} else if (contactsPage.getCount() != null && !Objects.equals(contactsPage.getCount(), (long) contacts.size())) {
throw new IllegalStateException(String.format("loaded %d contacts but response indicated %d contacts", contacts.size(), contactsPage.getCount()));
} else {
// done
}
}
log.info("{} contacts loaded using graph API", contacts.size());
return new ArrayList<>(contacts);
}
Initially, we did not put the loaded contacts in a Set by ID but just in a List. With the List we very often got more contacts than $count. My idea was, that there is some caching going on and some pages get fetched multiple times. Using the Set we can make sure, that we only have unique contacts in our local cache.
But using the Set, we sometimes have less contacts than $count, meaning some pages got skipped and we end up in the condition that throws the IllegalStateException.
Currently, we use microsoft-graph 5.8.0 and azure-identiy 1.4.2.
Have you experienced similar issues and can help us solve this problem?
Or do you have any idea what could be causing these inconsistent results?
Your help is very much appreciated!
I want to write an endpoint which always shows the newest messages of a redis stream (reactive).
The entities look like this {'key' : 'some_key', 'status' : 'some_string'}.
So I would like to have the following result:
Page is called, content would be for instance displaying an entity:
{'key' : 'abc', 'status' : 'status_A'}
the page is not closed
Then a new entity is added to the stream
XADD mystream * key abc status statusB
Now I would prefer to see each item of the Stream, without updating the Tab
{'key' : 'abc', 'status' : 'status_A'}
{'key' : 'abc', 'status' : 'status_B'}
When I try to mock this behavior it works and I get the expected output.
#GetMapping(value="/light/live/mock", produces = MediaType.TEXT_EVENT_STREAM_VALUE)
#ResponseBody
public Flux<Light> liveLightMock() {
List<Light> test = Arrays.asList(new Light("key", "on") , new Light("key", "off"),
new Light("key", "on") , new Light("key", "off"),
new Light("key", "on") , new Light("key", "off"),
new Light("key", "on") , new Light("key", "off"),
new Light("key", "on") , new Light("key", "off"));
return Flux.fromIterable(test).delayElements(Duration.ofMillis(500));
}
The individual elements of the list are displayed one after another with a 500ms Delay between items.
However, when I try to access Redis instead of the mocked variant, it no longer works. I try to test the partial functions successively. So that my idea works first the save (1) function must work, if the save function works, displaying old records without reactiv features must work (2) and last but not least if both work i kinda need to get the reactiv part going.
Maybe you guys can help me get the Reactive Part Working. Im working on it for days without getting any improvements.
Ty guys :)
Test 1) - Saving Function (Short Version)
looks like its working.
#GetMapping(value="/light/create", produces = MediaType.APPLICATION_JSON_VALUE)
#ResponseBody
public Flux<Light> createTestLight() {
String status = (++statusIdx % 2 == 0) ? "on" : "off";
Light light = new Light(Consts.LIGHT_ID, status);
return LightRepository.save(light).flux();
}
#Override
public Mono<Light> save(Light light) {
Map<String, String> lightMap = new HashMap<>();
lightMap.put("key", light.getKey());
lightMap.put("status", light.getStatus());
return operations.opsForStream(redisSerializationContext)
.add("mystream", lightMap)
.map(__ -> light);
}
Test 2) - Loading/Reading Function (Short Version)
seems to be working, but not reaktiv -> i add a new entity while a WebView was Open, the View showed all Items but didnt Updated once i added new items. after reloading i saw every item
How can i get getLightsto return something that is working with TEXT_EVENT_STREAM_VALUE which subscribes to the stream?
#Override
public Flux<Object> getLights() {
ReadOffset readOffset = ReadOffset.from("0");
StreamOffset<String> offset = StreamOffset.fromStart("mystream"); //fromStart or Latest
Function<? super MapRecord<String, Object, Object>, ? extends Publisher<?>> mapFunc = entries -> {
Map<Object, Object> kvp = entries.getValue();
String key = (String) kvp.get("key");
String status = (String) kvp.get("status");
Light light = new Light(key, status);
return Flux.just(light);
};
return operations.opsForStream()
.read(offset)
.flatMap(mapFunc);
}
#GetMapping(value="/light/live", produces = MediaType.TEXT_EVENT_STREAM_VALUE)
#ResponseBody
public Flux<Object> lightLive() {
return LightRepository.getLights();
}
Test 1) - Saving Function (Long Version)
The Endpoint & Saving Functions are part of Diffrent Classes.
String status = (++statusIdx % 2 == 0) ? "on" : "off"; flip flops the status from on to off, to on, to off, ...
#GetMapping(value="/light/create", produces = MediaType.APPLICATION_JSON_VALUE)
#ResponseBody
public Flux<Light> createTestLight() {
String status = (++statusIdx % 2 == 0) ? "on" : "off";
Light light = new Light(Consts.LIGHT_ID, status);
return LightRepository.save(light).flux();
}
#Override
public Mono<Light> save(Light light) {
Map<String, String> lightMap = new HashMap<>();
lightMap.put("key", light.getKey());
lightMap.put("status", light.getStatus());
return operations.opsForStream(redisSerializationContext)
.add("mystream", lightMap)
.map(__ -> light);
}
To Validate the Functions i
Delted the Stream, to Empty it
127.0.0.1:6379> del mystream
(integer) 1
127.0.0.1:6379> XLEN myStream
(integer) 0
Called the Creation Endpoint twice /light/create
i expected the Stream now to have two Items, on with status = on, and one with off
127.0.0.1:6379> XLEN mystream
(integer) 2
127.0.0.1:6379> xread STREAMS mystream 0-0
1) 1) "mystream"
2) 1) 1) "1610456865517-0"
2) 1) "key"
2) "light_1"
3) "status"
4) "off"
2) 1) "1610456866708-0"
2) 1) "key"
2) "light_1"
3) "status"
4) "on"
It looks like the Saving part is Working.
Test 2) - Loading/Reading Function (Long Version)
seems to be working, but not reaktiv -> i add a new entity and the page updates its values
#Override
public Flux<Object> getLights() {
ReadOffset readOffset = ReadOffset.from("0");
StreamOffset<String> offset = StreamOffset.fromStart("mystream"); //fromStart or Latest
Function<? super MapRecord<String, Object, Object>, ? extends Publisher<?>> mapFunc = entries -> {
Map<Object, Object> kvp = entries.getValue();
String key = (String) kvp.get("key");
String status = (String) kvp.get("status");
Light light = new Light(key, status);
return Flux.just(light);
};
return operations.opsForStream()
.read(offset)
.flatMap(mapFunc);
}
#GetMapping(value="/light/live", produces = MediaType.TEXT_EVENT_STREAM_VALUE)
#ResponseBody
public Flux<Object> lightLive() {
return LightRepository.getLights();
}
Calling /light/live -> i should have N entries
-> if I can see Entries, the normal Display is Working (non Reactive)
Calling /light/create twice -> the live Few should have added 2 Entries -> N+2Entries
Waiting 1 Minute just to be Safe
The View Should Show N+2 Entries for the Reactiv Part to be working
Refresh View from 1 (/light/live), should still show the same amount if Reactiv Works
Displaying the Information works (1), the Adding part of (2) worked, checked per Terminal, 4) didnt work
ergo the Display is working, but its not reactive
after i refreshed the Browser (5) i got the expected N+2 entries - so (2) worked aswell
There's a misconception here, reading from Redis reactively does not mean you have subscribed for new events.
Reactive will not provide you live updates, it will call Redis once and it will display whatever is there. So even if you wait for a day or two nothing is going to change in UI/Console, you will still seeing N entries.
You need to either use Redis PUB/SUB or you need to call Redis repetitively to get the latest update.
EDIT:
A working solution..
private List<Light> reactiveReadToList() {
log.info("reactiveReadToList");
return read().collectList().block();
}
private Flux<Light> read() {
StreamOffset<Object> offset = StreamOffset.fromStart("mystream");
return redisTemplate
.opsForStream()
.read(offset)
.flatMap(
e -> {
Map<Object, Object> kvp = e.getValue();
String key = (String) kvp.get("key");
String id = (String) kvp.get("id");
String status = (String) kvp.get("status");
Light light = new Light(id, key, status);
log.info("{}", light);
return Flux.just(light);
});
}
A reader that reads data from Redis on demand using reactive template and send it to the client as it sees using offset, it sends only one event at once we can send all of them.
#RequiredArgsConstructor
class DataReader {
#NonNull FluxSink<Light> sink;
private List<Light> readLights = null;
private int currentOffset = 0;
void register() {
readLights = reactiveReadToList();
sink.onRequest(
e -> {
long demand = sink.requestedFromDownstream();
for (int i = 0; i < demand && currentOffset < readLights.size(); i++, currentOffset++) {
sink.next(readLights.get(currentOffset));
}
if (currentOffset == readLights.size()) {
readLights = reactiveReadToList();
currentOffset = 0;
}
});
}
}
A method that uses DataReader to generate flux
public Flux<Light> getLights() {
return Flux.create(e -> new DataReader(e).register());
}
Now we've added an onRequest method on the sink to handle the client demand, this reads data from the Redis stream as required and sends it to the client.
This looks to be very CPU intensive maybe we should delay the calls if there're no more new events, maybe add a sleep call inside register method if we see there're not new elements in the stream.
I'm sorry this question header is not 100% correct. Because of that, I'll explain my scenario here.
I created a function to merge 4 data sets into one return format. Because that's the format front-end side needed. So this is working fine now.
public ReturnFormat makeThribleLineChart(List<NameCountModel> totalCount, List<NameCountModel>,p1Count, List<NameCountModel> p2Count, List<NameCountModel> average) {
ReturnFormat returnFormat = new ReturnFormat(null,null);
try {
String[] totalData = new String[totalCount.size()];
String[] p1Data = new String[p1Count.size()];
String[] p2Data = new String[p2Count.size()];
String[] averageData = new String[p2Count.size()];
String[] lableList = new String[totalCount.size()];
for (int x = 0; x < totalCount.size(); x++) {
totalData[x] = totalCount.get(x).getCount();
p1Data[x] = p1Count.get(x).getCount();
p2Data[x] = p2Count.get(x).getCount();
averageData[x] = average.get(x).getCount();
lableList[x] = totalCount.get(x).getName();
}
FormatHelper<String[]> totalFormatHelper= new FormatHelper<String[]>();
totalFormatHelper.setData(totalData);
totalFormatHelper.setType("line");
totalFormatHelper.setLabel("Uudet");
totalFormatHelper.setyAxisID("y-axis-1");
FormatHelper<String[]> p1FormatHelper= new FormatHelper<String[]>();
p1FormatHelper.setData(p1Data);
p1FormatHelper.setType("line");
p1FormatHelper.setLabel("P1 päivystykseen heti");
FormatHelper<String[]> p2FormatHelper= new FormatHelper<String[]>();
p2FormatHelper.setData(p2Data);
p2FormatHelper.setType("line");
p2FormatHelper.setLabel("P2 päivystykseen muttei yöllä");
FormatHelper<String[]> averageFormatHelper= new FormatHelper<String[]>();
averageFormatHelper.setData(averageData);
averageFormatHelper.setType("line");
averageFormatHelper.setLabel("Jonotusaika keskiarvo");
averageFormatHelper.setyAxisID("y-axis-2");
List<FormatHelper<String[]>> formatHelpObj = new ArrayList<FormatHelper<String[]>>();
formatHelpObj.add(totalFormatHelper);
formatHelpObj.add(p1FormatHelper);
formatHelpObj.add(p2FormatHelper);
formatHelpObj.add(averageFormatHelper);
returnFormat.setData(formatHelpObj);
returnFormat.setLabels(lableList);
returnFormat.setMessage(Messages.Success);
returnFormat.setStatus(ReturnFormat.Status.SUCCESS);
} catch (Exception e) {
returnFormat.setData(null);
returnFormat.setMessage(Messages.InternalServerError);
returnFormat.setStatus(ReturnFormat.Status.ERROR);
}
return returnFormat;
}
so, as you can see here, all the formatting is hardcoded. So my question is how to automate this code for list count. Let's assume next time I have to create chart formatting for five datasets. So I have to create another function to it. That's the thing I want to reduce. So I hope you can understand my question.
Thank you.
You're trying to solve the more general problem of composing a result object (in this case ReturnFormat) based on dynamic information. In addition, there's some metadata being setup along with each dataset - the type, label, etc. In the example that you've posted, you've hardcoded the relationship between a dataset and this metadata, but you'd need some way to establish this relationship for data dynamically if you have a variable number of parameters here.
Therefore, you have a couple of options:
Make makeThribleLineChart a varargs method to accept a variable number of parameters representing your data. Now you have the problem of associating metadata with your parameters - best option is probably to wrap the data and metadata together in some new object that is provided as each param of makeThribleLineChart.
So you'll end up with a signature that looks a bit like ReturnFormat makeThribleLineChart(DataMetadataWrapper... allDatasets), where DataMetadataWrapper contains everything required to build one FormatHelper instance.
Use a builder pattern, similar to the collection builders in guava, for example something like so:
class ThribbleLineChartBuilder {
List<FormatHelper<String[]>> formatHelpObj = new ArrayList<>();
ThribbleLineChartBuilder addDataSet(String describeType, String label, String yAxisId, List<NameCountModel> data) {
String[] dataArray = ... ; // build your array of data
FormatHelper<String[]> formatHelper = new FormatHelper<String[]>();
formatHelper.setData(dataArray);
formatHelper.setType(describeType);
... // set any other parameters that the FormatHelper requires here
formatHelpObj.add(formatHelper);
return this;
}
ReturnFormat build() {
ReturnFormat returnFormat = new ReturnFormat(null, null);
returnFormat.setData(this.formatHelpObj);
... // setup any other fields you need in ReturnFormat
return returnFormat;
}
}
// usage:
new ThribbleLineChartBuilder()
.addDataSet("line", "Uudet", "y-axis-1", totalCount)
.addDataSet("line", "P1 päivystykseen heti", null, p1Count)
... // setup your other data sources
.build()
I'm listening for connection changes through events pluging ("amq.rabbitmq.event", "connection.#").
It works properly so I'm adding at java side two additional parameters as clientproperties, to get the identity of the user that connects or disconnect.
However at c# side I can only access these properties as a list of byte[], and not sure on how to convert it to a Dictionary or so..
If I print all entries
if (args.BasicProperties.Headers.TryGetValue("client_properties", out object value))
{
var items = value as List<object>;
foreach(var item in items)
{
Console.WriteLine($"{item.GetType().ToString()}");
var bytes = item as byte[];
result.Add(Encoding.UTF8.GetString(bytes));
}
}
I can see this:
{<<"platform">>,longstr,<<"Java">>}
{<<"capabilities">>,table,[{<<"connection.blocked">>,bool,true},{<<"basic.nack">>,bool,true},{<<"exchange_exchange_bindings">>,bool,true},{<<"authentication_failure_close">>,bool,true},{<<"publisher_confirms">>,bool,true},{<<"consumer_cancel_notify">>,bool,true}]}
{<<"groupId">>,longstr,<<"1e6e935f0d4d9ec446d67dadc85cbafd10d1a095">>}
{<<"information">>,longstr,<<"Licensed under the MPL. See http://www.rabbitmq.com/">>}
{<<"version">>,longstr,<<"4.8.1">>}
{<<"copyright">>,longstr,<<"Copyright (c) 2007-2018 Pivotal Software, Inc.">>}
{<<"product">>,longstr,<<"RabbitMQ">>}
What kind of object format is and how can I parse this?:
{<<id>>,type,<<value>>}
Apparently ( as for an answer I got on Rabbit client google group for this questions ), client_properties is something that's not created to being read by the receiving party..
However is a really good way to have something like LWT ( Last Will and Testament ), then I am using it at the minute doing the parse by myself.
if (args.BasicProperties.Headers.TryGetValue("client_properties", out object value))
{
var items = value as List<object>;
foreach (var item in items)
{
var bytes = item as byte[];
//{<<id>>, type, <<value>>}
String itemStr = Encoding.UTF8.GetString(bytes);
var parts = itemStr.Split(",");
var key = CleanErlangString(parts[0]);
var value = CleanErlangString(parts[2]);
// Do things with key/value
}
}
ClearErlangFunction
private static string CleanErlangString(string toClean)
{
return toClean
.Replace("{", "").Replace("}", "")
.Replace("\"", "")
.Replace("<<", "").Replace(">>", "");
}
What I am doing to use it as LWT, is setting a custom property on client side and then obtaining it while reading events at "amq.rabbitmq.event", "connection.#". With that I know who have disconnected and even process something as LWT with my core server.
I hope this helps someone :)
I have a list of list which i am looking forward to run it using akka and would want to do a operation when all of the child lists are done processing. But the Complete is running before all child's are completed.
Basically i am trying to read all the sheets in the excel and then read each rows from the excel. For this i am looking to use akka to process each sheets seperately and also in each sheet i am looking to process each rows seperately.
Sample Code:
List<List<String>> workbook = new ArrayList<List<String>>();
List<String> Sheet1 = new ArrayList<String>();
Sheet1.add("S");
Sheet1.add("a");
Sheet1.add("d");
List<String> Sheet2 = new ArrayList<String>();
Sheet2.add("S");
Sheet2.add("a1");
Sheet2.add("d");
workbook.add(Sheet1);
workbook.add(Sheet2);
final ActorSystem system = ActorSystem.create("Sys");
final ActorMaterializer materializer = ActorMaterializer.create(system);
Source.from(workbook).map(sheet -> {
return Source.from(sheet).runWith(Sink.foreach(data -> {
System.out.println(data);
Thread.sleep(1000);
}), materializer).toCompletableFuture();
}).runWith(Sink.ignore(), materializer).whenComplete((a, b) -> {
System.out.println("Complete");
});
system.terminate();
The Current output is:
S
S
Complete
a
a1
d
d
The Expected output is:
S
S
a
a1
d
d
Complete
Could anyone please help ?
Your use of a "stream within a stream" may be overcomplicating the process.
You could instead use Flow.flatMapConcat. I can only provide an example in scala but hopefully it translates easily to java:
val flattenFlow : Flow[List[String], String, NotUsed] =
Flow[List[String].flatMapConcat(sheet => Source(sheet))
val Source[String] flattenedSource = Source(worksheet).via(flattenFlow)
There is a blog post with a example of using flatMapConcat in java but I don't know if my guessed type Flow.of(List<String>.class) is valid code.