We Keep Coding

Reindex selected _source fields using Rest high level client in java

I want to re_index only selected fields from my document in elasticsearch using Rest High level client.
I know the elasticsearch query to achieve that but I don't know it's equivalent query using rest client.
Following is the elasticsearch query which I am trying to implement using rest client -
{
"body" : {
"source" : {
"index" : "my source index name",
"_source" : "id, name, rollNo"
},
"dest" : {
"index" : "my destination index name"
}
}
}
To write its equivalent query using rest client in java, I have used the following code -
ReindexRequest reindexRequest = new ReindexRequest();
reindexRequest.setSourceIndices("source index name").setDestIndex("destination index name");
reindexRequest.setDocTypes("id", "name", "rollNo", "_doc");
client.reindex(reindexRequest,RequestOptions.DEFAULT);
But the above code is not working as expected. It's re_indexing all the fields of my document. I want only selective 3 fields to be re_indexed from each doc.

You need to use below code as setDocTypes is not used for source filtering.
As there is no direct method available for setting source filter so you need to change underlying search request suing below code.
ReindexRequest reindexRequest = new ReindexRequest();
reindexRequest.setSourceIndices("source index name").setDestIndex("destination index name");
reindexRequest.setDocTypes("_doc");
String[] include=new String[] {"id", "name", "rollNo"};
String[] exclude=new String[] {"test"};
reindexRequest.getSearchRequest().source().fetchSource(include, exclude);
client.reindex(reindexRequest,RequestOptions.DEFAULT);

Mention channel using ms graph api

I am working with ms graph and i need to integrate with ms teams. In one case i have to send a message to a channel which will contain a channel mention (#). Currently i have found how to tag a specific user using java but i cant figure out how to mention the whole channel. It is posible to do so in ms graph and if yes how?
To be more specific here is an example. Assume that i have a team named TestTeam and this team has a channel named testChannel. i need to send a message where will contain #testChannel and will send a notification to everyone in this channel.
Also in slack api i am able to do this action by using "<!channel>" in my message.
This is the code i am using in order to tag a user:
static void send_message(String text, String mentionName, String channelName, String teamName, String accessToken) {
ensureGraphClient(accessToken);
Team team = getTeam(accessToken, teamName);
Channel channel = getChannel(accessToken, channelName, team);
User user = getUser(accessToken, mentionName);
if (userInChannel(user, team, channel)) {
ChatMessage chatMessage = new ChatMessage();
ItemBody body = new ItemBody();
body.contentType = BodyType.HTML;
body.content = String.format("%s. <at id=\"1\">%s</at> ", text, mentionName);
chatMessage.body = body;
ChatMessageMention m = new ChatMessageMention();
m.id = 1;
IdentitySet st = new IdentitySet();
Identity ide = new Identity();
ide.id = user.id;
ide.displayName = user.displayName;
st.user = ide;
m.mentioned = st;
m.mentionText = mentionName;
List<ChatMessageMention> cmmt = new LinkedList<>();
cmmt.add(m);
chatMessage.mentions = cmmt;
graphClient.teams(team.id).channels(channel.id).messages()
.buildRequest()
.post(chatMessage);
}
}
I also tried something similar in order to tag channels but it did not worked:
body.contentType = BodyType.HTML;
body.content = String.format("%s. <at id=\"1\">%s</at> ", text, "myChannel");
chatMessage.body = body;
ChatMessageMention m = new ChatMessageMention();
m.id = 1;
IdentitySet st = new IdentitySet();
Identity ide = new Identity();
ide.id = "19%xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx%40thread.tacv2";
ide.displayName = "myChannel";
st.user = ide;
m.mentioned = st;
m.mentionText = "myChannel";
List<ChatMessageMention> cmmt = new LinkedList<>();
cmmt.add(m);
chatMessage.mentions = cmmt;
graphClient.teams(team.id).channels(channel.id).messages()
.buildRequest()
.post(chatMessage);

Introduction:
After playing around with it, here is the final result. And this is not final answer, but hope to lead for solution.
Lets start some introduction.
First lets define our post url:
POST https://graph.microsoft.com/beta/teams/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/channels/19:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx#thread.tacv2/messages
Now lets mentioning a user in graph api:
{
"body": {
"contentType": "html",
"content": "Hello World <at id=\"0\">Maytham Fahmi</at>"
},
"mentions": [
{
"id": 0, <- the index here should be the same as in the content message
"mentionText": "Maytham Fahmi",
"mentioned": {
"user": {
"displayName": "Maytham Fahmi",
"id": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" <- this should be user user object id
}
}
}
]
}
This is tested and working via Graph Api.
Now I come to the interested part. Sending to channel with channel mentioning looks like this:
{
"body": {
"contentType": "html",
"content": "Hello World <at id=\"0\">General</at>"
},
"mentions": [
{
"id": 0, <- the index here should be the same as in the content message
"mentionText": "General",
"mentioned": {
"conversation": {
"id": "19:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx#thread.tacv2",
"displayName": "General",
"conversationIdentityType#odata.type": "#Microsoft.Teams.GraphSvc.conversationIdentityType",
"conversationIdentityType": "channel"
}
}
]
}
The results in teams, that is sent from channel mentioning
This is also working from Graph api explorer.
The answer:
So as we can see for user mentioning we use User object under IdentitySet instance which works fine for user. I have testing it C# using Microsoft.Graph.
Here is comes the problem Conversation object does not exist under IdentitySet in C# using Microsoft.Graph. I am pretty sure this is the case for Java as well. If you have that object in Java then you should be able to make it. It is perhaps not in C#.
I have red different places, and sadly channel mentioning objects is not implemented in code yet, looks like a bug or a feature they will fix over time.
A work around solution is, you have graph api json payloads, you can make a Java client to post Jsons to the url with and your job is done.
Note: I have look at our solution, it does not support mentioning as well.
Inspiration links:
https://learn.microsoft.com/en-us/graph/api/channel-post-message?view=graph-rest-1.0&tabs=csharp#permissions
https://learn.microsoft.com/en-us/graph/api/resources/chatmessagemention?view=graph-rest-1.0
Note in case you need to send notification in general, you could use webhooks.

Spring boot Stram API large data

I'm using JPA with a Native Query that returns about 13k registers. I thought to use Stream API from Java 8, but the result is not coming.
I Can't paginate because the result will populate a combo box
My repository sends the Stream
I added the #Transactional(readOnly = true) to make it work
#Query(value = "select * from mytable", native = true)
Stream<MyTable> getTableStream()
#Transactional(readOnly = true)
public Stream<MyTable> getTableStream() {
return repository.getTableStream()
}
#GetMapping(value = "/table", produces = MediaType.APPLICATION_STREAM_JSON_VALUE)
#Transactional(readOnly = true)
public ResponseEntity<Stream<MyTable>> getMailingClient() {
Stream<MyTable> body = service.getTableStream();
return ResponseEntity.ok(body);
}
All links and resources I found about stream do not show the implementation of the return JSON with Spring rest API.
My frontend is Angular 6, and nearest I got was one custom object with no result

Loading all 13k records into a combo box sounds like a very slow solution. I would recommend implementing a search based on like query. Something like this:
#Query("SELECT * FROM mytable WHERE name like ':name%'")
Stream<MyTable> getTableStream(#Param("name") String name);
But if you really want to load all of the records, you can use a java.util.Collection or java.util.List instead of a stream.
Collection<MyTable> getTableStream();

Correct way to implement paging for Cassandra with CassandraRepository from Spring Data

I'm looking for a solution to implement paging for our Spring Boot based REST-Service with a Cassandra (version 3.11.3) database. We are using Spring Boot 2.0.5.RELEASE with spring-boot-starter-data-cassandra as a dependency.
As Spring Data's CassandraRepository<T, ID> interface does not extend the PagingAndSortingRepository we don't get the full paging functionality like we have with JPA.
I read the Spring Data Cassandra documentation and could find a possible way to implement paging with Cassandra and Spring Data as the CassandraRepository interface has the following method available Slice<T> findAll(Pageable pageable);. I am aware that Cassandra is not able to get a specific page adhoc and always needs page zero to iterate through all pages as it is documented in the CassandraPageRequest:
Cassandra-specific {#link PageRequest} implementation providing access to {#link PagingState}. This class allows creation of the first page request and represents through Cassandra paging is based on the progress of fetched pages and allows forward-only navigation. Accessing a particular page requires fetching of all pages until the desired page is reached.
In my usecase we have > 1.000.000 database entries and want to display them paged in our single page application.
My current approach looks like the following:
#RestController
#RequestMapping("/users")
public class UsersResource {
#Autowired
UserRepository userRepository;
#GetMapping
public ResponseEntity<List<User>> getAllTests(
#RequestParam(defaultValue = "0", name = "page") #Positive int requiredPage,
#RequestParam(defaultValue = "500", name = "size") int size) {
Slice<User> resultList = userRepository.findAll(CassandraPageRequest.first(size));
int currentPage = 0;
while (resultList.hasNext() && currentPage <= requiredPage) {
System.out.println("Current Page Number: " + currentPage);
resultList = userRepository.findAll(resultList.nextPageable());
currentPage++;
}
return ResponseEntity.ok(resultList.getContent());
}
}
BUT with this approach I have to find the requested page while fetching all database entries to memory and iterate until I found the correct page. Is there a different approach to find the correct page or do I have to use my current solution?
My Cassandra table definition looks like the following:
CREATE TABLE user (
id int, firstname varchar,
lastname varchar,
code varchar,
PRIMARY KEY(id)
);

What I have done is to create a page object that has the content and the pagingState hash.
In the initial page, we have the simple paging
Pageable pageRequest = CassandraPageRequest.of(0,5);
once the find is performed we get the slice
Slice<Group> slice = groupRepository.findAll(pageRequest);
with the slice you can get the paging state
page.setPageHash(getPageHash((CassandraPageRequest) slice.getPageable()));
where
private String getPageHash(CassandraPageRequest pageRequest) {
return Base64.toBase64String(pageRequest.getPagingState().toBytes());
}
finally returning a Page object with the List content and the pagingState as pageHash

See this below code. It may help.
#GetMapping("/loadData")
public Mono<DataTable> loadData(#RequestParam boolean reset, #RequestParam(required = false) String tag, WebSession session) {
final String sessionId = session.getId();
IMap<String, String> map = Context.get(HazelcastInstance.class).getMap("companygrouping-pageable-map");
int pageSize = Context.get(EnvProperties.class).getPageSize();
Pageable pageRequest;
if (reset)
map.remove(sessionId);
String serializedPagingState = map.compute(sessionId, (k, v) -> (v == null) ? null : map.get(session.getId()));
pageRequest = StringUtils.isBlank(serializedPagingState) ? CassandraPageRequest.of(0, pageSize)
: CassandraPageRequest.of(PageRequest.of(0, pageSize), PagingState.fromString(serializedPagingState)).next();
Mono<Slice<TagMerge>> sliceMono = StringUtils.isNotBlank(tag)
? Context.get(TagMergeRepository.class).findByKeyStatusAndKeyTag(Status.NEW, tag, pageRequest)
: Context.get(TagMergeRepository.class).findByKeyStatus(Status.NEW, pageRequest);
Flux<TagMerge> flux = sliceMono.map(t -> convert(t, map, sessionId)).flatMapMany(Flux::fromIterable);
Mono<DataTable> dataTabelMono = createTableFrom(flux).doOnError(e -> log.error("{}", e));
if (reset) {
Mono<Long> countMono = Mono.empty();
if (StringUtils.isNotBlank(tag))
countMono = Context.get(TagMergeRepository.class).countByKeyStatusAndKeyTag(Status.NEW, tag);
else
countMono = Context.get(TagMergeRepository.class).countByKeyStatus(Status.NEW);
dataTabelMono = dataTabelMono.zipWith(countMono, (t, k) -> {
t.setTotalRows(k);
return t;
});
}
return dataTabelMono;
}
private List<TagMerge> convert(Slice<TagMerge> slice, IMap<String, String> map, String id) {
PagingState pagingState = ((CassandraPageRequest) slice.getPageable()).getPagingState();
if (pagingState != null)
map.put(id, pagingState.toString());
return slice.getContent();
}

Cassandra supports forward pagination which means you can fetch first n rows then you can fetch rows between n+1 and 2n and so on until your data ends but you can't fetch rows between n+1 and 2n directly.

How to implement proper pagination in Google App Engine (Java)?

I tried to implement pagination in google app engine (Java), but I am not able to achieve. It is working only forward pagination and reverse pagination is not able to achieved.
I tried storing the previous cursor value through HTTP request as below:
JSP file:
<a href='/myServlet?previousCursor=${previousCursor}'>Previous page</a>
<a href='/myServlet?nextCursor=${nextCursor}'>Next page</a>
Servlet file:
String previousCursor = req.getParameter("previousCursor");
String nextCursor = req.getParameter("nextCursor");
String startCursor = null;
if(previousCursor != null){
startCursor = previousCursor;
}
if(nextCursor != null){
startCursor = nextCursor;
}
int pageSize = 3;
FetchOptions fetchOptions = FetchOptions.Builder.withLimit(pageSize);
if (startCursor != null) {
fetchOptions.startCursor(Cursor.fromWebSafeString(startCursor));
}
Query q = new Query("MyQuery");
PreparedQuery pq = datastore.prepare(q);
QueryResultList<Entity> results = pq.asQueryResultList(fetchOptions);
for (Entity entity : results) {
//Get the properties from the entity
}
String endCursor = results.getCursor().toWebSafeString();
req.setAttribute("previousCursor", startCursor);
req.setAttribute("nextCursor", endCursor);
With this I am able to retain the previous cursor value, but unfortunately the previous cursor seems to be invalid.
I also tried using reverse() method, but it is of no use. It work same as forward.
So is the any way to implement proper pagination (forward and backword) in google app engine (Java)?
I found similar one that was posted in 2010. Here also the answer was to use Cursor. But as I shown above it is not working.
Pagination in Google App Engine with Java

If you are familiar with JPA you can give it a try.
Have tested it and pagination works in GAE.
I think they support JPA 1.0 as of now.
What I tried was, created an Employee entity.
Created DAO layer and persisted few employee entities.
To have a paginated fetch, did this:
Query query = em.createQuery("select e from Employee e");
query.setFirstResult(0);
query.setMaxResults(2);
List<Employee> resultList = query.getResultList();
(In this example we get first page which has 2 entities. Argument to
setFirstResult would be start index and argument to setMaxResult would be your page size)
You can easily change the arguments to query.setFirstResult and setMaxResults
and have a pagination logic around it.
Hope this helps,
Regards,