We Keep Coding

Elasticsearch Java API implementation to fetch millions of records

I want to get all doc (millions) in elastic index based on some condition. I used below query in elastic.
GET /<index-name>/_search
{
"from" : 99550, "size" : 500,
"query" : {
"term" : { "CC_ENGAGEMENT_NUMBER" : "1967" }
}
}
And below are my java implementation.
public IndexSearchResult findByStudIdAndcollageId(final String studId, final String collageId,
Integer Page_Number_Start_Index, Integer Total_No_Of_Records) {
SearchSourceBuilder sourceBuilder = new SearchSourceBuilder();
List<Map<String, Object>> searchResults = new ArrayList<Map<String, Object>>();
IndexSearchResult indexSearchResult = new IndexSearchResult();
try {
QueryBuilder qurBd = new BoolQueryBuilder().minimumShouldMatch(2)
.should(QueryBuilders.matchQuery("STUD_ID", studId).operator(Operator.AND))
.should(QueryBuilders.matchQuery("CLG_ID", collageId).operator(Operator.AND));
sourceBuilder.from(Page_Number_Start_Index).size(Total_No_Of_Records);
sourceBuilder.query(qurBd);
sourceBuilder.sort(new FieldSortBuilder("ROLL_NO.keyword").order(SortOrder.DESC));
SearchRequest searchRequest = new SearchRequest();
searchRequest.indices("clgindex");
searchRequest.source(sourceBuilder);
SearchResponse response;
response = rClient.search(searchRequest, RequestOptions.DEFAULT);
response.getHits().forEach(searchHit -> {
searchResults.add(searchHit.getSourceAsMap());
});
indexSearchResult.setListOfIndexes(searchResults);
log.info("searchResultsHits {}", searchResults.size());
} catch (Exception e) {
log.error("search :: Search on clg flat index. {}", e.getMessage());
}
return indexSearchResult;
}
So if the limit from 99550 and size 500 then it will not fetch more that 1L records.
Error: "reason" : "Result window is too large, from + size must be less than or equal to: [100000] but was [100050]. See the scroll api for a more efficient way to request large data sets. This limit can be set by changing the [index.max_result_window] index level setting."
}
I don't want to change [index.max_result_window]. Only want solution at Java side to search all docs in index based on conditions by implementing elasticserach API.
Thanks in advance..

Creating POJO/beans dynamically and set values using CGLib

I have a requirement to parse a text file and generate JSON document. The text file has a pattern of text which contains a key which is a name and the value is a huge text of TSV with headers.
I could parse the text file and generate bean classes using the headers and now i want to set the data to this generated bean class. I am using reflection to do this.
Class<?> beanClass = BeanClassGenerator.beanGenerator(k, mapForBeanGeneration);
try {
Object beanClassObject = beanClass.newInstance();
lines.forEach(line -> {
if (line != null && !line.isEmpty() && !line.equals("null")) {
String[] lineData = line.split("\t");
System.out.println("LineData length :: " + lineData.length);
Method[] methods = beanClass.getMethods();
System.out.println("Methods length :: " + methods.length);
int index = 0;
for (Method m : methods) {
m.setAccessible(true);
if (m.getName().startsWith("set")) {
try {
if ((lineData.length <= index) && lineData[index] != null) {
m.invoke(beanClassObject, lineData[index]);
index++;
} else {
m.invoke(beanClassObject, " ");
}
} catch (IllegalAccessException | InvocationTargetException e) {
e.printStackTrace();
}
}
}
}
});
ObjectMapper om = new ObjectMapper();
System.out.println(om.writeValueAsString(beanClassObject));
} catch (InstantiationException | IllegalAccessException | JsonProcessingException e) {
e.printStackTrace();
}});
The problem with the approach is that most of the times all the column values may not have data it can be nulled.
I am wondering if there is an easier way of doing this. Any help is appreciated.
Here is the bean generation method.
public static Class<?> beanGenerator(final String className, final Map<String, Class<?>> properties) {
BeanGenerator beanGenerator = new BeanGenerator();
beanGenerator.setNamingPolicy(new NamingPolicy() {
#Override
public String getClassName(String prefix, String source, Object key, Predicate names) {
return className;
}
});
BeanGenerator.addProperties(beanGenerator, properties);
return (Class<?>) beanGenerator.createClass();
}
Here is the sample text file which needs to be converted to the JSON output.
<Data1>
Col1 col2 col3 col4 col5
even sense met has
root greatest spin mostly
gentle held introduced palace
cold equator remember grandmother
slightly butter depth like
distant second coast everyone
<Data2>
Col1 col2 col3 col4 col5 col6 col7 col8
greatest rope operation flies brown continent combination read
slightly diagram he grandfather where party fifty pour
well put plastic anyway refer careful correct furniture
how since army tongue birthday been clock official
table command specific distant cutting hill movie experience
national though stopped youth army underline five know
<Data3>
Col1 col2 col3 col4 col5 col6 col7 col8 col9 col9 col10
vessels characteristic ship joy than tomorrow high seven future trade
try gray fourth advice week stream motion musical whom tin
limited daughter large rice came home chicken wheat engine box
easy city pair strange stage visitor coach announced allow simple
jet therefore single during construction flag bigger muscle complex pleasure
income several coat range dull cattle damage jump present shake
JSON output:
[{
"<Data1>": [{
"col1": "",
"col2": "",
"col3": "",
"col4": ""
},
{
"col1": "",
"col2": "",
"col3": "",
"col4": ""
},
{
"col1": "",
"col2": "",
"col3": "",
"col4": ""
}
]
}, {
"<Data2>": [{
"col1": "",
"col2": "",
"col3": "",
"col4": "",
"col5": "",
"col6": "",
"col7": "",
"col8": ""
},
{
"col1": "",
"col2": "",
"col3": "",
"col4": "",
"col5": "",
"col6": "",
"col7": "",
"col8": ""
},
{
"col1": "",
"col2": "",
"col3": "",
"col4": "",
"col5": "",
"col6": "",
"col7": "",
"col8": ""
}
]
}]
I came up with a solution using the Maps.
Map<String, List<Map<String, String>>> finalMap = new HashMap<>();
metadataMap.forEach((k, v) -> {
List<Map<String, String>> datamap = new ArrayList<>();
String key = k;
String[] fields = v.getFields();
List<String> lines = v.getLines();
lines.forEach(line -> {
if (line != null && !line.isEmpty() && !line.equals("null")) {
String[] fieldData = line.split("\t");
Map<String, String> eachLineMap = new HashMap<>();
for (int index = 0; index < fields.length; index++) {
if (index < fieldData.length && (fieldData[index] != null && !fieldData[index].isEmpty())) {
eachLineMap.put(fields[index], fieldData[index]);
} else {
eachLineMap.put(fields[index], " ");
}
datamap.add(eachLineMap);
}
}
});
finalMap.put(key, datamap);
});
try {
output = new ObjectMapper().writeValueAsString(finalMap);
}catch(JsonProcessingException e){
e.printStackTrace();
}

You are going way overboard with your solution.
Your data is organized as an array of variable length arrays;
and does not require some crazy on-the-fly class generation solution.
As a side note,
on-the-fly class generation is not inherently crazy;
it is crazy to use on-the-fly class generation in this situation.
Do this:
Look at your data;
it is organized as follows:
first: outer key
second: exactly one line containing a variable number of space separated array of inner keys.
third: some number of lines containing values.
Design a solution to fix your problem
Read the outer key.
Use that value to create the outer key portion of your JSON.
Read the inner keys.
Store these in an array;
use LinkedList,
not ClownList (ArrayList).
Do this until the next empty line:
Read a line of values.
Write the inner JSON; use the inner keys as the keys for this.
Skip empty lines until one of the following:
If at end of file, write the ending portion of the JSON.
If you read the next outer key, goto to line 2 (Read the inner keys) above.
Write the code.

You don't need to write all that logic, you can just use Apache Commons BeanUtils; which provides a utility method (among MANY other utilities), that takes a Map of field names versus field values and populate a given bean with it:
BeanUtils.populate(target, fieldNameValueMap);
Then the only thing you need to implement is the logic to create the fieldNameValueMap Map; which you can do with this simple method:
Map<String, String> createFieldNameValueMap(String headerLine, String valuesLine) {
String[] fieldNames = headerLine.split("\t");
String[] fieldValues = valuesLine.split("\t");
return IntStream.range(0, fieldNames.length)
.mapToObj(Integer::new)
.collect(Collectors.toMap(idx -> fieldNames[idx], idx -> fieldValues[idx]));
}
You can test this solution with the following working demo:
import java.lang.reflect.InvocationTargetException;
import java.util.Map;
import java.util.stream.Collectors;
import java.util.stream.IntStream;
import org.apache.commons.beanutils.BeanUtils;
import lombok.Data;
public class DynamicBeanUtils {
static Map<String, String> createFieldNameValueMap(String headerLine, String valuesLine) {
String[] fieldNames = headerLine.split("\t");
String[] fieldValues = valuesLine.split("\t");
return IntStream.range(0, fieldNames.length)
.mapToObj(Integer::new)
.collect(Collectors.toMap(idx -> fieldNames[idx], idx -> fieldValues[idx]));
}
public static void main(String[] args) {
String headerLine = "booleanValue\tintValue\tstringValue\tdoubleValue\totherValue";
String valuesLine = "true\t12\tthis bean will be populated\t22.44\ttest string!!!";
Object target = new MyBean();
try {
BeanUtils.populate(target, createFieldNameValueMap(headerLine, valuesLine));
} catch (IllegalAccessException | InvocationTargetException e) {
// HANDLE EXCEPTIONS!
}
System.out.println(target);
}
#Data
public static class MyBean {
private String stringValue;
private double doubleValue;
private int intValue;
private boolean booleanValue;
private String otherValue;
}
}
This is the maven repository page for this dependency, so you can include it in your build: https://mvnrepository.com/artifact/commons-beanutils/commons-beanutils/1.9.3
I used Lombok in this solution as well, only to save me the pain of writing getter/setters/toString to test this solution; but it is not required for your solution.
Complete code on GitHub
Hope this helps.

I realized that instead of creating the POJOs with a complex approach. It is better to use the Maps and convert them to JSON using Jackson ObjectMapper. Posting for others who think this might be a useful approach.
public String convert(Map<String, ? extends Metadata> metadataMap) {
String output = "";
Map<String, List<Map<String, String>>> finalMap = new HashMap<>();
metadataMap.forEach((k, v) -> {
List<Map<String, String>> datamap = new LinkedList<>();
String key = k;
String[] fields = v.getFields();
List<String> lines = v.getLines();
lines.forEach(line -> {
if (line != null && !line.isEmpty() && !line.equals("null")) {
String[] fieldData = line.split("\t",-1);
Map<String, String> eachLineMap = new HashMap<>();
for (int index = 0; index < fields.length; index++) {
if (index < fieldData.length && (fieldData[index] != null && !fieldData[index].isEmpty())) {
eachLineMap.put(fields[index], fieldData[index]);
} else {
eachLineMap.put(fields[index], " ");
}
datamap.add(eachLineMap);
}
}
});
finalMap.put(key, datamap);
});
try {
output = new ObjectMapper().writeValueAsString(finalMap);
}catch(JsonProcessingException e){
e.printStackTrace();
}
return output;
}

Calling an actual API with vertx in Test

I want to check the data returned by the call, its suppose to be an array with name "data"
"success": true,
"serverName": "mma.tour.com",
"data": [
[
"01.08.2018",
[
"01.08"
],
"Ср",
8,
"09.08",
[....... etc
My api call goes like this
public static JsonObject clientTest() {
Vertx vertx = Vertx.vertx();
WebClient client = WebClient.create(vertx);
client
.get("https://www.mma-tour.com/tariffsearch/getResult?" +
"priceMin=0&priceMax=1500000&currency=533067&nightsMin=6&nightsMax=8" +
"&hotelClassId=269506&accommodationId=2&rAndBId=15350&tourType=1&" +
"locale=ru&cityId=786&countryId=1104&after=01.08.2018&before=01.08.2018&" +
"hotelInStop=false&specialInStop=false&version=2&tourId=1285&" +
"tourId=12689&tourId=12706&tourId=143330&tourId=9004247&" +
"tourId=4433&tourId=5736&tourId=139343&tourId=4434&tourId=12691&" +
"tourId=21301&tourId=12705&tourId=149827&tourId=4151426&hotelClassBetter=true&" +
"rAndBBetter=true&noTicketsTo=false&noTicketsFrom=false&searchTypeId=3&" +
"recommendedFlag=false&salePrivateFlag=false&onlineConfirmFlag=false&contentCountryId=1102")
.send(ar -> {
if (ar.succeeded()) {
// Obtain response
HttpResponse<Buffer> response = ar.result();
Tour tour = new Tour.TourBuilder().build();
JsonObject myTourJson = response.bodyAsJsonObject();
myTourJson.getJsonArray("data"); //here
} else {
}
});
return null;
}
Above i tried to get data and write a test to check some data is returned but it fails during debug doesnt even get to "data"

How do I look up a cognito user by their sub/UUID?

I want to look up a user in my Cognito user pool by their sub, which as far as I can tell, is just their UUID. I would like to do this in Java within a Lambda function but cannot find how to do this in AWS's documenation. Any thoughts?

Now it works.
http://docs.aws.amazon.com/cognito-user-identity-pools/latest/APIReference/API_ListUsers.html
"sub" in list of supported attributes.
Example for JavaScript:
var cog = new AWS.CognitoIdentityServiceProvider();
var filter = "sub = \"" + userSub + "\"";
var req = {
"Filter": filter,
"UserPoolId": "your pool id" // looks like us-east-9_KDFn1cvys
};
cog.listUsers(req, function(err, data) {
if (err) {
console.log(err);
}
else {
if (data.Users.length === 1){ //as far as we search by sub, should be only one user.
var user = data.Users[0];
var attributes = data.Users[0].Attributes;
} else {
console.log("Something wrong.");
}
}
});

As of today this is not possible with Cognito User Pools.
Users can only be looked up using their username or aliases. ListUsers API also allows users to be searched by providing search filters on some standard attributes but sub is not one of them.

// class var
protected final AWSCognitoIdentityProviderClient identityUserPoolProviderClient;
// initialize the Cognito Provider client. This is used to talk to the user pool
identityUserPoolProviderClient = new AWSCognitoIdentityProviderClient(new BasicAWSCredentials(AWS_ACCESS_KEY, AWS_SECRET_KEY));
identityUserPoolProviderClient.setRegion(RegionUtils.getRegion(USER_POOL_REGION));
// ...some init code omitted
// build the request
AdminGetUserRequest idRequest = new AdminGetUserRequest();
idRequest.withUserPoolId(USER_POOL_ID);
idRequest.withUsername(username);
// call cognito for the result
AdminGetUserResult result = identityUserPoolProviderClient.adminGetUser(idRequest);
// loop through results
List<UserType> userTypeList = result.getUsers();
// loop through them
for (UserType userType : userTypeList) {
List<AttributeType> attributeList = userType.getAttributes();
for (AttributeType attribute : attributeList) {
String attName = attribute.getName();
String attValue = attribute.getValue();
System.out.println(attName + ": " + attValue);
}
}

Old question, but you the username parameter is overloaded in Cognito's adminGetUser method. It is, unfortunately, not documented: adminGetUser SDK
Here's a snippet:
const params = {
UserPoolId: 'someUserPoolId'
Username: 'random-string-sub-uuid',
};
CognitoService.adminGetUser(params,(err, data) => {
console.log(data);
})
Returns:
{ Username: 'random-string-sub-uuid',
UserAttributes:
[ { Name: 'sub', Value: 'random-string-sub-uuid' },
{ Name: 'custom:attributeName', Value: 'someValue' },
{ Name: 'email_verified', Value: 'false' },
{ Name: 'name', Value: 'nameValue' },
{ Name: 'email', Value: 'user#stackoverflow.com' } ],
UserCreateDate: 2018-10-12T14:04:04.357Z,
UserLastModifiedDate: 2018-10-12T14:05:03.843Z,
Enabled: true,
UserStatus: 'CONFIRMED' }

FullCalendar - Adding Multiple Events or Add Event Source

I have looked through the post in stackoverflow to add events into FullCalendar, however I am really new and find it really difficult to understand without an example. In short, is anyone here able to dumb it down for me, in order to add an array of objects into the FullCalendar?
I would like to add appointments that I have created Appointment(Date date, String name, String phoneNo). So they are retrieved in list:
PersistenceManager pm = PMF.get().getPersistenceManager();
String query = "select from " + Appointment.class.getName();
query += " where merchant == '" + session.getAttribute("merchant") + "'";
List<Appointment> appointment = (List<Appointment>) pm.newQuery(query).execute();
How am I able to populate the FullCalendar plugin with the list I obtained? Thanks a lot!

if anyone encounters the same problem as I do - you have a list of java objects and want it to populate FullCalendar, here's the solution:
JSP Page
$(document).ready(function() {
var calendar = $('#calendar').fullCalendar({
header: {
left: 'prev,next today',
center: 'title',
right: 'month,agendaWeek,agendaDay'
},
selectable: true,
selectHelper: true,
select: function(start, end, allDay) {
var title = prompt('Event Title:');
if (title) {
calendar.fullCalendar('renderEvent',
{
title: title,
start: start,
end: end,
allDay: allDay
},
true // make the event "stick"
);
}
calendar.fullCalendar('unselect');
},
editable: true,
eventSources: [
{
url: '/calendarDetails',
type: 'GET',
data: {
start: 'start',
end: 'end',
id: 'id',
title: 'title',
allDay: 'allDay'
},
error: function () {
alert('there was an error while fetching events!');
}
}
]
});
});
Please take not of the URL, it is the servlet URL
Servlet
public class CalendarServlet extends HttpServlet {
public void doGet(HttpServletRequest req, HttpServletResponse resp)
throws IOException {
String something = req.getSession().getAttribute("merchant").toString(); //get info from your page (e.g. name) to search in query for database
//Get the entire list of appointments available for specific merchant from database
//Convert appointment to FullCalendar (A class I created to facilitate the JSON)
List<FullCalendar> fullCalendar = new ArrayList<FullCalendar>();
for (Appointment a : appointment) {
String startDate = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss").format(a.getDate());
startDate = startDate.replace(" ", "T");
//Calculate End Time
Calendar c = Calendar.getInstance();
c.setTime(a.getDate());
c.add(Calendar.MINUTE, 60);
String endDate = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss").format(c.getTime());
endDate = endDate.replace(" ", "T");
FullCalendar fc = new FullCalendar(startDate, endDate, a.getId(), a.getName() + " # " + a.getPhone(), false);
fullCalendar.add(fc);
}
//Convert FullCalendar from Java to JSON
Gson gson = new Gson();
String jsonAppointment = gson.toJson(fullCalendar);
//Printout the JSON
resp.setContentType("application/json");
resp.setCharacterEncoding("UTF-8");
try {
resp.getWriter().write(jsonAppointment);
} catch (IOException e) {
e.printStackTrace();
}
}
}
if you need more info on JSON or GSON, check the comments above.

Melvin you have tones of examples here in Stack, try searching for add event sources.
From my experience im fullcalendar, you can add events through JSON, well formed XML and array's and i think thats it. You can use ajax calls do retrieve does 3 types of formats.
In your server side you should create a method to return a String already with the XML/JSON/array built so you can pass to you ajax call.

take a look at https://github.com/mzararagoza/rails-fullcalendar-icecube
this is done in rails but i think what you are looking for is
dayClick: function(date, allDay, jsEvent, view) {
document.location.href=new_event_link + "?start_date=" + date;
},
full jquery
$('#calendar').fullCalendar({
dayClick: function(date, allDay, jsEvent, view) {
document.location.href=new_event_link + "?start_date=" + date;
},
header: {
left: 'prev,today,next',
center: 'title',
right: 'month,agendaWeek,agendaDay'
},
selectable: true,
selectHelper: true,
editable: false,
ignoreTimezone: false,
select: this.select,
eventClick: this.eventClick,
eventDrop: this.eventDropOrResize,
eventSources: [
{
url: '/event_instances.json',
data: {
custom_param1: 'something',
custom_param2: 'somethingelse'
},
error: function() {
alert('there was an error while fetching events!');
}
}
],
eventResize: this.eventDropOrResize
});