We Keep Coding

How to prevent IllegalStateException in reactor when you need to block

We have a synchronous process that needs to call two REST endpoints, whereas the result of the first is needed for the second. Using Springs WebClient the .block() causes the following exception:
java.lang.IllegalStateException: block()/blockFirst()/blockLast() are blocking, which is not supported in thread parallel-2
How can this be prevented?
Here is a simplified code snippet:
var job = webClient.createJob().block();
if (job == null || StringUtils.isBlank(job.getId())) {
throw new Exception("WebClient did not return with a job id");
}
batchRecords(job.getId(), records);// does some additional calls to the webClient
This works in the unit test, but when called through a #RestController the above exception is thrown.
EDIT:
The batchRecords method currently also has blocking Monos in it, so we can have a delay in between:
public void batchRecords(final String jobId, final List<InventoryRecord> records)
var recordCount = 0;
var inventoryPositions = new ArrayList<InventoryPosition>();
var recordIterator = records.iterator();
while (recordIterator != null && recordIterator.hasNext()) {
var inventoryRecord = recordIterator.next();
inventoryPositions.add(mapInventoryPosition(inventoryRecord));
recordCount++;
if (inventoryPositions.size() == batchSize) {
var response = createBatch(jobId, inventoryPositions);
Thread.sleep(sleepTime);
response.block();
inventoryPositions = new ArrayList<>();
}
}
}

You should do it reactively without blocking:
webClient.createJob()
.filter(job -> !StringUtils.isBlank(job.getId()))
.flatMap(job -> batchRecords(job.getId(), records))
.switchIfEmpty(Mono.error(new Exception("WebClient did not return with a job id")));
As soon as the createJob operation is finished, the result is filtered and provided to the flatMap operator. In case of an empty response (Mono.empty()) an exception is thrown.

The incoming request has too many parameters. The server supports a maximum of 2100 parameters in Grails

I use MSSQL witch has constaints. And I get error:
Class
com.microsoft.sqlserver.jdbc.SQLServerException
Message
The incoming request has too many parameters. The server supports a maximum of 2100 parameters. Reduce the number of parameters and resend the request.
In this plase:
final Set<User> unavailableUsers = ctx.delegationService.getUnavailableUserVariants()
if (unavailableUsers) {
inList 'key', unavailableUsers*.key
}
unavailableUsers has many Users about 2500.
How to solve this problem?
I try to do chunks of array
final Set<User> unavailableUsers = ctx.delegationService.getUnavailableUserVariants()
List<Set<User>> output = new ArrayList<>()
Set<User> currSet = null
for (CatalogUser value : unavailableUsers) {
if (currSet == null || currSet.size() == 1000)
output.add(currSet = new HashSet<User>());
currSet.add(value);
}
output.each {it ->
if (it) {
inList 'key', it*.key
}
}
But it doesn't help

Jenkins api too slow

Im using Jenkins on my local in docker from jenkins official docker hub (but I even tried jenkins we have on bluemix instance).
Im writing program (test driven currently) to trigger job from java and then get job id, using
jenkins api.
Properties jenkinsProps = new Properties();
InputStream jenkinsPropsIs = Files.newInputStream(jenkinsPropsFilePath);
jenkinsProps.load(jenkinsPropsIs);
// for building url
String jenkinsServerUrl = jenkinsProps.getProperty(JenkinsPropertiesKeys.KEY_JENKINS_SERVER_URL);
String jobName = jenkinsProps.getProperty(JenkinsPropertiesKeys.KEY_JENKINS_JOB_NAME);
String jobRemoteAccessToken = jenkinsProps.getProperty(JenkinsPropertiesKeys.KEY_JENKINS_JOB_ACCESS_TOKEN);
// for headers
String jenkinsUser = jenkinsProps.getProperty(JenkinsPropertiesKeys.KEY_JENKINS_USERNAME);
String jenkinsUserApiToken = jenkinsProps.getProperty(JenkinsPropertiesKeys.KEY_JENKINS_API_TOKEN);
String jenkinsCrumb = jenkinsProps.getProperty(JenkinsPropertiesKeys.KEY_JENKINS_CSRF_CRUMB);
// build parameters
Map<String, String> params = new LinkedHashMap<>();
params.put("param1", "test1");
params.put("param2", "test2");
params.put("param3", "test3");
// Jenkins cause - to identify which process had run this job
String procID = UUID.randomUUID().toString();
params.put("cause", procID);
String url = getJenkinsBuildWithParametersUrl(jenkinsServerUrl, jobName, jobRemoteAccessToken, params);
WebRequest request = new WebRequest(); // own HttpConnection based client
// setup Jenkins crumb to avoid CSRF
request.setHeader(HEADER_NAME_JENKINS_CRUMB, jenkinsCrumb);
// user authentification (Basic + base64 encoded user:apiToken)
setupAuthenticationHeader(request, jenkinsUser, jenkinsUserApiToken);
// execute POST request
request = request.post(url);
// asserts
assertNotNull(request);
assertEquals(201, request.getResponseCode());
/* GET JOB ID */
Thread.currentThread().sleep(8000); // !!! if less then 8sec, jenkins returns old job number
request.reset();
setupAuthenticationHeader(request, jenkinsUser, jenkinsUserApiToken);
url = getJenkinsLastBuildUrl(jenkinsServerUrl, jobName);
// execute get request to /api/json
request = request.get(url);
assertTrue(request.isOK());
// get note & compare with proc id, to match job
String jenkinsJobProcId = null;
JsonObject jenkinsLastBuildJson = request.getResponseAsJson();
JsonArray jenkinsActions = jenkinsLastBuildJson.get("actions").getAsJsonArray();
for (JsonElement action : jenkinsActions) {
JsonObject actionJson = action.getAsJsonObject();
if (actionJson.get("_class").getAsString().equals("hudson.model.CauseAction")) {
JsonArray causeActionJsonArray = actionJson.get("causes").getAsJsonArray();
for (JsonElement cause : causeActionJsonArray) {
JsonObject causeJson = cause.getAsJsonObject();
if (causeJson.get("_class").getAsString().equals("hudson.model.Cause$RemoteCause")) {
jenkinsJobProcId = causeJson.get("note").getAsString();
break;
}
}
if (!jenkinsJobProcId.isEmpty()) {
break;
}
}
}
System.out.println("LastBuild prodId : " + jenkinsJobProcId);
assertEquals(procID, jenkinsJobProcId);
// get jenkins build number
int lastBuildNumber = jenkinsLastBuildJson.get("number").getAsInt();
System.out.println("LastBuild buildNumber : " + lastBuildNumber);
assertTrue(lastBuildNumber > 0);
Once i trigger job, it takes like 8 sec, to job apear in /api/json.
Do you know what can be the problem ?
How to tune it up ?

Please check if you still need a delay between the two executions.
trigger the job
curl -X POST http://${JENKINS_HOTS}:${JENKINS_PORT}/job/${JOB_NAME}/build \
--user ${USER}:${PASSWORD} \
--data-urlencode json='{"parameter": [{"name":"delay", "value":"0sec"}]}'
get job info
curl http://${JENKINS_HOTS}:${JENKINS_PORT}/job/${JOB_NAME}/api/json \
--user ${USER}:${PASSWORD}
If you still need to wait around 8 seconds, check the setting of the quiet period in the job. If it's not yet enabled, enable it and set the period to 0 seconds. This should remove the delay between the executions.
Depending on the workload off the Jenkins instance it might be necessary, even with a period of zero seconds, that you need to wait a short period.

Using Java Stream in specific case with mapping and filtering response codes

My question is regarding lambda with JAVA8 and use of the filters.
It is done via Selenium of Java for testing for different response codes.
How can I use Lambda in greatest possible way to transform following function with Streams?
The code which I wanted to refactor is as follows into Streams , lambda of Java 8:
for (int i = 0; i < links.size(); i++) {
if (!(links.get(i).getAttribute("href") == null) && !(links.get(i).getAttribute("href").equals(""))) {
// Find HTTP Status-Code
try {
statusCode = getResponseCode(links.get(i).getAttribute("href").trim());
} catch (IOException e) {
e.printStackTrace();
}
// Check broken link
if (statusCode== 404) {
System.out.println("Broken of Link# "+i+" "+links.get(i).getAttribute("href"));
}
else if (statusCode== 400) {
System.out.println("Bad Request# "+i+" "+links.get(i).getAttribute("href"));
}
else if (statusCode== 401) {
System.out.println("Unauthorized# "+i+" "+links.get(i).getAttribute("href"));
}
else if (statusCode== 403) {
System.out.println("Forbidden# "+i+" "+links.get(i).getAttribute("href"));
}
else if (statusCode== 500) {
System.out.println("Internal server error# "+i+" "+links.get(i).getAttribute("href"));
}
}
}
What I have for now is:
List<AbstractMap.SimpleImmutableEntry<String,Integer>> variablename =
links.stream().map(WebElement::getAttribute("href"));
I was trying to do something along the lines of filtering out everything that is not 500,403,401,400,404 and only keeping the mapping or a Pair of that like (linkString, responseCode), but I am having a bit of troubles exactly how I could do it properly with Lambda?
EDIT1:
I didn't meant to put everything via stream, just to make as much use of it as I can in this example

It's fairly simple if you take it piece by piece, so:
// create a set of codes you want to include
Set<Integer> acceptableCodes = new HashSet<>(Arrays.asList(404, 400, 401, 403, 500));
// for (int i = 0; i < links.size(); i++) {
links.stream()
// convert to the href value as that's all we need later on
.map(link -> link.getAttribute("href"))
// filter out anything without a href
// if (!(links.get(i).getAttribute("href") == null) && !(links.get(i).getAttribute("href").equals(""))) {
.filter(href -> href != null)
.filter(href -> !href.equals(""))
// filter out non-matching status codes
.filter(href -> acceptableCodes.contains(getResponseCode(href))
.map(link -> new LinkWithCode(href , getResponseCode(href))
.collect(toList());
Putting it together without the comments so you can read it easier:
Set<Integer> acceptableCodes = new HashSet<>(Arrays.asList(404, 400, 401, 403, 500));
links.stream()
.map(link -> link.getAttribute("href"))
.filter(href -> href != null)
.filter(href -> !href.equals(""))
.filter(href -> acceptableCodes.contains(getResponseCode(href))
.map(link -> new LinkWithCode(href , getResponseCode(href))
.collect(toList());
LinkWithCode is a class you'll need to create to hold the link and the status code. This assumes getResponseCode isn't a heavyweight operation as it's happening twice.
Caveat: I haven't tested this or put it in an IDE so you might have some tidy up to do.

elasticsearch java bulk batch size

I want to use the elasticsearch bulk api using java and wondering how I can set the batch size.
Currently I am using it as:
BulkRequestBuilder bulkRequest = getClient().prepareBulk();
while(hasMore) {
bulkRequest.add(getClient().prepareIndex(indexName, indexType, artist.getDocId()).setSource(json));
hasMore = checkHasMore();
}
BulkResponse bResp = bulkRequest.execute().actionGet();
//To check failures
log.info("Has failures? {}", bResp.hasFailures());
Any idea how I can set the bulk/batch size?

It mainly depends on the size of your documents, available resources on the client and the type of client (transport client or node client).
The node client is aware of the shards over the cluster and sends the documents directly to the nodes that hold the shards where they are supposed to be indexed. On the other hand the transport client is a normal client that sends its requests to a list of nodes in a round-robin fashion. The bulk request would be sent to one node then, which would become your gateway when indexing.
Since you're using the Java API, I would suggest you to have a look at the BulkProcessor, which makes it much easier and flexibile to index in bulk. You can either define a maximum number of actions, a maximum size and a maximum time interval since the last bulk execution. It's going to execute the bulk automatically for you when needed. You can also set a maximum number of concurrent bulk requests.
After you created the BulkProcessor like this:
BulkProcessor bulkProcessor = BulkProcessor.builder(client, new BulkProcessor.Listener() {
#Override
public void beforeBulk(long executionId, BulkRequest request) {
logger.info("Going to execute new bulk composed of {} actions", request.numberOfActions());
}
#Override
public void afterBulk(long executionId, BulkRequest request, BulkResponse response) {
logger.info("Executed bulk composed of {} actions", request.numberOfActions());
}
#Override
public void afterBulk(long executionId, BulkRequest request, Throwable failure) {
logger.warn("Error executing bulk", failure);
}
}).setBulkActions(bulkSize).setConcurrentRequests(maxConcurrentBulk).build();
You just have to add your requests to it:
bulkProcessor.add(indexRequest);
and close it at the end to flush any eventual requests that might have not been executed yet:
bulkProcessor.close();
To finally answer your question: the nice thing about the BulkProcessor is also that it has sensible defaults: 5 MB of size, 1000 actions, 1 concurrent request, no flush interval (which might be useful to set).

you need to count your bulk request builder when it hits your batch size limit then index them and flush older bulk builds .
here is example of code
Settings settings = ImmutableSettings.settingsBuilder()
.put("cluster.name", "MyClusterName").build();
TransportClient client = new TransportClient(settings);
String hostname = "myhost ip";
int port = 9300;
client.addTransportAddress(new InetSocketTransportAddress(hostname, port));
BulkRequestBuilder bulkBuilder = client.prepareBulk();
BufferedReader br = new BufferedReader(new InputStreamReader(new DataInputStream(new FileInputStream("my_file_path"))));
long bulkBuilderLength = 0;
String readLine = "";
String index = "my_index_name";
String type = "my_type_name";
String id = "";
while((readLine = br.readLine()) != null){
id = somefunction(readLine);
String json = new ObjectMapper().writeValueAsString(readLine);
bulkBuilder.add(client.prepareIndex(index, type, id).setSource(json));
bulkBuilderLength++;
if(bulkBuilderLength % 1000== 0){
logger.info("##### " + bulkBuilderLength + " data indexed.");
BulkResponse bulkRes = bulkBuilder.execute().actionGet();
if(bulkRes.hasFailures()){
logger.error("##### Bulk Request failure with error: " + bulkRes.buildFailureMessage());
}
bulkBuilder = client.prepareBulk();
}
}
br.close();
if(bulkBuilder.numberOfActions() > 0){
logger.info("##### " + bulkBuilderLength + " data indexed.");
BulkResponse bulkRes = bulkBuilder.execute().actionGet();
if(bulkRes.hasFailures()){
logger.error("##### Bulk Request failure with error: " + bulkRes.buildFailureMessage());
}
bulkBuilder = client.prepareBulk();
}
hope this helps you
thanks