We Keep Coding

How to limit number of threads all concurrent methods at once

I am working on a legacy project which uses Java 8, Spring, HikariCP, and MySQL. Microservices' methods are triggered with a Kafka topic and start a reporting operation. Almost all triggered methods have this and some of them have the same usage inside their blocks.
new ForkJoinPool().submit(() -> { users.parallelStream().forEach(user ->
The application creates 8-9k threads and all of them try to get or create a record. However, the database couldn't handle these requests and started to throw exceptions and Zabbix sends mails about heap memory usage above %90:
Caused by: java.sql.SQLTransientConnectionException: HikariPool-2 -
Connection is not available, request timed out after 30000ms.
When I check the database and see the variable for max_connections = 600, but this is not enough.
I want to set a limit for thread count for the application level.
I tried setting these parameters but the thread size doesn't decrease.
SPRING_TASK_EXECUTION_POOL_QUEUE-CAPACITY , SPRING_TASK_EXECUTION_POOL_MAX-SIZE, -Djava.util.concurrent.ForkJoinPool.common.parallelism
Is there any property to solve this problem?

I have changed all new ForkJoinPool() to ForkJoinPool.commonPool() and use this parameter to control thread creation -Djava.util.concurrent.ForkJoinPool.common.parallelism after that I have fixed my problem.

Java Scheduler to Mail

Given that I have two scheduled component classes uploading files respectively.
I created a sending email method for each of them in order to send a reminder email to myself in case any uploading exceptions happened.
the flow like this:
Scheduler One --- if exception during uploading ---> sending a email after exception
Scheduler Two --- if exception during uploading ---> sending a email after exception
now I want to upgrade as
Scheduler One + Scheduler Two
--if exception--> sending a mail after two scheduler
Nonetheless, how can I do that?

You use case sounds really odd. Schedulers run independent. So if you want to share information (an exception was thrown) between both thos you have to store this information somewhere. A entry in a database or saving in in a global variable during runtime.
I would however suggest that you merge both of you scheduler into one. If they are not independent why divide the code? It saves you from creating theses hacks where the schedulers need to be connected

Why am I getting error creating object in Minio?

I'm trying to create an object in Minio. Sometimes it works and sometimes no.
When I execute :
minioClient.makeBucket(bucketName); // bucketName es a random unique string
I don´t get error, but if If ask Minio if bucket exists:
minioClient.bucketExists(bucketName)
it returns false
After that, If I try to launch once again minioClient.makeBucket(bucketName) because minioClient.bucketExists(bucketName) is returning false, then I get this error (simplified):
BucketAlreadyOwnedByYou
So that, when I get previous error, I try to create object in Minio :
minioClient.putObject(bucketName, objectName, is,null,customMetaData,null, null ); // is is an InputStream of the file I want to store in Minio
But I get this error, saying bucket does not exist;
ErrorResponse(code=NoSuchBucket, message=The specified bucket does not exist, bucketName=888bf891-064f-4270-b0f0-85514c0eae02, objectName=L7aEmF8Ppk, resource=/888bf891-064f-4270-b0f0-85514c0eae02/L7aEmF8Ppk, requestId=16733C7840E6854D, hostId=null)
Any help will be appreciated.
I'm using version 6.0.13 of Minio Java API

This not how MinIO should behave. I have only seen this error where it was either a programming error or a setup error.
When it is a setup error, the root problem is that instead of creating 1 server pool with 4 distributed nodes, people have created 4 server pools each with a single node. This means that when your application gets load balanced, it will sometimes hit the server pool where you created the bucket and sometimes not.
Another cause can be an non-consistent filesystem used as backend. For example NFS with certain settings can have behavior like this. The recommended filesystem is XFS.
Without knowing more about your setup I cannot tell you which one it is.

Kinesis Java consumer - keepalive/heartbeat?

Ive written a Java service that consumes from a Kinesis topic. The service starts / runs well and consumes happily as long as the data doesn't get too infrequent. If there is a gap of > 60-90 minutes it stops consuming. No errors are issued but subsequent data queues up in Kinesis and sits until the service is restarted.
Does Kinesis have some sort of heartbeat or keepalive message that needs to send during these quiet periods?
I looked through the configuration (KinesisClientLibConfiguration) and didn't see anything obvious. Hopefully this won't entail cycling the connection on an hourly basis.
Edit:
KinesisClientLibConfiguration kinesisClientLibConfiguration =
new KinesisClientLibConfiguration(config.getString("appname"),
config.getString("kinesis/stream_name"),
kinesisCredentialsProvider, localProvider,
localProvider, workerId);
kinesisClientLibConfiguration.withInitialPositionInStream(
InitialPositionInStream.TRIM_HORIZON);
Edit:
I managed to find some error output - lots of these:
com.amazonaws.services.kinesis.clientlibrary.lib.worker.ProcessTask call
SEVERE: ShardId shardId-000000000000: Caught exception:
com.amazonaws.services.kinesis.model.AmazonKinesisException: The security token included in the request is expired (Service: AmazonKinesis; Status Code: 400; Error Code: ExpiredTokenException; Request ID: cdb95cb6-23bb-0067-9c7b-1ad1125d7b7e)
These messages start precisely 60 minutes after the app starts. Found this reference saying to 'refresh 5 minutes before expiration'. Given that I have two kinds of credentials in this call (one for kinesis and one for dynamodb/cloud watch) I'll try a timer to .refresh().

(here's what worked)
Note: this involves two Credentials sources - a local and a remote. The local ones are for DynamoDB and CloudWatch. The remote is for Kinesis.
AWSCredentialsProvider localProvider = new ClasspathPropertiesFileCredentialsProvider("credentials");
STSAssumeRoleSessionCredentialsProvider stsRoleCredentials = new STSAssumeRoleSessionCredentialsProvider.Builder(
config.getString("kinesis/arn"), config.getString("kinesis/role_session_name"))
.withExternalId(config.getString("kinesis/external_id")).build();
KinesisClientLibConfiguration kinesisClientLibConfiguration = new KinesisClientLibConfiguration(
config.getString("appname"),
config.getString("kinesis/stream_name"),
stsRoleCredentials, localProvider, localProvider, workerId);
If you use a CredentialsProvider it will do the token refresh for you. After some exploration I found that the .refresh() calls I was making were to empty functions.

AWS Error Message: InvalidInstanceID.NotFound

I'm trying to start a Amazon EC2 cloud machine with [startInstance][2] method using aws-sdk in Java. My code is as follows.
public String startInstance(String instanceId) throws Exception {
List<String> instanceIds = new ArrayList<String>();
instanceIds.add(instanceId);
StartInstancesRequest startRequest = new StartInstancesRequest(
instanceIds);
startRequest.setRequestCredentials(getCredentials());
StartInstancesResult startResult = ec2.startInstances(startRequest);
List<InstanceStateChange> stateChangeList = startResult
.getStartingInstances();
log.trace("Starting instance '{}':", instanceId);
// Wait for the instance to be started
return waitForTransitionCompletion(stateChangeList, "running",
instanceId);
}
When I run the above code, i'm getting the following AWS error:
Status Code: 400, AWS Request ID: e1bd4795-a609-44d1-9e80-43611e80006b, AWS Erro
r Code: InvalidInstanceID.NotFound, AWS Error Message: The instance ID 'i-2b97ac
2f' does not exist
at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpCli
ent.java:538)
at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.ja
va:283)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:168
)
at com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.jav
a:5208)
at com.amazonaws.services.ec2.AmazonEC2Client.startInstances(AmazonEC2Cl
ient.java:2426)

AWS Error Message: The instance ID 'i-2b97ac2f' does not exist
You'll have to take the AWS response for granted here, i.e. the instance does not exist ;)
But seriously: Presumably you have already verified that you are actually running an instance with this ID in your account? Then this is most likely caused by targeting the wrong API endpoint, insofar an instance ID is only valid within a specific region (if not specified, the region defaults to 'us-east-1', see below).
In this case you need to specify the actual instance region via the setEndpoint() method of the AmazonEC2Client object within the apparently global ec2 variable before calling startInstances().
There are some examples regarding Using Regions with the AWS SDKs and all currently available AWS regional endpoint URLs are listed in Regions and Endpoints, specifically the Amazon Elastic Compute Cloud (EC2) defaults to 'us-east-1':
If you just specify the general endpoint (ec2.amazonaws.com), Amazon
EC2 directs your request to the us-east-1 endpoint.

We run a service (Qubole) that frequently spawns and then tags (and in some cases terminates) AWS instances immediately.
We have found that Amazon will, every once in a while, claim an instanceid as invalid - even though it has just created it. Retrying a few times with some sleep time thrown in usually solves the problem. Even a total retry interval of 15s proved insufficient in rare cases.
This experience comes from the useast region. We do not make api calls to different regions - so that is not an explanation. More likely - this is the infamous eventual consistency at work - where AWS is unable to provide read-after-write consistency for these api calls.

I am using the AWS ruby api and I noticed the same issue when creating an AMI image and its status is pending when I look in the AWS console but after a while the image is available for use.
Here is my script
image = ec2.images.create(:name => image_name, :instance_id => ami_id, :description => desc)
sleep 5 while image.state != :available
I sleep for about 5 sec for image to be in available but I get the error saying that the "AWS Error Message: InvalidInstanceID.NotFound". During my testing this is fine but most of the time this seems to be failing during continuous integration builds.

InvalidInstanceID.NotFound means the specified instance does not exist.
Ensure that you have indicated the region in which the instance is located, if it's not in the default region.
This error may occur because the ID of a recently created instance has not propagated through the system. For more information, see Eventual Consistency.