Im building a app that with spring boot that uses mysql as db, as im seen is that spring boot opens 10 connections with the db but uses only one.
Every time i run show processlist on db, 9 connections are sleep and only one is doing something.
There's a way to split all the process between the 10 opened connections?
My app need better mysql processing because every minute about 300 records is inserted, so i think spliting between this opened connections will get better results.
My aplication.yml:
security:
basic:
enabled: false
server:
context-path: /invest/
compression:
enabled: true
mime-types:
- application/json,application/xml,text/html,text/xml,text/plain
spring:
jackson:
default-property-inclusion: non-null
serialization:
write-bigdecimal-as-plain: true
deserialization:
read-enums-using-to-string: true
datasource:
platform: MYSQL
url: jdbc:mysql://localhost:3306/invest?useSSL=true
username: #
password: #
separator: $$
dbcp2:
test-while-idle: true
validation-query: SELECT 1
jpa:
show-sql: false
hibernate:
ddl-auto: update
naming:
strategy: org.hibernate.cfg.ImprovedNamingStrategy
properties:
hibernate:
dialect: org.hibernate.dialect.MySQL5Dialect
http:
encoding:
charset: UTF-8
enabled: true
force: true
There's a way to do this?
You can check my.ini,[mysqld] max_connections ,make sure more connections are allowed by your mysql.
You can adjust these config in your application.yml
spring.datasource.max-active
spring.datasource.max-age
spring.datasource.max-idle
spring.datasource.max-lifetime
spring.datasource.max-open-prepared-statements
spring.datasource.max-wait
spring.datasource.maximum-pool-size
spring.datasource.min-evictable-idle-time-millis
spring.datasource.min-idle
After all, I do not think this is a problem . Mysql can handle 1000+ insertion per second. 300 per minute is too few for pool to open more than one connection.
Related
I have a spring boot app that read/writes to postgres database. I use jooq and hikariCP to manage the database connections. My apps is connected to a patroni cluster consisting of two Postgresql 14.5 instances - one is the master and the other one is a read-only replica.
When the service is processing data and I trigger a failover in the database -
killing the leader, choosing new leader, then changing the old leader to a replica - I start getting exceptions like
with\norg.jooq.exception.DataAccessException: SQL [delete from \"public\".\"my_table\" where \"public\".\"my_table\".\"username\" = ?]; ERROR: cannot execute DELETE in a read-only transaction
it looks like a hikariCP/jdbc driver issue where it is still using the connections to the old-master-now-replica instead of evicting them and creating new connections to the new leader.
How to solve it ?
My configuration looks like this:
org.jooq:jooq:3.16.10
org.postgresql:postgresql:42.5.0
org.jooq:jooq-postgres-extensions:3.16.10
com.zaxxer:HikariCP:4.0.3
spring:
main:
allow-bean-definition-overriding: true
banner-mode: "off"
jooq:
sql-dialect: Postgres
jpa:
open-in-view: false
datasource:
type: com.zaxxer.hikari.HikariDataSource
driver-class-name: org.postgresql.Driver
url: "jdbc:postgresql://my-db-cluster:5432/my-database?tcpKeepAlive=true&ApplicationName=my-app"
username: ${DATASOURCE_USERNAME}
password: ${DATASOURCE_PASSWORD}
hikari:
minimumIdle: 0
maximumPoolSize: 10
auto-commit: false
autoconfigure:
exclude: org.springframework.boot.autoconfigure.r2dbc.R2dbcAutoConfiguration
I found a solution by adding &targetServerType=primary to jdbc url as docs states: https://jdbc.postgresql.org/documentation/use/
This question already has answers here:
Apparent connection leak detected with Hikari CP
(2 answers)
Closed last month.
In my Spring boot application,
datasource:
oracle:
hikari:
jdbc-url: URL
username: Username
password: password
minimum-idle: 3
maximum-pool-size: 10
max-lifetime: 0
connection-test-query: select 1 from dual
leak-detection-threshold: 10000
driver-class-name: oracle.jdbc.OracleDriver
idleTimeout: 0
keepaliveTime: 300000
With this config, I'm getting error:
com.zaxxer.hikari.pool.ProxyLeakTask : Connection leak detection triggered for oracle.jdbc.driver.T4CConnection#48063c71 on thread taskExecutor-1, stack trace follows
java.lang.Exception: Apparent connection leak detected
Previously reported leaked connection oracle.jdbc.driver.T4CConnection#48063c71 on thread taskExecutor-1 was returned to the pool (unleaked)
The memory usage in AppMetrix has jumped to 90% which is extremely high. Now, if I run any other job, the issue will be Heap out of memory.
Any suggestions on how this can be fixed?
I think it is due to a low value of hikari.leakDetectionThreshold property (10000 ms i.e 10 seconds). Try increasing it to 1 minute (60000 ms).
I have uploaded a saved configuration file in a BeanStalk application in a region to another BeanStalk application in another region.
While loading that config I got an error
Stack named 'awseb-e-sme7w3eym3-stack' aborted operation. Current
state: 'CREATE_FAILED' Reason: The following resource(s) failed to
create: [AWSEBLoadBalancer]
Creating load balancer failed Reason: Property Listeners cannot be
empty Any idea about this issue ?
See the config file
AWSConfigurationTemplateVersion: 1.1.0.0
EnvironmentConfigurationMetadata:
DateCreated: '1580272974000'
DateModified: '1580273310143'
Description: xxxxxxxxxxxxxxxxxxxxx
EnvironmentTier:
Name: WebServer
Type: Standard
OptionSettings:
AWSEBAutoScalingGroup.aws:autoscaling:updatepolicy:rollingupdate:
MaxBatchSize: '1'
MinInstancesInService: '1'
RollingUpdateEnabled: true
RollingUpdateType: Health
AWSEBAutoScalingLaunchConfiguration.aws:autoscaling:launchconfiguration:
EC2KeyName: xxxxxxxxxxxxxxxxxxx
AWSEBCloudwatchAlarmHigh.aws:autoscaling:trigger:
UpperThreshold: '60'
AWSEBCloudwatchAlarmLow.aws:autoscaling:trigger:
BreachDuration: '2'
LowerThreshold: '25'
MeasureName: CPUUtilization
Period: '1'
Statistic: Maximum
Unit: Percent
AWSEBLoadBalancerSecurityGroup.aws:ec2:vpc:
VPCId: vpc-xxxxxxxxxxxxxxxx
AWSEBV2LoadBalancerListener.aws:elbv2:listener:default:
ListenerEnabled: false
AWSEBV2LoadBalancerListener443.aws:elbv2:listener:443:
SSLCertificateArns: arn:aws:acm:us-east-2:xxxxxxxxxxx:certificate/xxxxxxx-xxxxx-xxxx-xxxx-xxxxxxxxxxxx
AWSEBV2LoadBalancerTargetGroup.aws:elasticbeanstalk:environment:process:default:
HealthCheckPath: /rest/account/ping
MatcherHTTPCode: '200'
Port: '80'
Protocol: HTTP
aws:autoscaling:launchconfiguration:
IamInstanceProfile: aws-elasticbeanstalk-ec2-role
SecurityGroups:
- sg-xxxxxxxxxxxxx
aws:ec2:instances:
InstanceTypes: t2.small
aws:ec2:vpc:
ELBSubnets: subnet-xxxxxxxxxxxxxxxxxx,subnet-xxxxxxxxxxxxxxxxxx,subnet-xxxxxxxxxxxxxxx
Subnets: subnet-xxxxxxxxxxxxxxxxx,subnet-xxxxxxxxxxxxxxxxx,subnet-xxxxxxxxxxxxxxxxx
aws:elasticbeanstalk:application:environment:
JDBC_CONNECTION_STRING: jdbc:mysql://xxxxxxxxxxxxxxxxxxxxxxxxxxxx?user=xxxxxxxx&password=xxxxxxxxxxx&rewriteBatchedStatements=true&characterEncoding=UTF-8
aws.accessKeyId: xxxxxxxxxxxxxxxxxx
aws.secretKey: xxxxxxxxxxxxxxxxxxxx
com.aws.secretManger.secret.name: xxxxxxxxxxxxxxx
com.aws.secretManger.secret.region: us-east-2
com.decsond.loggly.token: xxxxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx#xxxxx
com.decsond.metakey: xxxxxxxxxxxxxxxxx/XXX==
com.decsond.mode: debug
com.decsond.server.db.environment: aws
com.decsond.server.dpBinaryColumn: xxxxxxxxxxxx
com.decsond.server.environment: xxxxxxxxxx
com.decsond.server.type: pms
aws:elasticbeanstalk:container:tomcat:jvmoptions:
JVM Options: -XX:+CMSClassUnloadingEnabled -Dmvel.disable.jit=true -Ddrools.permgenThreshold=0
Xms: 512m
Xmx: 1024m
aws:elasticbeanstalk:environment:
LoadBalancerType: application
ServiceRole: arn:aws:iam::xxxxxxxxxxxxxx:role/aws-elasticbeanstalk-service-role
aws:elasticbeanstalk:healthreporting:system:
SystemType: enhanced
aws:elasticbeanstalk:managedactions:
ManagedActionsEnabled: true
PreferredStartTime: SAT:03:01
aws:elasticbeanstalk:managedactions:platformupdate:
InstanceRefreshEnabled: true
UpdateLevel: minor
aws:elasticbeanstalk:xray:
XRayEnabled: true
aws:elbv2:listener:443:
DefaultProcess: default
ListenerEnabled: true
Protocol: HTTPS
Rules: ''
SSLPolicy: ELBSecurityPolicy-2016-08
Platform:
PlatformArn: arn:aws:elasticbeanstalk:us-east-2::platform/Tomcat 8.5 with Java 8
running on 64bit Amazon Linux/3.3.1
Any idea about the issue ?
The most likely reason is that you are referencing objects in the region from where the config was saved from.
Is this the first EB application / environment in the new region?
If it is, it's worth first creating a test application and environment, using the features you want ... that will give EB a chance to create all the region specific behind-the-scenes magic it relies on.
I am using Spring cloud stream with RabbitMQ.
I want to be able to configure message and query properties from source code and not from a property file (as they mention in their docs).
For example with the classic Java Client for RabbitMq i can do something like that to create a queue with the properties i want:
//qName, passive, durable, exclusive auto-delete
channel.queueDeclare("myQueue", true, false, false, , false , null);
Any ideas on how can i achieve the same thing using Spring cloud stream?
Inside of "application.yml" you can add all this values , following is example
spring:
cloud:
stream:
instance-count: 1
bindings:
input:
consumer:
concurrency: 2
maxAttempts: 1
group: geode-sink
destination: jdbc-event-result
binder: rabbit
rabbit:
bindings:
input:
consumer:
autoBindDlq: true
republishToDlq: true
requeueRejected: false
rabbitmq:
username: ur-user-name
password: ur-password
host: rabbitmq-url-replace-here
port: 5672
datasource:
platform: mysql
url: jdbc:mysql-url-replace-here
username: ur-user-name
password: ur-password
driverClassName: com.mysql.jdbc.Driver
datasource:
tomcat:
max-wait: 300
min-idle: 10
max-idle: 100
aggregator:
groupCount: 2
batchSize: 1000
batchTimeout: 1000
Updated :
https://cloud.spring.io/spring-cloud-static/spring-cloud-stream-binder-rabbit/2.2.0.M1/spring-cloud-stream-binder-rabbit.html
https://github.com/spring-projects/spring-xd/blob/master/spring-xd-dirt/src/main/resources/application.yml
After digging in their documentation and with the help of #vaquar khan I found out that the only way to do it is from your property file.
application.yml
spring:
cloud:
stream:
bindings:
queue_name :
destination: queue_name
group: your_group_name
durable-subscription : true
This will declare a durable, non-deleting and non-exclusive queue.
As soon as I create model and try to run application, I run into error andsqlexception. Check below for my code, application.yml, build.gradle settings `and stack trace. Am I missing anything?
`I'm pretty new to Grails as well as stacktrace. Please pardon for mistakes.
`Build.gradle
runtime "mysql:mysql-connector-java:5.1.36"
`Application.yml
hibernate:
cache:
queries: false
use_second_level_cache: true
use_query_cache: false
region.factory_class: org.hibernate.cache.ehcache.SingletonEhCacheRegionFactory
dataSource:
pooled: true
driverClassName: com.mysql.jdbc.Driver
dialect: org.hibernate.dialect.MySQL5InnoDBDialect
environments:
development:
dataSource:
dbCreate: create-drop
url: jdbc:mysql://localhost:3306/newdb?useUnicode:yes&characterEncoding:UTF-8
username: root
password: YES
test:
dataSource:
dbCreate: update
url: jdbc:mysql://localhost:3306/newdb?useUnicode:yes&characterEncoding:UTF-8
username: root
password: YES
production:
dataSource:
dbCreate: none
url: jdbc:mysql://localhost:3306/newdb?useUnicode:yes&characterEncoding:UTF-8
username: root
password: YES
properties:
jmxEnabled: true
initialSize: 5
maxActive: 50
minIdle: 5
maxIdle: 25
maxWait: 10000
maxAge: 600000
timeBetweenEvictionRunsMillis: 5000
minEvictableIdleTimeMillis: 60000
validationQuery: SELECT 1
validationQueryTimeout: 3
validationInterval: 15000
testOnBorrow: true
testWhileIdle: true
testOnReturn: false
jdbcInterceptors: ConnectionState
defaultTransactionIsolation: 2 # TRANSACTION_READ_COMMITTED
`Model - Website.groovy
package newapp
class Website {
long Web_ID;
String Website;
long Company_ID;
// getters and setters here
static constraints = {
Web_ID(unique: true)
Website(unique: true)
}
}
`Stack Trace
| Running application...
2017-03-14 15:01:19.036 ERROR --- [ost-startStop-1] o.a.tomcat.jdbc.pool.ConnectionPool : Unable to create initial connections of pool.
java.sql.SQLException: Access denied for user 'root'#'localhost' (using password: YES)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:998)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3847)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3783)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:871)
at com.mysql.jdbc.MysqlIO.proceedHandshakeWithPluggableAuthentication(MysqlIO.java:1665)
at com.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:1207)
at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2249)
at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2280)
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':bootRun'.
> Process 'command 'C:\Program Files\Java\jdk1.8.0_66\bin\java.exe'' finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.
| Error Failed to start server (Use --stacktrace to see the full trace)
D:\BeanCracker\New Grails Workspace\NewApp>