I'm trying to test out MariaDB, Galera, and Corosync/Pacemaker to understand clustering with high-availability using CentOS 7 servers. The cluster size I am using for testing is 3 servers to prevent quorum issues for the most part. My tests and application are written in Java.
I have the clustering part down; it works in an active-active configuration and runs just fine. As well, I have HA set up using Pacemaker and corosync. I've done many tests with it in failing the slaves or bringing them up during the run. As well, I've written to all three at run-time regardless of if it was the master connection or one of the child connections. When I try to test the Master going down during run-time (to simulate a power outage, server crash, or whatever in the data center), the application immediately stops running. I get a java.net.SocketException error and the application closes with the other two connections being shut down successfully. I've used both the kill and stop commands in the terminal to test and see if it'll work (just in the off chance it would).
JDBC URL String
Below is the part of the code that connects the application to the cluster. It connects correctly and does work until I cause the first master to go down; the other two going down does not affect this.
public void connections() {
try {
bigConnec = DriverManager.getConnection(
"jdbc:mariadb:sequential:failover:loadbalance://"
+ "10.32.18.90,10.32.18.91,10.32.18.92/"+DB+"?autoReconnect=true&failOverReadOnly=false"
+ "&retriesAllDown=120",
"root", "PASS");
bigConnec.setAutoCommit(false);
} catch(SQLException e) {
System.err.println("Unable to connect to any one of the three servers! \n" + e);
System.exit(1);
}
...
}
There are three other connections made to each individual server so I can pull information more easily from them; that is what the ellipses indicate. The servers will exchange which is the "primary" node but the application will not connect to the next node in the list.
I feel like the issue is in the way I have my URL set up because everything works outside of the cluster. As well, nothing happens during testing when I shut down the child nodes. The most that happens is I'll get a warning that the URL lost connection to either or both of them. Is there a way to configure the URL in such a way to allow automatic failover to the next available node in the string or do I have to go about it some other way using individual connection URLs and Objects (or an array of Connection objects) or black magic and pixie dust that really only SysAdmins know some other way I have yet to try?
What I Have Tried
How to make MaxScale High Available with Corosync/Pacemaker (MariaDB Article)
Failover and High availability with MariaDB Connector/J (MariaDB Documentation)
What is the right MariaDB Galera jdbc URL properties for loadbalance (Stack Overflow)
HA Proxy Configuration with MariaDB Galera cluster (Stack Overflow)
Configuring Server Failover (MySQL Documentation)
Advanced Load-balancing and Failover Configuration (MySQL Documentation)
TL;DR
Problem: Java application is not automatically failing over despite being flagged for failover and sequential support in the JDBC URL (see JDBC URL String). Everything works with corosync and Pacemaker but I cannot get the Java application to transfer to the next available node to act as the primary connection when the current one goes down.
Question: Is the issue in the URL? A follow-up to that is, if so, would it be better to use three separate connections and use the first valid one or is there something I can do to allow the application to automatically rollover to the next available connection in the current URL?
Software/Equipment
MariaDB 10.1.24
corosync 2.4.0
Pacemaker 1.1.15
CentOS 7
Java 8 / Eclipse Neon.3 / Eclipse 4.6.3
MariaDB Connector/J 2.0.1
If there is any more information you need, please do tell me in the comments and I'll update this as soon as I can!
Related
I have been getting struggle to connect H2 database from a Spring Boot app by using the following connection string as mentioned on Database URL Overview section:
spring.datasource.url=jdbc:h2:tcp://localhost:9092/~/test-db
I also tried many different combination for tcp (server mode) connection, but still get error e.g. "Connection is broken: "java.net.SocketTimeoutException: connect timed out: localhost:9092" when running Spring Boot app.
#SpringBootApplication
public class Application {
// code omitted
#Bean(initMethod = "start", destroyMethod = "stop")
public Server h2Server() throws SQLException {
return Server.createTcpServer("-tcp", "-tcpAllowOthers", "-tcpPort", "9092");
}
}
So, how can I fix this problem and connect to H2 database via server mode?
You seem to be a little confused.
H2 can run in two different 'modes'.
Local mode
Local mode means H2 'just works', and you access this mode with the file: thing in the JDBC connect URL. The JDBC driver itself does all the database work, as in, it opens files, writes data, it does it all. There is no 'database server' at all. Or, if you prefer, the JDBC driver is its own server though it opens no ports.
Server mode
In this case you need a (separate) JVM and separately fire up H2 in server mode and then you can use the same library (still h2.jar) to serve as a JDBC server. In this mode, the two things are completely separate - if you want, you can run h2.jar on one machine to be the server, and run the same h2.jar on a completely different machine just to connect to the other H2 machine. The database server machine does the bulk of the work, with the 'client' H2 just being the JDBC driver. H2 is no different than e.g. mysql or postgres in such a mode: You have one 'app' / JVM process that runs as a database engine, allowing multiple different processes, even coming from completely different machines halfway around the world if you want to, to connect to it.
You access this mode with the tcp: thing in the JDBC string.
If you really want, you can run this mode and still have it all on a single machine, even a single JVM, but why would you want to? Whatever made you think this will 'solve lock errors' wouldn't be fixed by running all this stuff on a single JVM. There are only two options:
You're mis-analysing the problem.
You really do have multiple separate JVM processes (either one machine with 2 java processes in the activity monitor / ps auxww output / task manager, or 2+ machines) all trying to connect to a single database in which case you certainly do need this, yes.
How to do server mode right
You most likely want a separate JVM that starts before and that hosts the h2 database; it needs to run before the 'client' JVMs (the ones that will connect to it) start running. Catalina is not the 'server' you are looking for, it is org.h2.tools.Server, and if it says 'not found' you need to fix your maven imports. This needs be a separate JVM (you COULD write code that goes: Oh, hey, there isn't a separate JVM running with the h2 server so I'll start it in-process right here right now, but that means that process needs to stay in the air forever, which is just weird. Hence, you want a separate JVM process for this).
You haven't explained what you're doing. But, let's say what you're doing is this:
I have a CI script that fires up multiple separate JVMs, some in parallel even, which runs a bunch of integration and unit tests in parallel.
Even though they run in parallel (or perhaps intentionally so), you all want to run this off of a single DB. This is usually a really bad idea (you want tests to be isolated; that running them on their own continues to behave identically. You don't want a test to fail in a way that can only be reproduced if you run the same batch of 18 separate tests using the same run code, where one unrelated test fails in a specific fashion, whilst it's Tuesday, a full moon, and Beethoven is playing in your music player, and it's warmer than 24º in the room affecting the CPU's throttling, of course. Which is exactly what tends to happen if you try to re-use resources in multiple tests!) – still, you somehow really want this.
... then, edit the CI script to first Launch a JVM that hosts a H2 server, and once that's up and running, presumably run a process that fills this database with test data, and once that's done, then run all tests in parallel, and once those are all done, shut down the JVM, and delete the DB file.
Exactly how to do the third part is a separate question - if you need help with that, ask a new question and name the relevant tool(s) you are using to run this stuff, paste the config files, etc.
i've got a websphere 6.1 cluster environment which is composed of two nodes with 2 appservers each. Let's call NodeA including Server1(2809) & Server2(2810), NodeB including Server3(2811) & Server4(2812). Meanwhile, i created a cluster-scope datasource with JNDI local_db.
Right now i want to get database connection in a java client through WAS ORB call from above environment. The specific part of java code would look like this:
Hashtable env = new Hashtable();
env.put(Context.INITIAL_CONTEXT_FACTORY,"com.ibm.websphere.naming.WsnInitialContextFactory");
env.put(Context.PROVIDER_URL,"iiop://localhost:2809");
javax.sql.DataSource ds = (DataSource)initialContext.lookup("local_db");
Connection cn = ds.getConnection();
If above java client code gets run, will the database connection retrieve request follow load-balancing rule among the four connection pools of all application servers?
Moreover, if my java client gets one database connection successfully and then run a big SQL Query with large result return, as for the memory space occupation, which WAS application server would take care? only server1 due to port 2809 used above or the target server who returns the database connection?
BTW, if i put two server members for that PROVIDER_URL, such as iiop://localhost:2809, localhost:2810, does it mean load-balancing or failover?
Please help explain and correct me if i'm understanding wrongly!
Thanks
Let me start with the easy ones and proceed to the rest
Having two provider URLs' implies failover. If you can't connect to the first naming server, it connects to the second naming server and continues till the end of that list. Notice the Fail over is for connection to the naming server (not to the resource itself)
The look up is done on the server that you connect to. THe local_db represents a datasource (and its connection pool) on that server. You will one work with the server1 (as you are connecting to that NS) and will get connection from the datasource hosted on that server.
You will never get any connection from the other servers. In others words there is no load balancing (one request uses connection from server1, another uses a connection from server 2 etc). I believe this is what you mean by load balancing in your question above.
HTH
A DataSource is neither remotable nor serializable. Therefore, if you look up local_db from the server, it will return a javax.naming.Reference that the client uses to create a new DataSource instance with the same configuration as the one on the server. That also means that once the lookup has been performed, no JDBC method will ever send requests to the WebSphere server.
I have a very short Java application that just opens a connection to a remote MySQL database, reads some data, prints it, and exits. The most time-consuming part of the application is the database connection.
Currently I have only a single thread, and my only concern is to save the time of opening the connection.
I thought of several ways to make it faster, but it turned out they do not help:
Connection Pooling - doesn't help because the pool lives only only during a single run of the application. When the application is terminated, the pool is gone, and when I re-run the application, I have to re-open all the connections in the pool.
mysql-proxy - connects only to the local server: mysql-proxy for a remote MySQL server
TCP/IP server - I thought of holding a local TCP/IP server that will keep a persistent open connection and send it to a TCP/IP client on request. However, Connection objects cannot be serialized, so I have no way to pass the Connection object from client to server.
Any other option?
Generally connection to a DB is a most time-consuming operation. If the application is to be started and stopped then there is little that you can do.
Using connection-pooling in a web-server and call that by running your app which talks to the web server using JSON might be an option.
You said you have a very short application so your 3rd option might work if you put the database logic into you "option 3 TCP/IP server" and just forward the results to your connecting client. This is a typical application server pattern.
Another thing you should consider about network look up https://stackoverflow.com/q/3641155/1055715 which Marc B has mentioned in his comment.
It turns out the best solution is to use mysql-proxy with a script that handles connection pooling (a combination of my first two options). I found one such script here:
http://forge.mysql.com/tools/tool.php?id=151
It was probably written for an older version of mysql-proxy, so I had to fix it (if anyone need the fixed version - write me).
It works like a charm - I run the exact same application as before, the only change is in the connection string: instead of connecting to "qa-srv:3308" (the remote server) I connect to "127.0.0.1:4040" (the proxy server).
I've got a database that is mirrored using SQL 2008 Mirroring.
I have a java application, running on Linux, using the Microsoft SQL
type 4 JDBC drivers.
I have this setup duplicated as a QA environment.
On my QA environment, when I manually fail over the database,
providing a successful connection had already been made, the failover
was completely transparent. I did not have to implement anything in
order to get the application to talk to the new Principle (Old
mirror).
In the live environment however, the connections stop working once I
have manually failed over.
There are quite a few things different between the live and QA environments, but not anything I'd consider fundamental to this process, communication between databases is all very localised in both situations and there are no firewalls (Except the ones built into Windows Server 2008) between my java app and the Windows boxes running SQL 2008.
Does anyone have any ideas about how I can go about diagnosing this issue? Or can anyone tell me how this failover transparency occurs so I can work out how to diagnose this myself?
This behaviour is handled by the SQL Server provider as part of the Client Redirect behaviour. This was part of the SQL Server 2005 JDBC (Java Database Connectivity) 1.1 Driver, I'm unsure if that provider was directly used for the new type 4 drivers though?
In your live environment, is your application successfully connecting to the primary server before it fails over, so it is able to cache the failover partner and make use of that when the failover occurs?
To be sure, you can explicitly state the failover partner in the connection string, which is the recommended practice:
jdbc:sqlserver://serverA:1433; databaseName=AdventureWorks; integratedSecurity=true; failoverPartner=serverB
The full documentation of the redirect behaviour can be found here.
The addition of the failover documentation here suggests that it could be a manual consideration.
First of all, we are running a Java Web application running on WAS 5.1. Behind that, we use an Oracle data base. The problem that we're faced to is really simple, but after a couple of hours of Google search, I decided to ask you.
We have an application that is running on WAS. When we start the server, WAS sets his DataSource so that it points to the data base. Everything works fine, expect when the DBAs have to reboot the data base server. When they do, the data source is no longer valid and we have to manually restart all server and we are currently trying to correct that, if possible. We need to find a way to do it because we have 3 pre-production environnement for for our application, and there are two servers associated with it, one for the application and the other is a report generator web service. So, when the DBAs wants to reboot the server (and they usually don't tell us!) we have to reboot six servers. I was wondering if in Java, there was a way to reset the data source so that we don't need to restart the servers.
For you information, WebSphere is v5.1 and Oracle is 9g with Java 1.4.2.17.
We also use RAD:
Version: 6.0.1
Build id: 20050725_1800
You should configure your application server to always test the connection before leasing it out to a client. I'm not familiar with Websphere that much, but in WebLogic, you can set a jdbc sql statement such as select 1 from dual and the container removes stale connections from the connection pool.
Here is a link on how to do it in Websphere
http://www-01.ibm.com/support/docview.wss?uid=swg21439688
Based on what i read from your note, you should receive Stale connection exception as WAS has stale handles (in its pool) as the DB has been restarted.
The Data Source configuration can be configured to purge the entire pool once a stale connection is detected. The default policy is to purge the individual connection.
Adopting this would prevent you from restarting your WAS Servers.
There are a number of resources in this space
http://www-01.ibm.com/support/docview.wss?uid=swg21063645
HTH
Manglu