It is getting burdensome on my team to prototype tables in MySQL that back our Java business applications, so I'm campaigning to use SQLite to prototype new tables and test them in our applications.
These tables are usually lightweight parameters, holding 12 to 1000 records at most. When the Java applications use them they are likely to be doing so in a read-only capacity, and typically the data is ingested in memory and never read again.
Would I have a problem putting these prototype SQLite tables out on a network, as long as they are accessed via read-only and in small volume? Or should I copy them locally to everyone's machines? I know SQLite does not encourage concurrent access on a network, but I'd be surprised if more than one user would hit it a the same time given the number of users and the way our applications are architected.
If you are using a three-layer architecture, only the application server should have access to the database server. Therefore, you should have control over the connections (i.e. you can create a very small connection pool).
Embedded databases are not suited for lots (hundreds) of concurrent connections. Nevertheless, having into account the amount of data and that you will only focus on read-only queries, I doubt that would be a problem.
A major problem I foresee is that you can have serious problems in terms of SQL dialects. Usually embedded databases use the ANSI SQL standard, but mySQL and others allow you to use their own SQL dialects which are incompatible. It's usually a good practice to have a unit test that runs all the SQL queries against an embedded database to guarantee that they are ANSI-compliant. This way, you have a guarantee that you can use your application (automatically or manually) with the embedded database.
Related
I am a Java guy, I can work with Oracle Database, I know PLSQL, SQL. But I am not good at managing database servers. I think it is a completely different area.
My question is related to database replication. I googled it, found millions of answers but I am still confused.
I could see many times in my professional carrier that developers create complete (complicated) applications to keep sync the source database schema to a target one. It takes time to develop sync apps and very hard to maintain them, especially in case of any data structure modification for example in tables.
I could see that apps built with JPA, JDBC, Spring, myBatis, and PLSQL as well. Usually, they sync DBs during the night, scheduled by Cron, Quartz, Spring, etc. During the sync process usually, the source DB is only available for querying data, not for inserting and DB constraints and triggers are disabled.
These kinds of custom applications always scare me. I do not believe that there is no general, easy, and official way to keep sync two databases without developing a new application.
Now, I got a similar task and honestly, I would like to write zero lines of code related to this task. I believe that there are recommended and existing solutions, cover this topic offered by the database vendors.
That would be great if you can push me in the right direction. I feel that writing another new DB sync application is not the right way.
I need to focus on Oracle Database sync, but I would be happy to know a general, database vendor-independent way.
There are many ways to perform replication in a Oracle Database. Oracle has two replication techniques in the database "Advanced Replication" and "GoldenGate". GoldenGate us the new perferred method of replication which uses the redo logs files from the database. Both methods are geared for a Oracle DBA.
Often application developers will create a "interface" that will move data from one database to other. A interface is a program ( pl/sql, bash, c, etc ) that runs on a cron (database or system) that wakes on a event to move data. Interfaces are useful when data is needed to be process during replication.
Is Hibernate less effective in some environments, like a polygot company where several distributed systems are accessing the same db? If Acme Company has a python website reading from and writing to the same database as a java web app (web services), will Hibernate be a poor choice for the java web services app? In other words, does Hibernate caching and session management assume all db transactions for Acme will be using Hibernate? Do I need to be sensitive to certain ORM concerns at a company where several programming languages are writing a lot of updates to the same data concurrently? Is Hibernate more advantageous for a strict java shop using a java ee app server for nearly all of its business operations?
Hibernate does have some performance overhead over pure JDBC, but if you're using it cautiously it should be fine for most of use cases.
Hibernate does not assume that it handles all operations itself. The only thing I would worry about is second level cache if you need it. You won't have a way to keep it in sync if other apps access the same DB (but you don't have to use it).
Having said that, I must add that having multiple apps write to the same DB is not a good practice. I'd rather create one app that handles this DB and have others communicate with this one - this way it's much easier to keep the database consistent.
Does anyone know of a Java compatible Relational Database Management System, like Microsoft Access, that doesn't require a server side daemon to manage concurrent IO?
Without a server process somewhere, you're talking about a database library like HSQLDB, Derby or SQLite. They work reasonably well as long as you're not expecting lots of concurrent updates to be performant or stuff like that. Those DB servers that are so awkward to set up have a real purpose…
Be aware that if you're using a distributed filesystem to allow multiple users access to the database, you're going to need distributed locking to work (really very painful; too many SO questions to pick a good one to point to) or you're going to have only one process having a connection open at once (very limiting). Again, that's when a DB server makes sense.
I am writing a ETL project in JAVA. I will connect to the source database, get the data only once, do some transformations and Load the data to a target database.
The point is that I am not connecting to the source or the target database multiple times repeatedly. I just connect once (using JDBC), get the data I need and close the connection.
Should I still use the connection pooling?
Thank you for your views!
Connection pooling is used to get around the fact that many database drivers take a long time to create a connection. If you only need to use it shortly, and then discard it, the overhead might be substantial (both in time and cpu) if you need many connections. It is simply faster to reuse than to create a new.
If you do not have that need, there is no reason to set up a connection pool if you don't have it already. If you happen to have one already, then just use that.
My guess is that in some circonstances, using several threads and concurrent connections could improve the overvall throughput of your software allowing for exemple to use all CPU of your RDBMS server, or of the client ETL. This also could help using the fact that several tables could sit physically on differents hardware and thus could be accessed in parallel.
The real impact would really depend of the computers you use and the architecture of the database.
Be carefull that typically ETL have ordering constraints and doing several things at the same time should not violate theses constraints.
Edit : An exemple of this. You can configure Oracle to execute each requests using several cores or not. (Depending of configuration and licence if I understand right). So if one request is allowed to use only one core, using several connections at the same time will allow several requests as the same time and better use the CPU resources of the server.
I am developing a web application with php that needs to synchronize a local mysql database that a java desktop version of the web application is interacting with. At the same time i also need the local db to synchronize with the remote db. How do i do this without using other software like MySQL Compare. I will really appreciate the help. Thanx guys.
You clearly have a significant architecture issue. It needs to be planned very well. Two-way replication clearly isn't going to work unless you have thought it out very carefully and understand how to do conflict resolution and what impact that will have on your application. In particular, you can forget using AUTO_INCREMENT.
For one-way replication, you can use mk-table-sync, or use mysql replication in some way (there are a variety of possiblities).
You can also run another mysql instance on the server, use mk-table-sync to periodically synchronise it locally, and use mysql replication on that. This has some benefits, particularly if there are some tables you don't want to replicate.
You really need to think about how it's going to work, if you plan to do two-way synchronisation. It is possible that you may end up writing custom code to do it, as the conflict resolution mechanism may mandate it.