I have read-only access to a remote MySQL database, which contains a very large table (hundreds of millions of lines).
To get faster access to that table, I want to copy it to my local database.
What is the best way to do this?
"SELECT INTO OUTFILE" doesn't work, because I don't have the required permissions on the remote database.
I tried to use Java to SELECT all rows FROM the remote table, save them to a local text file, then use LOAD DATA INFILE; however, the select broke with
"Exception in thread "main" java.lang.OutOfMemoryError: Java heap space".
Use the mysqldump command on the remote database to extract the SQL statements of the database required. Then copy the extracted file to your local system and execute the sql file which will create the database in the local system.
Here is the mysqldump example
http://www.roseindia.net/tutorial/mysql/mysqlbackup/mysqldump.html
Try to set Synchronization, latest version of PHPMyAdmin provides an option to set synchronization. You need to set source DB as remote database and destination to your local database.
Setting up a PHP (and PHPMyAdmin too) on local machine is not a big task. if table is much bigger you may need to increase maximum execution time for phpmyadmin script.
Alternatively if you can access remote MySQL port then you can try to connect to remote db from your local machine as mysql -h remote_IP -u usernmae -pPassword. if it connects then you can definitely use mysqldump command on local machine. check this link
The problem with your Java program is likely to be because the MySQL JDBC driver stores the entire ResultSet in memory by default. With a huge table, this is highly likely to cause an OutOfMemoryError.
You can stop the MySQL driver from doing this by following the instructions in the ResultSet section of this page in the MySQL documentation (which I found via this blog post).
I was able to reproduce an OutOfMemoryError with a simple Java program that simply read each row out of a table with over 120 million rows. After making the changes suggested in the MySQL documentation, my Java program completed without any memory issues.
Related
Sorry Maybe this is the second time I am Asking this question because of not getting any answers .
this is my Code
try{
File f = new File("Database.sql");
if(f.exists()){
Class.forName("com.mysql.jdbc.Driver").newInstance();
Connection con = DriverManager.getConnection("jdbc:mysql://localhost/"+f.getName(),"","");
}else{
f.createNewFile();
System.out.println("file created");
//also do the connection
}
}catch(Exception ex){
System.out.println(ex.getMessage());
}
and Here is the error :
Communications link failure
The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
The purpose of this question is:
I am creating an Application that is for many users, the problem is they don't know anything about computer and I must make it as simple as possible.
So is there any way to connect with MYSQL like MS ACCESS via Directory Path ?
OR is there any other suggestion instead ?
Thanks .
AFAIK, you can't plug in a file name in the JDBC url for MySQL. MySQL needs to be running, and you need to connect to it via its TCP port. Something like:
jdbc:mysql://localhost:3306/yourDatabaseName
See http://dev.mysql.com/doc/refman/5.6/en/connector-j-reference-configuration-properties.html
Preliminary definitions
"Connect to a MySQL database file" is improper phrasing.
One actually connects to a MySQL database server, which allows access to a database. And so there must be a MySQL server to connect to, more about that below.
Your Database.sql file is a database dump file, that is to say, a dumb (plain text) file. You need a specialized process to extract data from it, after interpreting your SQL queries: a database server.
You might be assuming that one can connect to a file because you are used to working with MS Access files. I am not an expert neither in Java nor in MS Access, but it is my undestanding that accessing a MS Access "database" file from Java actually means connecting to some middleware server, such as this ODBC thingy from Microsoft.
The answer
There is no way to connect to a MySQL database server via a directory path. The only native ways are:
TCP
Local socket (Unix servers only)
Named pipes (Windows servers only)
Shared memory (Windows servers only)
There could be some third-party pieces of software around that provide other protocols, which I am not aware of, but they all reduce to the same problem: there must be a MySQL sever running somewhere.
On second thought there is actually one way to access MySQL data without an external MySQL server running: the embedded MySQL server C library. I never tried it myself, but it looks like a viable option for stand-alone applications. I do not believe, however, that it is a desirable solution if you plan to share the same MySQL data across several processes or computers.
The workarounds
Now I understand you are building a Java desktop application based on data that you have in the form of a SQL dump file, probably dumped from a MySQL server. If you want your users to be able to access this data from this Java application, I can see a few options:
Install a MySQL server on their computers and load this dump into it. Obvious as hell, but impractical, if I hear you well. Although I guess this installation could certainly be performed automatically by your Java application.
Install a MySQL server on a machine of your own, and make it accessible from your users' computers. Major drawback: it requires your users to be connected. You would also probably want to create a distinct database for each user.
Use an actually serverless database engine such as SQLite. This seems to be the best option for you. Its SQL syntax is virtually identical to MySQL for usual operations. There must be plenty of JDBC drivers for it. Again, I am not the best advisor in Java, but this one seems to be a serious candidate.
I'm using a JDBC driver to run "describe TABLE_NAME" on hive. It gives me the following error:
NativeException: java.sql.SQLException: Query returned non-zero code: 9, cause: FAILED:
Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask
return code 1 doesn't tell me very much. How do I figure out what the underlying reason is?
It's most likely because your Hive metastore is not setup properly. Hive uses a RDBMS metastore to store meta data about its tables. This includes things like table names, schemas, partitioning/bucketing/sorting columms, table level statistics, etc.
By default, Hive uses an embedded derby metastore which can only be accessed by one process at a time. If you are using that, it's possible that you have multiple sessions to Hive open that's causing this problem.
In any case, I would recommend you to set up a stand alone metastore for Hive. Embedded derby was chosen for its usability in running tests and a good out of the box metastore. However, in my opinion, it's not fit for production workflows. You can find instructions on how to configure MySQL as Hive metastore here.
Possibly you have another sesssion open. Since derby allows only one session per person.
You can check -
ps -wwwfu <your id>
kill the id which is running the hive connection.
It is because the table with the name you've specified is didn't exist in the database.
Try creating the table and again run the command. it will work. :)
I have a memory HSQLDB database with this connection URL:
jdbc:hsqldb:mem:test_database
It runs fine with my application but I need to configure this database in a SQL Client.
I can't because every clients complain that no host was found or there's no database.
I'm not sure if I'm filling all the information correctly in "host" and "database" fields or if it is a HSQLDB memory restriction.
Has anyone got the same error?? Thanks a lot.
With :mem: you define a database which is only accessible within the running java vm. This database resides in memory and cannot be accessed externally via host/port jdbc access.
Please read:
Running and Using Hsqldb
Advanced Topics
You can use the Database Manager provided by HSQLDB, just run in console
java -cp hsqldb.jar org.hsqldb.util.DatabaseManagerSwing
and connect to the jdbc:hsqldb:mem:test_database
So, I'm trying to figure out how to log the SQL statements that would be run, without actually having an active mySQL server.
The problem I'm trying to solve is right now we're writing data to both a remote mySQL instance, as well as a local (archive data). We're logging to a local as a backup in case the remote becomes unreachable/goes down/etc.
So, what we'd like to do instead is log the SQL statements locally (we're going through Spring JDBC Template w/variable replacement), so not quite as easy as taking the SQL we piece together ourselves and write it to a file.
I did find log4jdbc which looks great, except we'd still need a local mySQL instance active, even if it's just using the blackhole engine. I did think maybe we could just use the log4jdbc on the remote server, but if the connection goes away, will the JDBCTemplate even try and run the queries on the underlying JDBC driver objects before getting the failure? Most pooling mechanisms will validate the connection before returning it, so it'd still just fail before the query had a chance to run & get logged.
So, who has any bright ideas?
Run the remote MySQL instance with bin logging enabled.
The binary logs can be backed up and if necessary converted into SQL using the mysqlbinlog command, to later restore a database.
Mark O'Connors solution is probably the best in general, but the way we solved this was to simply write out the data as CSV files formatted to be ready for import via a load data infile statement.
I am having a problem with Java DB that I just don't know how to resolve. I am creating a DB and connecting to it using Java DB's native JDBC driver. If I relocate that database physically and try to connect to it using its new path, I consistently get XJ004 errors:
ERROR XJ004: Database 'blahblah' not found.
I am sure I am using the correct connection string. Is there any possibility the DB is somehow getting corrupted? Or is there some encoding of the DB path in the DB such that if you relocate a Java DB it gets confused?
I'm really at a loss here. :( Please help!
Jim
Have you verified that this error message isn't also used when there's no listener on the host machine ... and were you using JavaDB on your local machine before the relocation? Many database systems (and I'm not that familiar with JavaDB) ship set-up to only allow connections from localhost for security reasons. On PostgreSQL for instance, you have to allow TCP connections and bounce the daemon to obtain a remote connection.
Anyway ... since the problem started when you when remote, look for issues related to that first! (And if you can run your application on the remote machine, does that work?)
There must be a file named derby.log somewhere. Check the error there. If it is not detailed enough, try setting derby.stream.error.logSeverityLevel to a lower value. See the manual for more information.