I am not from Java ,so my question may be very easy but I need clear steps how to implement.
Existing project : Webmethods connecting to Oracle Data base to fetch certain properties file and insert log information into some tables.
Problem: Many a times data base goes down and hence delays in execution.
New Requirement: We have to replace existing oracle table with Hbase. I have writen code write file into Hbase using Pig. But I really don't know how to write the real time data into Hbase.
I found using Java client or Thrift connection I can write. I need very detailed explanation. I have to submit for an Project. Please help me out.
You have knowledge of Row oriented database and Hbase is column oriented database.But we have apache Phoenix.
Apache Phoenix is a relational database layer over HBase delivered as a client-embedded JDBC driver targeting low latency queries over HBase data. Apache Phoenix takes your SQL query, compiles it into a series of HBase scans, and orchestrates the running of those scans to produce regular JDBC result sets. The table metadata is stored in an HBase table and versioned, such that snapshot queries over prior versions will automatically use the correct schema. Direct use of the HBase API, along with coprocessors and custom filters, results in performance on the order of milliseconds for small queries, or seconds for tens of millions of rows.
This can easily solve your problem.
http://phoenix.apache.org/
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.sql.PreparedStatement;
import java.sql.Statement;
public class test {
public static void main(String[] args) throws SQLException {
Statement stmt = null;
ResultSet rset = null;
Connection con = DriverManager.getConnection("jdbc:phoenix:[zookeeper]");
stmt = con.createStatement();
stmt.executeUpdate("create table test (mykey integer not null primary key, mycolumn varchar)");
stmt.executeUpdate("upsert into test values (1,'Hello')");
stmt.executeUpdate("upsert into test values (2,'World!')");
con.commit();
PreparedStatement statement = con.prepareStatement("select * from test");
rset = statement.executeQuery();
while (rset.next()) {
System.out.println(rset.getString("mycolumn"));
}
statement.close();
con.close();
}
}
Related
We're using HSQLDB for a bunch of JUnit integration tests. For one particular test I need to load a bunch of data to verify some algorithms we run on the database. Each test in the suite will insert a large batch, test the algorithm and then delete all records. Unfortunately, HSQLDB will eventually throw an OutOfMemoryError, even though all records are cleared each time and the maximum number of records in the database at any given time does not change.
Here's a minimalist JUnit test to reproduce this. As you can see it just inserts and then deletes a bunch of rows. What is HSQLDB keeping in memory after the deletes that causes the Error? What can I change to be able to run the insert-deletes indefinitely (or at least aplenty to execute all tests)?
package mypackage;
import org.junit.Test;
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.PreparedStatement;
import java.sql.ResultSet;
import java.sql.SQLException;
/**
* Verifying HSQLDB OutOfMemoryError
*
* #author Ricardo van den Broek
*/
public class HsqlDbTest {
#Test
public void test() throws Exception {
executeSql("CREATE TABLE MY_TABLE (MY_COLUMN CLOB)");
String str = "TESTTESTTESTTESTTESTTESTTESTTEST";
String x = String.format("INSERT INTO MY_TABLE VALUES('%S')", str);
for (int i=0;i<1000;i++){
System.out.println("Starting batch: "+i);
for (int j=0; j<10000; j++) {
executeSql(x);
}
System.out.println("Inserted: "+getCount());
executeSql("DELETE FROM MY_TABLE");
System.out.println("After delete: "+getCount());
}
}
private void executeSql(String sql) throws SQLException {
Connection c = DriverManager.getConnection("jdbc:hsqldb:mem:a");
PreparedStatement ps = c.prepareStatement(sql);
ps.executeUpdate();
ps.close();
c.close();
}
private long getCount() throws Exception {
Connection c = DriverManager.getConnection("jdbc:hsqldb:mem:a");
PreparedStatement ps = c.prepareStatement("SELECT COUNT(*) FROM MY_TABLE");
ResultSet resultSet = ps.executeQuery();
if (resultSet.next()) {
long count = resultSet.getLong(1);
ps.close();
c.close();
return count;
}
throw new Exception("Should not happen");
}
}
Output:
Starting batch: 0
Inserted: 10000
After delete: 0
Starting batch: 1
# ...
# Omitting some repetition
# ...
Inserted: 10000
After delete: 0
Starting batch: 10
java.sql.SQLException: java.lang.OutOfMemoryError: Java heap space
Note: I'm aware I could increase the memory limit on the unit tests or shut the database down in between, but until someone can provide me with a reasonable explanation of why it does this, I feel I should be able to expect emptying the in-memory HSQLDB should also actually free the memory it was using.
This phenomenon is present in versions older than 2.5.1.
This is not a bug. In a mem: (memory-only) database with no files, when the column type is CLOB (as opposed to VARCHAR), the String objects are stored in a separate memory LOB store which is not cleared as rows are deleted. You need to perform a CHECKPOINT from time to time to clear the LOB store. Add executeSQL("CHECKPOINT") at the end of each run.
You can declare the column as LONGVARCHAR or any VARCHAR(n) with a large enough size to avoid using the CHECKPOINT statement.
This issue does not arise in file: databases, which use a file for the storage of LOBs, even when the table is a MEMORY table. In these databases, a CHECKPOINT releases the spaces taken up by deleted blocks inside the .lobs file. The spaces are then reused for new LOBs.
In the latest version 2.5.1, automatic checkpoints are also performed when lobs are deleted from mem: databases. Here, the hsqldb.log_size setting applies to the total amount (in MB) of deleted lob data that triggers a checkpoint.
In my company we are performing a database migration. For lots of reasons we cannot use a database migration tool, so we had to develop our migration tool to copy from a DB to another all rows contained in some specific tables. We have developed a tool using JDBC of multiple database. At the moment, we are migrating from DB2 to Oracle, with an intermediate step to H2. Same tables ha a Clob column. When we export this column from DB2 to H2 we get no errors or issues, but when we try to copy the Clob from H2 to Oracle using JDBC we get the following exception:
ClassCastException: cannot cast from org.h2.jdbc.JdbcClob to oracle.jdbc.Clob
Is there a way or a procedure to perform this kind of conversion? Something like a ClobCopy utility within different Clob types? Unfortunately we can do this task only using Java and Jdbc, no JPA or DB migration tools due to customer specifications.
This is an example of what I'm trying to do:
public class CopyTable {
public void doCopy(){
Connection h2 = getH2Connection(); //suppose this exists and works
Connection oracle = getOracleConnection(); //suppose this exists and works
String sqlSelect = "select * from tabletoexport";
String sqlInsert = "insert into tabletofill(ID, DATA) values (?,?)";
PreparedStatement select = h2.prepareStatement(sqlSelect);
PreparedStatement insert = oracle.prepareStatement(sqlInsert);
ResultSet rs = select.executeQuery();
while (rs.next()){
insert.setLong(1, rs.getLong("ID"));
insert.setClob(2, rs.getClob("DATA")); //this throws an exception
insert.executeUpdate();
}
}
}
The Clob interface has a getCharacterStream() method which returns a Reader, and the PreparedStatement interface has a setClob() method which takes a Reader. All you need to do to get the copy working is to use these methods.
In other words, replace the line
insert.setClob(2, rs.getClob("DATA")); //this throws an exception
with
insert.setClob(2, rs.getClob("DATA").getCharacterStream());
As for why the import from DB/2 to H2 didn't complain, perhaps the H2 JDBC driver doesn't assume that Clob values passed in to setClob come from H2, but the Oracle JDBC driver does assume that Clobs passed in in the same way are from Oracle. However, the Oracle JDBC can't reasonably make any such assumptions about a Reader, as these could come from anywhere
I recently implemented C3P0 in my database testing program (i'm using it to test different queries on our data in different DB formats... sqlite, mariadb, etc). The program was initially set up using a single sustained connection to do all queries. This worked fine with SQLite as I had to do an initial ATTACH on another table. When moving to C3P0 where it is necessary to open and close the connection on every query, how can one issue an ATTACH command and have it apply to subsequent queries? In my failure I did notice that the first query after the attach it seemed to apply.
Do I really need to interlace ATTACH test as TESTDB for EVERY query???
Running into a similar issue with setCatalog() for MariaDB. I get a "No Database selected" for every subsequent query after the first.
Do I really need to interlace ATTACH test as TESTDB for EVERY query???
No. As #MarkRotteveel suggested in a comment to the question, we can use a c3p0 connection customizer to tweak each connection as it is acquired for the pool. For example, if we create the class OurSQLiteConnectionCustomizer ...
package com.example.sqlite_pooled;
import java.sql.Connection;
import java.sql.SQLException;
import java.sql.Statement;
import com.mchange.v2.c3p0.AbstractConnectionCustomizer;
public class OurSQLiteConnectionCustomizer extends AbstractConnectionCustomizer {
public void onAcquire(Connection c, String pdsIdt) throws SQLException {
try (Statement st = c.createStatement()) {
st.execute("ATTACH DATABASE 'C:/__tmp/SQLite/test.sqlite' AS test");
}
}
}
... and we tell our ComboPooledDataSource to use it ...
cpds = new ComboPooledDataSource();
cpds.setConnectionCustomizerClassName("com.example.sqlite_pooled.OurSQLiteConnectionCustomizer");
... then whenever c3p0 acquires a new SQLite connection for the pool it will automatically perform the ATTACH DATABASE for us.
im using a embedded hsql database in my java programm.
I want to write a hsql statement like this:
statement.executeQuery("SELECT sum(Points) FROM Table");
At first i tried this one:
String column = "Points";
statement.executeQuery("SELECT sum(\""+column+"\") FROM \""+table+"\"");
java.sql.SQLException: Column not found: Points
Next one:
statement.executeQuery("SELECT sum(POINTS) FROM \""+table+"\"");
java.sql.SQLSyntaxErrorException: user lacks privilege or object not found: POINTS
Next try, should never been working but only for you :-)
statement.executeQuery("SELECT sum(\'"+column+"\') FROM \""+table+"\"");
java.sql.SQLSyntaxErrorException: incompatible data type in operation
if i try this one:
statement.executeQuery("SELECT \""+column+"\" FROM \""+table+"\"");
runs perfectly
Just to show you that my column exist in my table.
This statement:
SELECT sum("Points") as test FROM "MyTable"
runs in SQuirrel Client Version 3.7
Any idea with my problem?
The same statement that works in SQuirrel should work. This is the same statement with Java quoting:
statement.executeQuery("SELECT sum(\"Points\") FROM \"MyTable\"");
It looks like you are connecting to a different database from your program and from SQuirrel. Try using the same absolute path to the database file in SQuirrel and in your program. You cannot connect at the same time, so you need to shutdown the database in SQuirrel before you connect from your program.
These are my imports
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.sql.Statement;
This Statment also not run:
statement.executeQuery("SELECT sum(\"Points\") FROM \"MyTable\"");
SOLUTION for maybe others:
"SELECT sum(\"Punkte\") as TEST FROM \"Match_Stats\"");
while(table_01.next()){
players.get(i).setPoints(table_01.getInt("TEST"));
}
Alias TEST was the key.
I need to connect to a remote database using Database link using JDBC commands.
How can it be done?
If you already have the dblink setup, you can utilize it in your SQL (sent via jdbc) by addressing the required tables like such:
select * from SCHEMA.TABLE#DBLINK_NAME
Using this query inside of your java would look something like this
public ResultSet execQuery() throws SQLException, ClassNotFoundException{
//Load the database driver
Class.forName("oracle.jdbc.OracleDriver");
//Create connection to the database
Connection myConnection = DriverManager.getConnection(connectURL,userName,userPwd);
//Create a statement link to the database for running queries
Statement myQuery = myConnection.createStatement(ResultSet.TYPE_SCROLL_SENSITIVE,ResultSet.CONCUR_UPDATABLE);
//Create a resultSet to hold the returned query information
ResultSet myQueryResults = myQuery.executeQuery("select * from SCHEMA.TABLE#DBLINK_NAME");
return myQueryResults;
}
*java & oracle assumed
If you are asking about how to use JDBC to create a link between the DB you are talking to and another one, then it is "just SQL" that you (presumably) would execute in the same way as you would any other SQL statement. (If you tell us which DB you are using, we could talk about the actual SQL you need to execute.)
Otherwise, I don't think this makes sense. A DB link / Database link is a link from one database to another. But JDBC is for talking to a database from a Java client. It makes no sense (to me) to use DB link to connect a JDBC client to a database.
Please take a look at orajdbclink, on sourceforge
I am planning to to connect my oracle plsql sources to phoenix skin of hbase. It seems to me the unique way to create a connector between oracle and hbase for the moment...