I am trying to establish a JDBC connection to Hive so that I can view and create tables and query Hive tables from Eclipse. I used HiveClient sample code: https://cwiki.apache.org/confluence/display/Hive/HiveClient
Then I added all the required jars to the java build path inside eclipse and started Hive Thrift Server. Port 10000 is listening. I am using Cloudera QuickstartVM 4.6.1 and the eclipse that comes with it. Here's the error that I get in the IDE when I try to run the code.
Exception in thread "main" java.sql.SQLException: org.apache.thrift.transport.TTransportException: java.net.SocketException: Connection reset
at org.apache.hadoop.hive.jdbc.HiveStatement.executeQuery(HiveStatement.java:191)
at org.apache.hadoop.hive.jdbc.HiveStatement.execute(HiveStatement.java:127)
at org.apache.hadoop.hive.jdbc.HiveConnection.configureConnection(HiveConnection.java:108)
at org.apache.hadoop.hive.jdbc.HiveConnection.<init>(HiveConnection.java:103)
at org.apache.hadoop.hive.jdbc.HiveDriver.connect(HiveDriver.java:104)
at java.sql.DriverManager.getConnection(DriverManager.java:582)
at java.sql.DriverManager.getConnection(DriverManager.java:185)
at jdbc.Hive.main(Hive.java:24)
When I try connecting to Hive using beeline, I get the same error. However, when I eliminate the host name and port from the !connect command it works with the following error:
beeline> !connect jdbc:hive:// "" ""
scan complete in 4ms
Connecting to jdbc:hive://
14/03/21 18:42:03 WARN conf.HiveConf: DEPRECATED: Configuration property hive.metastore.local no longer has any effect. Make sure to provide a valid value for hive.metastore.uris if you are connecting to a remote metastore.
14/03/21 18:42:03 INFO metastore.HiveMetaStore: 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
14/03/21 18:42:04 INFO metastore.ObjectStore: ObjectStore, initialize called
14/03/21 18:42:05 INFO DataNucleus.Persistence: Property datanucleus.cache.level2 unknown - will be ignored.
What am I missing here!?
You have 2 options to connect hiveserver using jdbc
Option 1 : Hiveserver2
You are trying to connect hiveserver2, hiveserver version in cloudera manager is hivesever2, which is more secure than hiveserver. JDBC code you are using is hiveserver,Use the following code snippet for hiveserver2
Class.forName("org.apache.hive.jdbc.HiveDriver");
Connection con = DriverManager.getConnection("jdbc:hive2://localhost:10000/default", "hive", "");
Statement stmt = con.createStatement();
String tableName = "testHiveDriverTable";
stmt.execute("drop table if exists " + tableName);
stmt.execute("create table " + tableName + " (key int, value string)");
String sql = "show tables '" + tableName + "'";
If you look at the connection string, can see the hiveserver version 2(jdbc:hive2://localhost:10000/default", "", ""), second and third arguments are username and password, by default keep it empty string "".
For executing this program add hiveserver2 specific libraries.
Instead of writing your own programs for checking hiveserver2 jdbc connection, beeline hive client can be used as follows
> [testuser02#Abcd-Host1 ~]$ beeline
> beeline> !connect jdbc:hive2://Abcd-Host1:10000/default "" "" ""
>
> 0: jdbc:hive2://Abcd-Host1:10000/default> show tables;
+------------+
| tab_name |
+------------+
| sample_07 |
| sample_08 |
| test1 |
+------------+
3 rows selected (0.334 seconds)
Options 2: Hiveserver1
If you want to make use of your existing code(code for hiveserver1), which you are having https://cwiki.apache.org/confluence/display/Hive/HiveClient. You got to start a new hiveserver in your userspace in another port. Use the following command to start a hiveserver in a given port
nohup hive --service hiveserver -p 10001 &
Now change the port number to 10001 in jdbc connection and run it.
Related
When trying to issue "SHOW POOLS" or any stats query command on pgbouncer database via JDBC, facing the below exception.
org.postgresql.util.PSQLException: ERROR: unsupported pkt type: 80
at
org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2270)
at
org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1998)
at
org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:255)
at
org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:570)
at
org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:406)
at
org.postgresql.jdbc2.AbstractJdbc2Statement.executeQuery(AbstractJdbc2Statement.java:286)
JDBC code:
String connectionUrl = "jdbc:postgresql://"+ipaddress+":"+port+"/"+database;
con = DriverManager.getConnection(connectionUrl, userName, password);
statement = con.createStatement(); statement.executeQuery("SHOW POOLS");
JDBC Driver Version: 42.2.14 ;
PgBouncer Version: 1.14.0 ;
Postgres Version: 11.4;
PS:
Manually able to connect to pgbouncer database and issue all pgbouncer admin commands like SHOW POOLS or SHOW STATS. Just not able to execute the same from JDBC.
JDBC use extended query protocol by default, try simple protocol for such query
String connectionUrl = "jdbc:postgresql://"+ipaddress+":"+port+"/"+database+"?preferQueryMode=simple";
PGBouncer currently supports only the simple protocol - the packet type 80 is for 'Parse', which is the first step in the extended protocol. The message you see in the exception PSQLException actually comes from PGBouncer.
I'm running hive 2.1.1, hadoop 2.7.3 on Ubuntu 16.04.
It is totally fine to do HQL in the hive terminal.
hive> show databases;
OK
default
Time taken: 0.799 seconds, Fetched: 1 row(s)
hive> show tables;
OK
Time taken: 0.027 seconds
hive> create table test1(id int, name string);
OK
Time taken: 0.928 seconds
hive> show tables;
OK
test1
Time taken: 0.021 seconds, Fetched: 1 row(s)
hive>
This is the error I get from JDBC in beeline and Java:
Error: Error while processing statement: FAILED: Execution Error,
return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask.
MetaException(message:Got exception:
org.apache.hadoop.security.AccessControlException Permission denied:
user=hive2, access=WRITE,
inode="/user/hive/warehouse/test2":server:supergroup:drwxrwxr-x
I get the same error when I try insert query in both beeline and Java. It seems I can only read but not write through JDBC.
Here's how I log in in beeline:
Beeline version 2.1.1 by Apache Hive
beeline> !connect jdbc:hive2://localhost:10000
Connecting to jdbc:hive2://localhost:10000
Enter username for jdbc:hive2://localhost:10000: hive2
Enter password for jdbc:hive2://localhost:10000: ********
Here's piece of my Java code:
try(
Connection con = DriverManager.getConnection("jdbc:hive2://localhost:10000/default", "hive2", "password");
Statement stmt = con.createStatement();) {
...
}
I tried to set up the hive.exec.stagingdir property in hive-site.xml:
<property>
<name>hive.exec.stagingdir</name>
<value>/tmp/hive-staging</value>
</property>
Unfortunately it doesn't solve the problem.
Another solution I found is to log-in as a root user. But how could I log-in as a root user instead of user [hive2]?
How could I execute the write query (create, insert) through JDBC without getting the Permission denied error?
Thank you!
I need to be able to identify Type 4 jdbc workload on an IBM mainframe DB2 v10 zos database.
Our mainframe db2 monitor can filter on the following fields
SYSTEM ID
DB2 SUBSYSTEM ID
DATA SHARING GROUP
AUTHORIZATION ID
PLAN NAME
CONNECTION ID
OPERATOR ID
CORRELATION ID
DBRM/PACKAGE ID
BUFFER POOL ID
COLLECTION ID
LOCAL LOCATION
REQUESTING LOCATION
OTHER LOCATION
DATABASE.PAGESET
LOCK RESOURCE
I am guessing many of these values are not available to be changed.
However these items seems likely candidates
CONNECTION ID
CORRELATION ID
REQUESTING LOCATION
OTHER LOCATION
my questions are:-
i). Can the "likely candidates" be set in my java client jdbc code?
ii). How can I set these "likely candidates"?
If i cannot use any of these fields, then I have managed to amend the clientInfo associated with my JDBC connection, as shown in this snippet of jdbc trace
[jcc][Time:2015-12-10-14:39:24.851][Thread:main][Connection#3b6eb2ec] getClientInfo () called
[jcc][Time:2015-12-10-14:39:24.851][Thread:main][Connection#3b6eb2ec] getClientInfo () returned {ClientUser=XXXXXX00, ApplicationName=db2jcc_application, ClientHostname=L0513039, ClientAccountingInformation=JCC04130L0513039 '}
[jcc][SystemMonitor:stop] core: 0.28737999999999997ms | network: 0.0ms | server: 0.0ms
[jcc][SystemMonitor:start]
[jcc][Time:2015-12-10-14:39:24.852][Thread:main][Connection#3b6eb2ec] setClientInfo ({ApplicationName=crsJCC_application}) called
[jcc][SystemMonitor:stop] core: 3.613203ms | network: 0.0ms | server: 0.0ms
[jcc][SystemMonitor:start]
[jcc][Time:2015-12-10-14:39:24.856][Thread:main][Connection#3b6eb2ec] getClientInfo () called
[jcc][Time:2015-12-10-14:39:24.856][Thread:main][Connection#3b6eb2ec] getClientInfo () returned {ClientUser=XXXXXX00, ApplicationName=xxxxxx_application, ClientHostname=L0513039, ClientAccountingInformation=JCC04130L0513039 '}
[jcc][SystemMonitor:stop] core: 0.24718099999999998ms | network: 0.0ms | server: 0.0ms
[jcc][Time:2015-12-10-14:39:24.857][Thread:main][Connection#3b6eb2ec] createStatement () called
[jcc][Time:2015-12-10-14:39:24.863][Thread:main][Connection#3b6eb2ec] createStatement () returned Statement#5ebec15
[jcc
In this case I amended the ApplicationName within ClientInfo, what I would like is that the initial value as customised, e.g. that the initial value WAS'NT "db2jcc_application" but "started out as "xxxxxx_application", is this possible?
Working with the DB2Driver you should be able to set application name like this:
Properties p= new Properties();
p.put("user", "admin");
p.put("password", "secret");
p.put("clientProgramName", "xxxx_application");
Connection conn = DriverManager.getConnection(
"jdbc:db2://localhost:50000/yourdb", props);
If you're using a DB2DataSource, check out this information from IBM:
com.ibm.db2.jcc.DB2DataSource ds =
new com.ibm.db2.jcc.DB2DataSource();
ds.setDriverType(4);
ds.setServerName("localhost");
ds.setPortNumber(50000);
ds.setDatabaseName("sample");
ds.setUser("username");
ds.setPassword("password");
ds.setClientProgramName("My application");
As stated here:
clientProgramName
Specifies an application ID that is fixed for the duration of a physical connection for a client. The value of this property becomes the correlation ID on a DB2 for z/OS server. Database administrators can use this property to correlate work on a DB2 for z/OS server to client applications. The data type of this property is String. The maximum length is 12 bytes. If this value is null, the IBM DB2 Driver for JDBC and SQLJ supplies a value of db2jccthread-name.
I have web application. And I have jdbc database pooling system.
If my minIdle and initialSize value are 50, I have the following error:
[ WARN] [http-nio-8080-exec-25 03:11:33] (SqlExceptionHelper.java:logExceptions:144) SQL Error: 12519, SQLState: 66000
[ERROR] [http-nio-8080-exec-25 03:11:33] (SqlExceptionHelper.java:logExceptions:146) Listener refused the connection with the following error:
ORA-12519, TNS:no appropriate service handler found
might be this is because I should configure XE oracle to get many connections.
I do something like this:
ALTER SYSTEM SET processes=10000 scope=spfile
But this did not help me.
Also:
SQL> connect
Enter user-name: system
Enter password:
Connected.
SQL> select count(*) from v$process;
COUNT(*)
----------
44
SQL> show parameter processes;
NAME TYPE VALUE
------------------------------------ ----------- -----------------------
aq_tm_processes integer 0
db_writer_processes integer 1
gcs_server_processes integer 0
global_txn_processes integer 1
job_queue_processes integer 4
log_archive_max_processes integer 4
processes integer 100
SQL>
I have windows 7 x64. and this is Oracle Express edition.
If too many connection opened within a short time can cause this. You should check in your processes.
You may make system sleep for some time after some connection.
I am using Oracle database. I've written a small JDBC connection program in Java but I am facing an issue with the listener.
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.PreparedStatement;
import java.sql.ResultSet;
import java.sql.SQLException;
public class JdbcConnection {
public static void main(String[] args) throws SQLException,ClassNotFoundException {
String url = "jdbc:oracle:thin:#localhost:1521:orcl";
String user = "system";
String password = "password";
Connection connection = null;
Class.forName("oracle.jdbc.driver.OracleDriver");
connection = DriverManager.getConnection(url, user, password);
if(connection!=null){
System.out.println("Success in connnection");
} else {
System.out.println("failure in connection ");
}
}
}
I am getting the following exception:
C:\Users\Administrator\Desktop>java JdbcConnection
Exception in thread "main" java.sql.SQLException: Listener refused the connectio
n with the following error:
ORA-12505, TNS:listener does not currently know of SID given in connect descriptor
The Connection descriptor used by the client was:
localhost:1521:orcl
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java
:112)
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java
:261)
at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:387)
at oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:
441)
at oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:165)
at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtensio
n.java:35)
at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:801)
at java.sql.DriverManager.getConnection(Unknown Source)
at java.sql.DriverManager.getConnection(Unknown Source)
at JdbcConnection.main(JdbcConnection.java:18)
This is the output of lsnrctl status
LSNRCTL for 64-bit Windows: Version 12.1.0.1.0 - Production on 16-JUN-2015 13:43
:41
Copyright (c) 1991, 2013, Oracle. All rights reserved.
Welcome to LSNRCTL, type "help" for information.
LSNRCTL> status
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=EXTPROC1521)))
STATUS of the LISTENER
------------------------
Alias LISTENER
Version TNSLSNR for 64-bit Windows: Version 12.1.0.1.0 - Produ
ction
Start Date 16-JUN-2015 12:02:52
Uptime 0 days 1 hr. 40 min. 52 sec
Trace Level off
Security ON: Local OS Authentication
SNMP OFF
Listener Parameter File C:\app\orauser\product\12.1.0\dbhome_1\network\admin\l
istener.ora
Listener Log File C:\app\orauser\diag\tnslsnr\hydwemvm\listener\alert\lo
g.xml
Listening Endpoints Summary...
(DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(PIPENAME=\\.\pipe\EXTPROC1521ipc)))
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=hydwemvm)(PORT=1521)))
Services Summary...
Service "CLRExtProc" has 1 instance(s).
Instance "CLRExtProc", status UNKNOWN, has 1 handler(s) for this service...
The command completed successfully
If you know your oracle database SID, then use
jdbc:oracle:thin:#localhost:1521:orcl
otherwise use below in case you have service name
jdbc:oracle:thin:#localhost:1521/orcl
Also, make sure service name with the name ORCL should be up and running. If still doesn't work, then you need to restart your machine and try again above.
Still, not working ? Then, try following :
Login with SYSTEM user and register LOCAL_LISTENER by running below SQLs.
alter system set local_listener = '(ADDRESS=(PROTOCOL=TCP)(HOST=localhost)(PORT=1521))' scope = both;
alter system register;
How to check oracle SID and service name :
SELECT sys_context('USERENV', 'SID') FROM DUAL; -- It will return your oracle database SID
SELECT sys_context('USERENV', 'SERVICE_NAME') FROM DUAL; -- It will return your oracle database service name
If you want to know the default SID of your database use this query in sqlplus:
SELECT sys_context('USERENV', 'SID') FROM DUAL;
Use this value in the JDBC URL instead of "orcl".
Can you use the below URL?
Note the difference, this is to use the SERVICENAME instead of a SID.
jdbc:oracle:thin:#localhost:1521/orclservice
I am facing the same problem.
Try removing the LAN cable or disconnect your net connectivity and
restart the services of Listener and run the code.
It worked for me.