When I run this programm it show this mistake. Dose it because i donot run the service? How to writ config files for the following code?
com.zeroc.Ice.ObjectPrx obj = communicator.stringToProxy("IceStorm/TopicManager:tcp -p 10000");
com.zeroc.IceStorm.TopicManagerPrx topicManager = com.zeroc.IceStorm.TopicManagerPrx.checkedCast(obj);
You need to run the IceStorm service to be able to connect to it, IceStorm/TopicManager is an object hosted by IceStorm service.
if you want to define the topic manager using a property you should use communicator.propertyToProxy instead of communicator.stringToProxy and define the proxy in the configuration file used to initialize the configuration.
com.zeroc.Ice.Communicator communicator = com.zeroc.Ice.Util.initialize(args, "config.sub", extraArgs);
com.zeroc.IceStorm.TopicManagerPrx manager = com.zeroc.IceStorm.TopicManagerPrx.uncheckedCast(
communicator.propertyToProxy("TopicManager.Proxy"));
You should check IceStorm clock demo.
I try to connect with HBase in a remote server using java. Below is my java code
String zookeeperHost = "myserverIP";
String tableName = "User";
Configuration hconfig = HBaseConfiguration.create();
hconfig.setInt("timeout", 1200);
hconfig.set("hbase.zookeeper.quorum",zookeeperHost);
hconfig.set("hbase.zookeeper.property.clientPort", "2181");
TableName tname = TableName.valueOf(tableName);
try {
HBaseAdmin.checkHBaseAvailable(hconfig);
} catch (ServiceException e) {
e.printStackTrace();
}
HTableDescriptor htable = new HTableDescriptor(tname);
htable.addFamily( new HColumnDescriptor("Id"));
htable.addFamily( new HColumnDescriptor("Name"));
System.out.println( "Connecting..." );
Connection connection = ConnectionFactory.createConnection(hconfig);
HBaseAdmin hbase_admin = (HBaseAdmin)connection.getAdmin();
System.out.println( "Creating Table..." );
hbase_admin.createTable( htable );
System.out.println("Done!");
but I keep getting this exception
org.apache.hadoop.hbase.MasterNotRunningException:
com.google.protobuf.ServiceException: java.net.ConnectException: Connection >refused: no further information
How can I resolve this issue?
There could be two reasons for below exception
org.apache.hadoop.hbase.MasterNotRunningException
Remote VM hostname/IP is not accessible by client machine. In order to fix that, you need to update the inbound rules to allow port access. If in case of a VM host name, make sure you have added an entry in /etc/hosts to resolve the host name with IP address.
Second issue could be the HBase master is running. Make sure if HMaster service is running from your remote server.
Add below in 'hconfig'
hconfig.set("hbase.master", hbaseConnectionIpAddr);
hconfig.set("hbase.master.port", hbaseConnectionPortNum);
I am trying to connect to Hbase from Java. Hbase -Version 1.0.0
But I am unable to connect it. Kindly tell me what I am missing as I am new to Hbase.
Here is my code
public class HbaseAddRetrieveData{
public static void main(String[] args) throws IOException {
TableName tableName = TableName.valueOf("stock-prices");
Configuration conf = HBaseConfiguration.create();
conf.set("hbase.master","LocalHost:60000");
conf.set("hbase.zookeeper.property.clientPort", "2181");
conf.set("hbase.zookeeper.quorum", "LocalHost");
conf.set("zookeeper.znode.parent", "/hbase-unsecure");
System.out.println("Config set");
Connection conn = ConnectionFactory.createConnection(conf);
System.out.println("Connection");
Admin admin = conn.getAdmin();
if (!admin.tableExists(tableName)) {
System.out.println("In admin");
admin.createTable(new HTableDescriptor(tableName).addFamily(new HColumnDescriptor("cf")));
}
Table table = conn.getTable(tableName);
Put p = new Put(Bytes.toBytes("AAPL10232015"));
p.addColumn(Bytes.toBytes("cf"), Bytes.toBytes("close"), Bytes.toBytes(119));
table.put(p);
Result r = table.get(new Get(Bytes.toBytes("AAPL10232015")));
System.out.println(r);
}
Below is the error I am facing :
Exception in thread "main" org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the locations
at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:305)
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:131)
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:56)
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:287)
at org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:267)
at org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:139)
at org.apache.hadoop.hbase.client.ClientScanner.<init>(ClientScanner.java:134)
at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:823)
at org.apache.hadoop.hbase.MetaTableAccessor.fullScan(MetaTableAccessor.java:601)
at org.apache.hadoop.hbase.MetaTableAccessor.tableExists(MetaTableAccessor.java:365)
at org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:281)
at com.tcs.healthcare.HbaseRetrieveData.main(HbaseRetrieveData.java:32)
Kindly guide me through this
change L in localhost to l and 'H' to 'h'
Configuration conf = HBaseConfiguration.create();
conf.set("hbase.master","localhost:60000");
conf.set("hbase.zookeeper.property.clientPort", "2181");
conf.set("hbase.zookeeper.quorum", "localhost");
conf.set("zookeeper.znode.parent", "/hbase-unsecure");
If you are using remote machine, then conf.set("hbase.master","remotehost:60000"); even its not working then you check all the jars (which are remote) in the classpath
if you are using maven you can point to the jars which are there in cluster.
you can go to cluster and check below command to know the version of jars which are there in remote machine.
`hbase classpath`
Same version jars should be there in your client machine. For ex hbasex.jar in the remote machine and you are using hbasey.jar then it wont connect.
Moreover, please check from client machine you can able to ping server/cluster or not. Generally there will be firewall restrictions.
I am trying to setup a remote OrientDB Server and I am trying to enter vertices into it from local Java code.
When I try the following code:
private static final void dropDb() {
OrientGraphNoTx graph = new OrientGraphNoTx(ORIENT_URL);
graph.drop();
}
I get an Exception saying:
Exception in thread "main" com.orientechnologies.orient.core.exception.ODatabaseException: Cannot delete database
...
Caused by: java.lang.UnsupportedOperationException: Cannot delete a database in a remote server. Please use the console or the OServerAdmin class.
How do I go about deleting a remote graph, using Java, in OrientDB?
He jackofblades,
the Graph answers your question:
Please use the console or the OServerAdmin class
// CREATE A SERVER ADMIN CLIENT AGAINST A REMOTE SERVER
OServerAdmin serverAdmin = new OServerAdmin("remote:localhost/GratefulDeadConcerts").connect("admin", "admin");
serverAdmin.dropDatabase("GratefulDeadConcerts");
This is taken from the OrientDB Wiki Page
Patrick
Hi I am trying to write a java client for secure hbase.
I want to do kinit also from code itself for that i`m using the usergroup information class.
Can anyone point out where am I going wrong here?
this is the main method that Im trying to connect o hbase from.
I have to add the configuration in the CONfiguration object rather than using the xml, because the client can be located anywhere.
Please see the code below:
public static void main(String [] args) {
try {
System.setProperty(CommonConstants.KRB_REALM, ConfigUtil.getProperty(CommonConstants.HADOOP_CONF, "krb.realm"));
System.setProperty(CommonConstants.KRB_KDC, ConfigUtil.getProperty(CommonConstants.HADOOP_CONF,"krb.kdc"));
System.setProperty(CommonConstants.KRB_DEBUG, "true");
final Configuration config = HBaseConfiguration.create();
config.set(CommonConfigurationKeysPublic.HADOOP_SECURITY_AUTHENTICATION, AUTH_KRB);
config.set(CommonConfigurationKeysPublic.HADOOP_SECURITY_AUTHORIZATION, AUTHORIZATION);
config.set(CommonConfigurationKeysPublic.FS_AUTOMATIC_CLOSE_KEY, AUTO_CLOSE);
config.set(CommonConfigurationKeysPublic.FS_DEFAULT_NAME_KEY, defaultFS);
config.set("hbase.zookeeper.quorum", ConfigUtil.getProperty(CommonConstants.HBASE_CONF, "hbase.host"));
config.set("hbase.zookeeper.property.clientPort", ConfigUtil.getProperty(CommonConstants.HBASE_CONF, "hbase.port"));
config.set("hbase.client.retries.number", Integer.toString(0));
config.set("zookeeper.session.timeout", Integer.toString(6000));
config.set("zookeeper.recovery.retry", Integer.toString(0));
config.set("hbase.master", "gauravt-namenode.pbi.global.pvt:60000");
config.set("zookeeper.znode.parent", "/hbase-secure");
config.set("hbase.rpc.engine", "org.apache.hadoop.hbase.ipc.SecureRpcEngine");
config.set("hbase.security.authentication", AUTH_KRB);
config.set("hbase.security.authorization", AUTHORIZATION);
config.set("hbase.master.kerberos.principal", "hbase/gauravt-namenode.pbi.global.pvt#pbi.global.pvt");
config.set("hbase.master.keytab.file", "D:/var/lib/bda/secure/keytabs/hbase.service.keytab");
config.set("hbase.regionserver.kerberos.principal", "hbase/gauravt-datanode2.pbi.global.pvt#pbi.global.pvt");
config.set("hbase.regionserver.keytab.file", "D:/var/lib/bda/secure/keytabs/hbase.service.keytab");
UserGroupInformation.setConfiguration(config);
UserGroupInformation userGroupInformation = UserGroupInformation.loginUserFromKeytabAndReturnUGI("hbase/gauravt-datanode2.pbi.global.pvt#pbi.global.pvt", "D:/var/lib/bda/secure/keytabs/hbase.service.keytab");
UserGroupInformation.setLoginUser(userGroupInformation);
User user = User.create(userGroupInformation);
user.runAs(new PrivilegedExceptionAction<Object>() {
#Override
public Object run() throws Exception {
HBaseAdmin admins = new HBaseAdmin(config);
if(admins.isTableAvailable("ambarismoketest")) {
System.out.println("Table is available");
};
HConnection connection = HConnectionManager.createConnection(config);
HTableInterface table = connection.getTable("ambarismoketest");
admins.close();
System.out.println(table.get(new Get(null)));
return table.get(new Get(null));
}
});
System.out.println(UserGroupInformation.getLoginUser().getUserName());
} catch (Exception e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
I`m getting the following exception:
Caused by: org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException): GSS initiate failed
at org.apache.hadoop.hbase.security.HBaseSaslRpcClient.readStatus(HBaseSaslRpcClient.java:110)
at org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslRpcClient.java:146)
at org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupSaslConnection(RpcClient.java:762)
at org.apache.hadoop.hbase.ipc.RpcClient$Connection.access$600(RpcClient.java:354)
at org.apache.hadoop.hbase.ipc.RpcClient$Connection$2.run(RpcClient.java:883)
at org.apache.hadoop.hbase.ipc.RpcClient$Connection$2.run(RpcClient.java:880)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
at org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:880)
... 33 more
Any pointers would be helpful.
The above works nicely, but I've seen a lot of folks struggle with setting all of the right properties in the Configuration object. There's no de-facto list that I've found of exactly what you need and don't need and it is painfully dependent on your cluster configuration.
The surefire way is to have a copy of your HBase configurations in your classpath, since your client can be anywhere as you mentioned. Then you can add the resources to your object without having to specify all properties.
Configuration conf = HBaseConfiguration.create();
conf.addResource("core-site.xml");
conf.addResource("hbase-site.xml");
conf.addResource("hdfs-site.xml");
Here were some sources to back this approach:
IBM,
Scalding (Scala)
Also note that this approach doesn't limit you to actually use the internal Zookeeper principal and keytab, i.e. you can create keytabs for applications or Active Directory users and leave the internally generated keytabs for the daemons to authenticate amongst themselves.
Not sure if you still need help. I think setting the "hadoop.security.authentication" property is missing from your snippet.
I am using following code snippet to connect to secure HBase (on CDH5). You can give a try.
config.set("hbase.zookeeper.quorum", zookeeperHosts);
config.set("hbase.zookeeper.property.clientPort", zookeeperPort);
config.set("hadoop.security.authentication", "kerberos");
config.set("hbase.security.authentication", "kerberos");
config.set("hbase.master.kerberos.principal", HBASE_MASTER_PRINCIPAL);
config.set("hbase.regionserver.kerberos.principal", HBASE_RS_PRINCIPAL);
UserGroupInformation.setConfiguration(config);
UserGroupInformation.loginUserFromKeytab(ZOOKEEPER_PRINCIPAL,ZOOKEEPER_KEYTAB);
HBaseAdmin admins = new HBaseAdmin(config);
TableName[] tables = admins.listTableNames();
for(TableName table: tables){
System.out.println(table.toString());
}
in Jdk 1.8, you need set
"System.setProperty("javax.security.auth.useSubjectCredsOnly", "false");"
config.set("hbase.zookeeper.quorum", zookeeperHosts);
config.set("hbase.zookeeper.property.clientPort", zookeeperPort);
config.set("hadoop.security.authentication", "kerberos");
config.set("hbase.security.authentication", "kerberos");
config.set("hbase.master.kerberos.principal", HBASE_MASTER_PRINCIPAL);
config.set("hbase.regionserver.kerberos.principal", HBASE_RS_PRINCIPAL);
System.setProperty("javax.security.auth.useSubjectCredsOnly", "false");
UserGroupInformation.setConfiguration(config);
UserGroupInformation.loginUserFromKeytab(ZOOKEEPER_PRINCIPAL,ZOOKEEPER_KEYTAB);
HBaseAdmin admins = new HBaseAdmin(config);
TableName[] tables = admins.listTableNames();
for(TableName table: tables){
System.out.println(table.toString());
}
quote:
http://hbase.apache.org/book.html#trouble.client
question: 142.9
I think the best is https://scalding.io/2015/02/making-your-hbase-client-work-in-a-kerberized-environment/
To make the code work you don’t have to change any line from the one written in the top of this post, you just have to make your client able to access the full HBase configuration. This just implies to change your running classpath to:
/opt/cloudera/parcels/CDH-5.3.0-1.cdh5.3.0.p0.30/lib/hbase/conf:target/scala-2.11/hbase-assembly-1.0.jar
This will make everything run smoothly. It is specific for CDH 5.3 but you can adapt it for your cluster configuration.
PS No need in this:
conf.addResource("core-site.xml");
conf.addResource("hbase-site.xml");
conf.addResource("hdfs-site.xml");
Because HBaseConfiguration has
public static Configuration addHbaseResources(Configuration conf) {
conf.addResource("hbase-default.xml");
conf.addResource("hbase-site.xml");