Hi I am trying to write a java client for secure hbase.
I want to do kinit also from code itself for that i`m using the usergroup information class.
Can anyone point out where am I going wrong here?
this is the main method that Im trying to connect o hbase from.
I have to add the configuration in the CONfiguration object rather than using the xml, because the client can be located anywhere.
Please see the code below:
public static void main(String [] args) {
try {
System.setProperty(CommonConstants.KRB_REALM, ConfigUtil.getProperty(CommonConstants.HADOOP_CONF, "krb.realm"));
System.setProperty(CommonConstants.KRB_KDC, ConfigUtil.getProperty(CommonConstants.HADOOP_CONF,"krb.kdc"));
System.setProperty(CommonConstants.KRB_DEBUG, "true");
final Configuration config = HBaseConfiguration.create();
config.set(CommonConfigurationKeysPublic.HADOOP_SECURITY_AUTHENTICATION, AUTH_KRB);
config.set(CommonConfigurationKeysPublic.HADOOP_SECURITY_AUTHORIZATION, AUTHORIZATION);
config.set(CommonConfigurationKeysPublic.FS_AUTOMATIC_CLOSE_KEY, AUTO_CLOSE);
config.set(CommonConfigurationKeysPublic.FS_DEFAULT_NAME_KEY, defaultFS);
config.set("hbase.zookeeper.quorum", ConfigUtil.getProperty(CommonConstants.HBASE_CONF, "hbase.host"));
config.set("hbase.zookeeper.property.clientPort", ConfigUtil.getProperty(CommonConstants.HBASE_CONF, "hbase.port"));
config.set("hbase.client.retries.number", Integer.toString(0));
config.set("zookeeper.session.timeout", Integer.toString(6000));
config.set("zookeeper.recovery.retry", Integer.toString(0));
config.set("hbase.master", "gauravt-namenode.pbi.global.pvt:60000");
config.set("zookeeper.znode.parent", "/hbase-secure");
config.set("hbase.rpc.engine", "org.apache.hadoop.hbase.ipc.SecureRpcEngine");
config.set("hbase.security.authentication", AUTH_KRB);
config.set("hbase.security.authorization", AUTHORIZATION);
config.set("hbase.master.kerberos.principal", "hbase/gauravt-namenode.pbi.global.pvt#pbi.global.pvt");
config.set("hbase.master.keytab.file", "D:/var/lib/bda/secure/keytabs/hbase.service.keytab");
config.set("hbase.regionserver.kerberos.principal", "hbase/gauravt-datanode2.pbi.global.pvt#pbi.global.pvt");
config.set("hbase.regionserver.keytab.file", "D:/var/lib/bda/secure/keytabs/hbase.service.keytab");
UserGroupInformation.setConfiguration(config);
UserGroupInformation userGroupInformation = UserGroupInformation.loginUserFromKeytabAndReturnUGI("hbase/gauravt-datanode2.pbi.global.pvt#pbi.global.pvt", "D:/var/lib/bda/secure/keytabs/hbase.service.keytab");
UserGroupInformation.setLoginUser(userGroupInformation);
User user = User.create(userGroupInformation);
user.runAs(new PrivilegedExceptionAction<Object>() {
#Override
public Object run() throws Exception {
HBaseAdmin admins = new HBaseAdmin(config);
if(admins.isTableAvailable("ambarismoketest")) {
System.out.println("Table is available");
};
HConnection connection = HConnectionManager.createConnection(config);
HTableInterface table = connection.getTable("ambarismoketest");
admins.close();
System.out.println(table.get(new Get(null)));
return table.get(new Get(null));
}
});
System.out.println(UserGroupInformation.getLoginUser().getUserName());
} catch (Exception e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
I`m getting the following exception:
Caused by: org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException): GSS initiate failed
at org.apache.hadoop.hbase.security.HBaseSaslRpcClient.readStatus(HBaseSaslRpcClient.java:110)
at org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslRpcClient.java:146)
at org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupSaslConnection(RpcClient.java:762)
at org.apache.hadoop.hbase.ipc.RpcClient$Connection.access$600(RpcClient.java:354)
at org.apache.hadoop.hbase.ipc.RpcClient$Connection$2.run(RpcClient.java:883)
at org.apache.hadoop.hbase.ipc.RpcClient$Connection$2.run(RpcClient.java:880)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
at org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:880)
... 33 more
Any pointers would be helpful.
The above works nicely, but I've seen a lot of folks struggle with setting all of the right properties in the Configuration object. There's no de-facto list that I've found of exactly what you need and don't need and it is painfully dependent on your cluster configuration.
The surefire way is to have a copy of your HBase configurations in your classpath, since your client can be anywhere as you mentioned. Then you can add the resources to your object without having to specify all properties.
Configuration conf = HBaseConfiguration.create();
conf.addResource("core-site.xml");
conf.addResource("hbase-site.xml");
conf.addResource("hdfs-site.xml");
Here were some sources to back this approach:
IBM,
Scalding (Scala)
Also note that this approach doesn't limit you to actually use the internal Zookeeper principal and keytab, i.e. you can create keytabs for applications or Active Directory users and leave the internally generated keytabs for the daemons to authenticate amongst themselves.
Not sure if you still need help. I think setting the "hadoop.security.authentication" property is missing from your snippet.
I am using following code snippet to connect to secure HBase (on CDH5). You can give a try.
config.set("hbase.zookeeper.quorum", zookeeperHosts);
config.set("hbase.zookeeper.property.clientPort", zookeeperPort);
config.set("hadoop.security.authentication", "kerberos");
config.set("hbase.security.authentication", "kerberos");
config.set("hbase.master.kerberos.principal", HBASE_MASTER_PRINCIPAL);
config.set("hbase.regionserver.kerberos.principal", HBASE_RS_PRINCIPAL);
UserGroupInformation.setConfiguration(config);
UserGroupInformation.loginUserFromKeytab(ZOOKEEPER_PRINCIPAL,ZOOKEEPER_KEYTAB);
HBaseAdmin admins = new HBaseAdmin(config);
TableName[] tables = admins.listTableNames();
for(TableName table: tables){
System.out.println(table.toString());
}
in Jdk 1.8, you need set
"System.setProperty("javax.security.auth.useSubjectCredsOnly", "false");"
config.set("hbase.zookeeper.quorum", zookeeperHosts);
config.set("hbase.zookeeper.property.clientPort", zookeeperPort);
config.set("hadoop.security.authentication", "kerberos");
config.set("hbase.security.authentication", "kerberos");
config.set("hbase.master.kerberos.principal", HBASE_MASTER_PRINCIPAL);
config.set("hbase.regionserver.kerberos.principal", HBASE_RS_PRINCIPAL);
System.setProperty("javax.security.auth.useSubjectCredsOnly", "false");
UserGroupInformation.setConfiguration(config);
UserGroupInformation.loginUserFromKeytab(ZOOKEEPER_PRINCIPAL,ZOOKEEPER_KEYTAB);
HBaseAdmin admins = new HBaseAdmin(config);
TableName[] tables = admins.listTableNames();
for(TableName table: tables){
System.out.println(table.toString());
}
quote:
http://hbase.apache.org/book.html#trouble.client
question: 142.9
I think the best is https://scalding.io/2015/02/making-your-hbase-client-work-in-a-kerberized-environment/
To make the code work you don’t have to change any line from the one written in the top of this post, you just have to make your client able to access the full HBase configuration. This just implies to change your running classpath to:
/opt/cloudera/parcels/CDH-5.3.0-1.cdh5.3.0.p0.30/lib/hbase/conf:target/scala-2.11/hbase-assembly-1.0.jar
This will make everything run smoothly. It is specific for CDH 5.3 but you can adapt it for your cluster configuration.
PS No need in this:
conf.addResource("core-site.xml");
conf.addResource("hbase-site.xml");
conf.addResource("hdfs-site.xml");
Because HBaseConfiguration has
public static Configuration addHbaseResources(Configuration conf) {
conf.addResource("hbase-default.xml");
conf.addResource("hbase-site.xml");
Related
I tried Connecting the AWS Neptune with this Java code and got the error , NoHostAvailable Exception
approach 1:
public static void main(String[] args) throws Exception {
Cluster.Builder builder = Cluster.build();
builder.addContactPoint("endpoint");
builder.port(8182);
builder.enableSsl(true);
builder.keyStore("pem-file");
Cluster cluster = builder.create();
GraphTraversalSource g = traversal().withRemote(DriverRemoteConnection.using(cluster));
System.out.println(g.V().limit(10).toList());
cluster.close();
}}
approach 2:
Cluster cluster = Cluster.build("endpoint").
enableSsl(true).keyStore("pem").
handshakeInterceptor( r -> {
NeptuneNettyHttpSigV4Signer sigV4Signer = null;
try {
sigV4Signer = new NeptuneNettyHttpSigV4Signer("us-east-2", new
DefaultAWSCredentialsProviderChain());
} catch (NeptuneSigV4SignerException e) {
e.printStackTrace();
}
try {
sigV4Signer.signRequest(r);
} catch (NeptuneSigV4SignerException e) {
e.printStackTrace();
}
return r;
}).create();
Client client=Cluster.open("src\\conf\\remote-objects.yaml").connect();
client.submit("g.V().limit(10).toList()").all().get();
what ever I do, I am getting this error:
Sep 02, 2021 3:18:34 PM io.netty.channel.ChannelInitializer exceptionCaught
WARNING: Failed to initialize a channel. Closing:
java.lang.RuntimeException: java.lang.NullPointerException
org.apache.tinkerpop.gremlin.driver.Channelizer$AbstractChannelizer.initChannel(Channelizer.java:117)
Caused by: org.apache.tinkerpop.gremlin.driver.exception.NoHostAvailableException: All hosts
are considered unavailable due to previous exceptions. Check the error log to find the actual
reason.
I need the code or the document to connect my Gremlin code in .java file to AWS neptune. I am struggling and tried various number of ways,
1.created EC2 instance and did installed maven and apache still got error and code is running in Server(EC2), i want code to present in IntelliJ
it would be more helpful, if I get the Exact Code any way. what should be added in remote-objects.yaml.
if we require Pem-file to access Amazon Neptune, please help with the creation of it.
Assuming SSL is enabled but IAM is not, in terms of Java code, this is all you need to create the connection.
Cluster.Builder builder = Cluster.build();
builder.addContactPoint("localhost");
builder.port(8182);
builder.enableSsl(true);
builder.serializer(Serializers.GRAPHBINARY_V1D0);
cluster = builder.create();
drc = DriverRemoteConnection.using(cluster);
g = traversal().withRemote(drc);
You may need to add an entry to your /etc/hosts file to get the SSL certs to resolve correctly such as:
127.0.0.1 localhost my-neptune-cluster.us-east-1.neptune.amazonaws.com
If you find that using localhost with SSL enabled does not work then use the actual Neptune cluster DNS name and make the edit to your /etc/hosts file.
The last thing you will need to do is create access to the Neptune VPC from your local machine. One way is using an SSH tunnel as explained in this post
I have this very interesting issue that has been making me tear down my hair for the past two days.
I am using jUnit and an embedded Tomcat to test some API endpoints (Jersey).
I'm not a big fan of mocking and I made this setup to test the API responses in conditions as close as possible to production.
When the API receives a call, it should supply a response accordingly (found, not found, etc). This is where Hibernate comes in.
When I run this on my Tomcat set up in eclipse, or when I deploy the build (Maven) on a standalone Tomcat on the remote server, everything works fine, but when the API is called on the embedded Tomcat during the test I get this error:
java.lang.ClassCastException: com.models.listing.Listing cannot be
cast to com.models.listing.Listing
Yes, it's the same class name.
To retrieve the Listing object I use the Hibernate standard of getting a persisted object by ID
Listing listing = session.get(Listing.class, ID);
This is how the embedded tomcat setup looks like:
public void start(String appName) throws Exception {
File root = getRootFolder();
System.setProperty("org.apache.catalina.startup.EXIT_ON_INIT_FAILURE", "true");
Path tempPath = Files.createTempDirectory("tomcat-base-dir");
tomcat = new Tomcat();
tomcat.setPort(0);
tomcat.enableNaming();
tomcat.setSilent(true);
tomcat.setBaseDir(tempPath.toString());
tomcat.getHost().setDeployOnStartup(true);
tomcat.getHost().setAutoDeploy(true);
tomcat.getHost().setAppBase(tempPath.toString());
File webContentFolder = new File(root.getAbsolutePath(), "src/main/webapp/");
if (!webContentFolder.exists()) {
webContentFolder = Files.createTempDirectory("default-doc-base").toFile();
}
StandardContext ctx = (StandardContext) tomcat.addWebapp("/" + appName, webContentFolder.getAbsolutePath());
//Disable TLD scanning by default
if (System.getProperty(Constants.SKIP_JARS_PROPERTY) == null ) {
System.out.println("disabling TLD scanning");
StandardJarScanFilter jarScanFilter = (StandardJarScanFilter) ctx.getJarScanner().getJarScanFilter();
jarScanFilter.setTldSkip("*");
}
System.out.println("configuring app with basedir: " + webContentFolder.getAbsolutePath());
// Declare an alternative location for your "WEB-INF/classes" dir
// Servlet 3.0 annotation will work
File additionWebInfClassesFolder = new File(root.getAbsolutePath(), "target/classes");
WebResourceRoot resources = new StandardRoot(ctx);
WebResourceSet resourceSet;
if (additionWebInfClassesFolder.exists()) {
resourceSet = new DirResourceSet(resources, "/WEB-INF/classes", additionWebInfClassesFolder.getAbsolutePath(), "/");
System.out.println("loading WEB-INF resources from as '" + additionWebInfClassesFolder.getAbsolutePath() + "'");
} else {
resourceSet = new EmptyResourceSet(resources);
}
resources.addPreResources(resourceSet);
ctx.setResources(resources);
//start tomcat
tomcat.start();
}
And this is how the Hibernate configuration looks like:
private static SessionFactory buildSessionFactory() {
// setup the session factory
Configuration configuration = new Configuration();
//add annotated classes
configuration.addAnnotatedClass(Listing.class);
//connection properties
configuration.setProperty("hibernate.dialect", "org.hibernate.dialect.MySQL5InnoDBDialect");
configuration.setProperty("hibernate.connection.driver_class", "com.mysql.jdbc.Driver");
configuration.setProperty("hibernate.connection.url", "jdbc:mysql://<some IP>:3306/<some db name>");
configuration.setProperty("hibernate.connection.release_mode", "auto");
configuration.setProperty("hibernate.connection.username", "<some username>");
configuration.setProperty("hibernate.connection.password", "<some password>");
configuration.setProperty("hibernate.connection.provider_class", "org.hibernate.connection.C3P0ConnectionProvider");
//sql properties
configuration.setProperty("show_sql", "true");
configuration.setProperty("format_sql", "true");
//misc properties
configuration.setProperty("hibernate.hbm2ddl.auto", "validate");
configuration.setProperty("hibernate.current_session_context_class", "thread");
//c3p0 properties
configuration.setProperty("hibernate.c3p0.min_size", "1");
configuration.setProperty("hibernate.c3p0.max_size", "10");
configuration.setProperty("hibernate.c3p0.timeout", "100");
configuration.setProperty("hibernate.c3p0.max_statements", "50");
configuration.setProperty("hibernate.c3p0.idle_test_period", "1000");
configuration.setProperty("hibernate.c3p0.validate", "true");
//return null
return configuration.buildSessionFactory();
}
To summarize, the test starts in jUnit, the embedded tomcat instance fires up and with a REST client I send a request to an API endpoint. The endpoint responds after it retrieves the resource from Hibernate.
Dependency versions:
Glassfish Jersey 2.23
jUnit 4.11
Tomcat embedded 8.5.3
Hibernate 5.2.1
My best bet is some issue with the class loader.
I know that the JVM sees classes as different if they were loaded with different class loaders even if it's basically the same class from the same package and so on, but I just don't seem to find a way to make this work.
Maybe my hypothesis is completely off and I am missing something here, so if anyone has encountered something like this or has some suggestions (I already tried tens of "solutions") please jump in.
Thanks everyone for the help in advance!
I still didn't figure out the mystery and I just let go after some other "trial and error" ideas with class loaders and so on.
I "fixed" it by downgrading to hibernate 4.3.11
the code below allows me to create an embedded undertow servlet server, i have a problem setting the 'max-parameters' of the connector settings, the way i understand it undertow is normally configured via xml file.
public static String initCustomServer_(Servlet servlet,int preferedPort,String servletName,String[] resourceList,String... domainName){
String contextURL = null;
int curPort = preferedPort==-1?9001:preferedPort;
boolean initServ = false;
System.out.println("====servlet running in local mode====");
while(!initServ) {
try{
io.undertow.servlet.api.DeploymentInfo servletBuilder = io.undertow.servlet.Servlets.deployment()
.setClassLoader(servlet.getClass().getClassLoader())
.setContextPath(domainName.length==0?"/":"/"+domainName[0])
.setDeploymentName("test.war")
.addServlets(
io.undertow.servlet.Servlets.servlet(servletName, servlet.getClass()).addMapping("/"+servletName)
)
.setResourceManager(new io.undertow.server.handlers.resource.FileResourceManager(new File("src/dss_core/HTML5/webapp"), 1));
io.undertow.servlet.api.DeploymentManager manager = io.undertow.servlet.Servlets.defaultContainer().addDeployment(servletBuilder);
manager.deploy();
io.undertow.server.HttpHandler servletHandler = manager.start();
io.undertow.server.handlers.PathHandler path = io.undertow.Handlers.path(io.undertow.Handlers.redirect(domainName.length==0?"/":"/"+domainName[0]))
.addPrefixPath(domainName.length==0?"/":"/"+domainName[0], servletHandler);
io.undertow.Undertow server = io.undertow.Undertow.builder()
.addHttpListener(curPort, "localhost")
.setHandler(path)
.build();
server.start();
initServ = true;
contextURL = "http://localhost:"+curPort+(domainName.length==0?"":"/"+domainName[0])+"/"+servletName;
} catch (Exception ex) {
//creation of server at certain port fails therefore try again on another port
System.err.println(" server unable to initialize :" + ex.getMessage());
ex.printStackTrace();
curPort++;
}
}
return contextURL;
}
rather than using an xml like the one below how do i change configurations such as 'max-parameter' via embedded java code?
<server name="default-server">
<http-listener name="default" socket-binding="http" max-parameters="5000"/>
found here are list of stuff that i can configure via xml how can i set them via java code?
UPDATE 1: yay found some options in io.undertow.UndertowOptions, how ever this doesn't work as it is declared final, what now?
io.undertow.UndertowOptions.MAX_PARAMETERS = 10000;
after hours of research and trial and error finally i got it, my first idea was to simply get the code and compile it myself, negative side of that is that i'd have to download all source code then compile it, such proved to be trouble some and i decided to quit after seeing endless dependencies and hours of downloading their source code. configuring the server looked like this
io.undertow.Undertow server = io.undertow.Undertow.builder()
.addHttpListener(curPort, "localhost")
.setHandler(path)
.setServerOption(io.undertow.UndertowOptions.MAX_PARAMETERS, 10000)
.setServerOption(io.undertow.UndertowOptions.OPTION2, Value2)
.build();
setServerOption method and io.undertow.UndertowOptions class finally made sense, it's too bad undertow isn't very popular and not much sample code lying around, i hope i help anybody wishing to take the embedded road of undertow
I have created a java application in netbeans, and I intend to create an installer for the project.
For this I have created a jar file of the application, but I'm using the mysql database localhost.
How can I generate Jar File with Mysql localhost .
Can anyone help me please?
Thanks and regards
-----------------------------Edit---------------------------------------
Maybe not the best way to express myself, what I mean is that the database is created locally (localhost).
The connection of the application with the database is done this way:
Class.forName("com.mysql.jdbc.Driver");
return driverManager.getConnection("jdbc:mysql://localhost/database","root", "root");
I want to create a jar file of my application that has a database created locally.
I am going to explain a few things:
You do not need to hard - code the Connect URL into your code. This is why you are asking for ways of creating the database as localhost. I suggest you do not hard code the Connect URL in the code. Instead write it in an editable File either a Properties file or even a text file. Let the Application read the Editable file and pass the Parameters to the Code.
An Application running in your Local Machine where the database is will connect using Localhost. But a the same Application running remotely from another Machine whether in the Internet or Local access network will not Connect this way.That is why I am insisting on NOT Hard-Coding the Connect String.
The database Name, user, and Password Including the Host will change from time to Time depending on which environment the Application is running in. So again if the environment changes and the variables are not the same the Application will Not Connect to the database.
Suggestion:
User a Properties file:
db.host=192.168.1.23
db.user=root
db.password=root
db.dbname=database
Load the file as a Properties file:
Properties prop = new Properties();
InputStream input = null;
try {
input = new FileInputStream("config.properties");
// load a properties file
prop.load(input);
// get the property value and print it out
System.out.println(prop.getProperty("db.host"));
System.out.println(prop.getProperty("db.user"));
System.out.println(prop.getProperty("db.password"));
System.out.println(prop.getProperty("db.dbname"));
//PASS YOUR CONNECT STRING
Class.forName("com.mysql.jdbc.Driver");
Connection conn = DriverManager.getConnection("jdbc:mysql://" + prop.getProperty("db.host") + "/" + prop.getProperty("db.dbname"), prop.getProperty("db.user"), prop.getProperty("db.password"));
} catch (IOException ex) {
ex.printStackTrace();
} finally {
if (input != null) {
try {
input.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
This way you will never have to worry about what database the application is running on as you will just have to edit the config.properties and the Application will do the rest.
I hope I gave you an answer or better still other ideas on how to handle your situation.
I'm writing a code which manipulates data stored in HBase. I want to write also a test for this code. I want to use HBaseTestingUtility in my test, hence, in my #BeforeClass I create new instance of HBaseTestingUtility and I start the mini cluster:
#BeforeClass
public static void setUpClass() throws Exception {
utility = new HBaseTestingUtility();
utility.startMiniCluster();
}
It works good. However, I cannot connect to this embedded cluster in my code which is being tested. In the code I have:
Configuration config = HBaseConfiguration.create();
try (HBaseAdmin admin = new HBaseAdmin(config))
{
//code which manipulates the data
}
Unfortunately when new HBaseAdmin is created I'm getting a ConnectionError exception:
2013-11-21 11:20:35,778 WARN [main-SendThread(0:0:0:0:0:0:0:1:2181)] zookeeper.ClientCnxn (ClientCnxn.java:run(1089)) - Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused: no further information
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:692)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
When I try another apporach to create HBaseAdmin:
try (HBaseAdmin admin = new HBaseAdmin(HBaseCacheTest.utility.getConfiguration()))
{
//code which manipulates the data
}
it works (note that here I'm accessing the instance of HBaseTestingUtility via HBaseCacheTest.utility).
Obviously this is not a good approach as I don't want to have a production code which depends on the testing code.
One approach I see to work here is to make it possible in my production class to set the Configuration and use this setter method in my tests.
However, I believe that there should be another way to connect to the embeded mini cluster created with HBaseTestingUtility.
Any ideas?
You should not create a new Configuration object, but use the one provided by HBaseTestingUtility:
HBaseAdmin admin = testingUtility.getHBaseAdmin();
Also, if you need configuration only you can use this one:
testingUtility.getConfiguration()
Just rechecked it in my UT. I'm using HBase 0.96-hadoop2
Hope this helps :)