I'm trying to connect to my HDFS instance running on Cloudera. My first step was enabling Kerberos and creating Keytabs (as shown here).
In the next step i would like to authenticate with a keytab.
Configuration conf = new Configuration();
conf.set("fs.defaultFS", "hdfs://cloudera:8020");
conf.set("hadoop.security.authentication", "kerberos");
UserGroupInformation.setConfiguration(conf);
UserGroupInformation.loginUserFromKeytab("hdfs#CLOUDERA", "/etc/hadoop/conf/hdfs.keytab");
FileSystem fs = FileSystem.get(conf);
FileStatus[] fsStatus = fs.listStatus(new Path("/"));
for (int i = 0; i < fsStatus.length; i++) {
System.out.println(fsStatus[i].getPath().toString());
}
It fails with the following error
java.io.IOException: Login failure for hdfs#CLOUDERA from keytab
/etc/hadoop/conf/hdfs.keytab:
javax.security.auth.login.LoginException: Unable to obtain password
from user
The question is: how do I correctly handle the keytab? Do i have to copy it to my local machine?
When running a Hadoop client on Windows to reach a kerberized cluster, you need a specific "native library" (i.e. DLL).
As far as I can tell there is no good reason for that, because that lib is not actually used outside of some automated regression tests (!?!) so it's a pain inflicted to Hadoop users by Hadoop committers.
To add extra pain, there is no official build of that DLL (and of the Windows "stub" that enable its use from Java). You must either (a) build it yourself from source code -- good luck -- or (b) search the internet for a downloadable Hadoop-for-Windows runtime, and pray that is does not contain any malware.
The best option (for 64-bit Windows) is here: https://github.com/steveloughran/winutils
...and the ReadMe explains why you can reasonably trust that run-time. But if you are stuck with an older 32-bit Windows, then you are on your own.
Now let's assume you deployed that run-time on your Windows box under
C:\Some Dir\hadoop\bin\(the final bin is required; the embedded space is just extra fun)
You must point the Hadoop client to that run-time with a couple of Java properties:
"-Dhadoop.home.dir=C:/Some Dir/hadoop" "-Djava.library.path=C:/Some Dir/hadoop/bin"
(note the double quotes around Windows args as a whole, to protect embedded spaces in the paths, which have been translated to Java style for extra fun)(in Eclipse, just stuff these props under "VM Arguments", quotes included)
Now, there's the Kerberos config. If your KDC is your corporate Active Directory server, then Java should find the config parameters automatically. But if your KDC is a standalone "MIT Kerberos" install on Linux, then you have to find a valid /etc/krb5.conf file on the cluster, copy it on your Windows box, and have Java use it with an additional property...
"-Djava.security.krb5.conf=C:/Some Other Dir/krb5.conf"
Then let's assume you have created your keytab file on a Linux box, using ktutil (or an Active Directory admin created it for you with some AD command) and you dropped the file under C:\Some Other Dir\foo.keytab
Before anything else, if the keytab is for a real Windows account -- i.e. your own account -- or a Prod service account, then make sure that keytab is secure!! Use the Windows Security dialog box to restrict access to your account only (and maybe System, for backups). Because that file could enable anyone, on any machine, to authenticate on the cluster (and any Kerberos-enabled system, including Windows).
Now you can try to authenticate using
UserGroupInformation.loginUserFromKeytab("foo#BAR.ORG", "C:/Some Other Dir/foo.keytab");
If it does not work, enable the Kerberos debug traces with both an environment variable
set HADOOP_JAAS_DEBUG=true
...and a Java property
-Dsun.security.krb5.debug=true
(in Eclipse, set these in "Environment" and "VM Arguments" respectively)
Do you have set proper permissions?
chown hdfs:hadoop /etc/hadoop/conf/hdfs.keytab
chmod 440 /etc/hadoop/conf/hdfs.keytab
Related
On an NTFS volume on Windows 10 Version 1803 build 17134.523 with Developer Mode enabled, I have a file myfile. I can make symbolic links to this file with mklink. However, if I call Files.createSymbolicLink on java jre 1.8.0_201, if throws:
java.nio.file.FileSystemException: linkname: A required privilege is not held by the client.
at sun.nio.fs.WindowsException.translateToIOException(Unknown Source)
at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source)
at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source)
at sun.nio.fs.WindowsFileSystemProvider.createSymbolicLink(Unknown Source)
at java.nio.file.Files.createSymbolicLink(Unknown Source)
at CreateLinks.main(CreateLinks.java:15)
The same works without problems on the Windows Subsystem for Linux (WSL) with jre 1.8.0_191-8u191-b12-0ubuntu0.18.10.1-b12
How can I make this work on windows without going into WSL? And where is this exception thrown exactly?
Creating symbolic links requires SeCreateSymbolicLinkPrivilege, unless the system is in developer mode and WinAPI CreateSymbolicLink is called with the flag SYMBOLIC_LINK_FLAG_ALLOW_UNPRIVILEGED_CREATE (*). CMD's mklink command uses this flag in Windows 10. Apparently Java JRE version 1.8.0_201 does not.
As to WSL, it inherits the security context from which it is run. If run from a logon that has SeCreateSymbolicLinkPrivilege, recent versions of WSL will create normal Windows symlinks on a drvfs (e.g. NTFS) volume. Otherwise WSL uses a custom symlink type, which is based on an IO_REPARSE_TAG_LX_SYMLINK (0xA000001D) reparse point instead of the normal IO_REPARSE_TAG_SYMLINK (0xA000000C) reparse point. You can query the type of reparse point via the command fsutil reparsepoint query <filename>.
(*) The docs say "[s]pecify this flag to allow creation of symbolic links when the process is not elevated". More precisely, this flag allows creating symlinks without SeCreateSymbolicLinkPrivilege, which is only related to being "elevated" with the default system settings. Personally, I grant this privilege to the "Authenticated Users" group, in which case creating symlinks doesn't require elevating to full admin access.
it's not related to your Java, it's just related to your os. please see the following link:
How to create Soft symbolic Link using java.nio.Files
Win10 with UAC turned off - I had to set Local Policies > Security Options > User Account Control: Run all administrators in Admin Approval Mode = Disabled - otherwise - same FileSystemException: A required privilege is not held by the client
In the Java API example they create a Datastore by using DatastoreHelper.getOptionsfromEnv
But this creates the warning
WARNING: Not using any credentials
and leads ultimately to:
DatastoreException(null): beginTransaction 401
I set my environment variables to the following:
export DATASTORE_DATASET={Project-ID}
export DATASTORE_HOST="https://www.googleapis.com/datastore/v1/datasets/{Project-ID}"
export DATASTORE_SERVICE_ACCOUNT="{email address}"
export DATASTORE_PRIVATE_KEY_FILE="{path to local p12 keyfile}"
But still when I try to see what the credentials are:
println("Datastore helper: " +DatastoreHelper.getOptionsfromEnv
.dataset(datasetId).build().getCredential)
I get null, what could be missing?
Also is there either a way to set the Credentials inside the project (instead of using the getOptionsfromEnv)?
The problem was that even though I used
source ~/.bash_profile
to refresh my environemnt variables and the echo command showed me that they were indeed updated, apperently I needed to restart my terminal (using Mac OSX) for them to be also updated for sbt and Scala.
I am not sure why this is the case and if this is Scala specific but now I managed to authenticate and communicate with the server.
I managed to figure it out by using the local installation of the Datastore Server and continuing to have the same problems.
I will like to setup performance monitoring on an application running on Railo 4 using New Relic. I have consulted the java docs on Railo, Railo google groups, etc but no one seems to have a perfect step by step.
Here is what I have done so far:
Extracted newrelic into Railo's install folder.
Added this line to setenv.sh
export JAVA_OPTS="$JAVA_OPTS -javaagent:c/railo/newrelic/newrelic.jar"
Restarted the Railo-Tomcat service
Added this line to the onapplicationstart function
application.NewRelic = createObject( "java", "com.newrelic.api.agent.NewRelic" );
Added this line to the onrequeststart function
if ( structKeyExists( application, "NewRelic" ) ) {
application.NewRelic.setTransactionName( "CFML", CGI.SCRIPT_NAME );
}
My application is still not sending metrics to New Relic. I will appreciate a step by step instruction of what to do as I can't seem to find that anywhere else and I have no idea what to do.
You can't use setenv.sh on Windows. Instead modify the catalina.bat file, or use the Configure Tomcat utility in the Start Menu to set the javaagent option. These steps can be found in more detail in the New Relic documentation
We have more detailed instructions for installing with our supported platforms and frameworks in our documentation. To see a list of the supported frameworks we are compatible with check out https://docs.newrelic.com/docs/java/new-relic-for-java#h2-compatibility
We may be able to work with the Tomcat portion of your environment and you can find helpful installation information at https://docs.newrelic.com/docs/java/java-agent-manual-installation
Should you run into any obstacles, I suggest opening a ticket at http://support.newrelic.com
I have PHP and the PHP/Java bridge set up on Windows and Tomcat 7.0. All is OK there.
However, I am trying to write a php file with java calls to connect to my Weblogic server installed on my Windows 7 machine to play around with adding/deleting users/groups etc.
This is what I followed: http://docs.oracle.com/cd/E13222_01/wls/docs90/jmx/accessWLS.html I also looked at this one http://weblogic-wonders.com/weblogic/2009/10/11/creating-users-in-weblogic-server-embedded-ldap-programatically/ (but for the second one I get an error on the java:comp/env/jmx/runtime line)
However, I cannot get my set up to recognize "t3" protocols to connect to my weblogic. the weird thing is I can connect via a t3 protocol using jython in a separate standalone script and to the console that way.
There is a section in first link above to make sure that wljmxclient.jar is added to the classpath. I set up a classpath to add that. I even use PHP to exec Weblogic's setWLSenv.cmd right before the connection to my Weblogic server is made to set my classpath . . . it shows the jar file in question being added. Still won't work. I then copied the jar file to the JavaBirde/WEB-INF/lib folder. Still wont' work.
I did try IIOP protocol but then I started down an new path of errors where "not bound in this context" was indicated. I tried switching between the 3 Mservers mentioned in the Oracle doc link above. No luck there either.
I see that others have run into this "t3 protocol not supported" message but I don't see any solutions.
Here is my code:
<?php
require_once("java/Java.inc");
$output=exec("C:\Oracle\Middleware\wlserver_12.1\server\bin\setWLSenv.cmd");
$protocol = "t3";
$hostname = "localhost";
$port="7001";
$jndiroot="/jndi/";
$mserver="weblogic.management.mbeanservers.domainruntime";
$united = $jndiroot.$mserver;
$serviceURL = new java("javax.management.remote.JMXServiceURL",$protocol, $hostname, $port,$united);
$h = new java("java.util.Hashtable");
$h->put("javax.naming.Context.SECURITY_PRINCIPAL", "myUID");
$h->put("javax.naming.Context.SECURITY_CREDENTIALS", "myPWD");
$h->put("javax.management.remote.JMXConnectorFactory.PROTOCOL_PROVIDER_PACKAGES",
"weblogic.management.remote");
$connector = new java("javax.management.remote.JMXConnectorFactory");
$connector->connect($serviceURL,$h);
?>
This is how I FINALLY connected to my Weblogic:
1) I installed Tomcat 7.0 zip file instead of the installer and set it up manually. In my
setenv.bat file it looks like:
set CATALINA_HOME=C:\Program Files (x86)\Apache Software Foundation\apache-tomcat-7.0.34
set CATALINA_BASE=C:\Program Files (x86)\Apache Software Foundation\apache-tomcat-7.0.34
set JAVA_HOME=C:\Program Files (x86)\Java\jdk1.7.0_09
set JRE_HOME=C:\Program Files (x86)\Java\jdk1.7.0_09\jre
set JAVA_OPTS=-Djava.ext.dirs=C:\Oracle\Middleware\wlserver_12.1\server\lib
set CLASSPATH=.;C:\Oracle\Middleware\wlserver_12.1\server\lib\wljmxclient.jar
exit /b 0
2) I enabled IIOP protocol in weblogic through the weblogic console and added a userid/password which I used in num 3 for the userID/password
3) My php script on Tomcat 7 with the PHP/Java Bridge (not the JMXServerURL . . . nowhere did I find that you had to use "rmi" as the protocol")
$serviceURL = new java("javax.management.remote.JMXServiceURL","rmi","localhost","7001","/jndi/iiop://localhost:7001/weblogic.management.mbeanservers.edit");
//"service:jmx:iiop://localhost:7001/jndi/weblogic.management.mbeanservers.edit");
$env = new java("java.util.Hashtable");
$env->put("javax.management.remote.JMXConnectorFactory.PROTOCOL_PROVIDER_PACKAGES","weblogic.management.remote");
$env->put("javax.naming.Context.SECURITY_PRINCIPAL", "myuserID");
$env->put("javax.naming.Context.SECURITY_CREDENTIALS", "mypassword");
$jmxCon = java("javax.management.remote.JMXConnectorFactory")->newJMXConnector($serviceURL, $env);
$jmxCon->connect();
$connection = $jmxCon->getMBeanServerConnection();
$mbeans = $connection->queryNames(null,null);
foreach($mbeans as $key=>$value) {
echo $key. "=". $value;
}
It connected and printed out all the mbeans for my admin server and my two managed servers that I created for testing.
How can I write system preferences with Java, using Preferences.systemRoot()?
I tried with:
Preferences preferences = Preferences.systemRoot();
preferences.put("/myapplication/databasepath", pathToDatabase);
But I got this error message:
2010-maj-29 19:02:50 java.util.prefs.WindowsPreferences openKey
VARNING: Could not open windows registry node Software\JavaSoft\Prefs at root 0x80000002. Windows RegOpenKey(...) returned error code 5.
Exception in thread "AWT-EventQueue-0" java.lang.SecurityException: Could not open windows registry node Software\JavaSoft\Prefs at root 0x80000002: Access denied
at java.util.prefs.WindowsPreferences.openKey(Unknown Source)
at java.util.prefs.WindowsPreferences.openKey(Unknown Source)
at java.util.prefs.WindowsPreferences.openKey(Unknown Source)
at java.util.prefs.WindowsPreferences.putSpi(Unknown Source)
at java.util.prefs.AbstractPreferences.put(Unknown Source)
at org.example.install.Setup$2.actionPerformed(Setup.java:43)
I would like to do this, because I want to install an embedded JavaDB database, and let multiple users on the computer to use the same database with the application.
How to solve this? Can I invoke UAC and do this as Administrator from Java? And if I log in as Administrator when writing, can I read the values with my Java application if I'm logged in as a User?
You cannot write to any arbitrary registry location from java preferences - all preferences are stored under a subkey Software\Javasoft\Prefs. With user preferences mapping to the HKEY_CURRENT_USER hive, and system mapping to the HKEY_LOCAL_MACHINE hive.
To write to the registry, you could use the windows "REG" command line tool. This page details other ways of modifying the registry. including use of .reg files.
I had the same need - to write to the registry from java - I solved it by writing a small .NET command line utility to do it.
The Sun Windows JDK does ship with generic code to write to arbitrary portions of the registry (WindowsPreferences), but it's not public. This article describes how to access this class using reflection.
You can't edit Preferences.systemRoot() if User Account Control is turned on. Seems like Microsoft went and broke it. There is a workaround here, but it's not straightforward.
So I had this same issue, so I opened an issue with Oracle:
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=7043176
I was able to work around it myself, by writing a custom implementation of AbstractPreferences and a corresponding PreferencesFactory. What I did was on Windows have the system preferences write to the application data directory defined in the registry by:
HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Explorer\Shell Folders\Common AppData
I used Runtime.getRuntime().exec("reg query \""+key+ "\" /v \""+value+"\"") to get that (works even with UAC turned on).
That evaluates to "C:\ProgramData" on Windows 7 and "C:\Documents and Settings\All Users\Application Data" on XP. I added a subdirectory called "JavaPreferences" and wrote an implementation that uses a properties file as the backend.
As a side note, I had a similar issue with system preferences on Linux because the installer for the JRE was not run by root so I didn't have access to "/etc/.java". A ended up picking another custom directory and granting permissions for that.