I have a java program to run export command. While executing the export, it tries to look for the export directory in my local machine instead of Hdfs. The same program works fine for import. I have checked the file exists on hdfs.
Please help.
Below is the code I am executing:
public void executeSqoopLoad() throws UnsupportedEncodingException{
SqoopOptions options = new SqoopOptions();
String driver = "oracle.jdbc.driver.OracleDriver";
//options.setDriverClassName(driver);
options.setUsername(“user");
options.setPassword(“pass");
options.setConnectString("jdbc:oracle:thin:#host:1522:rptdev");
Configuration configuration = new Configuration(false);
Resource configResource;
try {
configResource = FileUtils.getFileResource("/Users/Moiz/git/jef/hadoop/hdfs-site.xml");
configuration.addResource(configResource.getInputStream());
configResource = FileUtils.getFileResource("/Users/Moiz/git/jef/hadoop/core-site.xml");
configuration.addResource(configResource.getInputStream());
FileSystem dfs = FileSystem.get(configuration);
String[] uriSplit = dfs.getUri().toString().split(":");
String newUri = uriSplit[0]+":"+uriSplit[1];
dfs.setWorkingDirectory(new Path(newUri+"/tmp"));
System.out.println(dfs.getWorkingDirectory());
System.out.println("Exists = " + dfs.exists(dfs.getWorkingDirectory()));
DateTime dt = new DateTime();
options.setCodeOutputDir("/tmp");
options.setClassName("SqoopLoad_"+null+dt.getYear()+dt.getMonthOfYear()+dt.getDayOfMonth()+dt.getMillisOfDay());
options.setVerbose(true);
// HDFS options
options.setExportDir(dfs.getWorkingDirectory()+"/TestDirectory");
System.out.println("Exists = " + dfs.exists(new Path(options.getExportDir())));
options.setInputFieldsTerminatedBy('\u0005');
options.setTableName(“SCHEMA.TEST_SQP");
options.setNumMappers(1);
options.setDirectMode(true);
System.setProperty(Sqoop.SQOOP_RETHROW_PROPERTY, "rethrow");
} catch (FileNotFoundException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
int ret = 100;
try{
ret = new ExportTool().run(options);
}catch (Exception e) {
System.out.println("Debug");
e.printStackTrace();
}
System.out.println("return code "+ ret);
}
java.lang.RuntimeException: java.io.FileNotFoundException: File
/tmp/TestDirectory/part-m-00000 does not exist
I got it to work. Had to re-import the sqoop jars through maven.
Related
I have this plugin that when installed, extracts some executables at some temporary location, and uses them. Here is my code:
public class StartCheck implements IStartup {
private BufferedReader buf=null;
public static String pathBandwidth;
public static String pathDeviceQuery;
public static String pathKernelLaunchOverhead;
public static String memoryLatency;
public void earlyStartup() {
// This method checks for presence of nvcc when Eclipse starts-up.
String command="nvcc --version";
Runtime run = Runtime.getRuntime();
Process pr;
try {
pr = run.exec(command);
pr.waitFor();
buf = new BufferedReader(new InputStreamReader(pr.getInputStream()));
//print-out the nvcc version
System.out.println(buf.readLine());
Preparation.return_val=true;
//extract the executables
Bundle bundle = Platform.getBundle("PTXAnalysis");
URL url_bandwidth = FileLocator.find(bundle, new Path("/Executables/bandWidth.out"), null);
URL url_deviceQuery = FileLocator.find(bundle, new Path("/Executables/deviceQuery.out"), null);
URL url_kernelLaunchOverhead = FileLocator.find(bundle, new Path("/Executables/empty"), null);
URL url_memoryLatency = FileLocator.find(bundle, new Path("/Executables/memLatency.out"), null);
try {
url_bandwidth = FileLocator.toFileURL(url_bandwidth);
url_deviceQuery = FileLocator.toFileURL(url_deviceQuery);
url_kernelLaunchOverhead = FileLocator.toFileURL(url_kernelLaunchOverhead);
url_memoryLatency = FileLocator.toFileURL(url_memoryLatency);
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
pathBandwidth=url_bandwidth.toString();
pathDeviceQuery=url_deviceQuery.toString();
pathKernelLaunchOverhead=url_kernelLaunchOverhead.toString();
memoryLatency=url_memoryLatency.toString();
}catch (IOException e) {
//disable all commands since no further task can be done, prompt user to install nvcc.
System.out.println("nvcc was not found on this computer. You won't be able to use the energy estimation plug-in");
EnergyEstimator.return_val=false;
Preparation.return_val=false;
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}
When I install the plugin, it gives me this location (one of the many), where the executable has been extracted:
/home/limafoxtrottango/.eclipse/org.eclipse.platform_4.4.1_2069420271_linux_gtx_x86_64/configuration/org.eclipse.osgi/460/0/.cp/Executables/bandWidth.out
Now, the problems: I can't find any such directory. I understand that it is a temporary directory, but it does not show-up even if Eclipse is running. I am using one of these paths to run an executable using ProcessBuilder. Here is the code:
public static void runExecutable(){
initializeArray();
path_result="/home/"+System.getProperty("user.name")+"/kernelLaunchOverhead.txt";
String path_executable=StartCheck.pathKernelLaunchOverhead.substring(path_result.indexOf('/'),path_result.lastIndexOf('/')+1); //path to the directory in which the executable is extracted
try {
fw = new FileWriter(path_result);
for(int i=0;i<arr.length;i++){
ProcessBuilder builder=new ProcessBuilder("./empty",Integer.toString(arr[i]));
builder.directory(new File(path_executable));
int av=0;
float sum=0;
while(av<10){
Process pr=builder.start();
stdin = pr.getInputStream();
isr = new InputStreamReader(stdin);
br = new BufferedReader(isr);
sum=sum+Float.parseFloat(line=br.readLine());
av++;
}
fw.write(arr[i]+" "+Float.toString(sum/10));
fw.write("\n");
}
fw.close();
} catch (IOException e1) {
// TODO Auto-generated catch block
e1.printStackTrace();
}
FillArrays(path_result);
BestLineFit();
saveModel("/home/"+System.getProperty("user.name")+"/KernelLaunchOverheadModel.txt");
}
On invoking this function, nothing happens. It does not even throw any FileNotFound exceptions. Normally, it should have found the executable in the directory, and run it. But after installing the plugin, nothing happens.
To re-iterate, the class StartCheck is successfully showing to me the path to which the executables have been extracted. But those paths do not exist anywhere on my system.
Directories starting with . (such as the .eclipse in the path you show) are hidden on Linux and macOS systems.
You can see them from the command line using ls -a
I am trying to read some property from the properties file, but my code is not reading the property file. The properties file is in some folder in my machine.
Here is my code:
public String getproperty(){
String extension="";
Properties prop = new Properties ();
String propname = "\\"+Any location in your machine+"\\fileExtension.properties";
Logger.debug("ReadFiles", " ----Property file path---- "+ propname, null);
ip = getClass().getClassLoader().getResourceAsStream(propname);
Logger.debug("ReadFiles", " ----ip value ---- "+ip, null);
try {
if(ip != null){
prop.load(ip);
Logger.debug("ReadFiles", " ----Property file loaded---- ", null);
}
extension = prop.getProperty("fileExt");
Logger.debug("ReadFiles", " ----Property = " + extension, null);
} catch (IOException e) {
Logger.error("ReadFiles", " ----Error while loading property file---- ", null);
e.printStackTrace();
}
finally{
try {
ip.close();
} catch (IOException e) {
e.printStackTrace();
}
}
return extension;
}
I am creating a jar placing it in lib folder of server (installed in my machine) and placing the properties file in my machine and given the same path in code. I have tried with absolute path and without absolute path.
Please try this example that uses an absolute path to the properties file.
package com.company;
import java.io.*;
import java.util.Properties;
public class Main {
public static void main(String[] args) {
Properties prop = new Properties();
InputStream input = null;
try {
input = new FileInputStream("/home/dac/gs-rest-service/javacode/src/main/java/com/company/config.properties");
prop.load(input);
String extension = prop.getProperty("fileExt");
System.out.println("ReadFiles ----Property = " + extension);
} catch (IOException ex) {
ex.printStackTrace();
} finally {
if (input != null) {
try {
input.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
}
}
Test
cat /home/dac/gs-rest-service/javacode/src/main/java/com/company/config.properties
#Mon Dec 26 17:31:30 CET 2016
dbpassword=password
database=localhost
dbuser=foobar
fileExt=.xml⏎
Run the program
ReadFiles ----Property = .xml
I have a problem with an exported jar file. When I run my project in Eclipse it runs fine, but when I run it as an exported jar from the console I receive the following error message:
java.io.FileNotFoundException: firstLaunch.properties (System can't find file)
or
java.io.FileNotFoundException: resources/config/firstLaunch.properties (System can't find file)
I tried to put it in to the resource folder and change syntax from firstLaunch.properties to /resource/config/firstLaunch.properties, but again it says the same thing but with a different path. I don't know why is this doing this.
Here is the code:
public void saveConfigFile(String file, String key, String value) {
Properties prop = new Properties();
OutputStream output = null;
try {
output = new FileOutputStream(file);
// set the properties value
prop.setProperty(key, value);
// save properties to project root folder
prop.store(output, null);
} catch (IOException io) {
io.printStackTrace();
} finally {
if (output != null) {
try {
output.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
}
My syntax for executing the method is
if (properties.loadConfigFile("firstLaunch.properties", "value").equals(properties.loadConfigFile("true.properties", "true"))) {
properties.saveConfigFile("port.properties", "port", "8795");
properties.saveConfigFile("ip.properties", "ip", temp[1]);
properties.saveConfigFile("firstLaunch.properties", "value", "false");
settings.port = properties.loadConfigFile("port.properties", "port");
settings.myIp = properties.loadConfigFile("ip.properties", "ip");
} else {
settings.port = properties.loadConfigFile("port.properties", "port");
settings.myIp = properties.loadConfigFile("ip.properties", "ip");
}
Your problem probably has to do with how you are referencing the file location. Add some detail/code examples as to how you are referencing the code, so then we can be sure to help. Having said that here is another way to reference a properties file:
Put it in the classpath like this:
private static Properties prop = new Properties();
private static String filename = "<name of file>.properties";
InputStream input = <ClassName>.class.getClassLoader().getResourceAsStream(filename);
try {
if (input==null) {
loggerOut.log(Level.SEVERE, "Sorry, unable to find " + filename);
}
prop.load(input);
loggerOut.info("XML In storage path: " prop.getProperty("<property in file>"));
fileNameAndPath = prop.getProperty("fileNameAndPathIN").trim();
logNameAndPath = logPath + logName;
} catch (IOException ex) {
ex.printStackTrace();
} finally {
if (input!=null) {
try {
input.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
I was trying to implement the functionality of transferring local files to a network drive using jcifs library but upon running the program on the command line I was receiving following exception:
Exception in thread "main" java.lang.NullPointerException
jcifs.smb.ServerMessageBlock.writeString(ServerMessageBlock.java: 202)
To understand the error I tried debugging the code on eclipse and while doing so at line:
NtlmPasswordAuthentication authentication = new NtlmPasswordAuthentication("xxxxxx.xx.com",username,password);
I received an exception stating ClassNotFoundException. But I have the jcifs.jar in the build path.
A quick google search for 'ntlmpasswordauthentication' classnotfoundexception landed me on two threads with same issue but no resolution.
Please let me know how can I resolve this.
Thank you
Here is the whole function, just in case needed:
private static void TransferFiles()
{
File transfer_files = new File (sourcepath);
File[] files = transfer_files.listFiles();
String username = properties.getProperty("user");
String password = properties.getProperty("password");
String source = sourcepath;
SmbFileOutputStream outputStream = null;
FileInputStream inputStream = null;
SmbFile copyFile = null;
byte[] buffer = new byte[16*1024*1024];
int length = 0;
jcifs.smb.NtlmPasswordAuthentication authentication = new NtlmPasswordAuthentication("xxxxxxx",username,password);
try
{
copyFile = new SmbFile(destinationpath,authentication);
}
catch (MalformedURLException e)
{
e.printStackTrace();
}
try
{
outputStream = new SmbFileOutputStream(copyFile);
}
catch (SmbException | MalformedURLException | UnknownHostException e)
{
e.printStackTrace();
}
try
{
inputStream = new FileInputStream(sourceFile);
}
catch (FileNotFoundException e1)
{
e1.printStackTrace();
}
try
{
while((length = inputStream.read(buffer))>0)
{
outputStream.write(buffer, 0, length);
}
}
catch (IOException e)
{
e.printStackTrace();
}
try
{
inputStream.close();
outputStream.close();
}
catch (IOException e)
{
e.printStackTrace();
}
I'm facing this error:
`Unable to load properties file for MultiWordNet
Exception in thread "main" java.lang.NumberFormatException: null
at java.lang.Integer.parseInt(Integer.java:417)
at java.lang.Integer.<init>(Integer.java:660)
at org.itc.mwn.MysqlDictionary.<init>(MysqlDictionary.java:85)`
This is the property file that MysqlDictionary.java is trying to read:
#------------------------------------------------------------
#Properties file properties MultiWordNet API
#Hostname of the MySQL server
MWN_HOSTNAME=localhost
#User
MWN_USER=root
#Password
MWN_PASSWD=
#Database name
MWN_DB=wordnet
#Cache of entity
CACHE_CAPACITY=1000
And,finally, this is the part where the code fails:
public MysqlDictionary() {
try {
connectionParameters = new Properties();
connectionParameters.load(new FileInputStream(new File("./conf/multiwordnet.properties")));
} catch (java.io.IOException ioee) {
System.err.println("Unable to load properties file for MultiWordNet");
}
/// connection drivers instance
try {
Class.forName("com.mysql.jdbc.Driver").newInstance();
//Class.forName("org.gjt.mm.mysql.Driver").newInstance();
} catch(ClassNotFoundException E){
System.err.println("Unable to load driver");
} catch(IllegalAccessException E){
System.err.println("Unable to load driver");
} catch(InstantiationException E){
System.err.println("Unable to load driver");
}
// MultiWordnet db connection
String host = connectionParameters.getProperty("MWN_HOSTNAME");
String user = connectionParameters.getProperty("MWN_USER");
String passwd = connectionParameters.getProperty("MWN_PASSWD");
String dbname = connectionParameters.getProperty("MWN_DB");
Integer cache = new Integer(connectionParameters.getProperty("CACHE_CAPACITY"));
//here is where the parsing fails, but the file is properly written!
try {
DEFAULT_CACHE_CAPACITY = cache.intValue();
String conn = "jdbc:mysql://" + host + "/" + dbname;
this.db = DriverManager.getConnection(conn,user,passwd);
this.stmt = db.createStatement();
System.err.println("Welcome to the MultiWordNet API\nConnection database ...OK\n");
} catch (SQLException E) {
System.out.println("Unable to establish multiwordnet Mysql DB connection on " + host + "(" + user + " - " + passwd + ")");
E.printStackTrace(System.out);
}
The strange thing is that the program started suddenly failing, after it was running correctly
When dealing with File in java don't struggle with relative paths. They are quite unpredictable since they depend on the current working directory on which you don't have total control. Use classpath instead, loading the resource using the class loader:
Properties properties = new Properties();
properties.load(getClass().getResourceAsStream("multiwordnet.properties"));
Your problem isn't not being able to parse an integer, but not being able to load your file.
Check out for irregular invisible chars, like <SHIFT>+<SPACE> or <CTRL>+<SPACE> or else, next to CACHE_CAPACITY=1000.
Sounds a bit profanatory, but I've often enough stumbled upon such things.
Check for spaces in your properties file or try as below
String cacheProp = connectionParameters.getProperty("CACHE_CAPACITY")
//For debug
System.out.println("cacheProp="+cacheProp);
Integer cache = new Integer(cacheProp.trim());
This code worked fine for me i kept the properties file in resources folder(src/main/resources)
public static void MysqlDictionary() {
try {
Properties p = new Properties();
p.load(MyTest.class.getResourceAsStream("/test.properties"));
String host = p.getProperty("MWN_HOSTNAME");
String user = p.getProperty("MWN_USER");
String passwd = p.getProperty("MWN_PASSWD");
String dbname = p.getProperty("MWN_DB");
String cache = p.getProperty("CACHE_CAPACITY");
int i = Integer.parseInt(cache);
System.out.println(host+"\t"+user+"\t"+passwd+"\t"+dbname+"\t"+i);
} catch (FileNotFoundException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
}