I am new to Hadoop, Mapr and Pivotal. I have written java code to write into pivotal but facing issue while writing into Mapr.
public class HadoopFileSystemManager {
private String url;
public void writeFile(String filePath,String data) throws IOException, URISyntaxException {
Path fPath = new Path(filePath);
String url = url = "hdfs://"+ip+":"+"8020";
FileSystem fs = FileSystem.get(new URI(url),new Configuration());
System.out.println(fs.getWorkingDirectory());
FSDataOutputStream writeStream = fs.create(fPath);
writeStream.writeChars(data);
writeStream.close();
}
}
This code works fine with pivoatal but fails with Mapr.
For Mapr i am using port = 7222.
I am getting the following error
"An existing connection was forcibly closed by the remote host"
Please let me know if am using the right port or anything needs to be changed in the code specific to Mapr.
I have stopped the iptables.
Any info is much appreciated.
Thanks
Heading
Try this code. But make sure you have MapR client setup in the node from where you are executing the test.
public class HadoopFileSystemManager {
private String url;
public void writeFile(String filePath,String data) throws IOException, URISyntaxException {
System.setProperty( "java.library.path", "/opt/mapr/lib" );
Path fPath = new Path(filePath);
String url = url = "hdfs://"+ip+":"+"8020";
FileSystem fs = FileSystem.get(new URI(url),new Configuration());
System.out.println(fs.getWorkingDirectory());
FSDataOutputStream writeStream = fs.create(fPath);
writeStream.writeChars(data);
writeStream.close();
}
}
Add the following to the classpath:
/opt/mapr/hadoop/hadoop-0.20.2/conf:/opt/mapr/hadoop/hadoop-0.20.2/lib/hadoop-0.20.2-dev-core.jar:/opt/mapr/hadoop/hadoop-0.20.2/lib/maprfs-0.1.jar:.:/opt/mapr/hadoop/hadoop-0.20.2/lib/commons-logging-1.0.4.jar:/opt/mapr/hadoop/hadoop-0.20.2/lib/zookeeper-3.3.2.jar
This statement in the above code: System.setProperty( "java.library.path", "/opt/mapr/lib" ); can be removed and can be supplied using -Djava.library.path too, if you are running your program from terminal when building.
/opt/mapr may not be your path to mapr files. If that's the case replace the path accordingly wherever applicable.
After comment:
If you are using Maven to build your project, try using the following in the pom.xml,
and with scope provided. MapR is compatible with the normal Apache Hadoop distribution too. So, while building you can use the same. Then when you run your program, you would supply the mapR jars in the classpath.
<dependency>
<groupid>hadoop</groupid>
<artifactid>hadoop</artifactid>
<version>0.20.2</version>
<scope>provided</scope>
</dependency>
Related
I am using apache commons VFS to connect to sftp server and write the content of files in /input directory into a single file in /output directory.The names of files in input directory is provided as a List. I am struggling to write Junit test case for it.My intention is that once the code gets executed, I will compare the contents of file in /input against content of file in /output
public void exportFile(List<String> fileNamesList){
for (String file : fileNamesList){
try(FileObject fileObject= //getsFileObject
OutputStream fileOutputStream= fileObject.resolveFile("/output/"+"exportfile.txt").getContent().getOutputStream(true);
)
fileObject.resolveFile("/input/"+file).getContent().getInputStream().transferTo(fileOutputStream);
}
}
I want to write Junit test case for the above. The below is my setup for test case
#BeforeAll
static void setUpSftpServer() throws IOException {
System.out.println("inside setup ssh");
sshd= SshServer.setUpDefaultServer();
sshd.setPort(1234);
sshd.setKeyPairProvider(new SimpleGeneratorHostKeyProvider());
sshd.setPublickeyAuthenticator(AcceptAllPublickeyAuthenticator.INSTANCE);
sshd.setSubsystemFactories(Arrays.asList(new SftpSubsystemFactory()));
sshd.start();
}
#Test
void exportFileTest() throws IOException, URISyntaxException {
System.out.println("Inside exportFile test");
FileObject fileObject=getFileObject();
when(sftpConfiguration.connect()).thenReturn(fileObject);
myobject.exportFile(Arrays.asList("a.txt"));
String actualContent=fileObject.resolveFile("/input/a.txt").getContentContent().getString("UTF-8");
String expectedContent=fileObject.resolveFile("/output/exportFile.txt").getContentContent().getString("UTF-8");
assertTrue(actualContent.equals(expectedContent));
}
static FileObject getFileObject() throws URISyntaxException, FileSystemException {
String userInfo = "uname" + ":" + "pwd";
SftpFileSystemConfigBuilder sftpConfigBuilder = SftpFileSystemConfigBuilder.getInstance();
FileSystemOptions options = new FileSystemOptions();
IdentityProvider identityInfo = new IdentityInfo(new File("/fake/path/to/key"), "test".getBytes());
sftpConfigBuilder.setIdentityProvider(options, identityInfo);
URI uri= new URI("sftp", userInfo, "127.0.0.1", Objects.requireNonNullElse(1234, -1), null, null, null);
FileObject fileObject= VFS.getManager().resolveFile(uri.toString(),options);
System.out.println("creating file object complete");
fileObject.resolveFile("/input").createFolder(); //create a folder in the path
fileObject.resolveFile("/output").createFolder();
//code to create a file called a.txt inside /input and write the string "abc" to the file
return fileObject;
}
But I am getting an exception like below
org.apache.commons.vfs2.FileSystemException: Unknown message with code "Could not get the user id of the current user (error code: -1)".
This exception I am getting at the line
FileObject fileObject= VFS.getManager().resolveFile(uri.toString(),options);
How do I write the unittest for this case correctly?
This is caused by the SftpFileSystem failing to run the command id -u which doesn't exist on some SSH connections such as Windows OpenSSH. It runs this command when attempting to detect the exec channel. Resolve this by adding the following SFTP configuration:
sftpConfigBuilder.setDisableDetectExecChannel(options, true);
See here.
I use geoip2 to determine the country by ip. During development and testing of the code, I have no problems, but when I run the compiled archive, I encounter a java.io.FileNotFoundException exception. I understand that this is because the path to the file is absolute, and in the archive it changes. Question: How do I need to change my code so that even from the archive I can access the file?
public static String getCountryByIp(String ip) throws Exception {
File database = new File(URLDecoder.decode(GeoUtils.class.getResource("/GeoLite2-Country.mmdb").getFile(),"UTF-8"));
DatabaseReader dbReader = new DatabaseReader.Builder(database).build();
InetAddress ipAddress = InetAddress.getByName(ip);
CountryResponse response = dbReader.country(ipAddress);
return response.getCountry().getName();
}
test.war/
test.war/WEB-INF/classes
You can try this
InputStream is = this.getClass().getClassLoader().getResourceAsStream("GeoLite2-Country.mmdb");
I tried to create a quick framework. in that I created below-mentioned classes:
Config file(All browsers path)
configDataProvider java class(reads the above file)
BrowserFactory class(has firefox browser object)
configDataProviderTest class(access data from dconfigDataProvider class)
now its not reading the paths mentioned in config.properties file.
I have provided all correct path and attached screenshots:
Looks like a problem is at your ConfigDataProvider class.
Firstly, you using Maven for building your project. Maven has defined project structure for code sources and for resources:
/src/main/java
/src/main/resorces
Thus, much better to put your .properties file there.
Second, you don't need to set the full path to your config file.
Relative path will be just enough. Something like below:
public class PropertiesFileHandler {
private static Logger log = Logger.getLogger(PropertiesFileHandler.class);
public static final String CONFIG_PROPERTIES = "src/main/resources/config.properties";
public static final String KEY = "browser.type";
public static BrowserType readBrowserType() {
BrowserType browserType = null;
Properties properties = new Properties();
try (InputStream inputStream = new BufferedInputStream(new FileInputStream(CONFIG_PROPERTIES))) {
properties.load(inputStream);
browserType = Enum.valueOf(BrowserType.class, properties.getProperty(KEY));
} catch (FileNotFoundException e) {
log.error("Properties file wasn't found - " + e);
} catch (IOException e) {
log.error("Problem with reading properties file - " + e);
}
return browserType;
}
}
Lastly, if you are building framework you don't need to put everything under src/main/test. This path specifies tests with future possibilities to be executed with maven default lifecycle - mvn test.
The core of your framework can look like:
Two things which I noticed:
Don't give path in your properties path within ""
all the path seperators should be replaced with double backward slash \\ or single forward slash /
I was using jboss server to deploy my applications and it works fine. My jboss server is corrupted, and I'm changing to tomcat 7.0.55 and its giving me error on my console
"java.io.FileNotFoundException: resourceedge-config.xml (The system cannot find the file specified)"
The resourceedge-config.xml is readingfrom this method:
public static Properties loadSystemConfiguration() throws FileNotFoundException, IOException{
return loadSystemConfiguration(ConfigReader.getFile_Prefix() + "-config.xml");
}
And it's also calling from my application filter class too which is:
//get the application context
ServletContext context = filterConfig.getServletContext();
String configFilePrefix = context.getInitParameter(ApplicationFilter.APPLICATION_CONFIG_FILE_PREFIX);
if(configFilePrefix != null){
ConfigReader.setFile_Prefix(configFilePrefix);
}
try{
configuration = ConfigReader.loadSystemConfiguration();
}catch(Exception e){
e.printStackTrace();
configuration = null;
}
The resourceedge-config.xml is place inside C:\apache-tomcat-7.0.55\bin
Please I need help on this to enable to read the file resourceedge-config.xml.
Thanks
My PC in running under Windows7 64 bit, and I have an utility (.exe, very legacy [~WinXP age], no sources available) that I'd like to invoke from java code deployed into Jetty.
If I launch utility from console I get no errors. If I launch utility via simple java wrapper:
import java.util.*;
import java.io.*;
public class Wrapper {
public static void main(String[] args) throws IOException {
System.out.println(System.getProperty("java.version"));
Runtime.getRuntime().exec("util.exe -opt1 -opt2");
}
}
I get no errors as well.
But in case I invoke this code (a bit more complex than Runtime.getRuntime().exec("util.exe") because I need to calculate absolute path to the binary file) from inside WAR I get IOException with following message:
CreateProcess error=216, This version of %1 is not compatible with the version of Windows you're running. Check your computer's system information to see whether you need a x86 (32-bit) or x64 (64-bit) version of the program, and then contact the software publisher
I've tried to launch jetty with -d32 option, tried to put different versions of java (both JRE/JDK, both 6/7, both 32/64 bit) into JAVA_HOME and PATH — and haven't succeed.
Did anyone face similar issues? Is it possible to resolve them?
[UPDATED]
I've attached some server-code.
CommandLine & FileUtils are part of Apache Commons. ApplicationContext relates to Spring Framework.
public class ImageLoader implements ApplicationContextAware {
private final static String UTIL_EXECUTABLE = "util.exe";
private final static String TEMP_FILE_PREFIX = "tmpFilePrefix";
private ApplicationContext applicationContext;
private File binaryPath;
#PostConstruct
public void init() throws Exception {
if (applicationContext instanceof WebApplicationContext) {
Resource binaryRoot = applicationContext.getResource(
"WEB-INF/classes/executable");
this.binaryPath = binaryRoot.getFile();
}
}
#Override
public void setApplicationContext(ApplicationContext applicationContext) throws BeansException {
this.applicationContext = applicationContext;
}
/**
* Download db image from device.
*/
public byte[] downloadImage(Device device) throws LoaderException {
try {
File file = downloadWindows(device);
return FileUtils.readFileToByteArray(file);
} catch (Exception ex) {
throw new LoaderException("Error downloading file: " + ex.getMessage(), ex);
}
}
private File downloadWindows(final Device device) throws Exception {
File tmpFile = File.createTempFile(TEMP_FILE_PREFIX, null);
CommandLine command = generateCommand(ActionType.download, tmpFile, device.getTargetIP(), "user", "pass");
Runtime.getRuntime().exec(command.toString());
return tmpFile;
}
protected CommandLine generateCommand(ActionType actionType, File file, String targetIP, String userName, String userPassword) throws IOException {
String bin = this.binaryPath.getPath() + "\\" + UTIL_EXECUTABLE;
// safe to use \\ because code only runs if WAR deployed under Windows
CommandLine commandLine = new CommandLine(bin.replace("\\", "\\\\"));
commandLine.addArgument(actionType.name());
commandLine.addArgument(file.getAbsolutePath().replace("\\", "\\\\"));
commandLine.addArgument(targetIP);
commandLine.addArgument(userName);
commandLine.addArgument(userPassword);
return commandLine;
}
}
enum ActionType {
download,
upload
}
Shame on me and my inattention. The problem actually was "elsewhere". The WAR was built by maven and the executable was processed like any other resource, so checksum was different in compare to original file. I excluded exe-files from filteing (in pom.xml) and it began to work correct.