We Keep Coding

File Not Found - Spark standalone cluster

I have two machines named: ubuntu1 and ubuntu2.
In ubuntu1, I started the master node in Spark Standalone Cluster and ubuntu2 I started with a worker (slave).
I am trying to execute the example workCount available on github.
When I submit the application, the worker send an error message
java.io.FileNotFoundException: File file:/home/ubuntu1/demo/test.txt does not exist.
My command line is
./spark-submit --master spark://ubuntu1-VirtualBox:7077 --deploy-mode cluster --clas br.com.wordCount.App -v --name"Word Count" /home/ubuntu1/demo/wordCount.jar /home/ubuntu1/demo/test.txt
The file test.txt has only to stay in one machine ?
Note: The master and the worker are in different machine.
Thank you

I got the same problem while loading the JSON file. I recognized by default windows storing file format as Textfile regardless of the name. identify the file format then you can load easily.
example: think you saved the file as test.JSON. but by default windows adding .txt to it.
check that and try to run again.
I hope your problem will get resolved with this idea.
Thank you.

You should put your file on hdfs by going to the folder and typing :
hdfs dfs -put <file>
Otherwise each node has to have access to it by having the same path folder existing on each machine.
Don't forget to change file:/ to hdfs:/ after you do that

Wait for non-java-process [winscp, sftp ]to complete file transfer in java

How to notify java/jvm that the file is being used by another process which is non-java process[i.e winSCP, C:D etc]. Consider I am transferring the file from my local machine to linux server to tmp directory. This tmp directory watch by java-watch-service when any entry is created[ when the files come to this direcotry java.nio.file.StandardWatchEventKinds.ENTRY_CREATE] in this directory the java will transfer this file to different server. The problem is java-watch-service pick this file as soon as ENTRY_CREATE is created without waiting for complete transfer.
This is working fine any-case of windows may be this is due to transfer-process and jvm process has same parent.

File upload from a web-based application to a linux server

It seems that my code won't work after learning that the machine that I'll be pointing the upload path to is a Linux box.
My use case is, a user logs in to the web app, chooses a file to upload, then click upload button. Is it possible to do this direct from the Java code to the Linux server using appropriate ssh or scp libraries if there is any?
EDIT: Here's my current code.
#Override
public void fileTransfer(File uploadedFile, String fileName, String pathTemp) {
File destFile = new File( pathTemp + File.separator + fileName);
try{
FileUtils.copyFile(uploadedFile, destFile);
String getTempFile = destFile.toString();
String tempPath = getTempFile.replace("\\", "\\\\");
File tempFile = new File(tempPath); // 1st file
String tempFileName = tempFile.getName();
String fileSave = getUploadPathSave().replace("\\", "\\\\");
tempFile.renameTo(new File(fileSave + tempFileName));
} catch (IOException ex) {
System.out.println("Could not copy file " + fileName);
ex.printStackTrace();
}
}

If your app is deployed at one place only (not mass distribution), the easiest way would be:
create samba share on linux machine
map samba share to logical drive on windows machine
do usual file copy with java functions.
attention: renameTo will not work between drives. You'll need to copy input stream to output stream or, better, use apache commons-io functions for that.

There are different possibilities:
If you can create a shared directory in linux and mount it under windows (see Samba. Then you can write to that directory like a local directory. File will go to the linux server.
Use a library like Jsch to upload the file from windows server to linux server.

There are certain things you can do:
1-> If you can program your linux server, then you can make a program that listens to user request on a port, and stores data in file. Then you can send you files to that port of server.
2-> The other way is you can use some sort of script to create ssh-connection to server and then you can just add file through ssh, but here your java program will not be useful.
I personally use my own program to share files between 2 machines in same network.
You can use it,if it will be useful for you: https://github.com/RishabhRD/xshare

Comparing file size of downloaded file with remote file on ftp server

I am using FTPClient to download files from remote FTP Server.
After downloading, I would like to compare the size of local file and remote file if they are same.
Once the file is downloaded, the size of downloaded file is different from the one in remote server.
Below is the code snippet
FileOutputStream output = new FileOutputStream(localFile.getAbsolutePath());
if( getFTPFileType()!=null ){
// set as binary
ftpClient.setFileType(getFTPFileType(), getFTPFileType());
ftpClient.setFileTransferMode(getFTPFileType());
}
if( getLog().isDebugEnabled()){
getLog().debug("FTP File Type "+getFTPFileType());
}
boolean success = ftpClient.retrieveFile(remoteFile.getName(), output);
If I download in windows environment, it works fine. But if I download in AIX server, there is difference between file sizes. I compared the downloaded files, the content is same.
Please advice.

IF the number of bytes difference in size ~= the number of lines in the file, it is due to the removal of windows carriage control characters: \r or ^M as they appear in vi.
And the content is not the same in the sense that checksums would fail.
Edit:
inside ftp: dir filename will return something like this
-rw-rw-rw- 1 user group 1224 Mar 4 20:22 twrite.c
I have no good idea on how to get your java class to send that command. Check your documentation. When you can get dir working, you can tokenize the string you get back, the fifth field is the number of bytes on the remote side.
Example using unix shell (java will let you run UNIX scripts), after fileA has been dowloaded from remote to local:
local_size=$(ls -l fileA | awk '{print $5}')
/usr/bin/ftp -n <<EOF > ftp.log
open remote_nodename
user username password
cd /directory/to/files
dir fileA
bye
END
remote_size=$(grep 'fileA' ftp.log | awk '{print $5}')
[ $remote_size -eq $local_size ] && echo 'OK' || echo 'NOTOK'
If you can emulate this with java classes okay, otherwise you need to use something like this which shows the use of runtime exec():
http://www.java-samples.com/showtutorial.php?tutorialid=8
Again, to vbe clear ftp does NOT run shell scripts on the remote side. ssh does.

how to get remote linux server file inputstream

I am trying to read a file which is in the remote linux server. But I do not know how to get the inputstream of the file using java.
How can this be done?

Assuming that by "remote linux server" you mean "remote linux shell", you should use an ssh library like JSch. You can find a file download example here.

Maybe SSHJ can help you? https://github.com/shikhar/sshj
Features of the library include:
reading known_hosts files for host key verification
publickey, password and keyboard-interactive authentication
command, subsystem and shell channels
local and remote port forwarding
scp + complete sftp version 0-3 implementation

Assuming you have a working connection to the server and access to the file, you can create a File object with the URI of the file:
File f = new File(uri);
FileInputStream fis = new FileInputStream(f);
The URI should be the URI to the file, for example "file://server/path/to/file".
See also the Javadoc for File(URI) .

It depends on how is the file available. Is it by HTTP, FTP, SFTP or through a server you wrote yourself ?
If your want to get the file through HTTP, you can use this :
HttpURLConnection connec = (HttpURLConnection)new URL("http://host/file").openConnection();
if(connec.getResponseCode() != connec.HTTP_OK)
{
System.err.println("Not OK");
return;
}
System.out.println("length = " + connec.getContentLength());
System.out.println("Type = " + connec.getContentType());
InputStream in = connec.getInputStream();
//Now you can read the file content with in
There is also Jsch library which is very good for SFTP / SCP

You can use any ssh java lib, as was mentioned in other answers, or mount directory with file as NFS share folder. After mounting you can use usual java API to acsess file.
Example