I am running a java application (app1) that runs multiple instances of another java application (app2) in parallel using ProcessBuilder and Fixed Thread Pool with the following code:
futures = new ArrayList<Future<String>>();
executor = Executors.newFixedThreadPool(10);
for (int i=0; i<cnt;i++)
{
ProcessBuilder builder = new ProcessBuilder();
builder.command("java","app2"); //I set the class path in the real code
builder.directory(new File("/Users/Evan/Documents/workspace/MyProj/ins_"+i));
builder.inheritIO();
ProcessRunner pr=new ProcessRunner (builder);
futures.add(executor.submit(pr));
}
Each run of app2 has different working directory (the value of i in the folder ins_i is different for each run). The code works, but now in app2, I want to read a file from the working directory of the current process (i.e. /Users/Evan/Documents/workspace/MyProj/ins_"+i). So in app2 code I need a statement that let me know the directory of the current process builder (basically the value of i in "/Users/Evan/Documents/workspace/MyProj/ins_"+i).
I tried this statement:
System.getProperty("user.dir")
and it did not work because it retrieved "/Users/Evan/Documents/workspace/MyProj"
Thank you and I appreciate any help
but now in app2, I want to read a file from the working directory of the current process
If you know the filename you want to read from within the working directory of app2, you can just create a File object, passing the file name of the file you wish to read. The parent folder defaults to the current working directory (which is defined by the ProcessBuilder)
File myFile = new File("FileToRead");
//read from myFile using FileInputStream
Related
I am trying to update a JSON file on my server resources folder in response to a form submission to my api endpoint.
I want to update my QueryMap.json from the controller QueryMapController.java
#CrossOrigin
#RequestMapping("/newQuery")
public String newQuery(#RequestParam("qid") String qid, #RequestParam("query") String query) throws Exception{
String fileName="QueryMap.json";
File file = new ClassPathResource(fileName).getFile();
JSONParser parser = new JSONParser();
FileReader fr = new FileReader(file);
Object ob = parser.parse(fr);
fr.close();
JSONArray arr = (JSONArray)ob;
JSONObject newob = new JSONObject();
newob.put("Query_id",qid);
newob.put("Query",query);
arr.add(newob);
FileWriter fw = new FileWriter(file.getAbsoluteFile(),false);
BufferedWriter out = new BufferedWriter(fw);
out.write(arr.toJSONString());
out.flush();
out.close();
fw.close();
return arr.toJSONString();
}
The above code is the Controller Method I want to use to update my .json file.
Apparently Everything seems to work. It works without errors and the program executes as if the file is updated but the actual file is not changed. As I restart the server the changes are gone. It seems the file is cached somewhere internally during runtime. How should I handle this.
Pardon my Custom Data Connection and writing code like this which doesn't strictly follow MVC guidelines. It has reasons behind to code this way.
tl;dr Place files you want to modify outside of your classpath because they are overwritten with the ones in your <projectDir>/src/main/resources directory when you build/package your project. If you want to see code for this check the end of the answer.
The reason this happens is most likely because when you restart your server, your project is re-build. During this process every class is re-compiled and the compiled classes are placed in a separate directory, along with all the resources (<projectDir>/target/classes/ when using maven). This replaces your changed version of the json file with the original one from your <projectDir>/src/main/resources directory.
You can verify this by debugging your newQuery Method and following these steps:
Check the location of your json file with File.getAbsolutePath(). For me it is <projectDir>/target/classes/QueryMap.json.
Check the content of the file, it should be in its original state.
Let your code do the changes, then re-check the file content. The changes should be there.
Stop the server and re-build your project (With maven mvn clean package).
The file content should be in its original state again, because the build process replaced <projectDir>/target/classes/QueryMap.json with <projectDir>/src/main/resources/QueryMap.json.
One solution to this would be to use the json file on your class path as a default and copy it to the current working directory where you can change it any way you want:
File editableQueryMap = new File("./QueryMap.json");
if (!editableQueryMap.exists()) {
File defaultQueryMap = new ClassPathResource("QueryMap.json").getFile();
try (OutputStream os = Files.newOutputStream(editableQueryMap.toPath())) {
Files.copy(defaultQueryMap.toPath(), os);
}
}
//do your changes with editableQueryMap here
For me, this puts the editable QueryMap.json file in the root of my project directory. If you use a jar file, it should place the file in the same directory as the jar (unless the working directory is changed).
I want just to print the content of streams to console. I wrote the following code but it does not print anything. Anyone can help me to read text file as stream in Spark?? Is there a problem related to Windows system?
public static void main(String[] args) throws Exception {
SparkConf sparkConf = new SparkConf().setAppName("My app")
.setMaster("local[2]")
.setSparkHome("C:\\Spark\\spark-1.5.1-bin-hadoop2.6")
.set("spark.executor.memory", "2g");
JavaStreamingContext jssc = new JavaStreamingContext(sparkConf, Durations.seconds(2));
JavaDStream<String> dataStream = jssc.textFileStream("C://testStream//copy.csv");
dataStream.print();
jssc.start();
jssc.awaitTermination();
}
UPDATE: The content of copy.csv is
0,0,12,5,0
0,0,12,5,0
0,1,2,0,42
0,0,0,0,264
0,0,12,5,0
textFileStream is for Monitoring the hadoop Compatible Directories. This operation will watch the provided directory and as you add new files in the provided directory it will read/ stream the data from the newly added files.
You cannot read text/ csv files using textFileStream or rather I would say that you do not need streaming in case you are just reading the files.
My Suggestion would be to monitor some directory (may be HDFS or local file system) and then add files and capture the content of these new files using textFileStream.
May be in your code may be you can replace "C://testStream//copy.csv" with C://testStream" and once your Spark Streaming job is up and running then add file copy.csv to C://testStream folder and see the output on Spark Console.
OR
may be you can write another command line Scala/ Java program which read the files and throw the content over the Socket (at a certain PORT#) and next you can leverage socketTextStream for capturing and reading the data. Once you have read the data, you further apply other transformation or output operations.
You can also think of leveraging Flume too
Refer to API Documentation for more details
This worked for me on Windows 7 and Spark 1.6.3: (removing the rest of code, important one is how to define the folder to monitor)
val ssc = ...
val lines = ssc.textFileStream("file:///D:/tmp/data")
...
print
...
This monitors directory D:/tmp/data, ssc is my streaming context
Steps:
Create a file say 1.txt in D:/tmp/data
Enter some text
Start the spart application
Rename the file to data.txt (i believe any arbitrary name will do as long as it's changed while directory is monitored by spark)
One other thing I noticed is that I had to change the line separator to Unix style (used Notepad++) otherwise file wasn't getting picked up.
Try below code, it works:
JavaDStream<String> dataStream = jssc.textFileStream("file:///C:/testStream/");
I'm trying to copy a file from local to hdfs in these three ways:
FileSystem fs = FileSystem.get(context.getConfiguration());
LocalFileSystem lfs = fs.getLocal(context.getConfiguration());
lfs.copyFromLocalFile(new Path("file:///pathToFile/file.properties"), new Path("/destPath/"));
fs.copyFromLocalFile(new Path("file:///pathToFile/file.properties"), new Path("/destPath/"));
fs.copyFromLocalFile(new Path("file:///pathToFile/file.properties"), new Path("/destPath/"));
But none of them are working.
I always get a FileNotFound exception for /pathToFile/file.properties, but the file exists on that path on Unix and has read and write permissions for the user that runs the Map/Reduce.
Any ideas what I'm missing here?
Job is running with Ozzie
CDH4
Thank you very much for your help.
opalo
Where is this code running?
If this code is running in a map or reduce method (as it seems because you have a Context instance), then you are executing on one of your slave nodes. Can all of your slave nodes see this path or can only the cluster's login node see the file?
If this code is in fact supposed to be running in a mapper or reducer, and the file is not local to these machines (and you do not want to put the file(s) into hdfs with a "hdfs fs -put" command), one option you have is to deploy the file(s) with your job using the hadoop distributed cache. You can do this programmatically using the DistributedCache class's static method addCacheFile, or through the command line if your main class implements the Tool interface by using the -files switch.
Programmatically (Copied from the documentation linked to above):
JobConf job = new JobConf();
DistributedCache.addCacheFile(new URI("/myapp/lookup.dat#lookup.dat"), ob);
DistributedCache.addCacheArchive(new URI("/myapp/map.zip", job);
DistributedCache.addFileToClassPath(new Path("/myapp/mylib.jar"), job);
DistributedCache.addCacheArchive(new URI("/myapp/mytar.tar", job);
DistributedCache.addCacheArchive(new URI("/myapp/mytgz.tgz", job);
DistributedCache.addCacheArchive(new URI("/myapp/mytargz.tar.gz", job);
From the command line if your main class implements Tool interface:
hadoop jar Your.jar Package.Path.To.MainClass -files comma,seperated,list,of,files program_argument_list_here
I am connecting XML with odbc:jdbc from java and updating A.xml file.
After the updating is done, I am copying A.xml to different location and renaming that file(lets say copy.xml).
Everything works fine.
But I noticed that if the original A.xml file has been opened during the time of update, the copy.xml file is empty.
Then I decided to write a macro in the A.xml file to auto save the worksheet if any column value changes. This macro works, but
the copy.xml file is still empty.
I am using
List<String> command = new ArrayList<String>();
command.add("cmd.exe");
command.add("/c");
command.add("copy A.xml copy.xml ");
ProcessBuilder processBuilder = new ProcessBuilder(command);
Process process = processBuilder.start();
to copy file.
Is there a different way to accomplish this.
Here is how you can copy a file using Java IO API:
import static java.nio.file.StandardCopyOption.*;
...
Files.copy(source, target, REPLACE_EXISTING);
Full documentation here
Also, you have to make sure the file is not being updated when you copy it over to the new place.
I have a java application from which i am calling a shell script. Can any one tell where to keep the script file in my application and what is the path to access the file in whole application.
i m keeping my script in the java package but when i m trying to access using path like com.abc.script.sh by running my java application through unix i ma getting error
java.io.IOException: error=2, No such file or directory
i am calling the script file with some argument with the following code
private static final String command = "com.abc.script.sh -db abc -scm TEST_xyz -bcp com.abc.out.txt -log /var/tmp -tab abc_$TABLENAME";
Process process = Runtime.getRuntime().exec(command);
and i am running the application from unix.
i need to pass the parameter to shell script file as well . the parameters are like hostname , table name...
where to keep the script file in my application
Your wrote, I can interprete this like:
Storing the content of the file into the memory
Storing the file into your .jar file
I think you mean the second.
You can place it in your jar file in every folder you want. I prefer a subfolder (not the root)
First put the file in your jar. Then you have to extract it to a temporary file (See the link if you want to know how to make a tempfile).
Then create an inputstream from the file in your jar and copy the data to the temp-file.
InputStream is = getClass().getResourceAsStream("/path/script.sh");
OutputStream os = new FileOutputStream(tempFile);
byte[] buffer = new byte[1024];
int bytesRead;
while ((bytesRead = is.read(buffer)) != -1)
{
os.write(buffer, 0, bytesRead);
}
buffer = null; // Clear the buffer
Then you have to execute your shellscript
Runtime.getRuntime().exec("terminalname " + tempFile.getAbsolutePath());
Maybe you can use this line to execute your script (I don't think this will work with your parameters):
java.awt.Desktop.getDestkop().open(tempFile);
I hope this is an answer for your question.
You could store your shell script as a resource (e.g. inside your jar file), then exec a shell and pipe the content of your script as standard input to the running shell.
Something like this (haven't tried it):
ProcessBuilder processBuilder = new ProcessBuilder( "/usr/bin/bash" );
Process process = processBuilder.start();
OutputStream outputStream = process.getOutputStream();
InputStream resourceStream = getClass().getResourceAsStream(
"/path/to/my/script.sh" );
IOUtils.copy( resourceStream, outputStream );
If you happen to be using the Spring framework for this project, a good and simple option is to store the shell script in a folder on your class path and use a ClassPathResource object to locate it.