We Keep Coding

How file manipulations perform during power outage

Linux machine, Java standalone application
I am having the following situation:
I have:
consecutive file write(which creates the destination file and writes some content to it) and file move.
I also have a power outage problem, which instantly cuts off the power of computer during these operations.
As a result, I am getting that the file was created, and it was moved as well, but the file content is empty.
The question is what under the hood can be causing this exact outcome? Considering the time sensitivity, may be hard drive is disabled before the processor and RAM during the cut out, but in that case, how is it possible that the file is created and moved after, but the write before moving is not successful?
I tried catching and logging the exception and debug information but the problem is power outage disables the logging abilities(I/O) as well.
try {
FileUtils.writeStringToFile(file, JsonUtils.toJson(object));
} finally {
if (file.exists()) {
FileUtils.moveFileToDirectory(file, new File(path), true);
}
}

Linux file systems don't necessarily write things to disk immediately, or in exactly the order that you wrote them. That includes both file content and file / directory metadata.
So if you get a power failure at the wrong time, you may find that the file data and metadata is inconsistent.
Normally this doesn't matter. (If the power fails and you don't have a UPS, the applications go away without getting a chance to finish what they were doing.)
However, if it does matter, you can do the following: to force the file to "sync" before you move it:
FileOutputStream fos = ...
// write to file
fs.getFD().sync();
fs.close();
// now move it
You need to read the javadoc for sync() carefully to understand what the method actually does.
You also need to read the javadoc for the method you are using to move the file regarding atomicity.

JMeter tries to use EOF in request

I have a HTTP request in a thread group that reads from a single column csv file to get values to populate a parameter in the request URL.
Below is my configuration for these:
There are 30 values in the csv data file.
My goal is to have each thread start at the beginning of the file once it gets to the end, effectively infinitely looping through the data values until the scheduler duration expires.
However, what actually happens is some requests try and use (see screenshot below) and therefore fail.
I have tried this but that just stops at the 30th iteration i.e. the end of the csv data file.
I assume I have some config option(s) wrong but I can't find anything online to suggest what they might be. Can anyone point me in the right direction (what i should be searching for?) or provide a solution?

Most probably it's test data issue, double check your CSV file and make sure it doesn't contain empty lines, if they are - remove them and your test should start working as expected.
For small files with only one column you can use __StringFromFile() function - it's much easier to set up and use.

Java File Marker API

I have a need for my application to be able to read large (very large, 100GB+) text files and process the content in these files potentially at different times. For instance, it might run for an hour and finish processing a few GBs, and then I shut it down and come back to it a few days later to resume processsing the same file.
To do this I will need to read the files into memory-friendly chunks; each chunk/page/block/etc will be read in, one at a time, processed, before then next chunk is read into memory.
I need the program to be able to mark where it is inside the input file, so if it shuts down, or if I need to "replay" the last chunk being processed, I can jump right to the point in the file where I am and continue processing. Specifically, I need to be able to do the following things:
When the processing begings, scan a file for a "MARKER" (some marker that indicates where we left off processing the last time)
If the MARKER exists, jump to it and begin processing from that point
Else, if the MARKER doesn't exist, then place a MARKER after the first chunk (for now, let's say that a "chunk" is just a line-of-text, as BufferedReader#readLine() would read in) and begin processing the first chunk/line
For each chunk/line processed, move the MARKER after the next chunk (thus, advancing the MARKER further down the file)
If we reach a point where there are no more chunks/lines after the current MARKER, we've finished processing the file
I tried coding this up myself and notice that BufferedReader has some interesting methods on it that sounds like they're suited for this very purpose: mark(), reset(), etc. But the Javadocs on them are kind of vague and I'm not sure that these "File Marker" methods will accomplish all the things I need to be able to do. I'm also completely open to a 3rd party JAR/lib that has this capability built into it, but Google didn't turn anything up.
Any ideas here?

Forget about markers. You cannot "insert" text without rewritting the whole file.
Use a RandomAccessFile and store the current position you are reading. When you need to open again the file, just use seek to find the position.

A Reader's "mark" is not persistent; it solely forms part of the state of the Reader itself.
I suggest that you not store the state information in the text file itself; instead, have a file alongside which stores the byte offset of the most recently processed chunk. That'll eliminate the obvious problems involving overwriting data in the original text file.

The marker of the buffered reader is not persisted after different runs of your application. I would neither change the content of that huge file to mark a position, since this can lead to significant IO and/or filesystem fragmentation, depending on your OS.
I would use a properties file to store the configuration of the program externally. Have a look at the documentation, the API is straight forward:
http://docs.oracle.com/javase/7/docs/api/java/util/Properties.html

How to etermine if a file is appended or outright changed

I'm wondering the best method for my program to determine if a file has been appended or completely changed.
What I am currently doing is using a LogIOThread I wrote that is using a FileInputStream.avail to check if a file has been appended. If so, I am appending to a non-editable JTextArea (actually a modified JTextArea with variable scroll features). My end-game was to have a autoscrolling JTextArea that is basically scrolling a log4j outputted logfile. This is working great for me right now. I'd like to adapt this to a non-log files.
What I would like to add to the LogIOThread is a monitor that will check if the file contents changed rather than just have new appended text. The first use case I am trying to solve is the file gets rewritten mid run. At the most basic level I figured I could use this function to reload my JTextArea close and reopen the FileInputStream, and start over if the file get overwritten.
I figured while I was doing that I might want to make it more robust to handle the second use case of mid-file insertions, like a properties file change. I figured I could get the textArea to do a line replace if I can figure out if a line changed.
I have a requirement (not set by me) to use Java6, so some of the new nio FileWatcher's are no good for me. Notifiers in general seem counter productive as I'm catching appends via FileInputStream. This also led me to discount other libs such as Commons and jnotify. I could be wrong though, and maybe FileInputStream.avail is not the best way for me anymore.
My first thought was that I could store the file size, and check if it drastically changed (to a value less than stored). Since a new created log file would need a fresh textArea, this doesn't seem to bad, and solves my first use case. This would not work for the second use case of a specific value change in the file. It also might be problematic for a file that gets recreated with a consistent size while my IOThread is sleeping, although I doubt it's likely.
My second thought was that I could continually check the file modified time, and if there is no appendable text, I would reread the file and do a line comparison? I would need to do this anyway if I'm going to change a line in the textArea unless I reload it every time. This seems horribly inefficient though.
Any suggestions? I'm not opposed to changing how the LogIOThread works if there is a suggestion to get new text and changes with something better than an avail + file modification check combo.

If the file size decreases it has certainly been overwritten. However it may also be overwritten with something as large or larger, so the converse does not hold.
Don't use available() as a surrogate for File.length(). See the Javadoc. That's not what it's for.

How about creating an intermediate OutputStream that reads the log data, makes it available to the JTextField and then routes that data to the file?

How to edit a specific attribute in file?

So say you have a file that is written in XML or soe other coding language of that sort. Is it possible to just rewrite one line rather than getting the entire file into a string, then changing then line, then having to rewrite the whole string back to the file?

In general, no. File systems don't usually support the idea of inserting or modifying data in the middle of a file.
If your data file is in a fixed-size record format then you can edit a record without overwriting the rest of the file. For something like XML, you could in theory overwrite one value with a shorter one by inserting semantically-irrelevant whitespace, but you wouldn't be able to write a larger value.
In most cases it's simpler to just rewrite the whole file - either by reading and writing in a streaming fashion if you can (and if the file is too large to read into memory in one go) or just by loading the whole file into some in-memory data structure (e.g. XDocument), making the changes, and then saving the file again. (You may want to consider saving to a different file then moving the files around to avoid losing data if the save operation fails for some reason.)
If all of this ends up being too expensive, you should consider using multiple files (so each one is smaller) or a database.

If the line you want to replace is larger than the new line that you want to replace it with, then it is possible as long as it is acceptable to have some kind of padding (for example white-space characters ' ') which will not effect your application.
If on the other hand the new content are larger than the content to be replaced you will need to shift all the data downwards, so you need to rewrite the file, or at least from the replaced line onwards.
Since you mention XML, it might be you are approaching your problem in the wrong way. Could it be that what you need is to replace a specific XML node? In which case you might consider using DOM to read the XML into a hierarchy of nodes and adding/updating/removing in there before writing the XML tree back to the file.