We Keep Coding

How to modify update a large file with small content changes at specific indexes

I need to modify a file. We've already written a reasonably complex component to build sets of indexes describing where interesting things are in this file, but now I need to edit this file using that set of indexes and that's proving difficult.
Specifically, my dream API is something like this
//if you'll let me use kotlin for a second, assume we have a simple tuple class
data class IdentifiedCharacterSubsequence { val indexOfFirstChar : int, val existingContent : String }
//given these two structures
List<IdentifiedCharacterSubsequences> interestingSpotsInFile = scanFileAsPerExistingBusinessLogic(file, businessObjects);
Map<IdentifiedCharacterSubsequences, String> newContentByPreviousContentsLocation = generateNewValues(inbterestingSpotsInFile, moreBusinessObjects);
//I want something like this:
try(MutableFile mutableFile = new com.maybeGoogle.orApache.MutableFile(file)){
for(IdentifiedCharacterSubsequences seqToReplace : interestingSpotsInFile){
String newContent = newContentByPreviousContentsLocation.get(seqToReplace);
mutableFile.replace(seqToReplace.indexOfFirstChar, seqtoReplace.existingContent.length, newContent);
//very similar to StringBuilder interface
//'enqueues' data changes in memory, doesnt actually modify file until flush call...
}
mutableFile.flush();
// ...at which point a single write-pass is made.
// assumption: changes will change many small regions of text (instead of large portions of text)
// -> buffering makes sense
}
Some notes:
I cant use RandomAccessFile because my changes are not in-place (the length of newContent may be longer or shorter than that of seq.existingContent)
The files are often many megabytes big, thus simply reading the whole thing into memory and modifying it as an array is not appropriate.
Does something like this exist or am I reduced to writing my own implementation using BufferedWriters and the like? It seems like such an obvious evolution from io.Streams for a language which typically emphasizes indexed based behaviour heavily, but I cant find an existing implementation.
Lastly: I have very little domain experience with files and encoding schemes, so I have taken no effort to address the 'two-index' character described in questions like these: Java charAt used with characters that have two code units. Any help on this front is much appreciated. Is this perhaps why I'm having trouble finding an implementation like this? Because indexes in UTF-8 encoded files are so pesky and bug-prone?

Invoking AS400 RPG From Java

I have a very limitied (0) knowledge on AS400 and RPG. But we have a urgent requirement where we need to invoke a RPG program from a java class. So I found that we can achieve it through JTOpen. But I am stuck at declaring the ProgramParameter list. I have the following information about RPG Program
Program name: ZM30000R
Parameters:
Branch 7,0 (Numeric)
Account type 2 (01-cheque,02 savings)
Account Number 20 (character)
Error code 7 (character)
DR/CR indicater 1 (character D,C)
But no clue about what is the intput and output .How to declare the ProgramParameter. I have done as below. I cannot test as well because I dont have connectivity to these systems.
// Create AS400 Text objects for the different lengths
// of parameters you are sending in.
AS400Text branchTxt = new AS400Text(7);
AS400Text accntTypeTxt = new AS400Text(2);
AS400Text accntNumberTxt = new AS400Text(20);
AS400Text errorCodeTxt = new AS400Text(7);
AS400Text DCIndicatorTxt = new AS400Text(1);
// declare and instantiate your parameter list.
ProgramParameter[] parmList = new ProgramParameter[5];
// assign values to your parameters using the AS400Text class to convert to bytes
// the second parameter is an integer which sets the length of your parameter output
parmList[0] = new ProgramParameter( branchTxt.toBytes(branch),7);
parmList[1] = new ProgramParameter( accntTypeTxt.toBytes(accntTypeTxt),2);
parmList[2] = new ProgramParameter( accntNumberTxt.toBytes(accntNumberTxt),20);
parmList[3] = new ProgramParameter( errorCodeTxt.toBytes(""),7);
parmList[4] = new ProgramParameter( DCIndicatorTxt.toBytes(indicator),5);
Any help will be really highly useful.
Thanks and Regards,
Srinivas

Well, I do have a clue just by the description of the parameters. Branch, account type and account number are IN. You need that information for a financial booking or transaction. The error code is appearently OUT. In my experience with financial systems it's reasonable normal that the API returns the way the amount is booked. Normally one would use the sign, but in financial systems the (D)ebit or (C)redit is the better way.
The API is very likely the API of a financial system. If that is true, then I'm missing the amount. Are you sure you've the complete description?
Notice that the first parameter is numeric. You're not in luck. The iSeries and RPG are not very forgiving about the type of a numeric. One can choose from Bit, Zoned, Packed, Decimal, Integer, Float and so on. If the RPG is really RPG instead of ILE RPG, then you can bring that down to Zoned, Packed and Byte.
I assume you've access to the iSeries. Then you can watch the program call, debug information and dump information. That will help you if you have to do "trial and error". If you don't have access, the road will be very hard. You'll receive an error in your java class if the program call is not succesfull. But it will be hard to identify the real error without the information from the iSeries yourself. Therefore, access is really required.

Your sample is mostly on the right track. But your branch parameter is numeric. So you should use AS400ZonedDecimal instead of AS400Text:
AS400ZonedDecimal branchNbr = new AS400ZonedDecimal(7,0)
The RPG program may be expecting packed instead of zoned. No big deal, just use AS400PackedDecimal instead.
As you construct your ProgramParameter object, your constructor requirements are different depending on if they are input or output parameters to your program. For input parameters, just pass the toBytes() results. There is no need to include the length. For output-only parameters, just pass the length.
I agree with Robert's answer that there is some missing information, but his assumptions on the outputness of the error code seems valid. I would guess, however, that the DCIndicator parameter is input since your sample passes a value. For the error code parameter, after your program call, you'll need to extract the value and do something with it. Given what you have already, here is how the program call would work. Take note that I specified a library name of "MyLibrary". That is for example purposes. You will have to determine which library your program object is in.
ProgramCall pgm = new ProgramCall(as400, QSYSObjectPathName.toPath("MyLibrary","ZM30000R","PGM"), parmList);
if (pgm.run() == true) {
String sErrorCode = (String) errorCodeTxt.toObject(parmList[3].getOutputData());
//Do something with your output data.
}
else {
AS400Message[] messageList = pgm.getMessageList();
for (int i=0; i<messageList.length; i++) {
String sMessageID = messageList[i].getID();
String sMessageText = messageList[i].getText();
//Do something with the error messages
}
}
Something else to consider is library lists. Does the RPG program expect certain libraries to be in the library list? If so, you should issue CommandCalls to add the libraries to the library list before calling the program.

FWIW: It's a lot easier to call IBM i host programs & service programs using PCML rather than ProgramCall.
The compilers will even generate the PCML document for you.
See http://javadoc.midrange.com/jtopen/com/ibm/as400/data/ProgramCallDocument.html for details.

If you don't have connectivity, then you really can't do what is asked. How do you test it? Is there numeric parameters or are they all character?

I need a fast key substitution algorithm for java

Given a string with replacement keys in it, how can I most efficiently replace these keys with runtime values, using Java? I need to do this often, fast, and on reasonably long strings (say, on average, 1-2kb). The form of the keys is my choice, since I'm providing the templates here too.
Here's an example (please don't get hung up on it being XML; I want to do this, if possible, cheaper than using XSL or DOM operations). I'd want to replace all #[^#]*?# patterns in this with property values from bean properties, true Property properties, and some other sources. The key here is fast. Any ideas?
<?xml version="1.0" encoding="utf-8"?>
<envelope version="2.3">
<delivery_instructions>
<delivery_channel>
<channel_type>#CHANNEL_TYPE#</channel_type>
</delivery_channel>
<delivery_envelope>
<chan_delivery_envelope>
<queue_name>#ADDRESS#</queue_name>
</chan_delivery_envelope>
</delivery_envelope>
</delivery_instructions>
<composition_instructions>
<mime_part content_type="application/xml">
<content><external_uri>#URI#</external_uri></content>
</mime_part>
</composition_instructions>
</envelope>
The naive implementation is to use String.replaceAll() but I can't help but think that's less than ideal. If I can avoid adding new third-party dependencies, so much the better.

The appendReplacement method in Matcher looks like it might be useful, although I can't vouch for its speed.
Here's the sample code from the Javadoc:
Pattern p = Pattern.compile("cat");
Matcher m = p.matcher("one cat two cats in the yard");
StringBuffer sb = new StringBuffer();
while (m.find()) {
m.appendReplacement(sb, "dog");
}
m.appendTail(sb);
System.out.println(sb.toString());
EDIT: If this is as complicated as it gets, you could probably implement your own state machine fairly easily. You'd pretty much be doing what appendReplacement is already doing, although a specialized implementation might be faster.

It's premature to leap to writing your own. I would start with the naive replace solution, and actually benchmark that. Then I would try a third-party templating solution. THEN I would take a stab at the custom stream version.
Until you get some hard numbers, how can you be sure it's worth the effort to optimize it?

Does Java have a form of regexp replace() where a function gets called?
I'm spoiled by the Javascript String.replace() method. (For that matter you could run Rhino and use Javascript, but somehow I don't think that would be anywhere near as fast as a pure Java call even if the Javascript compiler/interpreter were efficient)
edit: never mind, #mmyers probably has the best answer.
gratuitous point-groveling: (and because I wanted to see if I could do it myself :)
Pattern p = Pattern.compile("#([^#]*?)#");
Matcher m = p.matcher(s);
StringBuffer sb = new StringBuffer();
while (m.find())
{
m.appendReplacement(sb,substitutionTable.lookupKey(m.group(1)));
}
m.appendTail(sb);
// replace "substitutionTable.lookupKey" with your routine

You really want to write something custom so you can avoid processing the string more than once. I can't stress this enough - as most of the other solutions I see look like they are ignoring that problem.
Optionally turn the text into a stream. Read it char by char forwarding each char to an output string/stream until you see the # then read to the next # slurping out the key, substituting the key into the output: repeat until end of stream.
I know it's plain old brute for - but it's probably the best.
I'm assuming you have some reasonable assumption around '#' not just 'showing up' independant of your token keys in the input. :)

please don't get hung up on it being XML; I want to do this, if possible, cheaper than using XSL or DOM operations
Whatever's downstream from your process will get hung up if you don't also process the inserted strings for character escapes. Which isn't to say that you can't do it yourself if you have good cause, but does mean you either have to make sure your patterns are all in text nodes, and you also correctly escape the replacement text.
What exact advantage does #Foo# have over the standard &Foo; syntax already built into the XML libraries which ship with Java?

Text processing is going to always be bounded if you dont shift your paradigm. I dont know how flexible your domain is, so not sure if this is applicable, but here goes:
try creating an index into where your text substitution is - this is especially good if the template doesnt change often, because it becomes part of the "compile" of the template, into a binary object that can take in the value required for the substitutions, and blit out the entire string as a byte array. This object can be cached/saved, and next time, resubstitute in the new value to use again. I.e., you save on parsing the document every time. (implementation is left as an exercise to the reader =D )
But please use a profiler to check whether this is actually the bottleneck that you say it is before embarking on writing a custom templating engine. The problem may actually be else where.

As others have said, appendReplacement() and appendTail() are the tools you need, but there's something you have watch out for. If the replacement string contains any dollar signs, the method will try to interpret them as capture-group references. If there are any backslashes (which are used to escape the dollars sing), it will either eat them or throw an exception.
If your replacement string is dynamically generated, you may not know in advance whether it will contain any dollar signs or backslashes. To prevent problems, you can append the replacement directly to the StringBuffer, like so:
Pattern p = Pattern.compile("#([^#]*?)#");
Matcher m = p.matcher(s);
StringBuffer sb = new StringBuffer();
while (m.find())
{
m.appendReplacement("");
sb.append(substitutionTable.lookupKey(m.group(1)));
}
m.appendTail(sb);
You still have to call appendReplacement() each time, because that's what keeps you in sync with the match position. But this trick avoids a lot of pointless processing, which could give you a noticeable performance boost as a bonus.

this is what I use, from the apache commons project
http://commons.apache.org/lang/api/org/apache/commons/lang/text/StrSubstitutor.html

I also have a non-regexp based substitution library, available here. I have not tested its speed, and it doesn't directly support the syntax in your example. But it would be easy to extend to support that syntax; see, for instance, this class.

Take a look at a library that specializes in this, e.g., Apache Velocity. If nothing else, you can bet their implementation for this part of the logic is fast.

I wouldn't be so sure the accepted answer is faster than String.replaceAll(String,String). Here for your comparison is the implementation of String.replaceAll and the Matcher.replaceAll that is used under the covers. looks very similar to what the OP is looking for, and I'm guessing its probably more optomized than this simplistic solution.
public String replaceAll(String s, String s1)
{
return Pattern.compile(s).matcher(this).replaceAll(s1);
}
public String replaceAll(String s)
{
reset();
boolean flag = find();
if(flag)
{
StringBuffer stringbuffer = new StringBuffer();
boolean flag1;
do
{
appendReplacement(stringbuffer, s);
flag1 = find();
} while(flag1);
appendTail(stringbuffer);
return stringbuffer.toString();
} else
{
return text.toString();
}
}

... Chii is right.
If this is a template that has to be run so many times that speed matters, find the index of your substitution tokens to be able to get to them directly without having to start at the beginning each time. Abstract the 'compilation' into an object with the nice properties, they should only need updating after a change to the template.

Rythm a java template engine now released with an new feature called String interpolation mode which allows you do something like:
String result = Rythm.render("Hello #who!", "world");
The above case shows you can pass argument to template by position. Rythm also allows you to pass arguments by name:
Map<String, Object> args = new HashMap<String, Object>();
args.put("title", "Mr.");
args.put("name", "John");
String result = Rythm.render("Hello #title #name", args);
Since your template content is relatively long you could put them into a file and then call Rythm.render using the same API:
Map<String, Object> args = new HashMap<String, Object>();
// ... prepare the args
String result = Rythm.render("path/to/my/template.xml", args);
Note Rythm compile your template into java byte code and it's fairly fast, about 2 times faster than String.format
Links:
Check the full featured demonstration
read a brief introduction to Rythm
download the latest package or
fork it

Developing a (file) exchange format for java

I want to come up with a binary format for passing data between application instances in a form of POFs (Plain Old Files ;)).
Prerequisites:
should be cross-platform
information to be persisted includes a single POJO & arbitrary byte[]s (files actually, the POJO stores it's names in a String[])
only sequential access is required
should be a way to check data consistency
should be small and fast
should prevent an average user with archiver + notepad from modifying the data
Currently I'm using DeflaterOutputStream + OutputStreamWriter together with InflaterInputStream + InputStreamReader to save/restore objects serialized with XStream, one object per file. Readers/Writers use UTF8.
Now, need to extend this to support the previously described.
My idea of format:
{serialized to XML object}
{delimiter}
{String file name}{delimiter}{byte[] file data}
{delimiter}
{another String file name}{delimiter}{another byte[] file data}
...
{delimiter}
{delimiter}
{MD5 hash for the entire file}
Does this look sane?
What would you use for a delimiter and how would you determine it?
The right way to calculate MD5 in this case?
What would you suggest to read on the subject?
TIA.

It looks INsane.
why invent a new file format?
why try to prevent only stupid users from changing file?
why use a binary format ( hard to compress ) ?
why use a format that cannot be parsed while being received? (receiver has to receive entire file before being able to act on the file. )
XML is already a serialization format that is compressable. So you are serializing a serialized format.

Would serialization of the model (if you are into MVC) not be another way? I'd prefer to use things in the language (or standard libraries) rather then roll my own if possible. The only issue I can see with that is that the file size may be larger than you want.

1) Does this look sane?
It looks fairly sane. However, if you are going to invent your own format rather than just using Java serialization then you should have a good reason. Do you have any good reasons (they do exist in some cases)? One of the standard reasons for using XStream is to make the result human readable, which a binary format immediately loses. Do you have a good reason for a binary format rather than a human readable one? See this question for why human readable is good (and bad).
Wouldn't it be easier just to put everything in a signed jar. There are already standard Java libraries and tools to do this, and you get compression and verification provided.
2) What would you use for a delimiter and how determine it?
Rather than a delimiter I'd explicitly store the length of each block before the block. It's just as easy, and prevents you having to escape the delimiter if it comes up on its own.
3) The right way to calculate MD5 in this case?
There is example code here which looks sensible.
4) What would you suggest to read on the subject?
On the subject of serialization? I'd read about the Java serialization, JSON, and XStream serialization so I understood the pros and cons of each, especially the benefits of human readable files. I'd also look at a classic file format, for example from Microsoft, to understand possible design decisions from back in the days that every byte mattered, and how these have been extended. For example: The WAV file format.

Let's see this should be pretty straightforward.
Prerequisites:
0. should be cross-platform
1. information to be persisted includes a single POJO & arbitrary byte[]s (files actually, the POJO stores it's names in a String[])
2. only sequential access is required
3. should be a way to check data consistency
4. should be small and fast
5. should prevent an average user with archiver + notepad from modifying the data
Well guess what, you pretty much have it already, it's built-in the platform already:Object Serialization
If you need to reduce the amount of data sent in the wire and provide a custom serialization ( for instance you can sent only 1,2,3 for a given object without using the attribute name or nothing similar, and read them in the same sequence, ) you can use this somehow "Hidden feature"
If you really need it in "text plain" you can also encode it, it takes almost the same amount of bytes.
For instance this bean:
import java.io.*;
public class SimpleBean implements Serializable {
private String website = "http://stackoverflow.com";
public String toString() {
return website;
}
}
Could be represented like this:
rO0ABXNyAApTaW1wbGVCZWFuPB4W2ZRCqRICAAFMAAd3ZWJzaXRldAASTGphdmEvbGFuZy9TdHJpbmc7eHB0ABhodHRwOi8vc3RhY2tvdmVyZmxvdy5jb20=
See this answer
Additionally, if you need a sounded protocol you can also check to Protobuf, Google's internal exchange format.

You could use a zip (rar / 7z / tar.gz / ...) library. Many exists, most are well tested and it'll likely save you some time.
Possibly not as much fun though.

I agree in that it doesn't really sound like you need a new format, or a binary one.
If you truly want a binary format, why not consider one of these first:
Binary XML (fast infoset, Bnux)
Hessian
google packet buffers
But besides that, many textual formats should work just fine (or perhaps better) too; easier to debug, extensive tool support, compresses to about same size as binary (binary compresses poorly, and information theory suggests that for same effective information, same compression rate is achieved -- and this has been true in my testing).
So perhaps also consider:
Json works well; binary support via base64 (with, say, http://jackson.codehaus.org/)
XML not too bad either; efficient streaming parsers, some with base64 support (http://woodstox.codehaus.org/, "typed access API" under 'org.codehaus.stax2.typed.TypedXMLStreamReader').
So it kind of sounds like you just want to build something of your own. Nothing wrong with that, as a hobby, but if so you need to consider it as such.
It likely is not a requirement for the system you are building.

Perhaps you could explain how this is better than using an existing file format such as JAR.
Most standard files formats of this type just use CRC as its faster to calculate. MD5 is more appropriate if you want to prevent deliberate modification.

Bencode could be the way to go.
Here's an excellent implementation by Daniel Spiewak.
Unfortunately, bencode spec doesn't support utf8 which is a showstopper for me.
Might come to this later but currently xml seems like a better choice (with blobs serialized as a Map).

Inserting a Java string in another string without concatenation? [duplicate]

This question already has answers here:
How to format strings in Java
(10 answers)
Closed 5 years ago.
Is there a more elegant way of doing this in Java?
String value1 = "Testing";
String test = "text goes here " + value1 + " more text";
Is it possible to put the variable directly in the string and have its value evaluated?

String test = String.format("test goes here %s more text", "Testing");
is the closest thing that you could write in Java

A more elegant way might be:
String value = "Testing";
String template = "text goes here %s more text";
String result = String.format(template, value);
Or alternatively using MessageFormat:
String template = "text goes here {0} more text";
String result = MessageFormat.format(template, value);
Note, if you're doing this for logging, then you can avoid the cost of performing this when the log line would be below the threshold. For example with SLFJ:
The following two lines will yield the exact same output. However, the second form will outperform the first form by a factor of at least 30, in case of a disabled logging statement.
logger.debug("The new entry is "+entry+".");
logger.debug("The new entry is {}.", entry);

Rythm a java template engine now released with an new feature called String interpolation mode which allows you do something like:
String result = Rythm.render("Hello #who!", "world");
The above case shows you can pass argument to template by position. Rythm also allows you to pass arguments by name:
Map<String, Object> args = new HashMap<String, Object>();
args.put("title", "Mr.");
args.put("name", "John");
String result = Rythm.render("Hello #title #name", args);
Links:
Check the full featured demonstration
read a brief introduction to Rythm
download the latest package or
fork it

It may be done by some template-libaries. But beware, Strings are immutable in Java. So in every case at some low level the concatenation will be done.

You'll always have to use some form of concatenation for this (assuming value1 isn't a constant like you show here).
The way you've written it will implicitly construct a StringBuilder and use it to concatenate the strings. Another method is String.format(String, Object...)1, which is analogous to sprintf from C. But even with format(), you can't avoid concatenation.
1 Yes, I know the anchor link is broken.

What you want is called String interpolation. It is not possible in Java, although JRuby, Groovy and probably other JVM languages do that.
Edit: as for elegance, you can use a StringBuffer or check the other poster's solution. But at the low level, this will always be concatenation, as the other posters said.

You can use this free library. It gives you sprintf like functionality. Or use String.format static method provided you use Java 5 or newer.

Why do you think string concatenation isn't elegant?
If all you are doing is simple concatenation, I'd argue that code readability is more important and I'd leave it like you have it. It's more readable than using a StringBuilder.
Performance won't be the problem that most people think it is.
Read this from CodingHorror

I would use a StringBuffer.. it's a common practise when you are dealing with strings. It may seem a bit when you see it for the first time, but you'll get quickly used to it..
String test = new StringBuffer("text goes here ").append(value1).append(" more text").toString();
Strings are immutable thus a new instance is created after every concatenation. This can cause performance issues when used in loops.
StringBuffer is mutable version of String - that means you can create one, modify it as you want and you have still only one instance. When desired you can get a String representation of the StringBuffer by calling it's toString() method.

The problem is not if this is an elegant way or not. The idea behind using a template system may be that you put your template in a normal text file and don't have to change java code if you change your message (or think about i18ln).