We Keep Coding

Character encoding of parsed strings is wrong only after building jar

I am writing a program that generates PDF files of printable exams. I have all the exam questions stored in a JSON file. The catch is that the exam is in Czech, so there are many special characters (specifically ěščřžýáíé). When I run the program in Idea, it works perfectly - the output is exactly as it is supposed to be.
But when I build the jar executable, the generated files have chunks of wrong encoded text. Specifically anything that went through the JSON parser. Everything hard coded like headers etc. is encoded properly, so the mistake must be in the parser.
The JSON input file is encoded in UTF-8.
I use these two methods to parse the JSON file.
private static Category[] parseJSON(){
JSONParser jsonParser = new JSONParser();
Category[] categories = new Category[0];
try (FileReader reader = new FileReader("otazky.json")){
// Read JSON file
Object obj = jsonParser.parse(reader);
JSONArray categoryJSONList = (JSONArray) obj;
java.util.List<JSONObject> categoryList = new ArrayList<>(categoryJSONList);
categories = new Category[categoryJSONList.size()];
int i = 0;
for (JSONObject category : categoryList) {
categories[i] = parseCategoryObject(category);
i++;
}
} catch (ParseException | IOException e) {
e.printStackTrace();
}
return categories;
}
private static Category parseCategoryObject(JSONObject category) {
String categoryName = (String) category.get("name");
int generateCount = (int) (long) category.get("generateCount");
JSONArray questionsJSONArray = (JSONArray) category.get("questions");
java.util.List<JSONObject> questionJSONList = new ArrayList<>(questionsJSONArray);
Question[] questions = new Question[questionJSONList.size()];
int j = 0;
for (JSONObject question : questionJSONList) {
JSONArray answers = (JSONArray) question.get("answers");
String s = (String) question.get("question");
String[] a = new String[answers.size()];
for (int i = 0; i < answers.size(); i++) {
a[i] = answers.get(i).toString();
}
int c = (int) (long) question.get("correct");
Question q = new Question(s, a, c);
questions[j] = q;
j++;
}
return new Category(categoryName, questions, generateCount);
}
The output looks like this:
...
Právnà norma:
a) je obecnÄ› závaznĂ© pravidlo chovánĂ, kterĂ© nemusĂ mĂt urÄŤitou formu,
b) nemĹŻĹľe bĂ˝t součástĂ právnĂho pĹ™edpisu,
...
While I would need it to look like this:
...
Právní norma:
a) je obecně závazné pravidlo chování, které nemusí mít určitou formu,
b) nemůže být součástí právního předpisu,
...

Benjamin Urquhart suggested that I try using InputStringReader and FileInputStream instead of FileReader to read the file, because with FileReader you cannot specify the encoding (system default is used). I find those two methods hard to use, but I found an alternative - Files.readAllLines, which is fairly easy to use, and it worked.

How to deserialize avro files

I would like to read a hdfs folder containing avro files with spark . Then I would like to deserialize the avro events contained in these files. I would like to do it without the com.databrics library (or any other that allow to do it easely).
The problem is that I have difficulties with the deserialization.
I assume that my avro file is compressed with snappy because at the begining of the file (just after the schema), I have
avro.codecsnappy
written. Then it's followed by readable or unreadable charaters.
My first attempt to deserialize the avro event is the following :
public static String deserialize(String message) throws IOException {
Schema.Parser schemaParser = new Schema.Parser();
Schema avroSchema = schemaParser.parse(defaultFlumeAvroSchema);
DatumReader<GenericRecord> specificDatumReader = new SpecificDatumReader<GenericRecord>(avroSchema);
byte[] messageBytes = message.getBytes();
Decoder decoder = DecoderFactory.get().binaryDecoder(messageBytes, null);
GenericRecord genericRecord = specificDatumReader.read(null, decoder);
return genericRecord.toString();
}
This function works when I want to deserialise an avro file that doesn't have the avro.codecsbappy in it. When it's the case I have the error :
Malformed data : length is negative : -50
So I tried another way of doing it which is :
private static void deserialize2(String path) throws IOException {
DatumReader<GenericRecord> reader = new GenericDatumReader<>();
DataFileReader<GenericRecord> fileReader =
new DataFileReader<>(new File(path), reader);
System.out.println(fileReader.getSchema().toString());
GenericRecord record = new GenericData.Record(fileReader.getSchema());
int numEvents = 0;
while (fileReader.hasNext()) {
fileReader.next(record);
ByteBuffer body = (ByteBuffer) record.get("body");
CharsetDecoder decoder = Charsets.UTF_8.newDecoder();
System.out.println("Positon of the index " + body.position());
System.out.println("Size of the array : " + body.array().length);
String bodyStr = decoder.decode(body).toString();
System.out.println("THE BODY STRING ---> " bodyStr);
numEvents++;
}
fileReader.close();
}
and it returns the follwing output :
Positon of the index 0
Size of the array : 127482
THE BODY STRING --->
I can see that the array isn't empty but it just return an empty string.
How can I proceed ?

Use this when converting to string:
String bodyStr = new String(body.array());
System.out.println("THE BODY STRING ---> " + bodyStr);
Source: https://www.mkyong.com/java/how-do-convert-byte-array-to-string-in-java/

Well, it seems that you are on a good way. However, your ByteBuffer might not have a proper byte[] array to decode, so let's try the following instead:
byte[] bytes = new byte[body.remaining()];
buffer.get(bytes);
String result = new String(bytes, "UTF-8"); // Maybe you need to change charset
This should work, you have shown in your question that ByteBuffer contains actual data, as given in the code example you might have to change the charset.
List of charsets: https://docs.oracle.com/javase/7/docs/api/java/nio/charset/Charset.html
Also usful: https://docs.oracle.com/javase/7/docs/api/java/nio/ByteBuffer.html

iText - check if attachment exists / link attachment multiple times

Is there a way to check if an attachment is already present in the PDF document while creating the document (not after the document is saved to disk)? While parsing a XML to PDF I came across over multiple attachments which have the same content (Base64 String from XML > byte[]) and the same name. Currently the attachments are added multiple times, but I want to check if an attachment (with the same content or name) already exists (PdfWriter API?) and if YES, only a new Annotation should be created to the existing attachment.
NOTE: the check should happen while creating the PDF, not with a PdfReader and an existing PDF
EDIT:
Thanks to #Bruno Lowagie I got it working:
protected HashMap<String, PdfFileSpecification> cache = new HashMap<>();
private final byte[] BUFFER = new byte[1024];
public PdfFileSpecification getPdfFileSpecification(final PdfWriter pdfWriter, final String name, final byte[] data) throws IOException {
String hash = createMD5Hash(data);
PdfFileSpecification pdfFileSpecification = cache.get(hash);
if (pdfFileSpecification == null) {
pdfFileSpecification = PdfFileSpecification.fileEmbedded(pdfWriter, null, name, data);
cache.put(hash, pdfFileSpecification);
return pdfFileSpecification;
}
System.out.println(String.format("Name: %s Hash: %s", name, hash));
return pdfFileSpecification;
}
private String createMD5Hash(final byte[] data) {
MessageDigest messageDigest;
try {
messageDigest = MessageDigest.getInstance("MD5");
} catch (NoSuchAlgorithmException e) {
return null;
}
ByteArrayInputStream byteArrayInputStream = new ByteArrayInputStream(data);
try {
int i;
while ((i = byteArrayInputStream.read(BUFFER)) != -1) {
messageDigest.update(BUFFER, 0, i);
}
byteArrayInputStream.close();
} catch (IOException e) {
return null;
}
byte[] mdbytes = messageDigest.digest();
StringBuffer sb = new StringBuffer();
for (int i = 0; i < mdbytes.length; i++) {
sb.append(Integer.toString((mdbytes[i] & 0xff) + 0x100, 16).substring(1));
}
return sb.toString();
}
So every time I have to deal with a new attachment I do it like this:
PdfFileSpecification fs = getPdfFileSpecification(pdfWriter, name, data)
PdfAnnotation an = PdfAnnotation.createFileAttachment(pdfWriter, rectangle, name, fs);

Allow me to take your code and introduce some pseudo code to show you how I would do this:
protected Map<String, PdfFileSpecification> cache =
new HashMap<String, PdfFileSpecification>();
public void cellLayout(final PdfPCell pdfPCell, final Rectangle rectangle, final PdfContentByte[] pdfContentBytes) {
String hasheddata = createHash(attachment);
PdfFileSpecification fs = cache.get(hasheddata);
if (fs == null) {
fs = PdfFileSpecification.fileEmbedded(writer, null, displayname, attachment);
cache.put(hasheddata, fs);
}
PdfAnnotation an = PdfAnnotation.createFileAttachment(writer, rectangle, displayname, fs);
writer.addAnnotation(an);
}
This code won't compile because I left out some parts that aren't relevant to the problem. I only kept the stuff that explains the concept of creating the cache for the file specifications.
I create a hash of the attachment bytes to save memory. You will have to implement the createHash() method using the hashing algorithm of your choice. Before I create a new FileSpecification that will write bytes to the PdfWriter, I check if I can't reuse an already existing file specification. If one exists, I reuse it in an annotation. If it doesn't exist I create a new file specification.

Parsing list of json

I need to parse list of json stored in a single file !
What I have done so far is,
test.json file contains:
{"location":"lille","lat":28.4,"long":51.7,"country":"FR"}
with this file I have the code below
public class JsonReader {
public static void main(String[] args) throws ParseException {
JSONParser parser = new JSONParser();
try {
Object obj = parser.parse(new FileReader("c:\\test.json"));
JSONObject locationjson= (JSONObject) obj;
String location= (String) locationjson.get("location");
System.out.printf("%s",location);
long lat = (Long) locationjson.get("lat");
System.out.printf("\t%d",lat);
//similarly for other objects
This is a working code and I am able to print only one json in the file test.json
Now if I have to print a list of json in file: test1.json as shown below: each line is a single valid json and there are list of json in a single file. What I need is to parse each json and print it in each line. Will using a bean class work?
{"Atlas":{"location":"lille","lat":28.4,"long":51.7,"country":"FR"}}
{"Atlas":{"location":"luxum","lat":24.1,"long":54.7,"country":"LU"}}
{"Atlas":{"location":"ghent","lat":28.1,"long":50.1,"country":"BE"}}
{"Atlas":{"location":"alborg","lat":23.4,"long":53.7,"country":"DN"}}
Your help is appreciated !

The JSON should have a root node.
If you don't have that, you can read from the file line-by-line, and pass each line into the JSONParser wrapped in a StringReader (since the JSONParser.parse() method takes a Reader).
e.g.
BufferedReader in
= new BufferedReader(new FileReader("test.json"));
while(!done) {
String s = in.readLine();
if (s == null) {
done = true;
}
else {
StringReader sr = new StringReader(s);
// etc...
}
}
Edit: I've assumed you're using JSONParser. If you're using a different parser (which one?) then it may take a String argument.

JSONParser.parse() also takes the String as an aurgument.
Read the file with FileReader and for each line that you read use JSONParser.parse(String) method.

First of all:make sure you have created a class just for the son item for each row that has the string properties in the header file and synthesized in the implementation file.Then create a property array in the implementation file that will be doing the parsing.In the parse file..use NSJSONSerialization in your retrieve data method:
-(void)retrieveData{
NSURL * url =[NSURL URLWithString:getDataURL];
NSData *data =[NSData dataWithContentsOfURL:url];
json = [NSJSONSerialization JSONObjectWithData:data options:kNilOptions error:nil];
newsArray = [[NSMutableArray alloc]init];
for (int i = 0; i < json.count; i++){
{
NSString * cID =[[json objectAtIndex:i]objectForKey:#"Name1"];
NSString * cName = [[json objectAtIndex:i]objectForKey:#"name2"];
NSString * cState =[[json objectAtIndex:i]objectForKey:#"name3"];
NSString * cPopulation =[[json objectAtIndex:i]objectForKey:#"Edition"];
NSString * cCountry =[[json objectAtIndex:i]objectForKey:#"name4"];
City *myCity = [[City alloc]initWithCityID:cID andCityName:cName andCityState:cState andCityPopulation:cPopulation andCityCountry:cCountry];
[newsArray addObject:myCity];
}
[self.myTableView reloadData];
}
}
Then retrieve the son objects or arrays and enter them in the name 1 name 2 section to parse them to your table.Also make sure you have already setup your Table and assigned it a cell identifier and indexed to row.
-(UITableViewCell *)tableView:(UITableView *)tableView cellForRowAtIndexPath:(NSIndexPath *)indexPath{
//static NSString *strIdentifier=#"identity";
static NSString *cellIdentifier = #"Cell";
UITableViewCell *cell = [tableView dequeueReusableCellWithIdentifier:cellIdentifier];
if(cell == nil){
cell = [[UITableViewCell alloc]initWithStyle:UITableViewCellStyleSubtitle reuseIdentifier:cellIdentifier];
}
// cell.textLabel.text = [NSString stringWithFormat:#"Cell %d",indexPath.row];
City *currentCity =[newsArray objectAtIndex:indexPath.row];
cell.textLabel.text = currentCity.Name1;
cell.detailTextLabel.text = currentCity.Name2;
return cell;
}

Write XML file (using XStream) to filesystem in Java

I need to be able to serialize a string and then have it save in a .txt or .xml file. I've never used the implementation to read/write files, just remember I am a relative beginner. Also, I need to know how to deserialize the string to be printed out in terminal as a normal string.

XStream has facilities to read from and write to files, see the simple examples (Writer.java and Reader.java) in this article.

If you can serialize it to a txt file, just open an ObjectOutputStream and have it use String's own serialization capability for you.
String str = "serialize me";
String file = "file.txt";
try{
ObjectOutputStream out = new ObjectOutputStream(new FileOutputStream(file));
out.writeObject(str);
out.close();
ObjectInputStream in = new ObjectInputStream(new FileInputStream(file));
String newString = (String) in.readObject();
assert str.equals(newString);
System.out.println("Strings are equal");
}catch(IOException ex){
ex.printStackTrace();
}catch(ClassNotFoundException ex){
ex.printStackTrace();
}
You could also just open a PrintStream and syphon it out that way, then use a BufferedReader and readLine(). If you really want to get fancy (since this is a HW assignment after all), you could use a for loop and print each character individually. Using XML is more complicated than you need to serialize a String and using an external library is just overkill.

If you are beginning Java, then take some time to look through the Apache Commons project. There are lots of basic extensions to java that you will make use of many times.
I'm assuming you just want to persist a string so you can read it back later - in which case it doesn't necessarily need to be XML.
To write a string to a file, see org.apache.commons.io.FileUtils:
FileUtils.writeStringToFile(File file,String data)
To read it back:
FileUtils.readFileToString(File file)
References:
http://commons.apache.org/
http://commons.apache.org/io
http://commons.apache.org/io/api-release/org/apache/commons/io/FileUtils.html
Make sure you also look at commons-lang for lots of good basic stuff.

If you need to create a text file containing XML that represents the contents of an object (and make it bidirectional), just use JSON-lib:
class MyBean{
private String name = "json";
private int pojoId = 1;
private char[] options = new char[]{'a','f'};
private String func1 = "function(i){ return this.options[i]; }";
private JSONFunction func2 = new JSONFunction(new String[]{"i"},"return this.options[i];");
// getters & setters
...
}
JSONObject jsonObject = JSONObject.fromObject( new MyBean() );
String xmlText = XMLSerializer.write( jsonObject );
From there just wrote the String to your file. Much simpler than all those XML API's. Now, however, if you need to conform to a DTD or XSD, this is a bad way to go as it's much more free-format and conforms only to the object layout.
http://json-lib.sourceforge.net/usage.html
Piko

Is there any particular reason to use XStream? This would be extremely easy to do with something like JDOM if all you are doing is trying to serialize a string or two.
Ie, something like:
Document doc = new Document();
Element rootEl = new Element("root");
rootEl.setText("my string");
doc.appendChild(rootEl);
XMLOutputter outputter = new XMLOutputter();
outputter.output(doc);
Some of the details above are probably wrong, but thats the basic flow. Perhaps you should ask a more specific question so that we can understand exactly what problem it is that you are having?

From http://www.xml.com/pub/a/2004/08/18/xstream.html:
import com.thoughtworks.xstream.XStream;
class Date {
int year;
int month;
int day;
}
public class Serialize {
public static void main(String[] args) {
XStream xstream = new XStream();
Date date = new Date();
date.year = 2004;
date.month = 8;
date.day = 15;
xstream.alias("date", Date.class);
String decl = "\n";
String xml = xstream.toXML(date);
System.out.print(decl + xml);
}
}
public class Deserialize {
public static void main(String[] args) {
XStream xstream = new XStream();
Date date = new Date();
xstream.alias("date", Date.class);
String xml = xstream.toXML(date);
System.out.print(xml);
Date newdate = (Date)xstream.fromXML(xml);
newdate.month = 12;
newdate.day = 2;
String newxml = xstream.toXML(newdate);
System.out.print("\n\n" + newxml);
}
}
You can then take the xml string and write it to a file.

try something like this:
FileOutputStream fos = null;
try {
new File(FILE_LOCATION_DIRECTORY).mkdirs();
File fileLocation = new File(FILE_LOCATION_DIRECTORY + "/" + fileName);
fos = new FileOutputStream(fileLocation);
stream.toXML(userAlertSubscription, fos);
} catch (IOException e) {
Log.error(this, "Error %s in file %s", e.getMessage(), fileName);
} finally {
IOUtils.closeQuietly(fos);
}