We Keep Coding

Fatal Error :1:40: Content is not allowed in prolog

I have a super simple XML document encoded in UTF-16 LE.
<?xml version="1.0" encoding="utf-16"?><X id="1" />
I'm loading it in as such (using jcabi-xml):
BOMInputStream bomIn = new BOMInputStream(Main.class.getResourceAsStream("resources/test.xml"), ByteOrderMark.UTF_16LE);
String firstNonBomCharacter = Character.toString((char)bomIn.read());
Reader reader = new InputStreamReader(bomIn, "UTF-16");
String xmlString = IOUtils.toString(reader);
xmlString = xmlString.trim();
xmlString = firstNonBomCharacter + xmlString;
bomIn.close();
reader.close();
final XML xml = new XMLDocument(xmlString);
I have checked that there are no extra BOM/junk symbols (leading or anywhere) by saving out the file and inspecting it with a hex editor. The XML is properly formatted.
However, I still get the following error:
[Fatal Error] :1:40: Content is not allowed in prolog.
Exception in thread "main" java.lang.IllegalArgumentException: Invalid XML: "<?xml version="1.0" encoding="utf-16"?><X id="1" />"
at com.jcabi.xml.DomParser.document(DomParser.java:115)
at com.jcabi.xml.XMLDocument.<init>(XMLDocument.java:155)
at Main.getTransformedString(Main.java:47)
at Main.main(Main.java:26)
Caused by: org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 40; Content is not allowed in prolog.
at com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(Unknown Source)
at com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(Unknown Source)
at javax.xml.parsers.DocumentBuilder.parse(Unknown Source)
at com.jcabi.xml.DomParser.document(DomParser.java:105)
... 3 more
I have googled up and down for this error but they all say that it's the BOM's fault, which I have confirmed (to the best of my knowledge) to not be the case. What else could be wrong?

The following works for me:
try (InputStream stream = Test.class.getResourceAsStream("/Test.xml")) {
StreamSource source = new StreamSource(stream);
final XML xml = new XMLDocument(source);
}
With the input file's hex dump:
FF FE 3C 00 3F 00 78 00 6D 00 6C 00 20 00 76 00 65 00 72 00 73 00 69 00
6F 00 6E 00 3D 00 27 00 31 00 2E 00 30 00 27 00 20 00 65 00 6E 00 63 00
6F 00 64 00 69 00 6E 00 67 00 3D 00 27 00 55 00 54 00 46 00 2D 00 31 00
36 00 27 00 3F 00 3E 00 3C 00 58 00 20 00 69 00 64 00 3D 00 22 00 31 00
22 00 2F 00 3E 00
As far as I can tell, in your example you are converting the contents of the file to a string. But this is problematic because you actually throw away the encoding when you convert bytes to string. When the SAX parser converts the string to a byte array, it decides it will be UTF-8, but the prolog states that it is UTF-16 and so you have a problem.
Instead, when I use the StreamSource, it just automatically detects the fact that the file is encoded in UTF-16 LE from the BOM.
If you are not using java-7 or up and cannot use try-with-resources, then use the stream.close() as before.

Java serialization with empty and substrings

Had a look at the implementation and haven't been able to think of an explanation to this but maybe someone here will know.
public static void main(String[] args) throws Exception {
List<String> emptyStrings = new ArrayList<String>();
List<String> emptySubStrings = new ArrayList<String>();
for (int i = 0; i < 20000; i++) {
String actuallyEmpty = "";
String subStringedEmpty = " ";
subStringedEmpty = subStringedEmpty.substring(0, 0);
emptyStrings.add(actuallyEmpty);
emptySubStrings.add(subStringedEmpty);
}
System.out.println("Substring test");
// Write to files
long time = System.currentTimeMillis();
writeObjectToFile(emptyStrings, "empty.list");
System.out.println("Time taken to write empty list " + (System.currentTimeMillis() - time));
time = System.currentTimeMillis();
writeObjectToFile(emptySubStrings, "substring.list");
System.out.println("Time taken to write substring list " + (System.currentTimeMillis() - time));
//Read from files
time = System.currentTimeMillis();
List<String> readEmptyString = readObjectFromFile("empty.list");
System.out.println("Time taken to read empty list " + (System.currentTimeMillis() - time));
time = System.currentTimeMillis();
List<String> readEmptySubStrings = readObjectFromFile("substring.list");
System.out.println("Time taken to read substring list " + (System.currentTimeMillis() - time));
}
private static void writeObjectToFile(Object o, String file) throws Exception {
FileOutputStream out = new FileOutputStream(file);
ObjectOutputStream oout = new ObjectOutputStream(out);
oout.writeObject(o);
oout.flush();
oout.close();
}
private static <T> T readObjectFromFile(String file) throws Exception {
ObjectInputStream ois = null;
try {
ois = new ObjectInputStream(new FileInputStream(file));
return (T) ois.readObject();
} finally {
ois.close();
}
}
Ultimately these 2 lists contain 20,000 empty strings (one list contains "" empty strings and the other contains empty strings generated by substring(0,0)). But if you check the sizes of the serialized files generated (empty.list and substring.list) you will notice that the empty.list contains substantially more data.
I have noticed that the callers of remote EJB's which un-serialize these substring objects seem to have severe performance issues also.

The sizes of the lists are different because java uses a mechanism to store multiples references to the same object, like described:
References to other objects (except in transient or static fields)
cause those objects to be written also. Multiple references to a
single object are encoded using a reference sharing mechanism so that
graphs of objects can be restored to the same shape as when the
original was written.
see ObjectOutputStream
If you look the generated serialized file, you will see:
With 1 String empty inside:
empty.list:
ac ed 00 05 73 72 00 13 6a 61 76 61 2e 75 74 69
6c 2e 41 72 72 61 79 4c 69 73 74 78 81 d2 1d 99
c7 61 9d 03 00 01 49 00 04 73 69 7a 65 78 70 00
00 00 01 77 04 00 00 00 01 74 00 00 78
The string "" corresponds to the last three bytes (00 00 78)
substring.list
ac ed 00 05 73 72 00 13 6a 61 76 61 2e 75 74 69
6c 2e 41 72 72 61 79 4c 69 73 74 78 81 d2 1d 99
c7 61 9d 03 00 01 49 00 04 73 69 7a 65 78 70 00
00 00 01 77 04 00 00 00 01 74 00 00 78
Note that with one element the resulted file is the same.
But if we want to add more times the same object, we will be faced with other behavior.
Look the respective files with 2 times that string.
empty.list:
ac ed 00 05 73 72 00 13 6a 61 76 61 2e 75 74 69
6c 2e 41 72 72 61 79 4c 69 73 74 78 81 d2 1d 99
c7 61 9d 03 00 01 49 00 04 73 69 7a 65 78 70 00
00 00 02 77 04 00 00 00 02 74 00 00 71 00 7e 00
02 78
substring.list
ac ed 00 05 73 72 00 13 6a 61 76 61 2e 75 74 69
6c 2e 41 72 72 61 79 4c 69 73 74 78 81 d2 1d 99
c7 61 9d 03 00 01 49 00 04 73 69 7a 65 78 70 00
00 00 02 77 04 00 00 00 02 74 00 00 74 00 00 78
Note that substring continues "normal", two non related strings with different references. But empty has some extra bytes to handle the issue of same reference.
Six bytes from substring (00 00 74 00 00 78) versus eight bytes from emptylist (00 00 71 00 7e 00 02 78)
This goes wrong because every repeated string that you add, more extra bytes are added. So when you full your arrayList there will be a lot of extra bytes to make it possible to reconstruct as it's original way.
If you want to know why there is that sharing mechanism, I suggest you to take a look at this question:
What is the meaning of reference sharing in Serialization? How Enums are Serialized?

empty.list contains one String object and lots of references to it.
substring.list contains 2000 string objects, all of them are equal in content.
You could "fix" this by intern()ing the strings.
private void verify(String name, Supplier<String> stringSupplier) throws IOException, ClassNotFoundException {
List<String> inputStrings = new ArrayList<String>();
inputStrings.add(stringSupplier.get());
inputStrings.add(stringSupplier.get());
ByteArrayOutputStream boas = new ByteArrayOutputStream();
ObjectOutputStream emptyOut = new ObjectOutputStream(boas);
emptyOut.writeObject(inputStrings);
emptyOut.flush();
ObjectInputStream ois = new ObjectInputStream(new ByteArrayInputStream(boas.toByteArray()));
List<String> returnedStrings = (List<String>)ois.readObject();
if(returnedStrings.get(0) == returnedStrings.get(1)) {
System.out.println(name + " contains the same object");
} else {
System.out.println(name + " contains DIFFERENT objects");
}
}
#Test
public void test() throws IOException, ClassNotFoundException {
verify("empty string", new Supplier<String>() {
#Override
public String get() {
return "";
}
});
verify("sub string", new Supplier<String>() {
#Override
public String get() {
String data = " ";
return data.substring(0, 0);
}
});
verify("intern()ed substring", new Supplier<String>() {
#Override
public String get() {
String data = " ";
return data.substring(0, 0).intern();
}
});
}

Converting monochrome bitmap data into a little endian order hex string for Zebra printers

I have a 1bpp monochrome bitmap image loaded into memory. I have the image data loaded into a byte array. I am trying to print to a Zebra printer and following their manual I feel I mostly have the code working except that when I convert the image data into hexcode and send the command to the printer, the bounding box appears to print the correct size, but the image is hopelessly scrambled.
I suspect that this is because the following method that I found on StackOverflow for converting a byte array into hex is reading each byte value in Big Endian order when I need it to look at it in Little Endian order.
public static String bytesToHex(byte[] b) {
char hexDigit[] = {'0', '1', '2', '3', '4', '5', '6', '7',
'8', '9', 'a', 'b', 'c', 'd', 'e', 'f'};
StringBuffer buf = new StringBuffer();
for (int j=0; j<b.length; j++) {
buf.append(hexDigit[b[j] & 0x0f]);
buf.append(hexDigit[(b[j] >> 4) & 0x0f]);
}
return buf.toString().toUpperCase();
}
At this point I am stuck. Does anybody have any suggestions how I can modify the above the code to read each byte value in the proper order for reading bitmap image data?
EDIT:
The image, it is James Bond mangled with a poor ad-hoc Floyd-Steinberg dithering implementation (thats a different problem altogether)...
EDIT 2:
Well that didn't work, it looks like when I upload it then it gets converted to PNG.
Bitmap data in Hex:
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 01 FF FF FF 00 00 00 FF FF FF 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 01 FF FF FF 00 00 00 FF FF FE 00 00 ...
Hmmmm... I thought Monochrome 1bpp bitmaps only had two possible colors? Why did image4j ConvertUtil.convert1 introduce other colors? I wonder if this is the problem?
EDIT 3:
NVM, each bit determines the color in 1bpp, so each byte can represent up to 8 pixels. This is likely correct.

servlet request parameter character encoding

I have a Java servlet that receives data from an upstream system via a HTTP GET request. This request includes a parameter named "text". If the upstream system sets this parameter to:
TEST3 please ignore:
It appears in the logs of the upstream system as:
00 54 00 45 00 53 00 54 00 33 00 20 00 70 00 6c //TEST3 pl
00 65 00 61 00 73 00 65 00 20 00 69 00 67 00 6e //ease ign
00 6f 00 72 00 65 00 3a //ore:
(The // comments do not actually appear in the logs)
In my servlet I read this parameter with:
String text = request.getParameter("text");
If I print the value of text to the console, it appears as:
T E S T 3 p l e a s e i g n o r e :
If I inspect the value of text in the debugger, it appears as:
\u000T\u000E\u000S\u000T\u0003\u0000 \u000p\u000l\u000e\u000a\u000s\u000e\u0000
\u000i\u000g\u000n\u000o\u000r\u000e\u000:
So it seems that there's a problem with the character encoding. The upstream system is supposed to use UTF-16. My guess is that the servlet is assuming UTF-8 and therefore is reading twice the number of characters it should be. For the message "TEST3 please ignore:" the first byte of each character is 00. This is being interpreted as a space when read by the servlet, which explains the space that appears before each character when the message is logged by the servlet.
Obviously my goal is simply to get the message "TEST3 please ignore:" when I read the text request param. My guess is that I could achieve this by specifying the character encoding of the request parameter, but I don't know how to do this.

Use like this
new String(req.getParameter("<my request value>").getBytes("ISO-8859-1"),"UTF-8")

Try to use Filter for this
public class CustomCharacterEncodingFilter implements Filter {
public void init(FilterConfig config) throws ServletException {
}
public void doFilter(ServletRequest request, ServletResponse response, FilterChain chain)
throws IOException, ServletException {
request.setCharacterEncoding("UTF-8");
response.setCharacterEncoding("UTF-8");
chain.doFilter(request, response);
}
public void destroy() {
}
This should set encoding right for whole application

Looks like it was encoded with UTF-16LE (Little Endian) encoding, here is a class that successfully prints your string:
import java.io.UnsupportedEncodingException;
import java.math.BigInteger;
public class Test {
public static void main(String[] args) throws UnsupportedEncodingException {
String hex = "00 54 00 45 00 53 00 54 00 33 00 20 00 70 00 6c" +
"00 65 00 61 00 73 00 65 00 20 00 69 00 67 00 6e" +
"00 6f 00 72 00 65 00 3a"; // + " 00";
System.out.println(new String(new BigInteger(hex.replaceAll(" ", ""), 16).toByteArray(), "UTF-16LE"));
}
}
Output:
TEST3 please ignore?
Output with two zero's added to the input
TEST3 please ignore:
UPDATE
To get this working with your Servlet you can try:
String value = request.getParameter("text");
try {
value = new String(value.getBytes(), "UTF-16LE");
} catch(java.io.UnsupportedEncodingException ex) {}
UPDATE
see the following link, it verifies that the hex produced is in fact UTF-16LE

H.323, How to make a simple ring without media. This script was following Q.931 setup but still not working

Can anyone please help me to solve this? When i send this request i have seen in wireshark that packets are going to SJPhone in 1720 tcp port. But still SJPhone does not ring. I want to make it ring (no matter for media).
I would really appreciate your support. I must be missing the message protocol details to implement this. Please show me some positive pointers.
FYI: i have used this trace: http://www.vconsole.com/usermanuals/sample_isdn_trace.pdf
import java.io.*;
import java.net.*;
public class test
{
public static void main(String[] args) throws UnknownHostException, IOException
{
/* Step 1: simulate the Q.931 packets exchange */
byte st[]=new byte[256];
st[0]=0x08; // protocol discriminator
st[1]=0x02; // length (bytes) of call reference
st[2]=0x02; // call reference (1-15 bytes)
// message type
st[3]=0x05;
// information Elements
st[4]=0x6C; // calling party number
st[5]=0; // unknown
st[6]=0; // unknown
st[7]=1; // "1"
// information elements
st[8]=0x70; // called party number
st[9]=0; // unknown
st[10]=0; // unknown
st[11]=5; // "5"
System.out.println(st);
/* Step 2: by-pass it for testing with tcpdump */
Socket clientSocket = new Socket("localhost", 1720);
DataInputStream input = new DataInputStream(clientSocket.getInputStream());
DataOutputStream outToServer = new DataOutputStream(clientSocket.getOutputStream());
BufferedReader inFromServer = new BufferedReader(new InputStreamReader(clientSocket.getInputStream()));
outToServer.write(st);
String get;
get = inFromServer.readLine();
System.out.println("FROM SERVER: " + get);
clientSocket.close();
}
}
--More info:
-When SJPhone to SJPhone communicate i see this logs:
0000 00 00 03 04 00 06 00 00 00 00 00 00 00 00 08 00 ................
0010 45 00 00 3c 88 6c 40 00 40 06 b4 4d 7f 00 00 01 E..<.l#.#..M....
0020 7f 00 00 01 b4 a8 06 b8 56 f6 c4 b3 00 00 00 00 ........V.......
0030 a0 02 80 18 fe 30 00 00 02 04 40 0c 04 02 08 0a .....0....#.....
0040 02 a1 4c df 00 00 00 00 01 03 03 06 ..L.........
-When this test.java communicate to SJPhone i see this logs:
0000 00 00 03 04 00 06 00 00 00 00 00 00 00 00 08 00 ........ ........
0010 45 00 00 3c 1f ba 40 00 40 06 1d 00 7f 00 00 01 E..<..#. #.......
0020 7f 00 00 01 d6 ca 06 b8 8c e7 41 15 00 00 00 00 ........ ..A.....
0030 a0 02 80 18 fe 30 00 00 02 04 40 0c 04 02 08 0a .....0.. ..#.....
0040 02 a6 5e af 00 00 00 00 01 03 03 06 ..^..... ....
Note:
Friendly dump can be done using this command, to see realtime + save what you have seen: tcpdump -XX -s 0 -i lo | tee /tmp/log.log

So, the question was only to send a ring, without an actual call.
Here is some code that does the job. It was a lot of fun to see when it started working:
import java.io.DataInputStream;
import java.io.DataOutputStream;
import java.io.IOException;
import java.io.OutputStream;
import java.net.Socket;
/**
*
* #author martijncourteaux
*/
public class SJPhoneRinger
{
private static String setupOpenLogicChannel = "03|00|02|32|08|02|10|00|05|04|03|88|c0|a5|28|07|75|6e|6b|6e|6f|77|6e|6c|05|81|32|30|30|35|7e|02|11|05|20|88|06|00|08|91|4a|00|04|22|c0|00|00|00|00|0f|53|4a|20|4c|61|62|73|ae|20|53|4a|70|68|6f|6e|65|08|31|2e|36|30|2e|32|39|39|61|00|7f|00|00|01|06|b8|00|c4|a1|56|3c|d2|1d|b2|11|a8|ed|b7|8a|c2|c1|98|8f|00|c1|1d|80|04|11|00|f6|a1|56|3c|d2|1d|b2|11|a8|ed|b7|8a|c2|c1|98|8f|81|3a|0a|21|40|00|00|06|04|01|00|4e|0c|03|00|20|00|80|12|1e|00|01|00|7f|00|00|01|c0|04|00|7f|00|00|01|c0|05|00|2b|40|00|00|06|04|01|00|4c|10|00|00|00|00|09|69|4c|42|43|2d|31|33|6b|33|80|12|1e|40|01|00|7f|00|00|01|c0|04|00|7f|00|00|01|c0|05|04|2b|40|00|00|06|04|01|00|4c|10|00|00|00|00|09|69|4c|42|43|2d|31|35|6b|32|80|12|1e|40|01|00|7f|00|00|01|c0|04|00|7f|00|00|01|c0|05|08|1e|40|00|00|06|04|01|00|4c|20|13|80|12|1e|00|01|00|7f|00|00|01|c0|04|00|7f|00|00|01|c0|05|00|1e|40|00|00|06|04|01|00|4c|60|13|80|12|1e|00|01|00|7f|00|00|01|c0|04|00|7f|00|00|01|c0|05|00|16|00|00|0d|0e|0c|03|00|20|00|80|0b|06|00|01|00|7f|00|00|01|c0|05|00|20|00|00|0e|0c|10|00|00|00|00|09|69|4c|42|43|2d|31|33|6b|33|80|0b|06|40|01|00|7f|00|00|01|c0|05|04|20|00|00|0f|0c|10|00|00|00|00|09|69|4c|42|43|2d|31|35|6b|32|80|0b|06|40|01|00|7f|00|00|01|c0|05|08|13|00|00|10|0c|20|13|80|0b|06|00|01|00|7f|00|00|01|c0|05|00|13|00|00|11|0c|60|13|80|0b|06|00|01|00|7f|00|00|01|c0|05|00|01|00|01|00|01|00|01|00|6e|02|64|02|70|01|06|00|08|81|75|00|08|80|0d|00|00|3c|00|01|00|00|01|00|00|01|00|00|04|80|00|00|24|18|03|00|20|00|80|00|01|20|20|00|00|00|00|09|69|4c|42|43|2d|31|33|6b|33|80|00|02|20|20|00|00|00|00|09|69|4c|42|43|2d|31|35|6b|32|80|00|03|20|40|13|80|00|04|20|c0|13|00|80|06|00|04|00|00|00|01|00|02|00|03|00|04|07|01|00|32|80|d8|50|5f|02|80|01|80";
private static String alertingTSCA = "03|00|00|d8|08|02|90|00|01|28|07|75|6e|6b|6e|6f|77|6e|7e|00|c3|05|23|c0|06|00|08|91|4a|00|04|22|c0|00|00|00|00|0f|53|4a|20|4c|61|62|73|ae|20|53|4a|70|68|6f|6e|65|08|31|2e|36|30|2e|32|38|39|61|00|5e|e0|cc|44|98|ff|0d|0c|11|00|f6|a1|56|3c|d2|1d|b2|11|a8|ed|b7|8a|c2|c1|98|8f|01|00|01|00|04|c0|01|80|74|04|03|21|80|01|64|02|70|01|06|00|08|81|75|00|08|80|0d|00|00|3c|00|01|00|00|01|00|00|01|00|00|04|80|00|00|24|18|03|00|20|00|80|00|01|20|20|00|00|00|00|09|69|4c|42|43|2d|31|33|6b|33|80|00|02|20|20|00|00|00|00|09|69|4c|42|43|2d|31|35|6b|32|80|00|03|20|40|13|80|00|04|20|c0|13|00|80|06|00|04|00|00|00|01|00|02|00|03|00|04|06|01|00|32|40|1b|33|02|20|80";
private static String facilityTSCA = "03|00|00|62|08|02|10|00|62|1c|00|28|07|75|6e|6b|6e|6f|77|6e|7e|00|4b|05|26|90|06|00|08|91|4a|00|04|c4|a1|56|3c|d2|1d|b2|11|a8|ed|b7|8a|c2|c1|98|8f|86|01|00|13|05|80|11|00|f6|a1|56|3c|d2|1d|b2|11|a8|ed|b7|8a|c2|c1|98|8f|07|00|5e|e0|cc|44|ab|62|01|00|01|00|04|c0|01|80|08|02|03|21|80|01|02|20|a0";
private static String connectOpenLogicChannel = "03|00|00|b6|08|02|90|00|07|28|07|75|6e|6b|6e|6f|77|6e|7e|00|a1|05|22|c0|06|00|08|91|4a|00|04|00|5e|e0|cc|44|98|ff|22|c0|00|00|00|00|0f|53|4a|20|4c|61|62|73|ae|20|53|4a|70|68|6f|6e|65|08|31|2e|36|30|2e|32|38|39|61|00|c4|a1|56|3c|d2|1d|b2|11|a8|ed|b7|8a|c2|c1|98|8f|0d|1c|11|00|f6|a1|56|3c|d2|1d|b2|11|a8|ed|b7|8a|c2|c1|98|8f|41|02|21|40|00|03|06|04|01|00|4e|0c|03|00|20|00|80|12|1e|00|01|00|7f|00|00|01|c0|04|00|5e|e0|cc|44|c0|07|00|1d|00|00|0d|0e|0c|03|00|20|00|80|12|1e|00|01|00|5e|e0|cc|44|c0|06|00|5e|e0|cc|44|c0|07|00|01|00|01|00|02|80|01|80";
/**
* #param args the command line arguments
*/
public static void main(String[] args) throws IOException
{
Socket socket = new Socket("localhost", 1720);
System.out.println("Connected to SJPhone!");
DataOutputStream dos = new DataOutputStream(socket.getOutputStream());
DataInputStream dis = new DataInputStream(socket.getInputStream());
sendBytes(dos, setupOpenLogicChannel);
/* Following three packages were not needed. It worked without them.
* Of course, because it is only a sample ring request, SJPhone will tell you
* That the call is corrupted */
// sendBytes(dos, alertingTSCA);
// sendBytes(dos, facilityTSCA);
// sendBytes(dos, connectOpenLogicChannel);
try
{
Thread.sleep(10000); // Let it ring for ten seconds.
} catch (Exception e)
{
}
}
public static void sendBytes(OutputStream os, String bytes) throws IOException
{
String byteArrayStr[] = bytes.split("\\|");
byte bytesArray[] = new byte[byteArrayStr.length];
for (int i = 0; i < byteArrayStr.length; ++i)
{
bytesArray[i] = (byte) (Integer.parseInt(byteArrayStr[i],16));
}
os.write(bytesArray);
os.flush();
}
}