We Keep Coding

Offline XML Validation with Java

I need to figure out how to validate my XML files with schema's offline. After looking around for a couple of days, what I was able to find was basically that I needed to have an internal reference to the schema. I needed to find them, download them, and change the reference to a local system path. What I was unable to find was exactly how to do that. Where and how can I change the reference to point internally instead of externally? What is the best way to download the schemas?

There are three ways you could do this. What they all have in common is that you need a local copy of the schema document(s). I'm assuming that the instance documents currently use xsi:schemaLocation and/or xsi:noNamespaceSchemaLocation to point to a location holding the schema document(s) on the web.
(a) Modify your instance documents to refer to the local copy of the schema documents. This is usually inconvenient.
(b) Redirect the references so that a request for a remote file is redirected to a local file. The way to set this up depends on which schema validator you are using and how you are invoking it.
(c) Tell the schema processor to ignore the values of xsi:schemaLocation and xsi:noNamespaceSchemaLocation, and to validate instead against a schema that you supply using your schema processor's invocation API. Again the details depend on which schema processor you are using.
My preferred approach is (c): if only because when you are validating a source document, then by definition you don't fully trust it - so why should you trust it to contain a correct xsi:schemaLocation attribute?

XmlValidate is a simple but powerful command-line tool that can perform offline validation of single or multiple XML files against target schemas. It can scan local xml files by file name, directory, or URL.
XmlValidate automatically adds the schemaLocation based on the schema namespace and a config file that mapping to a local file. The tool will validate against whatever XML Schema is referenced in the config file.
Here are example mappings of namespace to target Schema in config file:
http://www.opengis.net/kml/2.2=${XV_HOME}/schemas/kml22.xsd
http://appengine.google.com/ns/1.0=C:/xml/appengine-web.xsd
urn:oasis:names:tc:ciq:xsdschema:xAL:2.0=C:/xml/xAL.xsd
Note that ${XV_HOME} token above is simply an alias for the top-level directory that XmlValidate is running from. The location can likewise be a full file path.
XmlValidate is an open-source project (source code available) that runs with the Java Runtime Environment (JRE). The bundled application (Java jars, examples, etc.) can be downloaded here.
If XmlValidate is run in batch mode against multiple XML files, it will provide a summary of validation results.
Errors: 17 Warnings: 0 Files: 11 Time: 1506 ms
Valid files 8/11 (73%)

You can set your own Implementation of ResourceResolver and LSInput to the SchemaFactory so that the call of
of LSInput.getCharacterStream() will provide a schema from a local path.
I have written an extra class to do offline validation. You can call it like
new XmlSchemaValidator().validate(xmlStream, schemaStream, "https://schema.datacite.org/meta/kernel-4.1/",
"schemas/datacite/kernel-4.1/");
Two InputStream are beeing passed. One for the xml, one for the schema. A baseUrl and a localPath (relative on classpath) are passed as third and fourth parameter. The last two parameters are used by the validator to lookup additional schemas locally at localPath or relative to the provided baseUrl.
I have tested with a set of schemas and examples from https://schema.datacite.org/meta/kernel-4.1/ .
Complete Example:
#Test
public void validate4() throws Exception {
InputStream xmlStream = Thread.currentThread().getContextClassLoader().getResourceAsStream(
"schemas/datacite/kernel-4.1/example/datacite-example-complicated-v4.1.xml");
InputStream schemaStream = Thread.currentThread().getContextClassLoader()
.getResourceAsStream("schemas/datacite/kernel-4.1/metadata.xsd");
new XmlSchemaValidator().validate(xmlStream, schemaStream, "https://schema.datacite.org/meta/kernel-4.1/",
"schemas/datacite/kernel-4.1/");
}
The XmlSchemaValidator will validate the xml against the schema and will search locally for included Schemas. It uses a ResourceResolver to override the standard behaviour and to search locally.
public class XmlSchemaValidator {
/**
* #param xmlStream
* xml data as a stream
* #param schemaStream
* schema as a stream
* #param baseUri
* to search for relative pathes on the web
* #param localPath
* to search for schemas on a local directory
* #throws SAXException
* if validation fails
* #throws IOException
* not further specified
*/
public void validate(InputStream xmlStream, InputStream schemaStream, String baseUri, String localPath)
throws SAXException, IOException {
Source xmlFile = new StreamSource(xmlStream);
SchemaFactory factory = SchemaFactory.newInstance(XMLConstants.W3C_XML_SCHEMA_NS_URI);
factory.setResourceResolver((type, namespaceURI, publicId, systemId, baseURI) -> {
LSInput input = new DOMInputImpl();
input.setPublicId(publicId);
input.setSystemId(systemId);
input.setBaseURI(baseUri);
input.setCharacterStream(new InputStreamReader(
getSchemaAsStream(input.getSystemId(), input.getBaseURI(), localPath)));
return input;
});
Schema schema = factory.newSchema(new StreamSource(schemaStream));
javax.xml.validation.Validator validator = schema.newValidator();
validator.validate(xmlFile);
}
private InputStream getSchemaAsStream(String systemId, String baseUri, String localPath) {
InputStream in = getSchemaFromClasspath(systemId, localPath);
// You could just return in; , if you are sure that everything is on
// your machine. Here I call getSchemaFromWeb as last resort.
return in == null ? getSchemaFromWeb(baseUri, systemId) : in;
}
private InputStream getSchemaFromClasspath(String systemId, String localPath) {
System.out.println("Try to get stuff from localdir: " + localPath + systemId);
return Thread.currentThread().getContextClassLoader().getResourceAsStream(localPath + systemId);
}
/*
* You can leave out the webstuff if you are sure that everything is
* available on your machine
*/
private InputStream getSchemaFromWeb(String baseUri, String systemId) {
try {
URI uri = new URI(systemId);
if (uri.isAbsolute()) {
System.out.println("Get stuff from web: " + systemId);
return urlToInputStream(uri.toURL(), "text/xml");
}
System.out.println("Get stuff from web: Host: " + baseUri + " Path: " + systemId);
return getSchemaRelativeToBaseUri(baseUri, systemId);
} catch (Exception e) {
// maybe the systemId is not a valid URI or
// the web has nothing to offer under this address
}
return null;
}
private InputStream urlToInputStream(URL url, String accept) {
HttpURLConnection con = null;
InputStream inputStream = null;
try {
con = (HttpURLConnection) url.openConnection();
con.setConnectTimeout(15000);
con.setRequestProperty("User-Agent", "Name of my application.");
con.setReadTimeout(15000);
con.setRequestProperty("Accept", accept);
con.connect();
int responseCode = con.getResponseCode();
if (responseCode == HttpURLConnection.HTTP_MOVED_PERM
|| responseCode == HttpURLConnection.HTTP_MOVED_TEMP || responseCode == 307
|| responseCode == 303) {
String redirectUrl = con.getHeaderField("Location");
try {
URL newUrl = new URL(redirectUrl);
return urlToInputStream(newUrl, accept);
} catch (MalformedURLException e) {
URL newUrl = new URL(url.getProtocol() + "://" + url.getHost() + redirectUrl);
return urlToInputStream(newUrl, accept);
}
}
inputStream = con.getInputStream();
return inputStream;
} catch (SocketTimeoutException e) {
throw new RuntimeException(e);
} catch (IOException e) {
throw new RuntimeException(e);
}
}
private InputStream getSchemaRelativeToBaseUri(String baseUri, String systemId) {
try {
URL url = new URL(baseUri + systemId);
return urlToInputStream(url, "text/xml");
} catch (Exception e) {
e.printStackTrace();
throw new RuntimeException(e);
}
}
}
prints
Try to get stuff from localdir: schemas/datacite/kernel-4.1/http://www.w3.org/2009/01/xml.xsd
Get stuff from web: http://www.w3.org/2009/01/xml.xsd
Try to get stuff from localdir: schemas/datacite/kernel-4.1/include/datacite-titleType-v4.xsd
Try to get stuff from localdir: schemas/datacite/kernel-4.1/include/datacite-contributorType-v4.xsd
Try to get stuff from localdir: schemas/datacite/kernel-4.1/include/datacite-dateType-v4.1.xsd
Try to get stuff from localdir: schemas/datacite/kernel-4.1/include/datacite-resourceType-v4.1.xsd
Try to get stuff from localdir: schemas/datacite/kernel-4.1/include/datacite-relationType-v4.1.xsd
Try to get stuff from localdir: schemas/datacite/kernel-4.1/include/datacite-relatedIdentifierType-v4.xsd
Try to get stuff from localdir: schemas/datacite/kernel-4.1/include/datacite-funderIdentifierType-v4.xsd
Try to get stuff from localdir: schemas/datacite/kernel-4.1/include/datacite-descriptionType-v4.xsd
Try to get stuff from localdir: schemas/datacite/kernel-4.1/include/datacite-nameType-v4.1.xsd
The print shows that the validator was able to validate against a set of local schemas. Only http://www.w3.org/2009/01/xml.xsd was not available locally and therefore fetched from the internet.

How do I send nested json POST request in java using jersey?

I am using a document converter api called cloudconvert. They don't have an official java library, but a third party java option. I needed a little customization so I cloned the github project and added it to my project. I am sending cloudconvert a .epub file and getting a .pdf file in return. If I use the default settings it works without issue and properly converts my .epub to a .pdf. Here is the code that makes it happen.
Here is what triggers the conversion:
// Create service object
CloudConvertService service = new CloudConvertService("api-key");
// Create conversion process
ConvertProcess process = service.startProcess(convertFrom, convertTo);
// Perform conversion
//convertFromFile is a File object with a .epub extension
process.startConversion(convertFromFile);
// Wait for result
ProcessStatus status;
waitLoop:
while (true) {
status = process.getStatus();
switch (status.step) {
case FINISHED:
break waitLoop;
case ERROR:
throw new RuntimeException(status.message);
}
// Be gentle
Thread.sleep(200);
}
//Download result
service.download(status.output.url, convertToFile);
//lean up
process.delete();
startConversion() calls:
public void startConversion(File file) throws ParseException, FileNotFoundException, IOException {
if (!file.exists()) {
throw new FileNotFoundException("File not found: " + file);
}
startConversion(new FileDataBodyPart("file", file));
}
Which calls this to actually send the POST request using jersey:
private void startConversion(BodyPart bodyPart) {
if (args == null) {
throw new IllegalStateException("No conversion arguments set.");
}
MultiPart multipart = new FormDataMultiPart()
.field("input", "upload")
.field("outputformat", args.outputformat)
.bodyPart(bodyPart);
//root is a class level WebTarget object
root.request(MediaType.APPLICATION_JSON).post(Entity.entity(multipart, multipart.getMediaType()));
}
Up to this point everything is working. My problem is that the when the conversion happens the .pdf that returns has very small margins. cloudconvert provides a way to change those margins. You can send in an optional json param converteroptions and set the margins manually. I have tested this out using postman and it works without issue, I was able to get a properly formatted margin document. So know this is possible. Here is the POSTMAN info I used:
#POST : https://host123d1qo.cloudconvert.com/process/WDK9Yq0z1xso6ETgvpVQ
Headers: 'Content-Type' : 'application/json'
Body:
{
"input": "base64",
"file": "0AwAAIhMAAAAA", //base64 file string that is much longer than this
"outputformat": "pdf",
"converteroptions": {
"margin_bottom": 75,
"margin_top": 75,
"margin_right": 50,
"margin_left": 50
}
}
Here are my attempts at getting the POST request formatted properly, I'm just not very experienced with jersey and the couple of answers I did find on stackoverflow didn't work for me.
Attempt 1, I tried adding the json string as a Multipart.field. It didn't give me any errors and still returned a converted .pdf file, but the margins didn't get changed so I must not be sending it back right.
private void startConversion(BodyPart bodyPart) {
String jsonString = "{\"margin_bottom\":75,\"margin_top\":75,\"margin_right\":50,\"margin_left\":50}";
MultiPart multipart = new FormDataMultiPart()
.field("input", "upload")
.field("outputformat", args.outputformat)
.field("converteroptions", jsonString)
.bodyPart(bodyPart);
root.request(MediaType.APPLICATION_JSON).post(Entity.entity(multipart, multipart.getMediaType()));
}
Attempt 2, when I had it working in POSTMAN it was using the 'input' type as 'base64' so I tried changing it to that but it this time it doesn't return anything at all, no request errors, just a timeout error at the 5 minute mark.
//I pass in a File object rather than the bodypart object.
private void startConversion(File file) {
byte[] encoded1 = Base64.getEncoder().encode(FileUtils.readFileToByteArray(file));
String encoded64 = new String(encoded1, StandardCharsets.US_ASCII);
String jsonString = "{\"margin_bottom\":75,\"margin_top\":75,\"margin_right\":50,\"margin_left\":50}";
MultiPart multipart = new FormDataMultiPart()
.field("input", "base64")
.field("outputformat", args.outputformat)
.field("file", encoded64)
.field("converteroptions", jsonString);
root.request(MediaType.APPLICATION_JSON).post(Entity.entity(multipart, multipart.getMediaType()));
}
Attempt 3, after some googling on how to properly send jersey json post requests I changed the format. This time it returned a 400 bad request error.
private void startConversionPDF(File file) throws IOException {
byte[] encoded1 = Base64.getEncoder().encode(FileUtils.readFileToByteArray(file));
String encoded64 = new String(encoded1, StandardCharsets.US_ASCII);
String jsonString = "{\"input\":\"base64\",\"file\":\"" + encoded64 + "\",\"outputformat\":\"pdf\",\"converteroptions\":{\"margin_bottom\":75,\"margin_top\":75,\"margin_right\":50,\"margin_left\":50}}";
root.request(MediaType.APPLICATION_JSON).post(Entity.json(jsonString));
}
Attempt 4, Someone said you don't need to manually use a jsonString you should use serializable java beans. So I created the corresponding classes and made the request like shown below. Same 400 bad request error.
#XmlRootElement
public class PDFConvert implements Serializable {
private String input;
private String file;
private String outputformat;
private ConverterOptions converteroptions;
//with the a default constructor and getters/setters for all
}
#XmlRootElement
public class ConverterOptions implements Serializable {
private int margin_bottom;
private int margin_top;
private int margin_left;
private int margin_right;
//with the a default constructor and getters/setters for all
}
private void startConversionPDF(File file) throws IOException {
byte[] encoded1 = Base64.getEncoder().encode(FileUtils.readFileToByteArray(file));
String encoded64 = new String(encoded1, StandardCharsets.US_ASCII);
PDFConvert data = new PDFConvert();
data.setInput("base64");
data.setFile(encoded64);
data.setOutputformat("pdf");
ConverterOptions converteroptions = new ConverterOptions();
converteroptions.setMargin_top(75);
converteroptions.setMargin_bottom(75);
converteroptions.setMargin_left(50);
converteroptions.setMargin_right(50);
data.setConverteroptions(converteroptions);
root.request(MediaType.APPLICATION_JSON).post(Entity.json(data));
}
I know this is quite the wall of text, but I wanted to show all the different things I tried so that I wouldn't waste anyone's time. Thank you for any help or ideas you might have to make this work. I really want to make it work with jersey because I have several other conversions I do that work perfectly, they just don't need any converteroptions. Also I know its possible because it works when manually running the process through POSTMAN.
Cloudconvert api documentation for starting a conversion
Github repo with the recommended 3rd party java library I am using/modifying

I finally figured it out. Hours of trial and error. Here is the code that did it:
private void startConversionPDF(File file) throws IOException {
if (args == null) {
throw new IllegalStateException("No conversion arguments set.");
}
PDFConvert data = new PDFConvert();
data.setInput("upload");
data.setOutputformat("pdf");
ConverterOptions converteroptions = new ConverterOptions();
converteroptions.setMargin_top(60);
converteroptions.setMargin_bottom(60);
converteroptions.setMargin_left(30);
converteroptions.setMargin_right(30);
data.setConverteroptions(converteroptions);
MultiPart multipart = new FormDataMultiPart()
.bodyPart(new FormDataBodyPart("json", data, MediaType.APPLICATION_JSON_TYPE))
.bodyPart(new FileDataBodyPart("file", file));
root.request(MediaType.APPLICATION_JSON).post(Entity.entity(multipart, multipart.getMediaType()));
}

change (or set ) encoding web service

I have a web service using JAX-WS API (a soap web service).
import javax.jws.WebService;
#WebService
public class Accounts {
public String getFullName(String userID){
AccountManager GFN = new AccountManager();
return GFN.getFullName(userID);
}
what can I do to change it's encoding to "UTF-8" (for non-English character)?
I fount something like "#Produces("text/html; charset=UTF-8")" but it is for JAX-RS (restful web service) i need something for JAX-WS.
Thanks.

To start, you need to get a hold of the SOAP message as early as possible. The best place to start is a javax.xml.ws.handler.LogicalHandler (as opposed to the more common type of handler, SOAPHandler).
The LogicalHandler intercepts the SOAP payload before the SOAPHandler, so it's the ideal place to do this
Within this handler, you have the liberty to whatever you want with the message, long before the encoding becomes a problem. Your code should look something like this
public class YourHandler implements LogicalHandler{
public boolean handleMessage(LogicalMessageContext context) {
boolean inboundProperty= (boolean)context.get(MessageContext.MESSAGE_INBOUND_PROPERTY);
if (inboundProperty) {
LogicalMessage lm = context.getMessage();
Source payload = lm.getPayload();
Source recodedPayload = modifyEncoding(payload); //This is where you change the encoding. We'll talk more about this
lm.setPayload(recodedPayload) //remember to stuff the payload back in there, otherwise your change will not be registered
}
return true;
}
}
So now you have the message. How to handle the change of encoding can be tricky, but it's completely up to you.
You have the option of setting the encoding on the entire message, or navigating (with xpath) to the field you're interested in and manipulating just that. Even for those two options, there are several ways of accomplishing both. I'm going to go the lazy route: set the encoding on the entire payload:
private Source modifyEncoding(Source payload){
StringWriter sw = new StringWriter();
StreamSource newSource = null;
try {
TransformerFactory transformerFactory = TransformerFactory.newInstance();
Transformer transformer = transformerFactory.newTransformer();
transformer.setOutputProperty(OutputKeys.ENCODING, "UTF-8"); //this determines the outcome of the transformation
StreamResult output = new StreamResult(sw);
transformer.transform(source, output);
StringReader sReader = new StringReader(sw.toString());
newSource = new StreamSource(sReader);//Stuff the re-encoded xml back in a Source
} catch(Exception e){
ex.printStackTrace();
}
return newSource;
}
After the LogicalHandler, you now have the SOAPHandler. Here, setting the encoding is more straightforward, but its behaviour is implementation-dependent. Your SOAPHandler can look like this:
public class YourSOAPHandler implements SOAPHandler{
public boolean handleMessage(SOAPMessageContext msgCtxt){
boolean inbound = (boolean)msgCtxt.get(MessageContext.MESSAGE_INBOUND_PROPERTY);
if (inbound){
SOAPMessage msg = msgCtxt.getMessage();
msg.setProperty(javax.xml.soap.SOAPMessage.CHARACTER_SET_ENCODING,"UTF-8");
msgCtxt.Message(msg); //always put the message back where you found it.
}
}
return true;
}

OpenSaml read metadata

I'm tring to read the metadata from an Idp using Open Saml 2. When i try to unmarshall the metadata openSaml show only this getter for attributes getUnknownAtrributes(). Looks like i am missing some point since when reading the Idp response SAML the code works very well. (it shows getAssertions() that returns a list of assertions).
I need to parse the metadata and find informations regarding the Idp.
Here the method
public Metadata metadataReader() {
ByteArrayInputStream bytesIn = new ByteArrayInputStream(ISSUER_METADATA_URL.getBytes());
BasicParserPool ppMgr = new BasicParserPool();
ppMgr.setNamespaceAware(true);
// grab the xml file
// File xmlFile = new File(this.file);
Metadata metadata = null;
try {
Document document = ppMgr.parse(bytesIn);
Element metadataRoot = document.getDocumentElement();
QName qName = new QName(metadataRoot.getNamespaceURI(), metadataRoot.getLocalName(),
metadataRoot.getPrefix());
Unmarshaller unmarshaller = Configuration.getUnmarshallerFactory().getUnmarshaller(qName);
metadata = (Metadata) unmarshaller.unmarshall(metadataRoot);
return metadata;
} catch (XMLParserException e) {
e.printStackTrace();
} catch (UnmarshallingException e) {
e.printStackTrace();
}
return null;
}

I suggest using a metadata provider to do the heavy lifting for you. FilesystemMetadataProvider is often a good fit.
I have a blog post that explains how to use it.

android, UTF8 - How do I ensure UTF8 is used for a shared preference

How do I ensure UTF8 is used for a shared preference menu? I have an android preferences menu that allows a user to set their name, amongst other things.
I need to know how to convert the data stored in shared preferences into UTF8 format
The preference menu is laid out in xml using utf8 encoding in a file called settings in the res/xml folder and looks like this:
<?xml version="1.0" encoding="utf-8"?>
<PreferenceScreen
xmlns:android="http://schemas.android.com/apk/res/android">
<EditTextPreference
android:key="#string/name_key"
android:title="#string/hof_name_title"
android:summary="#string/hof_name_summary"
android:defaultValue="Anonymous" />
<CheckBoxPreference
android:key="#string/new_game_preference_key"
android:title="#string/new_game_preference_title"
android:summary="#string/new_game_preference_summary"
android:defaultValue="false" />
</PreferenceScreen>
The class that handles this is
public class PrefsActivity extends PreferenceActivity implements OnSharedPreferenceChangeListener{
#Override
protected void onCreate(Bundle savedInstanceState) {
requestWindowFeature(Window.FEATURE_NO_TITLE);
getWindow().setFlags(WindowManager.LayoutParams.FLAG_FULLSCREEN,
WindowManager.LayoutParams.FLAG_FULLSCREEN);
super.onCreate(savedInstanceState);
addPreferencesFromResource(R.xml.settings);
}
#Override
protected void onPause() {
super.onPause();
// Unregister the listener whenever a key changes
getPreferenceScreen().getSharedPreferences()
.unregisterOnSharedPreferenceChangeListener(this);
}
#Override
protected void onResume() {
super.onResume();
// Set up a listener whenever a key changes
getPreferenceScreen().getSharedPreferences()
.registerOnSharedPreferenceChangeListener(this);
}
#Override
public void onSharedPreferenceChanged(SharedPreferences sharedPreferences,
String key) {
// Util.log_debug_message("#### Changed prefrence key = " + key);
if (key.equalsIgnoreCase(getString(R.string.name_key))){
update_webserver();
}else if (key.equalsIgnoreCase(getString(R.string.new_game_preference_key))){
update_webserver();
}
}
protected void update_webserver(){
Intent webServerReg = new Intent(this, RegService.class);
webServerReg.putExtra(Config.C2DM_REG_ID_EXTRA, C2DMessaging.getRegistrationId(this));
startService(webServerReg);
}
}
I assumed that setting <?xml version="1.0" encoding="utf-8"?> would ensure that text would be encoded in UTF8 format but this is not always the case. I am getting values in all sorts of formats. Some examples
"\nFe\xF1a", "L\xFAcia", "$'\xE5 \xEE'",
or
":-$B-)O:-):-P=-OB-):-D:-Q:-X:-!:-|\n:'(:-*XDo_O:-X:-C:-X:O:-X=-O;-):-):-);-):-D:OX-(o_O:-V:-#:-V:-X:-Do_O::-C
\xBF\xA1\xA1\xAB\xBB\xAE\xA9^\xA5\xA5?[\xA2}?\xA2\xA2\xA5\xA3?$\xBF\xA1\xAE\xA7><[\xA3~?=~~\xB0]\xA3?\xA7\xA1\\\xAB\xBB\xAE^``]]||||}{_|]|||\xB0\xB0?"
Yes that is one loooong set of rubish in the last example.
The problems I have start when trying to send the nmes as a json HTTP POST or PUT request to my webserver using the following code
public void update(String regId) throws AuthenticationException {
JSONObject mu_to_send = createJsonToPost(regId, false);
HttpResponse response;
try {
response = HttpRequest.httpPutJSONObject(this,
Config.UPDATE_USER_URL, mu_to_send);
int responseCode = response.getStatusLine().getStatusCode();
String responseJson;
try {
responseJson = EntityUtils.toString(response.getEntity());
String msg = "";
if (responseCode == 201) {
// mobile_user = new
// JSONObject(responseJson).getJSONObject("mobile_user");
// Broadcast.sendBroadcast(this,
// ResponseReceiver.ACTION_RESP,
// Intent.CATEGORY_DEFAULT, Config.C2DM_MESSAGE_EXTRA,
// Config.BROADCAST_WEBSERVER_USER_CREATED);
} else {
msg = "Failed to register with server! HTTP response code = "
+ responseCode;
}
} catch (ParseException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
} catch (ClientProtocolException e1) {
// TODO Auto-generated catch block
e1.printStackTrace();
} catch (IOException e1) {
// TODO Auto-generated catch block
e1.printStackTrace();
}
}
public JSONObject createJsonToPost(String regId, boolean set_password) {
JSONObject holder = new JSONObject();
JSONObject data = new JSONObject();
try {
data.put("auth", regId);
data.put("name", C2DMessaging.getName(this));
data.put("receive_new_game_notification",
C2DMessaging.getGameStartedNotificationPreference(this));
holder.put("mobile_user", data);
} catch (JSONException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
return holder;
}
public static HttpResponse httpPutJSONObject(Context context, String url,
JSONObject data) throws ClientProtocolException, IOException,
AuthenticationException {
DefaultHttpClient httpclient = getHttpClient(context);
HttpPut httpput = new HttpPut(url);
httpput.addHeader("User-Agent", Config.USER_AGENT);
httpput.addHeader("Accept", "application/json");
httpput.addHeader("Content-type", "application/json");
httpput.addHeader(new BasicScheme().authenticate(
getCredentials(context), httpput));
StringEntity se = new StringEntity(data.toString());
httpput.setEntity(se);
return httpclient.execute(httpput);
}
This causes all sorts of issues on my web server which expects valid JSON (i.e. UTF8)
as JSON should be sent encoded in UTF8 in the first place.
So
1) Why doesn't <?xml version="1.0" encoding="utf-8"?> actually set UTF8 encoding? when used in the layout?
2) How best to ensure that I always get valid UTF8 format characters sent to my web server?
Should this be handled by the put request or by the code that saves to the shared preference or or by the code that populates the json object or perhaps a combination of the above
? or something else?
This is a follow on from this RoR question Rails 3 - How to handle PG Error incomplete multibyte character

The key is to understand the difference between UTF-8 and Unicode.
Java processes characters and strings in memory using Unicode. Each character is stored in two bytes.
When text is transmitted between processes (eg to a web server) or it is written to/read from disk, the internal representation is converted into an over-the-wire format. This is the encoding or decoding. UTF-8 is the most popular, but other formats include:
UTF-16
ISO 8859-1
In your question, you mention that the XML files are encoded in utf-8: That is good, and you will be able to put foreign characters in the files, but that specifies the encoding only for that specific XML file.
These XML files will be compiled into Android resources and will contain the correct values (you can check it if you like in the debugger, or by preserving the intermediate Java resource files from the build chain).
The problem is almost certainly where you send data to and receive data from the HTTP server, specifically where that data is converted between the bytes on the network and a Java String. Currently you are not setting it in the request - this can be done as described in the documentation for Apache HTTPClient.
Although the server might already require/assume this, it's certainly a good thing to state clearly in the request.
You also need to ensure that the server (the one in Rails 3 - How to handle PG Error incomplete multibyte character):
Is expecting UTF-8
Decodes the request using a UTF-8 decoder
Encodes the response using UTF-8 encoding
(Sorry, but I don't know Ruby on Rails so I don't know how to specifically help there).
Back in the Android end, you also need to ensure that your HTTP library is decoding the response with the UTF-8 decoder. If you handle this yourself, ensure that the String constructor you use is this one, and the argument is "utf-8":
public String (byte[] data, String charsetName)
Once BOTH the client and the server are using UTF-8, your problems will be resolved.
To help debugging here, I suggest:
A number of logging statements on server and client that print the relevant strings as close as possible to the HTTP code
Running with the client configured to talk through a debugging proxy. Examine the request and response and check that they are indeed UTF-8. Proxies include:
Charles
WebScarab
Fiddler