I am posting Events to Google Calendar using their REST API from Java (GAE), NOT the client library.
This works very well, except that non-ASCII characters are showing up as question marks.
Things I have done to try to address this (some are clutching at straws):
The content-type is set to UTF-8: "Content-Type", "application/json; charset=UTF-8"
I have tried URL encoding the "summary" (which is the Event title) field within the JSON body
I tried html escaping the "summary" field within the JSON body
Searched StackOverflow, and I found this reference that seems like the same problem, but it's in the context of PHP, not Java, so (I think) String handling is quite different, and therefore it doesn't seem to directly apply. Basically this one says make sure that the summary value itself is UTF-8 encoded, and uses a utf8_encode function for which there isn't a direct Java comparison
I realize that one answer is probably just to use the Java library, but for various reasons I don't want to do that unless I absolutely have to.
Any ideas please? Thank you.
EDIT TO ADD CODE:
I create a Scribe Request like this (and, yes, I know that this is not what Scribe is intended for...):
String url = String.format("https://www.googleapis.com/calendar/v3/calendars/%s/events", inTargetCalendar.getId());
OAuthRequest req = new OAuthRequest(Verb.POST, url);
req.addQuerystringParameter("access_token", <TOKEN>);
JSONObject jBody = new JSONObject();
jBody.put( "start", <START> );
jBody.put( "end", <END> );
jBody.put( "summary", getSummary() );
log.info("*** getSummary() is: " + getSummary());
jBody.put( "colorId", getColorId() );
req.addPayload(jBody.toString());
req.addHeader("Content-Type", "application/json; charset=UTF-8");
Note that in this example, when I look at my log file, the logs show (correctly):
*** getSummary() is: Côte Brasserie
I then post the request like this:
Response response=null;
try {
response = request.send();
} catch (...) {...}
Per notes above, this works perfectly, except that the "ô" doesn't show up correctly. Specifically when I look at the Event through the Google calendar website it looks like this:
C?te Brasserie
I.e. the "ô" shows up as a "?". This is the same for other non-ASCII characters (from what I have seen).
The answer was: give up trying to do this without the SDK even though that's what I wanted, and just use the SDK... and it worked straight away - non-ASCII characters showing up fine now.
Related
I need to replace the spaces inside a string with the % symbol but I'm having some issues, what I tried is:
imageUrl = imageUrl.replace(' ', "%20");
But It gives me an error in the replace function.
Then:
imageUrl = imageUrl.replace(' ', "%%20");
But It still gives me an error in the replace function.
The I tried with the unicode symbol:
imageUrl = imageUrl.replace(' ', (char) U+0025 + "20");
But it still gives error.
Is there an easy way to do it?
String.replace(String, String) is the method you want.
replace
imageUrl.replace(' ', "%");
with
imageUrl.replace(" ", "%");
System.out.println("This is working".replace(" ", "%"));
I suggest you to use a URL Encoder for Encoding Strings in java.
String searchQuery = "list of banks in the world";
String url = "http://mypage.com/pages?q=" + URLEncoder.encode(searchQuery, "UTF-8");
I've ran into issues like this in the past with certain frameworks. I don't have enough of your code to know for sure, but what might be happening is whatever http framework you are using, in my case it was spring, is encoding the URL again. I spent a few days trying to solve a similar problem where I thought that string replace and the URI.builder() was broken. What ended up being the problem was that my http framework had taken my encoded url, and encoded it again. that means that any place it saw a "%20", it would see the '%' charictor and switch it out for '%' http code, "%25", resulting in. "%2520". The request would then fail because %2520 didn't translate into the space my server was expecting. While the issue apeared to be one of my encoding not working, it was really an issue of encoding too many times. I have an example from some working code in one of my projects below
//the Url of the server
String fullUrl = "http://myapiserver.com/path/";
//The parameter to append. contains a space that will need to be encoded
String param 1 = "parameter 1"
//Use Uri.Builder to append parameter
Uri.Builder uriBuilder = Uri.parse(fullUrl).buildUpon();
uriBuilder.appendQueryParameter("parameter1",param1);
/* Below is where it is important to understand how your
http framework handles unencoded url. In my case, which is Spring
framework, the urls are encoded when performing requests.
The result is that a url that is already encoded will be
encoded twice. For instance, if you're url is
"http://myapiserver.com/path?parameter1=param 1"
and it needs to be read by the server as
"http://myapiserver.com/path?parameter1=param%201"
it makes sense to encode the url using URI.builder().append, or any valid
solutions listed in other posts. However, If the framework is already
encoding your url, then it is likely to run into the issue where you
accidently encode the url twice: Once when you are preparing the URL to be
sent, and once again when you are sending the message through the framework.
this results in sending a url that looks like
"http://myapiserver.com/path?parameter1=param%25201"
where the '%' in "%20" was replaced with "%25", http's representation of '%'
when what you wanted was
"http://myapiserver.com/path?parameter1=param%201"
this can be a difficult bug to squash because you can copy the url in the
debugger prior to it being sent and paste it into a tool like fiddler and
have the fiddler request work but the program request fail.
since my http framework was already encoding the urls, I had to unencode the
urls after appending the parameters so they would only be encoded once.
I'm not saying it's the most gracefull solution, but the code works.
*/
String finalUrl = uriBuilder.build().toString().replace("%2F","/")
.replace("%3A", ":").replace("%20", " ");
//Call the server and ask for the menu. the Menu is saved to a string
//rest.GET() uses spring framework. The url is encoded again as
part of the framework.
menuStringFromIoms = rest.GET(finalUrl);
There is likely a more graceful way to keep a url from encoding twice. I hope this example helps point you on the right direction or eliminate a possability. Good luck.
Try this:
imageUrl = imageUrl.replaceAll(" ", "%20");
Replace spaces is not enought, try this
url = java.net.URLEncoder.encode(url, "UTF-8");
I programmed an Android App, which manages data to be stored and deleted in a MySQL database (on server). Whenever on the smartphone special characters ("ä", "ü", ...) are used the symbol is converted badly.
In the log I can see that the "Umlaut" (e.g. "ä") is transmitted properly. I also use in my php file "SET NAMES 'UTF-8'", see here:
function connect()
{
....
// Connecting to mysql database
$con = mysql_connect(DB_SERVER, DB_USER, DB_PASSWORD) or die(mysql_error());
mysql_query("SET NAMES 'utf8'");
// Selecting database
$db = mysql_select_db(DB_DATABASE) or die(mysql_error()) or die(mysql_error());
return $con;
}
If I start a request via Postman, I can add words with special characters, e.g. "TÄST", the json response looks as follows:
{"catId":"35","category":"T\u00c4ST"}
So the words are well converted to UTF-8. But if I add an Item via smartphone, the response from the server (to retrieve the added item) looks like this:
{"catId":"37","category":"T?ST"}
The position in my code, where I add the parameters for the JSON Object is that (note that "name" is the string content of the edit text field):
List<NameValuePair> params = new ArrayList<NameValuePair>();
...
params.add(new BasicNameValuePair("category", catName));
After that the HTTPRequest is send:
JSONObject json = jsonParser.makeHttpRequest(url_dest, "POST", params);
If I print out the params the word "Täst" is visible... But unfortunately I'm not able to check which json string is arriving on the server (due to my bad php knowledge). So where's the problem? In my android application or in the php files located on the server?
Do I have to encode the outgoing json object in any way?
Thanks a lot for your help.
To debug on Eclipse I suggest you:
http://projects.eclipse.org/projects/tools.pdt
It is quite easy to configure.
Control the encoding in the DB; maybe the VARCHAR default latin1_swedish_ci can't save that string.
Another way to solve it could be looking to the apache commons:
http://commons.apache.org/proper/commons-lang/javadocs/api-2.6/org/apache/commons/lang/StringEscapeUtils.html
in particular the functions provided by: escapeHtml4 and unescapeHtml4
#dafocus, if you set your Android app project to UTF-8 encoding in your IDE, it should ensure that UTF-8 is handled properly in the app. It looks like an encoding change somewhere. You might not need to convert TÄST to T\u00c4ST at all if the encoding is UTF-8 throughout. PHP used to be difficult with charsets. I would suggest you to look at the PHP to figure out if the handler page causes this (possible) and/or the DB - as #madx just said.
This question already has answers here:
JSON URL from StackExchange API returning jibberish?
(3 answers)
Closed 9 years ago.
I am trying to grab the data from a json file through java. If I navigate to the URL using my browser, everything displays fine, but if I try to get the data using java I get get a bunch of characters that cannot be interpreted or parsed. Note that this code works with other JSON Files. Could this be a server side thing with the way the JSON file is created? I tried messing around with different character sets and that did not seem to fix the problem.
public static void main(String[] args) throws Exception {
URL url = new URL("http://www.minecraftpvp.com/api/ping.json");
URLConnection connection = url.openConnection();
BufferedReader in = new BufferedReader(new InputStreamReader(connection.getInputStream()));
boolean hasLine = true;
while (hasLine) {
String line = in.readLine();
if (line != null) {
System.out.println(line);
} else {
hasLine = false;
}
}
}
The output I get from this is just a ton of strange characters that make no sense at all. Where if I change the url to something like google.com, it works fine.
EDIT: JSON URL from StackExchange API returning jibberish? Seemed to have answered my question. I tried searching before I asked to make sure the answer wasn't here and couldn't find anything. Guess I didn't look hard enough.
Yes that URL is returning gzip encoded content by default.
You can do one of three things:
Explicitly set the Accept-Encoding: header in your request. A web service should not return gzip compression unless it is listed as an accepted encoding in the request, so this website is not being very friendly. Your browser is setting it as accepted I suspect, that is why you can see it there. Just set it to an empty value and it should as per the spec return non-encoded responses, your mileage may vary on this one.
Or use the answer in this How to handle non-UTF8 html page in Java? that shows how to decompress the response. This should be the preferred option over #1.
And/or Ask the person hosting the service to implement the recommended scheme which is to only provide compressed responses if the client says it can handle them or if it can infer it from the browser fingerprint with high confidence.
Best of luck C.
You need to inspect the Content-Encoding header. The URL in question improperly returns gzip-compressed content even when you don't ask for it, and you'll need to run it through a decoder.
I am developing a small mail client in the Java Play Framework and I'm using SendGrid for the e-mails. When an e-mail is received, it gets posted to a url and I then parse the posted form using JsonNode. Now the problem is the "to", "from", "subject" fields of that form are automatically converted by SendGrid to UTF-8. Now comes the problem: apparently, the email message body is encoded in "ISO-8859-1". And I need to convert that String to "UTF-8". I already tried several ways of doing so, but most probably I'm doing something very wrong, since I always get strange characters for French or German words containing accents/umlauts (Example "Zürich" comes out as "Z?rich". The code I'm using for the conversion is the following:
byte[] msg = message.getBytes("ISO-8859-1");
byte[] msg_utf8 = new String(msg, "ISO-8859-1").getBytes("UTF-8");
message = new String(msg_utf8, "UTF-8");
Could you, please, suggest a solution? Thank you very much in advance!
Ok so I managed to get the raw byte request from SendGrid using the annotation and created the java String with the correct encodings:
#BodyParser.Of(BodyParser.Raw.class)
public static Result getmail() {
...
}
Now the problem is that for retrieving the file attachments from the request I would need the request to be parsed as MultipartFormData. With the annotation above set, I get a NullPointerException when calling, which was predictable:
request().body().asMultipartFormData().getFiles()
Does any of you have any idea on how I could get the same request again, but parsed with the #BodyParser.Of(Bodyparser.MultipartFormData.class) ? So I kind of need to combine the two annotations or find a way to convert the byte[] I get from the Raw parser to a MultiFormData. Thanks!
I need help. In my current development one of the requirements says:
The server will return 200-OK as a response(httpresponse).
If the panelist is verified then as a result, the server must also
return the panelist id of this panelist.
The server will place the panelist id inside the body of the 200-OK
response in the following way:
<tdcp>
<cmd>
<ack cmd=”Init”>
<panelistid>3849303</panelistid>
</ack>
</cmd>
Now I am able to put the httpresponse as
httpServletResponse.setStatus(HttpServletResponse.SC_OK);
And I can put
String responseToClient= "<tdcp><cmd><ack cmd=”Init”><panelistid>3849303</panelistid></ack></cmd></tdcp>";
Now what does putting the above xml inside the body of 200-OK response mean and how can it be achieved?
You can write the XML directly to the response as follows:
This example uses a ServletResponse.getWriter(), which is a PrintWriter to write a String to the response.
String responseToClient= "<tdcp><cmd><ack cmd=”Init”><panelistid>3849303</panelistid></ack></cmd></tdcp>";
httpServletResponse.setStatus(HttpServletResponse.SC_OK);
httpServletResponse.getWriter().write(responseToClient);
httpServletResponse.getWriter().flush();
You simply need to get the output stream (or output writer) of the servlet response, and write to that. See ServletResponse.getOutputStream() and ServletResponse.getWriter() for more details.
(Or simply read any servlet tutorial - without the ability to include data in response bodies, servlets would be pretty useless :)
If that's meant to be XML, Word has already spoiled things for you by changing the attribute quote symbol to ” instead of ".
It is worth having a look at JAXP if you want to generate XML using Java. Writing strings with < etc. in them won't scale and you'll run into problems with encodings of non-ASCII characters.