We Keep Coding

Decode google translate API response in JAVA

I need to write a small tool in JAVA which will translate text from English to French using the Google translate API. Everything works but I have an apostrophe decoding problem.
Original text:
Inherit Tax Rate
Text translated with Google translate API:
Taux d' imposition hérité
How it should be:
Taux d'imposition hérité
This is my translate method(sorry for the long method):
private String translate(String text, String from, String to) {
StringBuilder result = new StringBuilder();
try {
String encodedText = URLEncoder.encode(text, "UTF-8");
String urlStr = "https://www.googleapis.com/language/translate/v2?key=" + sKey + "&q=" + encodedText + "&target=" + to + "&source=" + from;
URL url = new URL(urlStr);
HttpsURLConnection conn = (HttpsURLConnection) url.openConnection();
InputStream googleStream;
if (conn.getResponseCode() == 200) {
googleStream = conn.getInputStream(); //success
} else
googleStream = conn.getErrorStream();
BufferedReader reader = new BufferedReader(new InputStreamReader(googleStream));
String line;
while ((line = reader.readLine()) != null) {
result.append(line);
}
JsonParser parser = new JsonParser();
JsonElement element = parser.parse(result.toString());
if (element.isJsonObject()) {
JsonObject obj = element.getAsJsonObject();
if (obj.get("error") == null) {
String translatedText = obj.get("data").getAsJsonObject().
get("translations").getAsJsonArray().
get(0).getAsJsonObject().
get("translatedText").getAsString();
return translatedText;
}
}
if (conn.getResponseCode() != 200) {
System.err.println(result);
}
} catch (IOException | JsonSyntaxException ex) {
System.err.println(ex.getMessage());
}
return null;
}
I'm using an XML writer to write the text and first I though that this has a problem, but I observed that the text is returned like this in the stream so I introduced the encoding parameter when I initialise the InputStreamReader:
BufferedReader reader = new BufferedReader(new InputStreamReader(googleStream, "UTF-8"));
But I receive the string with the same problem. Any ideas about what I can do?

I think this problem is solved by using the format parameter (docs). It defaults to html, but you can change it to text to receive unencoded data. Your request should look like this:
String urlStr = "https://www.googleapis.com/language/translate/v2?key=" + sKey + "&q=" + encodedText + "&target=" + to + "&source=" + from + "&format=text";

Youtube extraction URL only working for some videos

I'm having some trouble with this code here. What i've done essentially is parse the HTML of a video page and get an encoded MP4 URL. From there i've gathered key value pairs, and recreated a URL using the correct ones/format. This code does work but it only seems to work for non-licensed videos. I've compared the URLs for both the working address and the not working one, both have the exact same parameters, structure, not differences that i can see. The only data from the MP4 string that i'm not using is quality= and type=, both of which i've found shouldn't be in the actual url. I'm a little lost since this seems to work for some videos and not others and there is also no other data that could be used in the URL. What am i missing?
public static String getActualYTURL(String myURL) throws IOException {
CloseableHttpClient httpclient = HttpClients.createDefault();
HttpGet httpget = new HttpGet(myURL);
CloseableHttpResponse response = httpclient.execute(httpget);
//establish connection
String html = "";
InputStream in = response.getEntity().getContent();
BufferedReader reader = new BufferedReader(new InputStreamReader(in));
StringBuilder str = new StringBuilder();
String line = null;
while ((line = reader.readLine()) != null) {
str.append(line.replace("\\u0026", "&"));
}
in.close();
html = str.toString();
//get HTML for Youtube page
Pattern p = Pattern.compile("url_encoded_fmt_stream_map\":\"(.*?)?\"");
Matcher m = p.matcher(html);
ArrayList<String> matches = new ArrayList<String>();
m.find();
String urls[] = m.group().split(",");
//get map of encoded URLs
String encodedMP4URL = null;
for (String ppUrl : urls) {
String url = URLDecoder.decode(ppUrl, "UTF-8");
Pattern p1 = Pattern.compile("type=video/mp4");
Matcher m1 = p1.matcher(url);
if (m1.find()) {
encodedMP4URL = url;
}
}
//get MP4 encoded URL
HashMap <String, String> pairs = new HashMap<String, String>();
String[] temp = encodedMP4URL.split("&");
for (int i = 0; i < temp.length; i ++)
if (!temp[i].contains("url="))
pairs.put(temp[i].split("=")[0], temp[i].split("=")[1]);
else {
String URLPart = temp[i].split("\\?")[0] + "?";
pairs.put(URLPart.split("=")[0], URLPart.split("=")[1]);
String otherPart = temp[i].split("\\?")[1];
pairs.put(otherPart.split("=")[0], otherPart.split("=")[1]);
//deal with special case of first pair after url
}
//decode String into key value pairs
pairs.remove("quality");
pairs.remove("type");
//remove pairs that aren't used
StringBuilder realURL = new StringBuilder(pairs.get("url"));
pairs.remove("url");
//add url base then remove it from map
for (String s : pairs.keySet())
if (s.equals("s"))
realURL.append("signature=" + pairs.get(s) + "&");
//deal with special case "s" key needs to be "signature" in actual url
else
realURL.append(s + "=" + pairs.get(s) + "&");
//encode URL properly with required params
return realURL.toString();
}
Sample URL output: https://r16---sn-ab5l6nll.googlevideo.com/videoplayback?dur=298.608&mime=video%2Fmp4&source=youtube&ratebypass=yes&gir=yes&lmt=1479243873107622&id=o-AFZWFgdwCg66TqdZ2ZY823besbDXiB37zBB9ZwzPLwKe&key=yt6&itag=18&mm=31&mn=sn-ab5l6nll&ei=-uStWICxJ4TK8gT_xoLwDw&ms=au&ip=47.19.92.83&mt=1487791178&initcwndbps=922500&ipbits=0&mv=m&sparams=clen%2Cdur%2Cei%2Cgir%2Cid%2Cinitcwndbps%2Cip%2Cipbits%2Citag%2Clmt%2Cmime%2Cmm%2Cmn%2Cms%2Cmv%2Cpl%2Cratebypass%2Crequiressl%2Csource%2Cupn%2Cexpire&upn=mylzrCCRyNc&requiressl=yes&signature=12A12AC76CD7E14F402CC9EBE879103F1B2C55C870C.D86FB6D4D5D99C0DA732D4EC671EB522E9330D78&expire=1487812954&clen=26466943&pl=17&

Constant 401 error with OAuth1 (MCM API)

I've been trying to establish a connection with an API for more than a week now, to no avail. (Magic Card Market's, authentification documentation here and there). I'm supposed to receive a XML file.
I have what MCM call a "widget" access to their API, meaning that I don't have nor need a oauth_token (it's supposed to be an empty string) for the authorization header, and that I'm not supposed to receive nor use an access token/access secret.
The only things I do have are a consumer key (they call it app token sometimes) and a consumer secret.
Here is how I build my Authorization header :
private static String buildOAuthAuthorization(String method, String request)
throws UnsupportedEncodingException, NoSuchAlgorithmException, InvalidKeyException {
String mkmAppToken = APICredentials.appToken;
String mkmAppSecret = APICredentials.appSecret;
String realm = "https://www.mkmapi.eu/ws/v1.1/games";
String oauthVersion = "1.0";
String oauthConsumerKey = mkmAppToken;
String oauthToken = "";
String oauthSignatureMethod = "HMAC-SHA1";
String oauthTimestamp = Long.toString(System.currentTimeMillis() / 1000);
String oauthNonce = Long.toString(System.currentTimeMillis());
String paramString = "oauth_consumer_key=" + oauthConsumerKey
+ "oauth_nonce=" + oauthNonce
+ "oauth_signature_method=" + oauthSignatureMethod
+ "oauth_timestamp=" + oauthTimestamp
+ "oauth_token=" + oauthToken
+ "oauth_version=" + oauthVersion;
String baseString = method + "&" + rawUrlEncode(realm) + "&" + rawUrlEncode(paramString);
String signingKey = rawUrlEncode(mkmAppSecret) + "&";
Mac mac = Mac.getInstance("HMAC-SHA1");
SecretKeySpec secret = new SecretKeySpec(signingKey.getBytes(), mac.getAlgorithm());
mac.init(secret);
byte[] digest = mac.doFinal(baseString.getBytes());
byte[] oauthSignature = Base64.encode(digest, Base64.URL_SAFE);
String authorizationProperty = "OAuth "
+ "realm=\"" + realm + "\", "
+ "oauth_version=\"" + oauthVersion + "\", "
+ "oauth_timestamp=\"" + oauthTimestamp + "\", "
+ "oauth_nonce=\"" + oauthNonce + "\", "
+ "oauth_consumer_key=\"" + oauthConsumerKey + "\", "
+ "oauth_token=\""+ oauthToken + "\", "
+ "oauth_signature_method=\"" + oauthSignatureMethod + "\", "
+ "oauth_signature=\"" + oauthSignature + "\"";
System.out.println(authorizationProperty);
return authorizationProperty;
}
The actual request is in an AsyncTask :
public static class oAuthRequest extends AsyncTask<String, Integer, StringReader> {
private int lastCode;
#Override
protected StringReader doInBackground(String... requestURLs) {
String method = requestURLs[0];
String url = requestURLs[1];
StringReader result = null;
try {
String authProperty = buildOAuthAuthorization(method, url);
HttpURLConnection connection = (HttpURLConnection) new URL(url).openConnection();
connection.addRequestProperty("Authorization:", authProperty);
lastCode = connection.getResponseCode();
System.out.println("RESPONSE CODE 1 " + lastCode);
// Get content
BufferedReader rd = new BufferedReader(new InputStreamReader(lastCode == 200 ? connection.getInputStream() : connection.getErrorStream()));
StringBuilder sb = new StringBuilder();
String line;
while ((line = rd.readLine()) != null) {
sb.append(line);
}
rd.close();
result = new StringReader(sb.toString());
} catch (NoSuchAlgorithmException | InvalidKeyException | IOException e) {
e.printStackTrace();
}
return result;
}
}
It seems like no matter what I change, I'm always getting a 401.
Things I've tried :
oauthSignature as a String using Base64.encodeToString()
Nonce generation using SecureRandom
With and without the empty oauthToken
Another timestamp generation method (can't remember what though)
signing key with and without app token (theorically I need only the consumer secret, but you never know)
Using HttpsURLConnection instead of HttpURLConnection (the URI start in https, so I thought, hey. But no)
At least 2-3 other different implementations (one who was basically a copy/paste of the Java example in the documentation of course -- it still kind of is one now)
(Probably a lot of things I can't even remember)
At this point I'm wondering if maybe the issue comes from my keys, as I've tried to use the Postman app to test requests with the same results.

405 for URL: https://www.mutalyzer.nl/position-converter

i am trying to fetch few genomic coordinate using API from the web mutalyzer.nl using the method position converter, but i am getting 405 error. Can you please help?
here is my code:-
import java.lang.Object;
import org.apache.commons.*;
import java.util.regex.Pattern;
import java.util.regex.Matcher;
private long checkVariant() {
long totalLines = 0;
File tempInputFile = new File(filePath + "input.tsv");
File outputFile = new File(filePath + "temp.tsv");
outputFile.append("Query String" + "\t" + "Response");
tempInputFile.withReader("UTF-8") { buf ->
buf.eachLine { line ->
if (totalLines > 0) { // skip first line
String[] words = line.split("\\t");
// fire remote query and check response...
//https://www.mutalyzer.nl/position-converter?assembly_name_or_alias=GRCh37&description=NM_024675.3%3Ac.232G%3EA
def queryTerm = "description=" + words[1];
def query = "assembly_name_or_alias=GRCh37&" + queryTerm;
final Pattern pattern = Pattern.compile("<code>(.+?)</code>");
def urlString = "https://www.mutalyzer.nl/position-converter";
def url = new URL(urlString);
println urlString;
def connection = url.openConnection();
connection.setRequestMethod("POST");
connection.doOutput = true;
def writer = new OutputStreamWriter(connection.outputStream);
println query
writer.write(query);
println query
writer.flush();
writer.close();
connection.connect();
def recaptchaResponse = connection.content.text;
final Matcher matcher = pattern.matcher(recaptchaResponse);
matcher.find();
def answerString = matcher.group(1);
//def answer = answerString.replace("Human ", "").replace(" - UCSC Genome Browser v318", "");
println "QueryTerm :=" + words[1] + " : Response := " + answerString;
outputFile.append(System.getProperty("line.separator") + words[1] + "\t" + answerString);
}
totalLines = totalLines + 1;
}
}
return totalLines;
}
can any one tell me the exact issue?

A http 405 status code means the method is not supported, i.e you're trying to post. From looking at the web pages source its a GET. Try changing
connection.setRequestMethod("POST");
To
connection.setRequestMethod("GET");
Reference to HTTP 405- http://www.checkupdown.com/status/E405.html

What does the API of this website say? Do they support HTTP POST for their APIs.
I called a HTTP GET on following URL (remember that GET has no body in request)
https://www.mutalyzer.nl/position-converter?assembly_name_or_alias=GRCh37&description=NM_003002.3%3Ac.274G%3ET and was able to receive a HTML response from them.
I drilled there site and found they have SOAP Services and REST Services exposed https://www.mutalyzer.nl/webservices where example is given. (What you may be looking is https://www.mutalyzer.nl/soap-api#src.idp856784 submitBatchJob with GET)
Create a SOAP request to invoke soap service or HTTP request to invoke it using REst
https://mutalyzer.nl/json/submitBatchJob?data=Tk1fMDAzMDAyLjM6Yy4yNzRHPlQ=&process=PositionConverter&argument=hg19

Java, URL output different in java output

I have a question about URI and URL
when i pass a url is work good but result is worst need help!!
as my code look like this.
import java.io.*;
import java.net.*;
import java.net.URL;
public class isms {
public static void main(String[] args) throws Exception {
try {
String user = new String ("boo");
String pass = new String ("boo");
String dstno = new String("60164038811"); //You are going compose a message to this destination number.
String msg = new String("你的哈达哈达!"); //Your message over here
int type = 2; //for unicode change to 2, normal will the 1.
String sendid = new String("isms"); //Malaysia does not support sender id yet.
// Send data
URI myUrl = new URI("http://www.isms.com.my/isms_send.php?un=" + user + "&pwd=" + pass
+ "&dstno=" + dstno + "&msg=" + msg + "&type=" + type + "&sendid=" + sendid);
URL url = new URL(myUrl.toASCIIString());
URLConnection conn = url.openConnection();
conn.setDoOutput(true);
// Get the response
BufferedReader rd = new BufferedReader(new InputStreamReader(conn.getInputStream()));
String line;
while ((line = rd.readLine()) != null) {
// Print the response output...
System.out.println(line);
}
rd.close();
System.out.println(url);
} catch (Exception e) {
e.printStackTrace();
}
}
}
the output in web is different..
on my java output is
你的哈达哈达!
but on my the site is
ÄãµÄ¹þ´ï¹þ´ï!
Help!!

String user = new String ("boo");
You don't need to (and shouldn't) do new String in Java—String user = "boo"; is fine.
String msg = new String("你的哈达哈达!");
Writing non-ASCII characters in your source means that you have to get the -encoding flag to javac to match the encoding you have saved your text files with. It is possible you have saved the .java file as UTF-8 but not configured your build environment to use UTF-8 at compile time.
If you are not sure that you've got this right, you can use ASCII-safe \u escapes in the meantime:
String msg = "\u4F60\u7684\u54C8\u8FBE\u54C8\u8FBE!"; // 你的哈达哈达!
Finally:
URI myUrl = new URI("http://www.isms.com.my/isms_send.php?un=" + user + "&pwd=" + pass
+ "&dstno=" + dstno + "&msg=" + msg + "&type=" + type + "&sendid=" + sendid);
When you're putting a URI together you should URL-escape each of the parameters you include in the string. Otherwise any & or other invalid character in the value will break the query. This also allows you to choose what charset is used to create the query string.
String enc = "UTF-8";
URI myUrl = new URI("http://www.isms.com.my/isms_send.php?" +
"un=" + URLEncoder.encode(user, enc) +
"&pwd=" + URLEncoder.encode(pass, enc) +
"&dstno=" + URLEncoder.encode(dstno, enc) +
"&msg=" + URLEncoder.encode(msg, enc) +
"&type=" + URLEncoder.encode(Integer.toString(type), enc) +
"&sendid=" + URLEncoder.encode(sendid, enc)
);
What the right value for enc is depends on the service you are connecting to, but UTF-8 is a good guess.