I'm having some trouble with this code here. What i've done essentially is parse the HTML of a video page and get an encoded MP4 URL. From there i've gathered key value pairs, and recreated a URL using the correct ones/format. This code does work but it only seems to work for non-licensed videos. I've compared the URLs for both the working address and the not working one, both have the exact same parameters, structure, not differences that i can see. The only data from the MP4 string that i'm not using is quality= and type=, both of which i've found shouldn't be in the actual url. I'm a little lost since this seems to work for some videos and not others and there is also no other data that could be used in the URL. What am i missing?
public static String getActualYTURL(String myURL) throws IOException {
CloseableHttpClient httpclient = HttpClients.createDefault();
HttpGet httpget = new HttpGet(myURL);
CloseableHttpResponse response = httpclient.execute(httpget);
//establish connection
String html = "";
InputStream in = response.getEntity().getContent();
BufferedReader reader = new BufferedReader(new InputStreamReader(in));
StringBuilder str = new StringBuilder();
String line = null;
while ((line = reader.readLine()) != null) {
str.append(line.replace("\\u0026", "&"));
}
in.close();
html = str.toString();
//get HTML for Youtube page
Pattern p = Pattern.compile("url_encoded_fmt_stream_map\":\"(.*?)?\"");
Matcher m = p.matcher(html);
ArrayList<String> matches = new ArrayList<String>();
m.find();
String urls[] = m.group().split(",");
//get map of encoded URLs
String encodedMP4URL = null;
for (String ppUrl : urls) {
String url = URLDecoder.decode(ppUrl, "UTF-8");
Pattern p1 = Pattern.compile("type=video/mp4");
Matcher m1 = p1.matcher(url);
if (m1.find()) {
encodedMP4URL = url;
}
}
//get MP4 encoded URL
HashMap <String, String> pairs = new HashMap<String, String>();
String[] temp = encodedMP4URL.split("&");
for (int i = 0; i < temp.length; i ++)
if (!temp[i].contains("url="))
pairs.put(temp[i].split("=")[0], temp[i].split("=")[1]);
else {
String URLPart = temp[i].split("\\?")[0] + "?";
pairs.put(URLPart.split("=")[0], URLPart.split("=")[1]);
String otherPart = temp[i].split("\\?")[1];
pairs.put(otherPart.split("=")[0], otherPart.split("=")[1]);
//deal with special case of first pair after url
}
//decode String into key value pairs
pairs.remove("quality");
pairs.remove("type");
//remove pairs that aren't used
StringBuilder realURL = new StringBuilder(pairs.get("url"));
pairs.remove("url");
//add url base then remove it from map
for (String s : pairs.keySet())
if (s.equals("s"))
realURL.append("signature=" + pairs.get(s) + "&");
//deal with special case "s" key needs to be "signature" in actual url
else
realURL.append(s + "=" + pairs.get(s) + "&");
//encode URL properly with required params
return realURL.toString();
}
Sample URL output: https://r16---sn-ab5l6nll.googlevideo.com/videoplayback?dur=298.608&mime=video%2Fmp4&source=youtube&ratebypass=yes&gir=yes&lmt=1479243873107622&id=o-AFZWFgdwCg66TqdZ2ZY823besbDXiB37zBB9ZwzPLwKe&key=yt6&itag=18&mm=31&mn=sn-ab5l6nll&ei=-uStWICxJ4TK8gT_xoLwDw&ms=au&ip=47.19.92.83&mt=1487791178&initcwndbps=922500&ipbits=0&mv=m&sparams=clen%2Cdur%2Cei%2Cgir%2Cid%2Cinitcwndbps%2Cip%2Cipbits%2Citag%2Clmt%2Cmime%2Cmm%2Cmn%2Cms%2Cmv%2Cpl%2Cratebypass%2Crequiressl%2Csource%2Cupn%2Cexpire&upn=mylzrCCRyNc&requiressl=yes&signature=12A12AC76CD7E14F402CC9EBE879103F1B2C55C870C.D86FB6D4D5D99C0DA732D4EC671EB522E9330D78&expire=1487812954&clen=26466943&pl=17&
Related
public static String getsheetdata() throws IOException {
String name = null;
String email = null;
String phone = null;
String fin = null;
String address = null;
String car_registraion = null;
String question = null;
String pin = null;
String car_registraion_date = null;
String url = "https://sheets.googleapis.com/v4/spreadsheets/1BH-e3-XSZ9LjsQqELjZLpZbnB4DmIhrPy2VDAZsP9KM/values/lead!A2:J2?key=AIzaSyDJRy73ru1BSLFCb9nknUF8SlZd4LxwJAc";
URL obj = new URL(url);
HttpsURLConnection con = (HttpsURLConnection) obj.openConnection();
// optional default is GET
con.setRequestMethod("GET");
//add request header
con.setRequestProperty("User-Agent", "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_5_8; en-US) AppleWebKit/532.5 (KHTML, like Gecko) Chrome/4.0.249.0 Safari/532.5");
int responseCode = con.getResponseCode();
System.out.println("\nSending 'GET' request to URL : " + url);
System.out.println("Response Code : " + responseCode);
BufferedReader in = new BufferedReader(
new InputStreamReader(con.getInputStream()));
String inputLine;
StringBuffer response = new StringBuffer();
while ((inputLine = in.readLine()) != null) {
response.append(inputLine);
}
in.close();
System.out.println(response.toString());
//Read JSON response and print
JSONObject myResponse = new JSONObject(response.toString());
return inputLine;
}
I am getting following response
Sending 'GET' request to URL : https://sheets.googleapis.com/v4/spreadsheets/1BH-e3-XSZ9LjsQqELjZLpZbnB4DmIhrPy2VDAZsP9KM/values/lead!A2:J2?key=AIzaSyDJRy73ru1BSLFCb9nknUF8SlZd4LxwJAc
Response Code : 200
{ "range": "lead!A2:J2", "majorDimension": "ROWS", "values": [ [ "Lead Data Set 1 - Normal FOC Lead", "Bhupendra", "bhupendra+283273#abc.com", "2389432432", "90909892098988771", "Street123, Berlin", "1289243424321", "no comments", "10115", "12 / 12 / 2017" ] ]}
I need to fill the response data in following variables .
String name = null;
String email = null;
String phone = null;
String fin = null;
String address = null;
String car_registraion = null;
String question = null;
String pin = null;
String car_registraion_date = null;
Would appreciate if anyone can help me on it.
You can use any JSON to Java unmarshalling library to convert the JSON to Java object. Check options and examples
Create JSONObject of the response string you are getting and then extract values fields from JSONObject as JSONArray and then traverse through that JSONArray to get list of your object.
To add to Hiren's answer, you can try (using org.json):
JSONObject myResponse = new JSONObject(response.toString());
JSONArray jsonArr = (JSONArray) myResponse.get("values");
JSONArray requiredValues = jsonArr.getJSONArray(0);
String[] values = new String[requiredValues.length()];
for (int i = 0; i < requiredValues.length(); i++) {
values[i] = requiredValues.getString(i);
}
Now the "values" part of response will be stored in String[] values
for (int j = 0; j < values.length; j++) {
System.out.println(values[j]);
}
This will print
Lead Data Set 1 - Normal FOC Lead
Bhupendra
bhupendra+283273#abc.com
2389432432
90909892098988771
Street123, Berlin
1289243424321
no comments
10115
12 / 12 / 2017
You can assign it accordingly. Hope it helps.
I'm trying to get 2 values from Wikipedia with the next API link:
https://en.wikipedia.org/w/api.php?action=query&generator=random&grnnamespace=0&prop=pageimages&format=json&pithumbsize=500&utf8
because its generated by random, sometimes it doesn't return one of the values I need, but I will solve this later, currently I'm having a problem with accessing the two values I need in the Json, "title" and "source"
The returned Json is like this:
{"batchcomplete":"","continue":{"grncontinue":"0.360395277951|0.360395626487|10429617|0","continue":"grncontinue||"},"query":{"pages":{"38690716":{"pageid":38690716,"ns":0,"title":"Alaine Chartrand","thumbnail":{"source":"https://upload.wikimedia.org/wikipedia/commons/d/d4/Alaine_Chartrand.jpg","width":267,"height":400},"pageimage":"Alaine_Chartrand.jpg"}}}}
this is the code, can anybody figure out why does it go to JSONException?
String API = "https://en.wikipedia.org/w/api.php?action=query&generator=random&grnnamespace=0&prop=pageimages&format=json&pithumbsize=500&utf8";
//open connection with wikipedia.
HttpURLConnection httpcon = (HttpURLConnection) new URL(API).openConnection();
//read all the input from wikipedia.
BufferedReader in = new BufferedReader(new InputStreamReader(httpcon.getInputStream()));
String responseSB = in.lines().collect(Collectors.joining());
in.close();
JSONObject incomingJSON = new JSONObject(responseSB);
//crashes here
String mTitle = incomingJSON.getString("title");
String mUrl = incomingJSON.getString("source");
If you notice the JSON it is randomly generated but with specific format
Case 1
{
"batchcomplete": "",
"continue": {
"grncontinue": "0.720220803439|0.720221273467|12887566|0",
"continue": "grncontinue||"
},
"query": {
"pages": {
"4897672": {
"pageid": 4897672,
"ns": 0,
"title": "New Hope, Sunnyvale, Texas"
}
}
}
}
query and pages were always existed, and in pages the key is always randomly generate, so it is Map<String, JSONObject> map of String key and JSONObject as value, Then you need to get the title value from map values
String API = "https://en.wikipedia.org/w/api.php?action=query&generator=random&grnnamespace=0&prop=pageimages&format=json&pithumbsize=500&utf8";
//open connection with wikipedia.
HttpURLConnection httpcon = (HttpURLConnection) new URL(API).openConnection();
//read all the input from wikipedia.
BufferedReader in = new BufferedReader(new InputStreamReader(httpcon.getInputStream()));
String responseSB = in.lines().collect(Collectors.joining());
in.close();
JSONObject incomingJSON = new JSONObject(responseSB);
Map<String,JSONObject> map = (Map<String, JSONObject>) incomingJSON.getJSONObject("query").getJSONObject("pages");
map.forEach((k,v)->System.out.println(" The key is : "+k+" the title is : "+v.getString("title")));
Case 2 With source
{
"batchcomplete": "",
"continue": {
"grncontinue": "0.165621850014|0.165622038679|37982311|0",
"continue": "grncontinue||"
},
"query": {
"pages": {
"57529788": {
"pageid": 57529788,
"ns": 0,
"title": "Model Store",
"thumbnail": {
"source": "https://upload.wikimedia.org/wikipedia/commons/thumb/2/23/Australia_New_South_Wales_relief_location_map.png/500px-Australia_New_South_Wales_relief_location_map.png",
"width": 500,
"height": 443
},
"pageimage": "Australia_New_South_Wales_relief_location_map.png"
}
}
}
}
So source may not present in every response, handle with try catch
String API = "https://en.wikipedia.org/w/api.php?action=query&generator=random&grnnamespace=0&prop=pageimages&format=json&pithumbsize=500&utf8";
//open connection with wikipedia.
HttpURLConnection httpcon = (HttpURLConnection) new URL(API).openConnection();
//read all the input from wikipedia.
BufferedReader in = new BufferedReader(new InputStreamReader(httpcon.getInputStream()));
String responseSB = in.lines().collect(Collectors.joining());
in.close();
JSONObject incomingJSON = new JSONObject(responseSB);
Map<String,JSONObject> map = (Map<String, JSONObject>) incomingJSON.getJSONObject("query").getJSONObject("pages");
map.forEach((k,v)->{
System.out.println(" The key is : "+k+" the title is : "+v.getString("title"));
//use try catch to get source because you will not get the same response every time
String source = v.getJSONObject("thumbnail").getString("source");
});
}
You can't get title and source directly from JSON response because it has to contain multiple inner objects. Below is the code snap for reading title and source.
// new code
JSONObject incomingJSON = new JSONObject(responseSB);
JSONObject innerObject = incomingJSON.getJsonObject("query").getJsonObject("pages").getJsonObject("38690716");
String mTitle= innerObject.getString("title");
String mUrl= innerObject.getJsonObject("thumbnail").getString("source");
//crashes here
String mTitle = incomingJSON.getString("title");
String mUrl = incomingJSON.getString("source");
try this...
JSONObject incomingJSON = new JSONObject(responseSB);
JSONObject TitleObjects = incomingJSON.getJSONObject("query");
JSONObject j_Objects_01 = TitleObjects.getJSONObject("pages");
JSONObject j_Objects_02 = j_Objects_01.getJSONObject("38690716");
String mTitle = j_Objects_02.getString("title");
JSONObject j_Objects_03 = j_Objects_02.getJSONObject("thumbnail");
String mUrl = j_Objects_03.getString("source");
You should be aware that the page id will change and the thumbnail is optional.
// new code
JSONObject incomingJSON = new JSONObject(responseSB);
JSONObject pages = incomingJSON.getJSONObject("query").getJSONObject("pages");
Iterator<String> it = pages.keys();
while(it.hasNext()) {
JSONObject page = pages.getJSONObject(it.next());
String mTitle= page.getString("title");
if(page.keySet().contains("thumbnail")) {
String mUrl= page.getJSONObject("thumbnail").getString("source");
}
}
So because the ID kept changing, I've decided to go from another approach.
I've used the following code:
Pattern p = Pattern.compile("\"source\":\"(.*?)\",\"width");
Matcher m = p.matcher(responseSB);
if (m.find()) {
url = m.group(1);
}
p = Pattern.compile("\"title\":(.*?)\",\"thumbnail");
m = p.matcher(responseSB);
if (m.find()) {
description = m.group(1);
}
I need to write a small tool in JAVA which will translate text from English to French using the Google translate API. Everything works but I have an apostrophe decoding problem.
Original text:
Inherit Tax Rate
Text translated with Google translate API:
Taux d' imposition hérité
How it should be:
Taux d'imposition hérité
This is my translate method(sorry for the long method):
private String translate(String text, String from, String to) {
StringBuilder result = new StringBuilder();
try {
String encodedText = URLEncoder.encode(text, "UTF-8");
String urlStr = "https://www.googleapis.com/language/translate/v2?key=" + sKey + "&q=" + encodedText + "&target=" + to + "&source=" + from;
URL url = new URL(urlStr);
HttpsURLConnection conn = (HttpsURLConnection) url.openConnection();
InputStream googleStream;
if (conn.getResponseCode() == 200) {
googleStream = conn.getInputStream(); //success
} else
googleStream = conn.getErrorStream();
BufferedReader reader = new BufferedReader(new InputStreamReader(googleStream));
String line;
while ((line = reader.readLine()) != null) {
result.append(line);
}
JsonParser parser = new JsonParser();
JsonElement element = parser.parse(result.toString());
if (element.isJsonObject()) {
JsonObject obj = element.getAsJsonObject();
if (obj.get("error") == null) {
String translatedText = obj.get("data").getAsJsonObject().
get("translations").getAsJsonArray().
get(0).getAsJsonObject().
get("translatedText").getAsString();
return translatedText;
}
}
if (conn.getResponseCode() != 200) {
System.err.println(result);
}
} catch (IOException | JsonSyntaxException ex) {
System.err.println(ex.getMessage());
}
return null;
}
I'm using an XML writer to write the text and first I though that this has a problem, but I observed that the text is returned like this in the stream so I introduced the encoding parameter when I initialise the InputStreamReader:
BufferedReader reader = new BufferedReader(new InputStreamReader(googleStream, "UTF-8"));
But I receive the string with the same problem. Any ideas about what I can do?
I think this problem is solved by using the format parameter (docs). It defaults to html, but you can change it to text to receive unencoded data. Your request should look like this:
String urlStr = "https://www.googleapis.com/language/translate/v2?key=" + sKey + "&q=" + encodedText + "&target=" + to + "&source=" + from + "&format=text";
We are maintaining the Static URL in the xml file the same is getting reading in the java filter file.
URL hit may vary from Locale but currently I want for India location.
I am keep on reloading the same Page but header and Footer URL is getting changed as some other countries URL. Not able to understand how its getting changed.
Below is the code for the filtering from xml file,
try{
Map<String, String> configData = null;
configData = EcomConfigService.getConfigurationDataForLocale(
httpReq, locale);
// Here Locale will be loaded Example: INDIA as the country
String proxy = "";
String port = "";
String gwtShfFlag = "";
int shfTimeOutValue;
if (configData != null) {
proxy = configData.get("proxy");
port = configData.get("port");
gwtShfFlag = configData.get("gwtShfFlag");
//Setted the gwtShfFlag as "TRUE"
shfTimeOutValue = Integer.parseInt(configData.get("shf_timeout"));
logger.info("gwtShfFlag:: " + gwtShfFlag);
logger.info("SHF Timeout Value in Milliseconds:: " + shfTimeOutValue);
if(gwtShfFlag != null && gwtShfFlag.equalsIgnoreCase("true")){
HEADER_URL=configData.get("gwt_shared_header");//Loading the Header URL
FOOTER_URL=configData.get("gwt_shared_footer"); //Loading the Footer URL
HEADER_RESPONSIVE_URL=configData.get("gwt_shared_responsive_header"); //Static Header URL from GWT for INDIA
FOOTER_RESPONSIVE_URL=configData.get("gwt_shared_responsive_footer"); //Static Footer URL from GWT for INDIA
Proxy proxyTemp = new Proxy(Proxy.Type.HTTP, new InetSocketAddress(
proxy, Integer.parseInt(port)));
// Open connection and get response for Header.
URL urlH = new URL(HEADER_URL);
URLConnection conH = urlH.openConnection(proxyTemp);
conH.setConnectTimeout(shfTimeOutValue);
conH.setReadTimeout(shfTimeOutValue);
InputStream isH = conH.getInputStream();
BufferedReader brH = new BufferedReader(new InputStreamReader(isH));
StringBuilder sbH = new StringBuilder();
while ((inputLineH = brH.readLine()) != null) {
sbH.append(inputLineH);
}
// Open connection and get response for Footer.
URL urlF = new URL(FOOTER_URL);
URLConnection conF = urlF.openConnection(proxyTemp);
conF.setConnectTimeout(shfTimeOutValue);
conF.setReadTimeout(shfTimeOutValue);
InputStream isF = conF.getInputStream();
BufferedReader brF = new BufferedReader(new InputStreamReader(isF));
StringBuilder sbF = new StringBuilder();
while ((inputLineF = brF.readLine()) != null) {
sbF.append(inputLineF);
}
URL urlRH = new URL(HEADER_RESPONSIVE_URL);
URLConnection conRH = urlRH.openConnection(proxyTemp);
conRH.setConnectTimeout(shfTimeOutValue);
conRH.setReadTimeout(shfTimeOutValue);
InputStream isRH = conRH.getInputStream();
BufferedReader brRH = new BufferedReader(new InputStreamReader(isRH));
StringBuilder sbRH = new StringBuilder();
while ((inputLineRH = brRH.readLine()) != null) {
sbRH.append(inputLineRH);
}
URL urlRF = new URL(FOOTER_RESPONSIVE_URL);
URLConnection conRF = urlRF.openConnection(proxyTemp);
conRF.setConnectTimeout(shfTimeOutValue);
conRF.setReadTimeout(shfTimeOutValue);
InputStream isRF = conRF.getInputStream();
BufferedReader brRF = new BufferedReader(new InputStreamReader(isRF));
StringBuilder sbRF = new StringBuilder();
while ((inputLineRF = brRF.readLine()) != null) {
sbRF.append(inputLineRF);
}
// Set header and footer response in request scope.
httpReq.setAttribute("inputLineH", sbH.toString());
httpReq.setAttribute("inputLineF", sbF.toString());
httpReq.setAttribute("inputLineRH", sbRH.toString());
httpReq.setAttribute("inputLineRF", sbRF.toString());
Locales locales = localeList;
if (locales != null) {
for (Locale local : locales.getLocale()) {
String localeId = local.getLocaleId();
if ((localeId == locale) || (localeId.equals(locale))) {
if(local.getEnabled()){
logger.info("locale_id:: " + localeId);
logger.info("locale Enabled():: " + local.getEnabled());
List<Section> sections = local.getSections().getSection();
logger.info("sections:: " + sections.size());
httpReq.setAttribute("shfSections", sections);
}
}
}
}
isH.close();
brH.close();
isF.close();
brF.close();
isRH.close();
brRH.close();
brRF.close();
isRF.close();
}
}
} catch (Exception exception) {
logger.error("An Exception occured while calling the SHF urls"
+ exception.getMessage());
}
chain.doFilter(request, response);
}
I am trying to send a request to the Grooveshark API using POST Payload and their requested methods, and I have found a problem. Allow me to show you my code first.
public void getResponse() throws Exception
{
if(service.equals("Grooveshark")) link += getHmacMD5(privateGroovesharkKey, jsonInfo.toString());
if(requestedMethod.equals("GET")) infoURL = new URL(link+arguments);
else infoURL = new URL(link);
HttpURLConnection connection = (HttpURLConnection) infoURL.openConnection();
connection.setRequestMethod(requestedMethod);
connection.setRequestProperty("Accept-Charset", "UTF-8");
connection.setDoOutput(true);
connection.setDoInput(true);
connection.setUseCaches(false);
if(service.equals("Grooveshark"))
{
connection.setRequestProperty("Content-Type","application/json");
OutputStream output = connection.getOutputStream();
output.write(jsonInfo.toString().getBytes());
}
else if(requestedMethod.equals("POST") || requestedMethod.equals("PUT"))
{
OutputStream output = connection.getOutputStream();
output.write(arguments.getBytes());
}
connection.connect();
BufferedReader rd = new BufferedReader(new InputStreamReader(connection.getInputStream()));
StringBuilder sb = new StringBuilder();
String line;
while ((line = rd.readLine()) != null)
sb.append(line).append('\n');
setJsonResult(sb.toString());
System.out.println(jsonResult);
jsonFinal = new JSONObject(jsonResult);
connection.disconnect();
}
I have got that code up here in my project, and I can successfully send requested to any API Webservice that uses JSON in their responses. Now there's only a problem: In Android, it does not give me the WHOLE answer. I've tried running the code on a separate Java (no Android) project, and I get the following output. Although, if I run it on Android, the Log shows me the following:
{"header":{"hostname":"RHL073"},"result":{"songs":[{"SongID":5443351,"SongName":"??????\u00b7???? (FINAL FANTASY XII????)","ArtistID":713,"ArtistName":"Final Fantasy","AlbumID":898007,"AlbumName":"Final Fantasy XII Original Soundtrack","CoverArtFilename":"","Popularity":1214500005,"IsLowBitrateAvailable":tr
And it stops on that tr. Has it anything to do with the parsing of the file that I actually apply afterwards? I don't think it is, but just in case, here it is [This is how I call the search, JSONHandler being the object that contains the code provided above]:
public void performSearch() throws Exception
{
JSONObject search = new JSONObject();
search.put("method", method);
JSONObject header = new JSONObject();
header.put("wsKey", key);
JSONObject parameters = new JSONObject();
parameters.put("query", getSearchQuery());
parameters.put("country", "Portugal");
parameters.put("limit", limit);
parameters.put("offset", "");
search.put("header", header);
search.put("parameters", parameters);
JSONHandler jsonHandler = new JSONHandler(link, search, "Grooveshark", "POST", "");
JSONObject finalResult = jsonHandler.getJsonFinal();
JSONArray songs = finalResult.getJSONObject("result").getJSONArray("songs");
ArrayList<Result> allResults = new ArrayList<Result>();
for(int i = 0; i < songs.length(); i++)
{
JSONObject inner = (JSONObject) songs.get(i);
String name = inner.getString("SongName");
int ID = inner.getInt("SongID");
String artist = inner.getString("ArtistName");
Result res = new Result(name, artist, ID);
res.setAlbumName(inner.getString("AlbumName"));
boolean low = inner.getBoolean("IsLowBitrateAvailable");
int bit = 0;
if(low) bit = 1;
else bit = 0;
res.setIsLowBitRateAvailable(bit);
}
setResults(allResults);
}
As you can clearly see, I am using the json.org library. I really don't understand what's the problem here. Has anyone got any idea as to why?