We Keep Coding

Yahoo Finance URL not working (again since 05/2018)

Thanks to this valuable site, I found useful tips since 08/2017 to retrieve cookies and crumbs for Yahoo Finance site in order to solve my bulk quote download problem.
Nevertheless my program (written in Java) doesn't work anymore since end of May 2018.
I get the following error message :
CookieHandler retrieved cookie:
GUCS="AX62rEgH";$Path="/";$Domain=".yahoo.com" Added cookie using
cookie handler getContent on quote failed: java.io.IOException: Server
returned HTTP response code: 401 for URL:
https://query1.finance.yahoo.com/v7/finance/download/AC.PA?period1=1526594400&period2=1527631200&interval=1d&events=history&crumb=null
I think that the crumb search is failing..
FYI : I am a Java programmer "amateur" since 2003
Please advise if anybody knows how to solve this problem

Thanks to Maxzoom and Dave for their prompt answer. I apologize for the lack of details in my question.For that reason I am adding the complete java method I was using successfully until last month, thanks on one hand to the code of Serge dated Aug 27 2017
Since the method is too long in the comment page I will paste in a new answer Here is the method below:
public static String getQuote3(String quoteString, String stock) throws IOException {
int curByte;
char curChar;
String curQuote,z;
boolean priceFlag;
z="rr";
//////////////Search for cookies
try {
CookieManager manager = new CookieManager();
manager.setCookiePolicy(CookiePolicy.ACCEPT_ALL);
CookieHandler.setDefault(manager);
URL quoteURL = new URL("https://fr.finance.yahoo.com/quote"+stock+"/history?p="+stock);
URLConnection con = quoteURL.openConnection();
con.getContent();
// get cookies from underlying CookieStore
CookieStore cookieJar = manager.getCookieStore();
java.util.List <HttpCookie> cookies = cookieJar.getCookies();
for (HttpCookie cookie: cookies) {
System.out.println("CookieHandler retrieved cookie: " + cookie);
}
//now you can search for the crumb in the yahoo site:
String crumb = null;
InputStream inStream = con.getInputStream();
InputStreamReader irdr = new InputStreamReader(inStream);
BufferedReader rsv = new BufferedReader(irdr);
Pattern crumbPattern = Pattern.compile(".*\"CrumbStore\":\\{\"crum\":\"([^\"]+)\"\\}.*");
String line = null;
while (crumb == null && (line = rsv.readLine()) != null) {
Matcher matcher = crumbPattern.matcher(line);
if (matcher.matches() && matcher.group(1).length()< 12)
crumb = matcher.group(1);
if(crumb!= null)
{
System.out.println ("crumb= " + crumb) ;
}
}
rsv.close();
String quoteUrls = quoteString + crumb;
// create cookie
HttpCookie cookie = new HttpCookie("UserName", "John Doe");
// add cookie to CookieStore for a particular URL quoteURL = new URL(quoteUrls);
try {
cookieJar.add(quoteURL.toURI(), cookie);
System.out.println("Added cookie using cookie handler");
} catch(Exception e) {
System.out.println("Unable to set cookie using CookieHandler");
e.printStackTrace();
}
con.connect();
try {
DataInputStream quoteStream = new DataInputStream(quoteURL.openStream());
priceFlag = false;
curQuote = "";
while( (curByte = quoteStream.read()) != -1) {
curChar = (char) curByte;
curQuote += curChar;
}
System.out.println(curQuote);
priceFlagn = true;
return curQuote;
} catch (IOException e) {
System.err.println("getContent on quote failed: " + e);
priceFlagn = false;
}
} catch (MalformedURLException e) {
System.err.println("Yikes. URL exception");
}
return z;
}

How to enable cookies when opening a URL's connection

I am trying to use the following code to get the redirected URL and then do some processing on it. But when I print the redirected link, it goes to a page that informs the absence of cookies. how is it possible to enable cookies while opening the url's connection?
String url = "http://dx.doi.org/10.1137/0210059";
URLConnection con = new URL( url ).openConnection();
con.getInputStream();
String redirctedURL= con.getURL().toString();
System.out.println(redirctedURL);

When using java UrlConnection's, you should handle the cookies by yourself, to read and set the cookies, you can use the setRequestProperty() and getHeaderField() of URLConnection.
The remaining part is parsing the cookies by yourself, an example how the can be done is as follows:
Map<String, String> cookie = new HashMap<>();
public URLConnection doConnctionWithCookies(URL url) {
StringBuilder builder = new StringBuilder();
builder.append("&");
for(Map.Entry<String,String> entry : cookie.entrySet()) {
builder.append(urlenEncode(entry.getKey()))
.append("=")
.append(urlenEncode(entry.getValue()))
.append("&");
}
builder.setLength(builder.length() - 1);
URLConnection con = url.openConnection();
con.setRequestProperty("Cookie", builder.toString());
con.connect();
// Parse cookie headers
List<String> setCookie = con.getHeaderFields().get("set-cookie");
// HTTP Spec state header fields MUST be case INsentive, I expet the above to work with all servers
if(setCookie == null)
return con;
// Parse all the cookies
for (String str : setCookie) {
String[] cookieKeyValue = str.split(";")[0].split("=",2);
if (cookieKeyValue.length != 2) {
continue;
}
cookie.put(urlenDecode(cookieKeyValue[0]), urlenDecode(cookieKeyValue[1]));
}
return con;
}
public String urlenEncode(String en) {
return URLEncoder.encode(en, "UTF-8");
}
public String urlenDecode(String en) {
return URLDecoder.decode(en, "UTF-8");
}
The above implementation is a very stupid and crude way of implementation cookies, while it works, it totally ignores the fact that cookies can also have a host parameter to prevent identification cookies be shared across multiple hosts.
A better way than doing it yourself can be using a library dedicated for the task like Apache HttpClient.

how to form URL to make HTTP request in android?

I want to make a HTTP request to the url below.
http://192.168.1.60/api/check/merge?nodeid=2&tickets=2-0011,2-0010&saleId=140708120131003102,140708115753005302&firsttableid=1&layoutid=1&userid=1
I stored the tickets details and sales id details in the string array.
String tickets has {2-0011,2-0010} and String saleid has {140708120131003102, 140708115753005302}
How to form the URL like this by using that string array?
String mergeurl1 = "http://192.168.1.60/api/check/merge?nodeid=2&tickets=";
try
{
for(int i=0;i<tablenameurl.length;i++)
{
//System.out.println(ticketidurl[i]+"**"+salesidurl[i]);
if(tablenameurl.length==i && tablenameurl != null)
{
mergeurl1=mergeurl1+ticketidurl[i]+"&";
}
else
{
mergeurl1=mergeurl1+ticketidurl[i]+",";
}
}
mergeurl1=mergeurl1+"saleId=";
for(int i=0;i<tablenameurl.length;i++)
{
//System.out.println(ticketidurl[i]+"**"+salesidurl[i]);
if(tablenameurl.length==i)
{
mergeurl1=mergeurl1+salesidurl[i]+"&";
}
else
{
mergeurl1=mergeurl1+salesidurl[i]+",";
}
}
mergeurl1=mergeurl1+"&firsttableid=1&layoutid=1&userid=1";
System.out.println("URL");
System.out.println(mergeurl1);
}
catch(Exception e)
{
}

You have to use a URL Encoder for this purpose. The best way to do that would be to URLEncode and encode parts of the url. Dont encode the complete URL, it wont work then. You only have to encode the parts of the url that need to be passed as values. For example in this case encode 2-0011,2-0010

OK, assuming you have the URL String mergeurl1
URL url = new URL(mergeurl1);
HttpURLConnection urlConnection = (HttpURLConnection) url.openConnection();
If you want to obtain the response ...
InputStream in = new BufferedInputStream(urlConnection.getInputStream());
//go ahead to process your information with the inputstream
Finally close the HttpURLConnection by urlConnection.disconnect();
Please note that all the code the above cannot be run in your UI Thread, you may use a AsyncTask to cater that.
Here is the reference for you to learn about that:)
http://developer.android.com/reference/android/os/AsyncTask.html
http://developer.android.com/reference/java/net/HttpURLConnection.html
By the way, as the comment above mentioned, you better use a StringBuffer to append the String

Passing value from applet to PHP

I want to send values of two variables to a PHP file from a Java applet), and I tried the following code.
try {
URL url = new URL(getCodeBase(),"abc.php");
URLConnection con = url.openConnection();
con.setDoOutput(true);
PrintStream ps = new PrintStream(con.getOutputStream());
ps.print("score="+score);
ps.print("username="+username);
con.getInputStream();
ps.close();
} catch (Exception e) {
g.drawString(""+e, 200,100);
}
I got the following error:
java.net.UnknownServiceException:protocol doesn't support output

java.net.UnknownServiceException:protocol doesn't support output
Means that you are using a protocol that doesn't support output.
getCodeBase() refers to a file url, so something like
file:/path/to/the/applet
The protocol is file, which doesn't support outout. You are looking for a http protocol, which supports output.
Maybe you wanted getDocumentBase(), which actually returns the web page where the applet is, i.e.
http://www.path.to/the/applet

Here's some code I used with my own applet, to send values (via POST) to a PHP script on my server:
I would use it like this:
String content = "";
content = content + "a=update&gid=" + gid + "&map=" + getMapString();
content = content + "&left_to_deploy=" + leftToDeploy + "&playerColor=" + playerColor;
content = content + "&uid=" + uid + "&player_won=" + didWin;
content = content + "&last_action=" + lastActionCode + "&appletID=" + appletID;
String result = "";
try {
result = requestFromDB(content);
System.out.println("Sending - " + content);
} catch (Exception e) {
status = e.toString();
}
As you can see, I am adding up all my values to send into a "content" string, then calling my requestFromDB method (which posts my "request" values, and returns the server's response) :
public String requestFromDB(String request) throws Exception
{
// This will accept a formatted request string, send it to the
// PHP script, then collect the response and return it as a String.
URL url;
URLConnection urlConn;
DataOutputStream printout;
DataInputStream input;
// URL of CGI-Bin script.
url = new URL ("http://" + siteRoot + "/globalconquest/applet-update.php");
// URL connection channel.
urlConn = url.openConnection();
// Let the run-time system (RTS) know that we want input.
urlConn.setDoInput (true);
// Let the RTS know that we want to do output.
urlConn.setDoOutput (true);
// No caching, we want the real thing.
urlConn.setUseCaches (false);
// Specify the content type.
urlConn.setRequestProperty("Content-Type", "application/x-www-form-urlencoded");
// Send POST output.
printout = new DataOutputStream (urlConn.getOutputStream ());
printout.writeBytes (request);
printout.flush ();
printout.close ();
// Get response data.
input = new DataInputStream (urlConn.getInputStream ());
String str;
String a = "";
while (null != ((str = input.readLine())))
{
a = a + str;
}
input.close ();
System.out.println("Got " + a);
if (a.trim().equals("1")) {
// Error!
mode = "error";
}
return a;
} // requestFromDB
In my PHP script, I would only need to look at $_POST for my values. Then I would just print a response.
Note! Your PHP script MUST be on the same server as the applet for security reasons, or this will not work.

Using Java to send data to a form on a website hosted locally

I have a program in Java where I retrieve contents from a database.
Now I have a form in the program, and what I want to do is, on the press of a button, some string (text) content retrieved from the database, should be sent over to a website that I'm hosting locally. The content so sent, should be displayed on the website when refreshed.
Can someone guide me as to how I can achieve this (the sending of data to be displayed over the website)?
Will appreciate a lot, if you could kindly show some sample snippets or give me a reference to some tutorial that can help.
---- Okay so i found a link to a snippet that's supposed to do this, but im unable to understand at this stage as to how exactly this snippet works...can someone please guide me into knowing this better ?
here's the code
try {
// Construct data
String data = URLEncoder.encode("key1", "UTF-8") + "=" + URLEncoder.encode("value1", "UTF-8");
data += "&" + URLEncoder.encode("key2", "UTF-8") + "=" + URLEncoder.encode("value2", "UTF-8");
// Send data
URL url = new URL("http://hostname:80/cgi");
URLConnection conn = url.openConnection();
conn.setDoOutput(true);
OutputStreamWriter wr = new OutputStreamWriter(conn.getOutputStream());
wr.write(data);
wr.flush();
// Get the response
BufferedReader rd = new BufferedReader(new InputStreamReader(conn.getInputStream()));
String line;
while ((line = rd.readLine()) != null) {
// Process line...
}
wr.close();
rd.close();
} catch (Exception e) {
}

I'm not sure on how you store and manage any of the records but from Java you can send a HTTP Post to the Url (In your case http://localhost/, probably).
Have a look at http://www.exampledepot.com/egs/java.net/post.html for a snippet on how to do this.
Your Website could then store the received information in a database and display it when you refresh.
Update heres the function
Just a side not this is by no means the best way to do this and I have no idea on how this scales but for simple solutions this has worked for me in the past.
/**
* Posts a Set of forms variables to the Remote HTTP Host
* #param url The URL to post to and read
* #param params The Parameters to post to the remote host
* #return The Content of the remote page and return null if no data was returned
*/
public String post(String url, Map<String, String> params) {
//Check if Valid URL
if(!url.toLowerCase().contains("http://")) return null;
StringBuilder bldr = new StringBuilder();
try {
//Build the post data
StringBuilder post_data = new StringBuilder();
//Build the posting variables from the map given
for (Iterator iter = params.entrySet().iterator(); iter.hasNext();) {
Map.Entry entry = (Map.Entry) iter.next();
String key = (String) entry.getKey();
String value = (String)entry.getValue();
if(key.length() > 0 && value.length() > 0) {
if(post_data.length() > 0) post_data.append("&");
post_data.append(URLEncoder.encode(key, "UTF-8"));
post_data.append("=");
post_data.append(URLEncoder.encode(value, "UTF-8"));
}
}
// Send data
URL remote_url = new URL(url);
URLConnection conn = remote_url.openConnection();
conn.setDoOutput(true);
OutputStreamWriter wr = new OutputStreamWriter(conn.getOutputStream());
wr.write(post_data.toString());
wr.flush();
// Get the response
BufferedReader rd = new BufferedReader(new InputStreamReader(conn.getInputStream()));
String inputLine;
while ((inputLine = rd.readLine()) != null) {
bldr.append(inputLine);
}
wr.close();
rd.close();
} catch (Exception e) {
//Handle Error
}
return bldr.length() > 0 ? bldr.toString() : null;
}
You would then use the function as follows:
Map<String, String> params = new HashMap<String, String>();
params.put("var_a", "test");
params.put("var_b", "test");
params.put("var_c", "test");
String reponse = post("http://localhost/", params);
if(reponse == null) { /* error */ }
else {
System.out.println(reponse);
}

The big question is how will you authenticate the "update" from your Java program to your website?
You could easily write a handler on your website, say "/update" which saves the POST body (or value of a request parameter) to a file or other persistent store but how will you be sure that only you can set that value, instead of anybody who discovers it?