We Keep Coding

Yahoo Finance URL not working (again since 05/2018)

Thanks to this valuable site, I found useful tips since 08/2017 to retrieve cookies and crumbs for Yahoo Finance site in order to solve my bulk quote download problem.
Nevertheless my program (written in Java) doesn't work anymore since end of May 2018.
I get the following error message :
CookieHandler retrieved cookie:
GUCS="AX62rEgH";$Path="/";$Domain=".yahoo.com" Added cookie using
cookie handler getContent on quote failed: java.io.IOException: Server
returned HTTP response code: 401 for URL:
https://query1.finance.yahoo.com/v7/finance/download/AC.PA?period1=1526594400&period2=1527631200&interval=1d&events=history&crumb=null
I think that the crumb search is failing..
FYI : I am a Java programmer "amateur" since 2003
Please advise if anybody knows how to solve this problem

Thanks to Maxzoom and Dave for their prompt answer. I apologize for the lack of details in my question.For that reason I am adding the complete java method I was using successfully until last month, thanks on one hand to the code of Serge dated Aug 27 2017
Since the method is too long in the comment page I will paste in a new answer Here is the method below:
public static String getQuote3(String quoteString, String stock) throws IOException {
int curByte;
char curChar;
String curQuote,z;
boolean priceFlag;
z="rr";
//////////////Search for cookies
try {
CookieManager manager = new CookieManager();
manager.setCookiePolicy(CookiePolicy.ACCEPT_ALL);
CookieHandler.setDefault(manager);
URL quoteURL = new URL("https://fr.finance.yahoo.com/quote"+stock+"/history?p="+stock);
URLConnection con = quoteURL.openConnection();
con.getContent();
// get cookies from underlying CookieStore
CookieStore cookieJar = manager.getCookieStore();
java.util.List <HttpCookie> cookies = cookieJar.getCookies();
for (HttpCookie cookie: cookies) {
System.out.println("CookieHandler retrieved cookie: " + cookie);
}
//now you can search for the crumb in the yahoo site:
String crumb = null;
InputStream inStream = con.getInputStream();
InputStreamReader irdr = new InputStreamReader(inStream);
BufferedReader rsv = new BufferedReader(irdr);
Pattern crumbPattern = Pattern.compile(".*\"CrumbStore\":\\{\"crum\":\"([^\"]+)\"\\}.*");
String line = null;
while (crumb == null && (line = rsv.readLine()) != null) {
Matcher matcher = crumbPattern.matcher(line);
if (matcher.matches() && matcher.group(1).length()< 12)
crumb = matcher.group(1);
if(crumb!= null)
{
System.out.println ("crumb= " + crumb) ;
}
}
rsv.close();
String quoteUrls = quoteString + crumb;
// create cookie
HttpCookie cookie = new HttpCookie("UserName", "John Doe");
// add cookie to CookieStore for a particular URL quoteURL = new URL(quoteUrls);
try {
cookieJar.add(quoteURL.toURI(), cookie);
System.out.println("Added cookie using cookie handler");
} catch(Exception e) {
System.out.println("Unable to set cookie using CookieHandler");
e.printStackTrace();
}
con.connect();
try {
DataInputStream quoteStream = new DataInputStream(quoteURL.openStream());
priceFlag = false;
curQuote = "";
while( (curByte = quoteStream.read()) != -1) {
curChar = (char) curByte;
curQuote += curChar;
}
System.out.println(curQuote);
priceFlagn = true;
return curQuote;
} catch (IOException e) {
System.err.println("getContent on quote failed: " + e);
priceFlagn = false;
}
} catch (MalformedURLException e) {
System.err.println("Yikes. URL exception");
}
return z;
}

how to send file with httprequest in java?

how to send file with http request in java ?
i see question
Send image file using java HTTP POST connections
and
Upload files from Java client to a HTTP server
but them are too old question and not work any more .

A good example is given in Multi part File Upload example of Apache HttpClient
http://svn.apache.org/viewvc/httpcomponents/oac.hc3x/trunk/src/examples/MultipartFileUploadApp.java?view=markup
The part that actually posts the file is
String targetURL = cmbURL.getSelectedItem().toString();
// add the URL to the combo model if it's not already there
if (!targetURL
.equals(
cmbURLModel.getElementAt(
cmbURL.getSelectedIndex()))) {
cmbURLModel.addElement(targetURL);
}
PostMethod filePost = new PostMethod(targetURL);
filePost.getParams().setBooleanParameter(HttpMethodParams.USE_EXPECT_CONTINUE,
cbxExpectHeader.isSelected());
try {
appendMessage("Uploading " + targetFile.getName() + " to " + targetURL);
Part[] parts = {
new FilePart(targetFile.getName(), targetFile)
};
filePost.setRequestEntity(
new MultipartRequestEntity(parts, filePost.getParams())
);
HttpClient client = new HttpClient();
client.getHttpConnectionManager().
getParams().setConnectionTimeout(5000);
int status = client.executeMethod(filePost);
if (status == HttpStatus.SC_OK) {
appendMessage(
"Upload complete, response=" + filePost.getResponseBodyAsString()
);
} else {
appendMessage(
"Upload failed, response=" + HttpStatus.getStatusText(status)
);
}
} catch (Exception ex) {
appendMessage("ERROR: " + ex.getClass().getName() + " "+ ex.getMessage());
ex.printStackTrace();
} finally {
filePost.releaseConnection();
}
Hope it is of some help to you.

Why did my application stop connecting using HTTP, and how can I debug it?

I have a program that includes HTTP connection to a PLM application that runs on SQL Server. The program is scheduled to run daily. It collects data from few sources, then issues a query to the PLM to store the data, and finally reads the PLM's reply to verify if the data was properly stored.
The application ran OK, until we upgraded both the DB (into SQL Server 2012) and the PLM.
Since then the upgrade, when the program establishes the connection it receives OK status; however, the data setting query does not affect the data base, and there is no answer received. There are no error messages - just malfunction.
My major question is - how to debug it. I know whet I send and what I receive. How can I get more data on what happens in between?
I attach the code for review. What I didn't add here is the query itself, which is WML-like string. The PLM should fire an answer regardless the query it receives, even if it is an error message. However, I get only NULL.
public Boolean amlArasCommunication (String data , int targetDbType, String passWord)
{
final String url = "http://plm-srv/InnovatorServer/Server/InnovatorServer2012.aspx";
final String schemeUrl = "'http://schemas.xmlsoap.org/soap/envelope/'";
String answer =””;
String dataBase = data base name;
Writer wout;
HttpURLConnection amlConnection = null;
try
{
// instantiate the HttpURLConnection with the URL object - A new connection is
// opened every time by calling the openConnection method of the protocol
// handler for this URL. This is the point where the connection is opened.
amlConnection = (HttpURLConnection) new URL(url).openConnection();
amlConnection.setDoOutput(true);
amlConnection.setDoInput(true);
amlConnection.setRequestMethod("POST");
amlConnection.setRequestProperty("SOAPAction", "ApplyAML");
amlConnection.setConnectTimeout(10000);
amlConnection.setReadTimeout(10000);
amlConnection.setRequestProperty("AUTHUSER", "Admin");
amlConnection.setRequestProperty("AUTHPASSWORD", calcMD5(passWord));
amlConnection.setRequestProperty("DATABASE", dataBase);
String query = "<?xml version='1.0'?>\r\n" +
"<SOAP-ENV:Envelope xmlns:SOAP-ENV=" + schemeUrl + ">\r\n" +
" <SOAP-ENV:Body>\r\n" +
data +
" </SOAP-ENV:Body>\r\n" +
"</SOAP-ENV:Envelope>\r\n";
// instantiate OutputStreamWriter using the output stream, returned from getOutputStream, that writes
// to this connection. If an I/O error occurs while creating the output stream, IOException will be fired.
wout = new OutputStreamWriter(amlConnection.getOutputStream());
wout.write (query);
wout.close();
// At this point, we've sent all the data. The outputStream was closed, while the connection is still open
int result;
if ((result = amlConnection.getResponseCode()) == HttpURLConnection.HTTP_OK)
{
// Get the communication results from the PLM
InputStream ac = amlConnection.getInputStream();
InputStreamReader isr = new InputStreamReader (ac);
BufferedReader in = new BufferedReader(isr);
String readResult = in.readLine ();
int count = 0;
while (readResult != null)
{
answer += readResult + "\n";
readResult = in.readLine ();
count++;
}
in.close();
if (answer.contains("fault"))
System.out.println ("Error message: " + answer + "\nQuery: " + query);
else
log.message ("Lines count=" + count + "; com status=" + result + "; reply: " + answer, false);
}
else
// Error code is returned, or no status code is returned, do stuff in the else block
System.out.println("Connection failed with the following code: " + result);
}
catch (IOException e) { ; }
if (amlConnection != null)
amlConnection.disconnect ();
return true;
}

I think you will have to look into the log files of the PLM application to find out why you do not get an HTTP response. There might be a number of possible reasons why the application is not working anymore after the upgrade.
I guess that it will be difficult to debug the problem based on the client code only. As the server seems to accept your HTTP, I would expect that this event and errors would be written to a log file somewhere. You might also want to try some graphical tool like SOAP UI to test the SOAP service.

Generating Http 301 Response Message

I'm trying to create a little web server and have be able to generate Http response messages 200, 301, and 404.
I am able to get 200 and 404 to work, but I am having problems with 301.
When I try to access a page that has "permanently moved" my browser doesn't get redirected and I get a java.lang.NullPointerException from java.
The way I have it determine if the code should be a 301 is it checks a list of strings for the file the client is trying to access, and if the original file they're trying to access has been moved, it will be in the list, along with it's new name/location. So if the original file is "index5.html" and it's been moved to "index.html" then they will be in an array and "index5.html" will be in an index 1 before "index.html"
I'm also just testing this on my own machine so I'm using localhost for the URL and using port 9012.
Here is my code:
import java.io.*;
import java.net.*;
import java.util.*;
public final class HttpRequest implements Runnable {
final static String CarrLine = "\r\n";
Socket clientSocket;
// A list of files that have been moved.
// Even indexes (0, 2, 4, ...) are the original file names.
// Odd indexes (1, 3, 5, ...) are where the files of previous indexes moved to.
static String movedFiles[] = {"index5.html", "index.html", "page.html", "homepage.html"};
// This sets the Httprequest object socket equal to
// the socket the client comes in through
public HttpRequest(Socket socket) throws Exception {
this.clientSocket = socket;
}
// Here we define a new method that overwrites the
// previous method in the Runnables class. This is done
// so that when an Http request is attempted, and
// something goes wrong, our whole web server will
// not fail and crash.
#Override
public void run(){
try {
// This is where the method to actually start the Http request starts.
requestProcessing();
} catch (Exception ex) { System.out.print(ex); }
}
// This is our main processing method to take in out Http request
// and spit out a reponse header along with the requested data,
// if there is any.
void requestProcessing() throws Exception {
Boolean fileExists = false;
String CarrLine = "\r\n";
String statusCode = null;
String responseHeader = "HTTP/1.1 ";
String fileName, line = null;
String clientSentence = null;
ArrayList<String> records = new ArrayList<String>();
FileInputStream requestedFileStream = null;
File requestedFile;
// Starts input from client and establishes filters
BufferedReader inFromClient = new BufferedReader(new InputStreamReader(clientSocket.getInputStream()));
// Starts output stream for output to client through socket
DataOutputStream outToClient = new DataOutputStream(clientSocket.getOutputStream());
/*
// Reads in GET from client BufferedReader
while ( (line = inFromClient.readLine()) != null){
records.add(line);
break;
}*/
clientSentence = inFromClient.readLine();
// Parses and stores file name the client wants in a string
fileName = parseGET(clientSentence);
if (!existingFile(fileName)){
// Here is where the 301 response message is generated and
// retrieve the correct filename.
if (hasMoved(fileName) != -1){
statusCode = "301";
responseHeader = responseHeader + statusCode + " Moved Permanently\n";
responseHeader = responseHeader + "Location: localhost:9012/"
+ movedFiles[hasMoved(fileName)] + CarrLine;
}
// This generates the response header for the client
// if the file the client is looking for is not there (404).
else {
statusCode = "404";
responseHeader = responseHeader + statusCode + " Not Found: \n";
responseHeader = responseHeader + "Content-Type: text/html" + CarrLine;
}
}
// This generates the 200 status code response header
// to send to the client saying the file was found.
if (existingFile(fileName)) {
statusCode = "200";
responseHeader = responseHeader + statusCode + " OK: \n";
responseHeader = responseHeader + "Content-Type: " + fileType(fileName) + CarrLine;
requestedFileStream = openFileStream(fileName);
}
// Outputs the response message to the client through a data stream
outToClient.writeBytes(responseHeader);
outToClient.writeBytes(CarrLine);
// If the file the client is requesting exists,
// begin writing file out to client.
if (existingFile(fileName)){
fileWriteOut(requestedFileStream, outToClient);
requestedFileStream.close();
}
else if(hasMoved(fileName) != -1){
outToClient.writeBytes("File Moved");
}
// If the file the client is requesting does not exist,
// return a 404 message.
else {
outToClient.writeBytes("404: File not found!");
}
// Closes all open streams and sockets to the client.
inFromClient.close();
outToClient.close();
clientSocket.close();
}
// This parses the GET line from the client to get the filename the client is requesting
String parseGET(String clientString){
String temp[] = clientString.split(" /");
temp = temp[1].split(" ");
return temp[0];
}
// This is used to find the file the client is requesting.
// It will return null if no file was found/opened.
FileInputStream openFileStream(String file){
FileInputStream fileStream = null;
// Opening the file stream is in a try catch statment so that
// incase there was no file, the program doesn't crash
// and it'll alert the user on the console.
try {
fileStream = new FileInputStream(file);
} catch (FileNotFoundException e) {
System.out.println(e);
return null;
}
return fileStream;
}
// Determines the file type that is being sent to the client
// and returns the appropriate string
String fileType(String clientRequestFile){
// If the file ends in .html or .htm, it will return "text/html"
// so that it can be added to the response message.
if (clientRequestFile.endsWith(".html") || clientRequestFile.endsWith(".htm")){
return "text/html";
}
// If the file ends in .jpg, it will return "text/jpeg"
// so that it can be added to the response message.
if (clientRequestFile.endsWith(".jpg")){
return "text/jpg";
}
// If the file ends in .css, it will return "text/css"
// so that it can be added to the response message.
if (clientRequestFile.endsWith(".css")){
return "text/css";
}
// Returns this by default, if none of the above.
return "application/octet-stream";
}
// This creates a 2k buffer and writes out
// requested filed to the client.
static void fileWriteOut(FileInputStream clientStream, OutputStream toClient) throws Exception{
byte[] buffer = new byte[2048];
int bytes = 0;
while ((bytes = clientStream.read(buffer)) != -1){
toClient.write(buffer, 0, bytes);
}
}
// This determines whether or not a file that
// the client has requested exists or not.
// Returns a Boolean value.
static Boolean existingFile(String fileName){
File file = new File(fileName);
if (file.exists() && !file.isDirectory()){
return true;
}
return false;
}
// Determines if a file has been moved and if so,
// returns the index of the NEW file. Else it
// returns -1.
static int hasMoved(String fileName){
int i = 0;
for (i = 0; i < movedFiles.length; i=i+2){
if (movedFiles[i].equals(fileName)){
return i+1;
}
}
return -1;
}
}
Could someone point me in the right direction to doing this correctly?
Thank you!

Okay, I figured it out.
It was because I was trying to define the entire "URL" in the 301 response messages.
So it should have been:
responseHeader = responseHeader + "Location: /" + movedFiles[hasMoved(fileName)] + CarrLine;
Instead of:
responseHeader = responseHeader + "Location: localhost:9012/" + movedFiles[hasMoved(fileName)] + CarrLine;

Passing value from applet to PHP

I want to send values of two variables to a PHP file from a Java applet), and I tried the following code.
try {
URL url = new URL(getCodeBase(),"abc.php");
URLConnection con = url.openConnection();
con.setDoOutput(true);
PrintStream ps = new PrintStream(con.getOutputStream());
ps.print("score="+score);
ps.print("username="+username);
con.getInputStream();
ps.close();
} catch (Exception e) {
g.drawString(""+e, 200,100);
}
I got the following error:
java.net.UnknownServiceException:protocol doesn't support output

java.net.UnknownServiceException:protocol doesn't support output
Means that you are using a protocol that doesn't support output.
getCodeBase() refers to a file url, so something like
file:/path/to/the/applet
The protocol is file, which doesn't support outout. You are looking for a http protocol, which supports output.
Maybe you wanted getDocumentBase(), which actually returns the web page where the applet is, i.e.
http://www.path.to/the/applet

Here's some code I used with my own applet, to send values (via POST) to a PHP script on my server:
I would use it like this:
String content = "";
content = content + "a=update&gid=" + gid + "&map=" + getMapString();
content = content + "&left_to_deploy=" + leftToDeploy + "&playerColor=" + playerColor;
content = content + "&uid=" + uid + "&player_won=" + didWin;
content = content + "&last_action=" + lastActionCode + "&appletID=" + appletID;
String result = "";
try {
result = requestFromDB(content);
System.out.println("Sending - " + content);
} catch (Exception e) {
status = e.toString();
}
As you can see, I am adding up all my values to send into a "content" string, then calling my requestFromDB method (which posts my "request" values, and returns the server's response) :
public String requestFromDB(String request) throws Exception
{
// This will accept a formatted request string, send it to the
// PHP script, then collect the response and return it as a String.
URL url;
URLConnection urlConn;
DataOutputStream printout;
DataInputStream input;
// URL of CGI-Bin script.
url = new URL ("http://" + siteRoot + "/globalconquest/applet-update.php");
// URL connection channel.
urlConn = url.openConnection();
// Let the run-time system (RTS) know that we want input.
urlConn.setDoInput (true);
// Let the RTS know that we want to do output.
urlConn.setDoOutput (true);
// No caching, we want the real thing.
urlConn.setUseCaches (false);
// Specify the content type.
urlConn.setRequestProperty("Content-Type", "application/x-www-form-urlencoded");
// Send POST output.
printout = new DataOutputStream (urlConn.getOutputStream ());
printout.writeBytes (request);
printout.flush ();
printout.close ();
// Get response data.
input = new DataInputStream (urlConn.getInputStream ());
String str;
String a = "";
while (null != ((str = input.readLine())))
{
a = a + str;
}
input.close ();
System.out.println("Got " + a);
if (a.trim().equals("1")) {
// Error!
mode = "error";
}
return a;
} // requestFromDB
In my PHP script, I would only need to look at $_POST for my values. Then I would just print a response.
Note! Your PHP script MUST be on the same server as the applet for security reasons, or this will not work.