I want to search the following in Google query box:
http://www.cmu.edu/silicon-valley/ faculty directory
Unfortunately, the following code does not work:
Jsoup.connect("http://www.google.com/search?hl=en&q=http%3A%%2F%%2F%www.cmu.edu%2F%silicon-valley%2F%20faculty20directory").get();
nor does this one:
Jsoup.connect("http://www.google.com/search?hl=en&q=http%3A%%2F%%2F%www.cmu.edu%2F%silicon-valley%2F%20faculty20or20directory").get();
What am I missing here?
Edit: not working means Google didn't return any result as we see from browser.
Jsoup.connect("http://www.google.com/search?hl=en&q=http%3A%"%2F%%2F%www.cmu.edu%2F%silicon-valley%2F%20faculty").get();
The code above works though. It's equivalent to Googling "http://www.cmu.edu/silicon-valley/ faculty".
Edit: I have the following trick in my program, so bot-rule is not an issue:
.userAgent("Mozilla")
Jsoup.connect("http://www.google.com/search?hl=en&q=http%3A%2F%2Fwww.cmu.edu%2Fsilicon-valley%2F+faculty+directory") leads to a 403 error (Forbidden) as google forbis robots to access its results
You'll have to change the User Agent String if you want to do that
doc = Jsoup.connect("http://www.google.com/search?hl=en&q=http%3A%2F%2Fwww.cmu.edu%2Fsilicon-valley%2F+faculty+directory").header("User-Agent", "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.17 (KHTML, like Gecko) Chrome/24.0.1312.52 Safari/537.17").get() should work as expected, but could be against Google's Terms of Use.
Related
I am working on Yahoo stock data. Yesterday I got the stock data by using finance web service api. But today when I am trying to get the data from api I am getting the below error:
{
"p": {
"a": {
"href": "https://finance.yahoo.com/webservice/v1/symbols/msft,goog,appl,orcl,yhoo,tcs,amzn,INFY.NS/quote?bypass=true&format=json&view=detail",
"content": "https://finance.yahoo.com/webservice/v1/symbols/msft,goog,appl,orcl,yhoo,tcs,amzn,INFY.NS/quote?bypass=true&format=json&view=detail"
},
"content": "Moved Temporarily. Redirecting to"
}
}
Saying that it was moved temporarily.
Why am I getting this error? Did I reach the API limit for today?
NOTE:
Yesterday I kept it running to test the API request limit. But when I am trying to run today it showing the above error.
If the API limit for IP is reached then when do I get access to the data again?
This is the API which I am using:
http://finance.yahoo.com/webservice/v1/symbols/msft,goog,appl,orcl,yhoo,tcs,amzn,INFY.NS/quote?format=json&view=detail
As it was commented here: https://stackoverflow.com/a/38390559/6586718, you have to change the user-agent to a mobile device.
On Java, I do the following, and it's working (this is for XML, but the same can be applied to JSON):
URL url = new URL ("https://finance.yahoo.com/webservice/v1/symbols/" + stocks + "/quote");
HttpURLConnection urlc = (HttpURLConnection) url.openConnection ();
urlc.setRequestProperty ("User-Agent", "Mozilla/5.0 (Linux; Android 6.0; MotoE2(4G-LTE) Build/MPI24.65-39) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.81 Mobile Safari/537.36");
Document xml = DocumentBuilderFactory.newInstance ().newDocumentBuilder ().parse (urlc.getInputStream ());
try with this new one..
https://query.yahooapis.com/v1/public/yql?q=select%20*%20from%20csv%20where%20url%3D'http%3A%2F%2Fdownload.finance.yahoo.com%2Fd%2Fquotes.csv%3Fs%3DAAPL%26f%3Dsl1d1t1c1ohgv%26e%3D.csv'%20and%20columns%3D'symbol%2Cprice%2Cdate%2Ctime%2Cchange%2Ccol1%2Chigh%2Clow%2Ccol2'&format=json&env=store%3A%2F%2Fdatatables.org%2Falltableswithkeys
i create a program that take in input some information and search for the first 10 video on youtube, my problem is that the v2 version of Youtube API isn't support, so i change that
https://gdata.youtube.com/feeds/api/videos?q=
with new version,but i dont find that. Can you help me please?
Thanks a lot
EDIT
Thanks a lot , i have tried to do this command
https://www.googleapis.com/youtube/v3/search?part=snippet&maxResults=10&q=skyrim&key={YOUR_API_KEY}
i want search the first 10 video related to the keyword Skyrim.
I try this using eclipse and i have this error
Exception in thread "main" org.jsoup.HttpStatusException: HTTP errorfetching URL. Status=400, URL=https://www.googleapis.com/youtube/v3/search?part=snippet&maxResults=10&key={YOUR_API_KEY}&q=skyrim&max-results=10
at org.jsoup.helper.HttpConnection$Response.execute(HttpConnection.java:590)
at org.jsoup.helper.HttpConnection$Response.execute(HttpConnection.java:540)
at org.jsoup.helper.HttpConnection.execute(HttpConnection.java:227)
at org.jsoup.helper.HttpConnection.get(HttpConnection.java:216)
I run the query in this mode :
private static String QueryURL ="https://www.googleapis.com/youtube/v3/search?part=snippet&maxResults=10&key={YOUR_API_KEY}&q=";
Document doc = Jsoup.connect(QueryURL+stringa).userAgent("Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.94 Safari/537.36").get();
Where stringa is = Skyrim.
Thanks all for help
Start at the YouTube API v3 page: https://developers.google.com/youtube/v3/
I am trying to check if a service is available and always returns the same error:
java.io.IOException: Server returned HTTP response code: 403 for URL
Internet browsing proposed that it was necessary to indicate the "USER-AGENT" and so I did, but the error remains the same:
openConnection.addRequestProperty("User-Agent", "Mozilla / 5.0 (Windows NT 6.1; WOW64) AppleWebKit / 537.36 (KHTML, like Gecko) Chrome / 40.0.2214.91 Safari / 537.36");
The complete code is:
url = cadenaURL + cadenaEndpoint;
URLConnection openConnection = new URL(url).openConnection();
openConnection.connect();
is = openConnection.getInputStream();
if ("gzip".equals(openConnection.getContentEncoding()))
is = new GZIPInputStream(is);
and the error is in:
is = openConnection.getInputStream();
Someone could help me?
Thank You,
a greeting,
I am trying to check if a service is available
Simply put returning 403 means the service is not available. There's your answer.
There's a possibility you're using the wrong url and attempting to connect to the wrong service.
However it seems also that
is = openConnection.getInputStream();
This may not be enough to connect to an HTTP server. You need to properly format the request, you need to show more of the code and how you're using it for us to help you more.
Finally I changed URLConnection by HttpURLConnection and works perfect.
Thank You
I want to use Jsoup to access some data of a website located on a network server. Everytime I try to connect via a valid URL I'm retrieving a HttpStatusException with the following error:
Exception in thread "main" org.jsoup.HttpStatusException: HTTP error fetching URL. Status=500, URL=http://sv.thisismydomain.de/path/xyz.jsp (I've changed the URL)
This is my attempt:
System.out.println(Jsoup.connect(urlBase + urlLoginForm).userAgent(userAgent).timeout(10000).get().html());
I'm sure that this is the correct URL. The URL works fine if I copy it out of the StackTrace into my browser - so this can't be the problem.
This is the user agent I'm using:
String userAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) " +
"Chrome/30.0.1599.101 Safari/537.36";
Do you have any ideas? This drives me crazy!
Status 500 is a permanant error. It means the server encountered an unexpected condition which prevented it from fulfilling the request.No way around it other than handlin it in server. Since you are saying the url is working perfectly there can we certain possibilites that we could re-check. These may not exactly be a reason of 505.
1) When you say urlBase + urlLoginForm there could be chance to miss a \. Say you have urlBase = http://sv.thisismydomain.de/path and urlLoginForm = xyz.jsp when you construct it could be http://sv.thisismydomain.de/pathxyz.jsp instead of http://sv.thisismydomain.de/path/xyz.jsp
If urlLoginForm is a parameter list you should re-check how it is constructed.
**This should ideally return a 404 but since the domain part is correct chances are there it can explicitly fail with a 505.
2) The site you are trying to might be checking the source of the request. So you could rely on the referrer method of Jsoup in this case.
Document doc = Jsoup.connect(urlBase + urlLoginForm).referrer(urlBase + urlLoginForm).userAgent(userAgent).timeout(10000).get();
** ideally this should return a Forbidden 403 error or access denied.
3) Make sure get method is supported. try using post. Again this should return a Method 303, but just in case.. ;)
4) The URL doesn't show any issue. Since its behind a proxy you could try setting proxy properties before invoking jsoup.connect(). But again this should result in time out and not 505.
System.setProperty("http.proxyHost", "<your host ip>");
System.setProperty("http.proxyPort", "<proxy port>");
Sorry to give all these suggestion which are unrelated to 505. Since I don't have access to your URL this is the best I could suggest. :)
I'm developing an application which will allow users to sign in with their Facebook account. Currently, I'm using Eclipse SWT browser to display the login-browsing window, which works great. However, I'd prefer if this web-browsing window got treated (and therefore styled) as an iPhone. I've tried to force the HTTP-headers to mimic the iPhone's; without any change.
My current code looks somewhat like this:
String[] headers = { "Mozilla/5.0 (iPhone; U; CPU like Mac OS X; en) AppleWebKit/420+ (KHTML, like Gecko) Version/3.0 Mobile/1A543a Safari/419.3" };
browser.setUrl("https://m.facebook.com/dialog/oauth?client_id=appId&redirect_uri=url&state=state", "", headers);
However, the Facebook website STILL realizes that I'm not an iPhone, which makes the output looks like this:
Whilist I'm looking for this design: