I would like to write java application using jcef3 and emulate that as mobile browser.
I do somethink like that:
#Override
public boolean onBeforeResourceLoad(CefBrowser cefBrowser, CefRequest cefRequest) {
Map<String, String> headerMap = new HashMap<String, String>();
Map<String, String> newHeaderMap = new HashMap<String, String>();
cefRequest.getHeaderMap(headerMap);
for (Map.Entry<String, String> entry : headerMap.entrySet())
{
if (entry.getKey().equals("User-Agent")) {
newHeaderMap.put(entry.getKey(), "Mozilla/5.0 (Linux; Android 4.3; GT-I9300 Build/JSS15J) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/38.0.2125.114 Mobile Safari/537.36");
} else {
newHeaderMap.put(entry.getKey(), entry.getValue());
}
}
cefRequest.setHeaderMap(newHeaderMap);
return false;
}
Problem is with that one page that I would like browse like mobile detect that is some strange OS :) and I don't know why. If I download some plugin form chrome like user agent switcher and use the same UA then everything is ok, so I'm sure that that page use user agent verification.
Any idea?
Well, I solved that problem.
Generaly there are two ways to detect browser.
First one is request headers. My sample code show how to do that.
The secound one is javascript navigator property. That was the problem. Requests were sended with correct userAgent but javascript navigator.userAgent show bad user agent version that was different with that sended in request headers.
To do that correctly You should override javascript navigator.userAgent variable.
In CEF3 there is no problem with that to inject some javascript code that override that.
Related
Is there any way to exactly implement a "desktop mode" function into my mobile app when loading website content? I'm making a android app, and I want to have a page just with static website content just like in a web browser, I really like how mobile opera implemented this feature, So I just want to know if there is a way to do that. And if there is, then How?
(I'm making this project in Java and viewing the website using WebView from the library "WebKit")
I've also tried changing the User Agent, which didn't work on a static website.
This method helps you to set DesktopMode on webview
public void setDesktopMode(WebView webView,boolean enabled) {
String newUserAgent = webView.getSettings().getUserAgentString();
if (enabled) {
newUserAgent = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_5)\nAppleWebKit/605.1.15 (KHTML, like Gecko) CriOS/85\nVersion/11.1.1 Safari/605.1.15";
}
webView.getSettings().setUserAgentString(newUserAgent);
webView.getSettings().setUseWideViewPort(enabled);
webView.getSettings().setLoadWithOverviewMode(enabled);
webView.reload();
}
Call it like that
Mobile mode : setDesktopMode(webView, false);
Desktop mode : setDesktopMode(webView, true);
For Kotlin:
fun setDesktopMode(webView: WebView, enabled: Boolean) {
var newUserAgent: String? = webView.settings.userAgentString
if (enabled) {
newUserAgent = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_5)\nAppleWebKit/605.1.15 (KHTML, like Gecko) CriOS/85\nVersion/11.1.1 Safari/605.1.15"
}
webView.settings.apply {
userAgentString = newUserAgent
useWideViewPort = enabled
loadWithOverviewMode = enabled
}
webView.reload()
}
I am working on Yahoo stock data. Yesterday I got the stock data by using finance web service api. But today when I am trying to get the data from api I am getting the below error:
{
"p": {
"a": {
"href": "https://finance.yahoo.com/webservice/v1/symbols/msft,goog,appl,orcl,yhoo,tcs,amzn,INFY.NS/quote?bypass=true&format=json&view=detail",
"content": "https://finance.yahoo.com/webservice/v1/symbols/msft,goog,appl,orcl,yhoo,tcs,amzn,INFY.NS/quote?bypass=true&format=json&view=detail"
},
"content": "Moved Temporarily. Redirecting to"
}
}
Saying that it was moved temporarily.
Why am I getting this error? Did I reach the API limit for today?
NOTE:
Yesterday I kept it running to test the API request limit. But when I am trying to run today it showing the above error.
If the API limit for IP is reached then when do I get access to the data again?
This is the API which I am using:
http://finance.yahoo.com/webservice/v1/symbols/msft,goog,appl,orcl,yhoo,tcs,amzn,INFY.NS/quote?format=json&view=detail
As it was commented here: https://stackoverflow.com/a/38390559/6586718, you have to change the user-agent to a mobile device.
On Java, I do the following, and it's working (this is for XML, but the same can be applied to JSON):
URL url = new URL ("https://finance.yahoo.com/webservice/v1/symbols/" + stocks + "/quote");
HttpURLConnection urlc = (HttpURLConnection) url.openConnection ();
urlc.setRequestProperty ("User-Agent", "Mozilla/5.0 (Linux; Android 6.0; MotoE2(4G-LTE) Build/MPI24.65-39) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.81 Mobile Safari/537.36");
Document xml = DocumentBuilderFactory.newInstance ().newDocumentBuilder ().parse (urlc.getInputStream ());
try with this new one..
https://query.yahooapis.com/v1/public/yql?q=select%20*%20from%20csv%20where%20url%3D'http%3A%2F%2Fdownload.finance.yahoo.com%2Fd%2Fquotes.csv%3Fs%3DAAPL%26f%3Dsl1d1t1c1ohgv%26e%3D.csv'%20and%20columns%3D'symbol%2Cprice%2Cdate%2Ctime%2Cchange%2Ccol1%2Chigh%2Clow%2Ccol2'&format=json&env=store%3A%2F%2Fdatatables.org%2Falltableswithkeys
If I request the following URL
http://www.google.com/recaptcha/api/noscript?k=MYPUBLICKEY
I will get old no-script version of captcha, containing image of Google street number, like this
But if I'll do the same with HtmlUnit I will get some faked version of image, like this:
It happens all the time: real-world street number from browser and blackish distorted text from HtmlUnit. Public key is the same.
How can Google server distinguish between browser and HtmlUnit?
The HtmlUnit code is follows:
final WebClient webClient = new WebClient(BrowserVersion.FIREFOX_17);
final HtmlPage page = webClient.getPage("http://www.google.com/recaptcha/api/noscript?k=" + getPublicKey());
HtmlImage image = page.<HtmlImage>getFirstByXPath("//img");
ImageReader imageReader = image.getImageReader();
Process is observable with Fiddler.
And how about setting correct Headers for your request? User-Agent is a key here.
Headers are the way that backend can get client information (Firefox, Chrome etc) and what is it in your case? Set correct headers eg. for Firefox:
conn.setRequestProperty("User-Agent", " Mozilla/5.0 (Windows NT 6.1; WOW64; rv:8.0.1) Gecko/20100101 Firefox/8.0.1");
conn.setRequestProperty("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8");
This snipped if from my code using Apache HttpClient, you need to adapt it to your needs.
I know this is old post but, good way is to use
WebClient webClient = new WebClient(BrowserVersion.INTERNET_EXPLORER);
How you solve your problem?
I want to search the following in Google query box:
http://www.cmu.edu/silicon-valley/ faculty directory
Unfortunately, the following code does not work:
Jsoup.connect("http://www.google.com/search?hl=en&q=http%3A%%2F%%2F%www.cmu.edu%2F%silicon-valley%2F%20faculty20directory").get();
nor does this one:
Jsoup.connect("http://www.google.com/search?hl=en&q=http%3A%%2F%%2F%www.cmu.edu%2F%silicon-valley%2F%20faculty20or20directory").get();
What am I missing here?
Edit: not working means Google didn't return any result as we see from browser.
Jsoup.connect("http://www.google.com/search?hl=en&q=http%3A%"%2F%%2F%www.cmu.edu%2F%silicon-valley%2F%20faculty").get();
The code above works though. It's equivalent to Googling "http://www.cmu.edu/silicon-valley/ faculty".
Edit: I have the following trick in my program, so bot-rule is not an issue:
.userAgent("Mozilla")
Jsoup.connect("http://www.google.com/search?hl=en&q=http%3A%2F%2Fwww.cmu.edu%2Fsilicon-valley%2F+faculty+directory") leads to a 403 error (Forbidden) as google forbis robots to access its results
You'll have to change the User Agent String if you want to do that
doc = Jsoup.connect("http://www.google.com/search?hl=en&q=http%3A%2F%2Fwww.cmu.edu%2Fsilicon-valley%2F+faculty+directory").header("User-Agent", "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.17 (KHTML, like Gecko) Chrome/24.0.1312.52 Safari/537.17").get() should work as expected, but could be against Google's Terms of Use.
I'm developing an application which will allow users to sign in with their Facebook account. Currently, I'm using Eclipse SWT browser to display the login-browsing window, which works great. However, I'd prefer if this web-browsing window got treated (and therefore styled) as an iPhone. I've tried to force the HTTP-headers to mimic the iPhone's; without any change.
My current code looks somewhat like this:
String[] headers = { "Mozilla/5.0 (iPhone; U; CPU like Mac OS X; en) AppleWebKit/420+ (KHTML, like Gecko) Version/3.0 Mobile/1A543a Safari/419.3" };
browser.setUrl("https://m.facebook.com/dialog/oauth?client_id=appId&redirect_uri=url&state=state", "", headers);
However, the Facebook website STILL realizes that I'm not an iPhone, which makes the output looks like this:
Whilist I'm looking for this design: