I'm trying to make a program that submits a search query to Google and then opens the browser with the results.
I have managed to connect to Google but I'm stuck because I don't know how to insert the search query into the URL and submit it.
I have tried to use HtmlUnit but it doesn't seem to work.
This is the code so far:
URL url = new URL("http://google.com");
HttpURLConnection hr = (HttpURLConnection) url.openConnection();
System.out.println(hr.getResponseCode());
String str = "search from java!";
You can use the Java.net package to browse the internet. I have used an additional method to create the search query for google to replace the spaces with %20 for the URL address
public static void main(String[] args) {
URI uri= null;
String googleUrl = "https://www.google.com/search?q=";
String searchQuery = createQuery("search from Java!");
String query = googleUrl + searchQuery;
try {
uri = new URI(query);
Desktop.getDesktop().browse(uri);
} catch (IOException | URISyntaxException e) {
e.printStackTrace();
}
}
private static String createQuery(String query) {
query = query.replaceAll(" ", "%20");
return query;
}
The packages used are core java:
import java.awt.Desktop;
import java.net.URI;
import java.net.URISyntaxException;
Related
I am trying to get table data from the following url:
Get Data from this URL
and I wrote this code with the help of jaunt API
package org.open.browser;
import com.jaunt.Element;
import com.jaunt.Elements;
import com.jaunt.JauntException;
import com.jaunt.UserAgent;
public class ICICIScraperDemo {
public static void main(String ar[]) throws JauntException{
UserAgent userAgent = new UserAgent(); //create new userAgent (headless browser)
userAgent.visit("https://www.icicidirect.com/idirectcontent/Research/TechnicalAnalysis.aspx/companyprofile/inftec");
Elements links = userAgent.doc.findEvery("<div class=expander>").findEvery("<a>"); //find search result links
String url = null;
for(Element link : links) {
if(link.innerHTML().equalsIgnoreCase("Company Details")){
url = link.getAt("href");
}
}
/*userAgent = new UserAgent(); */ //create new userAgent (headless browser)
userAgent.visit(url);
System.out.println(userAgent.getSource());
Elements results = userAgent.doc.findEvery("<tr>").findEvery("<td>");
System.out.println(results);
}
}
But it didn't work.
Then I tried another API called htmlunit and wrote below code
public void htmlUnitEx(){
String START_URL = "https://www.icicidirect.com/idirectcontent/Research/TechnicalAnalysis.aspx/companyprofile/inftec";
try {
WebClient webClient = new WebClient(BrowserVersion.CHROME);
HtmlPage page = webClient.getPage(START_URL);
WebResponse webres = page.getWebResponse();
//List<HtmlAnchor> companyInfo = (List) page.getByXPath("//input[#id='txtStockCode']");
HtmlTable companyInfo = (HtmlTable) page.getFirstByXPath("//table");
for(HtmlTableRow item : companyInfo.getBodies().get(0).getRows()){
String label = item.getCell(1).asText();
System.out.println(label);
if(!label.contains("Registered Office")){
continue ;
}
}
}
But this also not giving the result .
Can someone please help how to get the data from the above url and other Anchor url in a single session?
Using HtmlUnit you can do this
String url = "https://www.icicidirect.com/idirectcontent/Research/TechnicalAnalysis.aspx/companyprofile/inftec";
try (final WebClient webClient = new WebClient(BrowserVersion.FIREFOX_60)) {
HtmlPage page = webClient.getPage(url);
webClient.waitForBackgroundJavaScript(1000);
final DomNodeList<DomNode> divs = page.querySelectorAll("div.bigcoll");
System.out.println(divs.get(1).asText());
}
Two things to mention:
you have to wait after the getPage call a bit because some parts are created by javascript/AJAX
there are many way to find elements on a page (see Finding a specific element). I have done only a quick hack to show the code is working.
I have a code which uses tor every time to get a new IP address, and then it opens a blog page, but then also the views counter of the blog do not increases?
import java.io.InputStream;
import java.net.*;
public class test {
public static void main (String args [])throws Exception {
System.out.println (test.getData("http://checkip.amazonaws.com"));
System.out.println (test.getData("***BLOG URL***"));
}
public static String getData(String ur) throws Exception {
String TOR_IP="127.0.0.1", TOR_PORT="9050";
System.setProperty("java.net.preferIPv4Stack" , "true");
System.setProperty("socksProxyHost", TOR_IP);
System.setProperty("socksProxyPort", TOR_PORT);
URL url = new URL(ur);
String s = "";
URLConnection c = url.openConnection();
c.connect();
InputStream i = c.getInputStream();
int j ;
while ((j = i.read()) != -1) {
s+=(char)j;
}
return s;
}
}
This I just made to understand what they have to pass this little auto script.
This is an evolving field, the blog sites try to detect and thwart cheating. Wordpress in particular excludes (https://en.support.wordpress.com/stats/):
visits from browsers that do not execute javascript or load images
In other words just hitting the page doesn't count. You need to fetch all the resources and possibly execute the JavaScript as well.
Creating an android app to receive data in json format from web server
in my app I should have url as string and use it to fetch data like below
private static final String my_url = "http://example.com/folder/showJsonData.php";
jsonObjectRequest = new JsonObjectRequest(Request.Method.POST, my_url ,new Response.Listener<JSONObject>() {
.
.
.
}
as you see my_url is not complicated or encoded so everyone can access it after decompiling apk.
the question is that how can I make it a little more complicated
please explain it with example.
Thanks
URL encoding is done in the same way on android as in Java SE;
try {
String url = "http://www.example.com/?id=123&art=abc";
String encodedurl = URLEncoder.encode(url,"UTF-8");
Log.d("TEST", encodedurl);
} catch (UnsupportedEncodingException e) {
e.printStackTrace();
}
i have a question about Jsoup library ...
i have this little program , which download ,parse and get the title of an HTML page which is google.com .
import java.io.File;
import java.io.IOException;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
public class HTMLParser{
public static void main(String args[]) {
// JSoup Example - Reading HTML page from URL
Document doc;
try {
doc = Jsoup.connect("http://google.com/").get();
title = doc.title();
} catch (IOException e) {
e.printStackTrace();
}
System.out.println("Jsoup Can read HTML page from URL, title : "+title);
}
}
The program is working very well,BUT the problem is :
when i try to parse a file from the ip adress 192.168.1.1(i change the google.com to 192.168.1.1 which is the adress of the router):
doc = Jsoup.connect("http://192.168.1.1/").get();
it does not work and shows me the error below :
org.jsoup.HttpStatusException: HTTP error fetching URL. Status=401, URL=http://192.168.1.1/
at org.jsoup.helper.HttpConnection$Response.execute(HttpConnection.java:537)
at org.jsoup.helper.HttpConnection$Response.execute(HttpConnection.java:493)
at org.jsoup.helper.HttpConnection.execute(HttpConnection.java:205)
at org.jsoup.helper.HttpConnection.get(HttpConnection.java:194)
at HTMLParser.main(HTMLParser.java:43)
first i think that the problem is related to "ussername and the password" so i change the address 192.168.1.1 to Username:Password#192.168.1.1 :
doc = Jsoup.connect("http://username:password#192.168.1.1/").get();
but it does not work , the program read the entire line as an adress.
if someone have any idea please help me !! and thanks for everybody
As with saka1029, you can request the URL with authentication. Then you use Jsoup.parse(String) to get the Document object.
Or you simply use Jsoup methods to send the request and get the response:
Getting HTML Source using Jsoup of a password protected website
Jsoup connection with basic access authentication
(I usually use javax.xml.bind.DatatypeConverter.printBase64Binary for the Base64 conversion.)
thank you very much saka1029;Griddoor. i read what you suggest , and it helps very much,
for me i use this solution :
URL url = new URL("http://user:pass#domain.com/url");
URLConnection urlConnection = url.openConnection();
if (url.getUserInfo() != null) {
String basicAuth = "Basic " + new String(new Base64().encode (url.getUserInfo().getBytes()));
urlConnection.setRequestProperty("Authorization", basicAuth);
}
InputStream inputStream = urlConnection.getInputStream();
from : Connecting to remote URL which requires authentication using Java
and used this method to read the inputstream:
StringWriter writer = new StringWriter();
IOUtils.copy(inputStream, writer);
String theString = writer.toString();
from : Read/convert an InputStream to a String
then i parse the theString with Jsoup.
I just started using YouTube API for Java and I'm having a tough time trying to figure out why things don't work since exception/stack trace is no where to be found. What I'm trying to do is to get list of videos uploaded by current user.
GoogleTokenResponse tokenFromExchange = new GoogleTokenResponse();
tokenFromExchange.setAccessToken(accessToken);
GoogleCredential credential = new GoogleCredential.Builder().setJsonFactory(JSON_FACTORY).setTransport(TRANSPORT).build();
credential.setFromTokenResponse(tokenFromExchange);
YouTube.Channels.List channelRequest = youtube.channels().list("contentDetails");
channelRequest.setMine(true);
channelRequest.setFields("items/contentDetails,nextPageToken,pageInfo");
ChannelListResponse channelResult = channelRequest.execute();
I don't see anything wrong with this code and also tried removing multiple things, but still not able to get it to work. Please let me know if you have run into a similar issue. The version of client library I'm using is v3-rev110-1.18.0-rc.
YouTube API has some working code and you can use it.
public static YouTubeService service;
public static String USER_FEED = "http://gdata.youtube.com/feeds/api/users/";
public static String CLIENT_ID = "...";
public static String DEVELOPER_KEY = "...";
public static int getVideoCountOf(String uploader) {
try {
service = new YouTubeService(CLIENT_ID, DEVELOPER_KEY);
String uploader = "UCK-H1e0S8jg-8qoqQ5N8jvw"; // sample user
String feedUrl = USER_FEED + uploader + "/uploads";
VideoFeed videoFeed = service.getFeed(new URL(feedUrl), VideoFeed.class);
return videoFeed.getTotalResults();
} catch (Exception ex) {
Logger.getLogger(YouTubeCore.class.getName()).log(Level.SEVERE, null, ex);
}
return 0;
}
This simple give you the number of videos a user has. You can read through videoFeed using printEntireVideoFeed prepared on their api page.