I'm learning java and I want to create a some kind of bot that will be doing operations on webpages, suchas clicking buttons filling textboxes etc. My question which library use to do it? I found out this bot works well:
https://github.com/DaveDuck321/SmashIt/blob/master/app/src/main/java/tk/smashr/smashit/KahootHandle.java
But I don't know how it's even done, it uses okhttp library, but it just sends http requests, but how author knew which request he need send to server to do stuff? Is it possible to use just reqest to bot any site or it's better to use something like htmlunit?
I'm learning java and I want to create a some kind of bot that will be doing operations on webpages, suchas clicking buttons filling textboxes etc.
Have a look at Selenium which makes it possible for you to control a web browser through all kinds of programming languages, including Java.
Further reading:
The Selenium ‘click’ command
How to press/click the button using Selenium if the button does not have the Id?
Related
My supervisor has tasked me with programmatically reducing a website's content by looking at the HTML tags to reveal only the core content. Importantly, this particular piece of the project must be written in Java.
Now having learnt about the differences betweenPlugins, Extensions, Applets, and Widgets, I think I want to use an Extension that calls a client-side Applet. My approach was going to be this:
Using the Google-Chrome API, I was going to display a button that
the user can click.
If clicked, the action is to launch a new browser tab that has the
Applet embedded within it.
The applet automatically sources the called tab's HTML code and
filters it.
Once filtered, the reduced copy of the original site appears.
So I have a few questions. To start, is it even possible to use an Extension with an Applet? Moreover, is it possible for an applet to look # another tabs HTML code? If not, is it possible to just reload the original tab with the Applet now embedded within it and complete the function. Thanks.
Javascript is already on most mobile web platforms. Java is not, and there is no reasonable way mobile customers will be able to install Java. Android, which runs many, but not all, mobile devices has a Java run time environment, and is basically a loader for Java apps. But an Apple iPhone is not an Android device... nor is a Windows Phone.
If you want to summarize content on the client, and in Javascript, as I see it you have two choices:
Succeed with some inner burst of genius where dozens of the best expert PhDs in Natural Language Computing have just begun exploring how to extract "true meaning" from text; OR
look at document.title and be done with it.
The 2nd approach assumes that the authors of web pages set titles and set a title appropriate for summarizing their website. This isn't a perfect assumption, but it is OK
most of the time. It is also a lot less expensive than #1
With the 1st approach you can get a head start with a "natural language toolkit" that can do things like scan text for unusual words and phrases. To get a rough idea of the kinds of software that have been built in this area, review wikipedia: Outline of natural language processing:: toolkits. A popular tookit for python is called NLTK. Whether you use a toolkit from java, or python, it means working on the server because the client will not have the storage, network speed, or CPU. For python there are server side app frameworks like django or web2py that can make building out a server app faster, and on Java there are servlets frameworks. Ultimately you'll need a lot of help, training, or luck and as I have hinted above it can easily be beyond the capabilities of a small team of fresh hires, and certainly way beyond what a single new developer eager to prove his/her capabilities can do in a few weeks on their own with limited help.
Most web pages have titles set like this near the beginning of the downloaded HTML:
<head><title>My Furry Kittens!</title></head>
You don't need to write a parser. If you are running in the browser, the title has been parsed into the DOM or Document Object Model already. The string "My Furry Kittens!" in this example would be available in the global variable document.title.
If you like, you could put a button into a plugin and let people push it to summarize the website. Or, they could just look up at the title. It is already on the page. Of course, if the goal is to scrape titles one can avoid writing a parser and use a "fake" headless scriptable browser like phantomJS or similar.
You can read more about document.title on the Mozilla Developer Network. MDN is a great reference for learning how web browsers work. They are the maintainers of the Mozilla Firefox browser. Most of what you can learn there will also work on Chrome, Internet Explorer, and various mobile platforms.
Good Luck!
How about implementing a local proxy server on the mobile device. The browser would just need to be configured to use the proxy, while the custom proxy implementation can transform the requested html however it likes.
need to code a bot that needs to do the following:
Go to a jsp page and search for something by:
writing something on a search box
clicking the search button(submit button)
clicking one of the the resulting buttons/links(same jsp page
with different output)
get the entire html of the new page(same jsp page with different
output)
The 4th one can be done with screen scraping and I do not think I need help with it. But I need some guidance to do the options from 1 to 3. Any links or just some keyword that will help me Google to learn about it will be appreciated. I plan to do this with java.
My suggestion is to use Selenium (http://docs.seleniumhq.org/download/).
Install Selenium IDE in your firefox, and it can record what you do on a website, store it into a script and reply it.
This video (http://www.youtube.com/watch?v=gsHyDIyA3dg) is gonna be helpful if you are a beginner.
And if you want to do it in Java, its easy, just export the scripts in Selenium IDE to JUnit Webdriver code.
Of course you can use Selenium Java webdriver in Java to write your program to operate on website directly.
Selenium automates browsers. That's it. What you do with that power is entirely up to you.
The above steps can be done by using selenium(which is a testing tool in java)
Even points 1 to 3 are screenscraping - you're figuring out (using either manual or automated means) what's there in the page and performing actions on them. You could try exploring the Apache HTTP Client for an easy way to run HTTP commands and get responses.
I hope you're doing this for legitimate means - screenscraping is almost always frowned upon if done without permission.
I have an HTML form that I want a user to fill out, and then send by pressing a button. Is there a way to have that button run a java program with all the HTML input as parameters? I looked into this a bit, and most people recommended servlets. Is there a way to do this without a servlet?
Edit: I'm not trying to use a servlet because of restrictions.
If you have a restriction against running a servlet you're pretty much out of luck in terms of using java. You could use a JSP, but that's just a servlet at the end of the day. You could use javascript to pop-up a mail-to with the form parameters entered as the body of the email but that would assume the user has a mail program configured. You could use some service like http://www.emailmeform.com/ (I have no idea if this one is good, I just googled it...).
Check out Java Web Start, might be what you're looking for.
Are you running the java client-side or server-side? The two posted answers are client-side answers. As for server-side, the tutorial here might help, depending on your server.
How can I open a webpage from a java application and enter username and password into it? I have seen questions here where people have referred to lobo or DJ Native Swing.
But as I am very new to java these libraries seem quite complex to me and I can not find a good tutorial in it, please refer to some good library with a solid tutorial which can be a beginning ground for me.
Note: I am developing a Java Swing application and show the user a page opening and user name and password being submitted.
What application do you develop? Web? Desktop/swing? Console? Mobile?
Take a look at Apache's HttpComponents project
It is not clear what you want to do.
1) Do you want to show it to the user?
join to WebKit browser in Java app on multiple platforms
2) You want to fetch some data from a web page that requires login?
then you should join to HttpClient login, search and get the XML content
Your question is not clear, are you trying to write a java application that will open a browser and enter data automatically?
If this is what you want then you can use selenium, a plugin for Mozilla. You can record your mouse movements, all your actions will get recorded as a jUnit test case.
You can then modify this junit test and read in the usernanme and password programatically from a file (or what ever you are trying to do.). This test can be run as a stand alone java application.
Although I've been programming for a few years I've only really dabbled in the web side of things, it's been more application based for computers up until now. I was wondering, in java for example, what library defined function or self defined function I would use to have a program launch a web browser to a certain site? Also as an extension to this how could I have it find a certain field in the website like a search box for instance (if it wasnt the current target of the cursor) and then populate it with a string and submit it to the server? (maybe this is a kind of find by ID scenario?!)
Also, is there a way to control whethere this is visible or not to the user. What I mean is, if I want to do something as a background task whilst the user carries on using the program, I will want the program to be submitting data to a webpage without the whole visual side of things that would interrupt the user?
This may be basic but like I say, I've never tried my hand at it so perhaps if someone could just provide some rough code outlines I'd really appreciate it.
Many thanks
I think Selenium might be what you are looking for.
Selenium allows you to start a Web browser, launch it to a certain website and interact with it. Also, there is a Java API (and a lot of other languages, by the way) allowing you to control the launched browser from a Java application.
There are some tweaking to do, but you can also launch Selenium in background, using a headless Web browser.
as i understand it you want to submit data to a server via the excisting webinterface?
in that case you need to find out how the URL for the request is build and then make a http-call using the corresponding URL
i advice reading this if it involves a POST submit