Reading website content via MATLAB - java

I need to read the content of a website via MATLAB but the problem is when I load the website as:
URL = java.net.URL(UrlNew);
urlConnection = URL.openConnection();
urlConnection.connect();
str = regexprep(fileread(filename),'\s*',' ');
The website redirects me to a page which I have first to press "I agree" button and then it goes to the requested page.
How can I handle this situation in MATLAB. I need to somehow click the "Agree" button in MATLAB to have access to the original webpage.
Thx

First you need to find out the value of the action attribute of the form on your original page. You can do this by examining the source code of the page, looking for the form tag containing the Agree button, or by manually clicking on the button and looking at the Network activity tab in the Chrome Developer Tools or Firebug in Firefox.
Once you find this, find out how to send POST requests in Java and simulate the clicking of the button by sending a POST request to the action URL of the form. If you collect the output from the socket after that, it should correspond to the page you want.

Related

how to pass data from activity to webview with external website

I am making an app for tracking courier. the user will enter tracking no.in apps textview and click submit this will open couriers website tracking page i.e fedex in a webview inside the App.
Now i want a way to pass those tracking no from my app to webview's website input field and click the submit button on the website automatically
I don't think this the way you should do that, you should look for a url that takes the tracking location number as a query and return the proper page
something like that http://example.com/over/there?number=441222
Update
I've looked at the page you've mentioned in the comments and all I can get is that it submits a post request as a form input
POST url: https://www.myutiitsl.com/PAN_ONLINE/PANTrackerSearch.action
parameter names are
appNo
panNo
I wish I had more knowledge as a web developer but this the way that could work with you try to make sure the post url is correct and the parameters I mentioned also correct and it should work with you

How to get Json data on page load OR get url generated from Json data on click event

I'm testing an existing website and I need to pick up the url of a link and the href text = "#".
If I click on the link, whatever it is opens then displays a perfectly normal URL in the browser, so the website is somehow translating "#" into a proper URL. I have several hundred items, all referenced by "#", but all opening completely different items when you click on them.
I need to know what it is before I click it as I have to be able to handle it opening as a pop up or in a separate window or as a load instruction to office etc. and therefore pass the URL to different bits of code to open, depending on the file extension.
I've tried:
teststring = "//div[#class='" + divClass.toString() + "']/div[2]/a";
LesTestString = driver.findElement(By.xpath(teststring)).getAttribute("href");
which returns 'myWebsite.com/my/path/here/#'; and
String LessLink = inputhandler.getAttribute("href");
which does the same.
when I look at the element in firebug it looks like this:
<div class="reference span5 ">
<a class="name math-formula" href="#"> 1.1 Lesson Player </a>
Any suggestions?
Added Information: It would seem that the URL is generated by the browser on click from various bits of information sent in a large Json string. So this questions should now be either (1) how do you pick up the url generated by the browser onclick, or (2) how do you pick up the Json sent from the server using java /selenium or something that will run in that environment?

Android, YUI, Moodle

The problem I'm having is the following:
I have an app with two separate modes: A WebView for browsing and a custom Canvas. The custom Canvas captures handwriting samples for language placement exams. Here's how it works. A user logs in to Moodle via the WebView. After they log in, they navigate to a Quiz inside Moodle. They click a link on one of the Quiz's questions and this launches an Intent which hides the WebView and shows the Canvas. The user then writes (using a stylus) on the Canvas. When a user is finished writing their essay (or whatever), they press a button that uploads an image file to Moodle. I am able to upload images to a point, it's getting them to show up in the HTML page that the user clicked the link in originally (see above) and to get Moodle to commit them to permanent storage that is the problem. Normally this is all accomplished through AJAX (really AJAJ since it's JavaScript and JSON) and when the user drops a file on this one component, the component refreshes and uploads the file.
Here is the problem: I need the WebView so that students can log in to Moodle through Shibboleth. But because the underlying JavaScript in the browser makes AJAX calls to the Moodle server and since the Java side of Android doesn't have access to the DOM, I have use the Apache HTTP components library to make some of the connections below basically to preserve the state of the HTML page in WebView.
In a desktop browser on, say, Windows, I use WebScarab to monitor the browser's requests and this is what I see: the browser uploads a file to Moodle via five successive calls to the following scripts:
POST https://[moodle website]/repository/repository_ajax.php [posts multipart form data]
POST https://[moodle website]/repository/draftfiles_ajax.php [posts some params]
GET https://[moodle website]/draftfile.php/[some_id]/user/draft/[some_id]/[somefilename.png] [returns an icon of the image for a filepicker from YUI]
POST https://[moodle website]/mod/quiz/processattempt.php [returns HTML page]
GET https://[moodle website]/mod/quiz/summary.php [returns HTML page]
Some of these scripts return, as you'd expect, JSON data since they're AJAX and not HTML. The final two calls (4 & 5) return HTML. Now, I can make all of those calls in succession in either the WebView or the Apache HTTP library, but if I do so with WebView, only JSON data is returned to the WebView in calls 1-3 (WebView treats the JSON data as a page and displays it wiping out whatever HTML page was displayed in it). If I capture and process the JSON data using the Apache HTTP library in Java, then the JavaScript components internal to the page do not get updated. If I split the calls so that I send only calls 4 & 5 to the WebView, the HTML merely returns WebView to the first question of the exam and Moodle acts as if I haven't uploaded anything.
I can verify that files are uploading if I manually refresh (press a link) the JavaScript UI elements in the page. I can't expect students to do this, though, because the link to do so is very tiny and it's not obvious that it does a refresh. I need a way to programmatically refresh this one element (it's part of YUI) or to get Android and the Java side to play more nicely with the JavaScript/DOM side.
My question is: does anyone know a way to 1) fire off a drag and drop event using YUI to an element inside an HTML page or 2) a way to consume the JSON data and pass it to an element inside the HTML page.
I'm banging my head against a wall trying to figure this out.
OK, so I figured out that: javascript:document.getElementsByClassName(\"[name of link here]\")[0].click() works in Chrome on the desktop but doesn't work if I pass it to WebView.loadURL(). I just need to be able to simulate that click event reliably in WebView. It appears not to support click(). Anyone have any ideas?
The winning code is:
el = document.getElementsByClassName("[some element]")[0];
var event = document.createEvent("MouseEvents");
event.initEvent("click", true, true);
el.dispatchEvent(event);
This selects the link at [some element] and thereby fires an AJAX request that refreshes the FilePicker. For those working with Moodle, I had to add the above code to the same quiz question that handles so it is invoked by putting that code in its own function and calling it with WebView.loadURL("javascript:myRefreshFunction()").

Submitting ASP form from Java application

I am writing a Java application that has a log in screen. Ideally, I would like to take the user supplied data (name, password), and submit it to an ASP form that can verify their credentials. I do not own the ASP form, I can only access the URL. I also do not want the user to be entering their credentials straight into the web form. They would enter their credentials into my program, and my program would put the data into the form and submit, and allow/deny the user based on the response.
Of course, the submit button on the ASP form is a POST request. However, constructing the URL (...login?username=name&password=pass) does not work, as the form must be submitted via the button with the text boxes filled in.
I have tried two approaches:
Using Java's URLConnection class. This does not seem to work because the form submitting is limited to the method I mentioned above, which is constructing the URL.
Using Javascript to access and edit the elements on the page. This has not worked either, because the Javascript is being run from my program, which is not a web browser, and therefore has no access to the 'document' or 'window' commonly used.
Other potential solutions I can think of:
Opening a browser to the login page but not giving it focus, running a script to fill out and submit the form, parsing the response, and then closing the browser. This would not involve the user at all, except for the input into the login page in my Java program.
Using a 3rd party Java library (suggestions? references to tutorials?).
Embedding the URL into my login screen (any help in this regard would be appreciated).
The things that cannot be changed are that my program is in Java, and that the login URL is an ASP form that hides the POST data from the URL.
Let me know if anything needs clarification. Any help is welcome.
try htmlunit, although it was designed for testing it would be ideal for this. You can use it in conjunction with Selenium webdriver
Why don't you open up your ASP form in an IFrame using javascript populate all the fields & then post it.
This should solve your problem.
Sfk is correct, i had a similar problem and manage to fill the form and submit it with Htmlunit .#sfk many thanks, you put me in the right path.
So with htmlunit
//for chrome simulation
WebClient webClient = new WebClient(BrowserVersion.CHROME_16);
//has getting an error from [http://www.google-analytics.com/ga.js with javascript on.
webClient.setJavaScriptEnabled(false);
HtmlPage page = webClient.getPage("http://yourtargetpage/Default.aspx");
//get the form by name, check page source for name
HtmlForm form = page.getFormByName("aspnetForm");
HtmlPasswordInput inputPass = form.getInputByName("your input password text field name");
HtmlTextInput userName = form.getInputByName("your input user text field name");
HtmlSubmitInput button=form.getInputByName("your target submit button");
//set username and password
userName.setText("myuser");
inputPass.setText("mypassword");
//click the submit button and get the returned page
HtmlPage page2 = button.click();
That´s it.. you got the reply page and sent the information on the fields..you can now parse the page and get the site response..

How can I direct a user to a new page dynamically based on a dropdown list with plain Wicket?

Here is my situation: the user selects a section (for example from a dropdown) such as "Section1," "Section2" or "Section3." Then he clicks the OK button (or some link).
What I need to happen: after he clicks on that button/link, he will be redirected to the selected section, e.g. www.homepage.com/docs#section2.
So far, I have not been able to process the form from Link's onClick method, nor have I been able to call some clickLink on Link from the Button method onSubmit().
I would prefer not to use AJAX or JavaScript. How can I do this?
That's because a Link doesn't submit the form. It just acts as a link to somewhere. To access your formdata you'll need to submit the form first. Try using a SubmitLink instead of a Link and call
getRequestCycle().setRequestTarget
(new RedirectRequestTarget("www.homepage.com/docs#section2"));
from the onSubmit function of the SubmitLink.
Judging from the Javadoc this should work but I can't test it right now.
A RequestTarget that will send a redirect url to the browser. Use this if you
want to direct the browser to some external URL, like Google etc, immediately.
Or if you want to redirect to a Wicket page. If you want to redirect with a
delay the RedirectPage will do a meta tag redirect with a delay.
Did you try Link.setAnchor(Component)?

Categories