HtmlUnit won't open a link unless I manually tell it to - java

This works:
HtmlPage page = (HtmlPage) browser.getPage("http://www.somewebsite.com/viewprofile.aspx?profile_id=107992814")
However if I put the URL in a variable like this:
String userPage = "http://www.somewebsite.com/" + profileAnchorLink.getHrefAttribute();
page = (HtmlPage) browser.getPage (userPage);
I get an error that starts off like this
Exception in thread "main" ======= EXCEPTION START ========
Exception class=[net.sourceforge.htmlunit.corejs.javascript.WrappedException]
com.gargoylesoftware.htmlunit.ScriptException: Wrapped com.gargoylesoftware.htmlunit.ScriptException: TypeError: Cannot read property "data" from undefined (https://www.gstatic.com/swiffy/v7.3.2/runtime.js#72)
Any ideas? I had an html web bot that worked beautifully but then I upgraded to Windows 10 and went through some messy problems, not sure if that has anything to do with it. I made a new project and re-imported the HtmlUnit libraries in case something was broken (kept the same workspace though not sure if that matters) and still to no avail.
The even weirder part is that sometimes it actually works. Initially my program wasn't even using the URL it was just going directly to the link but then something broke so I tried to do things a different way, the URL method was actually working but then it started to work only sometimes and now it doesn't work at all.
So I'm really quite lost on what's going on here.

Seems like the real problem was that I wasn't using getPage properly, after implementing the information from this answer (How to call getPage from HtmlUnit WebClient and have setTimeout not wait forever?) all is well...for now.

Related

How to Add a <script> into Head Using Selenium's JavascriptExecutor

Summary
I want to figure out a way to add a <script> tag into the head of DOM using Selenium's JavascriptExecutor, or any other way of doing this would be nice.
I have tried many ways and also found a few similar topics and none of them solved my problem which is why I felt the need to ask it on here.
For example :
Suggested solutions in this question did not solve my problem. Some people say it worked for them but nope, it didn't for me.
What I've been trying to execute?
Here is the small snippet of the code that I want to execute:
WebDriver driver = new FirefoxDriver();
JavascriptExecutor jse = (JavascriptExecutor) driver;
jse.executeScript("var s = document.createElement('script');");
jse.executeScript("s.type = 'text/javascript';");
jse.executeScript("s.text = 'function foo() {console.log('foo')}';");
jse.executeScript("window.document.head.appendChild(s);");
I just skipped the code above where you navigate to a webpage using driver.get() etc. and then try to execute the scripts.
Also, s.text would contain the actual script that I want to use so I just put there a foo() function just to give the idea.
The above code throws this error when you run it:
Exception in thread "main" org.openqa.selenium.JavascriptException: ReferenceError: s is not defined
So far I've tried every possible solution I could find on the Internet but none of them seems to work.
OP came up with the following solution:
jse.executeScript("var s=window.document.createElement('script');" +
"s.type = 'text/javascript';" + "s.text = function foo() {console.log('foo')};" +
"window.document.head.appendChild(s);");
For one, this line is invalid.
jse.executeScript("s.text = 'function foo() {console.log('foo')}';");
Note how you wrap single-quote text in single quotes. Use one set as "\""
I would personally do this by doing (edited to make it a global function):
using OpenQA.Selenium.Support.Extensions;
driver.ExecuteJavascript("window.foo = function foo() {console.log('foo')}");
It's as simple as that. You are registering foo as a method by doing this. After you execute this javascript, you can manually go in to the browser developer tools and call "foo()" to check. Additionally, you can check this by registering it directly in the console. Just enter "function foo() {console.log('foo')}" into your browser console, and then call "foo()".
No need to add this as a script tag.
EDIT #2: I fixed my above code suggestion so that the method is assigned to the window, and thus accessible globally, and outside of the anonymous script that javascript executor runs the code in. The original issues with this not working are resolved by this, at least in my testing of it.

HtmlUnit and HTTPS pages

I'm trying to make a program that checks avaliable positions and books the first avaliable one. I started writing it and i ran into a problem pretty early.
The problem is that when I try to connect with the site (which is https) the program doesn't do anything. It doesn't throw an error, it doesn't crash. And the weirdest thing is that it works with some https websites and with some it doesn't.
I've spent countless hours trying to resolve this problem. I tried using htmlunitdriver and it still doesn't work. Please help.
private final WebClient webc = new WebClient(BrowserVersion.CHROME);
webc.getCookieManager().setCookiesEnabled(true);
HtmlPage loginpage = webc.getPage(loginurl);
System.out.println(loginpage.getTitleText());
I'm getting really frustrated with this. Thank you in advance.
As far as i can see this has nothing to do with HttpS. It is a good idea to do some traffic analysis using Charles or Fiddler.
What you can see....
The page returned from the server as response to your first call to https://online.enel.pl/ loads some external javascript. And then the story begins:
This JS looks like
(function() {
var z = "";
var b = "766172205f3078666.....";
eval((function() {
for (var i = 0; i < b.length; i += 2) {
z += String.fromCharCode(parseInt(b.substring(i, i + 2), 16));
}
return z;
})());
})();
As you can see someone likes to hide the real javascript that gets processed.
Next step is to check the javascript after this simple decoding
It is really huge and looks like this
var _0xfbfd = ['\x77\x71\x30\x6b\x77 ....
(function (_0x2ea96d, _0x460da4) {
var _0x1da805 = function (_0x55e996) {
while (--_0x55e996) {
_0x2ea96d['\x70\x75\x73\x68'](_0x2ea96d['\x73\x68\x69\x66\x74']());
}
};
.....
Ok now we have obfuscated javascript. If you like you can start with http://ddecode.com/hexdecoder/ to get some more readable text but this was the step where i have stopped my analysis. Looks like this script does some really bad things or someone still believes in security by obscurity.
If you run this with HtmlUnit, this codes gets interpreted - yes the decoding works and the code runs. Sadly this code runs endless (maybe because of an error or some incompatibility with real browsers).
If you like to get this working, you have to figure out, where the error is and open an bug report for HtmlUnit. For this you can simply start with a small local HtmlFile and include the code from the first external javascript. Then add some log statements to get the decoded version. Then replace this with the decoded version and try to understand what is going on. You can start adding alert statements and check if the code in HtmlUnit follows the same path as browsers do. Sorry but my time is to limited to do all this work but i really like to help/fix if you can point to a specific function in HtmlUnit that works different from real browsers.
Without the URL that you are querying it is dificult to say what could be wrong. However, having worked with HTML unit some time back I found that it was failing with many sites that I needed to get data from. The site owners will do many things to avoid you using programs to access them and you might have to resort to using some lower level library like Apache HTTP components where you have more control over what is going on under the hood.
Also check if the website is constructed using JavaScript which is getting more and more popular but making it increasingly dificult to use programs to interrogate the content.

It seems that I found a simple bug for Springframework v4.2.0?

I try to use org.springframework.mock.web.MockHttpServletResponse in my project's unit-test, offered by spring-test-4.2.0.RELEASE.jar.
When I trigger its method of setContentType(), the whole thread progress goes dead/hung! until my #Test(timeout=5000) annotation breaks the test-case hunging!
I then try to replace the spring-mock package by a previous relase of spring-test-4.1.7.RELEASE.jar.
Guess what?! Everything goes fine!
Is it a silly bug of the mock-package of Springframework 4.2.0?
If it is, where should report it? since I can NOT find a bugzilla for Springframework on web?
OK friends, thanks for all of your quick response really!
I found the reason why:
spring-mock/test-package (spring-test-4.2.0.RELEASE.jar) may never works lonely! MockHttpServletResponse.setContentType() of v4.2.0 innerly uses org.springframework.http.MediaType, which locates inside spring-web-4.2.0.RELEASE.jar! (Which is missing)
But in v4.1.7, MockHttpServletResponse.setContentType() calls nothing outside the spring-test-4.2.0.RELEASE.jar.
Thus why spring-test-4.2.0.RELEASE.jar goes dead/hung, and spring-test-4.1.7.RELEASE.jar works fine!
It's surely NOT a bug!
However, I still can NOT understand why it goes dead/hung, instead of throws ClassNotFoundException in my TestCase?!
Thus, I raise another question: TestCase make up by JUnit + Spring-Mock package runs into a dead-hung, when other spring-jar files are missing! (Never a ClassNotFound exception!?)
Hope any of you may give me the correct answer!
Many thanks again!

Eclipse - Command line URL argument cutting off https

EDIT: Code for InstallCerts is here:
http://code.google.com/p/java-use-examples/source/browse/trunk/src/com/aw/ad/util/InstallCert.java
I am trying to run a java program in eclipse that takes a URL (eg https://myurl.com) as an argument. When I go to Run Configurations -> Arguments and paste in the URL, it looks fine. I click Apply.
The problem starts when I click Run. For some reason, eclipse removes the https: at the start of the URL and I get an error saying:
Exception in thread "main" java.lang.NumberFormatException: For input string: "//myurl.com"
Notice the lack of https:?
Anyway, if someone can point me in the right direction to resolve this, I'd be very grateful.
I should also point out, I tried to create a Variable as well, to hold the URL, but I got the same error. Likewise when I quoted the URL.
Thanks.
You are getting a "NumberFormatException" which means you are trying to convert a String to int in your code . I think thats the real problem.
Quote it.
"https://myurl.com"
EDIT : I just tried quote/unquote - it works either way. Can you post your code?
Cheers,Eugene
The trick is that class searches for port. Do not use https://, instead refer to secure port e.g. my.site.com:443!

In Java, starting a test in selenium with the DefaultSelenium object how do I find which browser the test is running on?

Consider a simple DefaultSelenium object
DefaultSelenium sel = new
DefaultSelenium("http://localhost:8080/myapp",4444,"*iexplore","/myAppLevel1");
Now my server is set with the option of -forcedBrowserMode "*firefox" in the command line when I start it up. However, I have 2 different batch files to start the Server, one forced in firefox, one forced to IE. FYI, the -forcedBrowserMode overrides the settings within of the instantiated java object.
The problem is from java, I can't seem to find a way to determine which browser my DefaultSelenium object is running on... I was thinking something like:
sel.getBrowserName();
But nothing like that exists. Are there any other creative ways of doing this?
I need to know because with a GWT web application, to click on a button you need to do it differently based on the browser. As well, you may wonder why I even use the -forcedBrowserMode, because then I can use custom setup firefox/ie installs to test on.
Thanks in advance for help!
I think that you can get the browser by executing some JavaScript, for example verify navigator.userAgent or any browser specific object, for example document.defaultView will be null in IE and not null in FF, something like this:
DefaultSelenium sel = ...
String res = sel.getEval("document.defaultView ? false : true");
boolean isIE = "true".equals(res);

Categories