I have a GWT based page that I would like to create an HTML snapshot for it using HtmlUnit.
The page loads using Ajax/JavaScript information on a product, so for about 1 second there is a Loading... message and then the content appears.
The problem is that HtmlUnit doesn't seem to capture the information and all I'm getting is the "Loading..." span.
Below is an experimental code with HtmlUnit where I try to give it enough time to wait for the loading of the data but it doesn't seem to change anything and I am still unable to capture the data loaded by the GWT javascript.
WebClient webClient = new WebClient();
webClient.setJavaScriptEnabled(true);
webClient.setThrowExceptionOnScriptError(false);
webClient.setAjaxController(new NicelyResynchronizingAjaxController());
WebRequest request = new WebRequest(new URL("<my_url>"));
HtmlPage page = webClient.getPage(request);
int i = webClient.waitForBackgroundJavaScript(1000);
while (i > 0)
{
i = webClient.waitForBackgroundJavaScript(1000);
if (i == 0)
{
break;
}
synchronized (page)
{
System.out.println("wait");
page.wait(500);
}
}
webClient.getAjaxController().processSynchron(page, request, false);
System.out.println(page.asXml());
Any ideas...?
Thank you for responding.
I actually should have reported this sooner that I have found the solution myself.
Apparently when initialising WebClient with FF:
WebClient webClient = new WebClient(BrowserVersion.FIREFOX_3_6);
It seem to be working.
When initialising WebClient with the default constructor it uses IE7 by default and I guess FF has better support for Ajax and is the recommended emulator to use.
I believe by default NicelyResynchronizingAjaxController will only resynchronize AJAX calls that were caused by a user action, by tracking which thread it originated from. Perhaps the GWT generated JavaScript is being called by some other thread which NicelyResynchronizingAjaxController does not want to wait for.
Try declaring your own AjaxController to synchronize with everything regardless of originating thread:
webClient.setAjaxController(new AjaxController(){
#Override
public boolean processSynchron(HtmlPage page, WebRequest request, boolean async)
{
return true;
}
});
As documentation states, waitForBackgroundJavaScript is experimental:
Experimental API: May be changed in next release and may not yet work perfectly!
The next approach has always worked for me, regardless of the BrowserVersion used:
int tries = 5; // Amount of tries to avoid infinite loop
while (tries > 0 && aCondition) {
tries--;
synchronized(page) {
page.wait(2000); // How often to check
}
}
Note aCondition is whatever you're checking for. EG:
page.getElementById("loading-text-element").asText().equals("Loading...")
None of the so far provided solutions worked for me. I ended up with Dan Alvizu's solution + my own hack:
private WebClient webClient = new WebClient();
public void scrapPage() {
makeWebClientWaitThroughJavaScriptLoadings();
HtmlPage page = login();
//do something that causes JavaScript loading
waitOutLoading(page);
}
private void makeWebClientWaitThroughJavaScriptLoadings() {
webClient.setAjaxController(new AjaxController(){
#Override
public boolean processSynchron(HtmlPage page, WebRequest request, boolean async)
{
return true;
}
});
}
private void waitOutLoading(HtmlPage page) {
while(page.asText().contains("Please wait while loading!")){
webClient.waitForBackgroundJavaScript(100);
}
}
Needless to say, "Please wait while loading!" should be replaced with whatever text is shown while your page is loading. If there is no text, maybe there is a way to check for existence of some gif (if that is used). Of course, you could simply provide a big enough milliseconds value if you're feeling adventurous.
Related
I'm using the following code to dynamically generate a download in Wicket, using the ResourceLink approach (since the download is not a static file, it needs to be generated on the fly, and I was told this was the correct approach):
IResource res = new AbstractResource() {
#Override
protected ResourceResponse newResourceResponse(Attributes attributes) {
ResourceResponse resourceResponse = new ResourceResponse();
resourceResponse.setContentType("application/pdf");
resourceResponse.setFileName("output.pdf");
resourceResponse.setContentDisposition(ContentDisposition.ATTACHMENT);
resourceResponse.setWriteCallback(new WriteCallback() {
#Override
public void writeData(Attributes attributes) throws IOException {
OutputStream outputStream = attributes.getResponse().getOutputStream();
try {
outputStream.write(generateDocument());
} catch (Exception e) {
//Generation failed... Here I'd like to either show a popup message or alter the current page to show an error somewhere in the page
}
}
});
return resourceResponse;
}
};
ResourceLink<Void> resLink = new ResourceLink<Void>("resLink", res);
myForm.add(resLink);
The comment in the code above shows where I'm having trouble. If the generation of the download fails (which can happen, if certain conditions are not met) I'd like to show an error message, either by showing a popup or altering the page to show some error text (but in either case I want to avoid leaving/reloading the entire page)
Is this possible?
Here's the link with the answer:
https://cwiki.apache.org/confluence/display/WICKET/AJAX+update+and+file+download+in+one+blow
Don't forger to use a try/catch with an error(e.getMessage()) inside the catch and a target.add(feedbackPanel) after catching the error.
I am not sure this is possible because you need to use non-Ajax request to be able to download as ATTACHMENT. But since it is non-Ajax request you will need to either reload the current page or redirect to another page in case of an error.
I try my best to describe my situation.
My wicket site contains list wicket component, where every list element has another list. Each element in lowest level list has ajax wicket link to download some file. All this works fine. I used to this AjaxBehaviour. Method startDownload of this behaviour is invoked within link onClick method.
public void startDownload(AjaxRequestTarget target) {
target.appendJavaScript("window.location.href='" + getCallbackUrl() +"'");
}
Method onRequest of this behaviour is:
#Override
public void onRequest() {
IRequestHandler fileTarget = new IRequestHandler() {
#Override
public void respond(IRequestCycle requestCycle) {
if (null != file) {
try {
FileInputStream inputStream = new FileInputStream(file);
WebResponse resp = (WebResponse) requestCycle.getResponse();
resp.setAttachmentHeader(fileName);
String contentType = FileUtils.getFileType(fileName);
if (contentType != null) {
resp.setContentType(contentType);
}
resp.setHeader("Pragma", "anytextexeptno-cache");
resp.setHeader("Cache-Control", "max-age=0");
Streams.copy(inputStream, requestCycle.getResponse().getOutputStream());
} catch (Exception e) {
logger.error(e.getMessage(), e);
}
}
}
Now i need to reload model and refresh some components in the page after download file action. I tried to add entire page to the AjaxRequestTarget in method onclick, after code invoked startDownload method. Reload page works fine but window with file to download doesn`t show.
I think that i have to do reload page in other, separate request (maybe i'm mistaken? ), because in this request i call 'window.location.href=....', but i don`t know how i can to enforce second request to reload page.
Does have someone some ideas what I do wrong ? And how can I resolve my problem ?
Seems you need something like this:
https://cwiki.apache.org/confluence/display/WICKET/AJAX+update+and+file+download+in+one+blow
It seems that my implementation is simmilar to this from cwiki.apache.org website. In onRequest method i used getComponent().getRequestCycle().scheduleRequestHandlerAfterCurrent(handler), and despite of this doesn`t work.
Is possible that reason of this is component, which cause request is added to target (because i add to target entire page and component - ajaxLink in this example, is child of this page)
Background:
Using WebSockets with JavaScript + Play! framework (2.2).
Can send and receive data fine in Chrome.
Can only receive data (from server) in Firefox as send() doesn't trigger any callbacks.
In addition to the send issue, and in Firefox only again, the tab for the page is always stuck on "connecting" while the spinner keeps spinning (see figure 1).
Misbehaving Browser:
Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:24.0)(Firefox 24.0)
(Figure 1. Firefox tab after page has loaded and data is shown)
Any time I refresh the web page, I receive the error below, attributed to the constant page loading behaviour I'm sure.
The connection to ws://localhost:9000/ was interrupted while the page was loading.
The entire JavaScript code:
$(function() {
var chatSocket = new WebSocket("#routes.Application.getMetaData().webSocketURL(request)");
var sendMessage = function() {
chatSocket.send(JSON.stringify({
id: "unique",
name: "a name",
age: 22
}));
}
var receiveEvent = function(event) {
var data = JSON.parse(event.data)
document.write(data.age);
document.write(data.name);
document.write(data.message);
document.write("\n");
sendMessage();
chatSocket.close();
}
chatSocket.onmessage = receiveEvent
})
Now In the past, I've been trying with MozWebSocket instead of the standard WebSocket, but I get nothing rendered on screen using that module therefore unless there is an angle I've missed, WebSocket is the better one to use.
The relevant Play! block:
public static WebSocket<JsonNode> getMetaData() {
return new WebSocket<JsonNode>() {
// Called when the Websocket Handshake is done.
public void onReady(WebSocket.In<JsonNode> in, WebSocket.Out<JsonNode> out) {
// For each event received on the socket,
in.onMessage(new Callback<JsonNode>() {
#Override
public void invoke(JsonNode jsonNode) {
System.out.println("Message Incoming!");
System.out.println(jsonNode.get("id"));
System.out.println(jsonNode.get("name"));
System.out.println(jsonNode.get("age"));
}
});
// When the socket is closed.
in.onClose(new Callback0() {
public void invoke() {
System.out.println("Disconnected");
}
});
ObjectNode node = Json.newObject();
node.put("message", "hello");
node.put("name", "client");
node.put("age", 1);
out.write(node);
//same result commented/uncommented
out.close();
}
};
}
So in Chrome, the flow would be:
document.write(...)
"Message Incoming!"
... JSON attributes
"Disconnected"
But in Firefox, the flow is:
document.write(...)
"Disconnected"
Any help in diagnosing these problems would be greatly appreciated. I have no intention of supporting IE, but having both Mozilla and Chrome working would be great.
Other JavaScript Warnings:
Below is a warning I occasionally get in Firefox's console while pointing at the "ws" protocol as the culprit. What its relevance is to my problem, I do not know.
Use of getPreventDefault() is deprecated. Use defaultPrevented instead.
You call document.write() after the document is loaded, which then implies document.open() which in turn replaces the document and by that unloads the old one and aborts stuff like timeouts or websockets.
Use something other than document.write() and you should be fine.
I'm working on upgrading our existing Wicket webapp to 1.5 and have hit a snag in our renderPage function that we use to render our HTML emails.
Previously we used the code referenced/listed in this StackOverflow question and this (currently broken but maybe fixed later) link but that code no longer works as a lot of those classes don't exist in 1.5.
I also found this email thread but it is light on the details and I don't know how to create the WebPage from my pageClass and parameters.
http://apache-wicket.1842946.n4.nabble.com/Render-WebPage-to-String-in-Wicket-1-5-td3622130.html
Here is my code:
// Renders a page under a temporary request cycle in order to get the rendered markup
public static String renderPage(Class<? extends Page> pageClass, PageParameters pageParameters)
{
//get the servlet context
WebApplication application = (WebApplication) WebApplication.get();
ServletContext context = application.getServletContext();
//fake a request/response cycle
MockHttpSession servletSession = new MockHttpSession(context);
servletSession.setTemporary(true);
MockHttpServletRequest servletRequest = new MockHttpServletRequest(application, servletSession, context);
MockHttpServletResponse servletResponse = new MockHttpServletResponse(servletRequest);
//initialize request and response
servletRequest.initialize();
servletResponse.initialize();
WebRequest webRequest = new ServletWebRequest(servletRequest);
BufferedWebResponse webResponse = new BufferedWebResponse(servletResponse);
webResponse.setAjax(true);
WebRequestCycle requestCycle = new WebRequestCycle(application, webRequest, webResponse);
requestCycle.setRequestTarget(new BookmarkablePageRequestTarget(pageClass, pageParameters));
try
{
requestCycle.getProcessor().respond(requestCycle);
if (requestCycle.wasHandled() == false)
{
requestCycle.setRequestTarget(new WebErrorCodeResponseTarget(HttpServletResponse.SC_NOT_FOUND));
}
}
finally
{
requestCycle.detach();
requestCycle.getResponse().close();
}
return webResponse.toString();
}
Specifically, the code breaks because the WebRequestCycle and BookmarkablePageRequestTarget classes no longer exist. I feel like I should be able to use the StringResponse class some how but I'm missing the link that would help me trigger a render on that response.
Any help would be appreciated. Thanks.
My Final Solution
Using the example that I was directed to by the answer below I ended up with the following code. I'm pasting it here as well so that if that link disappears or is changed with a future version of Wicket then people from the future will still be able to get the answer they need.
I ended up passing in a PageProvider because in some cases I needed to pass in an instantiated Page and in others a pageClass + parameters.
public static String renderPage(final PageProvider pageProvider)
{
final RenderPageRequestHandler handler = new RenderPageRequestHandler(pageProvider, RedirectPolicy.NEVER_REDIRECT);
final PageRenderer pageRenderer = Application.get().getPageRendererProvider().get(handler);
RequestCycle requestCycle = RequestCycle.get();
final Response oldResponse = requestCycle.getResponse();
BufferedWebResponse tempResponse = new BufferedWebResponse(null);
try
{
requestCycle.setResponse(tempResponse);
pageRenderer.respond(requestCycle);
}
finally
{
requestCycle.setResponse(oldResponse);
}
return tempResponse.getText().toString();
}
Check the source code of http://www.wicket-library.com/wicket-examples/mailtemplate/ example.
I want to login to a website (http://www.orkut.com) through
com.gargoylesoftware.htmlunit.WebClient
But when I click on the "Submit" button, it doesn't take me to the expected page that should come after login. Instead it returns the same login page again. In clear sense, there is some problem in login. When I try the same code with sites that doen't have javascript, it works fine so I think I am not able to handle scripts.
I am trying using the follwoing code:
public static void main(String[] args) {
final WebClient webClient = new WebClient();
try {
HtmlPage loginPage = webClient.getPage(new URL("https://www.google.com/accounts/ServiceLogin?service=orkut&hl=en-US&rm=false&continue=http%3A%2F%2Fwww.orkut.com%2FRedirLogin%3Fmsg%3D0%26page%3Dhttp%253A%252F%252Fwww.orkut.co.in%252FHome.aspx&cd=IN&passive=true&skipvpage=true&sendvemail=false"));
System.out.println(loginPage.getTextContent());
List<HtmlForm> forms = loginPage.getForms();
HtmlForm loginForm = forms.get(0);
HtmlInput username = loginForm.getInputByName("Email");
HtmlInput password = loginForm.getInputByName("Passwd");
HtmlInput submit = loginForm.getInputByName("signIn");
username.setNodeValue("username");
password.setNodeValue("password");
HtmlPage homePage = submit.click();
Thread.sleep(10 * 1000);
System.out.println(homePage.getTextContent());
}catch(Exception ex) {
ex.printStackTrace();
}
}
When we do click on the "submit" button, in actual it calls first this function
onsubmit="return(gaia_onLoginSubmit());"
specified as the attribute of the form below
<form id="gaia_loginform" action="https://www.google.com/accounts/ServiceLoginAuth?service=orkut" method="post"
onsubmit="return(gaia_onLoginSubmit());">
Can anyone help me in this.
NOTE: I WILL PAY FOR THE SOLUTION
According to their site the JavaScript support is provided by Mozilla Rhino, so maybe all you need is to add it to your classpath (and perhaps fiddle with some configurations).
Also, HtmlUnit has professional support