I'm trying to parse a phtml file from a project and save modifications to this file with Java. I actually use Jsoup API to parse this file. My problem is the saving step. Lets see an example :
PHTML FILE :
<div id="header">
<div id="logo">
<img src="images/logo.png" id="logo_pic" />
</div>
<div id="welcome">
<span id="welcome_title">
</span>
</div>
</div>
Java parsing :
File testFile = new File("C:\\Users\\root\\Desktop\\test.phtml");
try {
Document doc = Jsoup.parse(testFile, "UTF-8");
Element essai = doc.getElementById("welcome_title");
essai.appendText("Application NAME!");
PrintWriter writer = new PrintWriter(testFile, "UTF-8");
writer.write(doc.html());
writer.close();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
Result i want to have should be like this :
<div id="header">
<div id="logo">
<img src="images/logo.png" id="logo_pic" />
</div>
<div id="welcome">
<span id="welcome_title">
Application NAME!
</span>
</div>
</div>
Result i have for the moment :
<html>
<head></head>
<body>
<div id="header">
<div id="logo">
<img src="images/logo.png" id="logo_pic" />
</div>
<div id="welcome">
<span id="welcome_title"> Application NAME!</span>
</div>
</div>
</body>
</html>
My problem is that i must keep a phtml format with only div elements for my application. How can i reach the good result? Any idea? Thx for help!
Do this:
writer.write(essai.outerHtml());
instead of
writer.write(doc.html());
Related
Googling this I came across numerous examples, one of which was $q from Angular, to avoid recursively hitting my server side to check if request is complete with data or not.
I came across this link : Recursive $http.get in for loop
Which suggested to use $q, but I dont think that will resolve my issue.
JS:
function OrderFormController($scope, $http, $q) {
$('#loaderImage').show();
$http.get('/utilities/longProcess')
.success(function(data, status, headers, config) {
if(data != 'Still Processing'){
console.log('Data from server');
$scope.sampleJSON = data.pmdStructureWrapper;
$scope.sampleJSONDuplicates = data.pmdDuplicates;
$scope.$watch('sampleJSON', setTimeout(function() {
$('.panel-body li').each(function() {
if ($.trim($(this).text()) === "") {
$(this).hide();
}
});
}, 1000));
$('#loaderImage').hide();
}else{
console.log('rejected');
$q.reject(data);
}
})
.error(function(data, status, header, config) {
});
}
Controller:
#RequestMapping("/utilities/longProcess")
public CompletableFuture<String> asyncLongProcess(HttpServletResponse response, HttpServletRequest request) {
HttpSession session = request.getSession();
return CompletableFuture.supplyAsync(() -> session.getAttribute("CACHED_RESULT"))
.thenComposeAsync(obj -> {
if (obj == null) {
if(session.getAttribute("BACKGROUND_PROCESSING") == null) {
session.setAttribute("BACKGROUND_PROCESSING", true);
CompletableFuture.supplyAsync(() -> callURL(response, request))
.thenAccept(result -> session.setAttribute("CACHED_RESULT", result));
}
return CompletableFuture.completedFuture("Still Processing");
}
return CompletableFuture.completedFuture(obj.toString());
});
}
HTML :
<body ng-app ng-controller="OrderFormController">
<header>
<div class="container">
<div id="branding">
<h1><span class="highlight">Review </span> Result</h1>
<form class="form-logout" role="form" action="/logout">
<input type="submit" value="Logout" id="logout" class="btn btn-primary btn-lg pull-right">
</form>
</div>
</div>
</header>
<img src="../img/spinner.gif" id="loaderImage" style='display:none'>
<div class="col-md-12 col-lg-12" ng-cloak="">
<div class="panel with-nav-tabs panel-default">
<div class="panel-heading">
<ul class="nav nav-tabs">
<li class="active">Classes</li>
<li>Triggers</li>
<li>Visualforce pages</li>
<li>Duplicate Codes</li>
</ul>
</div>
<div class="panel-body">
<div class="tab-content">
<div class="tab-pane fade in active" id="tab1default">
<ul class="col-md-12 col-lg-12">
<li ng-click="showErrorDetails(key)" class="col-sm-12 col-md-4 col-lg-4 eachClassCell"
ng-repeat='(key,value) in sampleJSON'>
<div ng-if="key.indexOf('.cls') > -1">
<div title="{{key}}" class="classNameLabel">{{key}}</div>
<div title="Error count" class="errorContainer">
<span class="errorCount">{{value.pmdStructures.length}}</span>
<span class="errorMeter"
ng-repeat="eachClass in value.pmdStructures | limitTo: 10"></span>
</div>
</div>
</li>
</ul>
</div>
<div class="tab-pane fade" id="tab2default">
<ul class="col-md-12 col-lg-12">
<li ng-click="showErrorDetails(key)" class="col-sm-12 col-md-4 col-lg-4 eachClassCell"
ng-repeat='(key,value) in sampleJSON'>
<div ng-if="key.indexOf('.trigger') > -1">
<div title="{{key}}" class="classNameLabel">{{key}}</div>
<div title="Error count" class="errorContainer">
<span class="errorCount">{{value.pmdStructures.length}}</span>
<span class="errorMeter"
ng-repeat="eachClass in value.pmdStructures | limitTo: 10"></span>
</div>
</div>
</li>
</ul>
</div>
<div class="tab-pane fade" id="tab3default">
<ul class="col-md-12 col-lg-12">
<li ng-click="showErrorDetails(key)" class="col-sm-12 col-md-4 col-lg-4 eachClassCell"
ng-repeat='(key,value) in sampleJSON'>
<div ng-if="key.indexOf('.page') > -1">
<div title="{{key}}" class="classNameLabel">{{key}}</div>
<div title="Error count" class="errorContainer">
<span class="errorCount">{{value.pmdStructures.length}}</span>
<span class="errorMeter"
ng-repeat="eachClass in value.pmdStructures | limitTo: 10"></span>
</div>
</div>
</li>
</ul>
</div>
<div class="tab-pane fade" id="tab4default">
<ul class="col-md-12 col-lg-12">
<li ng-click="showDuplicateDetails(eachValue)" class="col-sm-12 col-md-4 col-lg-4 eachClassCell"
ng-repeat='eachValue in sampleJSONDuplicates'>
<div title="{{eachValue.duplicationInFile}}" class="classNameLabel">
{{eachValue.duplicationInFile}}
</div>
</li>
</ul>
</div>
</div>
</div>
</div>
</div>
<form>
</form>
<footer>
<p>Salesforce Free Code Review, Copyright © 2017</p>
</p>
</footer>
<script type="application/javascript">
</script>
</body>
What this does is to avoid timeout, it sends a Still Processing message to the client, as the timeout limit is 30000ms,
I know the first thing that comes to mind is do a lazy loading with some 30 data per request, but believe me I cant.
I have built a code review tool, so when the URL gets hit, it runs a code review on all the classes with a background thread and then returns the response in CompletableFuture after some 120 - 150 seconds. This response is unknown to the client until the page is refreshed again.
Even after refreshing the page, as the data is over 7MB the page takes almost 13-35 seconds to display the result.
The code works fine in my local, but when I hosted in heroku, which is shared server, I started getting timeouts.
There are two issues :
1) How can angularJS detect changes in response of the RestAPI ?(As there will be two responses sent, first the immediate on "Still Processing" and second the real result. This is one which I have to consume).
2) How to lazy load the data which is already in JSON format from the Controller?
Update 1:
I tried using OutputStream too:
Controller :
#RequestMapping("/utilities/longProcessStream")
public StreamingResponseBody asyncLongProcessStream(HttpServletResponse response, HttpServletRequest request) {
return new StreamingResponseBody() {
#Override
public void writeTo(OutputStream outputStream) throws IOException {
PMDController.this.callURL(response, request, outputStream);
}
};
}
But now the response changes to XML and the JS console.log('Data from server'); line is not printed untill full response is gathered from the server side.
When the response changes to XML how can I parse that to JSON format as my full JS depends on JSON?
After streaming why angular displaying result after full request is completed?
I found a library which reads StreamingBodyResponse, they internally change blob type to JSON which is exactly what i needed :
#RequestMapping("/utilities/longProcessStream")
public StreamingResponseBody asyncLongProcessStream(HttpServletResponse response, HttpServletRequest request) {
response.addHeader("Content-Type", MediaType.APPLICATION_JSON);
return new StreamingResponseBody() {
#Override
public void writeTo(OutputStream outputStream) throws IOException {
PMDController.this.callURL(response, request, outputStream);
}
};
}
JS:
$('#loaderImage').show();
$scope.sampleJSONClass = {};
$scope.sampleJSONTrigger = {};
$scope.sampleJSONPages = {};
oboe('/utilities/longProcessStream')
.done(function(data) {
var dataFromServer = data.pmdStructureWrapper;
if (Object.keys(dataFromServer)[0].endsWith('.cls')) {
$scope.sampleJSONClass[Object.keys(dataFromServer)[0]] = Object.values(dataFromServer)[0];
}
if (Object.keys(dataFromServer)[0].endsWith('.trigger')) {
$scope.sampleJSONTrigger[Object.keys(dataFromServer)[0]] = Object.values(dataFromServer)[0];
}
if (Object.keys(dataFromServer)[0].endsWith('.page')) {
$scope.sampleJSONPages[Object.keys(dataFromServer)[0]] = Object.values(dataFromServer)[0];
}
$scope.$apply();
$('#loaderImage').hide();
})
.fail(function() {
console.log('error');
});
This works perfectly.
I'm working on a project and I faced a problem. I need to scrape data from the website that contains following HTML code:
<div class="lin-curso" style="border: 0;">
<div class="lin-area-c3">
Vagas 2017
</div>
</div>
<div class="box10">
<div class="lin-area-c1">
L160
</div>
<div class="lin-area-c2">
Acupuntura
</div>
<div class="lin-area-c3">
[Lic-1º cic]
</div>
</div>
<div class="lin-curso">
<div class="lin-curso-c1">
</div>
<div class="lin-curso-c2">
3155
</div>
<div class="lin-curso-c3">
Instituto Politécnico de Setúbal - Escola Superior de Saúde
</div>
<div class="lin-curso-c4">
20
</div>
</div>
<br>
<div class="box10">
<div class="lin-area-c1">
9059
</div>
<div class="lin-area-c2">
Administração e Gestão de Empresas
</div>
<div class="lin-area-c3">
[Lic-1º cic]
</div>
</div>
<div class="lin-curso">
<div class="lin-curso-c1">
</div>
<div class="lin-curso-c2">
2270
</div>
<div class="lin-curso-c3">
Universidade Católica Portuguesa - Faculdade de Ciências Económicas e Empresariais
</div>
<div class="lin-curso-c4">
n.d.
</div>
</div>
<br>
<div class="box10">
<div class="lin-area-c1">
8056
</div>
<div class="lin-area-c2">
Administração e Gestão Pública
</div>
<div class="lin-area-c3">
[Lic-1º cic]
</div>
</div>
<div class="lin-curso">
<div class="lin-curso-c1">
</div>
<div class="lin-curso-c2">
4275
</div>
<div class="lin-curso-c3">
Instituto Superior de Ciências da Administração
</div>
<div class="lin-curso-c4">
20
</div>
</div>
<br>
<div class="box10">
<div class="lin-area-c1">
8194
</div>
<div class="lin-area-c2">
Administração da Guarda Nacional Republicana
</div>
<div class="lin-area-c3">
[Mest Integ]
</div>
</div>
<div class="lin-curso">
<div class="lin-curso-c1">
</div>
<div class="lin-curso-c2">
7510
</div>
<div class="lin-curso-c3">
Academia Militar
</div>
<div class="lin-curso-c4">
n.d.
</div>
</div>
<br>
<div class="box10">
<div class="lin-area-c1">
9672
</div>
<div class="lin-area-c2">
Administração e Marketing
</div>
<div class="lin-area-c3">
[Lic-1º cic]
</div>
</div>
BOX10 and line-curso should form an element and they don't.
Because in some lines there is only one BOX10 for one Lin-curso but there are lines that are like Lin-curso for one Box10 , if Box10 and Lin-curso were an element there wouldn't be a problem , is there a way I can associate those two ?
EDIT: The website link is this : http://www.dges.gov.pt/guias/indcurso.asp?letra=A
And the element is the ".inside"
Solution to this problem is fairly easy when you use sibling selector. In your case div with class box10 plays role of a header in the table and sibling divs with class lin-curso play role of table data rows. I would suggest firstly selecting all divs with class box10:
Elements boxes = doc.select("div.box10");
Then you can iterate over boxes and do two major things:
Extract data you are interested in from this div (it contains 3 child nodes, divs with classes lin-area-c1, lin-area-c2 and lin-area-c3)
Select sibling nodes with class lin-curso and extract data from them.
Jsoup provides a method called Element.nextElementSibling() that return sibling element to the element you called this method on. So when you call it on element div.box10 you will get sibling element div.lin-curso.
Sibling in this case means a node immediately following the specified node at the same tree level.
Exemplary solution
Below you can find exemplary code that parses given website and prints table to the console output:
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import org.jsoup.select.Elements;
import java.io.IOException;
final class TestMain {
public static void main(String[] args) throws IOException {
Document doc = Jsoup.connect("http://www.dges.gov.pt/guias/indcurso.asp?letra=A").get();
Elements boxes = doc.select("div.box10");
for (Element box : boxes) {
String linAreaC1 = box.select(".lin-area-c1").text();
String linAreaC2 = box.select(".lin-area-c2").text();
String linAreaC3 = box.select(".lin-area-c3").text();
System.out.printf("%s: %s %s%n", linAreaC1, linAreaC2, linAreaC3);
Element linCurso = box.nextElementSibling();
while (linCurso.hasClass("lin-curso")) {
String linCursoC2 = linCurso.select(".lin-curso-c2").text();
String linCursoC3 = linCurso.select(".lin-curso-c3").text();
String linCursoC4 = linCurso.select(".lin-curso-c4").text();
System.out.printf("%s\t%s\t%s%n", linCursoC2, linCursoC3, linCursoC4);
linCurso = linCurso.nextElementSibling();
}
System.out.println("==============================");
}
}
}
I hope it helps.
I have an older JSP web page which was working perfectly fine before on apace tomcat 7. Now I am trying to restore the page and I added it on server (with the same apache tomcat 7 server) and it does't work any more as it was suppose to.
Than I checked the source code on page and I saw that at some random point the html code is not written anymore.
If anyone can help me with this.
code where the html stops:
<!-- Portfolio Modals -->
<!-- Use the modals below to showcase details about your portfolio projects! -->
<!-- Portfolio Modal 1 -->
<%
for(int i=0; i<seznamID.size(); i++){
id=seznamID.get(i);
title=seznamTitle.get(i);
description=seznamDescription.get(i);
mainThumbnail=seznamMainThumbnail.get(i);
investor=seznamInvestor.get(i);
year=seznamYear.get(i);
%>
<div class="portfolio-modal modal fade" id="portfolioModal<%=id %>" tabindex="-1" role="dialog" aria-hidden="true">
<div class="modal-dialog">
<div class="modal-content">
<div class="close-modal" data-dismiss="modal">
<div class="lr">
<div class="rl">
</div>
</div>
</div>
<div class="container">
<div class="row">
<div class="col-lg-8 col-lg-offset-2">
<div class="modal-body">
<!-- Project Details Go Here -->
<h2><%=title %></h2><br/>
<div class="container">
<div class="row">
<div class="col-md-4">
<%
String sql4="SELECT name FROM webPage.images WHERE eng_references_ideng_references='"+id+"';";
Class.forName("com.mysql.jdbc.Driver");
Connection con4 = DriverManager.getConnection("jdbc:mysql://path_to_db?useUnicode=true&characterEncoding=UTF-8",
"username", "password");
Statement st4 = con4.createStatement();
ResultSet rs4= st4.executeQuery(sql4);
ArrayList <String>seznamImages=new ArrayList();
String images=null;
while (rs4.next()){
images=rs4.getString("name");
seznamImages.add(images);
}
con4.close();
rs4.close();
st4.close();
%>
<div class="fotorama" data-nav="thumbs" data-width="90%" data-ratio="800/600 " data-loop="true">
<img src="img/references/<%=mainThumbnail%>">
<%for (int j=0; j<seznamImages.size();j++){
String slika=seznamImages.get(j);
%>
<img src="img/references/<%=slika%>">
<%} %>
</div>
</div>
<div class="col-md-6">
<p><%=description%></p>
<table class="table">
<tr>
<td>Investor</td>
<td><%=investor %></td>
</tr>
<tr>
<td>implementation</td>
<td><%=year %></td>
</tr>
<tr>
<td>Actions</td>
<td>
<%
String sql3="select a.title , c.title, c.ideng_references from action a inner join action_references b on a.idaction = b.action_idaction inner join eng_references c on b.eng_references_ideng_references = c.ideng_references where c.ideng_references='"+id+"';";
Connection con3 = DriverManager.getConnection("jdbc:path_to_db?useUnicode=true&characterEncoding=UTF-8",
"username", "password");
Statement st3 = con3.createStatement();
ResultSet rs3=st3.executeQuery(sql3);
ArrayList <String>seznamAction=new ArrayList();
String action=null;
while(rs3.next()){
action=rs3.getString("title");
seznamAction.add(action);
}
con3.close();
st3.close();
rs3.close();
for(int j=0; j<seznamAction.size(); j++){
action=seznamAction.get(j);
out.print(action+"<br/>");
}
%>
</td>
</tr>
</table>
</div>
</div><br/>
<button type="button" class="btn btn-primary center" data-dismiss="modal"><i class="fa fa-times"></i> Close</button>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
source code from web page:
HTML stop generated because of exception in code.
Add:
Try {
Catch (Throwable e){e.printStackTrace();}
Surronding your code and find problem
I need to locate each item and click on it.
I tired this:
List<WebElement> allTournaments = driver.findElements(By.xpath("//*[#id='main-section']/div/div/section/div[1]/ul/li/div/div[1]/div"));
for (WebElement tournament : allTournaments) {
tournament.click();
}
But got the error "element not visible"
The xpath of first element is //[#id='main-section']/div/div/section/div1/ul/li1/div/div1/div
xpath of second element is //[#id='main-section']/div/div/section/div1/ul/li[2]/div/div1/div
like wise..
can anybody help..?
HTML:
<div class="header-toggle-area">
<div class="header-notification-area text-right ng-scope" ng-controller="notificationController">
</header>
<!--HEADER ENDS-->
<div class="content-wrap" style="padding-top: 60px;">
<aside id="main-sidebar">
<!--MAIN SIDEBAR ENDS-->
<main id="main-section">
<div class="main-content">
<div class="ng-scope" ng-controller="NewTournamentsController as tournaments">
<header class="member-title dt-title-main">
<section class="dashboard-tournaments-listing">
<div class="" ng-show="tournaments.Page==1">
<ul class="row dashboard-listing">
<!-- ngRepeat: running in tournaments.runningTournaments -->
<li class="dashboard-listing-items ng-scope" ng-repeat="running in tournaments.runningTournaments" style="">
<div class="dtslider-block">
<div class="dtslider-content">
<img ng-src="../../category/categoryDefault.jpg" alt="slider" src="../../category/categoryDefault.jpg"/>
<div class="hover-contents">
<div class="hc-timer">
<p> Tournament ends in </p>
<div class="hct-ticker ng-binding"> 04:30 - 26/11/16 </div>
</div>
<div class="hc-details">
<div class="hcd-block ng-hide" ng-show="running.JoinedPlayers>0">
<ul class="hc-icons">
</div>
</div>
</div>
Check if each element is visible and clickable as follows:
for (WebElement tournament : allTournaments) {
if (tournament.isDisplayed()
&& ExpectedConditions.elementToBeClickable(tournament).equals(true)) {
tournament.click();
}
}
I'm using Vaadin 7. When I try to add html content to CustomLayout it does not generate a paragraph inside a div element. Here is the input :
<div style="margin-top: 10px; padding-left: 50px;" class="col-lg-10">
<div class="bs-component">
<ul class="nav nav-pills">
<li class="active"><a href="#Myprofile" class="active"
data-toggle="tab">Profile</a></li>
<li class="">setting</li>
<li class="">Tasks</li>
</ul>
<div class="tabbable">
<div class="tab-content">
<div id="Myprofile" location="Myprofile" class="tab-pane active"></div>
<div id="setting" location="setting" class="tab-pane">
<p>My paragraph</p>
</div>
</div>
</div>
</div>
In java I use this :
String html; //This var contain my input
InputStream is = new ByteArrayInputStream(html.getBytes());
CustomLayout custom = null;
try {
custom = new CustomLayout(is);
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
Finaly I add the customLayout to my mainLayout. When I run the application I get this output :
<div style="margin-top: 10px; padding-left: 50px;" class="col-lg-10">
<div class="bs-component">
<ul class="nav nav-pills">
<li class="active"><a href="#Myprofile" class="active"
data-toggle="tab">Profile</a></li>
<li class="">setting</li>
<li class="">Tasks</li>
</ul>
<div class="tabbable">
<div class="tab-content">
<div id="Myprofile" location="Myprofile" class="tab-pane active"></div>
<div id="setting" location="setting" class="tab-pane">
</div>
</div>
</div>
</div>
My paragraph inside div with id=setting was not generated by Vaadin. Any Ideas to fix this problem ?
I found the source of the problem. I removed the location="setting" part in the div element and it worked. I don't know why It does that. Probably because CustomLayout when generating the html looks for elements to put in the location, if it does not find one it clean all the element content.
Don't you need the location="setting" attribute? Don't you insert a component for that attribute?