We Keep Coding

I need Freemarker documentation for data model when using #recurse with trees other than XML

I am trying to process a tree structure in Freemarker and would like use the #recurse, #visit directives but I can't find any good documentation on how to set up the data model. The only examples I can see are those that create a data model for an XML structure. I don't need it to be so detailed. My tree is very simple.
In trying to test the functionality I need, I built a unit test but when I run it I get
FreeMarker template error:
For "." left-hand operand: Expected a hash, but this has evaluated to a node
Here is the source code:
public class FreemarkerXmlTests {
static class Element implements TemplateNodeModel {
private final String name;
private final String text;
private Element parent;
private final List<Element> elements = new ArrayList<>();
public Element(String name) {
this(name, null);
}
public Element(String name, String text) {
this.name = name;
this.text = text;
}
public void add(Element element) {
element.parent = this;
this.elements.add(element);
}
public List<Element> getElements() {
return this.elements;
}
public String getName() {
return this.name;
}
public String getText() {
return this.text;
}
public String getTitle() {
return this.name;
}
public TemplateModel get(String key) {
return null;
}
#Override
public TemplateNodeModel getParentNode() throws TemplateModelException {
return this.parent;
}
#Override
public TemplateSequenceModel getChildNodes() throws TemplateModelException {
// TODO Auto-generated method stub
return new SimpleSequence(this.elements, cfg.getObjectWrapper());
}
#Override
public String getNodeName() throws TemplateModelException {
return this.name;
}
#Override
public String getNodeType() throws TemplateModelException {
return this.name;
}
#Override
public String getNodeNamespace() throws TemplateModelException {
return null;
}
}
private static Configuration cfg;
private static final String myTestTemplate = "<#recurse doc>\r\n" +
"\r\n" +
"<#macro book>\r\n" +
" Book element with title ${.node.title} \r\n" +
" <#recurse>\r\n" +
" End book\r\n" +
"</#macro>\r\n" +
"\r\n" +
"<#macro title>\r\n" +
" Title element\r\n" +
"</#macro>\r\n" +
"\r\n" +
"<#macro chapter>\r\n" +
" Chapter element with title: ${.node.title}\r\n" +
"</#macro>";
#BeforeClass
public static void classInit() throws IOException {
StringTemplateLoader stringTemplateLoader = new StringTemplateLoader();
stringTemplateLoader.putTemplate("myTestTemplate", myTestTemplate);
cfg = new Configuration(Configuration.VERSION_2_3_29);
cfg.setTemplateLoader(stringTemplateLoader);
cfg.setDefaultEncoding("UTF-8");
cfg.setTemplateExceptionHandler(TemplateExceptionHandler.RETHROW_HANDLER);
cfg.setLogTemplateExceptions(false);
cfg.setWrapUncheckedExceptions(true);
cfg.setFallbackOnNullLoopVariable(false);
}
#Test
public void basicXmlTest() throws TemplateException, IOException {
Element doc = new Element("doc");
Element book = new Element("book");
book.add(new Element("title", "Test Book"));
doc.add(book);
Element chapter1 = new Element("chapter");
chapter1.add(new Element("title", "Ch1"));
chapter1.add(new Element("para", "p1.1"));
chapter1.add(new Element("para", "p1.2"));
chapter1.add(new Element("para", "p1.3"));
book.add(chapter1);
Element chapter2 = new Element("chapter");
chapter2.add(new Element("title", "Ch2"));
chapter2.add(new Element("para", "p2.1"));
chapter2.add(new Element("para", "p2.2"));
chapter2.add(new Element("para", "p2.3"));
book.add(chapter2);
Map<String, Object> root = new HashMap<>();
// Put string "user" into the root
root.put("doc", doc);
Template temp = cfg.getTemplate("myTestTemplate");
Writer out = new OutputStreamWriter(System.out);
temp.process(root, out);
}
Any ideas?

Take a look at the freemarker.template.TemplateNodeModel interface. Your objects have to implement that, or they have to be wrapped (via the ObjectWrapper) into a TemplateModel the implements that. Then #recurse/#visit/?parent/?children/etc. will work with them.
Here's an example of implementing TemplateNodeModel for traversing JSON: https://github.com/freemarker/fmpp/blob/master/src/main/java/fmpp/models/JSONNode.java
Some templates where above is used:
https://github.com/freemarker/fmpp/tree/master/src/test/resources/tests/dl_json/src
As of the . operator, for that you need to implement TemplateHashModel (or its sub-interfaces, like TemplateHashModelEx2).

With the help of the examples posted by ddekany, I added the following:
implements TemplateHashModel to the Element class
static class Element implements TemplateNodeModel, TemplateHashModel {
a method to the Element class and the unit test worked:
#Override
public TemplateModel get(String key) throws TemplateModelException {
switch (key) {
case "title":
case "name":
return cfg.getObjectWrapper().wrap(this.name);
default:
throw new TemplateModelException("unknown hash get: " + key);
}
}

jsoup to get div elements with classes

I am new to Jsoup parsing and I want to get the list of all the companies on this page: https://angel.co/companies?company_types[]=Startup
Now, a way to do this is actually to inspect the page with the div tags relevant to what I need.
However, when I call the method :
Document doc = Jsoup.connect("https://angel.co/companies?company_types[]=Startup").get();
System.out.println(doc.html());
Firstly I cannot even find those DIV tags in my consol html output, (the ones which are supposed to give a list of the companies)
Secondly, even if I did find it, how can I find a certain Div element with class name :
div class=" dc59 frw44 _a _jm"
Pardon the jargon, I have no idea how to go through this.

The data are not embedded in the page but they are retrieved using subsequent API calls :
a POST https://angel.co/company_filters/search_data to get an ids array & a token named hexdigest
a GET https://angel.co/companies/startups to retrieve company data using the output from the previous request
The above is repeated for each page (thus a new token & a list of ids are needed for each page). This process can be seen using Chrome dev console in Network tabs.
The first POST request gives JSON output but the second request (GET) gives HTML data in a property of a JSON object.
The following extracts the company filter :
private static CompanyFilter getCompanyFilter(final String filter, final int page) throws IOException {
String response = Jsoup.connect("https://angel.co/company_filters/search_data")
.header("Content-Type", "application/x-www-form-urlencoded;charset=UTF-8")
.header("X-Requested-With", "XMLHttpRequest")
.data("filter_data[company_types][]=", filter)
.data("sort", "signal")
.data("page", String.valueOf(page))
.userAgent("Mozilla")
.ignoreContentType(true)
.post().body().text();
GsonBuilder gsonBuilder = new GsonBuilder();
Gson gson = gsonBuilder.create();
return gson.fromJson(response, CompanyFilter.class);
}
Then the following extracts companies :
private static List<Company> getCompanies(final CompanyFilter companyFilter) throws IOException {
List<Company> companies = new ArrayList<>();
URLConnection urlConn = new URL("https://angel.co/companies/startups?" + companyFilter.buildRequest()).openConnection();
urlConn.setRequestProperty("User-Agent", "Mozilla");
urlConn.connect();
BufferedReader reader = new BufferedReader(new InputStreamReader(urlConn.getInputStream(), "UTF-8"));
HtmlContainer htmlObj = new Gson().fromJson(reader, HtmlContainer.class);
Element doc = Jsoup.parse(htmlObj.getHtml());
Elements data = doc.select("div[data-_tn]");
if (data.size() > 0) {
for (int i = 2; i < data.size(); i++) {
companies.add(new Company(data.get(i).select("a").first().attr("title"),
data.get(i).select("a").first().attr("href"),
data.get(i).select("div.pitch").first().text()));
}
} else {
System.out.println("no data");
}
return companies;
}
The main function :
public static void main(String[] args) throws IOException {
int pageCount = 1;
List<Company> companies = new ArrayList<>();
for (int i = 0; i < 10; i++) {
System.out.println("get page n°" + pageCount);
CompanyFilter companyFilter = getCompanyFilter("Startup", pageCount);
pageCount++;
System.out.println("digest : " + companyFilter.getDigest());
System.out.println("count : " + companyFilter.getTotalCount());
System.out.println("array size : " + companyFilter.getIds().size());
System.out.println("page : " + companyFilter.getpage());
companies.addAll(getCompanies(companyFilter));
if (companies.size() == 0) {
break;
} else {
System.out.println("size : " + companies.size());
}
}
}
Company, CompanyFilter & HtmlContainer are model class :
class CompanyFilter {
#SerializedName("ids")
private List<Integer> mIds;
#SerializedName("hexdigest")
private String mDigest;
#SerializedName("total")
private String mTotalCount;
#SerializedName("page")
private int mPage;
#SerializedName("sort")
private String mSort;
#SerializedName("new")
private boolean mNew;
public List<Integer> getIds() {
return mIds;
}
public String getDigest() {
return mDigest;
}
public String getTotalCount() {
return mTotalCount;
}
public int getpage() {
return mPage;
}
private String buildRequest() {
String out = "total=" + mTotalCount + "&";
out += "sort=" + mSort + "&";
out += "page=" + mPage + "&";
out += "new=" + mNew + "&";
for (int i = 0; i < mIds.size(); i++) {
out += "ids[]=" + mIds.get(i) + "&";
}
out += "hexdigest=" + mDigest + "&";
return out;
}
}
private static class Company {
private String mLink;
private String mName;
private String mDescription;
public Company(String name, String link, String description) {
mLink = link;
mName = name;
mDescription = description;
}
public String getLink() {
return mLink;
}
public String getName() {
return mName;
}
public String getDescription() {
return mDescription;
}
}
private static class HtmlContainer {
#SerializedName("html")
private String mHtml;
public String getHtml() {
return mHtml;
}
}
The full code is also available here

Extracting the value of an element from a JSON

I'm trying to make a test where I get some documents based on the id of the batch they belong to. More specifically, I want to check that a specific batchPublicId is in the response body. I am using okhttp for the test.
This a shorter version of the json:
{
"_embedded": {
"invoices": [
{
"type": "INVOICE",
"publicId": "27bc8426-17cf-4fe5-9278-64108ae05e4b",
"deliveryStatus": null,
"processingStatus": "INITIATED",
"batchPublicId": "0000000000000000000000001"
}
]
}
}
I'm new to json and this is how far I got with the problem:
String invoicesJsonData = response.body().string();
JSONObject invoicesJsonObject = new JSONObject(invoicesJsonData);
Assert.assertTrue(invoicesJsonObject.getJSONObject("_embedded") !=null && invoicesJsonObject.getJSONObject("_embedded").has("invoices"));
I would like to verify that batchPublicId has the value mentioned in the json. Is there a way to do this? Thank you.

String invoicesJsonData = response.body().string();
JSONObject invoicesJsonObject = new JSONObject(invoicesJsonData);
JSONObject invoicesJsonObject1 = invoicesJsonObject.getJSONObject("_embedded");
JSONArray f2=invoicesJsonObject1.getJSONArray("invoices");
for(int i=0;i<f2.length();i++){
JSONObject obj=f2.getJSONObject(i);
if(obj.get("batchPublicId")!=null){
System.out.println(obj.get("batchPublicId"));
}

You can do something like this,Which worked out for me sometimes back.
String invoicesJsonData = response.body().string();
JSONObject invoicesJsonObject = new JSONObject(invoicesJsonData);
JSONObject invoicesJsonObject = json.getJSONObject("invoicesJsonObject");
String batchPublicId = invoicesJsonObject.getString("batchPublicId");
System.out.println( "batchPublicId: " + batchPublicId );
if(batchPublicId !=null){
// do something
}
Not sure about the syntax.Giving you a hint.

you can check any keys is there in json object or not like below :
if(jsonObject1.has("batchPublicId")){
String batchPublicId = jsonObject1.optString("batchPublicId");
Log.i(getClass().getSimpleName(), "batchPublicId=" + batchPublicId);}
has method is used to find any key is there in jsonobject or not.

In my opinion, a better approach for this would be to create a POJO from this JSON string, and extract the information you need using simply the getters
For example:
Wrapper class:
#JsonIgnoreProperties(ignoreUnknown = true)
#JsonRootName(value = "_embedded")
public class Embeded {
#JsonProperty("invoices")
private List<Invoice> invoices;
public Embeded() {}
public List<Invoice> getInvoices() {
return invoices;
}
public void setInvoices(List<Invoice> invoices) {
this.invoices = invoices;
}
}
Invoice class:
#JsonIgnoreProperties(ignoreUnknown = true)
public class Invoice {
#JsonProperty("type")
private String type;
#JsonProperty("publicId")
private String publicId;
#JsonProperty("deliveryStatus")
private String deliveryStatus;
#JsonProperty("processingStatus")
private String processingStatus;
#JsonProperty("batchPublicId")
private String batchPublicId;
public Invoice() {}
public String getType() {
return type;
}
public void setType(String type) {
this.type = type;
}
public String getPublicId() {
return publicId;
}
public void setPublicId(String publicId) {
this.publicId = publicId;
}
public String getDeliveryStatus() {
return deliveryStatus;
}
public void setDeliveryStatus(String deliveryStatus) {
this.deliveryStatus = deliveryStatus;
}
public String getProcessingStatus() {
return processingStatus;
}
public void setProcessingStatus(String processingStatus) {
this.processingStatus = processingStatus;
}
public String getBatchPublicId() {
return batchPublicId;
}
public void setBatchPublicId(String batchPublicId) {
this.batchPublicId = batchPublicId;
}
}
Test:
public void json_test() throws JsonParseException, JsonMappingException, IOException {
String json = "{"
+ "\"_embedded\": {"
+ "\"invoices\": ["
+ "{"
+ "\"type\": \"INVOICE\","
+ "\"publicId\": \"27bc8426-17cf-4fe5-9278-64108ae05e4b\","
+ "\"deliveryStatus\": null,"
+ "\"processingStatus\": \"INITIATED\","
+ "\"batchPublicId\": \"0000000000000000000000001\""
+ "}"
+ "]"
+ "}"
+ "}";
ObjectMapper mapper = new ObjectMapper();
mapper.configure(Feature.UNWRAP_ROOT_VALUE, true);
List<Invoice> invoices = mapper.readValue(json, Embeded.class).getInvoices();
Assert.assertTrue(StringUtils.equals(invoices.get(0).getBatchPublicId(), "0000000000000000000000001"));
}

If I understand your right, you just need to call:
Assert.assertTrue(invoicesJsonObject.getString("batchPublicId").equals("0000000000000000000000001"));"

If you want to create a test for JSON Validation, you can use the JSONAssert.
JSONAsset give the method assertEquals, that compare two json structures, strict identic or not.
final String expected_result = YOUR_EXPECTED_RESULT;
JSONAssert.assertEquals(YOUR_EXPECTED_JSON_RESULT, RESULT_FROM_RESPONSE_BODY, false);
The last boolean parameter defines if you want an strict comparation or just compare if your expected result is in result from response.

Splitting skips half a content when splitting by #

I have an android application that requests data from a text file. when I try to separate the data and initialize classes, I have serious trouble because half the data after splitting is missing. The code for the DataSource is:
indicators = new ArrayList<Indicator>();
try {
InputStream fileStream = getResources().openRawResource(
R.raw.indicators);
int fileLen = fileStream.available();
// Read the entire resource into a local byte buffer.
byte[] fileBuffer = new byte[fileLen];
fileStream.read(fileBuffer);
fileStream.close();
displayText = new String(fileBuffer);
String[] counts = displayText.split("#");
for(String i: counts) {
String[] temps = i.split(";");
indicators.add(new Indicator(temps[0],temps[1],temps[2]));
}
} catch (IOException e) {
// exception handling
}
for(Indicator i: indicators) Log.v("indicator", i.toString());
and the code for the Indicator class can be found here:
public class Indicator {
private String name;
private String isoCode;
private String topic;
public Indicator(String topic, String isoCode,String name) {
this.name = name;
this.isoCode = isoCode;
this.topic = topic;
}
public String getName() {
return this.name;
}
public String getIsoCode() {
return this.isoCode;
}
public String getTopic() {
return this.topic;
}
public String toString() {
return (this.topic + "," + this.isoCode + "," + this.name);
}
}
After doing this process the following log file comes up with a lot of content missing:
http://pastebin.com/1j5s1Z81
The file seems to be skipping every other entry and because of that, my entire software is messing up. The source file below is:
http://pastebin.com/eAzppMdb

I have not been able to solve the issue by ammending the command. I found a way around this problem however by using the BufferedReader class that read the opened string in. This has seemed to solve the issue.

Parsing files in java/android

I have a text file that has is set out like the following
Title - Welcome to the Dibb
Date - 13/03/11
Information - Hello and welcome to our website.
Title - Welcome to student room
Date - 06/05/11
Information - Hello and welcome to the student room. We are a online forum that allows previous and current students to ask questions.
I need to parse this text file and save things like the title line, date line and the rest will be saved as information. I know how to read the file and save the full file as a string but I am stuck on getting the select information.
CODE
This is the code I have used to read the text file
helloTxt.setText(readTxt());
}
private String readTxt() {
InputStream inputStream = getResources().openRawResource(R.raw.pages);
ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream();
int i;
try {
i = inputStream.read();
while (i != -1) {
byteArrayOutputStream.write(i);
i = inputStream.read();
}
inputStream.close();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
String str = byteArrayOutputStream.toString();
return str;
}

Read file line by line
BufferedReader br = new BufferedReader(new FileReader(file));
String line;
while ((line = br.readLine()) != null) {
// process the line.
}
br.close();
If you can guarantee that every line has as maximum one - then you can use following pattern.
String[] tokens = line.split("\s-\s");
For this line
Title - Welcome to the Dibb
It would give you
tokens[0] = "Title";
tokens[1] = "Welcome to the Dibb";

I try to write some classes which can help you to approach your issue
import java.io.IOException;
import java.io.InputStream;
import java.util.ArrayList;
import java.util.List;
public class Test4 {
private List<Information> parser(String data) {
List<Information> informations = new ArrayList<Information>();
String blocks[] = data.split("\n\r");
for(String block : blocks) {
String[] lines = block.split("\n");
Information information = new Information();
information.setTitle((lines[0].split("-"))[1].trim());
information.setDate((lines[1].split("-"))[1].trim());
information.setInfo((lines[2].split("-"))[1].trim());
informations.add(information);
}
return informations;
}
private void runner() throws IOException {
InputStream inputStream = getClass().getResourceAsStream("input.txt");
String input = "";
int cc;
while((cc = inputStream.read()) != -1) {
input += (char) cc;
}
List<Information> informations = parser(input);
for(Information information : informations) {
System.out.println(information);
}
}
/**
* #param args
* #throws IOException
*/
public static void main(String[] args) throws IOException {
Test4 test4 = new Test4();
test4.runner();
}
class Information {
private String title;
private String date;
private String info;
public String getTitle() {
return title;
}
public void setTitle(String title) {
this.title = title;
}
public String getDate() {
return date;
}
public void setDate(String date) {
this.date = date;
}
public String getInfo() {
return info;
}
public void setInfo(String info) {
this.info = info;
}
#Override
public String toString() {
return "Information [" + (title != null ? "title=" + title + ", " : "")
+ (date != null ? "date=" + date + ", " : "") + (info != null ? "info=" + info : "") + "]";
}
}
}