I am new to Jsoup parsing and I want to get the list of all the companies on this page: https://angel.co/companies?company_types[]=Startup
Now, a way to do this is actually to inspect the page with the div tags relevant to what I need.
However, when I call the method :
Document doc = Jsoup.connect("https://angel.co/companies?company_types[]=Startup").get();
System.out.println(doc.html());
Firstly I cannot even find those DIV tags in my consol html output, (the ones which are supposed to give a list of the companies)
Secondly, even if I did find it, how can I find a certain Div element with class name :
div class=" dc59 frw44 _a _jm"
Pardon the jargon, I have no idea how to go through this.
The data are not embedded in the page but they are retrieved using subsequent API calls :
a POST https://angel.co/company_filters/search_data to get an ids array & a token named hexdigest
a GET https://angel.co/companies/startups to retrieve company data using the output from the previous request
The above is repeated for each page (thus a new token & a list of ids are needed for each page). This process can be seen using Chrome dev console in Network tabs.
The first POST request gives JSON output but the second request (GET) gives HTML data in a property of a JSON object.
The following extracts the company filter :
private static CompanyFilter getCompanyFilter(final String filter, final int page) throws IOException {
String response = Jsoup.connect("https://angel.co/company_filters/search_data")
.header("Content-Type", "application/x-www-form-urlencoded;charset=UTF-8")
.header("X-Requested-With", "XMLHttpRequest")
.data("filter_data[company_types][]=", filter)
.data("sort", "signal")
.data("page", String.valueOf(page))
.userAgent("Mozilla")
.ignoreContentType(true)
.post().body().text();
GsonBuilder gsonBuilder = new GsonBuilder();
Gson gson = gsonBuilder.create();
return gson.fromJson(response, CompanyFilter.class);
}
Then the following extracts companies :
private static List<Company> getCompanies(final CompanyFilter companyFilter) throws IOException {
List<Company> companies = new ArrayList<>();
URLConnection urlConn = new URL("https://angel.co/companies/startups?" + companyFilter.buildRequest()).openConnection();
urlConn.setRequestProperty("User-Agent", "Mozilla");
urlConn.connect();
BufferedReader reader = new BufferedReader(new InputStreamReader(urlConn.getInputStream(), "UTF-8"));
HtmlContainer htmlObj = new Gson().fromJson(reader, HtmlContainer.class);
Element doc = Jsoup.parse(htmlObj.getHtml());
Elements data = doc.select("div[data-_tn]");
if (data.size() > 0) {
for (int i = 2; i < data.size(); i++) {
companies.add(new Company(data.get(i).select("a").first().attr("title"),
data.get(i).select("a").first().attr("href"),
data.get(i).select("div.pitch").first().text()));
}
} else {
System.out.println("no data");
}
return companies;
}
The main function :
public static void main(String[] args) throws IOException {
int pageCount = 1;
List<Company> companies = new ArrayList<>();
for (int i = 0; i < 10; i++) {
System.out.println("get page n°" + pageCount);
CompanyFilter companyFilter = getCompanyFilter("Startup", pageCount);
pageCount++;
System.out.println("digest : " + companyFilter.getDigest());
System.out.println("count : " + companyFilter.getTotalCount());
System.out.println("array size : " + companyFilter.getIds().size());
System.out.println("page : " + companyFilter.getpage());
companies.addAll(getCompanies(companyFilter));
if (companies.size() == 0) {
break;
} else {
System.out.println("size : " + companies.size());
}
}
}
Company, CompanyFilter & HtmlContainer are model class :
class CompanyFilter {
#SerializedName("ids")
private List<Integer> mIds;
#SerializedName("hexdigest")
private String mDigest;
#SerializedName("total")
private String mTotalCount;
#SerializedName("page")
private int mPage;
#SerializedName("sort")
private String mSort;
#SerializedName("new")
private boolean mNew;
public List<Integer> getIds() {
return mIds;
}
public String getDigest() {
return mDigest;
}
public String getTotalCount() {
return mTotalCount;
}
public int getpage() {
return mPage;
}
private String buildRequest() {
String out = "total=" + mTotalCount + "&";
out += "sort=" + mSort + "&";
out += "page=" + mPage + "&";
out += "new=" + mNew + "&";
for (int i = 0; i < mIds.size(); i++) {
out += "ids[]=" + mIds.get(i) + "&";
}
out += "hexdigest=" + mDigest + "&";
return out;
}
}
private static class Company {
private String mLink;
private String mName;
private String mDescription;
public Company(String name, String link, String description) {
mLink = link;
mName = name;
mDescription = description;
}
public String getLink() {
return mLink;
}
public String getName() {
return mName;
}
public String getDescription() {
return mDescription;
}
}
private static class HtmlContainer {
#SerializedName("html")
private String mHtml;
public String getHtml() {
return mHtml;
}
}
The full code is also available here
Related
I am trying to mock the getMeasureAggregator() of ResultSetRow object - I don't seem to succeed. I am very new in writing mockito unit test.
I want the BuilSQL.formatMeasuer() goes to case 2: so what I have decided to do was mocking row.getMeasureAggregator.
Here is my BuildSQL class:
public class SQLBuilder {
public static String buildSQL(JsonObject requestData, JsonObject queryInfo) throws AcquisitionException {
JsonArray jArray = queryInfo.get("columns").getAsJsonArray();
Set<String> columns = new HashSet<>(jArray.size());
for (int i = 0; i < jArray.size(); i++) {
columns.add(jArray.get(i).getAsString());
}
List<ResultSetRow> selectedRows = new ArrayList<>();
List<ResultSetRow> retrievedRows = null;
retrievedRows = MetaDataProvider.executeMetadataRequest(queryInfo, requestData); // this method returns a collection of RetrievedResultSetRow
for (ResultSetRow retrievedRow: retrievedRows) {
if (//some condition evaluates to true) {
selectedRows.add(retrievedRow);
}
}
String sql = "";
String select = "SELECT ";
for (int i = 0; i < selectedRows.size(); i++) {
ResultSetRow row = selectedRows.get(i);
select += formatMeasure(row.getMeatureName(), row.getMeasureAggregator());
}
select = select.substring(0, select.length() - 1);
return sql;
}
private static String formatMeasure(String measureName, int measureAggregator) {
switch(measureAggregator) {
case 1:
return "sum(\"" + measureName + "\")" + " AS \"" + measureName + "\",";
case 2:
return "COUNT(\"" + measureName + "\")" + " AS \"" + measureName + "\",";
return measureName;
}
}
here is my ResultSetRow class:
public class ResultSetRow {
private final int iRow;
private final int measureAggregator;
public ResultSetRow(JsonObject dimensionMetadata) {
this.iRow = dimensionMetadata.get("ROW").getAsInt();
this.measureAggregator = dimensionMetadata.get("MEASURE_AGGR").getAsInt();
}
public int getMeasureAggregator() {
return measureAggregator;
}
}
here is how I am mocking
#RunWith(PowerMockRunner.class)
#PrepareForTest({ HanaClientRequestUtils.class, RetrievedResultSetRow.class })
public class HanaSQLBuilderTest {
private ResultSetRow resultSetRow;
private requestData;
private queryInfo
#Test
public void formatMeatureExecuteCase2() throws Exception{
resultSetRow = Mockito.mock(ResultSetRow.class);
PowerMockito.when(resultSetRow.getMeasureAggregator()).thenReturn(2);
String querySQL = HanaSQLBuilder.buildSQL(requestData, queryInfo);
System.out.println(querySQL);
}
}
}
I am not sure why row.getMeasureAggregator() does not return 2?
Additionally to my comments above (I am coding by hand, forgive me some mistakes).
You can try with adding a retrievedRows as a parameter in Your buildSQL method:
public static String buildSQL(
JsonObject requestData,
JsonObject queryInfo,
List<ResultSetRow> retrievedRows) throws AcquisitionException {
// ... rest of Your code adopted to new parameter
}
And then provide Your retrievedRows filled with mocks
#Test
public void formatMeatureExecuteCase2() throws Exception{
resultSetRow = Mockito.mock(ResultSetRow.class);
PowerMockito.when(resultSetRow.getMeasureAggregator()).thenReturn(2);
List<ResultSetRow> retrievedRowsMock = new ArrayList<>(1);
retrievedRowsMock.add(resultSetRow);
String querySQL = HanaSQLBuilder.buildSQL(requestData, queryInfo, retrievedRowsMock);
System.out.println(querySQL);
}
}
I hope it will guide You to solution.
i have a method that puts some value(obtained from an excel file) into a hashmap with an array as the key
public HashMap<List<String>, List<String[]>> sbsBusServiceDataGnr() throws
IOException
{
System.out.println(engine.txtY + "Processing HashMap "
+ "sbsBusServiceData..." + engine.txtN);
int counterPass = 0, counterFail = 0, stopCounter = 0;
String dataExtract, x = "";
String[] stopInfo = new String[3];
List<String[]> stopsData = new ArrayList<String[]>();
List<String> serviceNum = new Vector<String>();
HashMap<List<String>, List<String[]>> sbsBusServiceData =
new HashMap<List<String>, List<String[]>>();
String dataPath = this.dynamicPathFinder(
"Data\\SBS_Bus_Routes.csv");
BufferedReader sbsBusServiceDataPop = new BufferedReader(
new FileReader(dataPath));
sbsBusServiceDataPop.readLine();
//Skips first line
while ((dataExtract = sbsBusServiceDataPop.readLine()) != null) {
try {
String[] dataParts = dataExtract.split(",", 5);
if (!dataParts[4].equals("-")){
if (Double.parseDouble(dataParts[4]) == 0.0){
sbsBusServiceData.put(serviceNum, stopsData);
String serviceNum1 = "null", serviceNum2 = "null";
if(!serviceNum.isEmpty()){
serviceNum1 = serviceNum.get(0);
serviceNum2 = serviceNum.get(1);
}
System.out.println("Service Number " + serviceNum1
+ ":" + serviceNum2 + " with " + stopCounter
+ " stops added.");
stopCounter = 0;
//Finalizing previous service
serviceNum.Clear();
serviceNum.add(0, dataParts[0]);
serviceNum.add(1, dataParts[1]);
//Adding new service
}
}
stopInfo[0] = dataParts[2];
stopInfo[1] = dataParts[3];
stopInfo[2] = dataParts[4];
stopsData.add(stopInfo);
//Adding stop to service
stopCounter++;
counterPass++;
}
catch (Exception e) {
System.out.println(engine.txtR + "Unable to process "
+ dataExtract + " into HashMap sbsBusServiceData."
+ engine.txtN + e);
counterFail++;
}
}
sbsBusServiceDataPop.close();
System.out.println(engine.txtG + counterPass + " number of lines"
+ " processed into HashMap sbsBusServiceData.\n" + engine.txtR
+ counterFail + " number of lines failed to process into "
+ "HashMap sbsBusServiceData.");
return sbsBusServiceData;
}
//Generates sbsBusServiceDataGnr HashMap : 15376 Data Rows
//HashMap Contents: {ServiceNumber, Direction},
// <{RouteSequence, bsCode, Distance}>
this method work for putting the values into the hashmap but i cannot seem to get any value from the hashmap when i try to call it there is always a nullpointerexception
List<String> sbsTest = new Vector<String>();
sbsTest.add(0, "10");
sbsTest.add(1, "1");
System.out.println(sbsBusServiceData.get(sbsTest));
try{
List<String[]> sbsServiceResults = sbsBusServiceData.get(sbsTest);
System.out.println(sbsServiceResults.size());
String x = sbsServiceResults.get(1)[0];
System.out.println(x);
} catch(Exception e){
System.out.println(txtR + "No data returned" + txtN + e);
}
this is a sample of the file im reading the data from:
SBS
How can i get the hashmap to return me the value i want?
Arrays are not suitable as keys in HashMaps, since arrays don't override Object's equals and hashCode methods (which means two different array instances containing the exact same elements will be considered as different keys by HashMap).
The alternatives are to use a List<String> instead of String[] as the key of the HashMap, or to use a TreeMap<String[]> with a custom Comparator<String[]> passed to the constructor.
If you are having fixed array size then the example I'm posting might be useful.
Here I've created two Object one is Food and Next is Product.
Here Food object is use and added method to get string array.
public class Product {
private String productName;
private String productCode;
public Product(String productName, String productCode) {
this.productName = productName;
this.productCode = productCode;
}
public String getProductName() {
return productName;
}
public void setProductName(String productName) {
this.productName = productName;
}
public String getProductCode() {
return productCode;
}
public void setProductCode(String productCode) {
this.productCode = productCode;
}
}
Food Model Class: Use as a Object instead of String[] and achieve String[] functionality.
public class Food implements Comparable<Food> {
private String type;
private String consumeApproach;
public Food(String type, String consumeApproach) {
this.type = type;
this.consumeApproach = consumeApproach;
}
public String getType() {
return type;
}
public void setType(String type) {
this.type = type;
}
public String getConsumeApproach() {
return consumeApproach;
}
public void setConsumeApproach(String consumeApproach) {
this.consumeApproach = consumeApproach;
}
public String[] FoodArray() {
return new String[] { this.type, this.consumeApproach };
}
//Implement compareTo method as you want.
#Override
public int compareTo(Food o) {
return o.getType().compareTo(this.type);
}
}
Using HashMap example
public class HashMapKeyAsArray {
public static void main(String[] args) {
HashMap<Food,List<Product>> map = dataSetLake();
map.entrySet().stream().forEach(m -> {
String[] food = m.getKey().FoodArray();
Arrays.asList(food).stream().forEach(f->{
System.out.print(f + " ");
});
System.out.println();
List<Product> list = m.getValue();
list.stream().forEach(e -> {
System.out.println("Name:" + e.getProductName() + " Produc Code:" + e.getProductCode());
});
System.out.println();
});
}
private static HashMap<Food,List<Product>> dataSetLake(){
HashMap<Food,List<Product>> data = new HashMap<>();
List<Product> fruitA = new ArrayList<>();
fruitA.add(new Product("Apple","123"));
fruitA.add(new Product("Banana","456"));
List<Product> vegetableA = new ArrayList<>();
vegetableA.add(new Product("Potato","999"));
vegetableA.add(new Product("Tomato","987"));
List<Product> fruitB = new ArrayList<>();
fruitB.add(new Product("Apple","123"));
fruitB.add(new Product("Banana","456"));
List<Product> vegetableB = new ArrayList<>();
vegetableB.add(new Product("Potato","999"));
vegetableB.add(new Product("Tomato","987"));
Food foodA = new Food("Fruits","Read To Eat");
Food foodB = new Food("Vegetables","Need To Cook");
Food foodC = new Food("VegetablesC","Need To Cook C");
data.put(foodA, fruitB);
data.put(foodB, vegetableB);
data.put(foodA, fruitA);
data.put(foodC, vegetableA);
return data;
}
Using TreeMap example
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
import java.util.TreeMap;
public class TreeMapKeyAsArray {
public static void main(String[] args) {
TreeMap<Food, List<Product>> map = dataSetLake();
map.entrySet().stream().forEach(m -> {
String[] food = m.getKey().FoodArray();
Arrays.asList(food).stream().forEach(f->{
System.out.print(f + " ");
});
System.out.println();
List<Product> list = m.getValue();
list.stream().forEach(e -> {
System.out.println("Name:" + e.getProductName() + " Produc Code:" + e.getProductCode());
});
System.out.println();
});
}
private static TreeMap<Food, List<Product>> dataSetLake() {
TreeMap<Food, List<Product>> data = new TreeMap<>();
List<Product> fruitA = new ArrayList<>();
fruitA.add(new Product("Apple", "123"));
fruitA.add(new Product("Banana", "456"));
List<Product> vegetableA = new ArrayList<>();
vegetableA.add(new Product("Potato", "999"));
vegetableA.add(new Product("Tomato", "987"));
List<Product> fruitB = new ArrayList<>();
fruitB.add(new Product("Apple", "123"));
fruitB.add(new Product("Banana", "456"));
List<Product> vegetableB = new ArrayList<>();
vegetableB.add(new Product("Potato", "999"));
vegetableB.add(new Product("Tomato", "987"));
Food foodA = new Food("Fruits", "Read To Eat");
Food foodB = new Food("Vegetables", "Need To Cook");
data.put(foodA, fruitB);
data.put(foodB, vegetableB);
data.put(foodA, fruitA);
data.put(foodB, vegetableA);
return data;
}
}
This is the service class.I am creating a XML file by reading value from database. Code is using three more pojo classes. Mt700, Header and Swift details. MT700 is main class for Header and swift details. Problem is I am able to store everything one time. Doesn't matter how many rows of data I have when the file get generated with one record it has only one header and one swift details. How can I make this work in loop? I think I have to use list but I am not sure how to use it to make it work.
public void generateEliteExtracts(int trdCustomerKy, Date lastRunDate, Date currentDate) throws TradeException {
FileOutputStream fout = null;
try {
MT700 mt700 = getMT700(trdCustomerKy,lastRunDate,currentDate);
if (null != mt700){
StringBuffer fileName = new StringBuffer(1024);
fileName.append(mConfiguration.getOutDirectory()).append(MT700_MSGTYPE)
.append(DOT).append(mConfiguration.getOutputFileExtn());
smLog.debug("Generated Extract for BankRef" + fileName.toString());
mTracer.log("Generated Extract for BankRef" + fileName.toString());
File xmlFile = new File(fileName.toString());
fout = new FileOutputStream(xmlFile);
fout.write(MT700_XMLHEADER.getBytes());
JAXBContext jaxbContext = JAXBContext.newInstance(MT700.class);
Marshaller marshaller = jaxbContext.createMarshaller();
marshaller.setProperty(Marshaller.JAXB_ENCODING, ENCODING_ASCII);
marshaller.setProperty(Marshaller.JAXB_FORMATTED_OUTPUT, Boolean.TRUE);
marshaller.setProperty(Marshaller.JAXB_FRAGMENT, Boolean.FALSE);
marshaller.setProperty("com.sun.xml.internal.bind.xmlDeclaration", Boolean.FALSE);
marshaller.marshal(mt700, fout);
IOUtils.closeQuietly(fout);
}
}catch(
Exception ex)
{
smLog.error("Caught unexpected error while creating extracts. ", ex);
throw new TradeException("Caught unexpected error while creating extracts.", ex);
} finally
{
IOUtils.closeQuietly(fout);
}
}
private MT700 getMT700(int trdCustomerKy, Date lastRunDate, Date currentDate) throws TradeException {
MT700 mt700 = new MT700();
AbInBevEliteExtractDAO dao = new AbInBevEliteExtractDAO(mConnection);
CompanyCodesHelper ccHelper = new CompanyCodesHelper(mConnection);
String cifCodes = ccHelper.getDescription(trdCustomerKy, "CIF Codes", "CIF Codes");
if (false == TradeUtil.isStringNull(cifCodes)) {
mTracer.log("Fetching records for CIFs: " + StringUtils.replace(cifCodes, PIPE, COMMA));
String[] codes = StringUtils.split(cifCodes, PIPE);
List<ExportAdvicesData> exportList = dao.getExportAdvices(trdCustomerKy, lastRunDate, currentDate, codes);
for (int i = 0; i < exportList.size(); i++) {
ExportAdvicesData exportData = exportList.get(i);
if ("XXLC".equalsIgnoreCase(exportData.getDocAcronym())) {
Header header = new Header();
header.setMessageType("N");
header.setVersionNo("1.0");
header.setRevisionNo("00");
header.setDocumentDate(DateUtil.formatDate(new Date(), DATE_FORMAT_YYYY_MM_DD_HHMMSS));
header.setBankId("BOA" + StringUtils.substring(exportData.getCustRef(), 0, 4));
header.setCustId("XOM");
SwiftDetails swiftTest = new SwiftDetails();
header.setDocumentType(MT700_MSGTYPE);
SwiftParserBankDocs parser = new SwiftParserBankDocs(exportData.getDocumentContent());
String bankRef = parser.getTagValue("21");
String custRef = parser.getTagValue("20");
if (TradeUtil.isStringNull(bankRef)) {
header.setCustRefNo("NONREF");
header.setBankRefNo(custRef);
} else {
header.setCustRefNo(custRef);
header.setBankRefNo(bankRef);
}
swiftTest.setTAG_27("1/1");
swiftTest.setTAG_20(custRef);
swiftTest.setTAG_23(EMPTY_STRING);
String issueDate = parser.getTagValue("31C");
swiftTest.setTAG_31C(getDateInYYMMDD(issueDate));
swiftTest.setTAG_40E("UCP LATEST VERSION");
String datePlaceOfExpiry = parser.getTagValue("31D");
swiftTest.setTAG_31D(getFormattedDatePlaceOfExpiry(datePlaceOfExpiry));
swiftTest.setTAG_50(parser.getTagValue("50"));
swiftTest.setTAG_59(parser.getTagValue("59"));
swiftTest.setTAG_32B(getCurrencyCdAmount(parser.getTagValue("32B")));
if (false == TradeUtil.isStringNull(exportData.getPositiveTolerance())) {
swiftTest.setTAG_39A(
exportData.getPositiveTolerance() + "/" + exportData.getPositiveTolerance());
} else {
swiftTest.setTAG_39A(EMPTY_STRING);
}
swiftTest.setTAG_39B(EMPTY_STRING);
swiftTest.setTAG_39C(EMPTY_STRING);
swiftTest.setTAG_41A(parser.getTagValue("41D"));
String tag42A = parser.getTagValue("42A");
swiftTest.setTAG_42A(tag42A);
if (TradeUtil.isStringNull(tag42A)) {
swiftTest.setTAG_42A(parser.getTagValue("42D"));
}
swiftTest.setTAG_42C(parser.getTagValue("42C"));
swiftTest.setTAG_42M(parser.getTagValue("42M"));
swiftTest.setTAG_42P(parser.getTagValue("42P"));
swiftTest.setTAG_43P(parser.getTagValue("43P"));
swiftTest.setTAG_43T(parser.getTagValue("43T"));
if (!(TradeUtil.isStringNull(parser.getTagValue("44A")))) {
swiftTest.setTAG_44A(parser.getTagValue("44A"));
}
if (!(TradeUtil.isStringNull(parser.getTagValue("44B")))) {
swiftTest.setTAG_44B(parser.getTagValue("44B"));
}
if (!(TradeUtil.isStringNull(parser.getTagValue("44E")))) {
swiftTest.setTAG_44E(parser.getTagValue("44E"));
}
if (!(TradeUtil.isStringNull(parser.getTagValue("44F")))) {
swiftTest.setTAG_44F(parser.getTagValue("44F"));
}
Date latestShipDate = exportData.getLatestShipDate();
if (null != latestShipDate) {
swiftTest.setTAG_44C(DateUtil.formatDate(latestShipDate, DATE_FORMAT_YYMMDD));
} else {
swiftTest.setTAG_44C(EMPTY_STRING);
}
swiftTest.setTAG_44D(parser.getTagValue("44D"));
swiftTest.setTAG_45A(parser.getTagValue("45") + BLANK_STRING + parser.getTagValue("45A")
+ BLANK_STRING + parser.getTagValue("45B"));
swiftTest.setTAG_46A(parser.getTagValue("46") + BLANK_STRING + parser.getTagValue("46A")
+ BLANK_STRING + parser.getTagValue("46B"));
swiftTest.setTAG_47A(parser.getTagValue("47") + BLANK_STRING + parser.getTagValue("47A")
+ BLANK_STRING + parser.getTagValue("47B"));
swiftTest.setTAG_71B(parser.getTagValue("71B"));
swiftTest.setTAG_48(parser.getTagValue("48"));
swiftTest.setTAG_49(parser.getTagValue("49"));
swiftTest.setTAG_50B(EMPTY_STRING);
swiftTest.setTAG_51A(EMPTY_STRING);
String issuingBank = parser.getAddress(SwiftParserBankDocs.ISSUING_BANK);
if (TradeUtil.isStringNull(issuingBank)) {
String errorMsg = "Issuing Bank address not found in bankdoc text, SWIFT content is possibly invalid, skipped processed record: "
+ exportData.getCustRef();
smLog.error(errorMsg);
mTracer.log("ERROR: " + errorMsg);
}
issuingBank = StringUtils.replace(issuingBank, CRLF, BLANK_STRING + CRLF);
swiftTest.setTAG_52A(issuingBank);
swiftTest.setTAG_53A(parser.getTagValue("53A"));
swiftTest.setTAG_78(parser.getTagValue("78"));
swiftTest.setTAG_57A(parser.getAddress("TO:"));
swiftTest.setTAG_72(parser.getTagValue("72"));
swiftTest.setTAG_40A(parser.getTagValue("40B"));
if (parser.is710Advice()) {
swiftTest.setTAG_20(parser.getTagValue("21"));
}
mt700.setSwift700(swiftTest);
mt700.setHeader(header);
} else if ("XAMD".equalsIgnoreCase(exportData.getDocAcronym())) {
Header header = new Header();
header.setMessageType("N");
header.setVersionNo("1.0");
header.setRevisionNo("00");
header.setDocumentDate(DateUtil.formatDate(new Date(), DATE_FORMAT_YYYY_MM_DD_HHMMSS));
header.setBankId("BOA" + StringUtils.substring(exportData.getCustRef(), 0, 4));
header.setCustId("XOM");
SwiftDetails swift = new SwiftDetails();
header.setDocumentType(MT707_MSGTYPE);
SwiftParserBankDocs parser = new SwiftParserBankDocs(exportData.getDocumentContent());
String custRef = parser.getTagValue("20");
String bankRef = parser.getTagValue("23");
if (TradeUtil.isStringNull(bankRef)) {
header.setBankRefNo("NONREF");
} else {
header.setBankRefNo(bankRef);
}
header.setCustRefNo(custRef);
swift.setTAG_20(custRef);
swift.setTAG_21(parser.getTagValue("21"));
swift.setTAG_23(EMPTY_STRING);
String issuingBank = parser.getAddress(SwiftParserBankDocs.ISSUING_BANK);
if (TradeUtil.isStringNull(issuingBank)) {
String errorMsg = "Issuing Bank address not found in bankdoc text, SWIFT content is possibly invalid, skipped processed record: "
+ exportData.getCustRef();
smLog.error(errorMsg);
mTracer.log("ERROR: " + errorMsg);
swift.setTAG_52A(EMPTY_STRING);
} else {
issuingBank = StringUtils.replace(issuingBank, CRLF, BLANK_STRING + CRLF);
swift.setTAG_52A(issuingBank);
}
swift.setTAG_31C(getDateInYYMMDD(parser.getTagValue("31C")));
swift.setTAG_30(getDateInYYMMDD(parser.getTagValue("30")));
swift.setTAG_26E(parser.getTagValue("26E"));
swift.setTAG_59(parser.getTagValue("59"));
swift.setTAG_31E(getDateInYYMMDD(parser.getTagValue("31E")));
swift.setTAG_79(parser.getTagValue("79"));
swift.setTAG_72(parser.getTagValue("72"));
swift.setTAG_32B(getCurrencyCdAmount(parser.getTagValue("32B")));
swift.setTAG_33B(getCurrencyCdAmount(parser.getTagValue("33B")));
swift.setTAG_34B(getCurrencyCdAmount(parser.getTagValue("34B")));
swift.setTAG_39A(parser.getTagValue("39A"));
swift.setTAG_39B(parser.getTagValue("39B"));
swift.setTAG_39C(parser.getTagValue("39C"));
swift.setTAG_44A(parser.getTagValue("44A"));
swift.setTAG_44B(parser.getTagValue("44B"));
swift.setTAG_44C(parser.getTagValue("44C"));
swift.setTAG_44D(parser.getTagValue("44D"));
swift.setTAG_44E(parser.getTagValue("44E"));
swift.setTAG_44F(parser.getTagValue("44F"));
mt700.setHeader(header);
mt700.setSwift700(swift);
}
}
}
return mt700;
}
This is MT700 POJO class. In this class I am calling header and swift details pojo classes.
#XmlRootElement(name = "MT700")
public class MT700 implements Serializable
{
/**
* serialVersionUID
*/
private static final long serialVersionUID = 1L;
private Header header;
private SwiftDetails swift700;
private String version = "1.0";
public Header getHeader()
{
return header;
}
#XmlElement(name = "Header")
public void setHeader(Header header)
{
this.header = header;
}
/**
* #return the swift700
*/
public SwiftDetails getSwift700()
{
return swift700;
}
#XmlElement(name = "Swift_Details_700")
public void setSwift700(SwiftDetails swift700)
{
this.swift700 = swift700;
}
public String getVersion()
{
return version;
}
#XmlAttribute(name = "Version")
public void setVersion(String version)
{
this.version = version;
}
}
This is Header class. I class similar to like this which has tags and that is swift details
#XmlRootElement(name = "Header")
#XmlType(propOrder = { "documentType", "messageType", "versionNo",
"revisionNo", "documentDate", "bankId", "custId", "custRefNo",
"bankRefNo" })
public class Header implements Serializable
{
private static final long serialVersionUID = 1L;
private String documentType;
private String messageType;
private String versionNo;
private String revisionNo;
private String documentDate;
private String bankId;
private String custId;
private String custRefNo;
private String bankRefNo;
I am not adding getter and setter for this class to make the post look simple
You are creating one MT700 instance and then in this loop, you are reassigning the header and swift fields each time through the loop:
MT700 mt700 = new MT700();
for (int i = 0; i < exportList.size(); i++) {
...
mt700.setHeader(header);
mt700.setSwift700(swift);
}
This means that the document you are outputting contains just the last header/swift returned from the database query.
You need to make one or more of these three into a list of some sort. Either your MT700 contains a list of headers and swifts, or more likely you want to have a list of MT700s each with one header and one swift.
In other words, you want to have a fourth type that will be the actual root of your XML document. That element will contain one MT700 element for each row found by the query. Each MT700 element will have a header element and a swift element.
So, more specifically, here is what you want to do:
#XmlRootElement
class MT700s {
#XmlElement(name = "MT700")
private List<MT700> mt700s = new ArrayList<>();
public List<MT700> getMT700s() { return mt700s; }
// Etc.
}
MT700s mt700s = new MT700s();
for (int i = 0; i < exportList.size(); i++) {
MT700 mt700 = new MT700();
...
mt700.setHeader(header);
mt700.setSwift700(swift);
mt700s.getMT700s().add(mt700);
}
How can we generate a number between a range using Json.
Like we have to generate a number between 0 to 50, how can we perform this in Java using a Json.
This is my Json Data
{
"rand": {
"type': "number",
"minimum": 0,
"exclusiveMinimum": false,
"maximum": 50,
"exclusiveMaximum": true
}
}
This is what I have tried in Java
public class JavaApplication1 {
public static void main(String[] args) {
try {
for (int i=0;i<5;i++)
{
FileInputStream fileInputStream = new FileInputStream("C://users/user/Desktop/V.xls");
HSSFWorkbook workbook = new HSSFWorkbook(fileInputStream);
HSSFSheet worksheet = workbook.getSheet("POI Worksheet");
HSSFRow row1 = worksheet.getRow(0);
String e1Val = cellE1.getStringCellValue();
HSSFCell cellF1 = row1.getCell((short) 5);
System.out.println("E1: " + e1Val);
JSONObject obj = new JSONObject();
obj.put("value", e1Val);
System.out.print(obj + "\n");
Map<String,Object> c_data = mapper.readValue(e1Val, Map.class);
System.out.println(a);
}
} catch (FileNotFoundException e) {
} catch (IOException e) {
}
}
}
Json Data is stored in excel sheet, from there I am reading it in Java program
Get a Json-reader like GSON.
Read in the JSON to an equivalent Object like
public class rand{
private String type;
private int minimum;
private boolean exclusiveMinimum;
private int maximum;
private boolean exclusiveMaximum;
//this standard-constructor is needed for the JsonReader
public rand(){
}
//Getter for all Values
}
and after reading in your JSON you can access your Data via your getter-methods
I think that Jackson may be of help here.
I suggest that you create a data model in Java that reflects the JSON. This can along the lines of:
// This is the root object. It contains the input data (RandomizerInput) and a
// generate-function that is used for generating new random ints.
public class RandomData {
private RandomizerInput input;
#JsonCreator
public RandomData(#JsonProperty("rand") final RandomizerInput input) {
this.input = input;
}
#JsonProperty("rand")
public RandomizerInput getInput() {
return input;
}
#JsonProperty("generated")
public int generateRandomNumber() {
int max = input.isExclusiveMaximum()
? input.getMaximum() - 1 : input.getMaximum();
int min = input.isExclusiveMinimum()
? input.getMinimum() + 1 : input.getMinimum();
return new Random().nextInt((max - min) + 1) + min;
}
}
// This is the input data (pretty much what is described in the question).
public class RandomizerInput {
private final boolean exclusiveMaximum;
private final boolean exclusiveMinimum;
private final int maximum;
private final int minimum;
private final String type;
#JsonCreator
public RandomizerInput(
#JsonProperty("type") final String type,
#JsonProperty("minimum") final int minimum,
#JsonProperty("exclusiveMinimum") final boolean exclusiveMinimum,
#JsonProperty("maximum") final int maximum,
#JsonProperty("exclusiveMaximum") final boolean exclusiveMaximum) {
this.type = type; // Not really used...
this.minimum = minimum;
this.exclusiveMinimum = exclusiveMinimum;
this.maximum = maximum;
this.exclusiveMaximum = exclusiveMaximum;
}
public int getMaximum() {
return maximum;
}
public int getMinimum() {
return minimum;
}
public String getType() {
return type;
}
public boolean isExclusiveMaximum() {
return exclusiveMaximum;
}
public boolean isExclusiveMinimum() {
return exclusiveMinimum;
}
}
To use these classes the ObjectMapper from Jackson can be used like this:
public static void main(String... args) throws IOException {
String json =
"{ " +
"\"rand\": { " +
"\"type\": \"number\", " +
"\"minimum\": 0, " +
"\"exclusiveMinimum\": false, " +
"\"maximum\": 50, " +
"\"exclusiveMaximum\": true " +
"} " +
"}";
// Create the mapper
ObjectMapper mapper = new ObjectMapper();
// Convert JSON to POJO
final RandomData randomData = mapper.readValue(json, RandomData.class);
// Either you can get the random this way...
final int random = randomData.generateRandomNumber();
// Or, you can serialize the whole thing as JSON....
String str = mapper.writeValueAsString(randomData);
// Output is:
// {"rand":{"type":"number","minimum":0,"exclusiveMinimum":false,"maximum":50,"exclusiveMaximum":true},"generated":21}
System.out.println(str);
}
The actual generation of a random number is based on this SO question.
In my play-framework-based web application users can download all the rows of different database tables in csv or json format. Tables are relatively large (100k+ rows) and I am trying to stream back the result using chunking in Play 2.2.
However the problem is although println statements shows that the rows get written to the Chunks.Out object, they do not show up in the client side! If I limit the rows getting sent back it will work, but it also has a big delay in the beginning which gets bigger if I try to send back all the rows and causes a time-out or the server runs out of memory.
I use Ebean ORM and the tables are indexed and querying from psql doesn't take much time. Does anyone have any idea what might be the problem?
I appreciate your help a lot!
Here is the code for one of the controllers:
#SecureSocial.UserAwareAction
public static Result showEpex() {
User user = getUser();
if(user == null || user.getRole() == null)
return ok(views.html.profile.render(user, Application.NOT_CONFIRMED_MSG));
DynamicForm form = DynamicForm.form().bindFromRequest();
final UserRequest req = UserRequest.getRequest(form);
if(req.getFormat().equalsIgnoreCase("html")) {
Page<EpexEntry> page = EpexEntry.page(req.getStart(), req.getFinish(), req.getPage());
return ok(views.html.epex.render(page, req));
}
// otherwise chunk result and send back
final ResultStreamer<EpexEntry> streamer = new ResultStreamer<EpexEntry>();
Chunks<String> chunks = new StringChunks() {
#Override
public void onReady(play.mvc.Results.Chunks.Out<String> out) {
Page<EpexEntry> page = EpexEntry.page(req.getStart(), req.getFinish(), 0);
ResultStreamer<EpexEntry> streamer = new ResultStreamer<EpexEntry>();
streamer.stream(out, page, req);
}
};
return ok(chunks).as("text/plain");
}
And the streamer:
public class ResultStreamer<T extends Entry> {
private static ALogger logger = Logger.of(ResultStreamer.class);
public void stream(Out<String> out, Page<T> page, UserRequest req) {
if(req.getFormat().equalsIgnoreCase("json")) {
JsonContext context = Ebean.createJsonContext();
out.write("[\n");
for(T e: page.getList())
out.write(context.toJsonString(e) + ", ");
while(page.hasNext()) {
page = page.next();
for(T e: page.getList())
out.write(context.toJsonString(e) + ", ");
}
out.write("]\n");
out.close();
} else if(req.getFormat().equalsIgnoreCase("csv")) {
for(T e: page.getList())
out.write(e.toCsv(CSV_SEPARATOR) + "\n");
while(page.hasNext()) {
page = page.next();
for(T e: page.getList())
out.write(e.toCsv(CSV_SEPARATOR) + "\n");
}
out.close();
}else {
out.write("Invalid format! Only CSV, JSON and HTML can be generated!");
out.close();
}
}
public static final String CSV_SEPARATOR = ";";
}
And the model:
#Entity
#Table(name="epex")
public class EpexEntry extends Model implements Entry {
#Id
#Column(columnDefinition = "pg-uuid")
private UUID id;
private DateTime start;
private DateTime finish;
private String contract;
private String market;
private Double low;
private Double high;
private Double last;
#Column(name="weight_avg")
private Double weightAverage;
private Double index;
private Double buyVol;
private Double sellVol;
private static final String START_COL = "start";
private static final String FINISH_COL = "finish";
private static final String CONTRACT_COL = "contract";
private static final String MARKET_COL = "market";
private static final String ORDER_BY = MARKET_COL + "," + CONTRACT_COL + "," + START_COL;
public static final int PAGE_SIZE = 100;
public static final String HOURLY_CONTRACT = "hourly";
public static final String MIN15_CONTRACT = "15min";
public static final String FRANCE_MARKET = "france";
public static final String GER_AUS_MARKET = "germany/austria";
public static final String SWISS_MARKET = "switzerland";
public static Finder<UUID, EpexEntry> find =
new Finder(UUID.class, EpexEntry.class);
public EpexEntry() {
}
public EpexEntry(UUID id, DateTime start, DateTime finish, String contract,
String market, Double low, Double high, Double last,
Double weightAverage, Double index, Double buyVol, Double sellVol) {
this.id = id;
this.start = start;
this.finish = finish;
this.contract = contract;
this.market = market;
this.low = low;
this.high = high;
this.last = last;
this.weightAverage = weightAverage;
this.index = index;
this.buyVol = buyVol;
this.sellVol = sellVol;
}
public static Page<EpexEntry> page(DateTime from, DateTime to, int page) {
if(from == null && to == null)
return find.order(ORDER_BY).findPagingList(PAGE_SIZE).getPage(page);
ExpressionList<EpexEntry> exp = find.where();
if(from != null)
exp = exp.ge(START_COL, from);
if(to != null)
exp = exp.le(FINISH_COL, to.plusHours(24));
return exp.order(ORDER_BY).findPagingList(PAGE_SIZE).getPage(page);
}
#Override
public String toCsv(String s) {
return id + s + start + s + finish + s + contract +
s + market + s + low + s + high + s +
last + s + weightAverage + s +
index + s + buyVol + s + sellVol;
}
1. Most of browsers wait for 1-5 kb of data before showing any results. You can check if Play Framework actually sends data with command curl http://localhost:9000.
2. You create streamer twice, remove first final ResultStreamer<EpexEntry> streamer = new ResultStreamer<EpexEntry>();
3. - You use Page class for retrieving large data set - this is incorrect. Actually you do one big initial request and then one request per iteration. This is SLOW. Use simple findIterate().
add this to EpexEntry (feel free to change it as you need)
public static QueryIterator<EpexEntry> all() {
return find.order(ORDER_BY).findIterate();
}
your new stream method implementation:
public void stream(Out<String> out, QueryIterator<T> iterator, UserRequest req) {
if(req.getFormat().equalsIgnoreCase("json")) {
JsonContext context = Ebean.createJsonContext();
out.write("[\n");
while (iterator.hasNext()) {
out.write(context.toJsonString(iterator.next()) + ", ");
}
iterator.close(); // its important to close iterator
out.write("]\n");
out.close();
} else // csv implementation here
And your onReady method:
QueryIterator<EpexEntry> iterator = EpexEntry.all();
ResultStreamer<EpexEntry> streamer = new ResultStreamer<EpexEntry>();
streamer.stream(new BuffOut(out, 10000), iterator, req); // notice buffering here
4. Another problem is - you call Out<String>.write() too often. Call of write() means that server needs to send new chunk of data to client immediately. Every call of Out<String>.write() have significant overhead.
Overhead appears because server needs to wrap response into chunked result - 6-7 bytes for each message Chunked response Format. Since you send small messages, overhead is significant.
Also, server needs to wrap your reply in TCP packet which size will be far less from optimal.
And, server needs to perform some internal action to send an chunk, this is also require some resources. As result, download bandwidth will be far from optimal.
Here is simple test: send 10000 lines of text TEST0 to TEST9999 in chunks. This takes 3 sec on my computer in average. But with buffering this takes 65 ms. Also, download sizes are 136 kb and 87.5 kb.
Example with buffering:
Controller
public class Application extends Controller {
public static Result showEpex() {
Chunks<String> chunks = new StringChunks() {
#Override
public void onReady(play.mvc.Results.Chunks.Out<String> out) {
new ResultStreamer().stream(out);
}
};
return ok(chunks).as("text/plain");
}
}
new BuffOut class. It's dumb, I know
public class BuffOut {
private StringBuilder sb;
private Out<String> dst;
public BuffOut(Out<String> dst, int bufSize) {
this.dst = dst;
this.sb = new StringBuilder(bufSize);
}
public void write(String data) {
if ((sb.length() + data.length()) > sb.capacity()) {
dst.write(sb.toString());
sb.setLength(0);
}
sb.append(data);
}
public void close() {
if (sb.length() > 0)
dst.write(sb.toString());
dst.close();
}
}
This implementation have 3 second download time and 136 kb size
public class ResultStreamer {
public void stream(Out<String> out) {
for (int i = 0; i < 10000; i++) {
out.write("TEST" + i + "\n");
}
out.close();
}
}
This implementation have 65 ms download time and 87.5 kb size
public class ResultStreamer {
public void stream(Out<String> out) {
BuffOut out2 = new BuffOut(out, 1000);
for (int i = 0; i < 10000; i++) {
out2.write("TEST" + i + "\n");
}
out2.close();
}
}