We Keep Coding

Get all matching partition which matches regex in hive

I have one hive table and it is partitions on multiple columns.
I need to fetch partition list with partial partition name.
Ex: table foo_table is partition on
| PARTITIONED BY ( |
| `dt` string COMMENT 'Custom Partition.', |
| `h` string COMMENT 'Custom Partition.', |
| `b` string COMMENT 'Custom Partition.', |
| `sv` string COMMENT 'Custom Partition', |
| `p` string COMMENT 'Custom Partition', |
| `dc` string COMMENT 'Custom Partition')
Now i need to fetch all the partition let say where dt=somevalue
Below code works if I give value of all the partition columns.
List<String> list = ...
list.add("dt=2021-02-01/h=19/b=30/sv=1/p=03/dc=aa")
List<Partition> partitions = HiveMetaStoreClient.getPartitionsByNames(database, tableName, list)
But if i want to fetch partition by only giveing dt=2021-02-01/h=19this doesn't work.
List<String> list = ...
list.add("dt=2021-02-01/h=19")
//OR
list.add("dt=2021-02-01/h=19/")
//OR
list.add("dt=2021-02-01/h=19/*")
//OR
list.add("dt=2021-02-01/h=19/b=*/sv=*/p=*/dc=*")
List<Partition> partitions = HiveMetaStoreClient.getPartitionsByNames(database, tableName, list)
How to achieve this?

You can use listPartitions method available in HiveMetaStoreClient.
public List<Partition> listPartitions(String db_name,
String tbl_name,
List<String> part_vals,
short max_parts)
throws NoSuchObjectException,
MetaException,
org.apache.thrift.TException
you can query for partitions in sequence of your partitions added in hive table.
Example: Table ProductByTypeHeight partitioned [type String, Height int]
Drawback: you can query for one type of partition at a time.
List<String> pvals1 = new ArrayList<>();
pvals1.add("fruit");
List<String> pvals2 = new ArrayList<>();
pvals2.add("vegetable");
pvals2.add("450");
List<List<String>> vals = new ArrayList<>();
vals.add(pvals1);
vals.add(pvals2);
for (List<String> pval:vals) {
List<Partition> pNameList = hiveMetaStoreConnector.listPartitions("productdb", "productbytypeheight", pval);
pNameList.forEach(partition -> {
List<String> partitionValues = partition.getValues();
for (int i = 0; i < partitionValues.size(); i++) {
System.out.print(partitionValues.get(i) + " ");
}
System.out.println();
});
}
Output:
fruit 600
fruit 450
fruit 400
vegetable 450

How to iterate an ArrayList to get the contents

ArrayList testdata_1 = new ArrayList<>();
for (int i = 0; i < testdata_1.size(); i++) {
System.out.println(testdata_1.get(i));
}
My output is
[]
[[Username,Password], [user_1, Test#100]]
I want to delete the first blank value ,iterate the arraylist, identify the key as username,fetch the value "user_1" and assign to a local string variable "username"
Datatable
| Username | user_1 |
| Password | Test#100 |
List> data = new ArrayList<>();
if(step.getRows()!=null)
step.getRows().forEach(row -> data.add(row.getCells()));

Resolved. Used data encapsulation to resolve this

How to extract "infobox company" data from wiki dumps

I have downloaded a big wiki dump XML file from https://dumps.wikimedia.org/enwiki/20170520/
I want to extract the metadata company name and parent company from this wikidumps. All the company data are located in the XML template like below:
{{Infobox company
| name =
| logo =
| type =
| industry =
| fate =
| predecessor = <!-- or: | predecessors = -->
| successor = <!-- or: | successors = -->
| founded = <!-- if known: {{Start date and age|YYYY|MM|DD}} in [[city]], [[state]], [[country]] -->
| founder = <!-- or: | founders = -->
| defunct = <!-- {{End date|YYYY|MM|DD}} -->
| hq_location_city =
| hq_location_country =
| area_served = <!-- or: | areas_served = -->
| key_people =
| products =
| owner = <!-- or: | owners = -->
| num_employees =
| num_employees_year = <!-- Year of num_employees data (if known) -->
| parent =
| website = <!-- {{URL|example.com}} -->
}}
I did some research and found about MediaWiki Parser.
Reference: https://github.com/dkpro/dkpro-jwpl/blob/master/de.tudarmstadt.ukp.wikipedia.parser/src/main/java/de/tudarmstadt/ukp/wikipedia/parser/tutorial/T1_SimpleParserDemo.java
https://dkpro.github.io/dkpro-jwpl/JWPLParser/
I tried to use this parser. But it requires the file to be converted in string. My wiki dump XML file is 60 GB in size. I can't convert this big file in string and keep in memory. Also, there is no description for the Mediawiki parser on how to find specific element like Infobox company, go inside it and extract name and other fields. Below is the sample code for Mediawiki parser:
public static void main(String[] args) throws IOException {
File file = new File("C:/Users/njaiswal/Downloads/accenture_data_from_wikidumps.xml");
String str = FileUtils.readFileToString(file);
// get a ParsedPage object
MediaWikiParserFactory pf = new MediaWikiParserFactory();
MediaWikiParser parser = pf.createParser();
ParsedPage pp = parser.parse(str);
// get the sections
for (Section section : pp.getSections()) {
System.out.println("section : " + section.getTitle());
System.out.println(" nr of paragraphs : " + section.nrOfParagraphs());
System.out.println(" nr of tables : " + section.nrOfTables());
System.out.println(" nr of nested lists : " + section.nrOfNestedLists());
System.out.println(" nr of definition lists: " + section.nrOfDefinitionLists());
for (Link link : section.getLinks(Link.type.INTERNAL)) {
System.out.println(" " + link.getTarget());
}
}
}
Is there any other parser that can solve my problem? Or can I use the same MediaWiki Parser to get to "Inbox company" and extract fields? Any help is appreciated. Thanks
Update: I tried to use wikiXMLj parser that Khalil suggested. I am able to get all the "Infobox" data, but I want to limit this to "Infobox company" data. Below is my code and output:
import edu.jhu.nlp.wikipedia.*;
public class Test {
public static void main(String[] args) throws Exception{
WikiXMLParser parser = WikiXMLParserFactory.getSAXParser("C:/Users/njaiswal/Downloads/enwiki-20170520-pages-articles-multistream.xml/enwiki-20170520-pages-articles-multistream.xml");
parser.setPageCallback(new PageCallbackHandler() {
public void process(WikiPage page) {
try {
InfoBox infobox=page.getInfoBox();
System.out.println(infobox.dumpRaw());
} catch (WikiTextParserException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
//do something with info box
}
});
parser.parse();
}
}
O/P:
{{Infobox Monarch
| name = Attila
| title = [[List of Hunnic rulers|Ruler]] of the [[Hunnic Empire]]
| place of burial =
}}
{{Infobox sea
| name = Aegean Sea
| image = Aegean Sea map.png
| caption = Map of the Aegean Sea
| pushpin_map = World
| pushpin_map_alt = World
| pushpin_label_position = right
}}
{{Infobox company
| name = Audi AG
| logo = Audi-Logo 2016.svg
| logo_size = 235
| image = Audi Ingolstadt.jpg
| image_size = 265
}}

I used before wikixmlj very simple dumb parser. this shall parse it perfectly:
// dumpPath should be like C:\your/Path/articles.xml.bz2"
WikiXMLParser wxsp = WikiXMLParserFactory.getSAXParser(dumpPath);
wxsp.setPageCallback(new PageCallbackHandler() {
#Override
public void process(WikiPage page) {
//System.out.println("info box:" + page.getInfoBox());
String regex = "\\{{Infobox company(.|\\n)+";
Pattern pattern = Pattern.compile(regex);
Matcher matcher = pattern.matcher(page.getInfoBox());
while (matcher.find()) {
System.out.println(matcher.group(0));}
}
});
wxsp.parse(); }
demo of the regex

QuerySolution, keep the "< >"

I'm currently stuck on my project on creating a Fuseki Triple Store Browser. I need to visualize all the data from a TripleStore and make the app browsable. The only problem is that the QuerySolution leaves out the "< >" that are in the triplestore.
If I use the ResultSetFormatter.asText(ResultSet) it returns this:
-------------------------------------------------------------------------------------------------------------------------------------
| subject | predicate | object |
=====================================================================================================================================
| <urn:animals:data> | <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> | <http://www.w3.org/1999/02/22-rdf-syntax-ns#Seq> |
| <urn:animals:data> | <http://www.w3.org/1999/02/22-rdf-syntax-ns#_1> | <urn:animals:lion> |
| <urn:animals:data> | <http://www.w3.org/1999/02/22-rdf-syntax-ns#_2> | <urn:animals:tarantula> |
| <urn:animals:data> | <http://www.w3.org/1999/02/22-rdf-syntax-ns#_3> | <urn:animals:hippopotamus> |
-------------------------------------------------------------------------------------------------------------------------------------
Notice that the some of the data contains the smaller/greater than signs "<" and ">". As soon as i try to parse the data from the ResultSet, it removes those sign, so that the data looks like this:
-------------------------------------------------------------------------------------------------------------------------------
| subject | predicate | object |
===============================================================================================================================
| urn:animals:data | http://www.w3.org/1999/02/22-rdf-syntax-ns#type | http://www.w3.org/1999/02/22-rdf-syntax-ns#Seq |
| urn:animals:data | http://www.w3.org/1999/02/22-rdf-syntax-ns#_1 | urn:animals:lion |
| urn:animals:data | http://www.w3.org/1999/02/22-rdf-syntax-ns#_2 | urn:animals:tarantula |
| urn:animals:data | http://www.w3.org/1999/02/22-rdf-syntax-ns#_3 | urn:animals:hippopotamus |
As you can see, the data doesn't contain the "<" and ">" signs.
This is how I parse the data from the ResultSet:
while (rs.hasNext()) {
// Moves onto the next result
QuerySolution sol = rs.next();
// Return the value of the named variable in this binding.
// A return of null indicates that the variable is not present in
// this solution
RDFNode object = sol.get("object");
RDFNode predicate = sol.get("predicate");
RDFNode subject = sol.get("subject");
// Fill the table with the data
DefaultTableModel modelTable = (DefaultTableModel) this.getModel();
modelTable.addRow(new Object[] { subject, predicate, object });
}
It's quite hard to explain this problem, but is there a way to keep the "< >" signs after parsing the data?

The '<>' are used by the formatter to indicate that the value is a URI rather than a string: so "http://example.com/" is a literal text value, whereas <http://example.com/> is a URI.
You can do the same yourself:
RDFNode node; // subject, predicate, or object
if (node.isURIResource()) {
return "<" + node.asResource().getURI() + ">";
} else {
...
}
But it's much easier to use FmtUtils:
String nodeAsString = FmtUtils.stringForRDFNode(subject); // or predicate, or object
What you need to do is get that code invoked when the table cell is rendered: currently the table is using Object::toString().
In outline, the steps needed are:
modelTable.setDefaultRenderer(RDFNode.class, new MyRDFNodeRenderer());
Then see http://docs.oracle.com/javase/tutorial/uiswing/components/table.html#renderer about how to create a simple renderer. Note that value will be an RDFNode:
static class MyRDFNodeRenderer extends DefaultTableCellRenderer {
public MyRDFNodeRenderer() { super(); }
public void setValue(Object value) {
setText((value == null) ? "" : FmtUtils.stringForRDFNode((RDFNode) value));
}
}

How to print object list to file with formatting in table format

I have to print list of objects to a text file with table format. For example, if I have list of Person(has getName,getAge and getAddress methods)objects, the text file should look like as below.
Name Age Address
Abc 20 some address1
Def 30 some address2
I can do this by manually writing some code, where I have to take care of spaces and formatting issues.
I am just curious whether are they APIs or tools to do this formatting work?

import java.util.*;
public class Test {
public static void main(String[] args) {
List<Person> list = new ArrayList<Person>();
list.add(new Person("alpha", "astreet", 12));
list.add(new Person("bravo", "bstreet", 23));
list.add(new Person("charlie", "cstreet", 34));
list.add(new Person("delta", "dstreet", 45));
System.out.println(String.format("%-10s%-10s%-10s", "Name", "Age", "Adress"));
for (Person p : list)
System.out.println(String.format("%-10s%-10s%-10d", p.name, p.addr, p.age));
}
}
class Person {
String name;
String addr;
int age;
public Person(String name, String addr, int age) {
this.name = name;
this.addr = addr;
this.age = age;
}
}
Output:
Name Age Adress
alpha astreet 12
bravo bstreet 23
charlie cstreet 34
delta dstreet 45

Use printf with padded fields to achive column alignments.
PrintWriter.printf to be specific

Library for printing Java objects as Markdown / CSV / HTML table using reflection: https://github.com/mjfryc/mjaron-etudes-java
dependencies {
implementation 'io.github.mjfryc:mjaron-etudes-java:0.2.1'
}
import pl.mjaron.etudes;
class Sample {
void sample() {
Cat[] cats = {cat0, cat1};
Table.render(cats, Cat.class)
.markdown() // or .csv() or .html()
.to(System.out) // Or to StrigBuilder | OutputStream | File.
// Optionally specify Left /Right / Center alignment.
.withAlign(VerticalAlign.Left)
.run(); // Or .runToString() to return it to String.
}
}
Sample Markdown output:
| name | legsCount | lazy | topSpeed |
|------|-----------|-------|----------|
| John | 4 | true | 35.24 |
| Bob | 5 | false | 75.0 |