I used the hadoop apache to create a counting Bloom Filter. However I get a NullPointerException when I am trying to add keys in it. I tried to change the class structure in many ways but still I get the same result.
Here is the code I did:
package package_name;
import java.io.IOException;
import java.util.ArrayList;
import java.util.Iterator;
import org.apache.hadoop.util.bloom.*;
public class CBF {
public static CountingBloomFilter CBF = new CountingBloomFilter();
public static void countingFilter (ArrayList<byte[]> CBF_Keys) throws IOException{
CBF_Keys= Keys.keyStringArray;
Iterator<byte[]> iter = CBF_Keys.iterator();
while (iter.hasNext()) {
byte[] temp = iter.next();
Key hadoop_key = new Key(temp, 2.0);
CBF.add(hadoop_key);
}
}
}
problem is CBF = new CountingBloomFilter(). We should use CountingBloomFilter(int vectorSize, int nbHash, int hashType) instead here, otherwise the HashFunction will not be constructed in parent class Filter.
Related
I have 100 CSV files with following content
name,price
book,12.4
bread,54.23
Each file show content in sorted by price order
I need to find 10 most expensive products through all these files.This is my code:
import org.apache.commons.io.FileUtils;
import org.junit.Assert;
import org.junit.Test;
import java.io.File;
import java.io.IOException;
import java.io.UncheckedIOException;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.util.List;
import static java.util.stream.Collectors.toList;
public final class FindBiggest extends Assert {
static class Data {
public Data(String str) {
final String[] split = str.split(",");
this.name = split[0];
this.price = Float.parseFloat(split[1]);
}
private final String name;
private final float price;
}
#Test
public void test() throws Exception {
final List<File> files = Files.walk(Paths.get("/tmp/"))
.filter(Files::isRegularFile)
.filter(path -> path.toString().endsWith(".csv"))
.map(Path::toFile)
.collect(toList());
final List<Data> collect =
files.stream()
.map(FindBiggest::content)
.map(Data::new)
.sorted((o1, o2) -> Float.compare(o1.price, o2.price))
.limit(10)
.collect(toList());
System.out.println(collect);
}
private static String content(final File file) {
try {
return FileUtils.readFileToString(file, StandardCharsets.UTF_8);
} catch (IOException e) {
throw new UncheckedIOException(e);
}
}
}
In case when I have a lot of csv files program throw UOM(Out of memory) how to implement program to sort content in all files non loading all data to memory ?
You'll need some sorted set limited by certain amount of items. Probably some 3rd party collections libraries provide it, otherwise you can make it somehow like this: Limited SortedSet. Important thing is that the method add of such sorted set must return false if collection is full and newly added element falls beyond the limit, and true otherwise.
Now, make a loop over the csv files. Inside the loop's body, read records from csv file and add them to the set until add returns false (it'd mean that collection is full and no new records from the current csv won't be larger then the current ones - time to proceed to next file).
When the loop is done, resulting set will be the answer.
I am trying to write json from stream to another stream with base64 field like that:
import com.fasterxml.jackson.core.JsonFactory;
import com.fasterxml.jackson.core.JsonGenerator;
import java.io.File;
import java.io.FileInputStream;
public class Exmaple {
public static void main (String[] args) throws Exception {
final JsonGenerator generator = new JsonFactory().createGenerator(System.out);
generator.writeStartObject();
generator.writeFieldName("data");
generator.writeBinary(new FileInputStream(new File("D:\\somePath\\pdf.pdf")), -1);
generator.writeEndObject();
}
}
Instead of pdf.pdf you can use any file.
But on outut I got:
{"data":"JVBERi0xLjMKJcfsj6IKMzAgMCBvYmoKPD
No " at the end despite generator.writeEndObject() is called
No = or == at the end of base64
No } at the end of object
Why does it happen? How to fix this???
It is needed to add generator.close() call which leads to flush stream.
This is the code I have written but the new built-in does not seem to work. I get the error:
Exception in thread "main" com.hp.hpl.jena.reasoner.rulesys.impl.LPRuleSyntaxException: Syntax error in backward rule: matematica Unknown builtin operation mysum
Can anyone tell me where the error is? Here is my code:
package JenaRules;
import java.io.BufferedReader;
import java.io.ByteArrayInputStream;
import java.io.IOException;
import java.io.InputStreamReader;
import java.util.Arrays;
import java.util.List;
import org.semanticweb.owlapi.model.OWLOntologyCreationException;
import org.semanticweb.owlapi.model.OWLOntologyStorageException;
import com.hp.hpl.jena.graph.Node;
import com.hp.hpl.jena.query.Query;
import com.hp.hpl.jena.query.QueryExecution;
import com.hp.hpl.jena.query.QueryExecutionFactory;
import com.hp.hpl.jena.query.QueryFactory;
import com.hp.hpl.jena.query.ResultSet;
import com.hp.hpl.jena.query.ResultSetFormatter;
import com.hp.hpl.jena.rdf.model.InfModel;
import com.hp.hpl.jena.rdf.model.Model;
import com.hp.hpl.jena.rdf.model.ModelFactory;
import com.hp.hpl.jena.rdf.model.Resource;
import com.hp.hpl.jena.reasoner.Reasoner;
import com.hp.hpl.jena.reasoner.rulesys.*;
import com.hp.hpl.jena.reasoner.rulesys.builtins.BaseBuiltin;
import com.hp.hpl.jena.util.FileManager;
import com.hp.hpl.jena.vocabulary.RDFS;
import com.hp.hpl.jena.vocabulary.ReasonerVocabulary;
public class RulesOntology_MT {
public static void main(String[] args) throws OWLOntologyStorageException,
OWLOntologyCreationException, IOException {
BuiltinRegistry.theRegistry.register(new BaseBuiltin() {
#Override
public String getName() {
return "mysum";
}
#Override
public int getArgLength() {
return 2;
}
#Override
public boolean bodyCall(Node[] args, int length, RuleContext context) {
checkArgs(length, context);
BindingEnvironment env = context.getEnv();
Node n1 = getArg(0, args, context);
Node n2 = getArg(1, args, context);
if (n1.isLiteral() && n2.isLiteral()) {
Object v1 = n1.getLiteralValue();
Object v2 = n2.getLiteralValue();
Node sum = null;
if (v1 instanceof Number && v2 instanceof Number) {
Number nv1 = (Number)v1;
Number nv2 = (Number)v2;
int sumInt = nv1.intValue()+nv2.intValue();
sum = Util.makeIntNode(sumInt);
return env.bind(args[2], sum);
}
}
return false;
}
});
// NON SERVE
// final String exampleRuleString2 =
// "[mat1: equal(?s ?p )\n\t-> print(?s ?p ?o),\n\t (?s ?p ?o)\n]"+
// "";
final String exampleRuleString =
"[matematica:"+
"(?p http://www.semanticweb.org/prova_rules_M#totale_crediti ?x)"+
" -> " +
"(?p rdf:type http://www.semanticweb.org/prova_rules_M#:Persona)"+
"(?e rdf:type http://www.semanticweb.org/prova_rules_M#:Esame)"+
"(?p http://www.semanticweb.org/prova_rules_M#:haSostenutoEsameDi ?e)"+
"(?e http://www.semanticweb.org/prova_rules_M/persona#crediti_esame ?cr)"+
"mysum(?cr,2)"+
"]";
System.out.println(exampleRuleString);
/* I tend to use a fairly verbose syntax for parsing out my rules when I construct them
* from a string. You can read them from whatever other sources.
*/
final List<Rule> rules;
try( final BufferedReader src = new BufferedReader(new InputStreamReader(new ByteArrayInputStream(exampleRuleString.getBytes()))) ) {
rules = Rule.parseRules(Rule.rulesParserFromReader(src));
}
/* Construct a reasoner and associate the rules with it */
// create an empty non-inferencing model
GenericRuleReasoner reasoner = (GenericRuleReasoner) GenericRuleReasonerFactory.theInstance().create(null);
reasoner.setRules(rules);
/* Create & Prepare the InfModel. If you don't call prepare, then
* rule firings and inference may be deferred until you query the
* model rather than happening at insertion. This can make you think
* that your Builtin is not working, when it is.
*/
InfModel infModel = ModelFactory.createInfModel(reasoner, ModelFactory.createDefaultModel());
infModel.prepare();
infModel.createResource(RDFS.Class);
//write down the result in RDFXML form
infModel.write(System.out);
}
}
Using the code that you provided, and Apache Jena 2.11.1, I cannot replicate the exception you are getting. Do note that when you call BuiltinRegistry.theRegistry.register(...), you are telling the reasoner that the builtin exists.
Solution
The exception that you are getting is likely because, in your actual code, you are not calling BuiltinRegistry.theRegistry.register(...) prior to calling Rule.parseRules(Rule.rulesParserFromReader(src));, so as far as the rule parser is concerned, you are using a Builtin which doesn't exist. To fix it, merely call register before parsing your rules. The toy example provided does not have this problem.
Using the example provided
I also noted that the provided code example did not include anything that would actually stimulate the rule to fire, so, in lieu of infModel.createResource(RDFS.Class);, I added the following lines:
final Resource s = infModel.createResource();
final Property p = infModel.createProperty("http://www.semanticweb.org/prova_rules_M#totale_crediti");
final Resource o = infModel.createResource();
infModel.add(s,p,o);
This stimulated the rule to fire, and led to the following exception trace:
com.hp.hpl.jena.reasoner.rulesys.BuiltinException: Error in clause of rule (matematica) mysum: builtin mysum not usable in rule heads
at com.hp.hpl.jena.reasoner.rulesys.builtins.BaseBuiltin.headAction(BaseBuiltin.java:86)
at com.hp.hpl.jena.reasoner.rulesys.impl.RETEConflictSet.execute(RETEConflictSet.java:184)
at com.hp.hpl.jena.reasoner.rulesys.impl.RETEConflictSet.add(RETEConflictSet.java:81)
at com.hp.hpl.jena.reasoner.rulesys.impl.RETEEngine.requestRuleFiring(RETEEngine.java:249)
at com.hp.hpl.jena.reasoner.rulesys.impl.RETETerminal.fire(RETETerminal.java:80)
at com.hp.hpl.jena.reasoner.rulesys.impl.RETEClauseFilter.fire(RETEClauseFilter.java:227)
at com.hp.hpl.jena.reasoner.rulesys.impl.RETEEngine.inject(RETEEngine.java:469)
at com.hp.hpl.jena.reasoner.rulesys.impl.RETEEngine.runAll(RETEEngine.java:451)
at com.hp.hpl.jena.reasoner.rulesys.impl.RETEEngine.add(RETEEngine.java:174)
at com.hp.hpl.jena.reasoner.rulesys.FBRuleInfGraph.performAdd(FBRuleInfGraph.java:654)
at com.hp.hpl.jena.graph.impl.GraphBase.add(GraphBase.java:202)
at com.hp.hpl.jena.rdf.model.impl.ModelCom.add(ModelCom.java:1138)
at SO.test(SO.java:108)
As a note: my test class is SO.java and line 108 is where we call infModel.add(s,p,o).
The exception that I get is different than the exception you encountered, but it is worth explaining. The implementation that you provided implements Builtin#bodyCall(...), but not Builtin#headAction(...). We can see the exception is thrown from BaseBuiltin#headAction(...). This default behavior assumes that you didn't implement the method because your Builtin doesn't support it. In the toy problem, this is correct behavior because the example implementation cannot be used in rule heads.
probably a very simple issue related to importing the correct libraries but I cannot seem to find anything to help me.
Basically getting this error when trying to run my code:
"Cannot find a class or type named "DataEntry"
This is my Code here:
//Variables
UnfoldingMap map;
List<Marker>countryMarkers;
HashMap<String, DataEntry> dataEntriesMap;
//Core methods...
void setup() {
size(800, 600);
smooth();
map = new UnfoldingMap(this);
MapUtils.createDefaultEventDispatcher(this, map);
//Read in GeoJSON File - Countries
List<Feature> countries = GeoJSONReader.loadData(this, "countries.geo.json");
countryMarkers = MapUtils.createSimpleMarkers(countries);
map.addMarkers(countryMarkers);//Add the countries to the map
//External Data source - CSV file
}
void draw() {
map.draw();
}
//Other methods required...
This is a list of all my Imports
import de.fhpotsdam.unfolding.mapdisplay.*;
import de.fhpotsdam.unfolding.utils.*;
import de.fhpotsdam.unfolding.marker.*;
import de.fhpotsdam.unfolding.tiles.*;
import de.fhpotsdam.unfolding.interactions.*;
import de.fhpotsdam.unfolding.ui.*;
import de.fhpotsdam.unfolding.*;
import de.fhpotsdam.unfolding.core.*;
import de.fhpotsdam.unfolding.data.*;
import de.fhpotsdam.unfolding.geo.*;
import de.fhpotsdam.unfolding.texture.*;
import de.fhpotsdam.unfolding.events.*;
import de.fhpotsdam.utils.*;
import de.fhpotsdam.unfolding.providers.*;
import processing.opengl.*;
import java.util.List;
import java.util.HashMap;
I am experimenting with GeoSpatial data using Processing and the Unfolding Map Library.
I'd appreciate any help guys.
I am not entirely familiar with the packages you are using but based on examples I've seen, it seems as though you are missing an inner class.
Try updating the code as indicated below noting the DateEntry inner class at the very bottom:
//Variables
UnfoldingMap map;
List<Marker>countryMarkers;
HashMap<String, DataEntry> dataEntriesMap;
//Core methods...
void setup() {
size(800, 600);
smooth();
map = new UnfoldingMap(this);
MapUtils.createDefaultEventDispatcher(this, map);
//Read in GeoJSON File - Countries
List<Feature> countries = GeoJSONReader.loadData(this, "countries.geo.json");
countryMarkers = MapUtils.createSimpleMarkers(countries);
map.addMarkers(countryMarkers);//Add the countries to the map
//External Data source - CSV file
}
void draw() {
map.draw();
}
public class DataEntry {
String countryName;
String id;
Integer year;
Float value;
}
I want to verify that a collection contains at least one non-null element. I have tried is(not(empty())), however this passes in the test below.
import org.junit.Test;
import java.util.ArrayList;
import java.util.Collection;
import static org.hamcrest.CoreMatchers.is;
import static org.hamcrest.MatcherAssert.assertThat;
import static org.hamcrest.Matchers.empty;
import static org.hamcrest.Matchers.not;
public class SandBoxTest {
#Test
public void shouldTestThis() {
Collection<Integer> collection = new ArrayList<Integer>();
collection.add(null);
assertThat(collection, is(not(empty())));
}
}
Is there an elegant/simple way to do this?
Things That Don't Work
#Test
public void should(){
Collection<String> collection = new ArrayList();
collection.add("gfas");
collection.add("asda");
assertThat(collection, contains(notNullValue()));
}
java.lang.AssertionError:
Expected: iterable containing [not null]
but: Not matched: "asda"
at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
import static org.hamcrest.MatcherAssert.assertThat;
import static org.hamcrest.Matchers.*;
...
assertThat(collection, hasItem(notNullValue(Integer.class)));
Unfortunately, there is a bug in Java 1.6 that means you might have to split it onto 2 lines as described here if you are using 1.6:
Matcher<Iterable<? super String>> matcher = hasItem(notNullValue(Integer.class));
assertThat(collection, matcher);
EDIT Here is the FEST Assert example you asked for:
import static org.fest.assertions.api.Assertions.assertThat;
...
assertThat(collection).doesNotContainNull();
FEST requires only a single static import so you get full IDE auto completion.
I just ran into the same problem and solved it as follows.
The basic idea is that if the collection has only null elements, converted to a set it will contain just one element and it will be null. If not so, then the collection contains at least one non-null element.
I wrote a matcher, and tried it with this test:
import org.hamcrest.Description;
import org.hamcrest.Factory;
import org.hamcrest.Matcher;
import org.hamcrest.TypeSafeMatcher;
import org.junit.Test;
import java.util.ArrayList;
import java.util.Collection;
import java.util.HashSet;
import java.util.Set;
import static org.hamcrest.CoreMatchers.is;
import static org.hamcrest.CoreMatchers.not;
import static org.junit.Assert.assertThat;
import static personal.CollectionOfNullsMatcher.collectionOfNulls;
public class SimpleTest {
#Test
public void should_check_collection_for_non_null_values() {
Collection<String> testedCollection = new ArrayList<String>();
testedCollection.add(null);
assertThat(testedCollection, is(collectionOfNulls()));
testedCollection.add("any");
assertThat(testedCollection, is(not(collectionOfNulls())));
}
}
class CollectionOfNullsMatcher extends TypeSafeMatcher<Collection> {
#Override
protected boolean matchesSafely(final Collection collection) {
Set<Object> set = new HashSet<Object>(collection);
return (set.size() == 1) && (set.toArray()[0] == null);
}
#Override
public void describeTo(final Description description) {
description.appendText("collection of nulls");
}
#Factory
public static <T> Matcher<Collection> collectionOfNulls() {
return new CollectionOfNullsMatcher();
}
}
Of course, in a real project, the matcher should be placed together with its brothers :)
Hope it helps.
You can try the following:
public void shouldTestThis() {
Collection<Integer> collection = new ArrayList<Integer>();
collection.add(null);
collection.removeAll(Collections.singleton(null)); // remove all "null" elements from collection
assertThat(collection, is(not(empty())));
}
As ajb notices, if you want to left your array unmodified, you should use an iterator and check each element until the end of the collection or a non null one.
You're on the right track, chaining Matcher instances. You just need the hasItem matcher (like I've suggested here) instead of contains.
Creates a matcher for Iterables that only matches when a single pass
over the examined Iterable yields at least one item that is matched by
the specified itemMatcher. Whilst matching, the traversal of the
examined Iterable will stop as soon as a matching item is found.
For example,
Collection<Integer> collection = new ArrayList<Integer>();
collection.add(null);
assertThat(collection, hasItem(is(not(nullValue()))));
will fail with
java.lang.AssertionError:
Expected: a collection containing is not null
but: was null
at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:8)
at com.example.ExampleTest.should(ExampleTest.java:21)
at [...]
whereas
Collection<Integer> collection = new ArrayList<Integer>();
collection.add(null);
collection.add("hey");
collection.add(null);
assertThat(collection, hasItem(is(not(nullValue()))));
// or shortened
assertThat(collection, hasItem(notNullValue()));
will succeed.