We Keep Coding

Collect all VersionOne assets and all their attributes using Java

I'm working on a task that requires me to export all assets and all their attribute values into a CSV file. I know that there is an option to export into Excel, but that one has its problems and we decide to give a chance to an API.
The problem I faced is that while I can get all assets of a specific type with the code
IServices services = new Services(connector);
IAssetType requestType = services.getMeta().getAssetType("Request");
Query query = new Query(requestType);
it isn't clean how to return all asset's attributes. There is a getAttributes() for the Asset object
QueryResult result = services.retrieve(query);
for (Asset asset : result.getAssets()) {
Map<String, Attribute> attributes = asset.getAttributes();
System.out.println(attributes.toString());
}
but it doesn't seem to return an attribute unless it is explicitly added into a query eg.
…
Query query = new Query(requestType);
IAttributeDefinition nameAttribute = requestType.getAttributeDefinition("Name");
IAttributeDefinition numberAttribute = requestType.getAttributeDefinition("Number");
query.getSelection().add(nameAttribute);
query.getSelection().add(numberAttribute);
QueryResult result = services.retrieve(query);
…
which doesn't make sense to me, since I may not even know all possible object's attributes!
I feel like getAttributes() method may not be suitable for this purpose, but what else to use then? Any ideas on how I can collect the data I need?

You can use Meta API query to retrieve metadata for specific asset type:
<Server Base URI>/meta.v1/Request

In general VersionOne APIs don't return all available attributes at once as a default. When using the VersionOne Rest api, the most important subset of attributes are returned and no custom fields. The VersionOne sdks are a wrapper around this api so it stands to reason that api business rules are fulfilled in the sdk. You will have to know the names of all possible attributes of an asset and explicitly request them. This includes custom fields (Custom_AttributeName). This can be queried by doing a meta query YourVersionOneInstance/meta.v1/YourAssetName. You then will have to iterate through this xml tree and get the attribute names and wrap the proper query plumbing around each attribute.

Add basic value to Ontology individuals #Jena

I have an Ontology with some Classes and everything setup to run. What is a good way to fill it up with Individuals and Data?? In Short do a one-way Mapping from Database (as Input) to an Ontology.
public class Main {
static String SOURCE = "http://www.umingo.de/ontology/bento.owl";
static String NS = SOURCE+"#";
public static void main(String[] args) throws Exception {
OntModel model = ModelFactory.createOntologyModel( OntModelSpec.OWL_MEM );
// read the RDF/XML file
model.read(SOURCE);
OntologyPreLoader loader = new OntologyPreLoader();
model = loader.init(model);
model.write(System.out,"RDF/XML");
}
}
My Preloader has a Method init with the goal to copy data from a database into the ontology. Here is the Excerpt.
public OntModel init(OntModel model) throws SQLException{
Resource r = model.getResource( Main.NS + "Tag" );
Property tag_name = model.createProperty(Main.NS + "Tag_Name");
OntClass tag = r.as( OntClass.class );
// statements allow to issue SQL queries to the database
statement = connect.createStatement();
// resultSet gets the result of the SQL query
resultSet = statement
.executeQuery("select * from niuu.tags");
// resultSet is initialised before the first data set
while (resultSet.next()) {
// it is possible to get the columns via name
// also possible to get the columns via the column number
// which starts at 1
// e.g., resultSet.getSTring(2);
String id = resultSet.getString("id");
String name = resultSet.getString("name");
Individual tag_tmp = tag.createIndividual(Main.NS+"Tag_"+id);
tag_tmp.addProperty(tag_name,name);
System.out.println("id: " + id);
System.out.println("name: " + name);
}
return model;
}
Everything is working, but I feel really unsure about this way to preload ontologies. Also every Individual should get its own ID so that i can match it with the database at a later point.
Can i simply define a Property ID and add it to every Individual?
I thought about Adding ID to "Thing" as it is the most basic Type in OWL ontologies.

At first sight it seems ok. One tip is to try convert the Jena model into a RDF serialization and run it through Protégé to get a more clear picture on how your ontology mapping looks like.
You can definitely make your own property to describe the id of every individual.
Beneath is an example on how you can create a similar property in turtle format.(I did not add the prefixes for OWL and rdfs since they are some common)
You can add this in Jena aswell if needed. (or load this into your model in Jena.)
#prefix you: <your domain> .
you:dbIdentificator a owl:DatatypeProperty .
you:dbIdentificator rdfs:label "<Your database identifcator>"#en .
you:dbIdentificator rdfs:comment "<Some valuable information if needed>"#en .
you:dbIdentificator rdfs:isDefinedBy <your domain> .
you:dbIdentificator rdfs:domain owl:Thing .
You could also add owl:Thing to every resource, but that is not the best practice because it is a vague definition of a resource. I would look around for vocabularies that defines more what the resource is. Take a look at GoodRelations. It is a very good defined vocabulary that can describe information even though it is not for commercial use. Especially check out the classes there.
Hope that answered some of your question.

Programatically generating URIs is always somewhat unsettling. If you have Guava, use Preconditions to make some fail-fast assertions about what's coming out of the database (so that your code will let you know if it gets out of alignment with your schema). Use the JDK's URLEncoder to ensure that the id you get from the database is converted to a URI-friendly format (Note that if your data contains characters that cannot be printed in xml and have no percent encoding, you'll need to manually handle them).
For your property/column values, use explicitly create the literal. This makes it very clear whether you are using plain literals, language literals, or typed literals:
// If things can have multiple names in multiple languages, for example
tag_tmp.addProperty(tag_name,model.createTypedLiteral(name, "en"));
Note that you may not wish to define your schema so that it implies things about owl:Thing, because that would have implications outside of your domain. Instead, define a domain-specific notion like a :DatabaseResource. Set the domains of your properties to be that and it's subclasses rather than thing. This way the use of your property implies that the subject with within your domain, rather than simply an owl individual (which is implied by the domain of owl:DatatypeProperty anyway).
EDIT: It's absolutely acceptable to create a representation of the database's unique ID and place it into the RDF model. If you are using owl2, you can define an OWL-2 Key on that property for your :DatabaseResources and keep the same semantics that you had in the database.
EDIT: Noting a portion of your post on the Jena mailing list:
I have a huge MYSQL-Database for read only purpose and want to extract some Data into the Ontology.
I would highly recommend using the TDB Java API to construct a Dataset that backed by your disk. I've worked on very large database exports before, and it's quite possible that your data size won't be tractable otherwise. TDB's indexing requires a lot of disk space, but the memory-mapped IO makes it very difficult to kill due to OOM errors. Finally, once you have constructed the database on disk, you won't have to perform this expensive import operation again (or could at least optimize it).
If you find database creation times to be prohibitive, then you may with to utilize the bulk loader in creative ways. This answer has an example of using the bulk loader from java.

Unboundid not returning requested LDAP attributes. Why?

I have written a program that reads a webservice, retrieving user data, and then is supposed to push that data to ActiveDirectory, thus updating the user's title, address, phone numbers, etc.
The problem is that when I perform the search using the Unboundid Connection class the requested attributes are not returned. Below is the search code:
SearchResult result = connection.search( properties.getProperty("ldap.search.baseDN"),
SearchScope.SUB, "(cn=" + userId + ")",
"personalTitle", "department", "company", "manager", "telephoneNumber",
"streetAddress", "I", "st", "postalCode", "c", "pager", "mobile",
"fax", "cn");
The above code locates the desired user and the cn attribute is returned as expected, but the other attributes all fail to return. If I connect to AD using JXplorer using the same connection credentials, I'm able to see all the desired attributes exist, but are simply not being returned.
I have tried substituting SearchRequest.ALL_OPERATIONAL_ATTRIBUTES, SearchRequest.ALL_USER_ATTRIBUTES and SearchRequest.REQUEST_ATTRS_DEFAULT rather than listing the fields explicitly, but with no success.
I have also looked at the 'Schema' object returned from 'connection.getSchema()' and can see that personalTitle should exist:
connection.getSchema().getAttributeType("personalTitle")
The above code returns:
1.2.840.113556.1.2.615 NAME 'personalTitle' SYNTAX '1.3.6.1.4.1.1466.115.121.1.15' SINGLE-VALUE
So maybe this is a user permission issue? Has anyone experienced this and know how to resolve it?
Thanks,
Mike

LDAP search result entries only include attributes that actually have values, so the behavior you are seeing from the UnboundID LDAP SDK is appropriate and correct. Even if you explicitly request a particular attribute, that attribute will only be included in the entry if it has one or more values.
I think that you're confused by JXplorer because it's reading the schema to determine what attributes could possibly be included in the entry based on its object classes and is showing them to you so that you can set values for those attributes in the editor. But that doesn't mean that the entry returned by the server actually includes any information about those attributes.
To verify this, you can use the ldap-debugger tool provided with the LDAP SDK to see the actual LDAP communication that occurs. Just run a command like:
tools/ldap-debugger --hostname {directory-server-address} \
--port {directory-server-port} --listenPort {listen-port}
This will create a very simple LDAP proxy server that decodes all requests and responses that pass through it. To use it, simply point JXplorer at the specified listen-port. You will see that when JXplorer retrieves the entry, the entry returned by the server will only contain attributes that actually have values.
If you want to figure out what all the possible attributes are that you can include in a given entry, then use the LDAPConnection.getSchema method to retrieve the server schema, then Schema.getObjectClass for each of the object classes in the target entry, and finally use the ObjectClassDefinition.getRequiredAttributes and ObjectClassDefinition.getOptionalAttributes methods to see what attribute types must and may be used in entries with that object class.

How to get webLinks having specific attribute in HTTPUnit

We have the API for getting all the WebLinks known as response.getLinks() assuming you have fetched the WebResponse in response. See the problem is I want to fetch only those anchor tags which have specific attribute like all those anchor tags which have data-class attribute and value is true.
//Qualify
? // Does not qualify
It is possible to get all the links and traverse them and identify the anchors of interest. But I dont want to do this long procedure.
Does any short method exist to get weblinks having specific property in HTTPUnit?

Fetching LDAP Schema in Java

I'm trying to fetch the schema of an LDAP entity. I've been advised to use Novell's open sourced LDAP library, which seems to be quite good. I found an example called GetAttributeSchema, which sounds really good, but I think I've got a parameter set incorrectly or something.
The results I'm getting are:
ou ( 2.5.4.11 NAME ( 'ou' 'organizationalUnitName' ) DESC 'Standard LDAP attribute type' SUP 'name' SYNTAX 1.3.6.1.4.1.1466.115.121.1.15 X-ORIGIN 'RFC 2256' )
objectClass ( 2.5.4.0 NAME 'objectClass' DESC 'Standard LDAP attribute type' EQUALITY 'objectIdentifierMatch' SYNTAX 1.3.6.1.4.1.1466.115.121.1.15 X-ORIGIN 'RFC 2256 (XXX: syntax should be ...38)' )
The results I'm expecting are [something to the effect of]:
cn
affiliations
streetaddress
phone
fax
etc
Just wondering if it's obvious to an LDAP expert what I'm missing or what I should be asking for.
Thanks!
==============================
SOLUTION:
ServiceableLDAPConnection lc = takeConnection();
LDAPSchema schema = lc.fetchSchema(lc.getSchemaDN());
Enumeration<?> enumeration = schema.getAttributeNames();
I found this site particularly useful: http://www.go4expert.com/forums/showthread.php?t=4814
==============================

You are getting the full schema definitions. Usually in LDAP queries for cn=schema, you get the entire schema as one object you will need to parse.