We Keep Coding

Writing a Jagged Array in HDF5 using the Java Native Library

I have tried numerous ways and followed some of the examples that are scattered around the web on how to write a jagged array (an array of arrays that may be of differing lengths) in HDF5.
Most of the examples are in C and rather low-level. Anyhow I can't seem to get it working and I just looked at the C-source code and it pretty much says that any variable-length datatypes that are not strings are not supported (if I understood correctly).
My miserable dysfunctional code (as is):
public void WIP_createVLenFloatDataSet( List<? extends Number> floats ) throws Exception
{
String group = "/test";
long groupId = createGroupIfNotExist( group );
MDataQualifier qualifier = new MDataQualifierImpl( group, "float", "0.0.0" );
long datasetId = openDataSet( qualifier );
long heapType = H5.H5Tcopy( MDataType.FLOAT_ARRAY.getHDFType() );
heapType = H5.H5Tvlen_create( heapType );
// heapType = H5.H5Tarray_create( heapType, 1, new long[]{1} );
if( !exists( datasetId ) )
{
long[] maxDims = new long[]{ HDF5Constants.H5S_UNLIMITED };
long dataspaceId = H5.H5Screate_simple( 1, new long[]{ 1 }, null );
// Create the dataset.
long datasetId1 = -1;
try
{
if( exists( m_fileId ) && exists( dataspaceId ) && exists( heapType ) )
{
long creationProperties = H5.H5Pcreate( HDF5Constants.H5P_DATASET_CREATE );
H5.H5Pset_chunk( creationProperties, /*ndims*/1, new long[]{ 1 } );
datasetId1 = H5.H5Dcreate( groupId, qualifier.getVersionedName(), heapType, dataspaceId, H5P_DEFAULT, creationProperties, H5P_DEFAULT );
// H5.H5Pclose( creationProperties );
}
}
catch( Exception e )
{
LOG.error( "Problems creating the dataset: " + e.getMessage(), e );
}
datasetId = datasetId1;
if( exists( datasetId ) )
{
// flushIfNecessary();
LOG.trace( "Wrote empty dataset {}", qualifier.getVersionedName() );
}
}
List<? extends Number> data = ( List<? extends Number> )floats;
// H5.H5Dwrite( datasetId, heapType, dataspaceId, memSpaceId, HDF5Constants.H5P_DEFAULT, Floats.toArray( data) );
ByteBuffer bb = ByteBuffer.allocate( data.size() * 4 );
floats.forEach( f -> bb.putFloat( f.floatValue() ) );
// H5.H5Dwrite( datasetId, heapType, H5S_ALL, H5S_ALL, H5P_DEFAULT, Floats.toArray( data ) );
H5.H5Dwrite( datasetId, heapType, H5S_ALL, H5S_ALL, H5P_DEFAULT, bb.array() );
}
Has anyone done this before and can at least confirm that it's not possible?
The most I can get out of HDF5 is the message "buf does not support variable length type".

Apparently the "glue code" of the JNI wrapper doesn't support this. If you want to use this feature you either have to implement your own JNI or wait for a newer version. The official JNI code is open source and can be found here.

Null Pointer Exception when replacing character " with no text then casting value to Float

I am writing a java program fetching JSON data through HTTP GET methods and it returns the following after I navigate the object tree:
{
"year":"2015",
"period":"M03",
"periodName":"March",
"value":"141178",
"footnotes":[{}]
}
Now I want to take value and caste it to a float, I tried to do this like such:
JSONParser parser = new JSONParser();
try
{
JSONObject
BLSemployment = ( JSONObject ) parser.parse( _RDATA );
BLSemployment = ( ( JSONObject ) BLSemployment.get( "Results" ) );
JSONArray
BLSemploymentseries = ( ( JSONArray ) BLSemployment.get( "series" ) );
BLSemployment = ( ( JSONObject ) BLSemploymentseries.get( 0 ) );
BLSemploymentseries = ( ( JSONArray ) BLSemployment.get( "data" ) );
for( int i = 0; i < 12; i++ )
{
BLSemployment = ( (JSONObject) BLSemploymentseries.get( i ) );
HistoricalNonFarmPayrollData[i] = Float.parseFloat( JSONValue.toJSONString( BLSemployment.get( "value" ) ).replace( "\"" , "" ) );
HistoricalNonFarmPayrollYear[i] = JSONValue.toJSONString( BLSemployment.get( "year" ) );
HistoricalNonFarmPayrollMonth[i] = JSONValue.toJSONString( BLSemployment.get( "periodName" ) );
}
}
catch ( ParseException pe )
{
System.out.println( pe );
}
However Now I get the error:
Exception in thread "main" java.lang.NullPointerException
at BLSFramework.getNonFarmPayrolls(playground.java:354)
at playground.main(playground.java:27)

RESOLVED: occasionally BLS website doesn't send data for periods of time and I was coding during that period of time.

Calculated group-by fields in MongoDB

For this example from the MongoDB documentation, how do I write the query using MongoTemplate?
db.sales.aggregate(
[
{
$group : {
_id : { month: { $month: "$date" }, day: { $dayOfMonth: "$date" }, year: { $year: "$date" } },
totalPrice: { $sum: { $multiply: [ "$price", "$quantity" ] } },
averageQuantity: { $avg: "$quantity" },
count: { $sum: 1 }
}
}
]
)
Or in general, how do I group by a calculated field?

You can actually do something like this with "project" first, but to me it's a little counter-intuitive to require a $project stage before hand:
Aggregation agg = newAggregation(
project("quantity")
.andExpression("dayOfMonth(date)").as("day")
.andExpression("month(date)").as("month")
.andExpression("year(date)").as("year")
.andExpression("price * quantity").as("totalAmount"),
group(fields().and("day").and("month").and("year"))
.avg("quantity").as("averavgeQuantity")
.sum("totalAmount").as("totalAmount")
.count().as("count")
);
Like I said, counter-intuitive as you should just be able to declare all of this under $group stage, but the helpers don't seem to work this way. The serialization comes out a bit funny ( wraps the date operator arguments with arrays ) but it does seem to work. But still, this is two pipeline stages rather than one.
What is the problem with this? Well by separating the stages the stages the "project" portion forces the processing of all of the documents in the pipeline in order to get the calculated fields, that means it passes through everything before moving on to the group stage.
The difference in processing time can be clearly seen by running the queries in both forms. With a separate project stage, on my hardware takes three times longer to execute than the query where all fields are calculated during the "group" operation.
So it seems the only present way to construct this properly is by building the pipeline object yourself:
ApplicationContext ctx =
new AnnotationConfigApplicationContext(SpringMongoConfig.class);
MongoOperations mongoOperation = (MongoOperations) ctx.getBean("mongoTemplate");
BasicDBList pipeline = new BasicDBList();
String[] multiplier = { "$price", "$quantity" };
pipeline.add(
new BasicDBObject("$group",
new BasicDBObject("_id",
new BasicDBObject("month", new BasicDBObject("$month", "$date"))
.append("day", new BasicDBObject("$dayOfMonth", "$date"))
.append("year", new BasicDBObject("$year", "$date"))
)
.append("totalPrice", new BasicDBObject(
"$sum", new BasicDBObject(
"$multiply", multiplier
)
))
.append("averageQuantity", new BasicDBObject("$avg", "$quantity"))
.append("count",new BasicDBObject("$sum",1))
)
);
BasicDBObject aggregation = new BasicDBObject("aggregate","collection")
.append("pipeline",pipeline);
System.out.println(aggregation);
CommandResult commandResult = mongoOperation.executeCommand(aggregation);
Or if all of that seems to terse to you, then you can always work with the JSON source and parse that. But of course, it has to be valid JSON:
String json = "[" +
"{ \"$group\": { "+
"\"_id\": { " +
"\"month\": { \"$month\": \"$date\" }, " +
"\"day\": { \"$dayOfMonth\":\"$date\" }, " +
"\"year\": { \"$year\": \"$date\" } " +
"}, " +
"\"totalPrice\": { \"$sum\": { \"$multiply\": [ \"$price\", \"$quantity\" ] } }, " +
"\"averageQuantity\": { \"$avg\": \"$quantity\" }, " +
"\"count\": { \"$sum\": 1 } " +
"}}" +
"]";
BasicDBList pipeline = (BasicDBList)com.mongodb.util.JSON.parse(json);

Another alternative is to use a custom aggregation operation class defined like so:
public class CustomAggregationOperation implements AggregationOperation {
private DBObject operation;
public CustomAggregationOperation (DBObject operation) {
this.operation = operation;
}
#Override
public DBObject toDBObject(AggregationOperationContext context) {
return context.getMappedObject(operation);
}
}
Then implement it in the pipeline like so:
Aggregation aggregation = newAggregation(
new CustomAggregationOperation(
new BasicDBObject(
"$group",
new BasicDBObject("_id",
new BasicDBObject("day", new BasicDBObject("$dayOfMonth", "$date" ))
.append("month", new BasicDBObject("$month", "$date"))
.append("year", new BasicDBObject("$year", "$date"))
)
.append("totalPrice", new BasicDBObject(
"$sum", new BasicDBObject(
"$multiply", Arrays.asList("$price","$quantity")
)
))
.append("averageQuantity", new BasicDBObject("$avg", "$quantity"))
.append("count",new BasicDBObject("$sum",1))
)
)
)
So it's basically just an interface that is consistent with that used by the existing pipeline helpers, but instead of using other helper methods it directly takes a DBObject definition to return in pipeline construction.

How to iterate DBObject values in Java

I have values that coming from MongoDB stored in a DBObject. and I needed to store that value in a set one by one. As being new to MongoDB I am not actually getting idea coherently how to proceed that.
String date = sdf.format(cal2.getTime());
List<String> dateList = new ArrayList<String>();
for (int i = 0; i < 6; i++) {
Date dateParsed = sdf.parse(date);
dateParsed.setDate(dateParsed.getDate() - i);
dateList.add(sdf.format(dateParsed));
}
Set<String> values2= new HashSet<String>();
for (String str : dateList) {
BasicDBObject find1 = new BasicDBObject("_ky", str);
DBObject values1= someDB.findOne(find1);
Iterator iter = values1.iterator(); /*giving error the method not found (becasue values1 is a dbObject)*/
while (iter.hasNext()) {
values2.add(//???//);
}
}
Any help on how can I iterate the DBObject- values1 and add those values in a set- values2 would be great.

You can call values1.keySet() and iterate over that and get() any values or use values1.toMap() and iterate that Map like you would any other.

The primary abstraction in the Mongo Java Driver is the DBObject which acts like a wrapper around Java's Map<String,Object>.
import java.util.Arrays;
import com.mongodb.BasicDBList;
import com.mongodb.BasicDBObject;
import com.mongodb.DBObject;
public class DBObjectKeySetDemo {
public static void main( String[] args ) {
DBObject dbo = new BasicDBObject( "firstName", "Robert" ).append( "lastName", "Kuhar" );
dbo.put( "age", 49 );
dbo.put( "hobbies", Arrays.asList( "Fly Fishing", "Board Games", "Roller Derby" ) );
DBObject browser = new BasicDBObject( "implementation", "Chrome" );
browser.put( "vendor", "Google" );
BasicDBList bookmarks = new BasicDBList();
bookmarks.add( new BasicDBObject( "name", "StackOverflow" ).append( "URL", "http://stackoverflow.com" ) );
bookmarks.add( new BasicDBObject( "name", "MMS" ).append( "URL", "https://mms.mongodb.com" ) );
browser.put( "bookmarks", bookmarks );
dbo.put( "browser", browser );
for ( String key : dbo.keySet() ) {
System.out.println( "key: " + key + " value: " + dbo.get( key ) );
}
System.out.println( "dbo: " + dbo );
}
}
The "gotcha" is that you can only work directly with "top level" element. For example, in the above example, through the Java API, you have no way to directly reference "browser.vendor". Through the Java API you would have to first get the "browser" sub-document, and then get the "vendor" field.
Clear? As mud? It helped me to just think of the abstraction as a Map<String,Object> where Object, in the case of a sub-document, may itself be a Map<String,Object>.

Groovy/Java: Ini4j insert multiple values to single parameter in different lines

I'm trying to add multiple-values option into my ini file from Groovy using ini4j with following codes (I tried some variants):
import org.ini4j.Wini
List valuesList = [ 'val1’, ‘val2’, ‘val3' ]
( new Wini( new File( "test.ini" ) ) ).with{
valuesList.each{
put( 'sectionNa'sectionName','optionName', it)
}
store()
}
import org.ini4j.Wini
List valuesList = [ 'val1’, ‘val2’, ‘val3' ]
( new Wini( new File( "test.ini" ) ) ).with{
Section sectionObject = get( ‘sectionName’ )
sectionObject .put( 'optionName', ‘val1’ )
sectionObject .put( 'optionName', ‘val2’ )
sectionObject .put( 'optionName', ‘val3’ )
}
store()
}
I got ini file like this one:
[sectionName]
optionName = val3
But I want to get:
[sectionName]
optionName = val1
optionName = val2
optionName = val3
Could you please advice me how to resolve my issue? Thanks In Advance!
Update 1
I still waiting more elegant solution. But I created direct ini file editing below. Please provide me any feedback about it:
List newLines = []
File currentFile = new File( "test.ini" )
List currentLines = currentFile.readLines()
int indexSectionStart = currentLines.indexOf( 'sectionName' )
(0..indexSectionStart).each{
newLines.add( currentLines[ it ] )
}
List valuesList = 'val1,val2,val3'.split( ',' )
valuesList.each{
newLines.add( "optionName = $it" )
}
( indexSectionStart + 1 .. currentLines.size() - 1 ).each{
newLines.add( currentLines[ it ] )
}
File newFile = new File( "new_test.ini" )
if ( newFile.exists() ) newFile.delete()
newLines.each {
newFile.append( it+'\n' )
}
And simply delete old file and rename new one. I implemented it because I didn't find any insertLine() like methods in standart File

Right, how's this:
import org.ini4j.*
List valuesList = [ 'val1', 'val2', 'val3' ]
new File( "/tmp/test.ini" ).with { file ->
new Wini().with { ini ->
// Configure to allow multiple options
ini.config = new Config().with { it.multiOption = true ; it }
// Load the ini file
ini.load( file )
// Get or create the section
( ini.get( 'sectionName' ) ?: ini.add( 'sectionName' ) ).with { section ->
valuesList.each {
// Then ADD the options
section.add( 'optionName', it )
}
}
// And write it back out
store( file )
}
}