I am new to memcache of GAE and I need a help in this.
Basically, I have a datastore which exceeded the Datastore Read Operations limit because of the fact that I didn't use memcache. My datastore has minimal writes but many reads and every time there's a write, it should be available for the read. Since, the site is up and I need a quick resolution for it so I need a design help in this. So the thing is, whenever there's a write in the datastore the new entry should get memcached. One more thing I would like to know that how the datastore can be replicated to the memcache. In parallel, I am working on it but since the site is up I am asking it here without any code in hand.
Thanks
UPDATE:
Java code looks like this:
MemcacheService memcache = MemcacheServiceFactory.getMemcacheService();
if(memcache.contains("LocationInfo"))
{
JSONArray js = new JSONArray((String)memcache.get("LocationInfo"));
result = new ArrayList<LocationInfo>();
for(int i = 0; i < js.length(); i++)
{
JSONObject jso = (JSONObject)js.get(i);
LocationInfo loc = new LocationInfo(jso);
result.add(loc);
}
}
else
{
q1= pm.newQuery(LocationInfo.class);
q1.setFilter(filter);
result = (List<LocationInfo>)q1.execute();
JSONArray js = new JSONArray();
for(LocationInfo loc : result)
{
js.put(loc.toJSON());
}
memcache.put("LocationInfo", js.toString());
}
from google.appengine.ext import db
from google.appengine.api import memcache
def top_arts(update = False):
key = 'top'
#Getting arts from memcache
arts = memcache.get(key)
#Check if key is defined in memcache
#or an update has been invoked
if update or not arts:
#Querying the Google Data store using GQL
arts = db.GqlQuery('SELECT * from Art ORDER BY created DESC LIMIT 10')
memcache.set(key, arts)
return arts
You can use the same function for reading from memcache and then writing data into
memcache
Eg:
for reading from memcache:-
arts = top_arts()
when writing into database:-
#write your entry in database
<some database code>
#update memcache with this new entry
top_arts(update=True)
Related
I am new to shapefile processing. Kindly guide me on how to achieve my below query.
I am using this shapefile tl_2018_us_aiannh.shp from census.gov : TIGER-LINE. I am to obtain the census block group entities like Block, Tract, County subdivision and County details from the shapefile based on the latitude and longitude provided by the user.
My requirement is to achieve this by shapefile alone and not through any API's.
Can someone help on which framework I can achieve this?
What I've tried/using so far:
I have used GeoTools to read the shapefile . Can I continue using the same? Will my requirement be achievable by this tool?
I have gone through a documentation from census.gov which states:
The Census Bureau assigns a code and these appear in fields such as
“TRACTCE”, where “CE” stands for Census. Finally, state-submitted
codes end in “ST”, such as “SLDLST”, and local education agency codes
end in “LEA”, as in “ELSDLEA”.
Which I tried in my code by:
File file = new File("D:\\tl_2018_us_aiannh.shp");
try {
Map<String, String> connect = new HashMap();
connect.put("url", file.toURI().toString());
DataStore dataStore = DataStoreFinder.getDataStore(connect);
String[] typeNames = dataStore.getTypeNames();
String typeName = typeNames[0];
System.out.println("Reading content " + typeName);
SimpleFeatureSource featureSource = dataStore
.getFeatureSource(typeName);
SimpleFeatureCollection collection = featureSource.getFeatures();
SimpleFeatureIterator iterator = collection.features();
try {
while (iterator.hasNext()) {
SimpleFeature feature = iterator.next();
GeometryAttribute sourceGeometry = feature
.getDefaultGeometryProperty();
String name = (String) (feature).getAttribute("TRACTCE");
Property property = feature.getProperty("TRACTCE");
System.out.println(property);
}
} finally {
iterator.close();
}
} catch (Throwable e) {
e.getMessage();
}
But I am receiving null as the value.
Any help would be much helpful.
I have found the solution to this. Hope this would be helpful to someone in need.
SimpleFeature is the type that has the attributes of shape files that you can check when you try to debug or print a line on runtime. You can use the SimpleFeature to get the property. The attributes can be achieved by:
try {
while (iterator.hasNext()) {
SimpleFeature feature = iterator.next();
Property intptlat = feature.getProperty("TRACTCE");
}
}
Make sure you are choosing the Block Groups as the layer type for download in Tiger-Line or which ever site is concerned, where you download the shape file.
Hy,
I’m new to milo (and OPC-UA) and try to implement an OPC-UA server with Historical Data Access. I reused the current milo server example and create a history node. On this node I can query (with the Prosys OPC UA Client) the empty history. I know that I have to implement the persistency of the history nodes by myself.
So far so good – but I could not found any information about to handle the history read request and how to return the response. More precisely how to add the HistoryData to an HistoryReadResult
#Override
public void historyRead(HistoryReadContext context, HistoryReadDetails readDetails, TimestampsToReturn timestamps,
List<HistoryReadValueId> readValueIds)
{
List<HistoryReadResult> results = Lists.newArrayListWithCapacity(readValueIds.size());
for (HistoryReadValueId readValueId : readValueIds){
//return 3 historical entries
DataValue v1 = new DataValue(new Variant(new Double(1)), StatusCode.GOOD, new DateTime(Date.from(Instant.now().minus(1, ChronoUnit.MINUTES))));
DataValue v2 = new DataValue(new Variant(new Double(2)), StatusCode.GOOD, new DateTime(Date.from(Instant.now().minus(2, ChronoUnit.MINUTES))));
DataValue v3 = new DataValue(new Variant(new Double(3)), StatusCode.GOOD, new DateTime(Date.from(Instant.now().minus(3, ChronoUnit.MINUTES))));
HistoryData data = new HistoryData(new DataValue[] {v1,v2,v3});
//???
HistoryReadResult result = new HistoryReadResult(StatusCode.GOOD, ByteString.NULL_VALUE, ??? );
results.add(result);
}
context.complete(results);
}
You're going to need access to the spec to successfully implement historical access services. Part 4 and Part 11.
The last parameter in the HistoryReadResult constructor is supposed to be a HistoryData structure. ExtensionObject is basically the container that structures are encoded and transferred in.
To create that ExtensionObject you would first create a HistoryData (or HistoryModifiedData, depends... see the spec) and then do something like ExtensionObject.encode(historyData) to get the object you need to finish building the HistoryReadResult.
Overrides historyRead is the correct way to do.
HistoryReadResult result = new HistoryReadResult(StatusCode.GOOD, ByteString.NULL_VALUE,ExtensionObject.encode(data) );
However method was not called by generic client such as UA-Expert before defining my variableNode with specific AccessLevel and Historizing mode like this :
Set<AccessLevel> acclevels = new LinkedHashSet<>();
acclevels.add(AccessLevel.CurrentRead);
acclevels.add(AccessLevel.CurrentWrite);
acclevels.add(AccessLevel.HistoryRead);
UaVariableNode node = new UaVariableNode.UaVariableNodeBuilder(server.getNodeMap())
.setNodeId(new NodeId(namespaceIndex, "HelloWorld/Test/" + name))
.setAccessLevel(ubyte(AccessLevel.getMask(acclevels)))
.setUserAccessLevel(ubyte(AccessLevel.getMask(acclevels)))
.setBrowseName(new QualifiedName(namespaceIndex, name))
.setDisplayName(LocalizedText.english(name))
.setDataType(typeId)
.setTypeDefinition(Identifiers.BaseDataVariableType)
.setHistorizing(true)
.build();
i am doing a task converting VB script written from Powerbuild to java,
i am struggled at converting the DataStore Object into java ,
i have something like this :
lds_appeal_application = Create DataStore
lds_appeal_application.DataObject = "ds_appeal_application_report"
lds_appeal_application.SetTransObject(SQLCA)
ll_row = lds_appeal_application.retrieve(as_ksdyh, adt_start_date, adt_end_date, as_exam_name, as_subject_code)
for ll_rc = 1 to ll_row
ldt_update_date = lds_appeal_application.GetItemDatetime(ll_rc, "sqsj")
ls_caseno = trim(lds_appeal_application.GetItemString(ll_rc, "caseno"))
ls_candidate_no = trim(lds_appeal_application.GetItemString(ll_rc, "zkzh"))
ls_subjectcode = trim(lds_appeal_application.GetItemString(ll_rc, "kmcode"))
ls_papercode = trim(lds_appeal_application.GetItemString(ll_rc, "papercode"))
ls_name = trim(lds_appeal_application.GetItemString(ll_rc, "mc"))
ll_ksh = lds_appeal_application.GetItemDecimal(ll_rc, "ks_h")
ll_kmh = lds_appeal_application.GetItemDecimal(ll_rc, "km_h")
simply speaking, a datasoure is created and a data table is point to it by sql query(ds_appeal_application_report). Finally using a for loop to retrieve information from the table.
in java way of doing, i use an entities manager to createnativequery and the query can result a list of object array. However, i just dont know how to retrieve the information like VB using the DataStore Object.
please give me some advice . Thanks
Using test cases I was able to see how ELKI can be used directly from Java but now I want to read my data from MongoDB and then use ELKI to cluster geographic (long, lat) data.
I can only cluster data from a CSV file using ELKI. Is it possible to connect de.lmu.ifi.dbs.elki.database.Database with MongoDB? I can see from the java debugger that there is a databaseconnection field in de.lmu.ifi.dbs.elki.database.Database.
I query MongoDB creating POJO for each row and now I want to cluster these objects using ELKI.
It is possible to read data from MongoDB and write it in a CSV file then use ELKI to read that CSV file but I would like to know if there is a simpler solution.
---------FINDINGS_1:
From ELKI - Use List<String> of objects to populate the Database I found that I need to implement de.lmu.ifi.dbs.elki.datasource.DatabaseConnection and specifically override the loadData() method which returns an instance of MultiObjectsBundle.
So I think I should wrap a list of POJO with MultiObjectsBundle. Now i'm looking at the MultiObjectsBundle and it looks like the data should be held in columns. Why columns datatype is List> shouldnt it be List? just a list of items you want to cluster?
I'm a little confused. How is ELKI going to know that it should look at the long and lat for POJO? Where do I tell ELKI to do this? Using de.lmu.ifi.dbs.elki.data.type.SimpleTypeInformation?
---------FINDINGS_2:
I have tried to use ArrayAdapterDatabaseConnection and I have tried implementing DatabaseConnection. Sorry I need thing in very simple terms for me to understand.
This is my code for clustering:
int minPts=3;
double eps=0.08;
double[][] data1 = {{-0.197574246, 51.49960695}, {-0.084605692, 51.52128377}, {-0.120973687, 51.53005939}, {-0.156876, 51.49313},
{-0.144228881, 51.51811784}, {-0.1680743, 51.53430039}, {-0.170134484,51.52834133}, { -0.096440751, 51.5073853},
{-0.092754157, 51.50597426}, {-0.122502346, 51.52395143}, {-0.136039674, 51.51991453}, {-0.123616824, 51.52994371},
{-0.127854211, 51.51772703}, {-0.125979294, 51.52635795}, {-0.109006325, 51.5216612}, {-0.12221963, 51.51477076}, {-0.131161087, 51.52505093} };
// ArrayAdapterDatabaseConnection dbcon = new ArrayAdapterDatabaseConnection(data1);
DatabaseConnection dbcon = new MyDBConnection();
ListParameterization params = new ListParameterization();
params.addParameter(de.lmu.ifi.dbs.elki.algorithm.clustering.DBSCAN.Parameterizer.MINPTS_ID, minPts);
params.addParameter(de.lmu.ifi.dbs.elki.algorithm.clustering.DBSCAN.Parameterizer.EPSILON_ID, eps);
params.addParameter(DBSCAN.DISTANCE_FUNCTION_ID, EuclideanDistanceFunction.class);
params.addParameter(AbstractDatabase.Parameterizer.DATABASE_CONNECTION_ID, dbcon);
params.addParameter(AbstractDatabase.Parameterizer.INDEX_ID,
RStarTreeFactory.class);
params.addParameter(RStarTreeFactory.Parameterizer.BULK_SPLIT_ID,
SortTileRecursiveBulkSplit.class);
params.addParameter(AbstractPageFileFactory.Parameterizer.PAGE_SIZE_ID, 1000);
Database db = ClassGenericsUtil.parameterizeOrAbort(StaticArrayDatabase.class, params);
db.initialize();
GeneralizedDBSCAN dbscan = ClassGenericsUtil.parameterizeOrAbort(GeneralizedDBSCAN.class, params);
Relation<DoubleVector> rel = db.getRelation(TypeUtil.DOUBLE_VECTOR_FIELD);
Relation<ExternalID> relID = db.getRelation(TypeUtil.EXTERNALID);
DBIDRange ids = (DBIDRange) rel.getDBIDs();
Clustering<Model> result = dbscan.run(db);
int i =0;
for(Cluster<Model> clu : result.getAllClusters()) {
System.out.println("#" + i + ": " + clu.getNameAutomatic());
System.out.println("Size: " + clu.size());
System.out.print("Objects: ");
for(DBIDIter it = clu.getIDs().iter(); it.valid(); it.advance()) {
DoubleVector v = rel.get(it);
ExternalID exID = relID.get(it);
System.out.print("DoubleVec: ["+v+"]");
System.out.print("ExID: ["+exID+"]");
final int offset = ids.getOffset(it);
System.out.print(" " + offset);
}
System.out.println();
++i;
}
The ArrayAdapterDatabaseConnection produces two clusters, I just had to play around with the value of epsilon, when I set epsilon=0.008 dbscan started creating clusters. When i set epsilon=0.04 all the items were in 1 cluster.
I have also tried to implement DatabaseConnection:
#Override
public MultipleObjectsBundle loadData() {
MultipleObjectsBundle bundle = new MultipleObjectsBundle();
List<Station> stations = getStations();
List<DoubleVector> vecs = new ArrayList<DoubleVector>();
List<ExternalID> ids = new ArrayList<ExternalID>();
for (Station s : stations){
String strID = Integer.toString(s.getId());
ExternalID i = new ExternalID(strID);
ids.add(i);
double[] st = {s.getLongitude(), s.getLatitude()};
DoubleVector dv = new DoubleVector(st);
vecs.add(dv);
}
SimpleTypeInformation<DoubleVector> type = new VectorFieldTypeInformation<>(DoubleVector.FACTORY, 2, 2, DoubleVector.FACTORY.getDefaultSerializer());
bundle.appendColumn(type, vecs);
bundle.appendColumn(TypeUtil.EXTERNALID, ids);
return bundle;
}
These long/lat are associated with an ID and I need to link them back to this ID to the values. Is the only way to go that using the ID offset (in the code above)? I have tried to add ExternalID column but I don't know how to retrieve the ExternalID for a particular NumberVector?
Also after seeing Using ELKI's Distance Function I tried to use Elki's longLatDistance but it doesn't work and I could not find any examples to implement it.
The interface for data sources is called DatabaseConnection.
JavaDoc of DatabaseConnection
You can implement a MongoDB-based interface to get the data.
It is not complicated interface, it has a single method.
I am creating a Java web app to manage meetings between a set of students and teachers. All of them already use Outlook to manage their email and personal calendar.
I would like to know if it's even possible to build the schedule feature of my web app using Exchange, Office365 or Sharepoint Team Calendar via REST service in order to check the availability and create a meeting for a student and one of the teachers available:
SharePoint 2013 REST service
So far, the most promising mechanism I have found is Microsoft Sharepoint Server's calendar, which collaborative features makes possible to create a meeting and check availability for a list of users. The downside is that it does not support one to one meetings but for the entire team (as far I have found).
My second option would be to require everyone in the group (students and teachers of the department) to make public their personal calendar so the web app be able to check the availability of both student and teacher and send a meeting request. The obvious problem is the privacy/security concern derived from this approach.
My last option (and by far the less favourite because it feels like re-inventing the wheel) is to build a proprietary calendar within the web app and send iCal requests to each person. The obvious problem with this approach is synchronisation between the two separated calendars.
In addition, this feature must be a pretty common need so there should be tons of blogs explaining how to take advantage of Exchange/Sharepoint/Office365 to implement it (other platforms are not considered since my employer's infrastructure is based on Microsoft). However, whether it is so obvious that nobody talks about it or I have not searched in the right place. Any advice to point me in the right direction?
Exchange natively shows user calendar availability exposed in EWS (Exchange Web Services), your network administrator must configure Exchange server enabling EWS.
But guess what... Office 365 (as I know) have EWS services enabled, due exchange is part of office 365 offer.
As EWS are normal Web services you should create a "service stub" or proxy in whatever you use in java to create services references mapping wsdl files.
Exchanged EWS is my preferred solution.
Hope this helps.
This is the reference page, this link show how to use the service references from C# to make the right API calls.
http://msdn.microsoft.com/en-us/library/exchange/aa494212(v=exchg.140).aspx
static void GetUserAvailability(ExchangeServiceBinding esb)
{
// Identify the time to compare free/busy information.
Duration duration = new Duration();
duration.StartTime = DateTime.Now;
duration.EndTime = DateTime.Now.AddHours(4);
// Identify the options for comparing free/busy information.
FreeBusyViewOptionsType fbViewOptions = new FreeBusyViewOptionsType();
fbViewOptions.TimeWindow = duration;
fbViewOptions.RequestedView = FreeBusyViewType.MergedOnly;
fbViewOptions.RequestedViewSpecified = true;
fbViewOptions.MergedFreeBusyIntervalInMinutes = 35;
fbViewOptions.MergedFreeBusyIntervalInMinutesSpecified = true;
MailboxData[] mailboxes = new MailboxData[1];
mailboxes[0] = new MailboxData();
// Identify the user mailbox to review for free/busy data.
EmailAddress emailAddress = new EmailAddress();
emailAddress.Address = "tplate#contoso.com";
emailAddress.Name = String.Empty;
mailboxes[0].Email = emailAddress;
mailboxes[0].ExcludeConflicts = false;
// Make the request.
GetUserAvailabilityRequestType request = new GetUserAvailabilityRequestType();
// Set the time zone of the request.
request.TimeZone = new SerializableTimeZone();
request.TimeZone.Bias = 480;
request.TimeZone.StandardTime = new SerializableTimeZoneTime();
request.TimeZone.StandardTime.Bias = 0;
request.TimeZone.StandardTime.DayOfWeek = DayOfWeekType.Sunday.ToString();
request.TimeZone.StandardTime.DayOrder = 1;
request.TimeZone.StandardTime.Month = 11;
request.TimeZone.StandardTime.Time = "02:00:00";
request.TimeZone.DaylightTime = new SerializableTimeZoneTime();
request.TimeZone.DaylightTime.Bias = -60;
request.TimeZone.DaylightTime.DayOfWeek = DayOfWeekType.Sunday.ToString();
request.TimeZone.DaylightTime.DayOrder = 2;
request.TimeZone.DaylightTime.Month = 3;
request.TimeZone.DaylightTime.Time = "02:00:00";
// Add the mailboxes to the request.
request.MailboxDataArray = mailboxes;
// Add the view options to the request.
request.FreeBusyViewOptions = fbViewOptions;
try
{
// Send the request and get the response.
GetUserAvailabilityResponseType response = esb.GetUserAvailability(request);
// Access free/busy information.
if (response.FreeBusyResponseArray.Length < 1)
{
throw new Exception("No free/busy response data available.");
}
else
{
foreach (FreeBusyResponseType fbrt in response.FreeBusyResponseArray)
{
if (fbrt.ResponseMessage.ResponseClass == ResponseClassType.Error)
{
Console.WriteLine(string.Format("Error: {0}", fbrt.ResponseMessage.MessageText));
}
else
{
// Show the free/busy stream.
FreeBusyView fbv = fbrt.FreeBusyView;
Console.WriteLine(string.Format("Merged free/busy data: {0}", fbv.MergedFreeBusy));
}
}
}
}
catch (Exception e)
{
// Perform error processing.
Console.WriteLine(e.Message);
Console.ReadLine();
}
}