We Keep Coding

How can I acquire JanusGraphManagement over a remote connection?

I have a docker container running the gremlin-server.
It was started via:
./bin/gremlin-server.sh conf/gremlin-server/gremlin-server.yaml
From within a docker container, running this image:
https://hub.docker.com/r/janusgraph/janusgraph
The server is up and is listening at port 8182
$ docker ps
6019adda6081 janusgraph/janusgraph "docker-entrypoint.s…" 2 days ago Up 26 hours 0.0.0.0:8182->8182/tcp
I am interested in using a schema and indexes.
Janus offers this here: https://docs.janusgraph.org/basics/schema/
The following Is the configuration I use to attempt to connect to the gremlin-server:
AbstractConfiguration config = new BaseConfiguration();
config.setListDelimiter('/');
// contents of conf/remote-graph.properties
config.setProperty("gremlin.remote.driver.sourceName", "g");
config.setProperty("gremlin.remote.remoteConnectionClass", "org.apache.tinkerpop.gremlin.driver.remote.DriverRemoteConnection");
// contents of conf/remote-objects.yaml:
config.setProperty("clusterConfiguration.hosts", databaseUrl);
config.setProperty("clusterConfiguration.port", 8182);
config.setProperty("clusterConfiguration.serializer.className", "org.apache.tinkerpop.gremlin.driver.ser.GryoMessageSerializerV3d0/");
config.setProperty("storage.backend", "cql");
config.setProperty("clusterConfiguration.serializer.config.ioRegistries", "org.janusgraph.graphdb.tinkerpop.JanusGraphIoRegistry");
When I call
GraphTraversalSource g = traversal().withRemote(config);
I get a traversal source and everything seems fine. However, to use the management stuff that Janus provides, I seem to need a JanusGraphManagement object. I cannot get the generic Graph object above and cast it to a JanusGraph. The docs suggest using a JanusGraphFactory: https://docs.janusgraph.org/basics/configuration/#janusgraphfactory
So I call
JanusGraph janusGraph = JanusGraphFactory.open(config);
I get the following stack trace:
Exception in thread "main" java.lang.IllegalArgumentException: Could not find implementation class: org.janusgraph.diskstorage.cql.CQLStoreManager
at org.janusgraph.util.system.ConfigurationUtil.instantiate(ConfigurationUtil.java:60)
at org.janusgraph.diskstorage.Backend.getImplementationClass(Backend.java:440)
at org.janusgraph.diskstorage.Backend.getStorageManager(Backend.java:411)
at org.janusgraph.graphdb.configuration.builder.GraphDatabaseConfigurationBuilder.build(GraphDatabaseConfigurationBuilder.java:50)
at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:161)
at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:132)
at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:112)
at com.activitystream.database.GraphMigration.migrateDatabase(GraphMigration.java:69)
at com.activitystream.runners.persistence.DataStores.migrateDatabase(DataStores.java:27)
at com.activitystream.runners.persistence.EntityPersistenceRunner.main(EntityPersistenceRunner.java:23)
Caused by: java.lang.ClassNotFoundException: org.janusgraph.diskstorage.cql.CQLStoreManager
at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581)
at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
at java.base/java.lang.Class.forName0(Native Method)
at java.base/java.lang.Class.forName(Class.java:315)
at org.janusgraph.util.system.ConfigurationUtil.instantiate(ConfigurationUtil.java:56)
... 9 more
Is it possible to modify the schema over a remote connection?
If it is not possible, how can one modify the schema?
Any insight would be appreciated.

You basically have two choices - either:
Interact with your JanusGraphManagement object by way of scripts sent to Gremlin Server (typically by way of a session but I guess you could package an entire "management script" together and submit it as one request) or
Bypass Gremlin Server and instantiation your JanusGraphManagement object locally as directed in the JanusGraph documentation.
There is no way to have return a JanusGraphManagement to your client as it is not a serializable object that can be sent back from the server.

MapReduce 2 job runs locally instead of being submitted to yarn

So i tried to configure my job to be submitted to yarn, but instead it runs locally:
config.set("yarn.resourcemanager.address", "ADDRESS:8032");
config.set("mapreduce.framework.name", "yarn");
config.set("fs.default.name", "hdfs://ADDRESS:8020");
If i set mapred.job.tracker it fails with:
Exception in thread "main" org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.RpcServerException): Unknown rpc kind in rpc headerRPC_WRITABLE
because its not MR1.
So why is the app not submitted to yarn?

Solved it by doing this:
config.set("yarn.resourcemanager.address", "ADDRESS:8032");
config.set("yarn.resourcemanager.scheduler.address", "ADDRESS:8030");
config.set("yarn.resourcemanager.resource-tracker.address", "ADDRESS:8031");
config.set("yarn.resourcemanager.admin.address", "ADDRESS:8033");
instead of:
config.set("yarn.resourcemanager.address", "ADDRESS:8032");

Hibernate connecting to DB2 at application startup on WAS Liberty on CICS

We're running a simple webapp on WebSphere Liberty, that uses Hibernate as persistence provider (included as a library in the WAR file).
When application is starting up Hibernate is initialized and it will open a connection to DB2 and issue some SQL statements. However, this fails when running on CICS and using JDBC Type 2 Driver DataSource. The following messages are logged (some extra line breaks for readability):
WARN org.hibernate.engine.jdbc.env.internal.JdbcEnvironmentInitiator -
HHH000342: Could not obtain connection to query metadata : [jcc][50053][12310][4.19.56]
T2zOS exception: [jcc][T2zos]T2zosCicsApi.checkApiStatus:
Thread is not CICS-DB2 compatible: CICS_REGION_BUT_API_DISALLOWED ERRORCODE=-4228, SQLSTATE=null
...
ERROR org.hibernate.hql.spi.id.IdTableHelper - Unable obtain JDBC Connection
com.ibm.db2.jcc.am.SqlException: [jcc][50053][12310][4.19.56] T2zOS exception: [jcc][T2zos]T2zosCicsApi.checkApiStatus:
Thread is not CICS-DB2 compatible: CICS_REGION_BUT_API_DISALLOWED ERRORCODE=-4228, SQLSTATE=null
at com.ibm.db2.jcc.am.kd.a(Unknown Source) ~[db2jcc4.jar:?]
...
at com.ibm.db2.jcc.t2zos.T2zosConnection.a(Unknown Source) ~[db2jcc4.jar:?]
...
at com.ibm.db2.jcc.DB2SimpleDataSource.getConnection(Unknown Source) ~[db2jcc4.jar:?]
at com.ibm.cics.wlp.jdbc.internal.CICSDataSource.getConnection(CICSDataSource.java:176) ~[?:?]
at org.hibernate.engine.jdbc.connections.internal.DatasourceConnectionProviderImpl.getConnection(DatasourceConnectionProviderImpl.java:122) ~[our-app.war:5.1.0.Final]
at org.hibernate.internal.SessionFactoryImpl$3.obtainConnection(SessionFactoryImpl.java:643) ~[our-app.war:5.1.0.Final]
at org.hibernate.hql.spi.id.IdTableHelper.executeIdTableCreationStatements(IdTableHelper.java:67) [our-app.war:5.1.0.Final]
at org.hibernate.hql.spi.id.global.GlobalTemporaryTableBulkIdStrategy.finishPreparation(GlobalTemporaryTableBulkIdStrategy.java:125) [our-app.war:5.1.0.Final]
at org.hibernate.hql.spi.id.global.GlobalTemporaryTableBulkIdStrategy.finishPreparation(GlobalTemporaryTableBulkIdStrategy.java:42) [our-app.war:5.1.0.Final]
at org.hibernate.hql.spi.id.AbstractMultiTableBulkIdStrategyImpl.prepare(AbstractMultiTableBulkIdStrategyImpl.java:88) [our-app.war:5.1.0.Final]
at org.hibernate.internal.SessionFactoryImpl.<init>(SessionFactoryImpl.java:451) [our-app.war:5.1.0.Final]
My current understanding is that when running on CICS and using JDBC Type 2 Drivers only some threads are capable of opening a DB2 connection. That would be the application threads (the ones processing HTTP requests) as well as worker threads servicing CICSExecutorService.
The current solution is to:
Disable JDBC metadata lookup in JdbcEnvironmentInitiator by
setting hibernate.temp.use_jdbc_metadata_defaults property to
false
Wrap execution of IdTableHelper#executeIdTableCreationStatements in a Runnable and submit it to CICSExecutorService.
Would you consider this solution to be sufficient and suitable for production? Or maybe you use some different approach?
Versions used:
CICS Transaction Server for z/OS 5.3.0
WebSphere Application Server 8.5.5.8
Hibernate 5.1.0
Update: Just to clarify, once our application is started, it can query DB2 with no problems (when servicing HTTP requests). The problem is only related to startup.

CICS TS v5.3 support for the JPA feature in Liberty was recently made available in a service-refresh (July 2016). Prior to that update, attempting to run JPA in applications would result in very similar problems to those you describe.
Although you are running hibernate and you are on a CICS-enabled thread, it does not have the API environment (which will allow the type 2 JDBC call to succeed). New detection logic was developed specifically (but not exclusively) for use with the DB2 JDBC type 2 driver and JPA. This update was shipped in a recent service refresh and might cure the issues you are seeing.
Try applying:
http://www-01.ibm.com/support/docview.wss?crawler=1&uid=swg1PI58375
The description says it is for 'Standard-mode Liberty' support, but it contains other developments as outlined above.

The following solution was tested to work ok.
The idea is to execute the SQL/DDL statements using CICSExecutorService#runAsCICS. The following extension is registered via hibernate.hql.bulk_id_strategy property.
package org.hibernate.hql.spi.id.global;
import java.util.concurrent.*;
import org.hibernate.boot.spi.MetadataImplementor;
import org.hibernate.engine.jdbc.connections.spi.JdbcConnectionAccess;
import org.hibernate.engine.jdbc.spi.JdbcServices;
import org.springframework.util.ClassUtils;
import com.ibm.cics.server.*;
public class CicsAwareGlobalTemporaryTableBulkIdStrategy extends GlobalTemporaryTableBulkIdStrategy {
#Override
protected void finishPreparation(JdbcServices jdbcServices, JdbcConnectionAccess connectionAccess, MetadataImplementor metadata, PreparationContextImpl context) {
execute(() -> super.finishPreparation(jdbcServices, connectionAccess, metadata, context));
}
#Override
public void release(JdbcServices jdbcServices, JdbcConnectionAccess connectionAccess) {
execute(() -> super.release(jdbcServices, connectionAccess));
}
private void execute(Runnable runnable) {
if (isCics() && IsCICS.getApiStatus() == IsCICS.CICS_REGION_BUT_API_DISALLOWED) {
RunnableFuture<Void> task = new FutureTask<>(runnable, null);
CICSExecutorService.runAsCICS(task);
try {
task.get();
} catch (InterruptedException | ExecutionException e) {
throw new RuntimeException("Failed to execute in a CICS API-enabled thread. " + e.getMessage(), e);
}
} else {
runnable.run();
}
}
private boolean isCics() {
return ClassUtils.isPresent("com.ibm.cics.server.CICSExecutorService", null);
}
}
Note that the newer JCICS API version has an overlaod for runAsCics method accepting a Callable, which might be useful to simplify the CICS branch of the execute method to something like this:
CICSExecutorService.runAsCICS(() -> { runnable.run(); return null; }).get();
A few alternatives tried:
Wrapping just the connection acquisition action (org.hibernate.engine.jdbc.connections.internal.DatasourceConnectionProviderImpl#getConnection) did not work as the connection was closed already when it was used in the main thread.
Wrapping the whole application startup (org.springframework.web.context.ContextLoaderListener#contextInitialized) led to classloading issues.
Edit: Eventually went with a custom Hibernate's MultiTableBulkIdStrategy implementation that does not run any SQL/DDL on startup (see project page on GitHub).

Response latency time increases steadily on Jetty 6 server

This is the basic scenario:
I have a Jetty server running on a AWS instance T2.medium.
This server contains 3 REST services: A,B and C.
My client, which is a java class running in my local eclipse
Creates around 400 objects for service A
After that it creates 5 B objects for each A (2000)
Finally another 5 C objects for each B (10000).
In the end, we have around 15,000 objects created.
The execution takes 45-50 minutes to create all of the 15,000 objects.
During the first 100 requests, everything is good and beautiful, it takes around 25 to 30ms to complete each request.
After 800 requests, things are not quite good, takes around 161ms to 182ms to complete each request.
This time keeps increasing until it reaches 1300ms to 1321ms to complete each request. In the end, after around 10,000 requests each one takes around 3000ms to complete.
I'm not running multiple threads to do the requests.
I'm not running other services in this VM.
I'm using MySql on a RDS instance.
*Ohhh and 1 more thing, if I restart the Jetty and try to create the objects again, the behaviour keeps repeating, it starts with 30ms and gets to 3000ms. This makes me think it has something to do with the threadpool on jetty.
The code I'm using to start the jetty is:
QueuedThreadPool threadPool = new QueuedThreadPool(100);
threadPool.setMinThreads(100);
threadPool.setMaxThreads(700);
threadPool.setMaxIdleTimeMs(3000);
httpServer = new Server(port);
httpServer.setThreadPool(threadPool);
ContextHandlerCollection contexts = new ContextHandlerCollection();
httpServer.setHandler(contexts);
Context root = new Context(contexts, "/", Context.SESSIONS);
GenericWebApplicationContext springContext = new GenericWebApplicationContext();
springContext.setParent(new ClassPathXmlApplicationContext("/education/applicationContext.xml"));
root.setAttribute(WebApplicationContext.ROOT_WEB_APPLICATION_CONTEXT_ATTRIBUTE,springContext);
Context html = new Context(contexts, "/html", Context.SESSIONS);
ResourceHandler htmlHandler = new ResourceHandler();
htmlHandler.setResourceBase("src/main/webapp");
html.setHandler(htmlHandler);
ServletHolder holder = new ServletHolder(CXFServlet.class);
root.addServlet(holder, "/rest/*");
try {
httpServer.start();
} catch (Exception e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
and after a couple of minutes I start to see the following exception:
**
20:30:34.556 [509681393#qtp-1095433972-667] DEBUG org.mortbay.log - EXCEPTION
java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:196) ~[na:1.7.0_65]
at java.net.SocketInputStream.read(SocketInputStream.java:122) ~[na:1.7.0_65]
at org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:382) ~[jetty-6.1.26.jar:6.1.26]
at org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:114) ~[jetty-6.1.26.jar:6.1.26]
at org.mortbay.jetty.bio.SocketConnector$Connection.fill(SocketConnector.java:198) [jett6.1.26.jar:6.1.26]
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:290) [jetty-6.1.26.jar:6.1.26]
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212) [jetty-6.1.26.jar:6.1.26]
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) [jetty-6.1.26.jar:6.1.26]
at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228) [jetty- 6.1.26.jar:6.1.26]
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) [jetty- util-6.1.26.jar:6.1.26]
20:30:34.555 [867433358#qtp-1095433972-664] DEBUG org.mortbay.log - EXCEPTION
20:30:34.563 [509681393#qtp-1095433972-667] DEBUG org.mortbay.log - EOF
**

Suddenly I'm getting "Hazelcast instance is not active!"

My application is running for months and working very well. Then suddenly I get the following error:
com.hazelcast.core.HazelcastInstanceNotActiveException: Hazelcast instance is not active!
at com.hazelcast.spi.impl.ProxyServiceImpl$ProxyRegistry.<init>(ProxyServiceImpl.java:220)
at com.hazelcast.spi.impl.ProxyServiceImpl$ProxyRegistry.<init>(ProxyServiceImpl.java:207)
at com.hazelcast.spi.impl.ProxyServiceImpl$1.createNew(ProxyServiceImpl.java:69)
at com.hazelcast.spi.impl.ProxyServiceImpl$1.createNew(ProxyServiceImpl.java:67)
at com.hazelcast.util.ConcurrencyUtil.getOrPutIfAbsent(ConcurrencyUtil.java:47)
at com.hazelcast.spi.impl.ProxyServiceImpl.getDistributedObject(ProxyServiceImpl.java:101)
at com.hazelcast.instance.HazelcastInstanceImpl.getDistributedObject(HazelcastInstanceImpl.java:285)
at com.hazelcast.instance.HazelcastInstanceImpl.getLock(HazelcastInstanceImpl.java:183)
at com.hazelcast.instance.HazelcastInstanceProxy.getLock(HazelcastInstanceProxy.java:77)
at br.com.xyz.lock.hazelcast.HazelcastLockManager.lock(HazelcastLockManager.java:37)
at br.com.xyz.lock.hazelcast.LockManagerFacade.lock(LockManagerFacade.java:24)
at br.com.xyz.recebe.negocio.NProcessadorMensagemRecebida.processamentoLock(NProcessadorMensagemRecebida.java:85)
at br.com.xyz.recebe.negocio.NProcessadorMensagemRecebida.processaArquivo(NProcessadorMensagemRecebida.java:74)
at br.com.xyz.recebe.processador.ProcessadorBase.processaArquivo(ProcessadorBase.java:75)
at br.com.xyz.recebe.processador.ProcessadorXml.processaArquivo(ProcessadorXml.java:16)
at br.com.xyz.recebe.processador.ProcessadorFacade.processaArquivo(ProcessadorFacade.java:34)
at br.com.xyz.recebe.mail.pdes.ProcessadorPDESMeRecebida.processar(ProcessadorPDESMeRecebida.java:77)
at gov.sefaz.util.pdes.ProcessadorDiretorioEntradaSaidaDaemon.processar(ProcessadorDiretorioEntradaSaidaDaemon.java:575)
at gov.sefaz.util.pdes.ProcessadorDiretorioEntradaSaidaDaemon.varrerDiretorioUsingStrategy(ProcessadorDiretorioEntradaSaidaDaemon.java:526)
at gov.sefaz.util.pdes.ProcessadorDiretorioEntradaSaidaDaemon.run(ProcessadorDiretorioEntradaSaidaDaemon.java:458)
at java.lang.Thread.run(Unknown Source)
I found some issues on Google that says it shutdown because other errors. But in my case there isn't any.
It shutdown without reason.
Has anyone seen this before?

The problem occurs if a hazelcast member doesn't shutdown normally (terminate), When you stop HazelcastInstance which queue proxy is bound to; then any operation on that queue after instance stopped should throw HazelcastInstanceNotActiveException.

You just need to wait for a few minutes and rerun your job.