We Keep Coding

FlinkCEP pattern detection doesn't happen in real time

I'm still new to Flink CEP library and yet I don't understand the pattern detection behavior.
Considering the example below, I have a Flink app that consumes data from a kafka topic, data is produced periodically, I want to use Flink CEP pattern to detect when a value is bigger than a given threshold.
The code is below:
public class CEPJob{
public static void main(String[] args) throws Exception {
StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
Properties properties = new Properties();
properties.setProperty("bootstrap.servers", "localhost:9092");
properties.setProperty("group.id", "test");
FlinkKafkaConsumer<String> consumer = new FlinkKafkaConsumer<String>("test", new SimpleStringSchema(),
properties);
consumer.assignTimestampsAndWatermarks(WatermarkStrategy.forMonotonousTimestamps());
DataStream<String> stream = env.addSource(consumer);
// Process incoming data.
DataStream<Stock> inputEventStream = stream.map(new MapFunction<String, Stock>() {
private static final long serialVersionUID = -491668877013085114L;
#Override
public Stock map(String value) {
String[] data = value.split(":");
System.out.println("Date: " + data[0] + ", Adj Close: " + data[1]);
Stock stock = new Stock(data[0], Double.parseDouble(data[1]));
return stock;
}
});
// Create the pattern
Pattern<Stock, ?> myPattern = Pattern.<Stock>begin("first").where(new SimpleCondition<Stock>() {
private static final long serialVersionUID = -6301755149429716724L;
#Override
public boolean filter(Stock value) throws Exception {
return (value.getAdj_Close() > 140.0);
}
});
// Create a pattern stream from our warning pattern
PatternStream<Stock> myPatternStream = CEP.pattern(inputEventStream, myPattern);
// Generate alert for each matched pattern
DataStream<Stock> warnings = myPatternStream .select((Map<String, List<Stock>> pattern) -> {
Stock first = pattern.get("first").get(0);
return first;
});
warnings.print();
env.execute("CEP job");
}
}
What happens when I run the job, pattern detection doesn't happen in real-time, it outputs the warning for the detected pattern of the current record only after a second record is produced, it looks like it's delayed to print to the log the warining, I really didn't understand how to make it outputs the warning the time it detect the pattern without waiting for next record and thank you :) .
Data coming from Kafka are in string format: "date:value", it produce data every 5 secs.
Java version: 1.8, Scala version: 2.11.12, Flink version: 1.12.2, Kafka version: 2.3.0

The solution I found that to send a fake record (a null object for example) in the Kafka topic every time I produce a value to the topic, and on the Flink side (in the pattern declaration) I test if the received record is fake or not.
It seems like FlinkCEP always waits for the upcoming event before it outputs the warning.

Drools timed rule execution via Java API

I want to create a time-based rule that is being triggered every 5 minutes, and Drools documentation states that:
Conversely when the Drools engine runs in passive mode (i.e.: using fireAllRules instead of fireUntilHalt) by default it doesn’t fire consequences of timed rules unless fireAllRules isn’t invoked again. However it is possible to change this default behavior by configuring the KieSession with a TimedRuleExecutionOption as shown in the following example
KieSessionConfiguration ksconf = KieServices.Factory.get().newKieSessionConfiguration();
ksconf.setOption( TimedRuleExecutionOption.YES );
KSession ksession = kbase.newKieSession(ksconf, null);
However, I am not accessing the KieSession object directly because I am using the Java REST API to send requests to a Drools project deployed on KieExecution Server like so (example taken directly from the Drools documentation):
public class MyConfigurationObject {
private static final String URL = "http://localhost:8080/kie-server/services/rest/server";
private static final String USER = "baAdmin";
private static final String PASSWORD = "password#1";
private static final MarshallingFormat FORMAT = MarshallingFormat.JSON;
private static KieServicesConfiguration conf;
private static KieServicesClient kieServicesClient;
public static void initializeKieServerClient() {
conf = KieServicesFactory.newRestConfiguration(URL, USER, PASSWORD);
conf.setMarshallingFormat(FORMAT);
kieServicesClient = KieServicesFactory.newKieServicesClient(conf);
}
public void executeCommands() {
String containerId = "hello";
System.out.println("== Sending commands to the server ==");
RuleServicesClient rulesClient = kieServicesClient.getServicesClient(RuleServicesClient.class);
KieCommands commandsFactory = KieServices.Factory.get().getCommands();
Command<?> insert = commandsFactory.newInsert("Some String OBJ");
Command<?> fireAllRules = commandsFactory.newFireAllRules();
Command<?> batchCommand = commandsFactory.newBatchExecution(Arrays.asList(insert, fireAllRules));
ServiceResponse<ExecutionResults> executeResponse = rulesClient.executeCommandsWithResults(containerId, batchCommand);
if(executeResponse.getType() == ResponseType.SUCCESS) {
System.out.println("Commands executed with success! Response: ");
System.out.println(executeResponse.getResult());
} else {
System.out.println("Error executing rules. Message: ");
System.out.println(executeResponse.getMsg());
}
}
}
so I'm a bit confused as to how i can pass this TimedRuleExecutionOption to the session?
I've already found a workaround by sending a FireAllRules command periodically but I'd like to know if I can configure this session option so that I don't have to add periodical triggering for every timed event I want to create.
Also, I've tried using FireUntilHalt instead of FireAllRules, but to my understanding that command blocks the execution thread on the server and I have to send a HaltCommand at some point, all of which I would like to avoid since I have a multi-threaded client that sends events to the server.

pass "-Ddrools.timedRuleExecution=true" while starting server instance where kie-server.war is deployed.

You can use drools cron function. It acts as a timer and invoke rule based on the cron expresion. Example to execute a rule every 5 minutes :
rule "Send SMS every 5 minutes"
timer (cron:* 0/5 * * * ?)
when
$a : Event( )
then
end
you can find explanation here

Which class need to be injected in Bamboo plugin to get diff in Pull Request?

I've created project using:
atlas-create-bamboo-plugin
I'm trying to get diffs (or commit list) on current build. Any ideas?

It might not be the easiest way to get release notes, but as part of a deployment step you can generate a URL with commits between this and the previous release. By querying the release note URL, you can get each commit message.
#Scanned
public class ReleaseNoteTask implements DeploymentTaskType {
private static final String BAMBOO_URL = "https://bamboo.url"; // URL to bamboo
#Override
public TaskResult execute(DeploymentTaskContext taskContext) {
final TaskResultBuilder taskResultBuilder = TaskResultBuilder.newBuilder(taskContext);
final BuildLogger buildLogger = taskContext.getBuildLogger();
long versionId = taskContext.getDeploymentContext().getDeploymentVersion().getId();
String deploymentProjectId = taskContext.getCommonContext().getEntityKey().toString().split("-")[0];
String releaseNotesPath = BAMBOO_URL + "/deploy/viewDeploymentVersionCommitsSnippet.action?pageSize=-1&versionId=" + versionId + "&deploymentProjectId=" + deploymentProjectId + "&decorator=nothing&confirm=true&os_authType=basic";
// You can use a library like jsoup to read HTML from releaseNotesPath and parse each commit message
buildLogger.addBuildLogEntry("Release notes: " + releaseNotesPath);
return taskResultBuilder.success().build();
}
}

How to get an Initial Contex from Wildfly 8

ADDED 7/23.
Many views: Not even a "that's dumb" question in response. Can anyone at least tell me why such an embarrassingly trivial question seems to have no answer anywhere.
Q:
--- Have Wildfly 8 running on local machine localhost:9990.
--- Have a Java program that need's Wildfly's IntialContext.
--- Every reference says use: "Context ctx = new InitialContext(env);"
--- Yet a week of searching turns up no set of properties that returns one.
And no example of a java program that gets one.
Does no one ever do this? Really need help
Original Msg Below
I know many people have asked how to get an Initial context from Wildfly 8. But I have yet to find a simple answer with a simple example.
Therefore, I hope someone can tell my why this doesn’t work.
I start Wildfly with standalone-full.xml
The three sections below have
A - Code summary of my test Class whose only purpose is to secure an Initial Context. (I only removed a lot of printing code that produced the next section.]
B - The Eclipse console output for a failure.
C - Cut and paste code. Just in case anyone can help me get this to work. I’d like to leave behind something the next new WF user can cut and past and run. The only difference from 1 above is that this version has all the static methods I used to format the output. NOTE: I know the comments I inserted about the less than sign sound dumb. BUT ... they are true.
A Code Summary
import java.util.Properties;
import javax.naming.CommunicationException;
import javax.naming.Context;
import javax.naming.InitialContext;
public class JmsTestGetJNDIContext {
//members
final private Properties env = new Properties() {
private static final long serialVersionUID = 1L;
{
/* These are Properties used by a standalone JavaClient to secure a WIldFly InitialContext()*/
put(Context.INITIAL_CONTEXT_FACTORY, "org.jboss.naming.remote.client.InitialContextFactory");
put(Context.PROVIDER_URL,"http-remoting://localhost:9990");
put(Context.SECURITY_PRINCIPAL,"userGLB");
put(Context.SECURITY_CREDENTIALS,"Open");
put("jboss.naming.client.ejb.context", true);
/*The above URL, ID and PW successfully open Wildfly's Admin Console*/
}
};
//constructor
private JmsTestGetJNDIContext (){
/*print "beg"*/
/*print "env"*/
try {
/*print "Requesting InitialContext"*/
Context ctx = new InitialContext(this.env);
/*print "JNDI Context: " + ctx)*/
/*print "end");
} catch (CommunicationException e) {
/* print "You forgot to start WildFly dummy!"*/
} catch (Exception e) {
/* print"caught: " + e.getClass().getName()*/
/*print e.getMessage()*/
/* "end")*/
}
static public void main (String[] args) {
/*print "beg"*/
JmsTestGetJNDIContext client = new JmsTestGetJNDIContext ();
/*print "end"*/
}
}
B - Console Output
JmsTestGetJNDIContext.main () beg
JmsTestGetJNDIContext.<init> () beg
JmsTestGetJNDIContext.<init> () These are Properties used to obtain IntialContext
Key: java.naming.provider.url
Value: http-remoting://localhost:9990
Key: java.naming.factory.initial
Value: org.jboss.naming.remote.client.InitialContextFactory
Key: jboss.naming.client.ejb.context
Value: true
Key: java.naming.security.principal
Value: userGLB
Key: java.naming.security.credentials
Value: Open
JmsTestGetJNDIContext.<init> () Requesting InitialContext
JmsTestGetJNDIContext.<init> () caught: javax.naming.NamingException
JmsTestGetJNDIContext.<init> () Failed to create remoting connection
JmsTestGetJNDIContext.<init> () end
JmsTestGetJNDIContext.main () end
Cut and Paste Code
package org.america3.gotest.xtra;
import java.util.Properties;
import javax.naming.CommunicationException;
import javax.naming.Context;
import javax.naming.InitialContext;
public class JmsTestGetJNDIContext {
//members
final private Properties env = new Properties() {
/**
* Properties used by a standalone JavaClient to secure
* a WIldFly InitialContext()*/
private static final long serialVersionUID = 1L;
{
put(Context.INITIAL_CONTEXT_FACTORY,"org.jboss.naming.remote.client.InitialContextFactory");
put(Context.PROVIDER_URL, "http-remoting://localhost:9990");
put(Context.SECURITY_PRINCIPAL, "userGLB");
put(Context.SECURITY_CREDENTIALS, "Open");
// The above URL, ID and PW successfully open Wildfly's Admin Console
put("jboss.naming.client.ejb.context", true);
}
};
//constructor
private JmsTestGetJNDIContext (){/*ignore*/String iAm = JmsTestGetJNDIContext.getIAm(" ", Thread.currentThread().getStackTrace());
P (iAm, "beg");
pProps(iAm, env);
try {
P (sp + iAm, "Requesting InitialContext");
Context ctx = new InitialContext(this.env);
P (sp + iAm, "JNDI Context: " + ctx);
P (sp + iAm, "end");
} catch (CommunicationException e) {
P (sp + iAm, "You forgot to start WildFly dummy!");
} catch (Exception e) {
P (sp + iAm, "caught: " + e.getClass().getName());
P (sp + iAm, e.getMessage());
P (iAm, "end");
}
}
static public void main (String[] args) {/*ignore*/String iAm = JmsTestGetJNDIContext.getIAm("",Thread.currentThread().getStackTrace());
P (iAm, "beg");
JmsTestGetJNDIContext client = new JmsTestGetJNDIContext ();
P (iAm , "end");
}
/*The remaining static methods are just to facilitate printing.
* They are normally in a Untility package I add to my projects.
* I put them here so this code would run for anyone.*/
static private void pProps (String leader, Properties p) {
StringBuffer sb = new StringBuffer ();
String s = JmsTestGetJNDIContext.padRight(leader, 45, ' ');
s = " " + s + "These are Properties used to obtain IntialContext"+"\n";
sb.append(s);
String skip = "";
for (Object key: p.keySet()) {
sb.append(skip + " " + JmsTestGetJNDIContext.padRight("\""
+ (String)key + "\"", 40, ' ')
+ " \"" + p.get(key) + "\"");
skip = "\n";
}
System.out.println(sb);
}
static private void P (String s, String s2) {
System.out.println(s + s2);
}
static public String getClassMethodName (StackTraceElement[] elements) {
String className = null;
for (int i = 0; i * elements.length; i++]i ) {
/* You need to type in a less than sign for the '*'
* because when I do, the editor will not show any code
* that comes after it.
* I have no idea why, but I've spent over an hour trying,
* and every time I type a less than sign all the following
* code dissappears!*/
className = elements[i].getClassName ();
if (className.startsWith ("org.america3")) {
int end = className.lastIndexOf ('.');
return className.substring (end + 1) + "." + elements[i].getMethodName ();
} else {
continue;
}
}
return "no project method found in elements beginning with org.america3" ;
}
static private String getIAm (String indent, StackTraceElement[] elements) {
StringBuffer sb = new StringBuffer ();
sb.append(JmsTestGetJNDIContext.getClassMethodName(elements));
sb.append(" ()");
return indent + JmsTestGetJNDIContext.padRight (sb.toString(), 45, ' ') ;
}
static public String padRight(String s, int width, char c){
if (s == null) return "Null String";
if(s.length() ** width){
/* You need to type in a greater than or equal sign for
* the '**'see above.*/
return s;
} else {
StringBuffer sb = new StringBuffer();
sb.append (s);
for(int i = 0; i *** (width - s.length()); i++){
/*You need to type in a less than sign the '***'. Again see above*/
sb.append(c);
}
return sb.toString();
}
}
static public String sp = " ";
}

A while ago I also struggled with remote EJBs in my CLI application. I excavated a small example project that I wrote then. It gets an InitialContext and calls a remote EJB named AddBrackets:
import java.util.Properties;
import javax.naming.Context;
import javax.naming.InitialContext;
import javax.naming.NamingException;
import de.dnb.test.ejb.AddBrackets;
public final class Application {
public static void main(String[] args) throws NamingException {
final Properties jndiProperties = initJndiProperties();
final AddBrackets addBrackets = getEjb(jndiProperties);
System.out.println(addBrackets.processText("Hello World"));
}
private static Properties initJndiProperties() {
final Properties jndiProperties = new Properties();
jndiProperties.put(Context.INITIAL_CONTEXT_FACTORY,
"org.jboss.naming.remote.client.InitialContextFactory");
jndiProperties.put("jboss.naming.client.ejb.context", true);
jndiProperties.put(Context.PROVIDER_URL, "http-remoting://localhost:8080/");
//jndiProperties.put(Context.SECURITY_PRINCIPAL, "test");
//jndiProperties.put(Context.SECURITY_CREDENTIALS, "test");
return jndiProperties;
}
private static AddBrackets getEjb(Properties jndiProps)
throws NamingException {
final Context jndiContext = new InitialContext(jndiProps);
final String interfaceName = AddBrackets.class.getName();
return (AddBrackets) jndiContext.lookup(
"ejbtest-app-1.0-SNAPSHOT/ejbtest-ejb-1.0-SNAPSHOT/AddBracketsBean!"
+ interfaceName);
}
}
I built this program as a Maven project which had a dependency on
<dependency>
<groupId>org.wildfly</groupId>
<artifactId>wildfly-ejb-client-bom</artifactId>
<version>8.2.1.Final</version>
<type>pom</type>
</dependency>
This dependency brings in Wildfly's remote client EJB implementation and adds the following jars to the class path (links are to Maven Central):
jboss-logging-3.1.4.GA.jar
jboss-marshalling-1.4.9.Final.jar
jboss-marshalling-river-1.4.9.Final.jar
jboss-remoting-4.0.7.Final.jar
jboss-sasl-1.0.4.Final.jar
jboss-ejb-api_3.2_spec-1.0.0.Final.jar
jboss-transaction-api_1.2_spec-1.0.0.Final.jar
xnio-api-3.3.0.Final.jar
xnio-nio-3.3.0.Final.jar
jboss-ejb-client-2.0.1.Final.jar
jboss-remote-naming-2.0.1.Final.jar
wildfly-build-config-8.2.1.Final.jar
I did no special configuration on Wildfly to run this example. I simply downloaded a vanilla Wildfly 8.2.1, unzipped it, set up an admin user with the add-user.sh script and deployed my EJB in an EAR. As you can see above access is granted without a username and a password.
You can find the complete project including the AddBrackets EJB on my bitbucket account.
When I tried to get my head around remote EJBs with Wildfly, I found the article JBoss EAP / Wildfly – Three ways to invoke remote EJBs really helpful. It clearly describes the three different methods to access remote EJBs on Wildfly.

According to your own answer the following jars are on your classpath:
jboss-remote-naming-1.0.7.final.jar
jboss-logging.jar
xnio-api-3.0.7.ga.jar
jboss-remoting-3.jar
jboss-ejb-client-1.0.19.final.jar
You write that the application throws the following exception:
java.lang.NoSuchMethodError:
org.jboss.remoting3.Remoting.createEndpoint(Ljava/lang/String;Lorg/xnio/OptionMap;)Lorg/jboss/remoting3/Endpoint;]
This exception is thrown when org.jboss.naming.remote.client.EndpointCache which is part of the jboss-remote-naming jar tries to call Remoting.createEndpoint() which is contained in the jboss-remoting jar.
As you explain in your answer the reason for this is that the Remoting class declares a 3-parameter version of the createEndpoint() method while the EndpointCache class tries to call a 2-parameter version which does not exist.
I checked the commit histories and declared dependencies of the jboss-remote-naming and the jboss-remoting projects to find out what is going wrong there. This is what I found out:
The 2-parameter version of createEndpoint() was only added in version 3.2 of jboss-remoting. The pom.xml for jboss-remote-naming-1.0.7.final says it depends on jboss-remoting 3.2.7.GA.
As there is no version number on your jboss-remoting-3.jar, I guess it is an older version. You should be able to check this by looking for a pom.xml in META-INF folder of your jboss-remoting-3.jar. This should contain the version number.
To solve your problem, I suggest to replace your jboss-remoting-3.jar with jboss-remoting-3.2.7ga.jar or to use the set of jars I listed in my other answer.

I’ve decided the problem isn’t coding or the JNDI InititialContext Properties.
I mean the fatal error is a NoSuchMethodError. Therefore, as I confirmed in the WildFly server logs, my main method never even tries to connect.
Here’s what I think explains the real problem.
And I think it explains why there are so many calls for help with this error:
java.lang.NoSuchMethodError:
org.jboss.remoting3.Remoting.createEndpoint(Ljava/lang/String;Lorg/xnio/OptionMap;)Lorg/jboss/remoting3/Endpoint;]
Also why none of those calls for help ever get a conclusive answer. Just people suggesting different jars.
And since all those answers fixed on jars, this is how I tested the Build Path I was using:
First I removed all jars from the Build Path. Then I ran my one line main program till all ClassNotFoundException were gone.
First Error
java.lang.ClassNotFoundException:
org.jboss.naming.remote.client.InitialContextFactory]
Added jboss-remote-naming-1.0.7.final.jar to class path
Next Error
java.lang.NoClassDefFoundError:
org/jboss/logging/Logger
Added jboss-logging.jar
Next Error
java.lang.NoClassDefFoundError:
org/xnio/Options
Added xnio-api-3.0.7.ga.jar
Next Error
java.lang.NoClassDefFoundError:
org/jboss/remoting3/spi/ConnectionProviderFactory
Added jboss-remoting-3.jar
Next Error
java.lang.NoClassDefFoundError:
org/jboss/ejb/client/EJBClientContextIdentifier
Added jboss-ejb-client-1.0.19.final.jar
FATAL ERROR (note: All NoClassDefFoundError have been cleared)
java.lang.NoSuchMethodError:
org.jboss.remoting3.Remoting.createEndpoint(Ljava/lang/String;Lorg/xnio/OptionMap;)Lorg/jboss/remoting3/Endpoint;]
Then I used Eclipse’s Project Explorer to verify:
That jboss-remoting3.jar has the org.jboss.remoting3.Remoting Class. It does. That’s why there is no NoClassDefFoundError left above.
And verified it had this method:
public Endpoint createEndpoint (String, Executor, OptionMap) note: 3 parameters.
BUT the above Error indicates something is calling:
public Endpoint createEndpoint (String, OptionMap) note: 2 parameters.
That’s why the program throws a NoSuchMethodError. It is looking for a 2 paramater version of org.jboss.remoting3.Remoting.createEndpoint(). And the Remoting Class I have only has a 3 parameter version.`
I know this sounds impossible but the only thing I can think is there is an inconsistency in the Java API???
Clearly something is calling org.jboss.remoting3.Remoting.createEndpoint with 2 parameters.
But my org.jboss.remoting3.Remoting Class only has a 3 parameter version of the createEndpoint() Method.
So I’m going to clean this all up and repost a question asking how to explain the existence of a Class calling for a 2 paramter org.jboss.remoting3.Remoting.createEndpoint Method when I have a jar whose org.jboss.remoting3.Remoting only offers a 3-parameter.

Here is your obligatory "that's a dumb question." Does the wildfly remote quickstart github repo answer the question for you? Their code, from RemoteEJB.java
final Hashtable<String, String> jndiProperties = new Hashtable<>();
jndiProperties.put(Context.URL_PKG_PREFIXES, "org.jboss.ejb.client.naming");
final Context context = new InitialContext(jndiProperties);
return (RemoteCalculator) context.lookup("ejb:/ejb-remote-server-side/CalculatorBean!" + RemoteCalculator.class.getName());

Accumulo scan/write not running in standalone Java main program in AWS EC2 master using Cloudera CDH 5.8.2

We are trying to run simple write/sacn from Accumulo (client jar 1.5.0) in standalone Java main program (Maven shade executable) as below in AWS EC2 master (described below) using Putty
public class AccumuloQueryApp {
private static final Logger logger = LoggerFactory.getLogger(AccumuloQueryApp.class);
public static final String INSTANCE = "accumulo"; // miniInstance
public static final String ZOOKEEPERS = "ip-x-x-x-100:2181"; //localhost:28076
private static Connector conn;
static {
// Accumulo
Instance instance = new ZooKeeperInstance(INSTANCE, ZOOKEEPERS);
try {
conn = instance.getConnector("root", new PasswordToken("xxx"));
} catch (Exception e) {
logger.error("Connection", e);
}
}
public static void main(String[] args) throws TableNotFoundException, AccumuloException, AccumuloSecurityException, TableExistsException {
System.out.println("connection with : " + conn.whoami());
BatchWriter writer = conn.createBatchWriter("test", ofBatchWriter());
for (int i = 0; i < 10; i++) {
Mutation m1 = new Mutation(String.valueOf(i));
m1.put("personal_info", "first_name", String.valueOf(i));
m1.put("personal_info", "last_name", String.valueOf(i));
m1.put("personal_info", "phone", "983065281" + i % 2);
m1.put("personal_info", "email", String.valueOf(i));
m1.put("personal_info", "date_of_birth", String.valueOf(i));
m1.put("department_info", "id", String.valueOf(i));
m1.put("department_info", "short_name", String.valueOf(i));
m1.put("department_info", "full_name", String.valueOf(i));
m1.put("organization_info", "id", String.valueOf(i));
m1.put("organization_info", "short_name", String.valueOf(i));
m1.put("organization_info", "full_name", String.valueOf(i));
writer.addMutation(m1);
}
writer.close();
System.out.println("Writing complete ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~`");
Scanner scanner = conn.createScanner("test", new Authorizations());
System.out.println("Step 1 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~`");
scanner.setRange(new Range("3", "7"));
System.out.println("Step 2 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~`");
scanner.forEach(e -> System.out.println("Key: " + e.getKey() + ", Value: " + e.getValue()));
System.out.println("Step 3 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~`");
scanner.close();
}
public static BatchWriterConfig ofBatchWriter() {
//Batch Writer Properties
final int MAX_LATENCY = 1;
final int MAX_MEMORY = 10000000;
final int MAX_WRITE_THREADS = 10;
final int TIMEOUT = 10;
BatchWriterConfig config = new BatchWriterConfig();
config.setMaxLatency(MAX_LATENCY, TimeUnit.MINUTES);
config.setMaxMemory(MAX_MEMORY);
config.setMaxWriteThreads(MAX_WRITE_THREADS);
config.setTimeout(TIMEOUT, TimeUnit.MINUTES);
return config;
}
}
Connection is established correctly but creating BatchWriter it getting error and it's trying in loop with same error
[impl.ThriftScanner] DEBUG: Error getting transport to ip-x-x-x-100:10011 : NotServingTabletException(extent:TKeyExtent(table:21 30, endRow:21 30 3C, prevEndRow:null))
When we run the same code (writing to Accumulo and reading from Accumulo) inside Spark job and submit to the YANK cluster it's running perfectly. We are struggling to figure out that but getting no clue. Please see the environment as described below
Cloudera CDH 5.8.2 on AWS environemnts (4 EC2 instances as one master and 3 child).
Consider the private IPs are like
Mater: x.x.x.100
Child1: x.x.x.101
Child2: x.x.x.102
Child3: x.x.x.103
We havethe follwing installation in CDH
Cluster (CDH 5.8.2)
Accumulo 1.6 (Tracer not installed, Garbage Collector in Child2, Master in Master, Monitor in child3, Tablet Server in Master)
HBase
HDFS (master as name node, all 3 child as datanode)
Kafka
Spark
YARN (MR2 Included)
ZooKeeper

Hrm, that's very curious that it runs with the Spark-on-YARN, but as a regular Java application. Usually, it's the other way around :)
I would verify that the JARs on the classpath of the standalone java app match the JARs used by the Spark-on-YARN job as well as the Accumulo server classpath.
If that doesn't help, try to increase the log4j level to DEBUG or TRACE and see if anything jumps out at you. If you have a hard time understanding what the logging is saying, feel free to send an email to user#accumulo.apache.org and you'll definitely have more eyes on the problem.