We Keep Coding

How to read file only once when app deployed on two nodes

I'm going to read files from SFTP location line by line:
#Override
public void configure() {
from(sftpLocationUrl)
.routeId("route-name")
.split(body().tokenize("\n"))
.streaming()
.bean(service, "build")
.to(String.format("activemq:%s", queueName));
}
But this application will be deployed on two nodes, and I think that in this case, I can get an unstable and unpredictable application work because the same lines of the file can be read twice.
Is there a way to avoid such duplicates in this case?

Camel has some (experimental) clustering capabilities - see here.
In your particular case, you could model a route which is taking the leadership when starting the directory polling, preventing thereby other nodes from picking the (same or other) files.

soluion is active passive mode . “In active/passive mode, you have a single master instance polling for files, while all the other instances (slaves) are passive. For this strategy to work, some kind of locking mechanism must be in use to ensure that only the node holding the lock is the master and all other nodes are on standby.”
it can implement with hazelcast, consul or zookeper
public class FileConsumerRoute extends RouteBuilder {
private int delay;
private String name;
public FileConsumerRoute(String name, int delay) {
this.name = name;
this.delay = delay;
}
#Override
public void configure() throws Exception {
// read files from the shared directory
from("file:target/inbox" +
"?delete=true")
// setup route policy to be used
.routePolicyRef("myPolicy")
.log(name + " - Received file: ${file:name}")
.delay(delay)
.log(name + " - Done file: ${file:name}")
.to("file:target/outbox");
}}
ServerBar
public class ServerBar {
private Main main;
public static void main(String[] args) throws Exception {
ServerBar bar = new ServerBar();
bar.boot();
}
public void boot() throws Exception {
// setup the hazelcast route policy
ConsulRoutePolicy routePolicy = new ConsulRoutePolicy();
// the service names must be same in the foo and bar server
routePolicy.setServiceName("myLock");
routePolicy.setTtl(5);
main = new Main();
// bind the hazelcast route policy to the name myPolicy which we refer to from the route
main.bind("myPolicy", routePolicy);
// add the route and and let the route be named Bar and use a little delay when processing the files
main.addRouteBuilder(new FileConsumerRoute("Bar", 100));
main.run();
}
}
Server Foo
public class ServerFoo {
private Main main;
public static void main(String[] args) throws Exception {
ServerFoo foo = new ServerFoo();
foo.boot();
}
public void boot() throws Exception {
// setup the hazelcast route policy
ConsulRoutePolicy routePolicy = new ConsulRoutePolicy();
// the service names must be same in the foo and bar server
routePolicy.setServiceName("myLock");
routePolicy.setTtl(5);
main = new Main();
// bind the hazelcast route policy to the name myPolicy which we refer to from the route
main.bind("myPolicy", routePolicy);
// add the route and and let the route be named Bar and use a little delay when processing the files
main.addRouteBuilder(new FileConsumerRoute("Foo", 100));
main.run();
}}
Source : Camel In Action 2nd Edition

Can't connect to a testcontainer Postgres instance

I've created a Postgres instance using testcontainers. The container starts but I cannot access it.
I have tried connecting at the containerized DB using DBeaver.
In the eclipse console everything seems fine:
01:29:34.662 [main] DEBUG com.github.dockerjava.core.command.AbstrDockerCmd - Cmd: com.github.dockerjava.core.command.CreateContainerCmdImpl#73386d72[name=,hostName=,domainName=,user=,attachStdin=,attachStdout=,attachStderr=,portSpecs=,tty=,stdinOpen=,stdInOnce=,env={POSTGRES_USER=test,POSTGRES_PASSWORD=test,POSTGRES_DB=ASIGDB_TEST}
Here is my code:
public class CustomPostgresContainer extends PostgreSQLContainer<CustomPostgresContainer>{
private static final String IMAGE_VERSION = "postgres:9.6";
private static CustomPostgresContainer customPostgresContainer;
private static final int EXPOSED_PORT = 5555;
private static final String DB_NAME = "ASIGDB_TEST";
private static final String DB_USER= "test";
private static final String DB_PASSWORD= "test";
public CustomPostgresContainer() {
super(IMAGE_VERSION);
}
public static CustomPostgresContainer getCustomPostgresContainerInstance() {
if(customPostgresContainer == null) {
return extracted().withExposedPorts(EXPOSED_PORT)
.withDatabaseName(DB_NAME)
.withUsername(DB_USER)
.withPassword(DB_PASSWORD);
}
return customPostgresContainer;
}
private static CustomPostgresContainer extracted() {
return new CustomPostgresContainer();
}
#Override
public void start() {
super.start();
}
#Override
public void stop() {
//do nothing, JVM handles shut down
}
}
I get:
Connection to localhost:5555 refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.
Does anyone know what is going on?

According to this link, withExposedPorts() --> this exposed port number is from the perspective of the container.
From the host's perspective Testcontainers actually exposes this on a random free port. This is by design, to avoid port collisions that may arise with locally running software or in between parallel test runs.
Because there is this layer of indirection, it is necessary to ask Testcontainers for the actual mapped port at runtime. This can be done using the getMappedPort method, which takes the original (container) port as an argument:
Integer firstMappedPort = container.getMappedPort(yourExposedPort);<br/>
Try to connect with DBeaver to the port that appears first.

Buffer and flush Apache Beam streaming data

I have a streaming job that with initial run will have to process large amount of data. One of DoFn calls remote service that supports batch requests, so when working with bounded collections I use following approach:
private static final class Function extends DoFn<String, Void> implements Serializable {
private static final long serialVersionUID = 2417984990958377700L;
private static final int LIMIT = 500;
private transient Queue<String> buffered;
#StartBundle
public void startBundle(Context context) throws Exception {
buffered = new LinkedList<>();
}
#ProcessElement
public void processElement(ProcessContext context) throws Exception {
buffered.add(context.element());
if (buffered.size() > LIMIT) {
flush();
}
}
#FinishBundle
public void finishBundle(Context c) throws Exception {
// process remaining
flush();
}
private void flush() {
// build batch request
while (!buffered.isEmpty()) {
buffered.poll();
// do something
}
}
}
Is there a way to window data so the same approach can be used on unbounded collections?
I've tried following:
pipeline
.apply("Read", Read.from(source))
.apply(WithTimestamps.of(input -> Instant.now()))
.apply(Window.into(FixedWindows.of(Duration.standardMinutes(2L))))
.apply("Process", ParDo.of(new Function()));
but startBundle and finishBundle are called for every element. Is there a chance to have something like with RxJava (2 minute windows or 100 element bundles):
source
.toFlowable(BackpressureStrategy.LATEST)
.buffer(2, TimeUnit.MINUTES, 100)

This is a quintessential use case for the new feature of per-key-and-windows state and timers.
State is described in a Beam blog post, while for timers you'll have to rely on the Javadoc. Nevermind what the javadoc says about runners supporting them, the true status is found in Beam's capability matrix.
The pattern is very much like what you have written, but state allows it to work with windows and also across bundles, since they may be very small in streaming. Since state must be partitioned somehow to maintain parallelism, you'll need to add some sort of key. Currently there is no automatic sharding for this.
private static final class Function extends DoFn<KV<Key, String>, Void> implements Serializable {
private static final long serialVersionUID = 2417984990958377700L;
private static final int LIMIT = 500;
#StateId("bufferedSize")
private final StateSpec<Object, ValueState<Integer>> bufferedSizeSpec =
StateSpecs.value(VarIntCoder.of());
#StateId("buffered")
private final StateSpec<Object, BagState<String>> bufferedSpec =
StateSpecs.bag(StringUtf8Coder.of());
#TimerId("expiry")
private final TimerSpec expirySpec = TimerSpecs.timer(TimeDomain.EVENT_TIME);
#ProcessElement
public void processElement(
ProcessContext context,
BoundedWindow window,
#StateId("bufferedSize") ValueState<Integer> bufferedSizeState,
#StateId("buffered") BagState<String> bufferedState,
#TimerId("expiry") Timer expiryTimer) {
int size = firstNonNull(bufferedSizeState.read(), 0);
bufferedState.add(context.element().getValue());
size += 1;
bufferedSizeState.write(size);
expiryTimer.set(window.maxTimestamp().plus(allowedLateness));
if (size > LIMIT) {
flush(context, bufferedState, bufferedSizeState);
}
}
#OnTimer("expiry")
public void onExpiry(
OnTimerContext context,
#StateId("bufferedSize") ValueState<Integer> bufferedSizeState,
#StateId("buffered") BagState<String> bufferedState) {
flush(context, bufferedState, bufferedSizeState);
}
private void flush(
WindowedContext context,
BagState<String> bufferedState,
ValueState<Integer> bufferedSizeState) {
Iterable<String> buffered = bufferedState.read();
// build batch request from buffered
...
// clear things
bufferedState.clear();
bufferedSizeState.clear();
}
}
Taking a few notes here:
State replaces your DoFn's instance variables, since
instance variables have no cohesion across windows.
The buffer and the size are just initialized as needed instead
of #StartBundle.
The BagState supports "blind" writes, so there doesn't need to be
any read-modify-write, just committing the new elements in the same
way as when you output.
Setting a timer repeatedly for the same time is just fine;
it should mostly be a noop.
#OnTimer("expiry") takes the place of #FinishBundle, since
finishing a bundle is not a per-window thing but an artifact of
how a runner executes your pipeline.
All that said, if you are writing to an external system, perhaps you would want to reify the windows and re-window into the global window before just doing writes where the manner of your write depends on the window, since "the external world is globally windowed".

The documentation for apache beam 0.6.0 says that StateId is "Not currently supported by any runner."

Cassandra in Java - Same cluster name everytime

I was trying to write a basic java program in eclipse that uses Cassandra java driver to connect to a Cassandra node.
I found this repository https://github.com/datastax/java-driver.
When I tried to run using -
package com.example.cassandra;
import com.datastax.driver.core.Cluster;
import com.datastax.driver.core.Session;
import com.datastax.driver.core.Host;
public class SampleConnection {
private Cluster cluster;
private Session session;
public void connect(String node){
cluster = Cluster.builder().addContactPoint(node).build();
session = cluster.connect("mykeyspace");
System.out.println(cluster.getClusterName());
}
public void close()
{
cluster.shutdown();
}
public static void main(String args[]) {
SampleConnection client = new SampleConnection();
client.connect("127.0.0.1");
client.close();
}
1) Encountered output in eclipse as
Exception in thread "main" com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: [/127.0.0.1])
Why it is refusing even to connect let alone create a table? (9042 port, configured in cassandra.yaml, is open & cassandra service is running)
2) Why, in my code, cluster.getClusterName(); is giving "cluster1" as cluster name everytime regardless of my cluster name in cassandra.yaml file?
However, when I tried to use the below code, it worked:
import com.datastax.driver.core.Cluster;
import com.datastax.driver.core.Host;
import com.datastax.driver.core.Session;
import com.datastax.driver.core.policies.DefaultRetryPolicy;
public class CassConnector {
private static Cluster cluster;
private static Session session;
public static Cluster connect(String node) {
return Cluster.builder().addContactPoint(node)
.withRetryPolicy(DefaultRetryPolicy.INSTANCE).build();
}
public static void main(String[] arg) {
cluster = connect("localhost");
session = cluster.connect("mykeyspace");
session.execute("CREATE KEYSPACE myks WITH REPLICATION = "
+ "{ 'class' : 'SimpleStrategy', 'replication_factor' : 1};" );
session.execute("USE mykeyspace");
String query = "CREATE TABLE emp(emp_id int PRIMARY KEY, "
+ "emp_name text, "
+ "emp_city text );";
session.execute(query);
System.out.println("Table created!");
session.close();
cluster.close();
}
What's the logical difference between these two approaches?

I assume you're referring to Cluster.getClusterName(). From the javadoc:
Note that this is not the Cassandra cluster name, but rather a name assigned to this Cluster object. Currently, that name is only used for one purpose: to distinguish exposed JMX metrics when multiple Cluster instances live in the same JVM (which should be rare in the first place). That name can be set at Cluster building time (through Cluster.Builder.withClusterName(java.lang.String) for instance) but will default to a name like cluster1 where each Cluster instance in the same JVM will have a different number.

Problems passing class objects through GWT RPC

I've run through the Google Web Toolkit StockWatcher Tutorial using Eclipse and the Google Plugin, and I'm attempting to make some basic changes to it so I can better understand the RPC framework.
I've modified the "getStocks" method on the StockServiceImpl server-side class so that it returns an array of Stock objects instead of String objects. The application compiles perfectly, but the Google Web Toolkit is returning the following error:
"No source code is available for type com.google.gwt.sample.stockwatcher.server.Stock; did you forget to inherit a required module?"
It seems that the client-side classes can't find an implementation of the Stock object, even though the class has been imported. For reference, here is a screenshot of my package hierarchy:
I suspect that I'm missing something in web.xml, but I have no idea what it is. Can anyone point me in the right direction?
EDIT: Forgot to mention that the Stock class is persistable, so it needs to stay on the server-side.

After much trial and error, I managed to find a way to do this. It might not be the best way, but it works. Hopefully this post can save someone else a lot of time and effort.
These instructions assume that you have completed both the basic StockWatcher tutorial and the Google App Engine StockWatcher modifications.
Create a Client-Side Implementation of the Stock Class
There are a couple of things to keep in mind about GWT:
Server-side classes can import client-side classes, but not vice-versa (usually).
The client-side can't import any Google App Engine libraries (i.e. com.google.appengine.api.users.User)
Due to both items above, the client can never implement the Stock class that we created in com.google.gwt.sample.stockwatcher.server. Instead, we'll create a new client-side Stock class called StockClient.
StockClient.java:
package com.google.gwt.sample.stockwatcher.client;
import java.io.Serializable;
import java.util.Date;
public class StockClient implements Serializable {
private Long id;
private String symbol;
private Date createDate;
public StockClient() {
this.createDate = new Date();
}
public StockClient(String symbol) {
this.symbol = symbol;
this.createDate = new Date();
}
public StockClient(Long id, String symbol, Date createDate) {
this();
this.id = id;
this.symbol = symbol;
this.createDate = createDate;
}
public Long getId() {
return this.id;
}
public String getSymbol() {
return this.symbol;
}
public Date getCreateDate() {
return this.createDate;
}
public void setId(Long id) {
this.id = id;
}
public void setSymbol(String symbol) {
this.symbol = symbol;
}
}
Modify Client Classes to Use StockClient[] instead of String[]
Now we make some simple modifications to the client classes so that they know that the RPC call returns StockClient[] instead of String[].
StockService.java:
package com.google.gwt.sample.stockwatcher.client;
import com.google.gwt.sample.stockwatcher.client.NotLoggedInException;
import com.google.gwt.user.client.rpc.RemoteService;
import com.google.gwt.user.client.rpc.RemoteServiceRelativePath;
#RemoteServiceRelativePath("stock")
public interface StockService extends RemoteService {
public Long addStock(String symbol) throws NotLoggedInException;
public void removeStock(String symbol) throws NotLoggedInException;
public StockClient[] getStocks() throws NotLoggedInException;
}
StockServiceAsync.java:
package com.google.gwt.sample.stockwatcher.client;
import com.google.gwt.sample.stockwatcher.client.StockClient;
import com.google.gwt.user.client.rpc.AsyncCallback;
public interface StockServiceAsync {
public void addStock(String symbol, AsyncCallback<Long> async);
public void removeStock(String symbol, AsyncCallback<Void> async);
public void getStocks(AsyncCallback<StockClient[]> async);
}
StockWatcher.java:
Add one import:
import com.google.gwt.sample.stockwatcher.client.StockClient;
All other code stays the same, except addStock, loadStocks, and displayStocks:
private void loadStocks() {
stockService = GWT.create(StockService.class);
stockService.getStocks(new AsyncCallback<String[]>() {
public void onFailure(Throwable error) {
handleError(error);
}
public void onSuccess(String[] symbols) {
displayStocks(symbols);
}
});
}
private void displayStocks(String[] symbols) {
for (String symbol : symbols) {
displayStock(symbol);
}
}
private void addStock() {
final String symbol = newSymbolTextBox.getText().toUpperCase().trim();
newSymbolTextBox.setFocus(true);
// Stock code must be between 1 and 10 chars that are numbers, letters,
// or dots.
if (!symbol.matches("^[0-9a-zA-Z\\.]{1,10}$")) {
Window.alert("'" + symbol + "' is not a valid symbol.");
newSymbolTextBox.selectAll();
return;
}
newSymbolTextBox.setText("");
// Don't add the stock if it's already in the table.
if (stocks.contains(symbol))
return;
addStock(new StockClient(symbol));
}
private void addStock(final StockClient stock) {
stockService.addStock(stock.getSymbol(), new AsyncCallback<Long>() {
public void onFailure(Throwable error) {
handleError(error);
}
public void onSuccess(Long id) {
stock.setId(id);
displayStock(stock.getSymbol());
}
});
}
Modify the StockServiceImpl Class to Return StockClient[]
Finally, we modify the getStocks method of the StockServiceImpl class so that it translates the server-side Stock classes into client-side StockClient classes before returning the array.
StockServiceImpl.java
import com.google.gwt.sample.stockwatcher.client.StockClient;
We need to change the addStock method slightly so that the generated ID is returned:
public Long addStock(String symbol) throws NotLoggedInException {
Stock stock = new Stock(getUser(), symbol);
checkLoggedIn();
PersistenceManager pm = getPersistenceManager();
try {
pm.makePersistent(stock);
} finally {
pm.close();
}
return stock.getId();
}
All other methods stay the same, except getStocks:
public StockClient[] getStocks() throws NotLoggedInException {
checkLoggedIn();
PersistenceManager pm = getPersistenceManager();
List<StockClient> stockclients = new ArrayList<StockClient>();
try {
Query q = pm.newQuery(Stock.class, "user == u");
q.declareParameters("com.google.appengine.api.users.User u");
q.setOrdering("createDate");
List<Stock> stocks = (List<Stock>) q.execute(getUser());
for (Stock stock : stocks)
{
stockclients.add(new StockClient(stock.getId(), stock.getSymbol(), stock.getCreateDate()));
}
} finally {
pm.close();
}
return (StockClient[]) stockclients.toArray(new StockClient[0]);
}
Summary
The code above works perfectly for me when deployed to Google App Engine, but triggers an error in Google Web Toolkit Hosted Mode:
SEVERE: [1244408678890000] javax.servlet.ServletContext log: Exception while dispatching incoming RPC call
com.google.gwt.user.server.rpc.UnexpectedException: Service method 'public abstract com.google.gwt.sample.stockwatcher.client.StockClient[] com.google.gwt.sample.stockwatcher.client.StockService.getStocks() throws com.google.gwt.sample.stockwatcher.client.NotLoggedInException' threw an unexpected exception: java.lang.NullPointerException: Name is null
Let me know if you encounter the same problem or not. The fact that it works in Google App Engine seems to indicate a bug in Hosted Mode.

GWT needs the .java file in addition to the .class file. Additionally, Stock needs to be in the "client" location of a GWT module.

The GWT compiler doesn't know about Stock, because it's not in a location it looks in. You can either move it to the client folder, or if it makes more sense leave it where it is and create a ModuleName.gwt.xml that references any other classes you want, and get your Main.gwt.xml file to inherit from that.
eg: DomainGwt.gwt.xml
<module>
<inherits name='com.google.gwt.user.User'/>
<source path="javapackagesabovethispackagegohere"/>
</module>
and:
<module rename-to="gwt_ui">
<inherits name="com.google.gwt.user.User"/>
<inherits name="au.com.groundhog.groundpics.DomainGwt"/>
<entry-point class="au.com.groundhog.groundpics.gwt.client.GPicsUIEntryPoint"/>
</module>

There's a better answer here: GWT Simple RPC use case problem : Code included
Basically, you can add parameters to your APPNAME.gwt.xml file so the compiler to give the compiler a path to the server-side class.

I was getting the same issue and the "mvn gwt:compile" output was not very helpful.
Instead, when I tried deploying to tomcat (via the maven tomcat plugin: mvn tomcat:deploy) I got helpful error messages.
A few things I had to fix up:
Make the object that is sent from the client to the server implement Serializable
Add an empty-arg constructor to that same object

Yes, it is sure that we need to use the Serialization for getting the server objects to the client. These modile?? file settings won't work to use the Stock class in the client side.
In your case you have only one class Stock and you can create a StockClient in client side. It is easy. But what will be the solution if anyone having more classes. Something like the properties of this class are also some other type of classes.
Example: stock.getEOD(date).getHigh();
getEOD will return another class with the given date and that class has the getHigh method.
What to do in such big cases? I don't think creating all classes implementing serialization in client side is good for that. Then we have to write code in both server and client. all classes two times.

Keying off of rustyshelf's answer above ...
In my case I needed to edit the ModuleName.gwt.xml file and add the following:
<source path='client'/>
<source path='shared'/>
I created my project with the New->Web Application Project wizard but unchecked the Generate project sample code option. I then created the shared package. Had I not unchecked that, the package would have been created for me and the xml file modified per the above.

There is a far more simple and easy solution for that. If you want to send an object of your custom designed class from server side to client side you should define this custom class in shared package.
For example for your case the you just have to carry the Stock.java class (by drag and drop) into
com.google.gwt.sample.stockwatcher.shared
package. However from your package hierarchy screenshot it seems that you had deleted this shared package. Just re-create this package and drop the Stock.java inside it and let the game begin.