We Keep Coding

JdbcIO.read is not returning results in apache beam

I am trying to read some data using jdbIO.read in apache beam and it works fine if I have code as follows.
Pipeline p = createPipeline(options);
p.apply(JdbcIO.<TestRow>read()
.withDataSourceConfiguration(JdbcIO.DataSourceConfiguration.create(dataSource))
.withQuery("query ")
.withCoder(SerializableCoder.of(TestRow.class))
.withRowMapper(new JdbcIO.RowMapper<TestRow>() {
#Override
public TestRow mapRow(ResultSet resultSet) throws Exception {
TestRow testRow = new TestRow();
//setters
return testRow;
}
}))
.apply(MapElements.via(new SimpleFunction<TestRow, String>() {
#Override
public String apply(TestRow input) {
return input.toString();
}
}));
But not getting any results, when I refactor this in a way to remove anonymous functions and put that call in separate class and extending DoFn class. the row mapper block is not getting executing at all.
PCollection<String> t = p
.apply(Create.<Long>of(1L))
.apply("Read Data", ParDo.of(readInput))
public abstract class ReadInput<S, T> extends DoFn<Long, TestRow> {
#DoFn.ProcessElement
public void processElement(#Element Long seq, final OutputReceiver<TestRow> receiver) {
getInput(receiver);
public class ReadInputOtc extends ReadInput<Long, TestRow>
#Override
protected void getInput(OutputReceiver<TestRow> receiver) {
JdbcIO.<TestRow>read()
.withDataSourceConfiguration(JdbcIO.DataSourceConfiguration.create(this.dataSource))
.withCoder(SerializableCoder.of(TestRow.class))
.withQuery("query ")
.withRowMapper(new JdbcIO.RowMapper<TestRow>() {
public TestRow mapRow(ResultSet resultSet) throws Exception {
TestRow testRow = new TestRow();
//setters
while (resultSet.next()) {
System.out.println(resultSet.getString("id"));
}
receiver.output(testRow);
return testRow;
}
});
}
thanks for your help

JdbcIO.<TestRow>read() just creates a reading PTransform, it does not actually do any reading. To do the read, it must be applied to the pipeline object (as you have in your first example) which produces a PCollection of records. PTransforms are not meant to be used within a DoFn, DoFns act on individual elements, not PCollections of elements.
If you are trying to remove anonomous classes, you could write your code as follows
[public static] class MuRowMapper extends JdbcIO.RowMapper<TestRow> {
#Override
public TestRow mapRow(ResultSet resultSet) throws Exception {
TestRow testRow = new TestRow();
...
return testRow;
}
}
[public static] class MyDoFn extends DoFn<MyRow, String> {
#DoFn.ProcessElement
public void processElement(#Element TestRow testRow,
final OutputReceiver<String> receiver) {
return receiver.output(testRow.toString());
}
}
Pipeline p = createPipeline(options);
p
.apply(JdbcIO.<TestRow>read()
.withDataSourceConfiguration(
JdbcIO.DataSourceConfiguration.create(dataSource))
.withQuery("query ")
.withCoder(SerializableCoder.of(TestRow.class))
.withRowMapper(new MyRowMapper()))
.apply(ParDo.of(new MyDoFn()));

Java Priority Queue Issue

Issue: I've got a priority queue to process actions. When I instantiate the actions and then add them to the queue it works, however when I instantiate them directly as I add them to the queue it no longer retains the priority.
This works - Executes by priority
public static void main(String[] args) {
final Action<Void> low = new LowAction();
final Action<Void> med = new MedAction();
final Action<Integer> high = new HighAction();
final Action<Boolean> walk = new WalkAction();
final ActionScheduler scheduler = new ActionScheduler(1,10);
scheduler.queue(high);
scheduler.queue(walk);
scheduler.queue(low);
scheduler.queue(med);
}
This does not work - Executes in the order I called them
public static void main(String[] args) {
final ActionScheduler scheduler = new ActionScheduler(1,10);
scheduler.queue(new HighAction());
scheduler.queue(new WalkAction());
scheduler.queue(new LowAction());
scheduler.queue(new MedAction());
}
ActionScheduler class
public class ActionScheduler {
private ExecutorService priorityJobPoolExecutor;
private ExecutorService priorityJobScheduler
= Executors.newSingleThreadExecutor();
private PriorityBlockingQueue<Action<?>> priorityQueue;
private Future<?> result;
public ActionScheduler(Integer poolSize, Integer queueSize) {
priorityJobPoolExecutor = Executors.newFixedThreadPool(poolSize);
priorityQueue = new PriorityBlockingQueue<>(queueSize);
priorityJobScheduler.submit(() -> {
while (true) {
try {
result = priorityJobPoolExecutor.submit(priorityQueue.take());
} catch (InterruptedException e) {
e.printStackTrace();
break;
}
}
});
}
public void schedule(Action<?> action) {
priorityQueue.offer(action);
}
public <T> Future<T> queue(Action<?> action) {
this.schedule(action);
return (Future<T>) result;
}
}
This also works correctly
public static void main(String[] args) {
final Action<Void> low = new LowAction();
final Action<Void> med = new MedAction();
final Action<Integer> high = new HighAction();
final Action<Boolean> walk = new WalkAction();
final ActionScheduler scheduler = new ActionScheduler(1,10);
scheduler.queue(new HighAction());
scheduler.queue(new WalkAction());
scheduler.queue(new LowAction());
scheduler.queue(new MedAction());
}
If anyone could offer any insight on why this is happening and how I can get it to execute by priority in both examples posted it would be greatly appreciated.
EDIT
Action Class
public abstract class Action<T> implements Callable<T>, Comparable<Action<?>> {
private final ActionContext context;
public Action(ActionContext context) {
this.context = context;
}
#Override
public int compareTo(Action action) {
if (action.getContext().getPriority() == this.getContext().getPriority()) {
return 0;
} else if (this.getContext().getPriority().ordinal() > action.getContext().getPriority().ordinal()) {
return -1;
} else {
return 1;
}
}
public ActionContext getContext() {
return context;
}
}
LowAction class
public class LowAction extends Action<Void> {
public LowAction() {
super(new ActionContext("low", Priority.LOW, true, false));
}
#Override
public Void call() throws Exception {
System.out.println("LOW");
return null;
}
}

There is one difference which I can see in the first approach you are binding your object with an actual type like final Action<**Void**> low = new LowAction(); on the other hand new LowAction() will be treat as a raw creation.
The internal working of the PriorityQueue is based on the Binary Heap.The elements of the priority queue are ordered according to the natural ordering, or by a comparator provided at construction time of the queue, depending on which constructor is used.

Can I add a new child class at run time?

I need to add the following class at run time:
public class MapperFunction extends Mapper<Integer, String, String, Integer> {
public MapperFunction(InputBlock s) {
super(s);
}
#Override
protected void map(Integer key, String value) {
Pattern pattern = Pattern.compile("[a-zA-Z]+");
Matcher matcher;
String str = value;
if (!str.equals("")) {
matcher = pattern.matcher(str);
while (matcher.find()) {
String word = matcher.group();
if (!MapperOut.containsKey(word))
MapperOut.put(word, 1);
else
MapperOut.put(word, (Integer) MapperOut.get(word) + 1);
}
}
}
}
or add the method map() to class during run time. I read the following question on stackoverflow Extending or adding new classes at runtime in Java
but I think my case differs a bit, anyway this is the parent class Mapper:
public abstract class Mapper<keyIn, valueIn, keyOut, valueOut> implements Runnable {
private RecordReader recordReader;
static AtomicInteger numberOfThreadsPerMapper = new AtomicInteger(2);
static Map<Object, Object> MapperOut = null;
static {
MapperOut = Collections.synchronizedMap(new TreeMap<>());
}
Mapper(InputBlock s) {
recordReader = new LineRecordReader(s);
}
protected abstract void map(keyIn key, valueIn value) throws IOException, InterruptedException;
#Override
public void run() {
run2();
}
public void run2() {
try {
while (recordReader.hasNext()) {
map((keyIn) recordReader.getKey(), (valueIn) recordReader.getValue());
}
// System.out.println("Thread out." + numberOfThreads.getAndIncrement());
} catch (Exception e) {
// e.printStackTrace();
}
}
}
Note that the parent class Mapper extends Thread.
My second question: if I can do that how can I create instance from it? Now this my call to create:
#Override
public void update(Observable o, Object arg) {
executorService.submit(new MapperFunction((InputBlock) arg));
}
My last question: if all that can happen is there any disadvantage (performance issue) since the application creates more instances from MapperFunction class()?

How do I create a event that fires up every time a new element is added to ArrayList

I want to create an method that fires up every time a new message is added to the groupchat arraylist.
Pseudo code:
public void listenForChange(){
while(true){
if(new message is added to the groupchat){
System.out.println(print the last added message);
}
}
}
What I have tried, but doesn't work:
public class Groupe{
ArrayList<String> groupechat;
int liveChange;
public void listenForChange() {
while(true){
if (groupchat.size() > liveChange){
liveChange= gruppenchat.size();
System.out.println(gruppenchat.get(liveChange-1));
}
}
}
Test class:
public class testGruppen extends Thread {
Gruppe gruppe;
public TestGroup(){
gruppe= new Gruppe("Monday");
}
public void run() {
System.out.println("listen");
gruppe.listenForChange();
}
public static void main(String[] args) {
testGruppen test = new testGruppen();
test.start();
test.gruppe.write("1"); // write just adds a new String to groupchat
test.gruppe.write("2");
test.gruppe.write("3");
test.gruppe.write("4");
}
}
Output: 4 instead of 1\n 2\n 3\n 4\n

What about using decorator:
public static void main(String... args) {
List<Integer> group = new FireEventListDecorator<>(new ArrayList<>());
for (int i = 0; i < 3; i++)
group.add(i);
}
public static class FireEventListDecorator<E> extends AbstractList<E> {
private final List<E> delegate;
public FireEventListDecorator(List<E> delegate) {
this.delegate = delegate;
}
#Override
public void add(int index, E element) {
delegate.add(index, element);
fireEvent(element);
}
#Override
public E get(int index) {
return delegate.get(index);
}
#Override
public int size() {
return delegate.size();
}
private void fireEvent(E element) {
System.out.println("add: " + element);
}
}

To avoid a CPU wasteful while (true) loop with polling, use a call-back method via an observer/listener pattern. One way to do this is to give your class that holds the ArrayList a PropertyChangeSupport instance, allow it to accept listeners, and then in the method that changes the ArrayList, notify listeners.
e.g.,
public class Group {
// property listened to: ADD_TEXT
public static final String ADD_TEXT = "add text";
// the support object
private PropertyChangeSupport support = new PropertyChangeSupport(this);
private List<String> chatText = new ArrayList<>();
public void addPropertyChangeListener(PropertyChangeListener listener) {
support.addPropertyChangeListener(ADD_TEXT, listener);
}
public void addText(String text) {
String oldValue = "";
String newValue = text;
chatText.add(text + "\n");
// notify listeners
support.firePropertyChange(ADD_TEXT, oldValue, newValue);
}
}
And then it can be used like so:
public class TestGroupChat {
public static void main(String[] args) {
final Group group = new Group();
group.addPropertyChangeListener(new GroupListener());
final String[] texts = {"Monday", "Tuesday", "Wednesday", "Thursday", "Friday"};
new Thread(() -> {
for (String text : texts) {
group.addText(text);
try {
TimeUnit.SECONDS.sleep(1);
} catch (InterruptedException e) {}
}
}) .start();
}
private static class GroupListener implements PropertyChangeListener {
#Override
public void propertyChange(PropertyChangeEvent evt) {
// call back method that is called any time the listened-to
// property has been changed
System.out.println("Notification: "+ evt.getNewValue());
}
}
}

You should take a look at LinkedBlockingQueue class.
This class is useful when you want to wake up a thread when a new element is added to a queue. In the example below, everytime you add a new message to the queue, the thread will print the message and wait for the next message.
public class Foo extends Thread {
LinkedBlockingQueue<String> messagesQueue;
public Foo(LinkedBlockingQueue<String> messagesQueue) {
this.messagesQueue = messagesQueue;
}
#Override
public voir run() {
while(true) {
String message = messagesQueue.take();
//The thread will sleep until there is a new message in the queue.
System.out.println(message);
}
}
}

Hystrix: How to capsule multi calls

I am using Hystrix to improve my services. How can I decapsulate the service calls into Hystrix. I know you can create for each call a special hystrix-class, but this would be too much work without using Spring!
I try to describe my problem with pseudocode:
public class HystrixController extends HystrixCommand {
public static void main(String[] args) throws Exception {
HystrixController hystrixController = new HystrixController();
System.out.print(hystrixController.execute());
}
private final ExampleService exampleService;
protected HystrixController() throws Exception {
super(HystrixCommandGroupKey.Factory.asKey("ExampleGroup"));
this.exampleService = new ExampleService();
}
// Call 1
public List getItemsAsList() {
return exampleService.getItemsByContractId(contractID);
}
// Call 2
public List getItemsByName() {
return exampleService.getItemsByName(contractID);
}
// How can I isolate the two calls ? The run() only allows me to use one.
#Override
protected List run() throws Exception {
return getItemsAsList();
}
}
In the example you can see it is only possible to execute only one call. I would like to have something like that:
public class HystrixController extends HystrixCommand {
public static void main(String[] args) throws Exception {
HystrixController hystrixController = new HystrixController();
System.out.print(hystrixController.execute(1));
System.out.print(hystrixController.execute(2));
}
private final ExampleService exampleService;
protected HystrixController() throws Exception {
super(HystrixCommandGroupKey.Factory.asKey("ExampleGroup"));
this.exampleService = new ExampleService();
}
// Call 1
public List getItemsAsList() {
return exampleService.getItemsByContractId(contractID);
}
// Call 2
public List getItemsByName() {
return exampleService.getItemsByName(contractID);
}
// Multi Threads
#Override
protected List run_getItemsAsList() throws Exception {
return getItemsAsList();
}
#Override
protected List run_getItemsByName() throws Exception {
return getItemsByName();
}
}
Thanks you in advance and I am sorry for my broken English