We Keep Coding

Obtain Master public DNS value from AWS EMR Cluster using the Java SDK

I need to obtain the master public DNS value via the Java SDK. The only information that I'll have at the start of the application is the ClusterName which is static.
Thus far I've been able to pull out all the other information that I need excluding this and this, unfortunately is vital for the application to be a success.
This is the code that I'm currently working with:
List<ClusterSummary> summaries = clusters.getClusters();
for (ClusterSummary cs: summaries) {
if (cs.getName().equals("test") && WHITELIST.contains(cs.getStatus().getState())) {
ListInstancesResult instances = emr.listInstances(new ListInstancesRequest().withClusterId(cs.getId()));
clusterHostName = instances.getInstances().get(0).toString();
jobFlowId = cs.getId();
}
}
I've removed the get for PublicIpAddress as wanted the full toString for testing. I should be clear in that this method does give me the DNS that I need but I have no way of differentiating between them.
If my EMR has 4 machines, I don't know which position in the list that Instance will be. For my basic trial I've only got two machines, 1 master and a worker. .get(0) has returned both the values for master and the worker on successive runs.
The information that I'm able to obtain from these is below - my only option that I can see at the moment is to use the 'ReadyDateTime' as an identifier as the master 'should' always be ready first, but this feels hacky and I was hoping on a cleaner solution.
{Id: id,
Ec2InstanceId: id,
PublicDnsName: ec2-54--143.compute-1.amazonaws.com,
PublicIpAddress: 54..143,
PrivateDnsName: ip-10--158.ec2.internal,
PrivateIpAddress: 10..158,
Status: {State: RUNNING,StateChangeReason: {},
Timeline: {CreationDateTime: Tue Feb 21 09:18:08 GMT 2017,
ReadyDateTime: Tue Feb 21 09:25:11 GMT 2017,}},
InstanceGroupId: id,
EbsVolumes: []}
{Id: id,
Ec2InstanceId: id,
PublicDnsName: ec2-54--33.compute-1.amazonaws.com,
PublicIpAddress: 54..33,
PrivateDnsName: ip-10--95.ec2.internal,
PrivateIpAddress: 10..95,
Status: {State: RUNNING,StateChangeReason: {},
Timeline: {CreationDateTime: Tue Feb 21 09:18:08 GMT 2017,
ReadyDateTime: Tue Feb 21 09:22:48 GMT 2017,}},
InstanceGroupId: id
EbsVolumes: []}

Don't use ListInstances. Instead, use DescribeCluster, which returns as one of the fields MasterPublicDnsName.

To expand on what was mentioned by Jonathon:
AmazonEC2Client ec2 = new AmazonEC2Client(cred);
DescribeInstancesResult describeInstancesResult = ec2.describeInstances(new DescribeInstancesRequest().withInstanceIds(clusterInstanceIds));
List<Reservation> reservations = describeInstancesResult.getReservations();
for (Reservation res : reservations) {
for (GroupIdentifier group : res.getGroups()) {
if (group.getGroupName().equals("ElasticMapReduce-master")) { // yaaaaaaaaah, Wahay!
masterDNS = res.getInstances().get(0).getPublicDnsName();
}
}
}

AWSCredentials credentials_profile = null;
credentials_profile = new
DefaultAWSCredentialsProviderChain().getCredentials();
AmazonElasticMapReduceClient emr = new
AmazonElasticMapReduceClient(credentials_profile);
Region euWest1 = Region.getRegion(Regions.US_EAST_1);
emr.setRegion(euWest1);
DescribeClusterFunction fun = new DescribeClusterFunction(emr);
DescribeClusterResult res = fun.apply(new
DescribeClusterRequest().withClusterId(clusterId));
String publicDNSName =res.getCluster().getMasterPublicDnsName();
Below is the working code to get the public DNS name.

how to extract data from restclient.entity.UserEntities in openfire rest API?

I am using this : Openfire Rest API
Now I am fetching users and groups using a java file.
In response I was expecting XML data, but it shows me strange data.
I am new to Java so I don't know how to extract data from this.
My Code is :
package bizrtc;
import api.restclient.RestClient;
import api.restclient.RestApiClient;
import api.restclient.entity.AuthenticationToken;
import api.restclient.entity.UserEntities;
import api.restclient.entity.GroupEntities;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
/**
*
* #author Rajan
*/
public class Bizrtc
{
/**
* #param args the command line arguments
*/
public static void main(String[] args)
{
// TODO code application logic here
AuthenticationToken authenticationToken = new AuthenticationToken("cn1ed9s8yEf5woQV");
// Set Openfire settings (9090 is the port of Openfire Admin Console)
RestApiClient restApiClient = new RestApiClient("192.168.50.50", 9090, authenticationToken);
// restApiClient.getUsers();
UserEntities users = restApiClient.getUsers();
System.out.println("The Groups are as below: "+restApiClient.getGroups());
System.out.println("Now fetching data from openfire Server");
System.out.println("The data is *******************************" + users.toString());
}
}
And when I run the program I get:
Dec 23, 2016 3:58:43 PM org.glassfish.jersey.filter.LoggingFilter log
INFO: 1 * Sending client request on thread main
1 > GET http://192.168.50.50:9090/plugins/restapi/v1/groups
1 > Authorization: cn1ed9s8yEf5woQV
1 > Content-Type: application/xml
Dec 23, 2016 3:58:44 PM org.glassfish.jersey.filter.LoggingFilter log
INFO: 1 * Client response received on thread main
1 < 200
1 < Access-Control-Allow-Credentials: true
1 < Access-Control-Allow-Headers: origin, content-type, accept, authorization
1 < Access-Control-Allow-Methods: GET, POST, PUT, DELETE, OPTIONS, HEAD
1 < Access-Control-Allow-Origin: *
1 < Content-Length: 3664
1 < Content-Type: application/xml
1 < Date: Fri, 23 Dec 2016 09:53:38 GMT
1 < Expires: Thu, 01 Jan 1970 00:00:00 GMT
1 < Set-Cookie: JSESSIONID=1bt213yrejbmfkpyfs53snplm;Path=/;HttpOnly
1 < X-Frame-Options: deny
The Groups are as below: api.restclient.entity.GroupEntities#1165b38
Now fetching data from openfire Server
The data is *******************************api.restclient.entity.UserEntities#4c12331b
How To print this users in XML or better to ARRAY format ??
How do i get the users from this response:api.restclient.entity.GroupEntities#1165b38
Do i have to convert this to string or something like that ?

Take a look at UserEntities Java code:
#XmlRootElement(
name = "users"
)
public class UserEntities {
List<UserEntity> users;
public UserEntities() {
}
public UserEntities(List<UserEntity> users) {
this.users = users;
}
#XmlElement(
name = "user"
)
public List<UserEntity> getUsers() {
return this.users;
}
public void setUsers(List<UserEntity> users) {
this.users = users;
}
}
It's a single POJO class with a list of users and mapped with JAXB annotations. This means you can easily convert your object to XML, JSON or whatever the lib enables.
XML way:
JAXBContext jaxbContext = JAXBContext.newInstance(UserEntities.class);
Marshaller jaxbMarshaller = jaxbContext.createMarshaller();
jaxbMarshaller.setProperty(Marshaller.JAXB_FORMATTED_OUTPUT, true);
StringWriter sw = new StringWriter();
jaxbMarshaller.marshal(users, sw);
String xmlString = sw.toString();
System.out.println(xmlString);
And if you want an array of UserEntity's, you already have its List:
final UserEntity[] arrayUsers = (UserEntity[]) users.getUsers().toArray();
Return example:
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<users>
<user>
<username>d</username>
<name>e</name>
<email>a#a.com</email>
<password>pass</password>
</user>
</users>

Java thread work unexpected

I am reading this topic about java Thread.
And there is an example:
import java.util.Vector;
class Producer extends Thread {
static final int MAXQUEUE = 5;
private Vector messages = new Vector();
#Override
public void run() {
try {
while (true) {
putMessage();
//sleep(5000);
}
} catch (InterruptedException e) {
}
}
private synchronized void putMessage() throws InterruptedException {
while (messages.size() == MAXQUEUE) {
wait();
}
messages.addElement(new java.util.Date().toString());
System.out.println("put message");
notify();
//Later, when the necessary event happens, the thread that is running it calls notify() from a block synchronized on the same object.
}
// Called by Consumer
public synchronized String getMessage() throws InterruptedException {
notify();
while (messages.size() == 0) {
wait();//By executing wait() from a synchronized block, a thread gives up its hold on the lock and goes to sleep.
}
String message = (String) messages.firstElement();
messages.removeElement(message);
return message;
}
}
class Consumer extends Thread {
Producer producer;
Consumer(Producer p) {
producer = p;
}
#Override
public void run() {
try {
while (true) {
String message = producer.getMessage();
System.out.println("Got message: " + message);
//sleep(200);
}
} catch (InterruptedException e) {
e.printStackTrace();
}
}
public static void main(String args[]) {
Producer producer = new Producer();
producer.start();
new Consumer(producer).start();
}
}
And the author said:
A possible output sequence:
Got message: Fri Dec 02 21:37:21 EST 2011
put message
put message
put message
put message
put message
Got message: Fri Dec 02 21:37:21 EST 2011
Got message: Fri Dec 02 21:37:21 EST 2011
Got message: Fri Dec 02 21:37:21 EST 2011
Got message: Fri Dec 02 21:37:21 EST 2011
Got message: Fri Dec 02 21:37:21 EST 2011
put message
put message
put message
put message
put message
Got message: Fri Dec 02 21:37:21 EST 2011
Got message: Fri Dec 02 21:37:21 EST 2011
Got message: Fri Dec 02 21:37:21 EST 2011
But when I run this code I got this result:
put message
put message
put message
put message
put message
put message
Got message: Tue Sep 24 16:44:59 CST 2013
Got message: Tue Sep 24 16:45:00 CST 2013
put message
Got message: Tue Sep 24 16:45:00 CST 2013
put message
Got message: Tue Sep 24 16:45:00 CST 2013
put message
..............
What is the problem?
Any one can explain it for me?

The author's entire point is that the order of tasks between different threads is unpredictable. He printed a possible output sequence, but many, many others are possible.

In addition to the already explained output I must add that the book you are reading doesn't seem to be a very good source to learn from. It teaches to:
extend Thread, a notorious bad practice;
invoke wait and notify on a Thread instance—another known, documented bad practice.
use the wait and notify mechanism in the first place, which has mostly been superseded by much more convenient and simpler java.util.concurrent tools such as CountDownLatch, Semaphore, and Phaser.

Your version of output is correct. Because "Got Message" is not possible without "put message". I mean, if there is no message in queue then how can you retrieve the message. Developer in his example might have given sample output which was not actual code run output but self made just for example.
Remember :
[Count of total "Got message" till particular line] will always be <= [Count of total "put message" till that very line]

The only strange thing is that there are 6 consecutive "put message" which seems to be impossible because the maximum queue size is 5.
But this is because the code sequence
producer.getMessage()
System.out.println("Got message: " + message);
is - of course - not atomic and has been interrupted by a thread switch.
By the way - always use notifyAll() instead of notify.

Bounced mails get Status header

I am trying to read the Status header of a bounced email. This site explains better what I am trying...
The original email is composed by several MultiParts objects, so I am reading it in java code:
private void test(MimeMessage message) throws IOException, MessagingException {
if (message.getContent() != null && message.getContent() instanceof Multipart) {
Multipart content = (Multipart) message.getContent();
for (int i = 0; i < content.getCount(); i++) {
BodyPart bodyPart = content.getBodyPart(i);
Enumeration headers = bodyPart.getAllHeaders();
while(headers.hasMoreElements()){
Header header = (Header) headers.nextElement();
LOGGER.info("Header: " + header.getName() + " value: " + header.getValue());
}
}
}
}
The email part I am analyzing:
Content-Description: Delivery report Content-Type: text/plain;
charset=utf-8 Content-Transfer-Encoding: 7bit
Reporting-MTA: dns; someLink.com
X-Postfix-Queue-ID: EC862F00D0 X-Postfix-Sender: rfc822;
receiver#email.com Arrival-Date: Wed, 7 Aug 2013
13:52:43 +0200 (CEST)
Final-Recipient: rfc822; noexisting#email.com
Original-Recipient: rfc822;noexisting#email.com Action:
failed Status: 5.1.1 Remote-MTA: dns; [somelink.com
Diagnostic-Code: smtp; 550-5.1.1 The email account that you tried to
reach does
not exist. Please try 550-5.1.1 double-checking the recipient's email
address for typos or 550-5.1.1 unnecessary spaces.
In my log file I can see only the 3 first headers:
> Header: Content-Description value: Delivery report
> Header: Content-Type value: text/plain; charset=us-ascii INFO
> Header: Content-Transfer-Encoding value: 7bit
Does anyone know why? How could I get the status header? Thanks

I couldn´t find the Status information in the header, and I will take it from the content. It is not an elegant solution, but at least it works.
If someone finds a better one, please let me know!
Java code:
StringWriter writer = new StringWriter();
IOUtils.copy(bodyPart.getInputStream(), writer);
LOGGER.info("Content inputstream: " + writer.toString());
Logs:
Content inputstream: Reporting-MTA: dns; srvvie-mx3.styria-multi-media.com
X-Postfix-Queue-ID: 2A1A8F00CF X-Postfix-Sender: rfc822;
Arrival-Date: Fri, 9 Aug 2013
11:14:02 +0200 (CEST)
Final-Recipient: rfc822; MAILER-DAEMON#domain.com
Original-Recipient: rfc822;MAILER-DAEMON#domain.com
Action: failed Status: 5.1.1 Remote-MTA: dns;
Diagnostic-Code: smtp; 550 5.1.1 Mailbox
does not exist

Pig error 1070 when doing UDF

I am trying to load up my own UDF in pig. I have made it into a jar using eclipse's export function. I am trying to run it locally so I can make sure it works before I put the jar on HDFS. When running it locally, I get the following error:
ERROR 1070: Could not resolve myudfs.MONTH using imports: [, org.apache.pig.builtin., org.apache.pig.impl.builtin.]
Script
REGISTER myudfs.jar;
--DEFINE MONTH myudfs.MONTH;
A = load 'access_log_Jul95' using PigStorage(' ') as (ip:chararray, dash1:chararray, dash2:chararray, date:chararray, getRequset:chararray, status:int, port:int);
B = FOREACH A GENERATE myudfs.MONTH(date);
DUMP B;
Function
package myudfs;
import java.io.IOException;
import org.apache.pig.EvalFunc;
import org.apache.pig.data.Tuple;
import org.apache.pig.impl.util.WrappedIOException;
#SuppressWarnings("deprecation")
public class HOUR extends EvalFunc<String>
{
public String exec(Tuple input) throws IOException {
if (input == null || input.size() == 0)
return null;
try{
String str = (String)input.get(0);
return str.substring(1, 3);
}catch(Exception e){
throw WrappedIOException.wrap("Caught exception processing input row ", e);
}
}
}
Working Directory
1.pig 2.pig bin myudfs.jar
pig.jar pig-withouthadoop.jar src/
Running command
pig -x local 2.pig
Structure of jar
0 Thu May 02 12:16:26 MDT 2013 META-INF/
68 Thu May 02 12:16:26 MDT 2013 META-INF/MANIFEST.MF
0 Thu May 02 12:05:50 MDT 2013 myudfs/
573 Thu May 02 12:15:10 MDT 2013 myudfs/HOUR.java
I am really close to start chucking monitors, so I am just looking for some help and direction. Let me know what could be wrong.

Your UDF class name is called HOUR
So shouldn't your pig latin be this?
B = FOREACH A GENERATE myudfs.HOUR(date);