I am trying to remotely profile alfresco running on a 64-bit linux server running a 1.8 JVM and Apache Tomcat 7.xx from my testing code but can't figure out how to programatically trigger snapshots.
What I want to do is to connect to the remote server, start profiling, and save a snapshot of that server's performance onto my local machine from my testing code which is written in Java.
I've already installed JProfiler 9.2 onto the linux server and can connect and take snapshots via the JProfiler GUI. The server also requires an SSH connection for security. I'd like to do this from my code, similar to how Controller.saveSnapshot(file) works for local JVMs.
Is this possible?
I know I can set triggers and have the remote profiler save snapshots on the server, but this isn't what I want to.
Additionally, I looked into using the command line controller but was unable to get it to connect to the server even with the correct arguments in the remote VM options.
I also tried to use ConnectionFactor.createRemoteConnection(), but don't see an argument that allows for a password to be input, so it fails.
You can access the JProfiler MBean programmatically. Below is an example on how to do that. I would run such a program on the remote machine and start it via SSH, because JMX connnections are difficult to tunnel through SSH.
import javax.management.MBeanServerConnection;
import javax.management.ObjectName;
import javax.management.remote.JMXConnector;
import javax.management.remote.JMXConnectorFactory;
import javax.management.remote.JMXServiceURL;
import java.util.Collections;
import java.util.HashMap;
import java.util.Map;
// Shows how to connect to the the JProfiler MBean programatically
// The profiled process has to be in offline profiling mode and the JMX server
// has to be started by passing -Djprofiler.jmxServerPort=[port] to the profiled JVM.
// This will not work in nowait mode because the MBean is not registered in that case.
public class MBeanProgrammaticAccessExample {
public static void main(String[] args) throws Exception {
if (args.length == 0) {
System.out.println("Specify the port as an argument that was passed to " +
"the profiled JVM with the VM parameter " +
"-Djprofiler.jmxServerPort=[port]");
}
String port = args[0];
// In this case the connection is made to a process on localhost, but it could
// be on a remote system as well. Note that the connection is made via JMX which
// does not work well with firewalls
System.out.println("Connecting to localhost:" + port);
JMXServiceURL jmxUrl = new JMXServiceURL(
"service:jmx:rmi:///jndi/rmi://localhost:" + port + "/jmxrmi");
JMXConnector connector = JMXConnectorFactory.newJMXConnector(jmxUrl,
Collections.<String, Object>emptyMap());
Map<String, Object> env = new HashMap<>();
// If you have protected the JMX server with a JMX password file by passing
// -Djprofiler.jmxPasswordFile=[file] to the profiled JVM, you can specify
// the password like this:
//env.put(JMXConnector.CREDENTIALS, new String[] {"username", "password"});
connector.connect(env);
MBeanServerConnection connection = connector.getMBeanServerConnection();
ObjectName objectName = new ObjectName(
"com.jprofiler.api.agent.mbean:type=RemoteController");
if (!connection.isRegistered(objectName)) {
throw new RuntimeException("JProfiler MBean not found.");
}
RemoteControllerMBean mbeanProxy = JMX.newMBeanProxy(connection,
objectName, RemoteControllerMBean.class, true);
// You can look up all available operations in the javadoc of
// com.jprofiler.api.agent.mbean.RemoteControllerMBean
System.out.println("Recording CPU data for 5 seconds ....");
mbeanProxy.startCPURecording(true);
// If you do not want a dependency on the JProfiler classes
// you can make the above call like this:
//connection.invoke(objectName, "startCPURecording", new Object[] {true},
// new String[] {Boolean.TYPE.getName()});
Thread.sleep(5000);
System.out.println("Saving snapshot to the working directory " +
"of the profiled JVM ....");
mbeanProxy.saveSnapshot("snapshot.jps");
connector.close();
System.out.println("Success");
}
}
Related
According to this tutorial, I am able to upload files on the website while running my tests locally and on the remote server.
As in the tutorial is:
For those of you doing this locally, all you need to do is use the
sendKeys command to type the local path of the file in any file field.
This works like a charm in all drivers. When moving this test to a
remote server (such as, for example, our Selenium 2 Cloud), all you
need to do is use the setFileDetector method to let WebDriver know
that you’re uploading files from your local computer to a remote
server instead of just typing a path.
on the remote server I have to use:
driver.setFileDetector(new LocalFileDetector());
...
upload.sendKeys("/Path/to/image.jpg");
and local just:
upload.sendKeys("/Path/to/image.jpg");
And this all works fine. Only the problem is, that there is no information how to determine if my tests are running local or on the remote server.
I have tried to determine instance of the webDriver:
WebDriver proxiedWebDriver = ((WebDriverFacade) getDriver()).getProxiedDriver();
if (proxiedWebDriver instanceof RemoteWebDriver) {
((RemoteWebDriver)proxiedWebDriver).setFileDetector(new LocalFileDetector());
}
but it seems like both(local and remote) cases are using RemoteWebDriver while running, because in every case I'm passing if condition.
How can I determine if my tests are running local or remote?
To get the address of the remote server you can use HttpCommandExecutor like this:
HttpCommandExecutor ce = (HttpCommandExecutor) ((RemoteWebDriver)driver).getCommandExecutor();
String remoteAddress = ce.getAddressOfRemoteServer().toString();
String localAddress = null;
try (Socket socket = new Socket()) {
socket.connect(new InetSocketAddress("google.com", 80));
localAddress = socket.getLocalAddress().getHostAddress();
} catch (IOException e) {
e.printStackTrace();
}
if (remoteAddress.contains("localhost") || remoteAddress.contains(localAddress)) System.out.println("Local machine");
else System.out.println("Remote machine");
The above code gets the Remote Server address (HUB) and compares it with your public IP address. It should give you the information if you are running local or remote server
I have been using Cassandra and its C++ driver to write APIs to insert and fetch data for some time now. However now that I've created my cluster, I want to develop monitoring tools for my cluster.
I want to build an application(preferably in C++ and I don't want to use a 3rd party application), which will store Cluster management specific attributes like memory utilization of each node in the cluster, latency of each operation, space occupied by each table on each node etc. I read about 'Metrics in Cassandra(https://cassandra.apache.org/doc/latest/operating/metrics.html) but I don't know how exactly to use them in building my application as I've not worked on Java before(excuse me for that!).
Can such application be built using C++? If it's a lot of work in C++, then it will be highly beneficial if you can share some Java code where these Cassandra Metrics have been used to monitor a Cassandra Cluster.
OS: RHEL
Cassandra version: 3.11.2
Cassandra 3.x uses the drop wizard api as you alluded to. If you can add the Jolokia jars to your deployment server(s) this will allow you to access the java jmx data using a simple http request. Jolokia exposes all the mbeans from java over a rest api.
It seems, there are no any c++ libs for JMX, but in Java it is pretty easy to get JMX metrics, all of you need is standard jdk. The following code demonstrates how to connect to cassandra node and get 'down' node count.
import java.util.HashMap;
import java.util.Map;
import javax.management.MBeanServerConnection;
import javax.management.ObjectName;
import javax.management.remote.JMXConnector;
import javax.management.remote.JMXConnectorFactory;
import javax.management.remote.JMXServiceURL;
public class Main {
public static void main( String[] args ) throws Exception {
String node = args[0];
String port = args[1];
String username = args[2];
String password = args[3];
JMXServiceURL url = new JMXServiceURL( "service:jmx:rmi:///jndi/rmi://" + node + ":" + port + "/jmxrmi" );
String[] credentials = {username, password};
Map<String, String[]> environment = new HashMap<>();
environment.put( JMXConnector.CREDENTIALS, credentials );
JMXConnector jmxConnector = JMXConnectorFactory.connect( url, environment );
MBeanServerConnection mbsc = jmxConnector.getMBeanServerConnection();//Get metrics bean
ObjectName oName = new ObjectName( "org.apache.cassandra.net:type=FailureDetector" );//create JMX object name
int downNodes = (int) mbsc.getAttribute( oName, "DownEndpointCount" ); //get number of unavailable nodes
System.out.println("Down node count: " + downNodes);
}
}
More details about jmx you can find in Oracle documentation
To obtain JMX object names and attribute names you can use jconsole tool, which is shipped together with jdk:
I'm working on a personal project for school where I have to user RMI to communicate between server and client.
Project info
The goal of my project is to retrieve stock info (from NYSE) for each day on the server at a specific time (after NYSE is closed). Each stock object is saved in a database. The information is retrieved over http and has nothing to do with RMI.
For the client it is also possible to fetch the stocks. When a user wants to fetch the stock object for the current day, it is directly fetched from the 3th party service. When a user, for example, wants to fetch Google's stock from last month, it is requested on the server over RMI. The server will the look for the stock object in the database and retrieve a Stock object and send it to the client.
Problem
When I start the client application, I have to login. The client will create a User object containing the username and password.
When I press the login button, it will take around 2 minutes before the main screen will be shown.
Below the source code where I setup the RMI connection.
Server (main.java)
public static void main(String[] args) throws UnknownHostException {
InetAddress IP= InetAddress.getLocalHost();
System.out.println("IP of my system is := "+IP.getHostAddress());
if(args.length == 1 && args[0].toLowerCase().equals("local")) {
System.out.println("Running on localhost");
System.setProperty("java.rmi.server.hostname", IP.getHostAddress());
} else {
System.out.println("rmi hostname is set to 37.97.223.70");
System.setProperty("java.rmi.server.hostname", "37.97.223.70");
}
try {
Registry reg = LocateRegistry.createRegistry(1099);
StockAppServer server = StockAppServer.getInstance();
reg.rebind("StockApp", server);
System.out.println("StockApp bound for StockAppServer object.");
} catch (RemoteException e) {
e.printStackTrace();
}
}
Based on the arguments that are passed to the application when it starts, I set the RMI hostname to my current IP address, or to the remote server address. The remote server address is a static IP, so this won't change.
Server (StockAppServer.java)
This class implements the interfaces that is used by the client to call methods on the server. So this class extends UnicastRemoteObject. When I start the server, registerStockTask() will be called. This method will fetch the ticker symbols (What are ticker symbols?) and then schedule a task to fetch all stock objects at a specific time.
private static StockAppServer _instance;
private List<User> loggedInUsers;
private List<Group> activeGroups;
private List<Notification> registeredNotifications;
private StockAppServer() throws IOException {
_instance = this;
this.loggedInUsers = new ArrayList<>();
this.activeGroups = new ArrayList<>();
this.registeredNotifications = new ArrayList<>();
this.registerStockTask();
clearActiveGroups();
checkForCompletedNotifications();
// Start the restful framework to allow incoming connections from the NodeJS server to manage new notification
Router.getInstance();
}
public static StockAppServer getInstance() {
try{
return _instance == null ? new StockAppServer() : _instance;
} catch (IOException e) {
e.printStackTrace();
}
return null;
}
Client (main.java)
public static void main(String[] arguments) throws Exception {
args = arguments;
Application.launch();
}
#Override
public void start(Stage primaryStage) throws Exception {
InetAddress IP= InetAddress.getLocalHost();
System.out.println("IP of my system is := "+IP.getHostAddress());
if(args.length == 1 && args[0].toLowerCase().equals("local")) {
// Program started with local command, expect that server is running on local host
reg = LocateRegistry.getRegistry(IP.getHostAddress(), 1099);
System.out.println("Attempting to connect to RMI server over 127.0.0.1");
} else {
// Program started without additional commands. Except that "the server" is available;
reg = LocateRegistry.getRegistry("37.97.223.70", 1099);
System.out.println("Attempting to connect to RMI server over 37.97.223.70");
}
try {
StockApp.getInstance().setServerInterfaces((IStockSend) reg.lookup("StockApp"), (IUserHandling) reg.lookup("StockApp"));
} catch(RemoteException e) {
AlertMessage.showException("Unable to connect to server.", e);
} catch (NotBoundException e) {
AlertMessage.showException("No server has been found with the name \"StockApp\" on the remote host.\nPlease try again later", e);
}
LoginController.showMenu();
//FileNotFoundException e = new FileNotFoundException("Couldn't find file blabla.txt");
//AlertMessage.showException("Something went wrong. Please try again later.", e);
}
How I tried to solve my problem
When I test my applications local, there is no problem. The login method will be finished within a few milliseconds and I will be represented the main screen.
I started by turning of my firewall on my macbook. No result, login method still takes around 2 seconds.
I turned off the firewall om my Ubuntu server. No result, both firewalls on server and macbook are turned off. Login method still takes around 2 seconds.
On the server runs (thanks to jenkins) another (unrelated) program. This program uses sockets instead of RMI. When this program is not running, the login method still takes around 2 minutes.
In StockAppServer.java, I called the following method:
super(1099);
This has the same outcome as the above steps I took.
I don't know what else I can try to solve my problem.
I tried to give as much code as possible for the RMI part. I you need any other source code, just ask and I can update this question. Also, the source code is available via github: https://github.com/juleskreutzer/GSO-Maatwerk. Make sure to run the program with -remote param.
Update 1 (9-1-2017)
As yanys requested in the comments, I should run the following command:
dscacheutil -q host -a name localhost
this returns the following output:
Mac:
name: localhost
ip_address: 127.0.0.1
Ubuntu:
dscacheutil: command not found
Update 2 (9-1-2017)
I checked with the provider of my VPS where I run the java server on. On their side everything should be OK. According to them, it shouldn't be a dns problem. After some research, I found out that RMI uses both DNS and reverse DNS. It this case, reverse DNS was the issue. Please see my answer on how I solved my problem.
As EJP pointed out in the comments on the question, it was an DNS problem.
I contacted the support of my hosting provider to see if I had some wrong settings. They helped me a lot in solving this problem.
First we tested the speed of my VPS, this is around 1000mbit download and upload speed. After we checked this, they said there was nothing wrong on their side.
After doing some research, I found out that RMI uses both DNS and Reverse DNS. The problem was that I didn't setup the reverse DNS on my server. I already have a domain name to use for reverse DNS.
I than did the following:
Create a A-record on my website that points to the IP address of the server. I named it vps.mydomain.com
Add the reverse DNS in the control panel of my server
Change the hostname of my server to vps.mydomain.com*
*My server runs Ubuntu 16.04, on ubuntu machines with systemd, you can use the command
sudo hostnamectl set-hostname new-name
to change the hostname
I have set up a local proxy server for request logging but my java code ignores it and connects directly (Windows XP, JDK 1.7). Web browsers work with it. So I wrote test code for discussion that seems to connect directly even if a (bogus) proxy is specified. With the bogus proxy, I would expect connection failure but the code succeeds, connecting directly:
System.setProperty("http.proxyHost", "localhost");
System.setProperty("http.proxyPort", "12345");
System.setProperty("http.nonProxyHosts", "noNonProxyHost.com");
URL url = new URL("http://docs.oracle.com/javase/7/docs/technotes/guides/net/proxies.html");
InputStream in = url.openStream();
System.out.println("Connection via bogus proxy succeeded");
The code is run as standalone Java, no Maven, no applet, no container. I have a direct internet connection.
In your case using java.net.URL(), if the proxy server cannot be reached at http.proxyHost and http.proxyPort then it simply falls back and tries to do a direct connect. If that succeeds, you'll see no exception thrown which is why your code works without error. You should see a pause while it tries to find the proxy though.
This sample code below happily fetches the URL and displays it, without error, even when run with bogus proxy settings. -Dhttp.proxyHost=bogus -Dhttp.proxyPort=2345 but will talk to my local proxy localhost port 8888 if set correctly
import java.io.*;
import java.net.URL;
import java.util.*;
public class URLClient {
private static String sUrl = "http://www.apache.org/";
public static void main(String[] args) {
try {
URL url = new URL(sUrl);
InputStream is = url.openStream();
java.util.Scanner s = new java.util.Scanner(is).useDelimiter("\\A");
String output = s.hasNext() ? s.next() : "";
System.out.println(output);
} catch(Throwable e) {
System.err.println("exception");
}
}
}
The problem I was originally having with http.proxyHost and http.proxyPort being ignored (Google led me to your question) was that those settings are completely ignored by apache.commons.httpClient because it uses its own sockets, as described here.
http://cephas.net/blog/2007/11/14/java-commons-http-client-and-http-proxies/
I have faced a similar problem recently. First of all, one part of the above answer from Daemon42 explains pretty well, why the bogus proxy server didn't lead to a failure of the program:
if the proxy server cannot be reached at http.proxyHost and http.proxyPort then it simply falls back and tries to do a direct connect. If that succeeds, you'll see no exception thrown which is why your code works without error. You should see a pause while it tries to find the proxy though.
Still, your actual question was, why the proxy server configured via the operating system is not used by the Java application. As stated in the Oracle documentation (https://docs.oracle.com/javase/8/docs/technotes/guides/net/proxies.html), the system proxy settings are not evaluated by Java by default. To do so, you have to set the value of the system property "java.net.useSystemProxies" to the value "true".
You can set that system property on the command line, or you can edit the JRE installation file jre/lib/net.properties, that way you have to change it only once on a given system.
I am trying to use Sqoop 2 to import data from a MySQL database to HDFS, basically following the instructions here. However, the Sqoop server is unable to make a connection to the MySQL database due to appropriate drivers not found.
Setup:
Here is some background on my setup:
Hadoop cluster: I have a three machine Hadoop cluster running CDH 4.4.0. Sqoop 2 was configured through the Cloudera Manager, and is running on the same machine as the Namenode.
I am developing on a Windows machine which is also where my MySQL database lives. The Hadoop cluster is a set of three Ubuntu Server machines.
MySQL database: I have a MySQL database running on my Windows machine, and I have checked that the MySQL database can be accessed from each of the machines in my Hadoop cluster.
Client application: My client application is an Eclipse project on my Windows machine which basically opens up a Sqoop client corresponding to a Sqoop server (I have verified that the Sqoop server and client are running on my Namenode).
Here is the main class of my client application.
package com.fc.SqoopImport;
import java.util.List;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.*;
import org.apache.sqoop.client.*;
import org.apache.sqoop.*;
import org.apache.sqoop.common.*;
import org.apache.sqoop.model.*;
import org.apache.sqoop.validation.Status;
import com.mysql.jdbc.*;
public class SqoopImport {
// utlity function to cycle through the connector and framework forms for errors
private static void printMessage(List<MForm> formList) {
for(MForm form : formList) {
List<MInput<?>> inputlist = form.getInputs();
if (form.getValidationMessage() != null) {
System.out.println("Form message: " + form.getValidationMessage());
}
for (MInput minput : inputlist) {
if (minput.getValidationStatus() == Status.ACCEPTABLE) {
System.out.println("Warning:" + minput.getValidationMessage());
} else if (minput.getValidationStatus() == Status.UNACCEPTABLE) {
System.out.println("Error:" + minput.getValidationMessage());
}
}
}
}
public static void main(String[] args) throws Exception {
String driver = "com.mysql.jdbc.Driver";
Class.forName(driver);
// location of the server running Sqoop 2 server
String urlSqoop2Server = "http://fc-01.fc.com:12000/sqoop/";
SqoopClient clientSqoop2 = new SqoopClient(urlSqoop2Server);
// dummy connection object
MConnection sqoopConnSAP = clientSqoop2.newConnection(1);
MConnectionForms sqoopConnSAPFrameworkForm = sqoopConnSAP.getFrameworkPart();
MConnectionForms sqoopConnSAPConnForm = sqoopConnSAP.getConnectorPart();
sqoopConnSAP.setName("SqoopConnSAP");
// Set the values for the connection form
sqoopConnSAPConnForm.getStringInput("connection.connectionString").setValue("jdbc:mysql://192.168.31.172:3306/dbsap");
sqoopConnSAPConnForm.getStringInput("connection.jdbcDriver").setValue("com.mysql.jdbc.Driver");
sqoopConnSAPConnForm.getStringInput("connection.username").setValue("root");
sqoopConnSAPConnForm.getStringInput("connection.password").setValue("1234");
sqoopConnSAPFrameworkForm.getIntegerInput("security.maxConnections").setValue(10);
Status statusConnSAP = clientSqoop2.createConnection(sqoopConnSAP);
if(statusConnSAP.canProceed()) {
System.out.println("Created. New connection ID: " + sqoopConnSAP.getPersistenceId());
} else {
System.out.println("Check for status and forms errors.");
printMessage(sqoopConnSAP.getConnectorPart().getForms());
printMessage(sqoopConnSAP.getFrameworkPart().getForms());
}
}
}
Error:
Running this project gives the following error:
Check for status and forms errors.
Form message: Can't connect to the database with given credentials: No suitable driver found for jdbc:mysql:192.168.31.172:3306/dbsap
Error:Can't load specified driver
Diagnosis:
The appropriate JDBC drivers (mysql-connector-java-5.1.26-bin.jar) is part of my Eclipse project, and for good measure, I have added this to the sqoop2 lib folder
/opt/cloudera/parcels/CDH-4.4.0-1.cdh4.4.0.p0.39/lib/sqoop2/client-lib
as well. However, this is the part I am not sure of, since the CDH4 documentation says]1 that in case Sqoop was installed using Cloudera Manager, the location of the appropriate JDBC driver should be added to HADOOP_CLASSPATH. So, I did
export HADOOP_CLASSPATH=/usr/lib/jdbcJars:HADOOP_CLASSPATH;
on my Hadoop Namenode, so that an echo $HADOOP_CLASSPATH gives /usr/lib/jdbcJars. Again, I am not entirely sure of the utility of this since my client application is not being developed on the Hadoop cluster.
The last thing that I have not tried yet is creating a new /usr/lib/sqoop/lib directory and adding the JDBC driver there.
Any help figuring this out would be appreciated.
Never ever alter content of parcel directory (/opt/cloudera/parcels/*). There are always different ways how to configure components. For example based on the official documentation, you need to copy the MySQL JDBC driver into /var/lib/sqoop2 directory on the node where you are running Sqoop2 server.
put the mysql-jdbc-driver into the dir:
/usr/lib/sqoop2/webapps/sqoop/WEB-INF/lib/mysql-connector-java-5.1.25.jar
and restart the sqoop2 server