I am trying to use Sqoop 2 to import data from a MySQL database to HDFS, basically following the instructions here. However, the Sqoop server is unable to make a connection to the MySQL database due to appropriate drivers not found.
Setup:
Here is some background on my setup:
Hadoop cluster: I have a three machine Hadoop cluster running CDH 4.4.0. Sqoop 2 was configured through the Cloudera Manager, and is running on the same machine as the Namenode.
I am developing on a Windows machine which is also where my MySQL database lives. The Hadoop cluster is a set of three Ubuntu Server machines.
MySQL database: I have a MySQL database running on my Windows machine, and I have checked that the MySQL database can be accessed from each of the machines in my Hadoop cluster.
Client application: My client application is an Eclipse project on my Windows machine which basically opens up a Sqoop client corresponding to a Sqoop server (I have verified that the Sqoop server and client are running on my Namenode).
Here is the main class of my client application.
package com.fc.SqoopImport;
import java.util.List;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.*;
import org.apache.sqoop.client.*;
import org.apache.sqoop.*;
import org.apache.sqoop.common.*;
import org.apache.sqoop.model.*;
import org.apache.sqoop.validation.Status;
import com.mysql.jdbc.*;
public class SqoopImport {
// utlity function to cycle through the connector and framework forms for errors
private static void printMessage(List<MForm> formList) {
for(MForm form : formList) {
List<MInput<?>> inputlist = form.getInputs();
if (form.getValidationMessage() != null) {
System.out.println("Form message: " + form.getValidationMessage());
}
for (MInput minput : inputlist) {
if (minput.getValidationStatus() == Status.ACCEPTABLE) {
System.out.println("Warning:" + minput.getValidationMessage());
} else if (minput.getValidationStatus() == Status.UNACCEPTABLE) {
System.out.println("Error:" + minput.getValidationMessage());
}
}
}
}
public static void main(String[] args) throws Exception {
String driver = "com.mysql.jdbc.Driver";
Class.forName(driver);
// location of the server running Sqoop 2 server
String urlSqoop2Server = "http://fc-01.fc.com:12000/sqoop/";
SqoopClient clientSqoop2 = new SqoopClient(urlSqoop2Server);
// dummy connection object
MConnection sqoopConnSAP = clientSqoop2.newConnection(1);
MConnectionForms sqoopConnSAPFrameworkForm = sqoopConnSAP.getFrameworkPart();
MConnectionForms sqoopConnSAPConnForm = sqoopConnSAP.getConnectorPart();
sqoopConnSAP.setName("SqoopConnSAP");
// Set the values for the connection form
sqoopConnSAPConnForm.getStringInput("connection.connectionString").setValue("jdbc:mysql://192.168.31.172:3306/dbsap");
sqoopConnSAPConnForm.getStringInput("connection.jdbcDriver").setValue("com.mysql.jdbc.Driver");
sqoopConnSAPConnForm.getStringInput("connection.username").setValue("root");
sqoopConnSAPConnForm.getStringInput("connection.password").setValue("1234");
sqoopConnSAPFrameworkForm.getIntegerInput("security.maxConnections").setValue(10);
Status statusConnSAP = clientSqoop2.createConnection(sqoopConnSAP);
if(statusConnSAP.canProceed()) {
System.out.println("Created. New connection ID: " + sqoopConnSAP.getPersistenceId());
} else {
System.out.println("Check for status and forms errors.");
printMessage(sqoopConnSAP.getConnectorPart().getForms());
printMessage(sqoopConnSAP.getFrameworkPart().getForms());
}
}
}
Error:
Running this project gives the following error:
Check for status and forms errors.
Form message: Can't connect to the database with given credentials: No suitable driver found for jdbc:mysql:192.168.31.172:3306/dbsap
Error:Can't load specified driver
Diagnosis:
The appropriate JDBC drivers (mysql-connector-java-5.1.26-bin.jar) is part of my Eclipse project, and for good measure, I have added this to the sqoop2 lib folder
/opt/cloudera/parcels/CDH-4.4.0-1.cdh4.4.0.p0.39/lib/sqoop2/client-lib
as well. However, this is the part I am not sure of, since the CDH4 documentation says]1 that in case Sqoop was installed using Cloudera Manager, the location of the appropriate JDBC driver should be added to HADOOP_CLASSPATH. So, I did
export HADOOP_CLASSPATH=/usr/lib/jdbcJars:HADOOP_CLASSPATH;
on my Hadoop Namenode, so that an echo $HADOOP_CLASSPATH gives /usr/lib/jdbcJars. Again, I am not entirely sure of the utility of this since my client application is not being developed on the Hadoop cluster.
The last thing that I have not tried yet is creating a new /usr/lib/sqoop/lib directory and adding the JDBC driver there.
Any help figuring this out would be appreciated.
Never ever alter content of parcel directory (/opt/cloudera/parcels/*). There are always different ways how to configure components. For example based on the official documentation, you need to copy the MySQL JDBC driver into /var/lib/sqoop2 directory on the node where you are running Sqoop2 server.
put the mysql-jdbc-driver into the dir:
/usr/lib/sqoop2/webapps/sqoop/WEB-INF/lib/mysql-connector-java-5.1.25.jar
and restart the sqoop2 server
Related
A few classmates and I are creating a Java project which requires a database. I have created a connection in MySQL and connected it to my Java project successfully using the following Connect class:
package com.example.javaworkoutgame.Model;
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.SQLException;
public class Connect {
static Connection con;
public Connect() {
connect();
}
// attempt to connect to MySQL database
public static void connect() {
try {
Class.forName("com.mysql.cj.jdbc.Driver");
System.out.println("Driver Loaded Successfully");
con = DriverManager.getConnection("jdbc:mysql://127.0.0.1:3306/lab3", "root",
"**********"); // not the actual password
System.out.println("Successful Connection");
} catch (ClassNotFoundException cnfe) {
System.err.println(cnfe);
} catch (SQLException sqle) {
System.err.println(sqle);
}
}
}
This code runs properly on my machine.
I committed and pushed the code to Bitbucket so my partners could access it. However, when they run the code on their computers, they get the following error message:
java.sql.SQLException: Access denied for user 'root'#'localhost' (using password: YES)
Is there something I need to change in MySQL workbench in order for other people to be able to access the database? I could not find any information on this.
The only thing I was able to try was found at this thread:
java.sql.SQLException: Access denied for user 'root'#'localhost' (using password: YES)
I opened a new .sql file and tried running the command:
GRANT ALL PRIVILEGES ON . TO 'root'#'localhost' IDENTIFIED BY '%password%' WITH GRANT OPTION;
(I replaced '%password%' with the actual password)
When I tried that I got the following error message:
Error Code: 1064. You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'IDENTIFIED BY '*********' WITH GRANT OPTION'
No, and you need to stop this line of thought and do some research first.
Your current configuration says that the mysql server is on the very same physical machine that the code is running on. You installed mysql on your dev machine, your friends need to install it on theirs, and each has their own unique database (nothing is shared).
You could, instead, take your mysql server, open it up to the world (which, for virtually all ways internet is made available in residential connections, requires messing with your router to 'open a port').
But then you have an open mysql server, and the username and password are on a public bitbucket site.
It also requires either a permanent IP (which few residential internet providers offer) or a dyndns service. More generally, hosting open MySQL servers that see lots of traffic gets your internet shut down, for good reason. You'd end up hosting a whole bunch of hackers. All hardware in your network will be p0wned and turned into bot nets. Hence, very very bad idea.
Good ways to solve this problem:
Everybody installs their own MySQL server. This is sensible; you're writing code and bound to make mistakes, it'd be real bad if all code you write is first-run and tested on live data. You don't want one of your friends to wipe your database. If you need some initial data to test with, set it up properly, and read up on how to make an SQL dump. With such a dump file you can reset any mysql server to that exact state - and that'd be how you and your friends develop: Set up the DB to be in that known state, write some code, and if you ruin the db by doing so, no problem. Just reset it again.
Set up a VPN between your friends. NOW you can share the IP your system has within the VPN (it'll be 10., 172.16., 192.168.* - if it's 127.0.0.1, it's localhost, i.e. everybody needs to install mysql on their own and nothing is shared, and if it's anything else, you're opening it to the world, which you don't want to do). Do not put the VPN username/password info anywhere in that bitbucket. And you need to trust your friends.
You should have a properties type file so that each person who is going to interact with the code has their local data without the need to replicate yours, in the same way you can have different values in the properties for test or production environments.
example of a property file:
system.properties
#BD
db.driver=com.mysql.cj.jdbc.Driver
db.user=user
db.pass=password
db.server=server_IP
db.port= port_IP
db.db = DB
Then you should have a procedure to read from java the properties inside the file
Utils.java
package com.example.javaworkoutgame.util;
import java.io.File;
import java.io.FileInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.util.Properties;
public final class Utils {
public static Properties getProperties() {
String path = String.format("PATH to your properties FILE/system.properties",
System.getProperty("user.dir"));
Properties properties = new Properties();
try (InputStream is = new FileInputStream(new File(path))) {
properties.load(is);
} catch (IOException e) {
throw new IllegalStateException(e);
}
return properties;
}
}
And finally you make a call to the function that gets the properties from your connection class
Connect.java
package com.example.javaworkoutgame.Model;
import com.example.javaworkoutgame.util.Utils;
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.SQLException;
public class Connect {
Properties properties = Utils.getProperties();
static Connection con;
public Connect() {
connect();
}
// attempt to connect to MySQL database
public static void connect() {
try {
String driver = properties.getProperty("db.driver");
String ip = properties.getProperty("db.ip");
String port = properties.getProperty("db.port");
String db = properties.getProperty("db.db");
String user = properties.getProperty("db.user");
String pass = properties.getProperty("db.pass"):
Class.forName(driver);
System.out.println("Driver Loaded Successfully");
con = DriverManager.getConnection("jdbc:mysql://"+ip+":"+port+"/"+db, user,
pass);
System.out.println("Successful Connection");
} catch (ClassNotFoundException cnfe) {
System.err.println(cnfe);
} catch (SQLException sqle) {
System.err.println(sqle);
}
}
}
About the MYSQL error, if your partners do not have a local mysql environment with the same values as you, they will experience the error you describe, since your configuration is a local configuration, if you need your partners to connect to your pc, you must open the ports of mysql and give them your public IP (not recommended)
I hope this answer helps you!
i am trying to implement java RMI client server connection, i made my server in NetBeans IDE and client class made in oracle database as java store procedure and use this via oracle function in my PLSQL block, everything is working fine but I stuck in little issue below:
I made this java store procedure in SYSTEM user and give dbms_java.grant_permission('SYSTEM','java.net.SocketPermission','192.168.43.25:*','connect,resolve') to SYSTEM user via SYSDBA, "*" is used for all available ports. I get perfect response from my server but till specific session expiry.
Whenever I logout from SYSTEM user and login back again, its again throws me an exception the Permission ("java.net.SocketPermission" "192.168.43.25:56792" "connect,resolve") has not been granted please guide me how can I get rid from this repeated task? because in development environment I'll manage it but how can deploy it on production with this issue. My Environment is Oracle DB 18c XE, Thank You!
Following is my java store procedure:
create or replace and compile java source named rmi as
package rmi;
import java.rmi.Remote;
import java.rmi.RemoteException;
import java.rmi.Naming;
import java.security.Policy;
public class RmiClient {
public RmiClient(){
}
public static String getRequestFromClient(){
Policy.setPolicy(new MyPolicy());
String result = "";
try {
IServer server = getServer();
result = server.getRequestFromClient();
} catch (Exception ex) {
result = ex.getMessage().toString();
}
return result;
}
private static IServer getServer() throws Exception {
Parameter configurationParameters = new Parameter();
IServer server = (IServer) Naming.lookup( configurationParameters.objectName);
return server;
}
}
Granting permissions with DBMS_JAVA is a little different than a typical Oracle grant. After granting a permission using DBMS_JAVA.GRANT_PERMISSION, you will need to COMMIT at some point after the grant as been executed for it to take affect permanently. Notice how there is a COMMIT after the DBMS_JAVA call in the Oracle documentation.
I am using a Java application that connects to DB2 over JDBC. It creates the connection such that the application name (sysibmadm.applications -> APPL_NAME) remains the JDBC default "db2jcc_application". Is there a way to externally set the APPL_NAME to a desired value? (e.g. in Oracle, I can pass -Doracle.jdbc.v$session.program=MyApplName to have a similar effect).
I want to do this so that I can clearly identify all the connections coming into my DB2. Currently all the java applications show up as "db2jcc_application".
I wrote a small piece of code to illustrate this ...
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.SQLException;
import java.util.Properties;
public class DB2ApplName {
public static void main(String[] args) throws ClassNotFoundException, SQLException, InterruptedException {
Class.forName("com.ibm.db2.jcc.DB2Driver");
Properties connProps = new Properties();
connProps.put("user", "<my db2 user>");
connProps.put("password", "<my db2 password>");
// Create a connection **WITHOUT** Application Name
Connection connWithoutApplName = DriverManager.getConnection("jdbc:db2://<ip>:<port>/<dbname>",
connProps);
// Add Application Name to the properties
connProps.put("clientProgramName", "MyApplName");
// Create a connection **WITH** Application Name
Connection connWithApplName = DriverManager.getConnection("jdbc:db2://<ip>:<port>/<dbname>",
connProps);
System.out.println("Sleeping for 60 seconds - check the connections.");
Thread.sleep(60000L); // During this time I will run query on DB2 to see connection details.
connWithoutApplName.close();
connWithApplName.close();
}
}
This code creates two connections, one wihtout ApplName set and one with ApplName set.
When the code is in Thread.sleep, I query the db ...
select appl_name from sysibmadm.applications
where client_nname = '<my client ip>' with ur;
APPL_NAME
------------------
db2jcc_application
MyApplName
As you can see the second connection has the application name correctly stamped, but the first connection simply has "db2jcc_application" as its name. But, of course, I could do this only if I could modify the application code - which I cannot.
The second way I found I can do this is by setting the JDBC URL in the application settings/properties in a particular fashion. Instead of simply using JDBCURL=jdbc:db2://<ip>:<port>/<dbname> in the application settings, if I set JDBCURL=jdbc:db2://<ip>:<port>/<dbname>:clientProgramName=MyApplName;, then the application name gets stamped correctly. Unfortunately, for certain applications that we have, we need to start multiple JVM processes, but I cannot set the above mentioned setting separately for each of these processes. I can set JVM parameters (such as -D parameters) separately for each process.
Thus, I am looking to see if there is a way to externally pass the Application Name so that the JDBC Driver will pick it up and stamp it on the connection.
Below is the list of available properties, which can be set via a JVM parameter or a properties file:
IBM Data Server Driver for JDBC and SQLJ configuration properties.
According to this link, you are not able to set the clientProgramName property in such a way.
I have a problem when accessing arangodb using Java driver.
When i run my application, it runs but neither it shows any response and nor it throws any error. Then i think it must be something on my connection to the remote arangodb server, and i tried to connect using Python arangodb, and it's success to connect.
Anyone knows what's the problem with my arangodb Java application? Is there any difference between java connector and python to arango? I'm using java driver arango 5.0.1
Here's what I've tried:
Java Driver :
public static void main(String[] args) {
// TODO Auto-generated method stub
#SuppressWarnings("deprecation")
ArangoDB arangodb = new ArangoDB.Builder().timeout(100).host("10.22.21.70",8599).user("xxx").password("xxx").useProtocol(Protocol.VST).build();
System.out.println("Test Start");
ArangoDatabase db = arangodb.db("data");
boolean tes = db.exists();
System.out.println("success");
arangodb.shutdown();
}
This program continues to run like this, without being terminated and throwing an error :
enter image description here
Using Python Arango :
import json
from arango import ArangoClient
from pyArango.connection import *
USER = 'xxx'
PASS = 'xxx'
HOST = '10.22.21.70'
DATABASE = 'data'
PORTT = 8599
client = ArangoClient(host=HOST,port=PORTT)
db = client.db(DATABASE,username=USER,password=PASS)
query = 'FOR s IN Email FILTER s.date>= "2018-11-07" LIMIT 0,2 RETURN s'
queryResult = db.aql.execute(query,batch_size=2,count=True)
res = queryResult.batch()
print(res)
And also i've tried to use telnet to the remote server, and it shows blank screen, which means no problem with the connection between my local host with the remote host.
Thanks
I'm using java driver arango 5.0.1
I had the same issue. Using v4.2.2 solved the problem. If someone could explain why drivers are not compatible, that would be great.
You are setting a timeout of 100ms, maybe you have higher network latencies while connecting to the remote server? To double check if this is the problem you can try without setting the timeout.
I am trying to remotely profile alfresco running on a 64-bit linux server running a 1.8 JVM and Apache Tomcat 7.xx from my testing code but can't figure out how to programatically trigger snapshots.
What I want to do is to connect to the remote server, start profiling, and save a snapshot of that server's performance onto my local machine from my testing code which is written in Java.
I've already installed JProfiler 9.2 onto the linux server and can connect and take snapshots via the JProfiler GUI. The server also requires an SSH connection for security. I'd like to do this from my code, similar to how Controller.saveSnapshot(file) works for local JVMs.
Is this possible?
I know I can set triggers and have the remote profiler save snapshots on the server, but this isn't what I want to.
Additionally, I looked into using the command line controller but was unable to get it to connect to the server even with the correct arguments in the remote VM options.
I also tried to use ConnectionFactor.createRemoteConnection(), but don't see an argument that allows for a password to be input, so it fails.
You can access the JProfiler MBean programmatically. Below is an example on how to do that. I would run such a program on the remote machine and start it via SSH, because JMX connnections are difficult to tunnel through SSH.
import javax.management.MBeanServerConnection;
import javax.management.ObjectName;
import javax.management.remote.JMXConnector;
import javax.management.remote.JMXConnectorFactory;
import javax.management.remote.JMXServiceURL;
import java.util.Collections;
import java.util.HashMap;
import java.util.Map;
// Shows how to connect to the the JProfiler MBean programatically
// The profiled process has to be in offline profiling mode and the JMX server
// has to be started by passing -Djprofiler.jmxServerPort=[port] to the profiled JVM.
// This will not work in nowait mode because the MBean is not registered in that case.
public class MBeanProgrammaticAccessExample {
public static void main(String[] args) throws Exception {
if (args.length == 0) {
System.out.println("Specify the port as an argument that was passed to " +
"the profiled JVM with the VM parameter " +
"-Djprofiler.jmxServerPort=[port]");
}
String port = args[0];
// In this case the connection is made to a process on localhost, but it could
// be on a remote system as well. Note that the connection is made via JMX which
// does not work well with firewalls
System.out.println("Connecting to localhost:" + port);
JMXServiceURL jmxUrl = new JMXServiceURL(
"service:jmx:rmi:///jndi/rmi://localhost:" + port + "/jmxrmi");
JMXConnector connector = JMXConnectorFactory.newJMXConnector(jmxUrl,
Collections.<String, Object>emptyMap());
Map<String, Object> env = new HashMap<>();
// If you have protected the JMX server with a JMX password file by passing
// -Djprofiler.jmxPasswordFile=[file] to the profiled JVM, you can specify
// the password like this:
//env.put(JMXConnector.CREDENTIALS, new String[] {"username", "password"});
connector.connect(env);
MBeanServerConnection connection = connector.getMBeanServerConnection();
ObjectName objectName = new ObjectName(
"com.jprofiler.api.agent.mbean:type=RemoteController");
if (!connection.isRegistered(objectName)) {
throw new RuntimeException("JProfiler MBean not found.");
}
RemoteControllerMBean mbeanProxy = JMX.newMBeanProxy(connection,
objectName, RemoteControllerMBean.class, true);
// You can look up all available operations in the javadoc of
// com.jprofiler.api.agent.mbean.RemoteControllerMBean
System.out.println("Recording CPU data for 5 seconds ....");
mbeanProxy.startCPURecording(true);
// If you do not want a dependency on the JProfiler classes
// you can make the above call like this:
//connection.invoke(objectName, "startCPURecording", new Object[] {true},
// new String[] {Boolean.TYPE.getName()});
Thread.sleep(5000);
System.out.println("Saving snapshot to the working directory " +
"of the profiled JVM ....");
mbeanProxy.saveSnapshot("snapshot.jps");
connector.close();
System.out.println("Success");
}
}