i have one jar file inside it there is lib folder which contain all the jar files which we mentaion in the pom file.
here the question is : how is read the all the external jar (pom file jars) files resources.
example : example.jar has dependies in lib folder file1.jar & file2.jar i want to read the resource of both file1.jar and file2.jar
how JVM loads the all the resources?
This is a very unusual situation, maybe much better approach which is usually used is "flatterning" the jars, so you won't have dependent jars in some folder inside an "outer" jar, but instead all packages from dependent jars will become packages of the outer jar residing next to your own code that probably is in the outer jar anyway.
Maven has shade plugin for this, and this is usually the way to go.
One noticeable exception is a spring boot applications packaged as JARs that work just like you've said (they put dependent jars into BOOT-INF/lib library, so technically its jars inside jar).
They have their own reasons to work like this which are way beyond the scope of this question, but the relevant point is that they had to create a special class loader that would handle this situation. Java out of the box can read classes from filesystem or from regular jar, but in theory java application can read the binary code from any place (Remote Filesystem, Database, Jar inside Jar whatever) as long as you implement the class loader that can find and load the resources from there.
In general I would recommend not to mess with Class Loaders which are pretty advanced concepts unless you really know what you're doing. Most of java programmers do not really create their own class loaders.
venkateswararao yeluru please follow below code to read data from jar file:
public class FirstExample {
// JDBC driver name and database URL
static final String JDBC_DRIVER = "com.mysql.jdbc.Driver";***`
strong text`***
static final String DB_URL = "jdbc:mysql://localhost/EMP";
// Database credentials
static final String USER = "username";
static final String PASS = "password";
public static void main(String[] args) {
Connection conn = null;
Statement stmt = null;
try {
// STEP 2: Register JDBC driver
Underclassmen("com.mysql.jdbc.Driver");
// STEP 3: Open a connection
System.out.println("Connecting to database...");
conn = DriverManager.getConnection(DB_URL, USER, PASS);
// STEP 4: Execute a query
System.out.println("Creating statement...");
stmt = conn.createStatement();
String sql;
sql = "SELECT id, first, last, age FROM Employees";
ResultSet rs = stmt.executeQuery(sql);
// STEP 5: Extract data from result set
while (rs.next()) {
// Retrieve by column name
int id = rs.getInt("id");
int age = rs.getInt("age");
String first = rs.getString("first");
String last = rs.getString("last");
// Display values
System.out.print("ID: " + id);
System.out.print(", Age: " + age);
System.out.print(", First: " + first);
System.out.println(", Last: " + last);
}
// STEP 6: Clean-up environment
rs.close();
stmt.close();
conn.close();
} catch (SQLException se) {
// Handle errors for JDBC
se.printStackTrace();
} catch (Exception e) {
// Handle errors for Class.forName
e.printStackTrace();
} finally {
// finally block used to close resources
try {
if (stmt != null)
stmt.close();
} catch (SQLException se2) {
} // nothing we can do
try {
if (conn != null)
conn.close();
} catch (SQLException se) {
se.printStackTrace();
} // end finally try
} // end try
System.out.println("Goodbye!");
}// end main
}// end FirstExample
Related
I am building a java app that gets data from the file system (parquet) in mapR cluster. I was initially using apache spark but the processing was quite slow.
So i decided to use the drill jdbc connection approach.
Following the documentation in mapR https://mapr.com/docs/52/Drill/Using-JDBC-Driver-App.html
Here is my code
Step 1;
placed the driver jar in a lib folder in my project path as shown;
[project directory][1]
[1]: https://i.stack.imgur.com/APsZi.png
step 2;
Imported the jar to my maven pom.xml
<dependency>
<groupId>DrillJDBC41</groupId>
<artifactId>DrillJDBC41</artifactId>
<scope>system</scope>
<version>1.0</version>
<systemPath>${project.basedir}\src\lib\DrillJDBC41.jar</systemPath>
</dependency>
step 3
my code implementation
try {
private static final String CONNECTION_URL = "jdbc:drill:zk=192.168.1.1:31010/drill/dev.maprcluster.com-drillbits;schema=dfs";
private static final String JDBC_DRIVER = "com.mapr.drill.jdbc41.Driver";
Connection con = null;
Statement stmt = null;
ResultSet rs = null;
// Define a plain query
String query = "SELECT * FROM `dfs.default`.`/storage/products/data/d/report/2019/07/12`" + " where unique_key = '00209220' LIMIT 30";
// Register the driver using the class name
Class.forName(JDBC_DRIVER);
// Establish a connection using the connection
// URL
try {
System.out.println("about establishing connection");
con = DriverManager.getConnection(CONNECTION_URL);
System.out.println("connection established");
}catch (Exception e){
System.out.println("EXCEPTION OO");
e.printStackTrace();
}
// Create a Statement object for sending SQL
// statements to the database
stmt = con.createStatement();
System.out.println("trying to execute query");
// Execute the SQL statement
rs = stmt.executeQuery(query);
// Display a header line for output appearing in
// the Console View
System.out.println("gotten result set");
// Step through each row in the result set
// returned from the database
while(rs.next()) {
// Retrieve values from the row where the
System.out.println("data is returned");
// cursor is currently positioned using
// column names
// Display values in columns 20 characters
// wide in the Console View using the
// Formatter
}
} catch (SQLException se) {
System.out.println("sql exception");
se.printStackTrace();
// Handle errors encountered during interaction
// with the data source
} catch (Exception e) {
e.printStackTrace();
// Handle other errors
} finally {
// Perform clean up
try {
System.out.println("entered finally block");
if (rs != null) {
rs.close();
}
if (stmt != null) {
stmt.close();
}
if (con != null) {
con.close();
}
} catch (SQLException se1) {
se1.printStackTrace();
}
} // End try
}
PROBLEM
The application builds fine but when i try to get data , it stops after printing this;
about establishing connection
and goes straight into the finally block without throwing any exception.
I am not sure what the problem is.
I also tried another another implementation using apache drill
<dependency>
<groupId>org.apache.drill.exec</groupId>
<artifactId>drill-jdbc-all</artifactId>
<version>1.1.0</version>
</dependency>
changed the driver to this;
private static final String JDBC_DRIVER_DRILL = "org.apache.drill.jdbc.Driver";
still the same problem.
No error thrown.
Output;
about establishing connection
entered finally block
UPDATE
I caught Throwable as advised and i was getting the following error;
java.lang.NoSuchMethodError: com.google.common.base.Stopwatch.createStarted()Lcom/google/common/base/Stopwatch;
at org.apache.drill.common.config.DrillConfig.create(DrillConfig.java:189)
at org.apache.drill.common.config.DrillConfig.create(DrillConfig.java:163)
at org.apache.drill.common.config.DrillConfig.forClient(DrillConfig.java:114)
at com.mapr.drill.drill.client.DRJDBCClient.openSession(Unknown Source)
at com.mapr.drill.drill.client.DRJDBCClient.<init>(Unknown Source)
at com.mapr.drill.drill.core.DRJDBCConnection.connect(Unknown Source)
at com.mapr.drill.jdbc.common.BaseConnectionFactory.doConnect(Unknown Source)
at com.mapr.drill.jdbc.common.AbstractDriver.connect(Unknown Source)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:270)
Looking at this line,
`private static final String CONNECTION_URL = "jdbc:drill:zk=192.168.1.1:31010/drill/dev.maprcluster.com-drillbits;schema=dfs";`
The documentation you provided a link to, stated it to be an example of connecting to a zookeeper cluster. And you copied the same server address provided in the example. There are many cluster managers like Yarn and Mesos. Ensure you are using the correct connection string to the right manager in your cluster and again, the server addresses should correspond to the respective addresses in your cluster.
Again, separate the aspect of the code that you want to use to get connection to a separate class and handle the logic in a single try-catch block, for the sake of separation of concern, instead of having it in a nested try-catch block in your service class.
This would enable you to test and debug more easily to understand what is happening.
Finally, since there's no static trace to inform of the error, the right approach is to debug the connection class (assuming you have extracted it out), then step through it to learn what is happening in that line. That would enable you to figure out what to do next.
I love Apache Drill, but is this really the best way to read Parquet data if you are worried about speed?
The AvroParquetReader might be a better approach for directly reading a Parquet-formatted file.
You can also use the ParquetFileReader class directly. Example here:
https://www.jofre.de/?p=1459
and here:
https://www.arm64.ca/post/reading-parquet-files-java/
I want to insert every filename of my local drive into a mysql database.
When I execute this code, it start perfectly.
Code that reads the directory for filenames:
public void main(String[] args)throws IOException {
JOptionPane.showMessageDialog(null, "IT MAY TAKE SOMETIME TO LOAD \n PLEASE WAIT FOR CLOSING POPUP!!");
String s="D:\\";
Path startingDir = Paths.get(s);
String pattern="*";
Finder finder = new Finder(pattern);
Files.walkFileTree(startingDir, finder);
JOptionPane.showMessageDialog(null,"close\n NOW PLEASE CLICK\nSEARCH MY FILE! BUTTON");
This is the code to insert the results into the database:
public void find(Path file) {
Path name = file.getFileName();
String st = file.toString();
if (name != null && matcher.matches(name)) {
try {
Connection conn=DriverManager.getConnection("jdbc:mysql://localhost/login","root","");
conn.createStatement();
String query =" INSERT INTO `search`(`path`) VALUES (?)";
PreparedStatement pst=conn.prepareStatement(query);
pst.setString(1,st );
pst.execute();
//myst.executeUpdate(query);
} catch (SQLException e) {
// TODO Auto-generated catch block
JOptionPane.showMessageDialog(null, e);
}
After some time, the scripts stops with this exception
com.mysql.jdbc.exception.jdbc4.MYSQLNonTransientConnectionException:
Data source rejected establishment of connection,
message from server:"Too Many connections"
Is there any way to solve this problem?
For every insert you create a new connection. If you keep doing this they build up and eventually you run out of connections to the database. This limit could be quite small e.g. 20.
Instead you can either
close the resources you have used. This means closing the PreparedStatement and the Connection
or more efficiently, create one Connection and one PrepareStatement ever and reuse it. The saves having to create and clean up resources which can be expensive (unless the driver does this recycling for you)
the easy fix would be to close the connection after you execute the query.
pst.close();
conn.close();
this should get the job done.
But it'd be better to reuse the connection.
I have a folder/stream of different complex XML files (each of size ~ 1GB). I know how to load an XMLfile data to Hive table (or any Hadoop data base).
But I want to know two things:
Can I load each xml file data to hive dynamically, i.e. without explicitly writing a create table command (because I get different XML files as a stream), is there any way which does this automatically.
"Stream of different complex xml Files --> Load to Hive tables (with out manually writing Create table command) --> Use the data which is loaded into Hive tables"
Instead of writing command line scripts to create hive tables, How can I write a java code to load xml data to Hive table.
Regarding your first question, AFAIK, it is not possible. Hive is intended to manage data that is stored within Hive tables (it is not always stored within tables, but metadata is added to the tables, pointing to the real data, that's the case of Hive external tables).
The only thing I think you can try is to create a single big table for all the data within your XML files, the already stored ones and the future ones; the trick is to put all the XML files under a common HDFS folder that it is used as the location of the create table command.
Regarding your second question, please refer to this code:
public final class HiveBasicClient {
private static String driverName = "org.apache.hadoop.hive.jdbc.HiveDriver";
private static Connection con;
private static Connection getConnection(String hiveServer, String hivePort, String hadoopUser, String hadoopPassword) {
try {
Class.forName(driverName);
} catch (ClassNotFoundException e) {
return null;
}
try {
return DriverManager.getConnection("jdbc:hive://" + hiveServer + ":" + hivePort + "/default?user=" + hadoopUser + "&password=" + hadoopPassword);
} catch (SQLException e) {
return null;
}
}
private static res doQuery(String query) {
try {
Statement stmt = con.createStatement();
ResultSet res = stmt.executeQuery(query);
res.close();
stmt.close();
return res;
} catch (SQLException ex) {
System.exit(0);
}
}
public static void main(String[] args) {
String hiveServer = args[0];
String hivePort = args[1];
String hadoopUser = args[2];
String hadoopPassword = args[3];
con = getConnection(hiveServer, hivePort, hadoopUser, hadoopPassword);
doQuery("create external table <table_name> (<list_of_columns>) row format serde '<your_xml_serde>' location `<your_xml_files_location>');
}
}
Hope it helps.
I am new to mysql and jdbc and I am getting the error in this title. I have been searching all day and cannot find a solution that works for me.
What I have tried:
uninstall/reinstall mysql, copy paste mysql-connector-java-5.1.25-bin.jar and ojdbc7.jar to same location as the .class file I am trying to run, rebuilt the program in a different directory, and probably a couple other things.
I am using notepad++ for coding and the windows command prompt to compile and run. it compiles fine but I try to run with
C:\Projects\bin>java -cp . ClientBase
The output is:
java.lang.ClassnNotFoundException: com.mysql.jdbc.Driver
at java.net.URLClassloader$1.run(URLClassLoader.java:336)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:432)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:356)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:188)
at ClientBase.main(ClientBase.java:21)
Goodbye.
// import packages
import java.sql.*;
// create class ClientBase
public class ClientBase{
// JDBC driver name and database URL
static final String JDBC_DRIVER = "com.mysql.jdbc.Driver";
static final String DB_URL = "jdbc:mysql://localhost/CLIENTBASE";
// Database credentials
static final String USER = "root";
static final String PASS = "";
// Begin method main
public static void main(String[] args) {
Connection conn = null;
Statement stmt = null;
try{
// register JDBC driver
Class.forName("com.mysql.jdbc.Driver");
// Open connection
System.out.println("Connecting to database...");
conn = DriverManager.getConnection(DB_URL, USER, PASS);
// Execute a query
System.out.println("Creating statement...");
stmt = conn.createStatement();
String sql;
sql = "SELECT id, name, address, address 2, city, phone, state, zip, fax FROM CLIENTBASE";
ResultSet rs = stmt.executeQuery(sql);
// Extract data from result set
while(rs.next()){
// Retrieve by column name
int id = rs.getInt("id");
String name = rs.getString("name");
String address = rs.getString("address");
String address2 = rs.getString("address2");
String city = rs.getString("city");
String phone = rs.getString("phone");
String state = rs.getString("state");
String zip = rs.getString("zip");
String fax = rs.getString("fax");
// Display values
System.out.print("ID: " + id);
System.out.print(" Name: " + name);
System.out.println("Address:" + address);
System.out.println(address2);
System.out.print("City:" + city);
System.out.print(" State: " + state);
System.out.println(" Zip: " + zip);
System.out.print("Phone: " + phone);
System.out.println(" Fax: " + fax);
} // end while
// clean up
rs.close();
stmt.close();
conn.close();
}catch(SQLException se){
// Handle errors for JDBC
se.printStackTrace();
}catch(Exception e){
// Handle errors for Class.forName
e.printStackTrace();
}finally{
// finally block used to close resources
try{
if(stmt!=null)
stmt.close();
}catch(SQLException se){
se.printStackTrace();
} // end finally
} // end try
System.out.println("Goodbye.");
} // End method main
} // end class ClientBase
I should also say that I am going off an online tutorial for this code. It is not exactly how they have it as I decided to make something a little different than theirs, but it is generally the same. I don't think it is a code problem though from what the error is.
Any help would be appreciated! I'm going crazy!
You need to add a connector library to the Runtime classpath:
java -cp .;mysql-connector-java-5.1.25-bin.jar ClientBase
My example uses Windows classpath separator ";", on other systems it may be different (":" on Linux/Mac). It also assumes, that mysql-connector-java-5.1.25-bin.jar is located on the same folder. If it's not the case, then put a path to the library instead of the plain name.
ClientBase stands for Java class file name here
c:\>javac Test.java
c:\>java -cp .;F:\CK\JavaTest\JDBCTutorial\mysql-connector-java-5.1.18-bin Test
I was also get same problem :-
How I solved for Linux system.
1.) Download file mysql-connector-java-5.1.18-bin.jar from given link or other.
2.) Then paste it to same folder or directory of your class or file (on where you want connection to present)
3.) Now use the following command by your linux command prompt.
java -cp .:mysql-connector-java-5.1.18-bin.jar YourFileName ..
Thats all you need to do... :)
if u use netbeans,do following
1.Open Netbeans IDE
2.Right-click your Project.
3.Select Properties.
4.On the left-hand side click Libraries.
5.Under "Compile" tab - click Add Jar/Folder button.
6.Select Downloaded "mysql-connector-java-5.1.25-bin.jar" file (Download Connector/J from dev.mysql.com)
7.Click OK
Run Again... Its work.
What did you import ?
From the documentation: http://dev.mysql.com/doc/connector-j/en/connector-j-usagenotes-connect-drivermanager.html
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.SQLException;
// Notice, do not import com.mysql.jdbc.*
// or you will have problems!
Comment:
Why are you using notepad++ ? install an IDE (Eclipse/Netbeans/IntelliJ) - it'll be much easier to locate such problems (un-included jars for example)
Add mysql.connector-java-x.x.x-bin.jar to your libraries folder. No need to import anything. Have a good one.
If You Are Using Swing you have to add External Jar of my-sql-connect
else if You are Using jsp servlet You have to add External jar and Copy that Jar into WEB_INF/lib project folder
for a school database project we are making a database program (user GUI and the database). Using Microsoft Access 2010 I created the database and populated it with some sample data, and saved it in .mdb format and placed it in my project folder.
When running it in eclipse the following code works fine, connects and even retrieves the query. However I find that I am unable to export the code to a jar and run it (which is required for the project, give them a working copy of your program on a CD or flash drive), and I'm also unable to port the code over to Netbeans to have it work, as well as trying to compile on a Linux machine.
I assume this is a problem with including drivers or trying to use Microsoft access. The error I get when running the jar or running on Netbeans is given below the code. So I ask either how do I include drivers to make the program portable, or how else can I approach this problem?
Thanks in advance
import java.sql.*;
public class JDBCTest {
static Connection connection;
static Statement statement;
public static void main(String args[]){
try {
Class.forName("sun.jdbc.odbc.JdbcOdbcDriver").newInstance();
String database = "jdbc:odbc:Driver={Microsoft Access Driver (*.mdb)};DBQ=TLDATABASEDBM.mdb";
connection = DriverManager.getConnection( database ,"","");
buildStatement();
executeQuery();
}catch(Exception e){
e.printStackTrace();
System.out.println("Error!");
}
}
public static void buildStatement() throws SQLException {
statement = connection.createStatement();
}
public static void executeQuery() throws SQLException {
boolean foundResults = statement.execute("SELECT * FROM tblStaff AS x WHERE City='Calgary'");
if(foundResults){
ResultSet set = statement.getResultSet();
if(set!=null) displayResults(set);
}else {
connection.close();
}
}
public static void displayResults(ResultSet rs) throws SQLException {
ResultSetMetaData metaData = rs.getMetaData();
int columns=metaData.getColumnCount();
String text="";
while(rs.next()){
for(int i=1;i<=columns;++i) {
text+=""+metaData.getColumnName(i)+":\t";
text+=rs.getString(i);
//text+="</"+metaData.getColumnName(i)+">";
text+="\n";
}
text+="\n";
}
System.out.println(text);
}
}
The error mentioned above:
java.sql.SQLException: [Microsoft][ODBC Driver Manager] Data source name not found and no default driver specified
at sun.jdbc.odbc.JdbcOdbc.createSQLException(JdbcOdbc.java:6957)
at sun.jdbc.odbc.JdbcOdbc.standardError(JdbcOdbc.java:7114)
at sun.jdbc.odbc.JdbcOdbc.SQLDriverConnect(JdbcOdbc.java:3073)
at sun.jdbc.odbc.JdbcOdbcConnection.initialize(JdbcOdbcConnection.java:323)
at sun.jdbc.odbc.JdbcOdbcDriver.connect(JdbcOdbcDriver.java:174)
at java.sql.DriverManager.getConnection(DriverManager.java:582)
at java.sql.DriverManager.getConnection(DriverManager.java:207)
at tldatabase.DataConnect.makeConnection(DataConnect.java:35)
at tldatabase.Main.main(Main.java:24)
I know the post was years ago but I felt like answering the question for those who are just experiencing this right now. It took me a while to know the answer to the question so here's the solution:
http://wiki.netbeans.org/FaqSettingHeapSize
Follow the "Running the 32-bit JVM".
All you have to do is find the netbeans.conf in the installation folder of your netbeans and change the directory from something like this:
netbeans_jdkhome="C:\Program Files\Java\jdk1.6.0_24"
to this:
netbeans_jdkhome="C:\Program Files (x86)\Java\jdk1.6.0_21"
The problem is netbeans might be running in 64 bit but MS Access only support 32-bit. So doing this would hopefully solve the problem. Also make sure to install this:
http://www.microsoft.com/download/en/details.aspx?displaylang=en&id=23734
The main problem lies in the line:
String database = "jdbc:odbc:Driver={Microsoft Access Driver (*.mdb)};DBQ=TLDATABASEDBM.mdb";
Make sure that the .mdb file is in the correct directory.
Check the file extension as .mdb or .mdbacc.
Also, if you want to use the same DSN every time, it is better to add the DSN(Data Source Name) into the respective system on which the mdb is stored.
I think that your app do not see TLDATABASEDBM.mdb in current directory. You can give full path to this file in connection string or add system DSN in ODBC Manager and then connect to it with connection string like: jdbc:odbc:TLDATABASEDBM
Honestly, I dont like what I am going to say... but, it solved the same issue for me... mysteriously... :(((
on the line where you are defining the database variable, I changed ...(.mdb)... into ...(.mdb, *.accdb)...
All the best for figuring out what difference that made!
package javaapplication1;
import java.sql.*;
public class MSaccess_archive {
public static void main(String[] args) {
try {
Class.forName("sun.jdbc.odbc.JdbcOdbcDriver");
// set this to a MS Access DB you have on your machine
String filename = "mdbTEST.mdb";
String database = "jdbc:odbc:Driver={Microsoft Access Driver (*.mdb)};DBQ=";
database+= filename.trim() + ";DriverID=22;}"; // add on to the end
// now we can get the connection from the DriverManager
Connection con = DriverManager.getConnection( database ,"","");
Statement stmt = con.createStatement();
stmt.execute("select * from student"); // execute query in table student
ResultSet rs = stmt.getResultSet(); // get any Result that came from our query
if (rs != null)
while ( rs.next() ){
System.out.println("Name: " + rs.getInt("Age") + " ID: "+rs.getString("Course"));
}
stmt.close();
con.close();
}
catch (Exception err) {
System.out.println("ERROR: " + err);
}
}
}