I have faced a scenario, the task was to read a file which contains 3 Millions IP Address.
There is MySQL Table which contains Id,PrimaryIP, PrimaryIP can by multiple IP separated by #, more over that PrimaryIP can also contain CIDR IP.
So totally, there are 8000 records, each record with multiple IP and CIDR IP.
Now, my task was to read that file, check it against with database and write the matching IP,ID to a file.
Initially, when i run my program, my program failed because: java.lang.OutOfMemoryError: Java heap space, so i have increased it by 3GB, still it was failing, then later i split the file into 6 subfiles, as 0.5 Millions each.
To find CIDR IP List, i have used Apache SubnetUtils.
Below is my code :
public static void main(String[] args) {
String sqlQuery = "SELECT id,PrimaryIP from IPTable where PrimaryIP != '' limit 100000;";
Connection connection = null;
Statement statement = null;
File oFile = new File("output.txt");
System.out.println(new Date());
try{
List<String> fileData = FileUtils.readLines(new File("input.txt"));
System.out.println("File Data Size : "+fileData.size());
Class.forName("com.mysql.jdbc.Driver");
connection = DriverManager.getConnection("jdbc:mysql://localhost/db?user=root&password=pwd");
statement = connection.createStatement();
ResultSet resultSet = statement.executeQuery(sqlQuery);
System.out.println("Started with MySQL Querying");
Map<String, Integer> primaryIPIDMap = new HashMap<String, Integer>();
while (resultSet.next()) {
primaryIPIDMap.clear();
int recordID = resultSet.getInt(1);
if (resultSet.getString(2).contains("#")) {
String primaryIP[] = resultSet.getString(2).split("#");
for (int i = 0; i < primaryIP.length; i++) {
if (primaryIP[i].contains("/")) {
String allIP[] = getAllIP(primaryIP[i]);
for (int allIPi = 0; allIPi < allIP.length; allIPi++) {
primaryIPIDMap.put(allIP[allIPi].intern(), recordID);
}
} else {
primaryIPIDMap.put(primaryIP[i].intern(), recordID);
}
}
} else {
primaryIPIDMap.put(resultSet.getString(2).intern(), recordID);
}
Iterator entries = fileData.iterator();
while (entries.hasNext()) {
String t = (String) entries.next();
if (primaryIPIDMap.containsKey(t)) {
FileUtils.writeStringToFile(oFile, recordID + "," + t);
}
}
primaryIPIDMap.clear();
}
resultSet.close();
statement.close();
connection.close();
} catch (Exception e) {
e.printStackTrace();
} finally {
try {
if (statement != null)
statement.close();
} catch (Exception se2) {
}
try {
if (connection != null)
connection.close();
} catch (Exception se) {
se.printStackTrace();
}
}
System.out.println("Finished");
System.out.println("End Time : "+new Date());
}
private static String[] getAllIP(String ip) {
return new SubnetUtils(ip).getInfo().getAllAddresses();
}
Can someone tell me the best practice to solve this.
Today it just 3 Millions, tomorrow it may be 5 Millions. I can't keep on creating subfile.
I fixed the problem using
Reading the input file line-by-line
I didn't change MySQL Table structure because it has a dependency in many places and table was not designed by me.
Related
I wrote a simple java program to use JDBC to run 20 queries to fetch data from a view
int connectionSize = 10;
ds.init(connectionSize, settings);
ExecutorService executor = Executors.newFixedThreadPool(10);
try {
stopwatch.start();
for (int i = 0; i < 20; i++) {
executor.execute(new Runnable() {
#Override
public void run() {
String sql = String.format("select * from viewA where userId in (%s)", randomUserIds(5));
PreparedStatement ps = null;
Connection conn = null;
try {
while ((conn = ds.getConnection()) == null) {
try {
Thread.sleep(10000);
} catch (InterruptedException e) {
logger.error(e.getMessage(), e);
}
}
logger.info("conn: " + conn.toString());
ps = conn.prepareStatement(sql);
ps.executeQuery();
ps.close();
} catch (Exception ex) {
ex.printStackTrace();
} finally {
if (conn != null) {
ds.returnConnection(conn);
}
}
}
});
}
executor.shutdown();
boolean finished = executor.awaitTermination(10, TimeUnit.MINUTES);
stopwatch.stop();
logger.log(Level.INFO, "Query Complete in " + stopwatch);
} catch (Exception ex) {
ex.printStackTrace();
}
e.g. select * from ViewA where userId in (random few user Ids)
I used a single connection, and inside a for loop executed the 20 queries in a sequential way
I set up 10 connections in a pool and running the 20 queries in 10 threads
I expected the second approach would use less time to finish the 20 queries, but after testing, the results show me these two approaches return similar time consumption.
I can confirm when I was running the second approach, it created 10 sessions in db.
Is the second approach supposed to give a better performance than the first one? What would be the problem to make the second performance same as the first one?
I am trying to pass the output of a ResultSet to Java HashMap.
Map<Integer, String> sIDpNumberHashMap = new HashMap<Integer, String>();
while (DBresult.next()) {
int sID = DBresult.getInt("slrid");
String pNumber = DBresult.getString("pNumber");
sIDpNumberHashMap.put(sID , pNumber );
System.out.println("Output1"+ sID + "\t" + pNumber + "\n");
}
System.out.println("Output2" + "\n" + sIDpNumberHashMap);
While the Output1 is showing all the records(from the DB). The put command only takes the last value from the ResultSet in.
Output1:
502332262 101E2571G103
502332262 101E2571G103
502332262 116E3139P001
502332262 117E3640G025
502332262 314B7159G003
502332262 117E3640G025
Output2:
{502332262=117E3640G025}
How do I make the put command to iterate over the results from the ResultSet?
All your IDs are identical (502332262), and HashMap doesn't allow duplicate keys. That's the reason you see only one entry in the HashMap (containing the last value you put in the Map).
If you want to allow duplicates, consider a different collection to hold the data. For example, you can use an ArrayList<SomeClass> where SomeClass contains the two properties you read from the DB.
I might be late but I believe someone can get an idea from what I did.
Recently I had an almost similar challenge where I wanted to build dynamic query results (whatever select query just returns its JSON list and question would vary a lot and I can't write method per query so I had to come up with something everyone will be calling).
Below is my sample code:
public <T> List<T> findWithDynamicQuery(String query){
Connection conn = null;
PreparedStatement statement = null;
List<Object> mResults = new ArrayList<>();
try {
conn = //use your connection parameters;
} catch (SQLException e) {
e.printStackTrace(out);
return (List<T>) mResults;
}
try {
statement = conn.prepareStatement(query);
ResultSet result = statement.executeQuery();
ResultSetMetaData metaData = result.getMetaData();
int column = metaData.getColumnCount();
while (result.next()) {
HashMap<Object, Object> rows = new HashMap<>();
for(int i = 1; i <= column; i++){
rows.put(metaData.getColumnLabel(i), result.getObject(i));
}
mResults.add(rows);
}
result.close();
return (List<T>) mResults;
} catch (SQLException e) {
e.printStackTrace(out);
}finally{
if(statement != null){
try {
statement.close();
} catch (SQLException e) {
}
}
if(conn != null){
try {
conn.close();
} catch (SQLException e) {
}
}
}
return (List<T>) mResults;
}
So for example if I have an array list of strings, how do I put them into a database sequentially.
For example if my array list is: "john", "sally", "rob"
How can I come put that into my database such that row 1 is john, row 2 is sally, row 3 is rob.
I was thinking something along the lines of:
for(int i = 0; i < array.size(); i++){
query = "insert into database values(i);
}
But I am not sure if that is the correct approach because I will also be inserting other values into the database other than the array.
I assume you mean a simple insertion;
First you have the database connection, Like this:
public class DBConnection {
final private String url = "jdbc:mysql://localhost/";
final private String databaseName = "test";
final private String user = "root";
final private String password = "159753";
public Connection Connect() {
Connection c = null;
try {
c = DriverManager.getConnection(url + databaseName, user, password);
} catch (SQLException ex) {
Logger.getLogger(DBConnection.class.getName()).log(Level.SEVERE, null, ex);
}
return c;
}}
Below is a simple example of insertion:
public static void main(String[] args) {
Connection con = new DBConnection().Connect();
String sql = "INSERT INTO nameTable (name) values (?)";
PreparedStatement stmt;
try {
stmt = con.prepareStatement(sql);
List<String> names = new ArrayList();
names.add("John");
names.add("Sally");
names.add("Rob");
for (String name : names) {
stmt.setString(1, name);
stmt.execute();
}
stmt.close();
con.close();
} catch (SQLException ex) {
System.err.println("Error = " + ex);
}
}}
PS: But you can use a Java Persistence API too. The wikibook is a good resource to use as reference
public void datuieguve(){
Scanner ievads = new Scanner(System.in);
String lietotIn = ievads.next();
// String paroleIn = ievads.next();
try {
quer = savien.createStatement();
String kverijs = "select lietotajvards from lietotaji where lietotajvards = '" +lietotIn+ "';";
rezult = quer.executeQuery(kverijs);
System.out.println(rezult.next());
String lietotajvards = rezult.getString("lietotajvards");
if (rezult.equals(lietotIn))
{
System.out.println("Yup");
}
else
{
System.out.println("Nope");
}
} catch(Exception ex) {
System.out.println("Kluuda: " +ex);
}
ievads.close();
}
Hi guys, I'm trying to create a code that will take user input and compare it with mysql database value. The problem is, it's not working and I'm stuck. I'd appreciate if someone could figure out what's wrong with my code and give me some tips or something, because I can't :)
I think I got down the connection part but the comparison is driving me nuts.
Make these changes on your code :
while(rezult.next()){
String lietotajvards = rezult.getString("lietotajvards");
if(lietotajvards.equals(lietotIn))
{
System.out.println("Yup");
}
...
}
Edit:
Its good practice to use prepared statement instead of create statement.
Ex:
Connection db = con.getConnection();
try {
PreparedStatement ps = ...;
try {
ResultSet rs = ...
try {
...
}
finally {
rs.close();
}
}
finally {
ps.close();
}
}
finally {
db.close();
}
OK, I know that Batch Processing allows to group related SQL statements into a batch and submit them with one call to the database. When you send several SQL statements to the database at once, you reduce the amount of communication overhead, thereby improving performance. In this particular situation (see code below) I don't think batching does it's sole purpose. Cause stmt.executeBatch() is called straight away after adding a batch(?) Wouldn't stmt.executeUpdate() do the same thing?
public void tableChanged(TableModelEvent e) {
int row = e.getFirstRow();
int col = e.getColumn();
model = (MyTableModel) e.getSource();
String stulpPav = model.getColumnName(col);
Object data = model.getValueAt(row, col);
Object studId = model.getValueAt(row, 0);
System.out.println("tableChanded works");
try {
new ImportData(stulpPav, data, studId);
bottomLabel.setText(textForLabel());
} catch (ClassNotFoundException e1) {
e1.printStackTrace();
} catch (SQLException e1) {
e1.printStackTrace();
}
}
public class ImportData {
public ImportData(String a, Object b, Object c)
throws ClassNotFoundException, SQLException {
Statement stmt = null;
try {
connection = TableWithBottomLine.getConnection();
String stulpPav = a;
String duom = b.toString();
String studId = c.toString();
System.out.println(duom);
connection.setAutoCommit(false);
stmt = connection.createStatement();
stmt.addBatch("update finance.fin set " + stulpPav + " = " + duom
+ " where ID = " + studId + ";");
stmt.executeBatch();
connection.commit();
} catch (BatchUpdateException e) {
e.printStackTrace();
} catch (SQLException e) {
e.printStackTrace();
} finally {
if (stmt != null)
stmt.close();
connection.setAutoCommit(true);
System.out.println("Data was imported to database");
}
}
}
In this case using batch has no advantage at all. It might even introduce additional overhead over a direct executeUpdate (but that is driver and database dependent).
However, don't assume that batching has advantages with all JDBC drivers. I haven't looked at the specifics of MySQL, but I know there are JDBC drivers where batching internally is a normal execute for each statement in the batch.
The code in your question however has a bigger problem: it is vulnerable to SQL injection.