I've written what seems to be a prohibitively slow program where it is taking many minutes rather than seconds to read and write data from a database. It appears to be slowing down a great deal in my runComparisons method below although im unsure what steps I should take to correct this.
full program
public static void runComparisons(String industryTitle, List<String> tickerName, TradingInterval tradingInterval) throws SQLException {
// Get primary performance values ticker return
for (String ticker : tickerName) {
String startDate = tradingInterval.beginDate;
//System.out.println("start date " + startDate);
String endDate = tradingInterval.endDate;
//System.out.println("end date " + endDate);
PreparedStatement statement = conn.prepareStatement
("select P.TransDate, P.openPrice, P.closePrice" +
" from PriceVolume P" +
" where Ticker = ? and TransDate>= ? and TransDate <= ?" +
"order by TransDate DESC");
statement.setString(1, ticker);
statement.setString(2, startDate);
statement.setString(3, endDate);
ResultSet result = statement.executeQuery();
String date;
double nextOpenPrice = 0.0;
double currentClosePrice;
//int numberDays = 1;
//int splitCounter = 0;
double divideAmount = 1;
String tickerSymbol = "";
// create list to contain dates for use in investment strategy
ArrayList<holdSplitVals> ascendingList = new ArrayList<>();
// process result set
while (result.next()) {
//tickerSymbol = result.getString("Ticker");
// reset split amount for next iteration
date = result.getString("TransDate");
double openPrice = result.getDouble("OpenPrice");
double closePrice = result.getDouble("ClosePrice");
currentClosePrice = closePrice;
// get closing/opening ratio
double splitAmount = getRatios(date, nextOpenPrice, currentClosePrice);
if (splitAmount > 0) {
divideAmount = divideAmount * splitAmount;
//splitCounter++;
}
// hold data in helper class
holdSplitVals data = new holdSplitVals();
data.date = date;
data.openPrice = openPrice / divideAmount;
data.closePrice = closePrice / divideAmount;
ascendingList.add(0, data);
nextOpenPrice = openPrice;
if (ascendingList.size() >= 1) {
outputTable(closePrice, openPrice, industryTitle, tickerSymbol, startDate, endDate);
}
}
}
}
private static void outputTable(double closePrice, double openPrice, String industryTitle, String tickerSymbol, String startDate, String endDate) throws SQLException {
double tickerNumber = (closePrice / openPrice) - 1;
DecimalFormat df = new DecimalFormat("#.####");
df.setRoundingMode(RoundingMode.CEILING);
String tickerNum = df.format(tickerNumber);
PreparedStatement writeStatement = writerconn.prepareStatement(
"insert into Performance(Industry, Ticker, StartDate, EndDate, TickerReturn)"
+ "values(?, ?, ?, ?, ?)");
writeStatement.setString(1, industryTitle);
writeStatement.setString(2, tickerSymbol);
writeStatement.setString(3, startDate);
writeStatement.setString(4, endDate);
writeStatement.setString(5, tickerNum);
writeStatement.executeUpdate();
}
private static double getRatios(String date, double nextOpenPrice, double currentClosePrice) {
double coRatio = currentClosePrice / nextOpenPrice;
// calculate splits
// for 2:1 split
if (Math.abs(coRatio - 2.0) < 0.20) {
//splitCounter++;
//splitDate = ("2:1 split on " + date);
//System.out.println("2:1 split on " + date + " " + currentClosePrice + "--> " + nextOpenPrice);
return 2.0;
// for 3:1 split
} else if (Math.abs(coRatio - 3.0) < 0.20) {
//splitCounter++;
//splitDate = ("3:1 split on "+ date);
//System.out.println("3:1 split on " + date + " " + currentClosePrice + "--> " + nextOpenPrice);
return 3.0;
// for 3:2 split
} else if (Math.abs(coRatio - 1.5) < 0.15) {
//splitCounter++;
//splitDate = ("3:2 split on " + date);
//System.out.println("3:2 split on " + date + " " + currentClosePrice + "--> " + nextOpenPrice);
return 1.5;
}
return 0;
}
}
Related
So I have some issue on my for loop. It only work on the first row of jtable and printed the same result to the whole rows. here's my code:
try {
int row = tblHasil.getRowCount();
for (int i = 0; i < row; i++) {
String tempA = (String) tblHasil.getModel().getValueAt(i, 3);
String tempB = (String) tblHasil.getModel().getValueAt(i, 4);
String tempC = (String) tblHasil.getModel().getValueAt(i, 5);
double a = Double.parseDouble(tempA);
double b = Double.parseDouble(tempB);
double c = Double.parseDouble(tempC);
VNilaiRataRata.setNilaiRataRata(a);
VTanggungan.setTanggungan(b);
VPenghasilan.setPenghasilan(c);
Rules.hitung_u();
Rules.hitung_z();
DecimalFormat df = new DecimalFormat("#########.###");
double hasildef = Double.parseDouble(df.format(Rules.defuzzifikasi()));
VKeterangan.setKeterangan(hasildef);
double keterangan = VKeterangan.getKeterangan();
double tidakLolos = VKeterangan.tidaklolos();
double cadangan = VKeterangan.cadangan();
double lolos = VKeterangan.lolos();
String desc;
if (lolos > cadangan && lolos > tidakLolos) {
desc = "Lolos";
tblHasil.setValueAt(desc, i, 6);
} else if (cadangan > tidakLolos && cadangan > lolos) {
desc = "Cadangan";
tblHasil.setValueAt(desc, i, 6);
} else {
desc = "Tidak Lolos";
tblHasil.setValueAt(desc, i, 6);
}
System.out.println("Nama : " + tblHasil.getModel().getValueAt(i, 1));
System.out.println("Hasil Fuzzi : " + hasildef);
System.out.println("Keterangan Value : " + keterangan);
System.out.println("Keterangan Tidak Lolos : " + tidakLolos);
System.out.println("Keterangan Cadangan : " + cadangan);
System.out.println("Keterangan Lolos : " + lolos);
System.out.println("Nilai Rata2 : " + VNilaiRataRata.getNilaiRataRata());
System.out.println("Tangungan : " + VTanggungan.getTanggungan());
System.out.println("Penghasilan : " + VPenghasilan.getPenghasilan());
System.out.println("Keterangan : " + desc);
System.out.println("================================");
System.out.println();
Connection cn = DriverManager.getConnection("jdbc:mysql://localhost/db_siswa", "root", "");
int nis = Integer.parseInt(tblHasil.getModel().getValueAt(i, 0).toString());
String sql = "UPDATE nilaisiswa SET hasil='" + desc + "' WHERE nis = '" + nis + "'";
cn.createStatement().executeUpdate(sql);
}
} catch (Exception e) {
e.printStackTrace();
java.util.logging.Logger.getLogger(Proses.class.getName()).log(Level.SEVERE, null, e);
}
And here's the result (marked with red circle) :
i don't know how to use this thing
I am trying to run the below query and insert the output of it into another table (postgres db):
String ssnQuery = "SELECT period_year, period_name, period_num, NULL as count_of_issues,
ledger_id,
balancing_segment,
Count(*) AS count_of_account_segments,
Sum(accounted_period_net_dr) AS balance_accounted_period_net_dr,
Sum(accounted_period_net_cr) AS balance_accounted_period_net_cr,
Round(Sum(accounted_period_net_dr_cr)) AS balance_accounted_period_net_dr_cr_diff,
Sum(entered_period_net_dr) AS balance_entered_period_net_dr,
Sum(entered_period_net_cr) AS balance_entered_period_net_cr,
Round(Sum(entered_period_net_dr_cr)) AS balance_entered_period_net_dr_cr_diff,
Sum(begin_balance_dr) AS begin_balance_dr,
Sum(begin_balance_cr) AS begin_balance_cr,
Round(Sum(net_beginning_balance)) AS net_beginning_balance,
Round(Sum(net_closing_balance)) AS net_closing_balance
FROM schema.tablename";
try {
pstmnt = financeConnection.prepareStatement(ssnQuery);
rs = pstmnt.executeQuery();
rsmd = rs.getMetaData();
for(int i=1; i<=rsmd.getColumnCount(); i++) {
columnNames.add(rsmd.getColumnName(i));
if(i == 1) {
queryColumns = rsmd.getColumnName(i);
} else if(i<7) {
queryColumns += ", " + rsmd.getColumnName(i);
} else {
queryColumns += ", value_" + (i-7);
}
}
while (rs.next()) {
queryValues = " ";
for(String colname: columnNames) {
if(queryValues.isEmpty()) {
queryValues = rs.getString(colname);
} else {
queryValues += rs.getString(colname) + ", ";
}
}
remCommas = queryValues.replaceAll(", $", "");
insertQuery = "INSERT INTO bdmerge.gen_audit_func_hive_results (run_id, run_date, run_date_ist" + queryColumns + ") VALUES (" + runid +"," + utcTimeStamp + "," + istTimeStamp + "," + remCommas + ")";
System.out.println("Final Insert query: " + insertQuery);
}
} catch (SQLException e) {
e.printStackTrace();
} catch(Exception e) {
e.printStackTrace();
}
To insert the output of above query, I formed the insert query according the column names in the destination table as below:
INSERT INTO schema.destinationTable (run_id, run_date, run_date_ist, source_system_type, source_system, module, source_table_name, period_year, period_name, period_num, count_of_issues, count_of_accounted_issues, count_of_entered_issues, value_0, value_1, value_2, value_3, value_4, value_5, value_6, value_7, value_8, value_9, value_10, value_11, value_12) VALUES (781,2018-11-12 08:15:32.0,2018-11-12 13:45:32.0, 2018, OCT-18, 10, null, 1, 1, 2073, ATRS, 6135, 6.2778220466107E11, 6.277946274560101E11, -1.2422795E7, 5.929031383587201E11, 5.9291556115366E11, -1.2422795E7, 3.931397937759116E13, 3.9313979377591164E13, 0.0)
But the destination table's columns:
run_id, count_of_issues, count_of_accounted_issues, count_of_entered_issues
are in numeric format (working on postgres db) and all others are varchar(1000).
In order to insert the varchar data, I need to enclose the column values from value_0 till value_12 in double quotes.
Without properly modifying the data, I am getting SQLException while inserting which is expected.
Is there any way I can just enclose those varchar column values from the resultSet in double quotes and insert them into destination table?
You need to add double quotes to the string values; Use Escape characters, like this:
String quotedStr = "Non Quoted," + " \"Quoted\".";
Modifying your code to address the problem:
while (rs.next()) {
queryValues = " ";
int i = 0;
for(String colname: columnNames) {
if(queryValues.isEmpty()) {
queryValues = rs.getString(colname);
} else {
if (i >= 7)
queryValues += "\"" + rs.getString(colname) +"\"" + ", ";
}
else
queryValues += rs.getString(colname) + ", ";
i++;
}
Connection conn = SqlConnection.getConnection("jdbc:mysql://localhost:3306/stocks");
Statement statement = conn.createStatement();
File path = new File("/Users/Zack/Desktop/JavaDB/BALANCESHEETS");
for(File file: path.listFiles()) {
if (file.isFile()) {
String fileName = file.getName();
String ticker = fileName.split("\\_")[0];
if (ticker.equals("ASB") || ticker.equals("FRC")) {
if (ticker.equals("ASB")) {
ticker = ticker + "PRD";
}
if (ticker.equals("FRC")) {
ticker = ticker + "PRD";
}
}
//CSVReader reader = new CSVReader(new FileReader(file));
//List entries = reader.readAll();
//ArrayList<String> entry = new ArrayList<String>();
Reader reader = new BufferedReader(new FileReader(file));
StringBuilder builder = new StringBuilder();
int c;
while ((c = reader.read()) != -1) {
builder.append((char) c);
}
String string = builder.toString();
ArrayList<String> stringResult = new ArrayList<String>();
if (string != null) {
String[] splitData = string.split("\\s*,\\s*|\\n");
for (int i = 0; i <splitData.length; i++) {
if (!(splitData[i] == null) || !(splitData[i].length() ==0)) {
stringResult.add(splitData[i].trim());
}
}
}
String columnName = null;
int yearCount = 0;
for (int i = 0; i < stringResult.size(); i++) {
int sL = stringResult.get(i).length();
for (int x = 0; x < sL; x++) {
if (Character.isLetter(stringResult.get(i).charAt(x))) {
yearCount = 0;
System.out.println("index: " + i);
columnName = stringResult.get(i);
columnName = columnName.replace(" ", "_");
System.out.println(columnName);
i++;
break;
}
}
yearCount++;
String value = stringResult.get(i);
System.out.println("Year: " + stringResult.get(yearCount) + " Value: " + value + " Stock: " + ticker + " Column: " + columnName );
if (!(columnName == null)) {
String writeValues = "INSERT INTO BalanceSheet (ticker, Year, " + columnName + ") "
+ "VALUE ('" + ticker + "','" + stringResult.get(yearCount) + "','" + value + "')";
String writeValues2 = "UPDATE BalanceSheet "
+ "SET ticker = '" + ticker + "', "
+ "Year = '" + stringResult.get(yearCount) + "', "
+ columnName + " = '" + value + "' "
+ "WHERE ticker = '" + ticker + "'";
statement.executeUpdate(writeValues2);
}
}
Towards the bottom of the code are two queries I tried, I'm trying to get all data organized by ticker and year into a table, "writeColumns" works but it's making a new row for every new "value" put into "columnName". My second attempt "writeColumns2" doesn't do anything.
I want to update the same row with a certain year for all values and then move onto the next year, then next ticker.
If I have understood your question correctly, you want to insert a row if it doesn't exists but update the values if it already does. You need to use ON DUPLICATE KEY UPDATE
String writeValues = "INSERT INTO BalanceSheet (ticker, Year, " + columnName + ") "
+ "VALUES (?,?,?) "
+"ON DUPLICATE KEY UPDATE " + columnName +"=?";
Statement statement = conn.prepareStatement(writeValues);
statement.setString(1,ticker);
statement.setString(2,stringResult.get(yearCount));
statement.setString(3, value);
This will solve your immidiate problem provided you create a UNIQUE index on ticker,year
However there are lot's of other issues here.
An update for each column - Currently you are doing an insert/update for each column on the table. What you are supposed to do is to insert update all the columns at one.
You are not using prepared statements addressed in my code sample
You shouldn't be doing this at all the best way to batch process data is to use MYSQL's built in LOAD DATA INFILE command. If your data is not in a format that can be easily imported into mysql, what your Java code can do is to transform it into a format that can be. Such a code will be a lot simpler and neater than what you have now
I have a method to show the values of table that satisfy the condition: show all value in table tblFuel if alarmType != 60 and nearest alarmType == 60, all of my code here:
public ArrayList<FuelData> BaoCaoDoXang(String accountID, String deviceID,
String fromTime, String toTime, String timezone)
throws DBException, SQLException {
ArrayList<FuelData> list = new ArrayList<FuelData>();
ResultSet rs = null;
DBConnection dbc = null;
Statement stmt = null;
FuelData reportFuel;
int alarmType = -1;//(1)initialize here*********************************************************1
try {
dbc = DBConnection.getDefaultConnection();
long epf = ConvertToEpoch(fromTime + " 00:00:00", timezone);
long epl = ConvertToEpoch(toTime + " 23:59:59", timezone);
String sql = "SELECT * "
+ " FROM tblFuel F INNER JOIN tblDoXang DX "
+ " ON DX.thoiGian = F.timestamp "
+ " where (F.timestamp BETWEEN " + epf + " and " + epl
+ ") " + " and F.accountID = '" + accountID
+ "' and F.deviceID = '" + deviceID
+ "' and F.alarmType = '" + alarmType//expectation value(4)*************************4
+ "' order by F.timestamp asc";
stmt = dbc.execute(sql);
rs = stmt.getResultSet();
double lastValue = 0;
int temp = -1;
while (rs.next()) {
// double fuel = rs.getDouble("nhienLieu");
temp = rs.getInt("alarmType");
if(temp != 60)
alarmType = temp;
if(temp == 60 && alarmType != -1)//(2)alarmType value after calculation*****************2
{
double currentValue = rs.getDouble("fuelLevel");
double changeValue = lastValue == 0 ? 0 : currentValue
- lastValue;
lastValue = currentValue;
reportFuel = new FuelData(rs.getString("accountID"),
rs.getString("deviceID"), rs.getInt("timestamp"),
rs.getDouble("latitude"), rs.getDouble("longitude"),
rs.getString("address"), currentValue,
rs.getDouble("odometerKM"), rs.getInt("status"),
changeValue,alarmType ,//(3)here************************************************3
rs.getDouble("nhienLieu"));
list.add(reportFuel);
/*
* if(fuel > 0){ changeValue = fuel; }
*/
}
}
} catch (SQLException sqe) {
throw new DBException("ReportByStatusCode", sqe);
} finally {
if (rs != null) {
try {
rs.close();
} catch (Throwable t) {
}
}
if (stmt != null) {
try {
stmt.close();
} catch (Throwable t) {
}
}
DBConnection.release(dbc);
}
return list;
}
I initialized my variable(alarmType) in (1) int alarmType = -1;//(1)initialize here*********************************************************1
and then checked the given condition and calculated in (2) and got the satisfied value in (3). I expected to put value of alarmType in (3)
double lastValue = 0;
int temp = -1;
while (rs.next()) {
// double fuel = rs.getDouble("nhienLieu");
temp = rs.getInt("alarmType");
if(temp != 60)
alarmType = temp;
if(temp == 60 && alarmType != -1)//(2)alarmType value after calculation*****************2
{
double currentValue = rs.getDouble("fuelLevel");
double changeValue = lastValue == 0 ? 0 : currentValue
- lastValue;
lastValue = currentValue;
reportFuel = new FuelData(rs.getString("accountID"),
rs.getString("deviceID"), rs.getInt("timestamp"),
rs.getDouble("latitude"), rs.getDouble("longitude"),
rs.getString("address"), currentValue,
rs.getDouble("odometerKM"), rs.getInt("status"),
changeValue,alarmType ,//(3)here************************************************3
rs.getDouble("nhienLieu"));
list.add(reportFuel);
/*
* if(fuel > 0){ changeValue = fuel; }
*/
}
}
to (4) and then execute the query
String sql = "SELECT * "
+ " FROM tblFuel F INNER JOIN tblDoXang DX "
+ " ON DX.thoiGian = F.timestamp "
+ " where (F.timestamp BETWEEN " + epf + " and " + epl
+ ") " + " and F.accountID = '" + accountID
+ "' and F.deviceID = '" + deviceID
+ "' and F.alarmType = '" + alarmType//expectation value(4)*************************4
+ "' order by F.timestamp asc";
. I have checked and recognized that the value return in (3) is true, however it seemed that the value in (4)(alarmType) which got from (1)(alarmType = -1), but not from (3) so my code return the wrong result. So my question is: how to retrieve correctly value from (3) to put to (4)?(In the other word: how to get value of alarmtype in while loop to put in in SQL query?).(other way) It is very easy if I add one more column in my table to put the values that calculated in (3)(by writing a service in somewhere), but what I can do without of doing that?
Do something like
alarmType
loopingConstruct {
sql String
execute sql string to get result set (rs)
loop through rs
get alarm type value and set it to alarm type
create FuelData object
add FuelData to the list
}
Looping construct context holds the modified value of alarmType during its execution which will be used in sql String.
Assumption - Result set (rs) returned contains only one row. If this is not the case then alarmType will have its value set to the value returned by the last row in the rs.
every time this starts my program freezes, and I can't figure out why.
It doesn't give any errors, it just freezes.
Is it possible I've created some kind of endless loop?
public static String[] DataVoorList(int coureur) throws SQLException{
ArrayList datalijst = new ArrayList();
String query = ""
+ "SELECT rd_datum, rd_locatie, rd_code "
+ "FROM racedag WHERE rd_code in( "
+ "SELECT i_rd_code "
+ "FROM inschrijvingen "
+ "WHERE i_c_nummer = " + coureur + ");";
ResultSet rs = Database.executeSelectQuery(query);
int i=0;
while (rs.next()){
String datum = rs.getString("rd_datum");
String locatie = rs.getString("rd_locatie");
String totaal = "" + datum + " - " + locatie;
datalijst.add(i, totaal);
i++;
int codeInt = rs.getInt("rd_code");
String code = ""+codeInt;
datalijst.add(i, code);
i++;
}
return Race.StringDataVoorList(datalijst);
}
public static String[] StringDataVoorList(ArrayList invoer){
int lengte = invoer.size();
String[] uitvoer = new String[lengte];
int i =0;
while (i < uitvoer.length){
uitvoer[i] = ""+invoer.get(i);
}
return uitvoer;
}
EDIT: I've solved the increment. However, it still freezes.
EDIT 2: I think I have located the problem (but I can be wrong)
public static String[] DataVoorList(int coureur) throws SQLException {
System.out.println("stap 1");
ArrayList datalijst = new ArrayList();
String query = ""
+ "SELECT rd_datum, rd_locatie, rd_code "
+ "FROM racedag WHERE rd_code in( "
+ "SELECT i_rd_code "
+ "FROM Inschrijvingen "
+ "WHERE i_c_nummer = " + coureur + ");";
ResultSet rs = Database.executeSelectQuery(query);
System.out.println("stap 2");
int i = 0;
while (rs.next()) {
String datum = rs.getString("rd_datum");
String locatie = rs.getString("rd_locatie");
String totaal = "" + datum + " - " + locatie;
datalijst.add(i, totaal);
System.out.println("stap 3");
i++;
int codeInt = rs.getInt("rd_code");
String code = "" + codeInt;
datalijst.add(i, code);
i++;
System.out.println("stap 4");
}
return Race.StringDataVoorList(datalijst);
(I've changed the while loop to a for loop)
public static String[] StringDataVoorList(ArrayList invoer) {
int lengte = invoer.size();
String[] uitvoer = new String[lengte];
for (int i = 0; i < uitvoer.length; i++) {
uitvoer[i] = "" + invoer.get(i);
}
return uitvoer;
}
}
this is being called from here:
public MijnRacedagenScherm() throws SQLException{
initComponents();
int gebruiker = Inloggen.getNummer();
String[] DataVoorList = Race.DataVoorList(2);
int lengte = DataVoorList.length;
System.out.println("resultaat is " + DataVoorList[0]);
int i = 0;
while (i < lengte) {
ListRacedagenCoureur.setListData(DataVoorList);
i = i + 2;
}
System.out.println("lengte is " + lengte);
}
This is a new screen, but in the previous screen I get a unreported SQL exception over this:
private void ButtonZienRacedagActionPerformed(java.awt.event.ActionEvent evt) {
new MijnRacedagenScherm().setVisible(true);
}
Well, um... In this section:
while (i < uitvoer.length){
uitvoer[i] = ""+invoer.get(i);
}
Where is i incremented?
Indeed it is, this
int i =0;
while (i < uitvoer.length){
uitvoer[i] = ""+invoer.get(i);
}
You never increment i.
As stated problem is in your while loop.
for loop is more suitable for iterating over indexed data type
for (int i = 0; i < uitvoer.length; i++) {
uitvoer[i] = ""+invoer.get(i);
}
How many rows are you processing? The way you append strings is quite slow, maybe it's not freezing anymore but just taking a long time to complete.