We Keep Coding

Why updating broadcast variable sample code didn't work?

I want to update broadcast variable every minute. So I use the sample code you give by Aastha in this question.
how can I update a broadcast variable in Spark streaming?
But it didn't work. The function updateAndGet() only works when the streaming application start. When I debug my code , it didn't went into the fuction updateAndGet() twice. So the broadcast variable didn't update every minute.
Why?
Here is my sample code.
public class BroadcastWrapper {
private Broadcast<List<String>> broadcastVar;
private Date lastUpdatedAt = Calendar.getInstance().getTime();
private static BroadcastWrapper obj = new BroadcastWrapper();
private BroadcastWrapper(){}
public static BroadcastWrapper getInstance() {
return obj;
}
public JavaSparkContext getSparkContext(SparkContext sc) {
JavaSparkContext jsc = JavaSparkContext.fromSparkContext(sc);
return jsc;
}
public Broadcast<List<String>> updateAndGet(JavaStreamingContext jsc) {
Date currentDate = Calendar.getInstance().getTime();
long diff = currentDate.getTime()-lastUpdatedAt.getTime();
if (broadcastVar == null || diff > 60000) { // Lets say we want to refresh every 1 min =
// 60000 ms
if (broadcastVar != null)
broadcastVar.unpersist();
lastUpdatedAt = new Date(System.currentTimeMillis());
// Your logic to refreshs
// List<String> data = getRefData();
List<String> data = new ArrayList<String>();
data.add("tang");
data.add("xiao");
data.add(String.valueOf(System.currentTimeMillis()));
broadcastVar = jsc.sparkContext().broadcast(data);
}
return broadcastVar;}}
//Here is the computing code submit to spark streaming.
lines.transform(new Function<JavaRDD<String>, JavaRDD<String>>() {
Broadcast<List<String>> blacklist =
BroadcastWrapper.getInstance().updateAndGet(jsc);
#Override
public JavaRDD<String> call(JavaRDD<String> rdd) {
JavaRDD<String> dd=rdd.filter(new Function<String, Boolean>() {
#Override
public Boolean call(String word) {
if (blacklist.getValue().contains(word)) {
return false;
} else {
return true;
}
}
});
return dd;
}});

Firebase with android 2 nodes with the same database reference, is it possible?

I am trying to avoid the asynchronous layout pefromaed naturally with Firebase. so I am trying to complete my method in the one dataonchanged method.
I need to know is it possible and how i can reference 2 nodes as you can see , using the one reference. iv tried something like
dataBaseLastIn = FirebaseDatabase.getInstance().getReference("clockInDate" + "clockOutDate");
//and
dataBaseLastIn = FirebaseDatabase.getInstance().getReference("clockInDate");
dataBaseLastIn = FirebaseDatabase.getInstance().getReference("clockOutDate");
//Method that works to set text to one reference but not both
protected void onStart() {
super.onStart();
final String uid = FirebaseAuth.getInstance().getCurrentUser().getUid();
dataBaseLastIn.addValueEventListener(new ValueEventListener(){
#Override
public void onDataChange(DataSnapshot dataSnapshot) {
dateinList.clear();
dateOutList.clear();
for (DataSnapshot postSnapshot : dataSnapshot.getChildren()) {
Date date = postSnapshot.getValue(Date.class);
String datt = date.getDate().toString().trim();
String user = date.getCurrUser().toString().trim();
if (DateNew.equals(datt) && user.equals(uid)) {
dateinList.add(date);
} else {
dateinList.remove(date);
}
DateOut dateOut = postSnapshot.getValue(DateOut.class);
String dattOut = dateOut.getDate().toString().trim();
String userOut = dateOut.getCurrUser().toString().trim();
if (DateNew.equals(dattOut) && userOut.equals(uid)) {
dateOutList.add(dateOut);
} else {
dateOutList.remove(date);
}
}
newList = dateinList;
outList = dateOutList;
try {
totalINList();
} catch (ParseException e) {
e.printStackTrace();
}
}
#Override
public void onCancelled(DatabaseError databaseError) {
}
});
}
//setting the text
public void totalINList() throws ParseException {
for(int i = 0 ; i < newList.size(); i++) {
timeDuration.add(newList.get(i).getClockTime());
}
Date e = newList.get(newList.size() - 1);
String timeIn = e.getClockTime();
txtCurrentTime.setText(timeIn);
for(int i = 0 ; i < outList.size(); i++) {
timeDurationOut.add(outList.get(i).getTime());
}
DateOut f = outList.get(outList.size() - 1);
String timeOut = f.getTime();
txtCurrentTimeOut.setText(timeOut);
}
May be something simple but i only get the value of the last reference i declare which obviously makes sense but was wondering is ther a way i can reference both nodes in the one

No, you cannot achieve something like with Firebase.
The initial response for most developers is to try and "fix" this asynchronous behavior, which I personally recommend against this. The web is asynchronous, and the sooner you accept that, the sooner you can learn how to become productive with modern web APIs.

Creating 140,000-row SQLite database in AsyncTask doInBackground taking many, many minutes

I haven't dealt with SQLite databases before last week. I last dealt with SQL many years ago, but I still have the gist of it.
The code below reads 140,000 words from an asset named dictionary.dic and inserts each into a SQLite database along with its status. My expectation was that it would take a good while, but it's been like 25 minutes on a 7" tablet and still not near finished (on P).
Should I say, "Hey, it's 1/7 of a million rows. It's gonna take awhile." But I can read all 140,000 words into an ArrayList<String> in 30 seconds. I realize there's overhead in creating the database, but many, many minutes?
Should I say, "Well, think how long it would take if not using AsyncTask" and accept it since it's a one-time task? But it's really obnoxious, taking so long. It's off-putting.
Should I say, "Why are you using a Scanner? No wonder it's taking so long?" and do some other asset access? Or is that not the real problem?
I also have never used AsyncTask. Am I misusing doInBackground? I've got a lot of code in there; not all MUST go there, but the loop is what it is and there's the hangup.
Is using database.Insert, which is called a "convenience method", what's causing the hangup? Should I be using a Cursor and query instead? I'm not entirely sure how I'd do that. Got my idea from Deitel's "Address Book" app in "Android for Programmers--App Driven...", but his database is empty at the outset.
I've given this plenty of thought. I just need someone with experience to look and say, "Well, HERE'S your problem." I can't justify starting redoing all the things I've thought of without some guidance about whether any of it is going to help.
public class DatabaseConnector //extends ArrayList<String>
{
public static Cursor cursor ;
Scanner scDict;
InputStream stream = null;
Context mContext;
AssetManager mAssets;
public static final String DATABASE_NAME = "Dictionary";
public static final String TABLE_NAME = "wordlist";
public static final String WORD_COLUMN_NAME = "word";
public static final String STATUS_COLUMN_NAME = "status";
public static final String [] columns = new String[]{WORD_COLUMN_NAME, STATUS_COLUMN_NAME};
private DatabaseOpenHelper ___databaseOpenHelper; // creates the database
private SQLiteDatabase ___database; // for interacting with the database
public DatabaseConnector(Context _context, AssetManager assets)
{
mContext = _context;
mAssets = assets;
___databaseOpenHelper = new DatabaseOpenHelper(_context, DATABASE_NAME, null, 1);
Log.w("DB connected", ___databaseOpenHelper.getDatabaseName());
createDbIfNecessary();
};
public void open() throws SQLException // opens/creates
{
___database = ___databaseOpenHelper.getWritableDatabase(); // create OR open
}
public void createDbIfNecessary(){
this.open();
if(getDbCount() < 140000){
try { stream = mAssets.open("dictionary.dic"); }
catch (IOException e) { System.out.println(Arrays.toString(e.getStackTrace())); }
MainActivity.setLblProgress("This one-time task takes awhile: loading letter... ");
LoadWords loadWords = new LoadWords();
loadWords.execute((Object[]) null);
this.close();
}
}
public void close(){
if(___database != null)
___database.close();
}
public int getDbCount(){
this.open();
return ___database.query(TABLE_NAME, columns, null, null, null, null, null).getCount();
}
public long insertWord(String _word)
{
ContentValues
__newWord;
__newWord = new ContentValues();
__newWord.put(WORD_COLUMN_NAME, _word);
__newWord.put(STATUS_COLUMN_NAME, true);
long __row = ___database.insert(TABLE_NAME, null, __newWord);
return __row; // -1 if can't insert
}
//////////////////////////////////////////////////////////////////////////////////////////////////
private class DatabaseOpenHelper extends SQLiteOpenHelper
{
public DatabaseOpenHelper(Context _context, String _name, CursorFactory _factory, int _version)
{ super(_context, _name, _factory, _version); }
#Override public void onCreate(SQLiteDatabase _db)
{
_db.execSQL( "CREATE TABLE " + TABLE_NAME +
"("
+ WORD_COLUMN_NAME + " TEXT primary key , " //not null, "
+ STATUS_COLUMN_NAME + " BOOLEAN" +
");"
); // execute query to create the ___database
}
} // end class DatabaseOpenHelper
//////////////////////////////////////////////////////////////////////////////////////////////////
private class LoadWords extends AsyncTask<Object, Integer, Void>
{
#Override
protected Void doInBackground(Object... params) {
long k = 0;
scDict = new Scanner(stream).useDelimiter("\r\n");
long count = getDbCount();
Log.w("Start load at " , "" + count);
String s = "";
while(k++ < count){
s = scDict.next();
}
Log.w("Add after " , s);
while (scDict.hasNext())
{
s = scDict.next();
publishProgress((Integer)(int)s.charAt(0));
insertWord(s) ;
return null;
}
protected void onProgressUpdate(Integer... progress) {
int c = (int)progress[0];
MainActivity.setLastLetterProcessed((char) c);
}
#Override
protected void onPostExecute(Void x)
{
MainActivity.popupMessage("Database has been created", mContext);
}
}
} // end class DatabaseConnector

You are attempting to do 140,000 individual database transactions. That might take weeks.
Instead, either wrap your entire thing in a single transaction, or batch the inserts into transactions (e.g., every 1000 words). You can create your own transaction bounds using this pseudo-Java:
db.beginTransaction();
try {
// do your SQL work here
db.setTransactionSuccesful();
}
catch (Exception e) {
// logging, event bus message to UI, whatever
}
finally {
db.endTransaction();
}

Thanks to #Commonsware, the 140,000 records now load in under a minute, as opposed to under an HOUR. All I did was use his "p-code" to surround my insert with a 1000-count loop:
protected Void doInBackground(Object... params) {
...
scDict = new Scanner(stream).useDelimiter("\r\n");
long count = getDbCount();
while (k++ < count)
s = scDict.next();
while (scDict.hasNext())
{
//////////////////////////////////////////////////////// start insert
int ki = 0;
try
{
___database.beginTransaction();
while (ki < MAX_TRANSACTIONS && scDict.hasNext())
{
//////////////////////////////////////////////////////// end
insertWord(scDict.next());
//////////////////////////////////////////////////////// start insert
++ki;
}
___database.setTransactionSuccessful();
}
catch(Exception e){ Log.w("Exception",e);}
finally
{
___database.endTransaction();
publishProgress((Integer) (int) s.charAt(0));
}
//////////////////////////////////////////////////////// end
}
return null;
}
...
}

Change Value of String Based on Media Metadata

I have a simple media player and in one of the classes I need to retrieve the metadata of the media. Thus far, I have successfully extracted the metadata, however when I attempt to use the values, the Strings returned are blank.
String album = "", artist = "", title = "", year = "";
...
public void addListItems(final Pane layout, final Stage stage, final Scene scene) {
String[] sList = s.split("\\\\");
try {
Media media = new Media(new File(s).toURI().toURL().toString());
media.getMetadata().addListener(new MapChangeListener<String, Object>() {
public void onChanged(Change<? extends String, ? extends Object> arg0) {
handleMetadata(arg0.getKey(), arg0.getValueAdded());
}
});
} catch (MalformedURLException e) {
e.printStackTrace();
}
System.out.println(album); //Here, album returns as ""
final Button lab = new Button(album + "--" + sList[sList.length - 1]);
}
...
public void handleMetadata(String key, Object value) {
if (key.equals("album")) {
album = value.toString(); //Here album returns as value.toString() correctly
} else if (key.equals("artist")) {
artist = value.toString();
} else if (key.equals("title")) {
title = value.toString();
} else if (key.equals("year")) {
year = value.toString();
}
}
I have done research on this topic, but the results I viewed did not help my circumstance. I greatly appreciate any assistance.

Java Vector<E> getting blocked with no apparent reason

I'm probably doing something wrong, but this is what happens:
I register a listener, wait for it to be called and when it executes and calls "Vector.clear()" it locks. The problem happens in the "pipeMsgEvent" method, here it is the, not so friendly-looking, code (if you want, go directly to "this is important" lines):
public class JxtaCustomer implements PipeMsgListener {
public static final int ANSWER_OK = 0;
public static final int ANSWER_TIMEOUT =-1;
public static final int ANSWER_NOT_ASKED =-2;
public static final int ANSWER_WORKING =-3;
public static final int REQ_SENT = 0;
public static final int REQ_INV_PIPE_ID =-1;
public static final int REQ_INV_PIPE_IO =-2;
public static final int REQ_INV_GROUP_ID =-3;
protected int answer;
protected JxtaGenericAdvertisement serviceAdv = null;
protected JxtaNode ThePeer = null;
JxtaBiDiPipe myBiDiPipe = null;
protected Vector<Entry> output = null;
public JxtaCustomer(JxtaGenericAdvertisement adv, JxtaNode peer)
{
ThePeer = peer;
serviceAdv = new JxtaGenericAdvertisement((Element)adv.getDocument(new MimeMediaType("text/xml")));
answer = ANSWER_NOT_ASKED;
Vector<Entry> output = new Vector<Entry>();
}
public void pipeMsgEvent(PipeMsgEvent event)
{
System.out.println("It Works! Uhuuu");
// We received a message
Message message = event.getMessage();
System.out.println("Lets GO! Uhuuu");
Message.ElementIterator elIt = message.getMessageElementsOfNamespace( serviceAdv.getName() );
System.out.println("A little bit more");
answer = ANSWER_WORKING;
System.out.println("enter");
// This is important: Here I get stuck, "clear"
// never appears on the screen
output.clear();
System.out.println("clear");
while (elIt.hasNext()) {
MessageElement mElem = elIt.next();
System.out.println(mElem.getElementName() + " " + mElem.toString());
output.add( new Entry( mElem.getElementName(), mElem.toString() ) );
}
System.out.println("leave");
answer = ANSWER_OK;
}
public int sendRequest(Vector<Entry> params)
{
try {
// Creating Pipe Advertisement
String pipeAdvType = PipeAdvertisement.getAdvertisementType();
PipeAdvertisement pipeAd = (PipeAdvertisement)AdvertisementFactory.newAdvertisement(pipeAdvType);
URI pipeURI = new URI(serviceAdv.getPipeID());
pipeAd.setPipeID( IDFactory.fromURI(pipeURI) );
pipeAd.setType(PipeService.UnicastType);
pipeAd.setName(serviceAdv.getName() + " Service Pipe");
pipeAd.setDescription("Created by " + serviceAdv.getName());
// Creating Group
JxtaGroup jxgp = ThePeer.getGroupFromID( serviceAdv.getGroupID() );
if (null == jxgp)
return REQ_INV_GROUP_ID;
PeerGroup group = jxgp.getPeerGroup();
if (null == group)
return REQ_INV_GROUP_ID;
// This is important: In the JxtaBiDiPipe call I register
// the callback, while I have access to the code, I think
// the problem is in my code
myBiDiPipe = new JxtaBiDiPipe( group, pipeAd, 30000, this);
if (myBiDiPipe.isBound()) {
Thread.sleep(500);
Message myMessage = new Message();
if (0 == params.size())
params.add( new Entry("dummy", "null") );
for (int i=0; i<params.size(); i++) {
String Key = params.get(i).getKey();
String Value = params.get(i).getValue();
StringMessageElement strMsgElem = new StringMessageElement( Key, Value, null);
myMessage.addMessageElement( serviceAdv.getName(), strMsgElem);
}
myBiDiPipe.sendMessage(myMessage);
answer = ANSWER_TIMEOUT;
return REQ_SENT;
}
} catch (URISyntaxException e){
e.printStackTrace();
return REQ_INV_PIPE_ID;
} catch (IOException e) {
return REQ_INV_PIPE_IO;
} catch (Exception e) {
e.printStackTrace();
}
return REQ_INV_PIPE_IO;
}
public Vector<Entry> getResponse()
{
Vector<Entry> results = new Vector<Entry>();
if (ANSWER_OK != answer)
return results;
// This is important: I always call "getState()" and check its value
// before calling this method, so you can say it's guaranteed to be
// called after answer = ANSWER_OK, which never happens because of the
// lock
for (int i=0; i<output.size(); i++)
results.add( output.get(i) );
return results;
}
public int getState()
{
int count = 10;
return answer;
}
}
I always create JxtaCustomer like this:
JxtaCustomer customer = new JxtaCustomer(adv, ThePeer);
Vector<Entry> params = new Vector<Entry>();
params.add( new Entry("key1", "value1") );
params.add( new Entry("key2", "value2") );
customer.sendRequest(params);
while(JxtaCustomer.ANSWER_OK != customer.getState()) {
// I was actually using synchronized locks and timed waits
// but since I'm stuck there I'm using this nonsense instead
}
Vector<Entry> response = customer.getResponse();
Here is where in the code I call "sleep", "wait" or I create a "synchronized" (this last one is never)
$ grep synchronized $(find . -name "*.java")
$ grep sleep $(find . -name "*.java")
./Src/net/ubi/jxta/JxtaCustomer.java: Thread.sleep(500);
$ grep wait $(find . -name "*.java")
./App/ExampleServices/SimpleSigner.java: boolean waiting = false;
./App/ExampleServices/SimpleSigner.java: waiting = true;
./App/ExampleServices/SimpleSigner.java: return (waiting && JxtaCustomer.ANSWER_OK == customer.getState());
./App/ExampleServices/SimpleSigner.java: if (!waiting)
./App/ExampleServices/SimpleSigner.java: waiting = false;
./App/ExampleServices/SimpleSigner.java: return "Error: Response not ready, wait more time\n";

Your constructor sets a local variable called output,and not the member variable, you have
Vector<Entry> output = new Vector<Entry>();
when it should be
output = new Vector<Entry>();
So, later on when you call output.clear() you're getting an Null Pointer Exception thrown, which is why the following lines of code don't get executed.
compiler warnings like "unused local variables" can help spot these sorts of mistakes.