This question might be independent from the Spark Streaming framework, but I might as well mention that I'm using it.
I am running a very simple Spark Streaming program in which I have a DStream in which each RDD is a JSON line with an 'id','value' and 'timestamp'. Each time that I receive a Json line, I store the value of these attributes in static java objects.
So, something like this:
//Outside of main()
...
private static String idpoint;
private static String value;
//Inside main()
...
//Inside RDD function
String idpoint=id;
String value=c;
Date date= new Date();
SimpleDateFormat sdf=new SimpleDateFormat("MM/dd/yyyy h:mm:ss a");
String formatedDate= sdf.format(date);
String final=idpoint+","+"value";
My goal now is to store each line in a table from my MySQL database, one JSON line per table line.
I already created a method to download the JDBC drive outside the main():
public static void sqlcode()
{
try
{
// Step 1: "Load" the JDBC driver
Class.forName("com.imaginary.sql.msql.MsqlDriver");
// Step 2: Establish the connection to the database
String url = "myjdbc string";
Connection conn =
DriverManager.getConnection(url,"user","password");
}
catch (Exception e)
{
System.err.println("D'oh! Got an exception!");
System.err.println(e.getMessage());
}
}
But if I declare this method in main(), it keeps reopening the connection to the database, since in Spark Streaming the main() method basicaly works like a while loop.
So how can I save these java objects to a MySQL table?
Hope I wasn't too confusing.
Thank you.
Related
My project is using Oracle database, and everything works just fine. For the purpose of testing I set up an H2 db. The following query now throws an error:
"SELECT * FROM ERESIS.ECH_HISFAB f WHERE f.FG_ETAT = 'A' AND TO_DATE(DT_INS) > '30-AUG-18' ORDER BY f.CD_MAT"
error:
Cannot parse "TIMESTAMP" constant "30-AUG-18"; SQL statement:
I can fix the error by setting up the string like TO_DATE('30-AUG-2018'), but changing the query kind of defeats the purpose since I already am sure the query works (but I need it to test the service). Is there any way to bypass this error without changing the query?
parsedatetime() should be able to convert string to TIMESTAMP, please try using -
"SELECT * FROM ERESIS.ECH_HISFAB f WHERE f.FG_ETAT = 'A' AND TO_DATE(DT_INS) > parsedatetime('30-AUG-2018', 'dd-MMM-yyyy') ORDER BY f.CD_MAT"
I had a similar issue with H2 (1.4.200), it has just one format for TO_DATE(input) method "DD MON YYYY".
After a debug I found that H2 uses an enum for date format when it's not provided as second parameter: org.h2.expression.function.ToDateParser.ConfigParam
TO_DATE("DD MON YYYY")
I did a solution to override it using reflection so that the sql code does not change.
In my unit test I created a utility method used to instantiate the workaround on #BeforeClass
private static boolean h2WorkaroundApplied = false; //utility to apply workaround just one time
protected static synchronized void applyH2ToOracleCompatibilityWorkaround() {
if (!h2WorkaroundApplied) {
fixH2ToDateFormat(); //apply to_date workaround
h2WorkaroundApplied = true; //disable future changes of same workaround
}
}
private static void fixH2ToDateFormat() {
try {
Class<?> classConfigParam = Arrays.stream(ToDateParser.class.getDeclaredClasses())
.filter(c -> "ConfigParam".equals(c.getSimpleName()))
.findFirst()
.orElseThrow(); //get the enum inner class
Object toDateEnumConstant = Arrays.stream(classConfigParam.getEnumConstants())
.filter(e -> "TO_DATE".equals(((Enum) e).name()))
.findFirst()
.orElseThrow(); //get the enum constant TO_DATE
Field defaultFormatStr = classConfigParam.getDeclaredField("defaultFormatStr"); //get the enum private field defaultFormatStr
defaultFormatStr.setAccessible(true);
defaultFormatStr.set(toDateEnumConstant, "YYYY-MM-DD"); //change with my date format used by production oracle DB.
} catch (Exception e) {
throw new UnsupportedOperationException(
"The current H2 version doesn't support this workaround. Tested with version 1.4.200", e);
}
}
I have an app that it stores data inside a database and shows them in a recyclerview. my recyclerview should contain images. I want the image URL's to be stored in the database. but I don't want to get the URL's from the user. I want to write the URL myself and inside the code and then insert them in the database. Is there a way to store data in database from inside the code and not from any edit text ?
Not sure that I understood you right but if you need to write strings hardcoded in code just make any class with only this strings as constants and write them into database on every open of application. But this is bad practice. You can just use it from that class. Something like YourConstantsClass.yourConstant;
I wrote the code like this:
public void insertwall(ModelWallpaper modelWallpaper) {
ContentValues c = new ContentValues();
c.put(Colwall_Url,modelWallpaper.getImageURL());
try {
this.getWritableDatabase().insert(TABLE_NAME, null, c);
} catch (Exception e) {
e.printStackTrace();
}
}
and in mainActivity:
wallpaperHelper = new WallpaperHelper(this) ;
wallpaperHelper.open();
ModelWallpaper modelWallpaper = new ModelWallpaper();
modelWallpaper.setImageURL("https://tpc.googlesyndication.com/pagead/imgad?id=CICAgKDb4cfN9wEQARgBMgjiJeT4IZ5nOw");
wallpaperHelper.insertwall(modelWallpaper);
wallpaperHelper.close();
Ok, so after trying really hard for some days i am still not able to make this work.
This is the problem: I have a JSP which hosts a Google chart that is going to be constructed from data that is going to be sent through a Servlet. I'm using the Google Visualization Java libraries to implement this servlet.
Then i have this helper function that, taking some data stored in a list of objects, constructs a datatable. Following is the class that implements said function:
public class DataTableGenerator {
public static DataTable generateDatatable(List<AccesoUrl> accesos) {
DataTable data = new DataTable();
ArrayList<ColumnDescription> cd = new ArrayList<>();
cd.add(new ColumnDescription("fecha", ValueType.DATE, "Fecha"));
cd.add(new ColumnDescription("navegador", ValueType.TEXT, "Navegador"));
cd.add(new ColumnDescription("ip", ValueType.TEXT, "IP"));
cd.add(new ColumnDescription("os", ValueType.TEXT, "Sistema Operativo"));
data.addColumns(cd);
try {
for(AccesoUrl acceso : accesos) {
GregorianCalendar calendario = new GregorianCalendar();
calendario.setTimeZone(TimeZone.getTimeZone("GMT"));
data.addRowFromValues(calendario, acceso.getNavegador(), acceso.getIp(), acceso.getSistemaoperativo());
}
} catch (TypeMismatchException e) {
System.out.println(e);
}
return data;
}
}
Now, this piece of code should work, but instead i am getting this exception on my web server:
com.google.visualization.datasource.base.TypeMismatchException: Value type mismatch.
at com.google.visualization.datasource.datatable.value.ValueType.createValue(Unknown Source)
at com.google.visualization.datasource.datatable.DataTable.addRowFromValues(Unknown Source)
I'm at wit's end now. I have tried every variation i could find on Google so that my JSP displays this data on a Google Table type of chart. I have tried sending a date as a string, a date as a string formatted as a javascript date (i.e "Date(2015,4,4)" or "new Date(2015,4,4)"). I have tried using the DateValue object that comes with the java visualization library to construct the date as well (i.e new DateValue(2015,4,4)). I have also tried passing a Java Date as well. Nothing works, everything throws a TypeMismatchException.
I am sure it's the date that's giving me trouble because as soon as i remove the date column from the datatable everything runs fine with no exception and i can get my data displayed on the Google chart table.
So can anybody please tell me what exactly do i have to do in my Java code to be able to construct a datatable with a date cell?
Even I faced same issue. But it seems addRowFromValues() doesn't support DateValue(). I siwtched to addRow(TableRow). Didnt see any error's.
You can use something like below,
Sample Code:
DataTable data = new DataTable();
ArrayList<ColumnDescription> cd = new ArrayList<ColumnDescription>();
cd.add(new ColumnDescription("date", ValueType.DATE, "Executed Date"));
cd.add(new ColumnDescription("value", ValueType.NUMBER, "Duration "));
data.addColumns(cd);
TableRow tr = new TableRow();
tr.addCell(new DateValue(2015,7,7));
data.addRow(tr);
I'm using the Domino Java API to query a database on a remote server. The server is processing the documents, and I'm trying to get their status. However, when I create the session, and run a query, even if I loop and check again every 30 seconds, my code will never see those documents update- it only sees the status at the time it created the first query. I have a few more loops, but the basic code outline is below- can someone tell me what I'm doing wrong?
Is there a way to update the current Database view from the Java API? The databases are not full text indexed, and cannot be due to outside constraints.
public static boolean queryDatabase(String adminFilePath, String targetItem)
NotesThread.sinitThread();
Session session =NotesFactory.createSession((String) null, (String) null, (String) null);
Registration Reg = s.createRegistration();
Reg.switchToID(adminFilePath, password);
DocumentCollection dc = getRecentDocsFromDB(session);
numResults=dc.getCount();
if (numResults > 0) {
//loop through documents to find what I'm looking for
//if the documents contain "done", finish, else:
Thread.sleep(60000);
session.recycle();
session=SessionFactory.newSession(adminFilePath, "password");
dc = getRecentDocsFromDB(session);
found = searchDocumentCollection(dc, targetItem);
//this is essentially doing the same thing again- create a session, get docs made in the
//past day or so, and loop through looking for the ones I need.
}
private static DocumentCollection getRecentDocsFromDB(Session session){
Database db = SessionFactory.openDatabase(session, server, database);
Calendar cal = Calendar.getInstance();
cal.add(Calendar.DATE, -1);
DateTime dt = session.createDateTime(cal);
DocumentCollection dc = searchNotesDBUsingDate(session, db,"Form=\"event\"", dt);
}
public static DocumentCollection searchNotesDBUsingDate(Session session,
Database database, String query, DateTime dt) throws NotesException {
DocumentCollection dc = null;
dc = database.search(query, dt);
return dc;
}
I've updated the code with a session.recycle() call. (Thanks for the suggestion!) In testing, it's not having any effect- the code is working for the first document, but then never sees a second document being called.
It's insane, because it seems to be caching the session anyway!
I tried to reproduce the problem, but I wasn't able to. In my tests, Database.search() always returns the latest documents -- even if a document is added after the database is opened. I suspect there is a subtle problem in your code (perhaps what Richard Schwartz suggested in his comment).
It may or may not be relevant, but I wasn't able to compile your version of getRecentDocsFromDB() because I don't have a SessionFactory class. My version just uses Session.getDatabase() like so:
private static DocumentCollection getRecentDocsFromDB(Session session, String server, String database) throws NotesException {
Database db = session.getDatabase(server, database);
Calendar cal = Calendar.getInstance();
cal.add(Calendar.DATE, -1);
DateTime dt = session.createDateTime(cal);
DocumentCollection dc = searchNotesDBUsingDate(session, db,"Form=\"event\"", dt);
return dc;
}
Also, as Richard mentioned, you are not reading a view. You are searching the database for all documents created (or modified) by the "event" form in the last 24 hours. And you are doing this in a tight loop. Even if you get it to work, this approach isn't the best for production. You might want to research the use of lotus.domino.View and lotus.domino.ViewNavigator.
I am new at java swing because of that in my project i couldn't connect and get data from Microsoft sql server to Jtable.
But in this time I have just done it but I still interesting that if I don't use code, if I want to use that without code what should I do? In 'Netbeans', if you click table contents there is a way for import data to form. that is the point what I couldn't.
I tried to use this button(import data to form) and I showed my sqlserver.jar and I couldn't.
I think is that what you want:
http://fahdshariff.blogspot.com.br/2010/02/display-any-resultset-in-jtable.html
But in java, you will not get without some code.
I ran into a similar problem and the code below helped so I could retrieve the desired data. Edit the code so it will get the information you want it to retrieve.
private class RemoveBookings implements ActionListener {
String bookingID;
#Override
public void actionPerformed(ActionEvent ae) {
if (theMainViewDisplay.bookingData.getSelectedRow() != -1) {
final DefaultTableModel model = (DefaultTableModel) theMainViewDisplay.bookingData.getModel();
// Get stored varible
int index = theMainViewDisplay.bookingData.getSelectedRow();
System.out.println(index);
theMainViewDisplay.bookingID = model.getValueAt(index, 0).toString();
theMainViewDisplay.pickUpLocation = model.getValueAt(index, 2).toString();
theMainViewDisplay.dropOffLocation = model.getValueAt(index, 3).toString();
System.out.println(theMainViewDisplay.bookingID);
// Remove booking from database
model.removeRow(theMainViewDisplay.bookingData.getSelectedRow());
theBackendModel.thePowList.remove(bookingID);
}
}
}