We Keep Coding

FileUtils.copyURLToFile Getting Stuck

I'm using a Jave program to get NSE share price data from NSE's website like this for example:
url = new URL("https://archives.nseindia.com/archives/equities/bhavcopy/pr/PR071122.zip");
f = new File("NSEData.zip");
try {
FileUtils.copyURLToFile(url, f);
} catch (Exception e) {
}
The above code works for dates where market data exists, like 07/11/22 . However, where data does not exist, like on 08/11/22, the url is broken and the copyURLToFile line gets stuck indefinitely during runtime (replacing 071122 with 081122 in the url/code above will cause it to get stuck). Is there an easy way to get the program to recognize that the url for a certain date is broken (eg. https://archives.nseindia.com/archives/equities/bhavcopy/pr/PR081122.zip) and therefore ignore and continue past the try block without getting stuck?
My current workaround is to check whether a certain date is a market holiday using a DayOfWeek check as well as a HashSet containing a list of public holidays, but this is not perfect.

So, basically your URL is returning 500 error upon requesting for invalid date. You can simply use the another method available in FileUtils
https://commons.apache.org/proper/commons-io/javadocs/api-2.5/org/apache/commons/io/FileUtils.html#copyURLToFile(java.net.URL,%20java.io.File,%20int,%20int)
Example code : (Adjust timeouts as per your requirement)
var url = new URL("https://archives.nseindia.com/archives/equities/bhavcopy/pr/PR081122.zip");
var f = new File("NSEData.zip");
try {
FileUtils.copyURLToFile(url, f, 5000, 5000);
} catch (Exception e) {
}

Copying file or Replacing SQLite database is working fine in all versions of Android except Android Pie [duplicate]

I have a database saved in my apps assets folder and I copy the database using the below code when the app first opens.
inputStream = mContext.getAssets().open(Utils.getDatabaseName());
if(inputStream != null) {
int mFileLength = inputStream.available();
String filePath = mContext.getDatabasePath(Utils.getDatabaseName()).getAbsolutePath();
// Save the downloaded file
output = new FileOutputStream(filePath);
byte data[] = new byte[1024];
long total = 0;
int count;
while ((count = inputStream.read(data)) != -1) {
total += count;
if(mFileLength != -1) {
// Publish the progress
publishProgress((int) (total * 100 / mFileLength));
}
output.write(data, 0, count);
}
return true;
}
The above code runs without problem but when you try to query the database you get an SQLite: No such table exception.
This issue only occurs in Android P, all earlier versions of Android work correctly.
Is this a known issue with Android P or has something changed?

Was having a similar issue, and solved this adding this to my SQLiteOpenHelper
#Override
public void onOpen(SQLiteDatabase db) {
super.onOpen(db);
db.disableWriteAheadLogging();
}
Apparently Android P sets the PRAGMA Log thing different. Still no idea if will have side effects, but seems to be working!

My issues with Android P got solved by adding
'this.close()' after this.getReadableDatabase() in createDataBase() method as below.
private void createDataBase() throws IOException {
this.getReadableDatabase();
this.close();
try {
copyDataBase();
} catch (IOException e) {
throw new RuntimeException(e);
}
}

This issue seems to lead to a crash much more often on Android P than on previous versions, but it's not a bug on Android P itself.
The problem is that your line where you assign the value to your String filePath opens a connection to the database that remains open when you copy the file from assets.
To fix the problem, replace the line
String filePath = mContext.getDatabasePath(Utils.getDatabaseName()).getAbsolutePath();
with code to get the file path value and then close the database:
MySQLiteOpenHelper helper = new MySQLiteOpenHelper();
SQLiteDatabase database = helper.getReadableDatabase();
String filePath = database.getPath();
database.close();
And also add an inner helper class:
class MySQLiteOpenHelper extends SQLiteOpenHelper {
MySQLiteOpenHelper(Context context, String databaseName) {
super(context, databaseName, null, 2);
}
#Override
public void onCreate(SQLiteDatabase db) {
}
#Override
public void onUpgrade(SQLiteDatabase db, int oldVersion, int newVersion) {
}
}

I ran into a similar issue. I was copying a database but not from an asset. What I found is that the problem had nothing to do with my database file copying code at all. Nor did it have to do with files left open, not closed, flushing or syncing. My code typically overwrites an existing unopen database. What appears to be new/diffferent with Android Pie and different from previous releases of Android, is that when Android Pie creates a SQLite database, it sets journal_mode to WAL (write-ahead logging), by default. I've never used WAL mode and the SQLite docs say that journal_mode should be DELETE by default. The problem is if I overwrite an existing database file, let's call it my.db, the write-ahead log, my.db-wal, still exists and effectively "overrides" what's in the newly copied my.db file. When I opened my database, the sqlite_master table typically only contained a row for android_metadata. All the tables I was expecting were missing. My solution is to simply set journal_mode back to DELETE after opening the database, especially when creating a new database with Android Pie.
PRAGMA journal_mode=DELETE;
Perhaps WAL is better and there's probably some way to close the database so that the write-ahead log doesn't get in the way but I don't really need WAL and haven't needed it for all previous versions of Android.

Unfortunately, the accepted answer just "happens to work" in very concrete cases, but it doesn't give a consistently working advice to avoid such an error in Android 9.
Here it is:
Have single instance of SQLiteOpenHelper class in your application to access your database.
If you need to rewrite / copy the database, close the database (and close all connections to this database) using SQLiteOpenHelper.close() method of this instance AND don't use this SQLiteOpenHelper instance anymore.
After calling close(), not only all connections to the database are closed, but additional database log files are flushed to the main .sqlite file and deleted. So you have one database.sqlite file only, ready to be rewritten or copied.
After copying / rewriting etc. create a new singleton of the SQLiteOpenHelper, which getWritableDatabase() method will return new instance of the SQLite database! And use it till next time you will need your database to be copied / rewritten...
This answer helped me to figure that out: https://stackoverflow.com/a/35648781/297710
I had this problem in Android 9 in my AndStatus application https://github.com/andstatus/andstatus which has quite large suite of automated tests that consistently reproduced "SQLiteException: no such table" in Android 9 emulator before this commit:
https://github.com/andstatus/andstatus/commit/1e3ca0eee8c9fbb8f6326b72dc4c393143a70538 So if you're really curious, you can run All tests before and after this commit to see a difference.

Solution without disabling the WAL
Android 9 introduces a special mode of SQLiteDatabase called Compatibility WAL (write-ahead loggin) that allows a database to use "journal_mode=WAL" while preserving the behavior of keeping a maximum of one connection per database.
In Detail here:
https://source.android.com/devices/tech/perf/compatibility-wal
The SQLite WAL mode is explained in detail here:
https://www.sqlite.org/wal.html
As of the official docs the WAL mode adds a second database file called databasename and "-wal". So if your database is named "data.db" it is called "data-wal.db" in the same directory.
The solution is now to save and restore BOTH files (data.db and data-wal.db) on Android 9.
Afterwards it is working as in earlier versions.

I had the same thing I had an application in version 4 of android, and when updating my mobile that has android 9, then I was 2 days trying to find the error, thanks for the comments in my case I just had to add this.close ();
private void createDataBase () throws IOException {
     this.getReadableDatabase ();
     this.close ();
     try {
         copyDataBase ();
     } catch (IOException e) {
         throw new RuntimeException (e);
     }
}
ready running for all versions !!

First, thank you for posting this question. I had the same thing happen. All was working well, but then when testing against Android P Preview I was getting crashes. Here's the bug that I found for this code:
private void copyDatabase(File dbFile, String db_name) throws IOException{
InputStream is = null;
OutputStream os = null;
SQLiteDatabase db = context.openOrCreateDatabase(db_name, Context.MODE_PRIVATE, null);
db.close();
try {
is = context.getAssets().open(db_name);
os = new FileOutputStream(dbFile);
byte[] buffer = new byte[1024];
while (is.read(buffer) > 0) {
os.write(buffer);
}
} catch (IOException e) {
e.printStackTrace();
throw(e);
} finally {
try {
if (os != null) os.close();
if (is != null) is.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
The issue I ran into was this code works just fine BUT in SDK 28+ openOrCreateDatabase no longer automatically creates the android_metadata table for you. So if you do a query of "select * from TABLE" it will not find that TABLE because the query starts to look after the "first" table which should be the metadata table. I fixed this by manually adding the android_metadata table and all was well. Hope someone else finds this useful. It took forever to figure out because specific queries still worked fine.

Similar issue, only Android P device affected. All previous versions no problems.
Turned off auto restore on Android 9 devices.
We did this to troubleshoot. Would not recommend for production cases.
Auto restore was placing a copy of the database file in the data directory before the copy database function is called in the database helper. Therefore the a file.exists() returned true.
The database that was backed up from the development device was missing the table. Therefore "no table found" was in fact correct.

Here's the perfect solution for this problem:
Just override this method in your SQLiteOpenHelper class:
#Override
public void onOpen(SQLiteDatabase db) {
super.onOpen(db);
if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.JELLY_BEAN) {
db.disableWriteAheadLogging();
}
}

It seems that you don't close the output stream. While it probably does not explain why the db is not really created (unless Android P added a multi MB buffer) it is a good practice to use a try-with-resource, something like :
// garantees that the data are flushed and the resources freed
try (FileOutputStream output = new FileOutputStream(filePath)) {
byte data[] = new byte[1024];
long total = 0;
int count;
while ((count = inputStream.read(data)) != -1) {
total += count;
if (mFileLength != -1) {
// Publish the progress
publishProgress((int) (total * 100 / mFileLength));
}
output.write(data, 0, count);
}
// maybe a bit overkill
output.getFD().sync();
}

In version P, the major change is WAL (Write Ahead Log). The following two steps are required.
Disable the same by the following line in config.xml in the values folder under resources.
false
Make the following change in the DBAdapter class in createDatabase method. Otherwise phones with earlier Android versions crash.
private void createDataBase() throws IOException {
if (android.os.Build.VERSION.SDK_INT < android.os.Build.VERSION_CODES.P) {
this.getWritableDatabase();
try {
copyDataBase();
} catch (IOException e) {
throw new RuntimeException(e);
}
}
}

The issue occurring in Android Pie,
Solution is:
SQLiteDatabase db = this.getReadableDatabase();
if (db != null && db.isOpen())
db.close();
copyDataBase();

Simplest answer to use following line for Database file path in Android PIE and above:
DB_NAME="xyz.db";
DB_Path = "/data/data/" + BuildConfig.APPLICATION_ID + "/databases/"+DB_NAME;

Use SQLite for data collection

I'm quite new at android and SQL. I'm making an app, which can get data form the accelerometer, and the I'm storing them in a SQLLite database. Afterwards I intend to get the database out so I can plot the data. I have two questions:
How can I save the data as a usable file on the SD-card? I've seen some topics, but I couldn't get any off it to work. I think I need some examples/tutorials.
Secondly, after some collection, the app starts to lag. I guess it is the storing method, which is like this in the DB-class:
public long createEntry(float x, float y, float z, float t) {
ContentValues cv = new ContentValues();
cv.put(X_DATA,x);
cv.put(Y_DATA,y);
cv.put(Z_DATA,z);
cv.put(TIME_DATA,t);
return ourDatabase.insert(DATABASE_TABLE, null, cv);
}
I hope you'll help me.

For the first part of your question, performing all the inserts in a transaction should make it a lot faster. To do this, before you start inserting, call the following code:
ourDatabase.beginTransaction()
Then once you've finished inserting data, call:
ourDatabase.setTransactionSuccessful();
ourDatabase.getDb().endTransaction();
To answer the second part of question, you need to retrieve the data from the database and create a CSV file from it:
StringBuilder csv = new StringBuilder();
Cursor cursor = this.db.query("Data", new String[] { "x","y", "z", "t" }, null, null, null, null, null);
while (cursor.moveToNext()) {
csv.append(cursor.getFloat(0))
.append(",")
.append(cursor.getFloat(1))
.append(",")
.append(cursor.getFloat(2))
.append(",")
.append(cursor.getFloat(3))
.append("\n");
}
cursor.close();
File outputFile = new File("/sdcard/mycsv.csv");
FileWriter writer;
try {
writer = new FileWriter(outputFile);
writer.write(csv.toString());
writer.close();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
If you're collecting a lot of rows (tens of thousands perhaps), you might need to stream this CSV out to the disk at the same time as writing it.

Serialization and Data Structures

Hi all I need an advice.
I'll explain my problem. I want take data from web , elaborate the result, keep it and serialize on file.
I need to restore and use data from file , somehow. I dont want that file is been overwritten and lose old data.I need have a sort of list of Object in the File where i can search the last , use Method of another class to find some values etc , insert or similar, etc.....
In this case i used ArrayList but i dont know if it was the best choise.
I tried to do this but i have a problem know. In the specific if use a Class called Data and i want a serilizate file that keep ArrayList. So when i call method save(Object obj) of FileStructureClass(a class that i made to save, load file ) in this method i need to check if file already has an ArrayList so if it's true i can add in that Arraylist the Object , passed as parameter , else i return a new ArrayList. Of course i do a cast from Object to Data Class when i add in ArrayList.
I would fix this problem and then after find a better solution (if there is) to my problem.
The Data Class contains only 3 String and 1 GregoriusCalendar. Keep in mind(for the choise as ArrayList as Solution) that i need save file 1 time at day(i do a check with last element of the arraylist and do check with actual GregoriusCalendar..if past 1 day i can insert the element in arraylist).
After Explain the situation i list my problems
When i try to save for the FIRST time a Data Object in the Arraylist i have an error java.io.EOFException , i think that the problem is in tmp= ArrayList)ois.readObject(); but i cant find a solution. Dont happen when i insert manually a DataObject in the ArrayList and i use a method to insert a second one
According to you , ArrayList is a valid solution for my situation?
This method check if the file has data or not.
If it's empty i create a new one ArrayList and return it
otherwise i read the ArrayList already store in the file , and i return it
public ArrayList<Dati> check() {
ArrayList<Dati> tmp = new ArrayList<Dati>();
ObjectInputStream ois;
try{
fileInput = new FileInputStream("prova.dat");
ois = new ObjectInputStream(fileInput);
if (ois.readObject() == null) {
Logger.getLogger("file is empty");
ois.close();
return tmp;
}
//The error that i recive arrives from the under line
// (Impossible load file check method: java.io.EOFException
tmp = (ArrayList<Dati>) ois.readObject();
ois.close();
} catch (IOException e) {
System.out.println("Impossibile caricare i dati metodo check: "+e);
}
catch (ClassNotFoundException e) {
System.err.println("error");
}
return tmp;
}
//This method recive data of file that contains ArrayList<Data>
// and add to this a Data Object gave as Parameter
public void save(Object obj){
try{
ArrayList<Data> temp = check();
temp.add((Data) obj);
ObjectOutputStream os =
new ObjectOutputStream(new FileOutputStream("prova.dat"));
os.writeObject(temp);
os.flush();
os.close();
}
catch(IOException e){
System.out.println("Impossible save datas: "+e);
}
}
public Object load(String path){
Object obj=null;
try{
fileInput=new FileInputStream(path);
ois=new ObjectInputStream(fileInput);
obj=ois.readObject();
ois.close();
}
catch(IOException e){
System.out.println("Impossible load file: "+e);
}
catch(ClassNotFoundException e){
System.err.println();
}
return obj;
}

When i try to save for the FIRST time a Data Object in the Arraylist
i have an error java.io.EOFException , i think that the problem is
in tmp= (ArrayList)ois.readObject(); but i cant find a solution. Dont
happen when i insert manually a DataObject in the ArrayList and i
use a method to insert a second one
This seems to be correct. Looking at javadoc, it does seem like ois.readObject() returns null when there is nothing in the file. One approach I can think of is to initialize the file with an empty ArrayList in setup phase.
According to you , ArrayList is a valid solution for my situation?
I don't see why not. But it depends on what you will do with the list after reading it. If you have to search it often and it is large then you may consider a different data structure.

Remember the values entered on standalone app on the client side

We have a standalone java swing app, in which the user can print something that he drew, on a printer by giving its IP.
Now the requirement is that the app needs to remember the ip that was given the last time by this user.
What I could think of till now is (a brute one though) - keep a log file kind of storage on the client machine, and that everytime the app comes up it reads the last submitted one.
Any suggestions would be helpful.
Thanks in advance.

Here's a tutorial on using the Java Preferences API to achieve what you want.
From the article:
The Java Preferences API provides a
systematic way to handle user and
system preference and configuration
data, e.g. to save user settings,
remember the last value of a field
etc.
I would use this approach over writing any data out explicitly to a file because its platform agnostic.

More or Less that's it. Still you can review the source code for HistoryTextField component of jEdit.
http://www.jedit.org/api/org/gjt/sp/jedit/gui/HistoryTextField.html
A Sample from jEdit source:
public boolean save(Map<String, HistoryModel> models)
{
Log.log(Log.MESSAGE,HistoryModel.class,"Saving history");
File file1 = new File(MiscUtilities.constructPath(
jEdit.getSettingsDirectory(), "#history#save#"));
File file2 = new File(MiscUtilities.constructPath(
jEdit.getSettingsDirectory(), "history"));
if(file2.exists() && file2.lastModified() != historyModTime)
{
Log.log(Log.WARNING,HistoryModel.class,file2
+ " changed on disk; will not save history");
return false;
}
jEdit.backupSettingsFile(file2);
String lineSep = System.getProperty("line.separator");
BufferedWriter out = null;
try
{
out = new BufferedWriter(new OutputStreamWriter(
new FileOutputStream(file1), "UTF-8"));
if(models != null)
{
Collection<HistoryModel> values = models.values();
for (HistoryModel model : values)
{
if(model.getSize() == 0)
continue;
out.write('[');
out.write(StandardUtilities.charsToEscapes(
model.getName(),TO_ESCAPE));
out.write(']');
out.write(lineSep);
for(int i = 0; i < model.getSize(); i++)
{
out.write(StandardUtilities.charsToEscapes(
model.getItem(i),
TO_ESCAPE));
out.write(lineSep);
}
}
}
out.close();
/* to avoid data loss, only do this if the above
* completed successfully */
file2.delete();
file1.renameTo(file2);
}
catch(IOException io)
{
Log.log(Log.ERROR,HistoryModel.class,io);
}
finally
{
IOUtilities.closeQuietly(out);
}
historyModTime = file2.lastModified();
return true;
}

Since it is a Swing app., you might launch it using Java Web Start then persist the data using the PersistenceService. Here is a demo. of the PersistenceService.

i dont really recommend this, but you could use the registry also.