The database of my application need to be filled with a lot of data,
so during onCreate(), it's not only some create table sql
instructions, there is a lot of inserts. The solution I chose is to
store all this instructions in a sql file located in res/raw and which
is loaded with Resources.openRawResource(id).
It works well but I face to encoding issue, I have some accentuated
caharacters in the sql file which appears bad in my application. This
my code to do this:
public String getFileContent(Resources resources, int rawId) throws
IOException
{
InputStream is = resources.openRawResource(rawId);
int size = is.available();
// Read the entire asset into a local byte buffer.
byte[] buffer = new byte[size];
is.read(buffer);
is.close();
// Convert the buffer into a string.
return new String(buffer);
}
public void onCreate(SQLiteDatabase db) {
try {
// get file content
String sqlCode = getFileContent(mCtx.getResources(), R.raw.db_create);
// execute code
for (String sqlStatements : sqlCode.split(";"))
{
db.execSQL(sqlStatements);
}
Log.v("Creating database done.");
} catch (IOException e) {
// Should never happen!
Log.e("Error reading sql file " + e.getMessage(), e);
throw new RuntimeException(e);
} catch (SQLException e) {
Log.e("Error executing sql code " + e.getMessage(), e);
throw new RuntimeException(e);
}
The solution I found to avoid this is to load the sql instructions
from a huge static final String instead of a file, and all
accentuated characters appear well.
But isn't there a more elegant way to load sql instructions than a big
static final String attribute with all sql instructions?
I think your problem is in this line:
return new String(buffer);
You're converting the array of bytes in to a java.lang.String but you're not telling Java/Android the encoding to use. So the bytes for your accented characters aren't being converted correctly as the wrong encoding is being used.
If you use the String(byte[],<encoding>) constructor you can specify the encoding your file has and your characters will be converted correctly.
The SQL file solution seems perfect, it's just that you need to make sure that the file is saved in utf8 encoding otherwise all the accentuated characters will be lost. If you don't want to change the file's encoding then you need to pass an extra argument to new String(bytes, charset) defining the file's encoding.
Do prefer to use file resources instead of static final String to avoid having all those unnecessary bytes loaded into memory. In mobile phones you want to save all memory possible!
I am using a different approach:
Instead of executing loads of sql statements (which will take long time to complete), I build my sqlite database on the desktop, put it in the assets folder, create an empty sqlite db in android and copy the db from the assets folder into the database folder. This is a huge increase in speed. Note, you need to create an empty database first in android, and then you can copy and overwrite it. Otherwise, Android will not allow you to write a db into the datbase folder. There are several examples on the internet.
BTW, seems this approach works best, if the db has no file extension.
It looks like you are passing all your sql statements in one string. That's a problem because execSQL expects "a single statement that is not a query" (see documentation [here][1]). Following is a somewhat-ugly-but-working solution.
I have all my sql statements in a file like this:
INSERT INTO table1 VALUES (1, 2, 3);
INSERT INTO table1 VALUES (4, 5, 6);
INSERT INTO table1 VALUES (7, 8, 9);
Notice the new lines in between text(semicolon followed by 2 new lines)
Then, I do this:
String text = new String(buffer, "UTF-8");
for (String command : text.split(";\n\n")) {
try { command = command.trim();
//Log.d(TAG, "command: " + command);
if (command.length() > 0)
db.execSQL(command.trim());
}
catch(Exception e) {do whatever you need here}
My data columns contain blobs of text with new lines AND semicolons, so I had to find a different command-separator. Just be sure to get creative with the split str: use something you know doesn't exist in your data.
HTH
Gerardo
[1]: http://developer.android.com/reference/android/database/sqlite/SQLiteDatabase.html#execSQL(java.lang.String, java.lang.Object[])
Related
I have multiple 100MB raw files with series of user activities in CSV format. I only want to download the first 100 lines of the files.
The problem is that each file may have different CSV header columns, and data values, because they are user activities from multiple subdomains using different activity tracking providers. This means that each line can be 50 characters long or 500 characters long and it is unknown until I read them all.
S3 supports getObject API with Range parameter which you can use to download the specific ranges of XX bytes of the file.
https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObject.html#API_GetObject_RequestSyntax
If I use this API to parse first 1Mb of files, iterate each byte until I see 100 new lines character \n, would that technically work? Is there something that I have to be careful about this approach? (e.g. multibyte chars?)
There is no built-in way, byte-range fetches are the best way forward.
As you're not sure of the header or line length in each case, downloading 1MB chunks until you have 100 line is a safe & efficient approach.
Multibyte chars etc. won't be important as at this level, you're purely looking to stop reading after 100 \n characters. Depending on the source of your files, however, I would be also conscious of \r\n and \r as being valid line endings.
I've written the below Java code for getting the last n bytes, feel free to use it as a starting point for getting the first n bytes:
public String getLastBytesOfObjectAsString(String bucket, String key, long lastBytesCount) {
try {
final ObjectMetadata objectMetadata = client.getObjectMetadata(bucket, key);
final long fileSizeInBytes = objectMetadata.getContentLength();
long rangeStart = fileSizeInBytes - lastBytesCount;
if (rangeStart < 0) {
rangeStart = 0;
}
final GetObjectRequest getObjectRequest =
new GetObjectRequest(bucket, key).withRange(rangeStart);
try (S3Object s3Object = client.getObject(getObjectRequest);
InputStream inputStream = s3Object.getObjectContent()) {
return new String(inputStream.readAllBytes());
}
} catch (Exception ex) {
...
}
}
You can use smart_open like this:
from smart_open import open
with open('s3://bucket/path/file.csv', 'r') as f:
csv_reader = csv.DictReader(f, delimiter=',')
data = ''
for i, row in enumerate(csv_reader):
data += row +'\n'
if i > 100:
store(data)
You will need to open another file in your localmachine with write permission to store a 100 lines or as many as you like. If you want the 1st lines from multiple files, you can do the same but using the boto3 function for listing the files and send the path/file name to a function using smart_open.
s3client = boto3.client('s3')
listObj = s3client.list_objects_v2(Bucket=bucket, Prefix=prefix)
for obj in listObj['Contents']:
smart_function(obj['Key'])
The obj['Key'] contains the path and file name of each file in that Bucket+Path(Prefix)
So I've created this simple app that asks the user questions (it's like a truth or dare game).
The questions are saved in a .json-file in SD-Card > Android > data > [package name] > files.
I want to make it possible, however, to add or delete those questions by manually editing or exchanging that .json-file in which the questions are saved (so that those new questions then appear in the app).
I already understand that the app needs to call the getExternalFilesDir() method somewhere, so that a directory that I can find in the device's file manager is created in the first place. However - as you probably have guessed - when I open that directory, the "files" folder is empty. (I know the .json-files must be in there since the app is working fine!)
Now my question:
Is there any way I can view and edit those .json-files? I don't care if the solution includes using a PC or some program.
Or should I maybe not save the questions in a .json-file but in a .txt-file instead? Or should I use a different directory to save my question files to begin with??
I would appreciate a detailed answer, since I'm obviously new.
Thanks in advance!
P.S.: I can include code on request.
EDIT: I don't want to edit those files from code, I already know how to do that. My question in particular is about whether there is a possibility to edit the json files using a text editor for example or exchanging the files using copy paste.
Code:
public void copy() {
Context Context = getApplicationContext();
String DestinationFile = Context.getExternalFilesDir(null).getPath() + File.separator + "fragenAdvanced.json";
if (!new File(DestinationFile).exists()) {
try {
CopyFromAssetsToStorage(Context, "fragenAdvanced.json", DestinationFile);
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
private void CopyFromAssetsToStorage(Context Context, String SourceFile, String DestinationFile) throws IOException {
InputStream IS = Context.getAssets().open(SourceFile);
OutputStream OS = new FileOutputStream(DestinationFile);
CopyStream(IS, OS);
OS.flush();
OS.close();
IS.close();
}
private void CopyStream(InputStream Input, OutputStream Output) throws IOException { //brauche ich das?
byte[] buffer = new byte[5120];
int length = Input.read(buffer);
while (length > 0) {
Output.write(buffer, 0, length);
length = Input.read(buffer);
}
}
This is how I save the file to storage.
there is 3 way to save your user data but your way is not secure because the user has access on it.
the first way is saving user data in SharedPreferences , it's very simple to use , but if user clears the app catch, the data will be deleted ( you can just convert your json to string and save it in sharedPreferences , for converting string to json you can use Gson library )
second way is using SQlite and save data on local database , it will be deleted by clear stroge and still not good for important data
the final way is using remote database and get data from api
I am new to databases in Java and i am trying to export the data from 1 table and store it in a text file. At the moment the code below writes to the text file however all on one line? can anyone help?
My Code
private static String listHeader() {
String output = "Id Priority From Label Subject\n";
output += "== ======== ==== ===== =======\n";
return output;
}
public static String Export_Message_Emails() {
String output = listHeader();
output +="\n";
try {
ResultSet res = stmt.executeQuery("SELECT * from messages ORDER BY ID ASC");
while (res.next()) { // there is a result
output += formatListEntry(res);
output +="\n";
}
} catch (Exception e) {
System.out.println(e);
return null;
}
return output;
}
public void exportCode(String File1){
try {
if ("Messages".equals(nameOfFile)){
fw = new FileWriter(f);
//what needs to be written here
//fw.write(MessageData.listAll());
fw.write(MessageData.Export_Message_Emails());
fw.close();
}
}
Don't use a hard coded value of "\n". Instead use System.getProperty("line.separator"); or if you are using Java 7 or greater, you can use System.lineSeparator();
Try String.format("%n") instead "\n".
Unless you're trying to practice your Java programming (which is perfectly fine of course!), you can export all the data from one table and store it in a file by using the SYSCS_UTIL.SYSCS_EXPORT_TABLE system procedure: http://db.apache.org/derby/docs/10.11/ref/rrefexportproc.html
I'm gonna assume you are using Windows and that you are opening your file with notepad. If that is correct then it is not really a problem with your output but with the editor you are viewing it with.
Try a nicer editor, ie. Notepad++
Do as the other answers suggest and use System.getProperty("line.separator"); or similar.
Use a Writer implementation such as, PrintWriter.
Personally I prefer "\n" over the system line separator, which on Windows is "\r\n".
EDIT: Added option 3
I want to add a help screen to my Codename One App.
As the text is longer as other strings, I would like put it in a separate file and add it to the app-package.
How do I do this? Where do I put the text file, and how can I easily read it in one go into a string?
(I already know how to put the string into a text area inside a form)
In the Codename One Designer go to the data section and add a file.
You can just add the text there and fetch it using myResFile.getData("name");.
You can also store the file within the src directory and get it using Display.getInstance().getResourceAsStream("/filename.txt");
I prefer to have the text file in the filesystem instead of the resource editor, because I can just edit the text with the IDE. The method getResourceAsStream is the first part of the solution. The second part is to load the text in one go. There was no support for this in J2ME, you needed to read, handle buffers etc. yourself. Fortunately there is a utility method in codename one. So my working method now looks like this:
final String HelpTextFile = "/helptext.txt";
...
InputStream in = Display.getInstance().getResourceAsStream(
Form.class, HelpTextFile);
if (in != null){
try {
text = com.codename1.io.Util.readToString(in);
in.close();
} catch (IOException ex) {
System.out.println(ex);
text = "Read Error";
}
}
The following code worked for me.
//Gets a file system storage instance
FileSystemStorage inst = FileSystemStorage.getInstance();
//Gets CN1 home`
final String homePath = inst.getAppHomePath();
final char sep = inst.getFileSystemSeparator();
// Getting input stream of the file
InputStream is = inst.openInputStream(homePath + sep + "MyText.txt");
// CN1 Util class, readInputStream() returns byte array
byte[] b = Util.readInputStream(is);
String myString = new String(b);
I need to parse a java file (actually a .pdf) to an String and go back to a file. Between those process I'll apply some patches to the given string, but this is not important in this case.
I've developed the following JUnit test case:
String f1String=FileUtils.readFileToString(f1);
File temp=File.createTempFile("deleteme", "deleteme");
FileUtils.writeStringToFile(temp, f1String);
assertTrue(FileUtils.contentEquals(f1, temp));
This test converts a file to a string and writtes it back. However the test is failing.
I think it may be because of the encodings, but in FileUtils there is no much detailed info about this.
Anyone can help?
Thanks!
Added for further undestanding:
Why I need this?
I have very large pdfs in one machine, that are replicated in another one. The first one is in charge of creating those pdfs. Due to the low connectivity of the second machine and the big size of pdfs, I don't want to synch the whole pdfs, but only the changes done.
To create patches/apply them, I'm using the google library DiffMatchPatch. This library creates patches between two string. So I need to load a pdf to an string, apply a generated patch, and put it back to a file.
A PDF is not a text file. Decoding (into Java characters) and re-encoding of binary files that are not encoded text is asymmetrical. For example, if the input bytestream is invalid for the current encoding, you can be assured that it won't re-encode correctly. In short - don't do that. Use readFileToByteArray and writeByteArrayToFile instead.
Just a few thoughts:
There might actually some BOM (byte order mark) bytes in one of the files that either gets stripped when reading or added during writing. Is there a difference in the file size (if it is the BOM the difference should be 2 or 3 bytes)?
The line breaks might not match, depending which system the files are created on, i.e. one might have CR LF while the other only has LF or CR. (1 byte difference per line break)
According to the JavaDoc both methods should use the default encoding of the JVM, which should be the same for both operations. However, try and test with an explicitly set encoding (JVM's default encoding would be queried using System.getProperty("file.encoding")).
Ed Staub awnser points why my solution is not working and he suggested using bytes instead of Strings. In my case I need an String, so the final working solution I've found is the following:
#Test
public void testFileRWAsArray() throws IOException{
String f1String="";
byte[] bytes=FileUtils.readFileToByteArray(f1);
for(byte b:bytes){
f1String=f1String+((char)b);
}
File temp=File.createTempFile("deleteme", "deleteme");
byte[] newBytes=new byte[f1String.length()];
for(int i=0; i<f1String.length(); ++i){
char c=f1String.charAt(i);
newBytes[i]= (byte)c;
}
FileUtils.writeByteArrayToFile(temp, newBytes);
assertTrue(FileUtils.contentEquals(f1, temp));
}
By using a cast between byte-char, I have the symmetry on conversion.
Thank you all!
Try this code...
public static String fetchBase64binaryEncodedString(String path) {
File inboundDoc = new File(path);
byte[] pdfData;
try {
pdfData = FileUtils.readFileToByteArray(inboundDoc);
} catch (IOException e) {
throw new RuntimeException(e);
}
byte[] encodedPdfData = Base64.encodeBase64(pdfData);
String attachment = new String(encodedPdfData);
return attachment;
}
//How to decode it
public void testConversionPDFtoBase64() throws IOException
{
String path = "C:/Documents and Settings/kantab/Desktop/GTR_SDR/MSDOC.pdf";
File origFile = new File(path);
String encodedString = CreditOneMLParserUtil.fetchBase64binaryEncodedString(path);
//now decode it
byte[] decodeData = Base64.decodeBase64(encodedString.getBytes());
String decodedString = new String(decodeData);
//or actually give the path to pdf file.
File decodedfile = File.createTempFile("DECODED", ".pdf");
FileUtils.writeByteArrayToFile(decodedfile,decodeData);
Assert.assertTrue(FileUtils.contentEquals(origFile, decodedfile));
// Frame frame = new Frame("PDF Viewer");
// frame.setLayout(new BorderLayout());
}