Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 7 years ago.
Improve this question
How can I validate every data in Microsoft Excel before I import it in the database using Java? Should I use loops? But wouldn't it take much time in looping all data in the excel? example if i have more than 3000 data in Excel. What is the best way to validate bad user input in excel? For example bad date format or bad employee id format.
You gave very little detail. I have done something similar to this many times. So I can at least provide some advice.
Assuming you know that each column should contain data of a given format you can validate each input cell with RegEx, i.e. the cell either matches a given RegEx or it doesn't.
If match then import
If not match then
2a. If bad format then correct format
2b. If invalid data do something
2c. Reject entire row?
2d. Prompt for user action?
All of the excel files I had to deal with were machine generated based on user input. So while I could have bad data the format was always correct. If you are dealing with human generated files then you are going to have to assume that at least some of the data will be bad format.
As to your question on speed, 3000 rows is a drop if the bucket. For my project I was forced to use Access/VBA which is dam slow. I was dealing with many files of 10,000+ plus rows with upwards of 50+ columns. The entire run time of the process was around 5 minutes to have the program access the website, pull the files, and load them into the database.
Java is orders of magnitude faster than Access/VBA. The only way you will know if your run time is reasonable is to run some test. Likely, I could have optimized the run time but as the code was only ran once a week there was no need.
Related
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 12 days ago.
Improve this question
I am want to be able to store per user multiple types of binary files. Could be pdf, photos or very small video ~2MB.
I have in mind 2 approaches:
Use MySQL and have a BLOB column in a table and add in the column these different types of files.
Use MySQL to store metadata about the binary files but store the actual files in the filesystem.
I think (1) is simpler to implement but (2) allows for easier access of the files from everywhere e.g. even for download links.
What I was not sure though is if we can consider the binary files as documents and hence using e.g. Cassandra or any other NoSQL store is a better choice. What are the downsides of treating the binary files as "documents"?
The downside for this approach with Cassandra, is depending on the table structure, your partitions could get too big. The prevailing wisdom is to keep your partition sizes < 100MB. If this table is partitioned on something unique like video_id, then each movie is its own partition, and that shouldn't be a problem.
But if there's a category or playlist system where multiple videos are getting stored in the same partition, that could exceed that limit and read performance would degrade.
tl;dr;
Regardless of database choice, option #2 is the best practice. Storing binary files in a database almost always leads to problems (corruption, slow reads, higher ops maintenance). Storing the metadata or file location data in the DB, and using that to reference the binary files is a much friendlier solution with fewer opportunities for failure.
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 7 years ago.
Improve this question
I am going to write a game in which I have often have to check if a string of letters is actually a word or not. My question is about how to do this the fastest with the least computation-able power as possible (for instance an old smart-phone). With if possible not much start-up time to make it a quick and responsive app.
I once did this look-up by first reading in a word-file with almost all words into an appropriate sized hash-map of around 650,000 words*. (* might be more, I am not sure if this is the exhausted list yet).
Would a SQL database be appropriate here? I am thinking of buying a book about it so I can learn and implement one. Also I have no idea how you could create a hash-map, save it for later and then load one. Is that too much of a hacker solution or is that technique used more often? So would it make sense for me to learn SQL or do it with saving a hashmap and then later restoring it.
A database SQL could be appropriate if you plan to query it every time you need to check a word, but this is not the fastest solution; querying every single word slows down the response time but it should use less memory if the words number is high (you must measure the memory consumed by the db vs the memory consumed by the map). Checking if a word is inside a map is not so computationally expensive, it must calculate the hash and iterate over the array of items with the same hash.
Personally I would choose a map if the memory requirements of keeping all the words in memory can be satisfied. You can store the dictionary as plain text file (one line -> one word) and read it in a background thread when the application starts.
If memory is an issue, this seems like a good use for a B-Tree. This allows for O(log n) search time while searching a large amount of records with minimal memory usage. For this sort of application it sounds like loading the entire thing into memory is not going to be a good idea.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 8 years ago.
Improve this question
I want to make a dialogue tree (conversation tree) in Java using libGDX. Should I use lots of conditionals (if,else etc...) and move on to the next dialogue or is there a better way to read a file such as XML that already have the dialogues inside? Also, I want the solution that would consume the least possible memory amount because I am going to write it for Android.
Example of the dialogue tree:
(Q: Question, A: Answer , C:Choice ,AC:Action)
Q:Hi is there anyway that i can help you?
A:You own me 5 dollars!
C1:Ask politely to return them to you, C2:Τhreaten her , C3:Draw your gun
A1:No way get out of here , A2:Call the security , A3:Call the cops
AC1-2:Exit the building //end of choices 1-2
C3.1:Draw your gun and shoot the cops , C3.2:Jump from the window
AC3.1:Arrested , AC3.2:Dead
If your game is going to have little dialogue, I would use Strings for it, but if you base it around the dialogues, I would use a SQLite or similar database to store them in it. I don't know whether it will be the most efficient way to do that, but that what's occurred to me while reading your question:
You could use e.g. column 1 for the question, and columns 2, 3, 4, 5 for the possible answers. You can get information about using SQLite in libGDX here
You could make into some method actor, whom he pass an id to access the SQLite for the question and the answers to that question, then assign some variables, and use a switch statement, if you don't want to have much if-elseif etc.
Note: I think that SQLite is mostly used when you want data to be saved and used in the future; if the data for example change every 10 minutes, I think it would be better to use JSON, because in SQLite making connections to the database every 10 minutes may take some time. I think this is not the case; in my opinion the purpose of JSON and SQLite is completely different. I would use for example:
JSON = I would use it to send or/and receive data betwen server & client or configuration files etc.
SQLite = I would use it to store data.
This is only my opinion, and I not say that SQLite is better or worse than JSON.
PS 1: the photo is taken from the Internet
PS 2: I also believe that you should read https://stackoverflow.com/tour
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 9 years ago.
Improve this question
for some test data scenario i need to read file containing 100000 lines of row and process each row with some condition and then based on condition need to output the data in text format
for this i am planning to store all lines of data in some pojo then pojo to List
My worry is to having pojo of 100000 row in memory . this is just for testing case .
I think using InputSteam to read the file will be better since you still fetch rows one by one. You can read one line per time, and process your confition and then output.
Storing too much Objects in List may encounter an Out of Memory Error.
In any case, its a bad design to store all 100000 rows as POJO in memory. Some of the possible solutions are:
Read one row at a time and process it.
Rather than reading from a file one record at a time and processing it using java, use some scripting language to populate a database table, and then from your java code you can process the records from the table.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 6 years ago.
Improve this question
I have been learning how to build android apps this summer. I am currently trying to work on xml parsing which falls under java in this case. I have a few questions that are mostly conceptual and one specific one.
First, in most of the examples I have seen pages already in xml are used. Can I use a page in regular html format and with whatever the program does turn it to xml and then parse it? Or is that what is normally done anyway?
Secondly, I could use a little explanation on how the parser actually works and saves the data so I will better know how to use it (extract it from whatever it is saved in), when the parsing is done.
So for my specific example I am trying to work with some weather data from the NWS. My program will take the data from this page, and after some user input take you to a page like this, which sometimes will have various alerts. I want to select certain ones. This is what I could use help with. I haven't really coded anything on that yet because I don't know what I am doing.
If I need to clarify or rephrase anything in here I am happy too and let me know. I am trying to be a good contributor on here!
Yes you can parse HTML and there are many parsers available too, there is a question about it here Parse HTML in Android, then we have an answer here about parsing html https://stackoverflow.com/a/7114346/826657
Although its a bad idea, as the tag names aren't well named, so you will have to write lots of code searching attributes for a specific data tag, so you always have to prefer XML,for saving lots of code space and also time.
Here is a text from CodingHorror which says at general parsing html is a bad idea.
http://www.codinghorror.com/blog/2009/11/parsing-html-the-cthulhu-way.html
Here is something which explains parsing an XML document using XML PullParser http://www.ibm.com/developerworks/library/x-android/