I have a file and i read data from this and put them in a jTable. The problem is that when the file has many data (e.g 300.000 rows), my application needs a lot of memory (350MB). Is there any efficient way in order to load many rows in a JTable?
I created a Default Model and a Jtable like that:
DefaultTableModel model = new DefaultTableModel(array, colNames);
JTable data_table = new JTable();
data_table.setModel(model);
The array 'array' contains the data and the array 'colNames' the names of the Columns.
Use an embedded database, and store only the record ID in the model and then only if you need to filter/sort rows.
Absent enough information to offer a particular solution, you may be able to identify a reasonable partition function for your data's primary key, e.g. a String prefix or a Date range. Use an adjacent control to update the TableModel based on the selection. In this example, buttons are used to change a chart's data model. To minimize latency, Use SwingWorker. Aside from the memory problem, you may want to filter the table or load the file into an in-memory database such as H2 Database.
If you want to reduce the memory consuption of your program.
You could use java.nio.channels.FileChannel and map parts of the file to your main memory.
But you would have to reload/change current mapping, depending on your view.
Take a look at the javadoc of the map method.
I don' know what exactly your file is, but if you just use it as a simplistic replacement for an actual database. You're better off with using a real one.
Related
I'm working on a project with an existing cassandra database.
The schema looks like this:
partition key (big int)
clustering key1 (timestamp)
data (text)
1
2021-03-10 11:54:00.000
{a:"somedata", b:2, ...}
My question is: Is there any advantage storing data in a json string?
Will it save some space?
Until now I discovered disadvantages only:
You cannot (easily) add/drop columns at runtime, since the application could override the json string column.
Parsing the json string is currently the bottleneck regarding performance.
No, there is no real advantage to storing JSON as string in Cassandra unless the underlying data in the JSON is really schema-less. It will also not save space but in fact use more because each item has to have a key+value instead of just storing the value.
If you can, I would recommend mapping the keys to CQL columns so you can store the values natively and accessing the data is more flexible. Cheers!
Erick is spot-on-correct with his answer.
The only thing I'd add, would be that storing JSON blobs in a single column makes updates (even more) problematic. If you update a single JSON property, the whole column gets rewritten. Also the original JSON blob is still there...just "obsoleted" until compaction runs. The only time that storing a JSON blob in a single column makes any sense, is if the properties don't change.
And I agree, mapping the keys to CQL columns is a much better option.
I don't disagree with the excellent and already accepted answer by #erick-ramirez.
However there is often a good case to be made for using frozen UDTs instead of separate columns for related data that is only ever going to be set and retrieved at the same time and will not be specifically filtered as part of your query.
The "frozen" part is important as it means less work for cassandra but does mean that you rewrite the whole value each update.
This can have a large performance boost over a large number of columns. The nice ScyllaDB people have a great post on that:
If You Care About Performance, Employ User Defined Types
(I know Scylla DB is not exactly Cassandra but I've seen multiple articles that say the same thing about Cassandra)
One downside is that you add work to the application layer and sometimes mapping complex UDTs to your Java types will be interesting.
I am wondering how I would store my custom network level in a MySQL table. I could make four columns, 'level', 'exp', 'expreq' and 'total'. Only this will take up four columns, and as I am storing name, rank and other data in the same table it will be too many columns in the end. Are there better ways? Should I make another table?
In a relational data model, and for expansion ability you have to do it in a different table. by which the master can point to the detailed table where you can have as many attributes as you can.
BUT
This has an obvious impact on the memory when it becomes large, in addition to that, this approach is usually being replaced by less-normalized version of the tables by introducing concepts like "Custom Fields"
OR
If it is me, and this table will be accessible by certain programming language, I would store them in JSON format in very simple table. and let the program do the processing overhead
I would assume that I should use a jTable. I tried this, but I can't for the life of me figure out how to append, insert and delete rows without a ton of overrides and complicated code. I find it hard to believe that Oracle doesn't have an easier way to do it.
Here's the premise. I have a few arrayLists. Each contain n amount of items and I want to be able to add these items' properties in the form of strings to the jtable and once i surpass a certain number of rows, I want the jTable to scroll.
So that's the reason I need to be able to add and remove rows.
As discussed in How to Use Tables: Creating a Table Model, DefaultTableModel has convenient methods to add, insert and remove rows. Simply update your model using any of these methods and your view will be updated accordingly.
Addendum: There's an example here.
Take a look at GlazedLists. It makes working with dynamically changing data and sowing it in JTables/JLists/JTrees, etc, very simple.
I'm a .net developer that would like to try develop some simple apps in Java. I would like to know how to do databinding in Java.
How can I show a query result in a JTable?
GustlyWind's comment is the best place to look on how to use a JTable. The key for the data is getting the items in a model. You can use the DefaultTableModel which would require that your table results are put into a 2D Array or Vector. Or you could implement your own model that uses other custom objects from your application or a different underlying data structure.
Either way, as you loop through the ResultSet of your query you are going to have to pull the relevant data and stick it in some sort of Collection.
How do I enable JTable icons and behaviors for sorting table rows by a column, without letting it use a comparison predicate to do the sorting? That is to say, how do I tell the table headers to show the arrow for ascending/descending sort order in the column being used, and get it to call appropriate methods when sort order/column change?
I am trying to create an (editable, filterable, sortable) JTable backed by an SQL query or view. The rows may not fit in memory, and may not map cleanly to java objects, so I want to do all sorting/filtering within SQL. I have already written the code for changing a query to accommodate sorting by column, filtering by values, and visible columns.
To use this, I am planning to write a JTableModel based on a ResultSet with TYPE_SCROLL_SENSITIVE, and CONCUR_UPDATABLE, so changes to the DB get propagated to the ResultSet. I will periodically (several times a second) force a refresh of the visible JTable from the ResultSet, so changes to the database become visible to the user. User changes to the table will be passed to the updateable ResultSet after validation.
I've looked a little bit at how sorting is done normally, but most implementations seems to rely on the JTable creating a javax.swing.RowSorter with a Comparator predicate, or on maintaining a sorted list of rows that fires events when changed. So, my questions:
ORM frameworks are NOT an answer to this question, because the data do not map well to entity objects. Also, the DBMS I am using is H2.
EDIT: Sortable JTable libraries based on applying Comparators or sorting predicates to row objects are also unsuitable, unfortunately. I do not believe I will be able to hold all objects in memory in order to perform sorting. This problem prevents me from using the SwingX JXTables, GlazedLists, or similar libraries. I wish I could, but I can't. Period.
** I will be dealing with many thousand rows, potentially millions, with numerous columns. Yes, I really DO need to use SQL to do the sorting and filtering.**
Questions: (in descending importance)
How do I show indicators for which column is used to sort rows?
How do I get the JTable to fire appropriate events when the column headers are LEFT-clicked to change sort order?
Is there an easier way to force the JTable to update when the database changes?
Is there a library that would make all this considerably easier (connecting DB queries or views and JTables)?
Am I going to run into horrible, horrible problems when I design the system like this?
I have never used it myself but JIDE Data Grids provides a DatabaseTableModel that provides filtering and sorting support using SQL WHERE and ORDER BY.
In answer to 1 and 2, check out SwingX, which already includes a table class with built-in sorting (and filtering). You may be able to adapt this.
Am I going to run into horrible, horrible problems when I design the system like this?
From experience, yes. I worked on a project almost exactly the same as this, where someone had designed a JTable that supposedly 'magically' bound to a database table. This coupled display logic and database access together in one big horrible mess, which we replaced entirely with reflection-driven table models and separate record CRUD operations.
You say that ORM is not the answer...
If the format of the data doesn't change, then it's worth considering anyway. Your 'entity' classes need not represent real-world entities.
If (as I suspect) your entity format changes, it might be worth considering:
A flexible map-based Record class which stores records as key-value pairs;
Dynamically-built table models for your display logic, built by querying record keys, plugged into SwingX tables to get sort and filter for free;
A similarly-designed Repository class which encapsulates your database access separately from the table itself, responsible for loading and saving Records. This acts as an adapter between your updateable ResultSet and the view (although I'd check whether using a ResultSet this way is going to require an open database connection whilst data is visible...).
This separation into 'a table that displays and sorts records' and 'a repository that manages the data' means:
You can reuse the table for non-database-bound data;
You can display database-bound records in things other than tables;
You won't go mad trying to build and test the thing :)
You should be able to subclass javax.swing.RowSorter in order to create a row sorter that does the sorting in the database. From the API docs:
"RowSorter implementations typically don't have a one-to-one mapping with the underlying model, but they can. For example, if a database does the sorting, toggleSortOrder might call through to the database (on a background thread), and override the mapping methods to return the argument that is passed in."
http://docs.oracle.com/javase/6/docs/api/javax/swing/RowSorter.html
Leaving aside the database stuff there's a class called SortableTable that's a part of JIDE Grids. It displays the sorting with a little ^ or v in the table header, and supports sorting by more than 1 column (1v, 2v, etc.).