I am working for a log analyzer system,which read the log of tomcat and display them by a chart/table in web page.
(I know there are some existed log analyzer system,I am recreating the wheel. But this is my job,my boss want it.)
Our tomcat log are saved by day. For example:
2011-01-01.txt
2011-01-02.txt
......
The following is my manner for export logs to db and read them:
1 The DB structure
I have three tables:
1)log_current:save the logs generated today.
2)log_past:save the logs generated before today.
The above two tables own the SAME schema.
+-------+-----------+----------+----------+--------+-----+----------+----------+--------+---------------------+---------+----------+-------+
| Id | hostip | username | datasend | method | uri | queryStr | protocol | status | time | browser | platform | refer |
+-------+-----------+----------+----------+--------+-----+----------+----------+--------+---------------------+---------+----------+-------+
| 44359 | 127.0.0.1 | - | 0 | GET | / | | HTTP/1.1 | 404 | 2011-02-17 08:08:25 | Unknown | Unknown | - |
+-------+-----------+----------+----------+--------+-----+----------+----------+--------+---------------------+---------+----------+-------+
3)log_record:save the information of log_past,it record the days whose logs have been exported to the log_past table.
+-----+------------+
| Id | savedDate |
+-----+------------+
| 127 | 2011-02-15 |
| 128 | 2011-02-14 |
..................
+-----+------------+
The table shows log of 2011-02-15 have been exported.
2 Export(to db)
I have two schedule work.
1) day work.
at 00:05:00,check the tomcat log directory(/tomcat/logs) to find all the latest 30 days log files(of course it include logs of yesterday.
check the log_record table to see if logs of one day is exported,for example,2011-02-16 is not find in the log_record,so I will read the 2011-02-16.txt,and export them to log_past.
After export log of yesterday,I start the file monitor for today's log(2011-02-17.txt) not matter it exist or not.
2)the file monitor
Once the monitor is started,it will read the file hour by hour. Each log it read will be saved in the log_current table.
3 tomcat server restart.
Sometimes we have to restart the tomcat,so once the tomcat is started,I will delete all logs of log_current,then do the day work.
4 My problem
1) two table (log_current and log_past).
Because if I save the today's log to log_past,I can not make sure all the log file(xxxx-xx-xx.txt) are exported to db. Since I will do a check in 00:05:00 every day which make sure that logs before today must be exported.
But this make it difficult to query logs accros yestersay and today.
For example,query from 2011-02-14 00:00:00 to 2011-02-15 00:00:00,these log must be at log_past.
But how about from 2011-02-14 00:00:00 to 2011-02-17 08:00:00 ?(suppose it is 2011-02-17 09:00:00 now).
It is complex to query across tables.
Also,I always think my desing for the table and work manner(schedule work of export/read) are not perfect,so anyone can give a good suggestion?
I just need to export and read log and can do a almost real-time analysis where real-time means I have to make logs of current day visiable by chart/table and etc.
First of all, IMO you don't need 2 different tables log_current and log_past. You can insert all the rows in the same table, say logs and retrieve using
select * from logs where id = (select id from log_record where savedDate = 'YOUR_DATE')
This will give you all the logs of the particular day.
Now, once you are able to remove the current and past distinction between tables using above way, I think the problem you are asking here would be solved. :)
Related
I am making a game and I require a ranking system. I already save all the stats like kills, deaths, wins, innocent shots, etc with MySQL. I am clueless a the moment on how I would be able to rank everyone. I want to have it over MySQL but they will be updated very quickly. I was thinking I could load all ranks in a HashMap when the game starts but that would be very ineffective since there are thousands of players. I want to also use most of the stats to work this out. Could someone explain to me how I would be able to do this? Thanks!
One way would be to use mysql Events to trigger a stored procedure. The stored procedure would execute the ranking and store the rank in the db. You would then just set the event trigger time to whatever you wanted, say 10 minutes.
mysql events: https://dev.mysql.com/doc/refman/5.7/en/events.html
CREATE
[DEFINER = { user | CURRENT_USER }]
EVENT
[IF NOT EXISTS]
event_name
ON SCHEDULE schedule
[ON COMPLETION [NOT] PRESERVE]
[ENABLE | DISABLE | DISABLE ON SLAVE]
[COMMENT 'comment']
DO event_body;
schedule:
AT timestamp [+ INTERVAL interval] ...
| EVERY interval
[STARTS timestamp [+ INTERVAL interval] ...]
[ENDS timestamp [+ INTERVAL interval] ...]
interval:
quantity {YEAR | QUARTER | MONTH | DAY | HOUR | MINUTE |
WEEK | SECOND | YEAR_MONTH | DAY_HOUR | DAY_MINUTE |
DAY_SECOND | HOUR_MINUTE | HOUR_SECOND | MINUTE_SECOND}
Then you would setup a stored procedure to generate the rank and set it for your players: https://dev.mysql.com/doc/refman/5.7/en/stored-programs-views.html
An example procedure would be:
CREATE PROCEDURE simpleproc (OUT param1 INT)
BEGIN
SELECT COUNT(*) INTO param1 FROM t;
END
You can make the procedures as complex as you need to.
I'm working on a mini-project that can upload certain files to a server. I'm using java and mysql. According to things I read the 'right way' is to save the file to a folder in my server and put the file path in mysql.
A Part of my code:
File source = new File(fileChooser.getSelectedFile().getPath());
File dest = new File(destChooser.getSelectedFile().getPath());
String tempUpdate1 = "Insert into images (ID, file_name, file_path)"
+ "VALUES ('"+ID+"' , '"+fileChooser.getSelectedFile().getName()+"' , '"+destChooser.getSelectedFile().getPath()+"')";
conn = Connect.ConnectDB();
Statement stemt2 = conn.createStatement();
stemt2.executeUpdate(tempUpdate1);
FileUtils.copyFileToDirectory(source, dest);
JOptionPane.showMessageDialog(null, "File Uploaded Successfully!");
then i tried running it. It successfully copied the file to my desired folder. The problem is in my
mysql table where i save the path of the file.
The table's like this:
| ID | file_name | file_path |
| 1 | sample.docx | C:UsersMEDesktopest |
| 2 | sample.jpg | C:UsersMEDesktopest |
I tried seeing the output of the file path myself using JOptionPane it returned normal with the separators, but using the same variable and puting it in mysql, that's what i get as seen above, no separators.
Am i really doing it right? I did what is instructed on the topics related to this but no one seems to be complaining with the path. I'm wondering if did something wrong somewhere.
I'm planning to access the file using the file path as my project progress. Just would like to ask if the path file is accessible since i dont see any separator such as '//', '/' or '\'
it is almost good.
imagine that you have two files named same way. you can have only one of them on your server. I would save file under coded name like time in milis at moment of saving action.
Your table would contain then original name sample.jpg and file name 49473996034732 for example (in separate column of course). This way you can save as many sample.jpg as you like. Request link should point to file name 49473996034732 so you always will know for which file is request.
Then in response, when you set content to proper type you can set user friendly name.
modified table
| ID | file_name | server_file_name | file_path |
| 1 | sample.docx | 49473996034732 | C:UsersMEDesktopest |
| 3 | sample.jpg | 49473996034850| C:UsersMEDesktopest |
| 2 | sample.jpg | 49473996034988 | C:UsersMEDesktopest |
second thing is that i noticed is that you allow user to pick destination folder for containing file. Shouldn't is be fixed location? This way it would be more like server behavior. But im guessing this is just an example code fast codded in Swing
I have two machines that will be an application server in each.
The machine X is dynamic sources. The machine Y is static sources.
Thus, the user is always connected to "x.com".
When he does one upload an image, I need to send this information to "y.com".
How can I pass (at the time of upload) the byte image server x.com to save on y.com ?
See here what I started doing:
http://forum.primefaces.org/viewtopic.php?f=3&t=30239&p=96776#p96776
Balusc answered very well here:
Simplest way to serve static data from outside the application server in a Java web application
But my case is slightly diferent.
I appreciate any help!
Thank you!
I think the simplest way is to create a database table on X.com to track all images your user store on Y.com, for example:
+----------+-------------------------+
| user_id | image_path |
+----------+-------------------------+
| 0 | /images/image_xxxxx.jpg |
| 0 | /images/image_xxxxx.jpg |
| 2 | /images/image_xxxxx.jpg |
| 2 | /images/image_xxxxx.jpg |
| 3 | /images/image_xxxxx.jpg |
+----------+-------------------------+
and then serve on X.com all your images redirecting the browser to Y.com
X.com:
<img src="Y.com/images/image.xxxxx.jpg" />
Use share network disk like: samba or NFS.
Ootionaly you can consider to setup rsyncs if you have Linux/U*x hosts
I have a CSV datasource something like this:
User,Site,Requests
user01,www.facebook.com,54220
user01,plusone.google.com,2015
user01,www.twitter.com,33564
user01,www.linkedin.com,54220
user01,weibo.com,2015
user02,www.twitter.com,33564
user03,www.facebook.com,54220
user03,plusone.google.com,2015
user03,www.twitter.com,33564
In the report I want to display the first 3 rows (max) for each user, while the other rows will only contribute to the group total. How do I limit the report to only print 3 rows per group?
e.g
User Site Requests
user01 | www.facebook.com | 54220
plusone.google.com | 2015
www.twitter.com | 33564
| 146034
user02 | www.twitter.com | 33564
| 33564
user03 | www.facebook.com | 54220
user03 | plusone.google.com | 2015
user03 | www.twitter.com | 33564
| 89799
It is really just the line limiting I am struggling with, the rest is working just fine.
I found a way to do it, if anyone can come up with a more elegant answer I would be happy to see it, as this feels a bit hacky!
for each item in detail band:
<reportElement... isRemoveLineWhenBlank="true">
<printWhenExpression><![CDATA[$V{userGroup_COUNT} < 4]]></printWhenExpression>
</reportElement>
where userGroup is the field I am grouping by. I only seemed to need the isRemoveLineWhenBlank attribute for the first element.
you may consider to use subreport by querying the grouping fields in the main report and then passing the grouping fields as parameters into the subreport; the merits of this method is to avoid the report engine to actually looping through all un-required rows (although they are not shown) and spending unnecessary server-to-server or server-to-client bandwidth especially when the dataset returned is large
Is it possible index a complete database without mentioning the table names explicitly in the data-config.xml as new tables are added everyday and I cannot change the data-config.xml everyday to add new tables.
Haven table names based on the date smells like there is something wrong in your Design. But given this requirement in your question you can add Data to your solr server without telling you have a DB. You just have to make sure you hav a unique ID for the data record in you solr Server with whom you can identify the corresponding record in your DB, something like abcd_2011_03_19.uniqueid. You can post the data to solr in Java in solrj or just plain xml or json.
Example:
--------------
| User Input |
--------------
|post
V
-----------------------------------
| My Backend (generate unique id) |
-----------------------------------
|post(sql) |post (e.g. solrj)
V V
------ --------
| DB | | solr |
------ --------
My ascii skillz are mad :D