We are using apache commons logging in our application for all our logging requirements.
A funny use case popped up this week that I had a question about.
We started receiving FIX messages from clients where a particular field in the fix message is populated based on the values in a free form textarea on an application that our client has. The client is allowed to enter in any text they want including special characters and new lines etc.
We log the fix message we receive back but when we receive a FIX message that includes this tag that has new line characters in it, only the part of the fix message up until the new line character is logged. Is there anyway to tell the logging framework to ignore new line characters and log the entire string whether or not it contains new line characters?
are you sure the message isn't being logged on a new line? We do a similar thing and new lines are logged without any extra configuration. Are you grep'ing for these lines in the log to view them? I ask because they will show on a new line, and therefore not in the output of grep unless you add '-A 5' (or how many lines you want to see after) flag to your grep statement to see the new lines after the matching one.
Related
I have a task to implement a logging method that will take all program content up to logging method call and save it to file. The method should not overwrite already present info in file, but rather add new content to it.
What I need to write:
all program messages and output;
all user inputs in the way they appear on console (the program inquires it several times);
My thought on it is to create a StringBuilder object and start appending everything to it. Once logging method is invoked, ask for a file to save log to and save contents of StringBuilder to it. Then flush StringBuilder and continue to gather information. On second invocation, if the same filename is provided, just append new info gathered by StringBuilder.
However, this means that I will need to place gathering invocations all over the place where program output and user input are. Seems like not very optimal to me.
Are there any ideas on how to reach my goal differently and more optimally?
Thank you in advance.
Best regards,
Vadim
UPDATE: I actually was able to redirect system.out to gather everything to ByteArrayStream and then write to file on demand. But I still don't understand how to do it for inputstream. I don't need to redirect it, I still have to input eveyrthing from keyboard, it's just values that have to make it to logs in correct places. Am still searching for a solution.
Do not reinvent the wheel. Go for a logging framework. There is one integrated into java anyway.
Alternatively you can use log4j, and there are other such frameworks out there.
I am writing simple socket chat using console to output messages. The problem is that when one user types a message, and at the same time getting one, his input interrupted:
I: writing my input here, but
Other_user: hi! here is a message for you.
I: it splits to different lines, which is 1) very inconvenient 2) cannot see which simbols i deleted if press backspace
So, what i am asking is, how can I avoid this
(something like: if message is received, check input for symbols; if there are, remember them, delete last stroke in console, print message, and then recreate that stroke).
EDIT: attached picture
hard to tell without specific code, but an option is to use two threads, one to handle the socket input, one for output. attach these to System.in and System.out respectively. it seems like you might be using only one thread to do both.
I'm new to Spark and the Hadoop ecosystem and already fell in love with it.
Right now, I'm trying to port an existing Java application over to Spark.
This Java application is structured the following way:
Read file(s) one by one with a BufferedReader with a custom Parser Class that does some heavy computing on the input data. The input files are of 1 to maximum 2.5 GB size each.
Store data in memory (in a HashMap<String, TreeMap<DateTime, List<DataObjectInterface>>>)
Write out the in-memory-datastore as JSON. These JSON files are smaller of size.
I wrote a Scala application that does process my files by one worker but that is obviously not the most performance benefit I can get out of Spark.
Now to my problem with porting this over to Spark:
The input files are line-based. I usually have one message per line. However, some messages depend on preceding lines to form an actual valid message in the Parser. For example it could happen that I get data in the following order in an input file:
{timestamp}#0x033#{data_bytes} \n
{timestamp}#0x034#{data_bytes} \n
{timestamp}#0x035#{data_bytes} \n
{timestamp}#0x0FE#{data_bytes}\n
{timestamp}#0x036#{data_bytes} \n
To form an actual message that out of the "composition message" 0x036, the parser also needs the lines from message 0x033, 0x034 and 0x035. Other messages could also get in between these set of needed messages. The most messages can be parsed by reading a single line though.
Now finally my question:
How to get Spark to split my file correctly for my purposes? The files can not be Split "randomly"; they must be split in a way that makes sure that all my messages can be parsed and the Parser will not wait for input that he will never get. This means that each composition message (messages that depend on preceding lines) need to be in one split.
I guess there are several ways to achieve a correct output but I'll throw some ideas that I had into this post as well:
Define a manual Split algorithm for the file input? This will check that the last few lines of a split do not contain the start of a "big" message [0x033, 0x034, 0x035].
Split the file however spark wants but also add a fixed number of lines (lets say 50, that will do the job for sure) from the last split to the next split. Multiple data will be handled by the Parser class correctly and would not introduce any issues.
The second way might be easier, however I have no clue how to implement this in Spark. Can someone point me into the right direction?
Thanks in advance!
I saw your comment on my blogpost on http://blog.ae.be/ingesting-data-spark-using-custom-hadoop-fileinputformat/ and decided to give my input here.
First of all, I'm not entirely sure what you're trying to do. Help me out here: your file contains lines containing the 0x033, 0x034, 0x035 and 0x036 so Spark will process them separately? While actually these lines need to be processed together?
If this is the case, you shouldn't interpret this as a "corrupt split". As you can read in the blogpost, Spark splits files into records that it can process separately. By default it does this by splitting records on newlines. In your case however, your "record" is actually spread over multiple lines. So yes, you can use a custom fileinputformat. I'm not sure this will be the easiest solution however.
You can try to solve this using a custom fileinputformat that does the following: instead of giving line by line like the default fileinputformat does, you parse the file and keep track of encountered records (0x033, 0x034 etc). In the meanwhile you may filter out records like 0x0FE (not sure if you want to use them elsewhere). The result of this will be that Spark gets all these physical records as one logical record.
On the other hand, it might be easier to read the file line by line and map the records using a functional key (e.g. [object 33, 0x033], [object 33, 0x034], ...). This way you can combine these lines using the key you chose.
There are certainly other options. Whichever you choose depends on your use case.
I'm writing a multithreaded socket chat. Is there any way to get the console input at a given time before the user has hit enter?
Let's say I'm typing a message, and while I'm typing it, the server sends a message to me. The server's message will get printed right after the message I'm currently writing.
How it looks:
Me>Hey!
Server>Heya!
Me>How are yServer>Hello!
If possible, I want to save the "How are y" string, remove it and paste it again after the server output:
Me>Hey!
Server>Heya!
Server>Hello!
Me>How are y
your problems boils down to two separate issues. number one is reading input from stdin or console character by character. this is tricky by itself, since most common approaches to reading input only do entire lines.
the second issues you're facing is going back and forth on the screenm because you have to update the current input and the new incoming messages this is quite possible, but the way to do it depends heavily on your terminal type and it's capabilities:
http://unix.stackexchange.com/questions/43945/whats-the-difference-between-various-term-variables
doing this correctly for all possible $TERM values on all OSs is rather tricky as well. you might do better using a library such as jcurses:
https://github.com/sunhong/jcurses
that abstracts this away from you.
I'm trying to generate a 'nice' log message with using escaping characters or an xml pattern etc.
What I'd like to output is something along the lines of:
ABAS : value
B : value
Cse : value
I've achieved this using \t but I figure there must be a cleaner way. I've looked at .info which takes an argument and using the {} as a way of inserting the values but I can't seem to find out how to add the line breaks or tabbing.
so far I have
logger.info(A : {} \nBasdas : {} \nC : asds ) and so on.
thanks for any help.
slf4j is the log frontend and only intended to provide log level, message etc. to the backend, most likely logback in your case. You shouldn't format your messages in the frontend expecting any special format in the actual log output, because exactly that can be somewhat freely configured by the backend one uses. Especially indentation over some independent lines doesn't work, because you can't know how lines start, if your logger names are part of lines, where the msg is printed within a line and all that stuff. Just look at the logback configuration and what is possible, how do you want to tell as the log message issuing programmer which of those possibilities are used during runtime in any environment of your software? You simply can't and therefore shouldn't assume too much.
So what you want is simply not possible, besides embedding tabs or newlines there's nothing to format log messages in slf4j for a good reason. And you can't count on your tabs as well, because how those are presented to a user looking at your log file depends totally on the text editor or whatever one uses. It may even convert tabs to spaces, show them with a width of 1 or 10 or whatever.
Log statements spanning multiple lines may be considered bad practice at all.