I want to do something which seems really straightforward: just pass a lot of logging commands (maybe all, but particularly WARN and ERROR levels) through a method in a simple utility class. I want to do this in particular so that during testing I can suppress the actual output to logging by mocking the method which does this call.
But I can't find out how, with ch.qos.logback.classic.Logger, to call a single method with the Level as a parameter ... obviously I could use a switch command based on this value, but in some logging frameworks there's a method or two which lets you pass the logging Level as a parameter. Seems a bit "primitive" not to provide this.
The method might look a bit like this:
Logger.log( Level level, String msg )
Later
Having now looked up the "XY problem" I understand the scepticism about this question. Dynamic logging is considered bad, at least in Java (possibly less so in Python)... now I know and understand that the preferred route is to configure the logging configuration appropriately for testing.
One minor point, though, if I may: although I haven't implemented this yet with this particular project, I generally find "just" tracing the stacktrace back to the beginning of the particular Thread insufficient, and this is what logback does (with Exceptions passed at WARN or ERROR levels). I plan to implement a system for recording "snapshots" of Threads when they run new Threads... which can then be listed (right back to the start of the app's first Thread) if an error occurs. This is, if you like, another justification for using something to "handle" outgoing log calls. I suppose that if I want to implement something like this I will instead have to try to extend some elements of logback in some way.
Related
I'm trying to log the methods call chain of some classes instances of a Java program using Log4j2. Each instance will behave differently based on the inputs they will recive (obviously).
The problem I'm facing is differentiating the logging messages of each instances for easly reconstructing the methods call chains I was talking about.
In addition to logging the entering and the leaving of each method, which is the default logging schema I'm using, I've tried to:
add method call parameters to the log: too few informations and not very readable;
add more informations about the method behaviour: too verbose and not very readable;
adding instance hash code to all the logging messages, which will become something like LOG.trace("{} - Leaving constructor(index:{})", System.identityHashCode(this), index);: good solution, but I've to add some boilerplate code to all the logging methods, which makes the code a little less readable;
using per-instance loggers (so not the per-class/static one) and adding the instance hash code in the logger name (so the logger name will be Classname#hashcode): seems the best solution in clean code terms, but I didn't found any way for declaring logger settings (like logging threshold) for multiple loggere, i.e. for all the loggers which name starts with Classname.
Which one do you think will be the best solution, or do you have any other way to suggest?
For this requirement, you can easily use an nested thread context: Look at "Fish Tagging" in https://logging.apache.org/log4j/2.x/manual/thread-context.html .
Excerpt:
ThreadContext.push(UUID.randomUUID().toString()); // Add the fishtag;
logger.debug("Message 1");
.
.
.
logger.debug("Message 2");
.
.
ThreadContext.pop();
I am struggling with the following problem and ask for help.
My application has a logger module. This takes the trace level and the message (as string).
Often should be messages constructed from different sources and/or different ways (e.G. once using String.format in prior of logging, other times using .toString methods of different objects etc). Therefore: the construction method of the error messages cannot be generalized.
What I want is, to make my logger module effective. That means: the trace messages would only then be constructed if the actual trace level gets the message. And this by preventing copy-paste code in my application.
With C/C++, by using macros it was very easy to achive:
#define LOG_IT(level, message) if(level>=App.actLevel_) LOG_MSG(message);
The LOG_MSG and the string construction was done only if the trace level enabled that message.
With Java, I don't find any similar possibility for that. That to prevent: the logging would be one line (no if-else copy-pastes everywhere), and the string construction (expensive operation) only be done if necessary.
The only solution I know, is to surrond every logger-calls with an IF-statement. But this is exactly what I avoided previously in the C++ app, and what I want to avoid in my actual Java-implementation.
My problem is, on the target system only Java 1.6 is available. Therefore the Supplier is not a choice.
What can I do in Java? How can this C/C++ method easily be done?
Firstly, I would encourage you to read this if you're thinking about implementing your own logger.
Then, I'd encourage you to look at a well-established logging API such as SLF4j. Whilst it is possible to create your own, using a pre-existing API will save you time, effort and above all else provide you with more features and flexibility out of the box (I.e file based configuration, customisability (look at Mapped Diagnostic Context)).
To your specific question, there isn't a simple way to do what you're trying to do. C/C++ are fundamentally different to java in that the preprocessor allows for macros like you've created above. Java doesn't really have an easy-to-use equivalent, though there are examples of projects that do make use of compile time code generation which is probably the closest equivalent (i.e. Project Lombok, Mapstruct).
The simplest way I know of to avoid expensive string building operations whilst logging is to surround the building of the string with a simple conditional:
if ( logger.isTraceEnabled() )
{
// Really expensive operation here
}
Or, if you're using Java 8, the standard logging library takes a java.util.function.Supplier<T> argument which will only be executed if the current log level matches that of the logging method being called:
log.fine(()-> "Value is: " + getValue());
There is also currently a ticket open for SLF4j to implement this functionality here.
If you're really really set on implementing your own logger, the two above features are easy enough to implement yourself, but again I'd encourage you not to.
Edit: Aspectj compile time weaving can be used to achieve something similar to what you're trying to achieve. It would allow you to wrap all your logging statements with a conditional statement in order to remove the boilerplate checking.
Newest logging libraryies, including java.util.logging, have a second form of methods, taking a Supplier<String>.
e.g. log.info( ()->"Hello"); instead of log.info("Hello");.
The get() method of the supplier is only called if the message has effectively to be logged, therefore your string is only constructed in that case.
I think the most important thing to understand here is that the C/C++ macro solution, does not save computational effort by not constructing the logged message, in case the log level was such that the message would not be logged.
Why is so? Simply because the macro method would make the pre-processor substitute every usage of the macro:
LOG_IT(level, message)
with the code:
if(level>=App.actLevel_) LOG_MSG(message);
Substituting anything you passed as level and anything you passed as message along with the macro itself. The resulting code to be compiled will be exactly the same as if you copied and pasted the macro code everywhere in your program. The only thing macros help you with, is to avoid the actual copying and pasting, and to make the code more readable and maintainable.
Sometimes they manage to do it, other times they make the code more cryptic and thus harder to maintain as a result. In any case, macros do not provide deferred execution to save you from actually constructing the string, as Java8 Logger class does by using lambda expressions. Java defers the execution of the body of a lambda until the last possible time. In other words, the body of the lambda is executed after the if statement.
To go back to your example in C\C++, you as a developer, would probably want the code to work regardless of the log level, so you would be forced to construct a valid string message and pass it to the macro. Otherwise in certain log levels, the program would crash! So, since the message string construction code must be before the call to the macro, you will execute it every time, regardless of the log level.
So, to make the equivalent to your code is quite simple in Java 6! You just use the built-in class: Logger. This class provides support for logging levels automatically, so you do not need to create a custom implementation of them.
If what you are asking is how to implement deferred execution without lambdas, though, I do not think it is possible.
If you wanted to make real deferred execution in C\C++ you would have to make the logging code such, as to take a function pointer to a function returning the message string, you would make your code execute the function passed to you by the function pointer inside the if statement and then you would call your macro passing not a string but a function that creates and returns the string! I believe the actual C\C++ code to do this is out of scope for this question... The key concept here, is that C\C++ provide you the tools to make deferred execution, simply because they support function pointers. Java does not support function pointers, until Java8.
I am struggling a little bit to understand how to implement a Hystrix Metrics Publisher plugin.
Having read the documentation, it is still not clear how things are supposed to work together.
My goal is to write a plugin that will collect every metrics published by hystrix and write these metrics to a file on disk.
This file will later be collected and processed by an external tool giving us a good historical basis of the circuit’s behavior and problems.
The system where hystrix is running is a normal spring application. This said, I am also somewhat new on the java platform (although I am comfortable with the java language).
I thought that a first step towards understanding how the plugin could be implemented would be looking at the already implemented publishers. With this in mind, I looked at some of the implementations of hystrix-contrib directory.
I have chosen hystrix-codahale-metrics-publisher and hystrix-servo-metrics-publisher.
Both of them have a main class (servo is HystrixServoMetricsPublisher) which seems to register for receiving all possible kinds of metrics and some classes to deal with each kind of metric.
By looking at what I will call the main class, I see that, for example, there is a method called getMetricsPublisherForCommand that must return an implementation of the interface HystrixMetricsPublisherCommand.
Now questions start:
Question 1 I am assuming that once a plugin is registered every execution of every command on the context where the plugin is registered, and by the word command we can understand every execution of the execute() method of every class which inherits from HistrixCommand on that context, will generate a call to the getMetricsPublisherForCommand() method of my plugin. Is it true?
If so, there are a lot of low level implementations in hystrix such as thread pools and other, Should my getMetricsPublisherForCommand() implementation be thread-safe or I am guaranteed to receive calls in a sequential order? On what thread will my getMetricsPublisherForCommand() be executed?
Question 2 By looking at the documentation I am still not sure about what exactly the implementation of HystrixMetricsPublisherCommand to be returned by getMetricsPublisherForCommand() has to do. This is due to the fact that the HystrixMetricsPublisherCommand interface only specifies a method called initialize (). if it specified a method called, say, publish() I would conclude that the hystrix engine would call my custom getMetricsPublisherForCommand() method to get a metrics publisher on which it would call a publish() method to perform the custom publishing. But the initialize () method seens to be called only once when this given object is returned and I have found no other method the engine would call afterwards.
Also, by reading the documentation, I am under the impression that the implementation of HystrixMetricsPublisherCommand returned by getMetricsPublisherForCommand() will be somehow a singleton which completely breaks my understanding about how the thing is supposed to work.
The documentation say this:
The initialize() method will be called once-and-only-once to indicate when this instance can register with external services, start publishing metrics etc.
If you look at the servopublisher however you will notice that, unless I am completely and absolutely confused, the publishing stuff is performed right from the constructor. Now, if initialize() will be called to make some setup, how can I code my logic from the constructor where, unless the object is a singleton, it will be executed before any method including initialize () will have a chance to be called? In the other hand ,,, if this is a singleton, how can it run its constructor for every hystrix command?
May be I have missed something, I don't know ... but I need to understand conceptually what is going on here in order to implement my logic the right way. Thanks for your patience and I hope I have made myself clear enough in this long question.
First, recommend staying within the one (concise) question format.
Second, recommend using an existing implementation such as the default CodaHale (formerly DropWizard) implementation (which publishes to Graphite repository for Grafana consumption for example) to get it working.
HystrixPlugins.reset();
final WebApplicationContext springContext =
WebApplicationContextUtils.getWebApplicationContext(sce.getServletContext());
HystrixPlugins plugins = HystrixPlugins.getInstance();
plugins.registerCommandExecutionHook(...);
// Good idea to use properties to enable/disable metrics generally...
// Using Spring type example...
if (hystrixMetricsEnabled.get()) {
plugins.registerMetricsPublisher(new HystrixCodaHaleMetricsPublisher(
getRegistry(springContext, sce.getServletContext())));
...
Otherwise the Hystrix documentation and full source of classes involved are publicly available:
https://github.com/Netflix/Hystrix/wiki/Plugins#metricspublisher
In my Java application, I wish to get the name (and actual runtime class) of different methods as they execute in a specific thread. Its different then getting a regular stack trace. Its like I am on observer looking at the execution of my program and see the names of different methods appear as they are run.
One (naive and practically infeasible) way is to put a print message at the beginning of each method to print its name (class, line number, etc.). I can use the exception trace to get all the relevant info to print, BUT I would have to add at least one line to the beginning of each method which is not very elegant. Furthermore, if I miss/forget to add a line at the beginning of any method, that method name would not be displayed. Also, this strategy is not future-proof. Is there any instrumentation/other technique that will help me accomplish this.
You should have a look at Aspect Oriented Programmation and all of the frameworks implementing it.
If you are using spring you can have a look at the simple trace interceptor that will do exactly what you want : http://static.springsource.org/spring/docs/1.2.9/api/org/springframework/aop/interceptor/SimpleTraceInterceptor.html
Aside from AOP and the Spring Listener mentioned in the other answer below, you could use maybe some trickier code generation framework such as ASM or CGLib. You'd be writing code to rewrite each of your classes at runtime and append these print instructions at the beginning of your methods. But that requires deeper knowledge of bytecode.
I came across one very good library for parsing CUE files. But when I started to read its source code, I realized that it is almost unreadable:
public void setParent(final CueSheet parent) {
FileData.logger.entering(FileData.class.getCanonicalName(), "setParent(CueSheet)", parent);
this.parent = parent;
FileData.logger.exiting(FileData.class.getCanonicalName(), "setParent(CueSheet)");
}
every method has logger.entering() and logger.exiting() messages. Isn't that too much?
There's another java library for parsing audio tags. It also had like 15 log messages for each file it read. It was annoying so I commented out every call to logger. And the library became twice as fast, because they used a lot of string concatenation for log messages.
So the question is: should I really log everything, even if it is not large enterprise application? Because these libraries obviously don't need any logging, except for error messages. And my experience shows that loggers are terrible tool for debugging. Why should I use it?
How to know when is too much logging? When you know that the logged information isn't important in the long term, such as for straightforward debug actions or bug correction, or for when the application doesn't deal with too much important information.
Sometimes you need to log almost everything. Is performance or full possibility of analysis the most important part of an application? It really depends.
I've worked in the past with some integration with a lot of different webservices, like 10 in a same app. We logged all xml requests and responses. Is this an overhead? In the long term, I don't think so because we worked with a lot of credit card operations and should have every process made with the server logged. How to know what happened when there was a bug?
You wouldn't believe what I've seen in some of the xml responses. I've even received a xml without closing tags, from a BIG airplane company. Were the "excessive logs" a bad practice? Say that to your clients when you have to prove that the error came from the other vendor.
Ideally, you use a logger that allows logging levels; log4j has fatal/error/warn/debug/info, for example. That way, if you set the level to "only show errors", you don't lose speed to the software building log messages you didn't need.
That said, it's only too much logging until you wind up needing something that would have been logged. It sounds like most of the logging that's slowing you down should be "trace" level, though; it's showing you what a profiler would have.
Most logging libraries incorporate a means to confirm that logging is enabled before processing an instruction:
For example:
public void foo(ComplicatedObject bar) {
Logger.getInstance(Foo.class).trace("Entering foo(" + bar + ")");
}
Could be quite costly depending on the efficiency of the bar.toString() method. However, if you instead wrap that in a check for the logging level before doing the string concatenation:
static {
Logger log = Logger.getInstance(Foo.class);
public void foo(ComplicatedObject bar) {
if (log.isTraceEnabled()) {
log.trace("Entering foo(" + bar + ")");
}
}
Then the string concatenation only occurs if at least one appender for the class is set to Trace. Any complicated log message should do this to avoid unnecessary String creation.
This level of logging is canonically bad - in fact, I saw code exactly like this in the Daily WTF a few days ago.
But logging is in general a Very Good Thing.
It depends, it this code for an application, or a library? For an application, logger are useful once the code is in production. It should not be used to debug, but to help you replicate a bug. When a user tells you that your application crashed, you always want the maximum logging information.
I agree that it makes the code less readable. It even make the application slower!
It's a total different game for a library. You should have consistent logging with adequate level. The library SHOULD inform the development team when an error occurs.
Logging should provide you with information that a stack trace can't in order to track down a problem. This usually means that the info is some kind of historical trace of what the program did, as opposed to what state it's in at the time of failure.
Too much historical data will be ignored. If you can safely deduce that a method was called without having to actually log its entry and exit, then it's safe to remove those logging entries.
Another bad sign is if your logging files start to use up a huge amounts of disk space. You're not only sacrificing space, you're probably slowing down the app too much.
To answer the question, why should I use loggers?
Have you ever encountered a piece of software where the only error indicated presented to the end user is Fatal error occured. Would it not be nice to find out what have caused it?
Logging is a tool that can really help you narrow these kind of problems in the field.
Remember, end-user systems don't have nice IDE's to debug and the end-users usually are not knowledgeable enough to run these tools. However end-users, in most cases, are capable of copying log configuration files ( written by us, clever programmers ) into predefined location and fetch log files and email them back to us ( poor soles for having to parse megabytes of log output ) when they encounter problems.
Having said this, logging should be highly configurable and under normal conditions produce minimal output. Also, guards should protect finer level logging from consuming too many resources.
I think in the example that you have provided all logging should have been done on a TRACE level. Also, because nothing bad can really happen between function entry point and exit, it probably make sense to have only one log statement there.
Over the years I've swayed backwards and forwards between promoting logging everything at the appropriate levels (trace, info, etc...) and thinking that any is a complete waste of time. In reality it depends on what is going to be useful to track down or required (logging can be a cheap way of maintaining an audit trail).
Personally, I tend to log entry/exit at a component or service level and then log significant points in the processing such as a business logic decision or a call on another service/component. Of course errors are always logged, but once only and at the place they were handled (the stack trace and exception message should have sufficient info to diagnose the problem) and any service/component interface should always handle an errors (even if it is just converting it into another more appropriate to the caller).
The problem with logging stuff on the off chance something goes wrong is that you end up with too much information that it is impossible to identify the issue, especially if it is running under a server as you end up with loads of intertwined log entries. Obviously you can get around that by incorporating a request id in the entry and using some software to filter on that. Of course you also have the case where your application is distributed and/or cluster and you have multiple logs.
Nowadays I would never actually write trace entering/exiting entries code, the code just gets in the way and it is so much easier to use something like aspectj if it is really needed. Using aspectj also would guarantee to be consistent (you can change the log format in one place rather than having to change every operation) and accurate (in case some refactoring adds a new paramater and teh developer forgets to add it to the logging).
One thing I have thought about doing or looking to see if someone already has is a logger that will hold the entries in memory, then if an error is encountered they are written, if the operation succeeds the entries are just discarded. If anyone knows of one (ideally for log4j) please let me know, alternatively I have a few ideas on how to implement this if anyone is interested in doing one.
This is where log levels are helpful. In general, log levels in the order of verbosity and priority are TRACE, DEBUG, INFO, WARN, ERROR, FATAL.
The developer has to take a conscious call to use the correct log level while logging in the code.
While creating an instance of Logger we have to pass the correct log level by choosing it from a config (always prefer config). This decides which levels to be logged. For example, while creating the logger, if the config for log level is set to "INFO", anything below "INFO" (TRACE, DEBUG) won't be logged.
For instance, in the example you mentioned above, a TRACE OR DEBUG level would make more sense.
In runtime in production, the config for log level should always be set to INFO.
When an issue occurs in production and if the developer wants to find out the root cause, they can request for changing the log level to TRACE or DEBUG (mostly inside a QA environment where they can replicate the scenario), to see what exactly is happening (The app sometimes has to be restarted to have the log level changed, but it is helpful).
Log levels is a great practice, as most of the times, we won't be able to launch a debugger in the landscapes. As we are skipping the unnecessary file writes by choosing a higher log level, the performance won't take a hit