I have a Java application running on Google App Engine standard environment.
I am able to log from it (using JUL). In standard environment, all the application log lines from a single web request are grouped into a single entry in the request_log. Everything runs great
However, now I have a requirement to add custom labels to a log entry for a request. For example, what is user ID associated with it.
Stackdriver documentation (https://cloud.google.com/logging/docs/setup/java) gives example how to "enhance" log entries with custom labels. However, it appears that the page does not apply to standard environment.
Is it possible to add labels (or any information associated with the log entry other than app log lines) to a log entry in request_log and how? If not, what are the alternatives?
The log enhancer would allow you to add a custom labels although it would be hard coded as this function (enhanceLogEntry(LogEntry.Builder logEntry)) is called at the end of the request when the log is being populated. Supplying a value from the request to appear in the request_log made by the application would not be possible.
I do not see how it would be limited to the flexible environment, you should be able to do it from the standard environment as far as I can tell.
Alternatively, I believe the best path would be to write your own logs entries by using the Stackdriver Logging Client Libraries within your request code.
Related
I am using OpenTelemetry java auto instrumentation in my spring boot app. Is there a way to make the application logs part of the spans that are created?
My autoconfig settings are as below:
-Dotel.traces.exporter=jaeger
-Dotel.metrics.exporter=none
-Dotel.exporter.jaeger.endpoint=http://localhost:14250
-Dotel.resource.attributes=service.name=myService
-javaagent:C:/path/to/opentelemetry-javaagent-1.0.1-all.jar
OpenTelemetry ships logs separately to the telemetry data obtained from auto instrumentation, and does not interleave log data I'm afraid. We ship our logs via the use of FluentBit (https://medium.com/opentelemetry/introducing-the-fluentbit-exporter-for-opentelemetry-574ec133b4b4).
You may wish to use manual instrumentation and add spans, span attributes and/or events to pertinent code blocks, to add log like context to the metadata utilised downstream.
As you are using Spring Boot, it would be advisable to use one of the starter dependencies, such as opentelemetry-otlp-exporter-starter (https://github.com/open-telemetry/opentelemetry-java-instrumentation/tree/main/instrumentation/spring/starters/otlp-exporter-starter), which should get you most of the way there. You want to use the #WithSpan annotation to decorate your methods, which will enable you to obtain the current span easily. See https://opentelemetry.lightstep.com/java/.
The official docs have a few examples, that may help, but be aware that the API and SDK are changing rapidly, so examples don't always work - https://opentelemetry.io/docs/java/manual_instrumentation/.
Detailed information regarding OpenTelemetry and logging: https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/logs/overview.md
Adding logs to spans depends a bit on the backend you are using for collecting the traces/spans and visualizing them.
I used Jaegar, which interprets OTEL events as logs in the UI, and so I wrote a custom log appender which put app logs into an event, which was picked up subsequently in the UI.
More details here:
https://stackoverflow.com/a/68739794/2715083
Executive Summary
How do I filter by the servlet in which the log message was invoked? (presently using 2.0 beta8)
Why on earth I would want to do that...
I have several existing web applications. They were written to rely on a proprietary logging system. I have re-implemented a key class from the proprietary system from scratch and added it as a class the proprietary system as a jar and log4j 2.0 as jars in tomcat, thereby utilizing the class loading load order in tomcat to divert the proprietary system into log4j. This succeeds and my log4j config now controls everything (Yay!).
But... (There's always a "But"!)
I was very pleased until I discovered that with all 4 applications deployed in the same container, they were not coordinating their writes to the single log file in the single configuration I had placed in conf/log4j2.xml (and specifed by passing -Dlog4j.configurationFile=/mnt/kui/tomcat/conf/log4j2.xml on the command line). I found some log messages with much earlier time stamps (hours earlier) in the middle of the log file. Out of order logs (and possibly overwritten log lines?) are not desirable of course.
I actually don't want them all in one file anyway and would prefer a log per application controlled by a single config file. Initially I thought this would be easy to achieve since log4j automatically sets up a LoggingContext with the name of the web application.
However I can't seem to find a filter implementation that will allow me to filter on the LoggingContext. I understand that from each application's perspective there is only one logging context (I think), but the same config file is read by 4 applications so from the config perspective LoggingContext is not unique.
I'm looking for a way to route each application to it's own file without having a config file for every application, or having to add classes to all the applications or edit war files (including web.xml). I'm sooo... close but It's not working.
Just to complicate matters, there is a jar file we wrote that is shared among all 4 applications that uses this logging too and one application has converted to using log4j directly in it's classes (but it still uses proprietary classes that reference the proprietary logging class that I replaced).
I have already seen http://logging.apache.org/log4j/2.x/manual/logsep.html and my case seems closest to '"Shared" Web Applications and REST Service Containers' but that case doesn't seem very well covered by that page.
You may want to look at the RoutingAppender which can be used to separate log files based on data in your ThreadContextMap. You could use the web app name as a unique key.
About the out of order logs, there was an issue with FastFileAppender in older betas. If append was false, the old file was not truncated but new log events would start to overwrite the old file from the beginning. (So after your most recent log event you would see yesterday's log events, for example). What version are you using?
I am using Log4j logging framework to insert the log into oracle database.But the insert query in the log4j properties file is taking a lot of time to execute and making the application very slow.When I removed the logging statements from the java code, the application worked fine.At first, I thought that the insertion into DataBase is taking time , but writing the log on an external file also takes a lot of time.
Can anyone please suggest a solution?
Thank You,
Dhaval Maheshwari.
If you application is under development then log level should be debug and before logging you should check for isDebugEnabled() and then log your string.
but If your application is in production then log level should be info and you must log minimal information in log file.
Always use atleast two log level in your application one for debuggnig
mode(for development environment) and another for production mode and
production log should be minimal.
This is the way you can speed up you applicaiton.
and second thing if you want to persist your logs into database then
create a scheduler task whose responsibility would be reading logs
from flat file and persisting them into database and schedule this to
run only once in a day.
I suggest not to follow the technique u r following now.
First of all I am not sure why u r trying to log the output of log4j in DB.
Anyways if it is that necessary try something like this. Let the logfile write into a file as it is and later run a thread to dump this file from the disk when file is closed to the database as a batch process.
In this case your application will be separated from the latency of DB.
There are other solutions also using a JMS.
Where you can write it to a JMS queue and the consumer on the other hand can read the queue and write it a DB.
It depends on the kind of problem you are trying to solve though.
See of it helps
In Logging there are levels included in. For example in production only log application level exceptions and errors[ERROR level].
If it's tracking logs(Such as user actions) don't write them to files, directly add them to database. Hope this helps.
I'm working with core java and IBM Websphere MQ 6.0. We have a standalone module say DBcomponent that hits the database and fetches a resultset based on the runtime query. The query is passed to the application via MQ messaging medium. We have a trigger configured for the queue which invokes the DBComponent whenever a message is available in the queue. The DBComponent consumes the message, constructs the query and returns the resultset to another queue. In this overall process we use log4j to log statements on a log file for auditing.
The connection is pooled to the database using Apache pool. I am trying to check whether the log messages are logged correctly using a sample program. The program places the input message to the queue and checks for the logs in the log file. Its expected for the trigger method invocation to complete before i try to check for the message in log file, but every time my program to check for log message gets executed first leading my check to failure.
Even if i introduce a Thread.sleep(time) doesn't solves the case. How can i make it to keep my method execution waiting until the trigger operation completes?
Any suggestion will be helpful.
I suggest you go and read up about the concurrency primitives that Java offers you. http://tutorials.jenkov.com/java-concurrency/index.html seems to cover the bases, the Thread Signalling chapter in particular.
I would recommend against relying on log4j (or any logging functionality) even in a simple test program.
Have your test run as you would expect it to, putting debugging/tracing statements in the log as you see fit (be liberal about it, log4j is very fast!) Then, when it's done, check the log yourself.
Writing log parsing will only complicate your goals.
Write your test, view the result, view the logs. If you want automated testing, consider setting up a functional test. You can set up tests free using Selenium. (http://seleniumhq.org/) There's no need to write your own functional testing/parsing stuff when there's easy to configure, easy to use, easy to customize frameworks out there! :-)
There is only one file. And it is written simultaneously as web app copies run.
How do you filter only one session log messages from other log lines?
Using a servlet filter with either NDC or MDC information is the best way I've seen. A quick comparison of the two is available at http://wiki.apache.org/logging-log4j/NDCvsMDC.
I've found MDC has worked better for me in the past. Remember that you'll need to update your log4j properties file to include whichever version you prefer (pattern definitions at http://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/PatternLayout.html).
A full example of configuring MDC with a servlet filter is available at http://veerasundar.com/blog/2009/11/log4j-mdc-mapped-diagnostic-context-example-code/.
A slightly easier to configure, but significantly inferior option: You could opt to just print out the thread ID (via the properties file) for each request and make sure that the first thing you log about each request is a session identifier. It isn't as proper (or useful), but it can work for low-volume applications.
You could set a context message including the identifier of the specific app instance using org.apache.log4j.NDC, like this:
String appInstanceId = "My App Instance 1";
org.apache.log4j.NDC.push(appInstanceId);
// handle request
org.apache.log4j.NDC.clear();
You can set up the context during the initialization of your web app instance, or inside the doPost() method of your servlets. As its name implies, you can also nest contexts within contexts with multiple push calls at different levels.
See the section "Nested Diagnostic Contexts" in the Log4J manual.
Here is a page that sets up an MDC filter for web-app -> http://rtner.de/software/MDCUserServletFilter.html
Being a servlet filter it will free you from managing MDC/NDC in each of your servlets.
Of course, you should modify it to save information more pertinent to your web-app.
If you want to differentiate sessions in the same application then the MDC is the way to go. But if you want to differentiate the web applications writing to the same file, then MDC won't help because it works on a thread basis. In such case I used to make my own appender which knows which application instance it serves. This can be done through appender configuration properties. Such appender would stick application name into each logging event as a property before writing it into the media, and then you can use a layout to show this property value in the text file it writes to. Using MDC in such case won't work because every thread will have to MDC.put(applicationName) and that is quite ugly. MDC is only good for single process, not for several processes. If someone knows the other way, I'd like to hear.