I am using OpenTelemetry java auto instrumentation in my spring boot app. Is there a way to make the application logs part of the spans that are created?
My autoconfig settings are as below:
-Dotel.traces.exporter=jaeger
-Dotel.metrics.exporter=none
-Dotel.exporter.jaeger.endpoint=http://localhost:14250
-Dotel.resource.attributes=service.name=myService
-javaagent:C:/path/to/opentelemetry-javaagent-1.0.1-all.jar
OpenTelemetry ships logs separately to the telemetry data obtained from auto instrumentation, and does not interleave log data I'm afraid. We ship our logs via the use of FluentBit (https://medium.com/opentelemetry/introducing-the-fluentbit-exporter-for-opentelemetry-574ec133b4b4).
You may wish to use manual instrumentation and add spans, span attributes and/or events to pertinent code blocks, to add log like context to the metadata utilised downstream.
As you are using Spring Boot, it would be advisable to use one of the starter dependencies, such as opentelemetry-otlp-exporter-starter (https://github.com/open-telemetry/opentelemetry-java-instrumentation/tree/main/instrumentation/spring/starters/otlp-exporter-starter), which should get you most of the way there. You want to use the #WithSpan annotation to decorate your methods, which will enable you to obtain the current span easily. See https://opentelemetry.lightstep.com/java/.
The official docs have a few examples, that may help, but be aware that the API and SDK are changing rapidly, so examples don't always work - https://opentelemetry.io/docs/java/manual_instrumentation/.
Detailed information regarding OpenTelemetry and logging: https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/logs/overview.md
Adding logs to spans depends a bit on the backend you are using for collecting the traces/spans and visualizing them.
I used Jaegar, which interprets OTEL events as logs in the UI, and so I wrote a custom log appender which put app logs into an event, which was picked up subsequently in the UI.
More details here:
https://stackoverflow.com/a/68739794/2715083
Related
I traversed too many articles but didn't find anything about auto-instrumentation of Quarkus application,
Manual Instrumentation for Quarkus application is given below.
adding this implementetion will automatically add all the required dependency for tracing to app.
implementation 'io.quarkus:quarkus-opentelemetry-exporter-otlp'
In application.properties after mentioning required configurations , we will be able to send traces to collector.
quarkus.application.name=manualInstrumentationForQuarkus
quarkus.opentelemetry.enabled=true
quarkus.opentelemetry.tracer.exporter.otlp.endpoint=http://localhost:4317
quarkus.opentelemetry.tracer.exporter.otlp.headers=Authorization=Bearer my_secret
Please answer me with any autoinstrumentation method where we can use agent to get traces from application without adding any dependency.
The recommended way to use OpenTelemetry on Quarkus is using the extension dependency you mention.
The OpenTelemetry Java Agent would provide some output but there are many context switching situations that will fail with the agent, leading to missing spans.
I have a Java application running on Google App Engine standard environment.
I am able to log from it (using JUL). In standard environment, all the application log lines from a single web request are grouped into a single entry in the request_log. Everything runs great
However, now I have a requirement to add custom labels to a log entry for a request. For example, what is user ID associated with it.
Stackdriver documentation (https://cloud.google.com/logging/docs/setup/java) gives example how to "enhance" log entries with custom labels. However, it appears that the page does not apply to standard environment.
Is it possible to add labels (or any information associated with the log entry other than app log lines) to a log entry in request_log and how? If not, what are the alternatives?
The log enhancer would allow you to add a custom labels although it would be hard coded as this function (enhanceLogEntry(LogEntry.Builder logEntry)) is called at the end of the request when the log is being populated. Supplying a value from the request to appear in the request_log made by the application would not be possible.
I do not see how it would be limited to the flexible environment, you should be able to do it from the standard environment as far as I can tell.
Alternatively, I believe the best path would be to write your own logs entries by using the Stackdriver Logging Client Libraries within your request code.
I am very much new to multi-tenancy. We have an application based on Java, Spring, Hibernate/JPA etc. which doesn't support multi-tenancy.
Now, we want to convert that application into multi-tenant one. I have read about multi-tenancy and even wrote one standalone application using hibernate with separate schema approach. Link referred is here.
I think about the logging part which is bound to be changed now as log files will be maintained per tenant(client) now. So, for each tenant a separate log file will be there.Also, log file for a particular tenant shouldn't be accessed by another tenant.
Is there any logging API specific to support multi-tenancy? If not, how should i go ahead with implementing logging in multi-tenant application? What should be taken care of while implementing logging in multi-tenant application.
you can use MDC (mapped diagnostic context) support to route logging for each tenant into a separate file/dir/whatever.
you can read up on the concept here. it exists in slf4/logback and log4j
simply put, you set some property like tenantName in the MDC at the beginning of every request processing according to the specific tenant making the request and then use this property in your logging configuration to determine the log file into which log messages are written.
Executive Summary
How do I filter by the servlet in which the log message was invoked? (presently using 2.0 beta8)
Why on earth I would want to do that...
I have several existing web applications. They were written to rely on a proprietary logging system. I have re-implemented a key class from the proprietary system from scratch and added it as a class the proprietary system as a jar and log4j 2.0 as jars in tomcat, thereby utilizing the class loading load order in tomcat to divert the proprietary system into log4j. This succeeds and my log4j config now controls everything (Yay!).
But... (There's always a "But"!)
I was very pleased until I discovered that with all 4 applications deployed in the same container, they were not coordinating their writes to the single log file in the single configuration I had placed in conf/log4j2.xml (and specifed by passing -Dlog4j.configurationFile=/mnt/kui/tomcat/conf/log4j2.xml on the command line). I found some log messages with much earlier time stamps (hours earlier) in the middle of the log file. Out of order logs (and possibly overwritten log lines?) are not desirable of course.
I actually don't want them all in one file anyway and would prefer a log per application controlled by a single config file. Initially I thought this would be easy to achieve since log4j automatically sets up a LoggingContext with the name of the web application.
However I can't seem to find a filter implementation that will allow me to filter on the LoggingContext. I understand that from each application's perspective there is only one logging context (I think), but the same config file is read by 4 applications so from the config perspective LoggingContext is not unique.
I'm looking for a way to route each application to it's own file without having a config file for every application, or having to add classes to all the applications or edit war files (including web.xml). I'm sooo... close but It's not working.
Just to complicate matters, there is a jar file we wrote that is shared among all 4 applications that uses this logging too and one application has converted to using log4j directly in it's classes (but it still uses proprietary classes that reference the proprietary logging class that I replaced).
I have already seen http://logging.apache.org/log4j/2.x/manual/logsep.html and my case seems closest to '"Shared" Web Applications and REST Service Containers' but that case doesn't seem very well covered by that page.
You may want to look at the RoutingAppender which can be used to separate log files based on data in your ThreadContextMap. You could use the web app name as a unique key.
About the out of order logs, there was an issue with FastFileAppender in older betas. If append was false, the old file was not truncated but new log events would start to overwrite the old file from the beginning. (So after your most recent log event you would see yesterday's log events, for example). What version are you using?
I was going through details of CAS project and found that it is using something called inspektr. I googled for some time and tried to find more details about its usage. But I did not get any information.
Can anyone provide more details about it and its usage.
Thanks in advance.
Inspektr can be found here: https://github.com/dima767/inspektr with details for usage here: https://github.com/dima767/inspektr/wiki/Inspektr-Auditing
As I understand the project, it collects information from your web flow and allows you to save said data through the use of the #Audit annotations provided. If the configuration is copied from that CAS project you linked, nearly everything's configured to log to a file. Sample data logged would be the Client's IP, remote IP, the action being performed (as configured via Spring and the #Audit annotation), as well as various other things.
If you're familiar with Spring Aspects, it should be a breeze to look through the Inspektr source code to find other uses.
Inspektr is a framework that allows us to drive audit records from Annotations utilizing an Aspect that is provided with the framework. This works for Spring Managed Beans only!
Here the github project website:
https://github.com/dima767/inspektr/wiki/Inspektr-Auditing
A good practical reference for config: https://wiki.jasig.org/display/CASUM/Auditing+and+Statistics+Via+Inspektr
The base principal here is that Inspektr allows for logging of these audit frames into the console, database, the application server log ,we can even define our own managers to log to a different medium if required.