I am using EFK.
Could someone help here on how to enable fluentd plugins.
I have installed the following 3 plugins as part of my work.
gem install fluent-plugin-concat
gem install fluent-plugin-detect-exceptions
gem install fluent-plugin-detect-exceptions-with-error
gem install fluent-plugin-record-modifier
And I am seeing the list of gems installed here:
opsmxuser#opsmx-ind-setup:~/docker-compose-efk$ fluent-gem list
*** LOCAL GEMS ***
.
.
fluent-plugin-concat (2.4.0)
fluent-plugin-detect-exceptions (0.0.12)
fluent-plugin-detect-exceptions-with-error (0.0.3)
fluent-plugin-record-modifier (2.0.1)
fluentd (1.6.2, 0.14.25)
.
.
But these 3 plugins are not enabled. When we are using these configurations into fluent.conf file getting following error messages into fluentd container logs.
2019-08-05 12:34:55 +0000 [info]: parsing config file is succeeded path="/fluentd/etc/fluent.conf"
2019-08-05 12:34:55 +0000 [error]: config error file="/etc/fluent/fluent.conf" error_class=Fluent::ConfigError error="Unknown filter plugin 'concat'. Run 'gem search -rd fluent-plugin' to find plugins"
2019-08-02 12:46:23 +0000 [error]: config error file="/fluentd/etc/fluent.conf" error_class=Fluent::ConfigError error="Unknown output plugin 'detect_exceptions'. Run 'gem search -rd fluent-plugin' to find plugins"
2019-08-05 12:34:55 +0000 [info]: parsing config file is succeeded path="/fluentd/etc/fluent.conf"
2019-08-05 12:34:55 +0000 [error]: config error file="/fluentd/etc/fluent.conf" error_class=Fluent::ConfigError error="Unknown filter plugin 'record_modifier'. Run 'gem search -rd fluent-plugin' to find plugins"
Add to fluentd config #type copy directive.
Example below:
<match **>
#type copy
<store>
#type exec
#id load2clickhouse
command bash /usr/local/bin/insert_ch.sh {{ template "fluentdInsertTable" $ }}
format json
<buffer>
#type memory
chunk_limit_size 32m
queue_limit_length 32
flush_at_shutdown true
flush_interval 15s
flush_thread_count 4
</buffer>
</store>
<store>
#type elasticsearch
host ip
port 5044
logstash_format true
</store>
</match>
I also encountered the same problem today.
I follow the fluent github issue, build a image with plugin in it
build your own image
it works for me
Related
Im using kubenetes and I have a pod contain ignite db I added to the pod another container - sscaling/jmx-prometheus-exporter:latest
I read in git I should run this
To run as a javaagent download the jar and run:
java -javaagent:./jmx_prometheus_javaagent-0.14.0.jar=8080:config.yaml -jar yourJar.jar
but I didnt understand, should I also download the file for the kubenrtes container as well?
can someone assist how can I continue from here
I have the following default configuration:
---
hostPort: localhost:5555
username:
password:
rules:
- pattern: ".*"
I have a Spring Web Application in a DevOps repository, with a .yml that looks like this (as generated by the tool in the DevOps web client):
# Build your Java project and deploy it to Azure as a Linux web app
# Add steps that analyze code, save build artifacts, deploy, and more:
# https://learn.microsoft.com/azure/devops/pipelines/languages/java
trigger:
- master
variables:
Version: '0.0.1'
# Azure Resource Manager connection created during pipeline creation
azureSubscription: '-snipped-'
# Web app name
webAppName: 'SlackCentralTestApp'
# Environment name
environmentName: 'SlackCentralTestApp'
# Agent VM image name
vmImageName: 'ubuntu-latest'
stages:
- stage: Build
displayName: Build stage
jobs:
- job: MavenPackageAndPublishArtifacts
displayName: Maven Package and Publish Artifacts
pool:
vmImage: $(vmImageName)
steps:
- task: Maven#3
displayName: 'Maven Package'
inputs:
mavenPomFile: 'pom.xml'
publishJUnitResults: true
testResultsFiles: '**/surefire-reports/TEST-*.xml'
javaHomeOption: 'JDKVersion'
jdkVersionOption: '1.11'
mavenVersionOption: 'Default'
mavenAuthenticateFeed: false
effectivePomSkip: false
sonarQubeRunAnalysis: false
- task: CopyFiles#2
displayName: 'Copy Files to artifact staging directory'
inputs:
SourceFolder: '$(System.DefaultWorkingDirectory)'
Contents: '**/target/SlackbotTest-$(Version)-SNAPSHOT.?(war|jar)'
TargetFolder: $(Build.ArtifactStagingDirectory)
- upload: $(Build.ArtifactStagingDirectory)
artifact: drop
- stage: Deploy
displayName: Deploy stage
dependsOn: Build
condition: succeeded()
jobs:
- deployment: DeployLinuxWebApp
displayName: Deploy Linux Web App
environment: $(environmentName)
pool:
vmImage: $(vmImageName)
strategy:
runOnce:
deploy:
steps:
- task: AzureWebApp#1
displayName: 'Azure Web App Deploy: SlackCentralTestApp'
inputs:
azureSubscription: 'Azure for Students (4998490e-1bc4-43fc-a370-80744706d1f5)'
appType: 'webAppLinux'
appName: 'SlackCentralTestApp'
package: '$(Pipeline.Workspace)/drop/target/SlackbotTest-$(Version)-SNAPSHOT.jar'
runtimeStack: 'JAVA|11-java11'
startUpCommand: 'java -jar $(Pipeline.Workspace)/drop/target/SlackbotTest-$(Version)-SNAPSHOT.jar'
The deployment process seems to be successful as seen in this screenshot, yet when I take a look at the server log, I sadly get greeted with the following error:
2020-04-14T09:03:55.658599508Z Initializing App Insights applicationinsights-agent-codeless-2.5.0.jar....
2020-04-14T09:03:55.669449167Z STARTUP_FILE=
2020-04-14T09:03:55.676322304Z STARTUP_COMMAND=java -jar /home/vsts/work/1/drop/target/SlackbotTest-0.0.1-SNAPSHOT.jar
2020-04-14T09:03:55.676350305Z No STARTUP_FILE available.
2020-04-14T09:03:55.676428905Z Running STARTUP_COMMAND: java -jar /home/vsts/work/1/drop/target/SlackbotTest-0.0.1-SNAPSHOT.jar
2020-04-14T09:03:55.681352232Z Error: Unable to access jarfile /home/vsts/work/1/drop/target/SlackbotTest-0.0.1-SNAPSHOT.jar
2020-04-14T09:03:55.694784805Z Finished running startup command 'java -jar /home/vsts/work/1/drop/target/SlackbotTest-0.0.1-SNAPSHOT.jar'. Exiting with exit code 1.
This would lead me to conclude that the path as given in the 'startUpCommand' property from the .yml file is incorrect, yet I have not been able to find what the correct path should be.
I have attempted the following:
Specify no directories, only the filename. Leads to the same result, sadly.
Use the 'find' command in Bash to find any .jars on the Web App, which tells me that there are none.
Manually building the application from the Web App does not seem to be an option either, as it's a Java 11 application and the java SE that seems to be included is version 1.8, regardless of the version I specify during the creation of the Web App resource in Azure.
Update 1:
Modified the version of hadoop to 2.x but the error is still there.
Original:
I generated tpcds test data into Ceph with hive-testbench.
Currently, the data is located at root directory of storage system, and is in folder tpcds.
For example, the result of hdfs dfs -ls / is
drwxrwxrwx - root root 0 2019-08-05 10:44 /hive
drwxrwxrwx - root root 0 2019-08-05 10:44 /tmp
drwxrwxrwx - root root 0 2019-08-05 10:44 /tpcds
drwxrwxrwx - root root 0 2019-08-05 10:44 /user
drwxrwxrwx - root root 0 2019-08-05 10:44 /warehouse
The result of s3cmd ls s3://tpcds is:
DIR s3://tpcds/hive/
DIR s3://tpcds/tmp/
DIR s3://tpcds/tpcds/
DIR s3://tpcds/user/
DIR s3://tpcds/warehouse/
For s3cmd ls s3://tpcds, the bucket name is tpcds.
When the data is ready, the next setup is to create external table in Hive to get access to those data. The reason why I show the storage layout is to make sure by you guys that the issue I met has nothing to do with the path.
The command used is hive -i settings/load-flat.sql -f ddl-tpcds/text/alltables.sql -d DB=tpcds_text_7 -d LOCATION=tpcds/7, however, I met issue below:
exec.DDLTask: org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:Exception thrown flushing changes to datastore)
at org.apache.hadoop.hive.ql.metadata.Hive.createDatabase(Hive.java:433)
at org.apache.hadoop.hive.ql.exec.DDLTask.createDatabase(DDLTask.java:4243)
For the stack version: Hive 2.3.2, Hadoop 3.1.2.
Currently, the most possible reason from my side is about the hadoop version, I'm going to degrade it to hadoop 2.7 to see if same error occurs.
And at the same time, any comment is welcomed. Thanks for your help in advance.
Since the issue is solved, I'd like to post the solution here for further visitors who might meet same issue.
The hive I used to initialize schema of mysql metastore is 3.1.1. After that, I just replaced the hive folder with hive 2.3.2. This kind of downgrade is not graceful, the metastore created before is not consistent with hive 2.3.2 and that's the reason why I met the issue.
I reverted the hive folder to 3.1.1 and everything turns fine.
I am trying to run a basic robotframework test with RemoteSwingLibrary, but I can't seem to get it to work.
I was able to run the following test just fine with SwingLibrary:
*** Settings ***
Library SwingLibrary
*** Test Cases ***
Foobar Test
Start Application fully.qualified.name.of.my.class
I am running it from within Eclipse via the robot framework standalone jar. I put robotframework-3.0.2.jar and swinglibrary-1.9.7.jar on my class path, created a new run configuration using my projects default classpath with the main class being org.robotframework.RobotFramework. In the arguments, I specify the name of my test file.
So, that seems to work fine. Then I tried to do almost exactly the same thing with RemoteSwingLibrary:
*** Settings ***
Library RemoteSwingLibrary
*** Test Cases ***
Foobar Test
Start Application my_app java fully.qualified.name.of.my.class
I replaced the swinglibrary jar on my classpath with remoteswinglibrary-2.2.1.jar and I ran the same test. The output says:
console: Failed to install '': java.nio.charset.UnsupportedCharsetException: cp0.
==============================================================================
Remotetest
==============================================================================
Foobar Test | FAIL |
RemoteSwingLibraryTimeoutError: Agent port not received before timeout
------------------------------------------------------------------------------
Remotetest | FAIL |
1 critical test, 0 passed, 1 failed
1 test total, 0 passed, 1 failed
==============================================================================
Output: C:\<path>\output.xml
Log: C:\<path>\log.html
Report: C:\<path>\report.html
(note that the "console: Failed to install..." part is normal and I don't think I need to be concerned about it)
When I dig into the report for the Start Application Keyword, it says
13:48:40.070 INFO Link to stdout
13:48:40.070 INFO Link to stderr
13:48:40.070 INFO -javaagent:"C:<pathToProject>\__pyclasspath__"=127.0.0.1:63110
13:48:40.071 INFO Starting process:
java fully.qualified.name.of.my.class
13:49:40.145 INFO Failed to start application: Traceback (most recent call last):
File "__pyclasspath__/RemoteSwingLibrary.py", line 307, in start_application
File "__pyclasspath__/RemoteSwingLibrary.py", line 353, in _application_started
File "__pyclasspath__/RemoteSwingLibrary.py", line 373, in _get_agent_address
RemoteSwingLibraryTimeoutError: Agent port not received before timeout
13:49:40.155 INFO Waiting for process to complete.
13:49:40.165 INFO Process completed.
13:49:40.165 INFO STDOUT: Error occurred during initialization of VM
agent library failed to init: instrument
13:49:40.175 INFO STDERR: Error opening zip file or JAR manifest missing : C:<pathToProject>\__pyclasspath__
Picked up JAVA_TOOL_OPTIONS: -javaagent:"C:<pathToProject>\__pyclasspath__"=127.0.0.1:63110
Picked up _JAVA_OPTIONS: -Djava.security.policy="C:\<UserFolder>\
AppData\Local\Temp\2\grant_all_f_vfit.policy"
13:49:40.195 FAIL RemoteSwingLibraryTimeoutError: Agent port not received before timeout
I'm not sure if this hunch is really founded on anything, but my suspicion is that there's a classpath issue. I would think that when I call the Start Application keyword with "java fully.qualified.name.of.my.class" I would probably have to pass along my classpath. I was hoping that maybe I wouldn't since the non-remote version seems to handle that just fine (probably because its running my app in the same JVM as I launched robot framework in, and that already included everything I needed in my classpath).
So if that's the issue, I can probably fix it by using the -cp option in my java command. However, the classpath for this particular project is actually quite long (my project is dependent on a lot of other projects) and I would rather not have to list it all out. Also, when I add a new dependency down the road, I would rather not have to update the classpath in my test file as well.
Is there some magic way that I can pass the current classpath along to the remote app? Or perhaps some other solution to this problem?
Update
I tried again with -cp and my giant classpath and it gave me the same errors, so its possible that my classpath has nothing to do with whatever is going on.
I also tried specifying the port number explicitly and it gives me slightly different output (though it still doesn't work). I also discovered that there's a debug setting that gives a little more detail. Here's me new setup:
*** Settings ***
Library RemoteSwingLibrary debug=True port=8242
*** Test Cases ***
Foobar Test
Start Application my_app java -cp <classpath> <fully.qualified.name.of.my.class> remote_port=8242
And here is my new output:
13:48:40.070 INFO Link to stdout
13:48:40.070 INFO Link to stderr
13:48:40.070 DEBUG Picked old JAVA_TOOL_OPTIONS=''
13:48:40.070 DEBUG Picked old _JAVA_OPTIONS=''
13:48:40.070 INFO -javaagent:"C:<pathToProject>\__pyclasspath__"=127.0.0.1:51509:DEBUG
13:48:40.070 DEBUG Set JAVA_TOOL_OPTIONS='-javaagent:"C:<pathToProject>\__pyclasspath__"=127.0.0.1:8242:APPORT=8242:DEBUG'
13:48:40.070 DEBUG SET _JAVA_OPTIONS='-Djava.security.policy="C:\<UserFolder>\AppData\Local\Temp\4\grant_all_haud6t.policy"'
13:48:40.071 INFO Starting process:
java -cp <classpath> <fully.qualified.name.of.my.class>
13:48:40.071 DEBUG Process configuration:
cwd: C:<PathToProject>
shell: True
stdout: C:<PathToProject>\<someBigPath>.txt
stderr: C:<PathToProject>\<someBigPath>.txt
alias: my_app
env: None
13:48:40.100 DEBUG Returned old JAVA_TOOL_OPTIONS=''
13:48:40.100 DEBUG Returned old _JAVA_OPTIONS=''
13:48:40.102 INFO connection to started application at 127.0.0.1:8242
13:48:40.120 DEBUG remote swinglibrary instantiated
13:48:40.121 DEBUG remote services instantiated
13:48:40.121 INFO waiting for api at 127.0.0.1:8242
13:49:40.145 INFO Failed to start application: Traceback (most recent call last):
File "__pyclasspath__/RemoteSwingLibrary.py", line 307, in start_application
File "__pyclasspath__/RemoteSwingLibrary.py", line 357, in _application_started
File "__pyclasspath__/RemoteSwingLibrary.py", line 344, in _wait_for_api
RuntimeError: Connecting to api at 127.00.1:8242 has failed: ''
13:49:40.155 INFO Waiting for process to complete.
13:49:40.165 INFO Process completed.
13:49:40.165 INFO STDOUT: Error occurred during initialization of VM
agent library failed to init: instrument
13:49:40.175 INFO STDERR: Error opening zip file or JAR manifest missing : C:<pathToProject>\__pyclasspath__
Picked up JAVA_TOOL_OPTIONS: -javaagent:"C:<pathToProject>\__pyclasspath__"=127.0.0.1:8242:APPORT=8242:DEBUG
Picked up _JAVA_OPTIONS: -Djava.security.policy="C:\<UserFolder>\
AppData\Local\Temp\2\grant_all_f_vfit.policy"
13:49:40.195 FAIL Connecting to api at 127.0.0.1:8242 has failed: ''
13:49:40.195 DEBUG Traceback (most recent call last):
File "__pyclasspath__/RemoteSwingLibrary.py", line 498, in run_keyword
File "__pyclasspath__/RemoteSwingLibrary.py", line 307, in start_application
File "__pyclasspath__/RemoteSwingLibrary.py", line 357, in _application_started
File "__pyclasspath__/RemoteSwingLibrary.py", line 344, in _wait_for_api
Yet another update
I think that this may have something to do with the "__pyclasspath__" that shows up in the error message. I put my java command in a .bat file preceded by a "set" command to list my environment variables. I can run the .bat file from the command line just fine (it launches the java app).
Then I modified my .robot file to do:
Foobar Test
Start Application my_app cd robotframework/tests && Run.bat remote_port=8242
I get essentially the same error output that I got before:
Error opening zip file or JAR manifest missing : C:<pathToProject>\__pyclasspath__
Picked up JAVA_TOOL_OPTIONS: -javaagent:"__pyclasspath__"=127.0.0.1:8242:APPORT=8242:DEBUG
Picked up _JAVA_OPTIONS: -Djava.security.policy="C:\\AppData\Local\Temp\1\grant_all_lgirs7.policy"
The environment that my batch script prints out appears the same except for JAVA_TOOL_OPTIONS and _JAVA_OPTIONS (which match with what the error output says). There is no environment variable called __pyclasspath__ so I'm wondering if that's the issue.
I set JAVA_TOOL_OPTIONS in my cmd environment to match the output that I saw from my robot run and now I get the same error message when I try to run my .bat script from the command line, so that environment variable seems to be the issue.
Well, I came up with a fix, though its definitely not an ideal fix.
I found out that when the framework kicks of my remote app, it sets:
JAVA_TOOL_OPTIONS=-javaagent:"C:\<pathToProject>\__pyclasspath__
But __pyclasspath__ is not actually an environment variable. I figured out that the javaagent has to point to the remote framework library, so I decided to just set it explicitly.
So now I changed my robot file to something like this:
*** Settings ***
Library RemoteSwingLibrary
*** Test Cases ***
Foobar Test
Start Application my_app run.bat remote_port=1234
And then made a file called run.bat that does:
set JAVA_TOOL_OPTIONS=-javaagent:"C:\<pathToProject\lib\remoteswinglibrary-2.2.1.jar"=127.0.0.1:1234:APPORT=1234
java -cp <classPath> fully.qualified.name.of.class
This is obviously sort of a hacky solution and it will take a little work to make it portable (I'll need to get rid of the hardcoded "pathToProject" and also make it support Linux environments as well). It also requires hardcoding in a port, whereas it would be nice if the robotframework could just select a port for me.
So, I would definitely like to see a better solution if anyone can find one, but for now, this will at least work.
I am new to Apache OFBiz. I have downloaded and installing the OFBiz by instructions. I have loaded the data successfully.
But when am running OFBiz using gradlew it stopped at birt-container. I have added the console information here.
Build process stopped at 91% with the message of "Started container birt-container" for 5 hours and I have stopped the process. Still no idea why it is stopping here.
2017-05-04 11:01:42,143 |main |CatalinaContainer |I| Connector HTTP/1.1 # 8443 - secure [org.apache.coyote.http11.Http11NioProtocol] started.
2017-05-04 11:01:42,143 |main |CatalinaContainer |I| Started Apache Tomcat/8.0.37
2017-05-04 11:01:42,143 |main |ContainerLoader |I| Started container catalina-container
2017-05-04 11:01:42,143 |main |ContainerLoader |I| Starting container birt-container
2017-05-04 11:01:42,143 |main |BirtContainer |I| Start BIRT container
2017-05-04 11:01:42,152 |main |BirtContainer |I| Startup BIRT platform
2017-05-04 11:01:43,034 |main |BirtContainer |I| Create factory object
2017-05-04 11:01:43,052 |main |BirtContainer |I| Create report engine
2017-05-04 11:01:43,130 |main |BirtContainer |I| BIRT supported formats: xlsx, postscript, docx, pptx, pdf, ppt, xls_spudsoft, doc, html,
odp, xls, odt, ods
2017-05-04 11:01:43,130 |main |ContainerLoader |I| Started container birt-container
> Building 91% > :ofbiz
The Gradle percentage is bit misleading here, it does not reach 100% as long as OFBiz is running. I think you just assumed after building the service will be up. Not like that. Once the Service is up it will be in 91%. But the Service has been started. You can access the E-Commerce and other services. Once you are running service get the following URL (ex).
localhost:8443/ap/control/main
You will be redirected to admin login page. Give user as "admin" and password as "ofbiz". Do the changes and test your data as you want in Accounting, Manufacturing, HR , etc..
Note: After completed your process stop the process using "stopofbiz" shell script file under tools folder. After stop the service you can able to see the build will be reach 100% in service console.
This is a duplicate to Installing OfBiz in debug mode hangs
I think OFBiz does not hang and that you should be able to access and work with it without problems. The Gradle percentage is bit misleading here, it does not reach 100% as long as OFBiz is running.
See also https://lists.apache.org/thread.html/26543a82adf1f8d9a8bec47975e8d87f750c2de2f2da7e242b04a72e#%3Cuser.ofbiz.apache.org%3E
Prepare OFBiz:
Clean all:
gradlew cleanAll loadDefault testIntegration
Note: Depending on your Internet connection speed it might take a long time for this step to complete if you are using OFBiz for the first time as it needs to download all dependencies. So please be patient!
MS Windows: gradlew cleanAll loadDefault
Unix-like OS: ./gradlew cleanAll loadDefault
Start OFBiz:
MS Windows: gradlew ofbiz
Unix-like OS: ./gradlew ofbiz
Reference: After download http://www.apache.org/dyn/closer.lua/ofbiz/apache-ofbiz-16.11.02.zip , unzip, look for
apache-ofbiz-16.11.02/tools/documentation/README.md.html