We Keep Coding

Fluent Bit Java Multiline parser on docker container logs in Kubernetes cluster (via New Relic integration)

We're using New Relic Fluent Bit integration to send Kubernetes pod logs to New Relic. Some pods are running Java apps so we'd like to apply java multiline parsing. Unfortunately this fluent-bit conf catch logs but multiline java parsing added in a FILTER block is not working. How can we do?
apiVersion: v1
kind: ConfigMap
metadata:
name: fluent-bit-config
namespace: newrelic
labels:
k8s-app: newrelic-logging
data:
# fluent bit config file
fluent-bit.conf: |
[SERVICE]
Flush 1
Log_Level info
Daemon off
Parsers_File parsers.conf
HTTP_Server On
HTTP_Listen 0.0.0.0
HTTP_Port 2020
#INCLUDE input-kubernetes.conf
#INCLUDE filter-kubernetes.conf
#INCLUDE output-newrelic.conf
input-kubernetes.conf: |
[INPUT]
Name tail
Tag kube.*
Path /var/log/containers/*.log
Parser docker
DB /var/log/flb_kube.db
Mem_Buf_Limit 5MB
Skip_Long_Lines On
Refresh_Interval 10
filter-kubernetes.conf: |
[FILTER]
Name record_modifier
Match *
Record cluster_name ${CLUSTER_NAME}
[FILTER]
Name kubernetes
Match kube.*
Kube_URL https://kubernetes.default.svc.cluster.local:443
Merge_Log Off
[FILTER]
Name multiline
Match kube.*
multiline.key_content log
multiline.parser java
output-newrelic.conf: |
[OUTPUT]
Name newrelic
Match *
licenseKey ${LICENSE_KEY}
endpoint ${ENDPOINT}
parsers.conf: |
[PARSER]
Name docker
Format json
Time_Key time
Time_Format %Y-%m-%dT%H:%M:%S.%L
Time_Keep On

Spring: Initialize multiple connection pools at startup

I want to initialize multiple connection pools at context startup by reading connection parameters from a database table. Basically I want to address following two things
Read connection properties from database not the properties file.
Their are multiple connection pool(rows in db) details.
So my question is How can I iterate over list of rows returned by database in spring-context file and create multiple data-source objects and store them (Lets say in a map) with a unique key?
Databse table structure are somewhat like following:
+--------------+----------------+---------------+
| DBSERVERNAME | DBDRIVERCLASS | DBMINPOOLSIZE |
+--------------+----------------+---------------+
| Server1 | Mysql-Driver | 10 |
| Server2 | Oracle-Driver | 20 |
| Server3 | DB2-Driver | 10 |
+--------------+----------------+---------------+
Let me know if more details are needed. Thanks.

How to create a rule that passes only the Request Path - urlrewritefilter

Since we use ajax in our pages, We need to serve search bots requests from our Ajax Prerender service.
To do that I created a rule to modify crawling requests.
<OUR_DOMAIN>/stocks?_escaped_fragment_= is converted to http://www.MY_SEO_SERVER.com/MY_TOKEN_e291b9f/stocks
I used the following rule:
<rule>
<name>escaped_fragment</name>
<!--<condition type="request-filename" operator="notfile"/>-->
<condition type="query-string" operator="equal">.*escaped_fragment.*</condition>
<from>(.*)$</from>
<to type="proxy" last="true" qsappend="true">http://www.MY_SEO_SERVER.com/TOKEN_e291b9f78$1</to>
</rule>
The Problem is that it generates a wrong URL: http://www.MY_SEO_SERVER.com/MY_TOKEN_e291b9f/stockshttp://www.MY_SEO_SERVER.com/MY_TOKEN_e291b9f.
Would love to get advice on how to build the rule correctly.
I am using urlrewritefilter - 4.0.3
Log:
DEBUG|http-apr-8080-exec-2|o.t.w.f.u.u.ServerNameMatcher | looking for hostname match on current server name localhost
DEBUG|http-apr-8080-exec-2|o.t.w.f.u.UrlRewriteFilter | checking for status path on /stocks
DEBUG|http-apr-8080-exec-2|o.t.w.f.urlrewrite.UrlRewriter| processing request for /stocks
DEBUG|http-apr-8080-exec-2|o.t.w.f.urlrewrite.RuleBase | escaped_fragment (rule 0) run called with /stocks
DEBUG|http-apr-8080-exec-2|o.t.w.f.urlrewrite.RuleBase | matched "from"
DEBUG|http-apr-8080-exec-2|o.t.w.f.urlrewrite.Condition | evaluating "_escaped_fragment_=" against .*escaped_fragment.*
DEBUG|http-apr-8080-exec-2|o.t.w.f.urlrewrite.RuleBase | conditions match
DEBUG|http-apr-8080-exec-2|o.t.w.f.u.s.MatcherReplacer | found 1
DEBUG|http-apr-8080-exec-2|o.t.w.f.u.s.MatcherReplacer | replaced sb is http://www.MY_SEO_SERVER.com/MY_TOKEN_e291b9f/stocks
DEBUG|http-apr-8080-exec-2|o.t.w.f.u.s.MatcherReplacer | found 1
DEBUG|http-apr-8080-exec-2|o.t.w.f.u.s.MatcherReplacer | replaced sb is http://www.MY_SEO_SERVER.com/MY_TOKEN_e291b9f
DEBUG|http-apr-8080-exec-2|o.t.w.f.u.RuleExecutionOutput | needs to be
proxied from http://www.MY_SEO_SERVER.com/MY_TOKEN_e291b9f/stockshttp://www.MY_SEO_SERVER.com/MY_TOKEN_e291b9f
DEBUG|http-apr-8080-exec-2|o.t.w.f.urlrewrite.UrlRewriter| rule is last
INFO|http-apr-8080-exec-2|o.t.w.f.u.RequestProxy | execute,
target is http://www.MY_SEO_SERVER.com/MY_TOKEN_e291b9f/stockshttp://www.MY_SEO_SERVER.com/MY_TOKEN_e291b9f
INFO|http-apr-8080-exec-2|o.t.w.f.u.RequestProxy | response commit state: false
INFO|http-apr-8080-exec-2|o.t.w.f.u.RequestProxy | checking url
INFO|http-apr-8080-exec-2|o.t.w.f.u.RequestProxy | seting up the host configuration
INFO|http-apr-8080-exec-2|o.t.w.f.u.RequestProxy | config is HostConfiguration[host=http://www.MY_SEO_SERVER.com]
INFO|http-apr-8080-exec-2|o.t.w.f.u.RequestProxy | setting proxy request parameter:connection, value: keep-alive

Its seems like a bug with UrlRewriteFilter. adding "/" to the and did the trick.
This is to final rule
<rule>
<name>escaped_fragment</name>
<!--<condition type="request-filename" operator="notfile"/>-->
<condition type="query-string" operator="equal">.*escaped_fragment.*</condition>
<from>^/(.*)$</from>
<to type="proxy" last="true" qsappend="true">http://www.MY_SEO_SERVER.com/TOKEN_e291b9f78/$1</to>
</rule>

Jetty + OSGi results in strange cache errors

I'm trying to run an embedded jetty instance from an OSGi server.
When the server starts I can see the following in the log:
Started o.e.j.w.WebAppContext#1f437060{/browser,bundle://201.0:24/browser,AVAILABLE}
The first request is successful but later requests will result in a stack trace, e.g.
WARN | ResourceCache | Could not load bundle://201.0:24/browser/index.html true 1415275444922 java.nio.HeapByteBuffer[pos=0 lim=9 cap=9] java.nio.HeapByteBuffer[pos=0 lim=29 cap=29]
WARN | ResourceCache |
java.io.FileNotFoundException: \browser\index.html (Det går inte att hitta sökvägen)
at java.io.RandomAccessFile.open(Native Method)[:1.8.0_20]
at java.io.RandomAccessFile.<init>(RandomAccessFile.java:243)[:1.8.0_20]
at org.eclipse.jetty.util.BufferUtil.readFrom(BufferUtil.java:408)[201:com.test.mybundle:1.0.0]
at org.eclipse.jetty.server.ResourceCache.getIndirectBuffer(ResourceCache.java:296)[201:com.test.mybundle:1.0.0]
at org.eclipse.jetty.server.ResourceCache$Content.getIndirectBuffer(ResourceCache.java:478)[201:com.test.mybundle:1.0.0]
at org.eclipse.jetty.server.ResourceCache$Content.getInputStream(ResourceCache.java:525)[201:com.test.mybundle:1.0.0]
at org.eclipse.jetty.server.HttpOutput.sendContent(HttpOutput.java:427)[201:com.test.mybundle:1.0.0]
at org.eclipse.jetty.server.HttpOutput.sendContent(HttpOutput.java:345)[201:com.test.mybundle:1.0.0]
at org.eclipse.jetty.servlet.DefaultServlet.sendData(DefaultServlet.java:887)[201:com.test.mybundle:1.0.0]
at org.eclipse.jetty.servlet.DefaultServlet.doGet(DefaultServlet.java:493)[201:com.test.mybundle:1.0.0]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:735)[201:com.test.mybundle:1.0.0]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)[201:com.test.mybundle:1.0.0]
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:698)[201:com.test.mybundle:1.0.0]
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:505)[201:com.test.mybundle:1.0.0]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:138)[201:com.test.mybundle:1.0.0]
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:582)[201:com.test.mybundle:1.0.0]
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:213)[201:com.test.mybundle:1.0.0]
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1096)[201:com.test.mybundle:1.0.0]
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:432)[201:com.test.mybundle:1.0.0]
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:175)[201:com.test.mybundle:1.0.0]
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1030)[201:com.test.mybundle:1.0.0]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:136)[201:com.test.mybundle:1.0.0]
at org.eclipse.jetty.server.Dispatcher.forward(Dispatcher.java:261)[201:com.test.mybundle:1.0.0]
at org.eclipse.jetty.server.Dispatcher.forward(Dispatcher.java:101)[201:com.test.mybundle:1.0.0]
at org.eclipse.jetty.servlet.DefaultServlet.doGet(DefaultServlet.java:546)[201:com.test.mybundle:1.0.0]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:735)[201:com.test.mybundle:1.0.0]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)[201:com.test.mybundle:1.0.0]
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:698)[201:com.test.mybundle:1.0.0]
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:505)[201:com.test.mybundle:1.0.0]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:138)[201:com.test.mybundle:1.0.0]
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:564)[201:com.test.mybundle:1.0.0]
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:213)[201:com.test.mybundle:1.0.0]
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1096)[201:com.test.mybundle:1.0.0]
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:432)[201:com.test.mybundle:1.0.0]
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:175)[201:com.test.mybundle:1.0.0]
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1030)[201:com.test.mybundle:1.0.0]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:136)[201:com.test.mybundle:1.0.0]
at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52)[201:com.test.mybundle:1.0.0]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)[201:com.test.mybundle:1.0.0]
at org.eclipse.jetty.server.Server.handle(Server.java:445)[201:com.test.mybundle:1.0.0]
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:268)[201:com.test.mybundle:1.0.0]
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:229)[201:com.test.mybundle:1.0.0]
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.run(AbstractConnection.java:358)[201:com.test.mybundle:1.0.0]
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:601)[201:com.test.mybundle:1.0.0]
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:532)[201:com.test.mybundle:1.0.0]
at java.lang.Thread.run(Thread.java:745)[:1.8.0_20]
My guess is that the path prefix bundle:// is not supported. Could this be the problem, and if so, how could this be resolved?

ServiceMix already has a build in Web Feature. You just need install the war or http feature which will also install a Jetty Server. That's all you need to do, no extras to install in a ServiceMix/Karaf Container.

POST data from one page to another

I am trying to implement some data flow as follows
______________ _____________ _____________
| myecom.com | submit | myecom.com | add additional | payment.com |
| product.html |------------>| process.php |----------------->| pay.php |
| myform | form data | | data & submit | |
-------------- ------------- -------------
|
______________ |
| myecom.com |<--------------------------------
| receiver.php | success or failure info
--------------
An ecommerce site receive some info from user
Submit the info to an internal processor
The internal processor processes the data
The processed data along with some additional data is submitted to an external processor
The external processor sends back some success/failure report
My goal is to perform step 4 without informing the user what data is being submitted to external processor. In another word, I want to POST some data to the external processor from the internal processor.
FYI, simply cURL will not do as it does not redirect. The page must redirect to payment.com/pay.php. Also, it need to to be PHP specific, any technology like Java, etc. will do. I am considering storing the data in session using cURL and then javascript redirect. But is there any other ways?
Thanks a lot for your time
Khalid

How about doing it like this instead?
______________ _____________ _____________
| myecom.com | (1) submit | myecom.com | (2) add additional | payment.com |
| product.html |------------>| process.php |-------------------->| pay.php |
| myform | form data | | data & submit | |
-------------- ------------- -------------
| ^ |
| | |
(4) redirect | ---------------------------------
| (3) success or failure info
v
______________
| myecom.com |
| receiver.php |
--------------
Explanation: Your internal processor sends the user data plus the additional data to the external processor. Since it's your server that makes the POST request to the external processor, the user never sees the request and thus cannot see the data that is being sent.
I'm not a PHP programmer, but a quick Google query on "PHP post request" revealed these two links, see if they help you:
How do I send a POST request with PHP?
Three Ways to Make a POST Request from PHP