When creating a web service server using Netbeans, Maven, Metro and Tomcat, how can I use relative filepaths in the wsit configuration?
For example, I have this line inside the wsit file:
<sc:KeyStore wspp:visibility="private" location="SERVER_KeyStore.jks" type="JKS" storepass="*****" alias="*****"/>
where should I put the jks file so it matches that location?
Finally, I found the answer.
when providing the
keystore/trustore name and location in
the wsit-*.xml files, please note that
they'll be loaded as resources
scanning the META-INF directory in
your package (WEB-INF/classes/META-INF
when using war packages on JBoss
Application Server 5).
from JBossWS - Stack Metro User Guide
In my case that means adding a META-INF folder to my resources folder and add <include>**/*.jks</include> to the pom file.
I saw some number of questions for wsit security configuration, most of them deal with externalizing SSL configuration, rather than hardcoding into wsdl file. Just because there may be development and production environment, and all in all hardcoded configuration is bad anyway. I spend several days with this issue and found only some (often monstrous) hints here in stackoverflow and various other forums. But the solution turned to be not so complex indeed. I just leave it here for someone (it matches also original question, because it will allow having jks anywhere, also having external config file as well).
Say, you have wsit policy in your wsdl files like this:
<wsp1:Policy wsu:Id="MyBinding_IWebServicePolicy">
<wsp1:ExactlyOne>
<wsp1:All>
<sc:KeyStore wspp:visibility="private" type="JKS" storepass="pass" alias="some-alias" keypass="pass" location="keystore.jks"/>
<sc:TrustStore wspp:visibility="private" type="JKS" peeralias="other-alias" storepass="pass" location="truststore.jks"/>
</wsp1:All>
</wsp1:ExactlyOne>
</wsp1:Policy>
You need to use CallbackHandler instead.
Adjusted policy:
<wsp1:Policy wsu:Id="MyBinding_IWebServicePolicy">
<wsp1:ExactlyOne>
<wsp1:All>
<sc:KeyStore wspp:visibility="private" callbackHandler="com.my.KeyStoreHandler"/>
<sc:TrustStore wspp:visibility="private" callbackHandler="com.my.TrustStoreHandler"/>
</wsp1:All>
</wsp1:ExactlyOne>
</wsp1:Policy>
And handler might look like this (I use scala, but you may translate this to java easily):
import javax.security.auth.callback.{ CallbackHandler => ICallbackHandler, Callback }
import com.sun.xml.wss.impl.callback.{ KeyStoreCallback, PrivateKeyCallback }
import java.security.{ PrivateKey, KeyStore }
import java.io.FileInputStream
abstract class CallbackHandler extends ICallbackHandler {
def conf: Config // getting external configuration
def handle(callbacks: Array[Callback]): Unit = callbacks foreach {
// loads the keystore
case cb: KeyStoreCallback =>
val ks = KeyStore.getInstance(conf.getString("type"))
val is = new FileInputStream(conf.getString("file"))
try ks.load(is, conf.getString("store-password").toCharArray) finally is.close()
cb.setKeystore(ks)
// loads private key
case cb: PrivateKeyCallback =>
cb.setAlias(conf.getString("alias"))
cb.setKey(cb.getKeystore.getKey(conf.getString("alias"), conf.getString("key-password").toCharArray).asInstanceOf[PrivateKey])
// other things
case cb => // I didn't need anything else, but just in case
}
}
class TrustStoreHandler extends CallbackHandler {
lazy val conf = getMyTrustStoreConfig
}
class KeyStoreHandler extends CallbackHandler {
lazy val conf = getMyKeyStoreConfig
}
In java just use if (cb isinstanceof Class) instead of case cb: Class =>, the other code is practically java without semicolons.
Related
I am new Python and I want to define the following logging code in python in one place so that it can be used in other python files(class or non-class). In case of Java, we use log4j or slf4j as part of jar files and we simply define as Logger logger = Logger.getLogger("class name"), I want achieve exactly like this so that I can use loger in any python module/s. Currently I define the config in all the classes.
import logging
logging.basicConfig(
level=logging.INFO,
format="%(asctime)s [%(levelname)s] %(message)s",
handlers=[
logging.FileHandler("debug.log"),
logging.StreamHandler()
]
)
I have also gone through this SO link Using logging in multiple modules
Also please suggest the most recommended approach for medium to large project having more number of python files.
As per this link https://fangpenlin.com/posts/2012/08/26/good-logging-practice-in-python/, the author has mentioned a better practice. There may be better recommended approach.
You have to do in the following manner.
Define logging.json file as mentioned in the link.
Define a python file called LogSetup.py and add below the code.
def setup_logging(default_path='logging.json', default_level=logging.DEBUG, env_key='LOG_CFG'):
"""Setup logging configuration"""
path = default_path
value = os.getenv(env_key, None)
if value:
path = value
if os.path.exists(path):
with open(path, 'rt') as f:
config = json.load(f)
logging.config.dictConfig(config)
else:
logging.basicConfig(level=default_level)
Call the above file like this LogSetup.setup_logging() in the main method.
I provide below the code snippet.
logger = logging.getLogger(name)
if __name__ == '__main__':
LogSetup.setup_logging()
logger.debug("This is a debug message ...")
It looks good for me, others may recommend the better approaches.
One option to make your logging handler available everywhere, is to make it a built-in identifier:
a.py:
import logging
import builtins
logging.basicConfig(
level=logging.INFO,
format="%(asctime)s %(filename)s [%(levelname)s] %(message)s",
handlers=[
logging.FileHandler("/tmp/debug.log"),
logging.StreamHandler()
]
)
builtins.logger = logging.getLogger(__name__)
logger.info("module a!")
import b
b.py:
logger.info(msg='This is from module b.py')
def foo():
logger.info('Log message from b.foo()')
foo()
Output:
2020-02-14 20:18:17,549 a.py [INFO] module a!
2020-02-14 20:18:17,550 b.py [INFO] This is from module b.py
2020-02-14 20:18:17,550 b.py [INFO] Log message from b.foo()
I'm writing an intelij plugin and would like to download the platform specific artefact at runtime.
I've loaded the platform specific jar into a class loader but the ChromiumExtractor cannot access the nested resources when prefixed with "/". So I can access the resource as "chromium-mac.zip" but the library cannot.
I've tried to unzip the nested zipped chromium artefact into the correct directory but this does not leading to a working solution. So now I've been trying to piece together the way the library extracts the artefact but it's rather tedious as the code is obfuscated.
Does the jxbrowser plugin have some support for retrieving the artefact at runtime. Could such support be added (jxbtrowser devs use SO for support questions etc, this is a message to them :D ) ?
Approach taken :
// inside intelij plugin . The plugin has the jxbrowser-6.6.jar
// and license.jar loaded into the classloader. the platform specific
// artefact will be retrieved manual).
val cl = URLClassLoader(arrayOf(URL("file://.../jxbrowser-mac-6.6.jar")), Browser::class.java.classLoader)
val backup = Thread.currentThread().contextClassLoader
try {
Thread.currentThread().contextClassLoader = cl
// can access like this
Thread.currentThread().contextClassLoader.getResource("chromium-mac.zip")
val ce = ChromiumExtractor.create()
// cannot access as resource is retrieved "/chromium-mac.zip" ?
ce.extract(BrowserPreferences.getChromiumDir())
browser = Browser()
} finally {
Thread.currentThread().contextClassLoader = backup
}
The following does the trick, The resource jar had to be in the same class loader as the client jar (as well as the license). It would be nice if JxBrowser added a helper for this that is capable of performing the download and initialising chromium, perhaps taking just a path for a persistent storage directory.
private fun initializeJxBrowser(): Browser {
if(ChromiumExtractor.create().shouldExtract(BrowserPreferences.getChromiumDir())) {
val cl = URLClassLoader(arrayOf(
URL("file:.../license.jar"),
URL("file:.../jxbrowser-mac-6.6.jar"),
URL("file:../jxbrowser-6.6.jar")
))
cl.loadClass("com.teamdev.jxbrowser.chromium.BrowserContext")
.getMethod("defaultContext")
.invoke(null)
}
return Browser()
}
I'm trying to write a UDF for Hadoop Hive, that parses User Agents. Following code works fine on my local machine, but on Hadoop I'm getting:
org.apache.hadoop.hive.ql.metadata.HiveException: Unable to execute method public java.lang.String MyUDF .evaluate(java.lang.String) throws org.apache.hadoop.hive.ql.metadata.HiveException on object MyUDF#64ca8bfb of class MyUDF with arguments {All Occupations:java.lang.String} of size 1',
Code:
import java.io.IOException;
import org.apache.hadoop.hive.ql.exec.UDF;
import org.apache.hadoop.hive.ql.metadata.HiveException;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.*;
import com.decibel.uasparser.OnlineUpdater;
import com.decibel.uasparser.UASparser;
import com.decibel.uasparser.UserAgentInfo;
public class MyUDF extends UDF {
public String evaluate(String i) {
UASparser parser = null;
parser = new UASparser();
String key = "";
OnlineUpdater update = new OnlineUpdater(parser, key);
UserAgentInfo info = null;
info = parser.parse(i);
return info.getDeviceType();
}
}
Facts that come to my mind I should mention:
I'm compiling with Eclipse with "export runnable jar file" and extract required libraries into generated jar option
I'm uploading this "fat jar" file with Hue
Minimum working example I managed to run:
public String evaluate(String i) {
return "hello" + i.toString()";
}
I guess the problem lies somewhere around that library (downloaded from https://udger.com) I'm using, but I have no idea where.
Any suggestions?
Thanks, Michal
It could be a few things. Best thing is to check the logs, but here's a list of a few quick things you can check in a minute.
jar does not contain all dependencies. I am not sure how eclipse builds a runnable jar, but it may not include all dependencies. You can do
jar tf your-udf-jar.jar
to see what was included. You should see stuff from com.decibel.uasparser. If not, you have to build the jar with the appropriate dependencies (usually you do that using maven).
Different version of the JVM. If you compile with jdk8 and the cluster runs jdk7, it would also fail
Hive version. Sometimes the Hive APIs change slightly, enough to be incompatible. Probably not the case here, but make sure to compile the UDF against the same version of hadoop and hive that you have in the cluster
You should always check if info is null after the call to parse()
looks like the library uses a key, meaning that actually gets data from an online service (udger.com), so it may not work without an actual key. Even more important, the library updates online, contacting the online service for each record. This means, looking at the code, that it will create one update thread per record. You should change the code to do that only once in the constructor like the following:
Here's how to change it:
public class MyUDF extends UDF {
UASparser parser = new UASparser();
public MyUDF() {
super()
String key = "PUT YOUR KEY HERE";
// update only once, when the UDF is instantiated
OnlineUpdater update = new OnlineUpdater(parser, key);
}
public String evaluate(String i) {
UserAgentInfo info = parser.parse(i);
if(info!=null) return info.getDeviceType();
// you want it to return null if it's unparseable
// otherwise one bad record will stop your processing
// with an exception
else return null;
}
}
But to know for sure, you have to look at the logs...yarn logs, but also you can look at the hive logs on the machine you're submitting the job on ( probably in /var/log/hive but it depends on your installation).
such a problem probably can be solved by steps:
overide the method UDF.getRequiredJars(), make it returning a hdfs file path list which values are determined by where you put the following xxx_lib folder into your hdfs. Note that , the list mist exactly contains each jar's full hdfs path strings ,such as hdfs://yourcluster/some_path/xxx_lib/some.jar
export your udf code by following "Runnable jar file exporting wizard" (chose "copy required libraries into a sub folder next to the generated jar". This steps will result in a xxx.jar and a lib folder xxx_lib next to xxx.jar
put xxx.jar and the folders xxx_lib to your hdfs filesystem according to your code in step 0.
create a udf using: add jar ${the-xxx.jar-hdfs-path}; create function your-function as $}qualified name of udf class};
Try it. I test this and it works
So I have a Java application that use to be packaged as a war and then deployed to Tomcat, but now I have it setup so it all runs straight from a jar file using embedded Jetty as follows:
class JettyServer extends ServerTrait {
val server = new Server()
val connector = new ServerConnector(server)
connector.setPort(BigSenseServer.config.options("httpPort").toInt)
server.setConnectors(Array(connector))
val context = new ServletContextHandler()
context.setContextPath(BigSenseServer.webRoot)
context.addServlet(new MasterServlet().getClass, "/*")
context.addEventListener(new InitLoggingListener())
context.addEventListener(new DBUpdateListener())
val fileContext = new WebAppContext()
fileContext.setContextPath(BigSenseServer.contentRoot)
fileContext.setResourceBase(BigSenseServer.getClass.getResource("/io/bigsense/web").toExternalForm)
val handlers = new HandlerCollection()
handlers.setHandlers(Array( fileContext, context, new DefaultHandler()))
server.setHandler(handlers)
override def startServer() {
server.start
server.join
}
override def stopServer() {
server.stop
}
}
The webroot is /bigsense/api and the contentRoot is /static. In this configuration, static files like CSS and Javascript are served from the /io/bigsense/web package (kept in src/main/resources in SBT). In my original, the context-root was /bigsense and the servlet was mapped to api/*, so all the static content could be served directly from /bigsense/{js,css,whatever}.
I couldn't figure out how to get Jetty to do the same thing, but the current setup listed above works fine and I adjusted all my templates to get that static path from the same config object (which has a backend in a property file).
I want to create an embedded Tomcat implementation as well and I've read several guides, but they all seem to want a real webapp base directory. I can't find any examples which either just map a servlet directory without a webapp base or take the webapp base from the classpath (in a jar) instead of a real physical directory. I've tried things similar to the following:
EDIT Got the servlet working with the following. Now I just need the ServletContextListneres and a way to server files from the jar:
class TomcatServer extends ServerTrait {
val tomcat = new Tomcat()
tomcat.setPort(BigSenseServer.config.options("httpPort").toInt)
val tmp = new File(System.getProperty("java.io.tmpdir"))
val ctx = tomcat.addContext(BigSenseServer.webRoot,tmp.getAbsolutePath)
Tomcat.addServlet(ctx,"bigsense",new MasterServlet())
ctx.addServletMapping("/*","bigsense")
override def startServer() = {
tomcat.start()
tomcat.getServer().await()
}
override def stopServer() = tomcat.stop
}
So I have a similar setup to Jetty for the main servlet. I can't find any functions on the context object for adding in the two ServletContextListner objects I have. I also need to be able to serve my static context from the jar on contentRoot (/static).
I researched and looked into the PlayN game framework and I liked it a lot. I program in Scala and actually don't know Java but it's not usually a problem since they work together great.
I've set up a basic project in eclipse and imported all the libraries and dependencies. I even translated over the base maven project code. Here's the two files:
Zeitgeist.scala
package iris.zeit.core
import playn.core.PlayN._
import playn.core.Game
import playn.core.Image
import playn.core.ImageLayer
class Zeitgeist extends Game {
override def init (){
var bgImage: Image = assets().getImage("images/bg.png")
var bgLayer: ImageLayer = graphics().createImageLayer(bgImage)
graphics().rootLayer().add(bgLayer)
}
override def paint (alpha: Float){
//painting stuffs
}
override def update(delta: Float){
}
override def updateRate(): Int = {
25
}
}
Main.scala
package iris.zeit.desktop
import playn.core.PlayN
import playn.java.JavaPlatform
import iris.zeit.core.Zeitgeist
object Main {
def main(args: Array[String]){
var platform: JavaPlatform = JavaPlatform.register()
platform.assets().setPathPrefix("resources")
PlayN.run(new Zeitgeist())
}
}
The cool thing is it works! A window comes up perfectly. The only problem is I can't seem to load images. With the above line, "assets().getImage("images/bg.png")" it pops out
Could not load image: resources/images/bg.png [error=java.io.FileNotFoundException: resources/images/bg.png]
I've played around with the location of my resources file to no avail. I was even able to find bg.png myself with java.io.File. Am I doing something wrong? Is there something I'm forgetting?
Looking at the code of JavaAssetsManager, it looks like it is trying to load a resource and not a file. So you should check that your images are actually in the classpath and at the path you give ("resources/images/bp.png")
Alternatively, you can use getRemoteImage and pass a File URL. As you succeeded in using a java.io.File, you can just get the URL with method toUri of File (toUrl is deprecated).
This almost certainly doesn't work because you're doing this:
platform.assets().setPathPrefix("resources")
That means you're saying your source folder looks like this:
src/main/resources/resources/images/bg.png
src/main/resources/resources/images/pea.png
src/main/resources/resources/images
I imagine it actually looks like one of these:
src/main/resources/assets/images/bg.png <-- 'assets' the default prefix
src/main/resources/assets/images/pea.png
src/main/resources/assets/images
or:
src/main/resources/images/bg.png <-- You have failed to put a subfolder prefix in
src/main/resources/images/pea.png
src/main/resources/images
You can either do this, if you have no prefix:
plat.assets().setPathPrefix("")
Or just put your files in the assets sub-folder inside the resources folder.
It's worth noting that the current implementation calls:
getClass().getClassLoader().getResource(...)
Not:
getClass().getResource(...)
The difference is covered elsewhere, but the tldr is that plat.assets.getImage("images/pea.png") will work, but plat.assets.getImage("/images/pea.png") will not.