We Keep Coding

How to define logging in one place in Python?

I am new Python and I want to define the following logging code in python in one place so that it can be used in other python files(class or non-class). In case of Java, we use log4j or slf4j as part of jar files and we simply define as Logger logger = Logger.getLogger("class name"), I want achieve exactly like this so that I can use loger in any python module/s. Currently I define the config in all the classes.
import logging
logging.basicConfig(
level=logging.INFO,
format="%(asctime)s [%(levelname)s] %(message)s",
handlers=[
logging.FileHandler("debug.log"),
logging.StreamHandler()
]
)
I have also gone through this SO link Using logging in multiple modules
Also please suggest the most recommended approach for medium to large project having more number of python files.

As per this link https://fangpenlin.com/posts/2012/08/26/good-logging-practice-in-python/, the author has mentioned a better practice. There may be better recommended approach.
You have to do in the following manner.
Define logging.json file as mentioned in the link.
Define a python file called LogSetup.py and add below the code.
def setup_logging(default_path='logging.json', default_level=logging.DEBUG, env_key='LOG_CFG'):
"""Setup logging configuration"""
path = default_path
value = os.getenv(env_key, None)
if value:
path = value
if os.path.exists(path):
with open(path, 'rt') as f:
config = json.load(f)
logging.config.dictConfig(config)
else:
logging.basicConfig(level=default_level)
Call the above file like this LogSetup.setup_logging() in the main method.
I provide below the code snippet.
logger = logging.getLogger(name)
if __name__ == '__main__':
LogSetup.setup_logging()
logger.debug("This is a debug message ...")
It looks good for me, others may recommend the better approaches.

One option to make your logging handler available everywhere, is to make it a built-in identifier:
a.py:
import logging
import builtins
logging.basicConfig(
level=logging.INFO,
format="%(asctime)s %(filename)s [%(levelname)s] %(message)s",
handlers=[
logging.FileHandler("/tmp/debug.log"),
logging.StreamHandler()
]
)
builtins.logger = logging.getLogger(__name__)
logger.info("module a!")
import b
b.py:
logger.info(msg='This is from module b.py')
def foo():
logger.info('Log message from b.foo()')
foo()
Output:
2020-02-14 20:18:17,549 a.py [INFO] module a!
2020-02-14 20:18:17,550 b.py [INFO] This is from module b.py
2020-02-14 20:18:17,550 b.py [INFO] Log message from b.foo()

How to load a custom transformer in Spark 2.4

I'm trying to create a custom transformer in Spark 2.4.0. Saving it works fine. However, when I try to load it, I get the following error:
java.lang.NoSuchMethodException: TestTransformer.<init>(java.lang.String)
at java.lang.Class.getConstructor0(Class.java:3082)
at java.lang.Class.getConstructor(Class.java:1825)
at org.apache.spark.ml.util.DefaultParamsReader.load(ReadWrite.scala:496)
at org.apache.spark.ml.util.MLReadable$class.load(ReadWrite.scala:380)
at TestTransformer$.load(<console>:40)
... 31 elided
This suggests to me that it can't find my transformer's constructor, which doesn't really make sense to me.
MCVE:
import org.apache.spark.sql.{Dataset, DataFrame}
import org.apache.spark.sql.types.{StructType}
import org.apache.spark.ml.Transformer
import org.apache.spark.ml.param.ParamMap
import org.apache.spark.ml.util.{DefaultParamsReadable, DefaultParamsWritable, Identifiable}
class TestTransformer(override val uid: String) extends Transformer with DefaultParamsWritable{
def this() = this(Identifiable.randomUID("TestTransformer"))
override def transform(df: Dataset[_]): DataFrame = {
val columns = df.columns
df.select(columns.head, columns.tail: _*)
}
override def transformSchema(schema: StructType): StructType = {
schema
}
override def copy(extra: ParamMap): TestTransformer = defaultCopy[TestTransformer](extra)
}
object TestTransformer extends DefaultParamsReadable[TestTransformer]{
override def load(path: String): TestTransformer = super.load(path)
}
val transformer = new TestTransformer("test")
transformer.write.overwrite().save("test_transformer")
TestTransformer.load("test_transformer")
Running this (I'm using a Jupyter notebook) leads to the above error. I've tried compiling and running it as a .jar file, with no difference.
What puzzles me is that the equivalent PySpark code works fine:
from pyspark.sql import SparkSession, DataFrame
from pyspark.ml import Transformer
from pyspark.ml.util import DefaultParamsReadable, DefaultParamsWritable
class TestTransformer(Transformer, DefaultParamsWritable, DefaultParamsReadable):
def transform(self, df: DataFrame) -> DataFrame:
return df
TestTransformer().save('test_transformer')
TestTransformer.load('test_transformer')
How can I make a custom Spark transformer that can be saved and loaded?

I can reproduce your problem in spark-shell.
Trying to find the source of the problem I looked into DefaultParamsReadable and DefaultParamsReader sources and I could see they utilize Java reflection.
https://github.com/apache/spark/blob/v2.4.0/mllib/src/main/scala/org/apache/spark/ml/util/ReadWrite.scala
lines 495-496
val instance =
cls.getConstructor(classOf[String]).newInstance(metadata.uid).asInstanceOf[Params]
I think scala REPLs and Java reflection aren't good friends.
If you run this snippet (after yours):
new TestTransformer().getClass.getConstructors
you'll get the following output:
res1: Array[java.lang.reflect.Constructor[_]] = Array(public TestTransformer($iw), public TestTransformer($iw,java.lang.String))
It is true! TestTransformer.<init>(java.lang.String) doesn't exist.
I found 2 workarounds,
Compiling your code with sbt and creating a jar, then including in spark-shell with :require, worked for me (You mentioned you tried a jar, I don't know how though)
Pasting the code in spark-shell with :paste -raw , worked fine as well. I suppose -raw prevents from REPL doing shenanigans to your classes.
See: https://docs.scala-lang.org/overviews/repl/overview.html
I'm not sure how you can adapt any of these to Jupyter but I hope this info is useful for you.
NOTE: I actually used spark-shell in spark 2.4.1

Java ClassLoader Not Finding Expected Resource

I have a jar file that contains the following:
LibJar Contents
dir1
|dir1-1
| |Class1-1-1
| |LClass1-1-2
|Ldir1-2
|LClass1-2-1
Ldir2
|LClass2-1
My java program (we can call it ProgJar, but I also run it in Netbeans IDE) has the following package structure:
ProgJar
dir1
|dir1-1
| |Class-1-1
| |PClass1-1-2 Different file name from LibJar
Pdir2
|PClass2-1
The only shared package structure between ProgJar and LibJar is "dir1/dir1-1/Class1-1-1". Everything else prefixed with a P is unique to ProgJar and everything prefixed with a L is unique to LibJar.
I use LibJar as a library in ProgJar.
This is the snippet of code I run in ProjJar:
ClassLoader clP = Pdir2.PClass2-1.class.getClassLoader();
ClassLoader clL = Ldir2.LClass2-1.class.getClassLoader();
URL u1 = clP.getResource("dir1/dir1-1");
URL u2 = clL.getResource("dir1/dir1-1");
System.out.printf(u1.toExternalForm());
System.out.printf(u2.toExternalForm());
When I run this in Netbeans I get the following output:
Netbeans Output:
jar:file:/C:/path/to/project/lib/LibJar.jar!/dir1/dir1-1
jar:file:/C:/path/to/project/lib/LibJar.jar!/dir1/dir1-1
When I run as a ProgJar as a built jar outside of netbeans, I get:
Jar Output:
jar:file:/C:/path/to/ProgJar/ProgJar.jar!/dir1/dir1-1
jar:file:/C:/path/to/ProgJar/ProgJar.jar!/dir1/dir1-1
What I expect to see is the following:
Netbeans Output:
jar:file:/C:/path/to/project/build/classes/dir1/dir1-1
jar:file:/C:/path/to/project/lib/LibJar.jar!/dir1/dir1-1
Jar Output:
jar:file:/C:/path/to/ProgJar/ProgJar.jar!/dir1/dir1-1
jar:file:/C:/path/to/ProgJar/libs/LibJar.jar!/dir1/dir1-1
I read through a few different articles, but this one seems somewhat relevant to this particular issue:
http://jeewanthad.blogspot.com/2014/02/how-to-solve-java-classpath-hell-with.html
How am I able to achieve my specified output?

Below code is not doing what you are expecting it to do:
ClassLoader clP = Pdir2.PClass2-1.class.getClassLoader();
ClassLoader clL = Ldir2.LClass2-1.class.getClassLoader();
Here clP amd clL are same Classloader instances(you system/application classloader to be specific).To verify, just see (clP == clL) should return true.
What you want to do is, use a custom classloader(URLClassLoader should do) to load your library. Then, the system classloader that loaded your ProgJar and your custom classloader will be different. Then rest of your code should work as expected.

How to save a Java object in Jython/Python

I'm building a Python UI using Tkinter. For the needs of the program, I've to connect Python with Java to do some stuff, so I'm using a simple Jython script as a linker. I cant use Tkinter with Jython because it's not supported.
Python (ui.py) -> Jython (linker.py) -> Java (compiled in jars)
To call the Jython function in Python I use subprocess as follows:
ui.py:
cmd = 'jython linker.py"'
my_env = os.environ
my_env["JYTHONPATH"] = tons_of_jars
subprocess.Popen(cmd, shell=True, env=my_env)
Then, in the Jython file, linker.py, I import the Java classes already added on the JYTHONPATH, and I create an object with the name m and call to some functions of the Java class.
linker.py:
import handler.handler
m = handler.handler(foo, moo, bar)
m.schedule(moo)
m.getFullCalendar()
m.printgantt()
The thing is that I've created a m object, that will be destroyed after the execution of jython linker.py ends.
So the question is: Is possible to save that m object somewhere so I can call it from ui.py whenever I want? If it's not possible, is there any other way to do this?
Thanks in advance.

I finally solved it by using ObjectOutputStream.
from java import io
def saveObject(x, fname="object.bin"):
outs = io.ObjectOutputStream(io.FileOutputStream(fname))
outs.writeObject(x)
outs.close()
def loadObject(fname="object.bin"):
ins = io.ObjectInputStream(io.FileInputStream(fname))
x=ins.readObject()
ins.close()
return x

Why does Jython not find this module? [duplicate]

This question already has answers here:
Closed 12 years ago.
Possible Duplicate:
How to run a python script from java?
I am running a Python script using Jython and got this error:
Exception in thread "main" Traceback (innermost last): File "C:\Facebook\LoginPython\Facebook.py", line 5, in ? ImportError: no module named cookielib
Why doesn't this work?

A little bit more about using Jython - had my share of problems with that as well. Note that this may not be the best way to do it, but it works fine for me.
I assume you want to call a function foo in module bar from your Java code that takes a string argument and returns a string:
PythonInterpreter interpreter = new PythonInterpreter();
// Append directory containing module to python search path and import it
interpreter.exec("import sys\n" + "sys.path.append(pathToModule)\n" +
"from bar import foo");
PyObject meth = interpreter.get("foo");
PyObject result = meth.__call__(new PyString("Test!"));
String real_result = (String) result.__tojava__(String.class);
The sys.path.append() part is only necessary if your module isn't part of the Python search path by default, which may very well be the problem if you get Import or Module not find errors.
Also you need to cast the objects between the java and python versions, you'll need to look that up if necessary, so far I only needed primitive types that were easy to cast, not sure if it's as easy for arbitrary java objects.

Use Jython to run Python on the JVM. Use PyDev to develop with Python (or Jython) on Eclipse.