I just found out about a pretty weird behaviour of Scala scoping when bytecode generated from Scala code is used from Java code. Consider the following snippet using Spark (Spark 1.4, Hadoop 2.6):
import java.util.Arrays;
import java.util.List;
import org.apache.spark.SparkConf;
import org.apache.spark.api.java.JavaSparkContext;
import org.apache.spark.broadcast.Broadcast;
public class Test {
public static void main(String[] args) {
JavaSparkContext sc =
new JavaSparkContext(new SparkConf()
.setMaster("local[*]")
.setAppName("test"));
Broadcast<List<Integer>> broadcast = sc.broadcast(Arrays.asList(1, 2, 3));
broadcast.destroy(true);
// fails with java.io.IOException: org.apache.spark.SparkException:
// Attempted to use Broadcast(0) after it was destroyed
sc.parallelize(Arrays.asList("task1", "task2"), 2)
.foreach(x -> System.out.println(broadcast.getValue()));
}
}
This code fails, which is expected as I voluntarily destroy a Broadcast before using it, but the thing is that in my mental model it should not even compile, let alone running fine.
Indeed, Broadcast.destroy(Boolean) is declared as private[spark] so it should not be visible from my code. I'll try looking at the bytecode of Broadcast but it's not my specialty, that's why I prefer posting this question. Also, sorry I was too lazy to create an example that does not depend on Spark, but at least you get the idea. Note that I can use various package-private methods of Spark, it's not just about Broadcast.
Any idea of what's going on ?
If we reconstruct this issue with a simpler example:
package yuvie
class X {
private[yuvie] def destory(d: Boolean) = true
}
And decompile this in Java:
[yuvali#localhost yuvie]$ javap -p X.class
Compiled from "X.scala"
public class yuvie.X {
public boolean destory(boolean);
public yuvie.X();
}
We see that private[package] in Scala becomes public in Java. Why? This comes from the fact that Java private package isn't equivalent to Scala private package. There is a nice explanation in this post:
The important distinction is that 'private [mypackage]' in Scala is
not Java package-private, however much it looks like it. Scala
packages are truly hierarchical, and 'private [mypackage]' grants
access to classes and objects up to "mypackage" (including all the
hierarchical packages that may be between). (I don't have the Scala
spec reference for this and my understating here may be hazy, I'm
using [4] as a reference.) Java's packages are not hierarchical, and
package-private grants access only to classes in that package, as well
as subclasses of the original class, something that Scala's 'private
[mypackage]' does not allow.
So, 'package [mypackage]' is both more and less restrictive that Java
package-private. For both reasons, JVM package-private can't be used
to implement it, and the only option that allows the uses that Scala
exposes in the compiler is 'public.'
Related
I have trouble understanding the "code smell" reported by Sonarqube in my scala class. The message is something like:
com.foo.bar.MyObject$.MODULE$ isn't final and can't be protected from malicious code
And the first line in my code package... is tagged as having the issue (see below). I don't understand if Sonarqube wants Object to be marked final (which doesn't make sense in Scala) or if it's something to do with the imports. Can anyone help me understand?
package com.foo.bar
import com.foo.baz
Object MyObject {
val myVal = ...
def method = ...
}
This is because of how objects are turned to bytecode. If you want to get rid of this message, consider doing this:
package com.foo.bar
import com.foo.baz
final object MyObject {
val myVal = ...
def method = ...
}
this will do effectively nothing from perspective of scala code bit your linter will shut up, you can check your bytecode which will now reflect that variable
com.foo.bar.MyObject$.MODULE$
Is now final. This is because object instance turned into static _.MODULE$ variable inside class MyObject.
OK so this is some kind of theoretical question for you guys.
I am experimenting with cglib's Enchancer - creating a proxy for a class.
My code is running in a Felix OSGi container.
The hierarchy looks kind of similar to that:
// Bundle A;
// Imports-Package: javax.xml.datatype
// Exports-Package: a.foo
package a.foo;
public class Parent {
protected javax.xml.datatype.XMLGregorianCalendar foo;
... -> getter/setter;
}
// Bundle B
// Imports-Package: a.foo
// DOES NOT IMPORT PACKAGE javax.xml.datatype !!!
package b.bar;
import a.foo.Parent;
public class Child extends Parent {
protected String bar;
... -> getter/setter;
}
// Bundle B
// Code extracted from https://github.com/modelmapper/modelmapper/blob/master/core/src/main/java/org/modelmapper/internal/ProxyFactory.java#L59
public Child enchance() {
Enhancer enhancer = new Enhancer();
enhancer.setSuperclass(Child.class);
enhancer.setUseFactory(true);
enhancer.setUseCache(true);
enhancer.setNamingPolicy(NAMING_POLICY);
enhancer.setCallbackFilter(METHOD_FILTER);
enhancer.setCallbackTypes(new Class[] { MethodInterceptor.class, NoOp.class });
try {
return enhancer.createClass();
} catch (Throwable t) {
t.printStackTrace();
}
}
From OSGi point of view - the two bundles - Bundle A and Bundle B are fully functional.
The package imports/exports are bnd generated. Although BundleA does not import explicitly the javax.xml.datatype package - I can create instances of Child without any problem.
So far so good.
But when I try to call the enchance() method and create a Child proxy - cglib throws a NoClassDefFoundError: javax.xml.datatype.XMLGregorianCalendar
OK, I get this - BundleB's classloader indeed cannot load this class and in fact - cglib's Enchancer seems to be using BundleB's classloader (Child's class type classloader) in order to create the proxy.
On the other hand - for handling modularity the OSGi container is doing the so called classloading delegation - instead of BundleB's classloader, the OSGi runtime delegates the loading of the parent class Parent to BundleA's classloader, which knows how to load all of its fields.
This is why BundleB does not need to explicitly import the javax.xml.datatype package and does not need to know how to load the XMLGregorianCalendar class and still be able to work with Child objects.
I was wondering - isn't such "delegating" approach suitable in the cglib's use case as well?
Please note that I don't know ANYTHING about byte code manipulation and that might sound like a very stupid question to some.
But I really don't understand - why isn't cglib able to delegate loading of the Parent to Parent's own classloader?
Is such mechanism really not available in cglib? Why? Is cglib not used in combination with OSGi? If so then why?
The Child class does not need to import javax.xml.datatype so long as it does not access the javax.xml.datatype.XMLGregorianCalendar field and you are just using the Child class in the normal way. However in order to generate a proxy class, CGLib will need to have visibility of the internals of the full inheritance hierarchy including the javax.xml.datatype.XMLGregorianCalendar in order to generate the bytecode for the new type. Therefore an import of the package will be required.
Unfortunately bnd cannot predict that you will be doing bytecode generation on the Child class so it does not add the import of javax.xml.datatype – it only add the imports required for normal usage.
In general it is a bad idea to inherit from a class imported from another bundle. Java inheritance creates a very tight coupling from the subclass to the superclass, which means you are exposed to the internals of the superclass.
To your last question: CGLib is fairly widely used in OSGi for things like mocking objects during testing. It is less used in production because there is nearly always a better solution than bytecode generation, such as proper usage of the service registry.
I tried combining the OSGi Class Loader Bridge idea that is described here:
https://www.infoq.com/articles/code-generation-with-osgi
... that solves a similar problem with code generation frameworks running within OSGi, with another idea that came to me recently.
The idea is to keep track of class loaders of class types that are found in the parent type hierarchy of the user's type. We can later use these class loaders as fallback for loading types that are otherwise unknown to the Bundle's class loader of the user's type.
We can then tell CGLIB's Enhancer to use this new class loader for resolving.
The idea is presented here:
https://github.com/modelmapper/modelmapper/pull/294
I would love to hear the opinion of experienced OSGi specialists about this though.
But so far this seems to work.
Until proven wrong, I accept my own answer.
I want to write some tests for my compiler but can't get past an error.
I'm following an example from 'Implementing DSL with Xtext and Xtend' by L. Bettini (great book btw). I've downloaded the code for the 'entities' DSL from https://github.com/LorenzoBettini/packtpub-xtext-book-examples and the tests in EntitiesGenerator.xtend work great.
If I write a test for the default DSL (MyDsl) using the same code, I've got an error:
org.eclipse.xtext.xbase.compiler.CompilationTestHelper cannot be resolved to a type.
or, if I add org.eclipse.xtext.xbase.junit (2.4.1) to the list of required plug-ins, I get
Discouraged access: The type CompilationTestHelper is not accessible due to restriction on required project org.xtext.example.myDsl.tests
I can allow access to it, but then get a runtime error anyway. If I try to add org.eclipse.xtext.xbase.lib as well, only org.eclipse.xtext.xbase.lib.source appears in the list. I don't know it that matters. In any case, adding it doesn't change anything.
What do I need to do to make it work?
I'm using Juno with Xtext 2.4.1., Java 1.7.
The content of the test class:
package org.xtext.example.myDsl.tests
import com.google.inject.Inject
import org.eclipse.xtext.junit4.InjectWith
import org.eclipse.xtext.junit4.XtextRunner
import org.eclipse.xtext.xbase.compiler.CompilationTestHelper // error here
import org.xtext.example.myDsl.MyDslInjectorProvider
import org.junit.Test
import org.junit.runner.RunWith
#RunWith(typeof(XtextRunner))
#InjectWith(typeof(MyDslInjectorProvider))
class MyDslGeneratorTest {
#Inject extension CompilationTestHelper
#Test
def void testGeneratedCode() {
'''
Hello some1!
Hello some2!
'''.assertCompilesTo(
'''some content''')
}
}
Thank you in advance!
the xtext guys mark stuff that may be changed NOT as api. this is why you get this warning.
it should work anyway. (although it is meant to be used for xbase languages only)
P.S: you have to add a dependency to jdt.core too
I am new in Java, searched for this question in google and stackoverflow, found some posts, but still I can't understand.
I want to use DLL libary (C++) methods from Java. I use JNA for this purpose. JNA found my library but it can't find my method:
Exception in thread "main" java.lang.UnsatisfiedLinkError: Error looking up function 'LoadCurrentData': The specified procedure could not be found.
My code:
package javaapplication1;
import com.sun.jna.Library;
import com.sun.jna.Native;
import com.sun.jna.Platform;
import com.sun.jna.Pointer;
public class JavaApplication1 {
public interface LibPro extends Library {
LibPro INSTANCE = (LibPro) Native.loadLibrary(
(Platform.isWindows() ? "LibPro" : "LibProLinuxPort"), LibPro.class);
public short LoadCurrentData();
}
public static void main(String[] args) {
LibPro sdll = LibPro.INSTANCE;
sdll.LoadCurrentData(); // call of void function
}
}
I looked in my DLL with Depency Walker Tool and saw that my function name has prefix and suffix - it looks like _LoadCurrentData#0
Thanks for response!
P.S. I found good example which works http://tutortutor.ca/cgi-bin/makepage.cgi?/articles/rjna (Listing 6).
I'd say that you need to apply correct name mapper, as you noticed function name got mangled, you need to register CallMapper that will implement the same mangling as your compiler.
Here is a revelant entry from JNA homepage:
Use a dump utility to examine the names of your exported functions to make sure they match (nm on linux, depends on Windows). On Windows, if the functions have a suffix of the form "#NN", you need to pass a StdCallFunctionMapper as an option when initializing your library interface. In general, you can use a function mapper (FunctionMapper) to change the name of the looked-up method, or an invocation mapper (InvocationMapper) for more extensive control over the method invocation.
Here is a possibly revelant question: renaming DLL functions in JNA using StdCallFunctionMapper
I have a Java library that I am working on with a directory structure which looks like the following:
/com
/example
/LibX
Server.java
Client.java
Now, from a project which is using the above classes, it seems to me that importing com.example.LibX.* and using Client client = new Client(...); is a bit ambiguous, as "Client" could mean anything. Therefore, I tried the following, only to receive "package not found" errors:
import com.example.*;
LibX.Client client = new LibX.Client(...);
It is possible to do what I described? Or is there another way to remove the ambiguity without using com.example.LibX.Client?
Java packages are not hierarchical, even if they may sometimes look like it. You can't import a "tree" of packages, as you're suggesting. You either need to import a specific package with a wildcard, or you import the specific class name, or you use fully-qualified class names in your code directly.
The following isn't ambiguous, and is considered best practice:
import com.example.LibX.Client;
...
Client client = new Client(...);
In a world where IDEs can organise your imports for you, there's no reason not to state them all explicitly.
Your concern about ambiguity is unnecessary - if you have an ambiguous reference your class won't compile -
e.g.
import java.util.Date;
import java.sql.Date;
public class Test {
private Date date;
}
won't compile. So if you can compile the class then by definition you don't have an ambiguous reference.
Incidentally LibX.Client is a bit confusing. Usually classnames are capitalized, package names lowercased, so if you did that (if LibX was a top-level package and you were giving the full name) it looks more like an inner class reference, as in Andy's response above.
It's possible if you're willing to group Client and Server as static nested classes.
public class LibX {
public static class Client {
//...
}
public static class Server {
//...
}
}
Now you can import mypak.LibX and then do new LibX.Client(). This has the unfortunate drawbacks of forcing you to group your classes and also creating the additional empty LibX class.
There's no such mechanism like what you described. Your only other possibility is to use single class imports:
import com.example.LibX.Client;