I currently have class which have some fields initialized in declaration, like this:
public class SomeClass implements Externalizable {
private long id;
private final List<Hit> hits = new ArrayList<>();
#Override
public void writeExternal(ObjectOutput out) throws IOException {
out.writeLong(id);
out.writeInt(hits.size());
for (int i = 0; i < hits.size(); i++) {
out.writeObject(hits.get(i));
}
}
#Override
public void readExternal(ObjectInput in) throws IOException, ClassNotFoundException {
id = in.readLong();
int size = in.readInt();
for (int i = 0; i < size; i++) {
hits.add((Hit) in.readObject()); //<--Nullpointer here, hits == null
}
}
}
And this class is used in filebased chronicle-map configured like this:
ChronicleMap<Long, SomeClass> storage = ChronicleMapBuilder
.of(Long.class, SomeClass.class)
.averageValueSize(avgEntrySize)
.entries(entries)
.createPersistedTo(new File(path));
The problem is that when I restart my application I get NullpointerException when chronicle tries to read saved map, because hits field wasn't initialized, meaning it is null.
I did some investigation and found that before calling readExternal chronical creates object of this class using UNSAFE.allocateInstance(in ExternalizableMarshaller):
protected E getInstance() throws Exception {
return (E) NativeBytes.UNSAFE.allocateInstance(classMarshaled);
}
So basically this is the reason why its not initialized. What I am trying to understand why it is using such approach instead of MethodHandle or reflection?
And maybe there is another way to fix this without modifying SomeClass, like some chronicle configuration property maybe?
This appears to be an issue with version 2.x which is no longer supported.
In version 3.x it should call the default constructor if one exists. It will use Unsafe if there is no default constructor. I have added a test case which shows this works in 3.x
https://github.com/OpenHFT/Chronicle-Map/blob/master/src/test/java/net/openhft/chronicle/map/externalizable/ExternalizableTest.java
For version 2.x I suggest you need to check whether the list is null and set it as needed.
Related
I'm essentially asking the same as this old question, but for Java 14 instead of Java 8. To spare answerers the trouble of navigating to the old question, I'll rephrase it here.
I want to get the name of a function from a referenced method. The following Java code should give you the idea:
public class Main
{
public static void main(String[] args)
{
printMethodName(Main::main);
}
private static void printMethodName(Consumer<String[]> theFunc)
{
String funcName = // somehow get name from theFunc
System.out.println(funcName)
}
}
The equivalent in C# would be:
public class Main
{
public static void Main()
{
var method = Main.Main;
PrintMethodName(method)
}
private static void PrintMethodName(Action action)
{
Console.WriteLine(action.GetMethodInfo().Name);
}
}
According to the accepted answer of the old question, this was not possible in Java 8 without considerable work, such as this solution. Is there a more elegant solution in Java 14?
Getting a method info from a method reference never was a goal on the JDK developer’s side, so no effort was made to change the situation.
However, the approach shown in your link can be simplified. Instead of serializing the information, patching the serialized data, and restoring the information using a replacement object, you can simply intercept the original SerializedLambda object while serializing.
E.g.
public class GetSerializedLambda extends ObjectOutputStream {
public static void main(String[] args) { // example case
var lambda = (Consumer<String[]>&Serializable)GetSerializedLambda::main;
SerializedLambda sl = GetSerializedLambda.get(lambda);
System.out.println(sl.getImplClass() + " " + sl.getImplMethodName());
}
private SerializedLambda info;
GetSerializedLambda() throws IOException {
super(OutputStream.nullOutputStream());
super.enableReplaceObject(true);
}
#Override protected Object replaceObject(Object obj) throws IOException {
if(obj instanceof SerializedLambda) {
info = (SerializedLambda)obj;
obj = null;
}
return obj;
}
public static SerializedLambda get(Object obj) {
try {
GetSerializedLambda getter = new GetSerializedLambda();
getter.writeObject(obj);
return getter.info;
} catch(IOException ex) {
throw new IllegalArgumentException("not a serializable lambda", ex);
}
}
}
which will print GetSerializedLambda main. The only newer feature used here, is the OutputStream.nullOutputStream() to drop the written information immediately. Prior to JDK 11, you could write into a ByteArrayOutputStream and drop the information after the operation which is only slightly less efficient. The example also using var, but this is irrelevant to the actual operation of getting the method information.
The limitations are the same as in JDK 8. It requires a serializable method reference. Further, there is no guaranty that the implementation will map to a method directly. E.g., if you change the example’s declaration to public static void main(String... args), it will print something like lambda$1 when being compiled with Eclipse. When also changing the next line to var lambda = (Consumer<String>&Serializable)GetSerializedLambda::main;, the code will always print a synthetic method name, as using a helper method is unavoidable. But in case of javac, the name is rather something like lambda$main$f23f6912$1 instead of Eclipse’s lambda$1.
In other words, you can expect encountering surprising implementation details. Do not write applications relying on the availability of such information.
I'm using flink to read data from kafka and convert it to protobuf. The problem I'm facing is when I run the java application I get the below error. If I modify the unknownFields variable name to something else, it works but it's hard to make this change on all protobuf classes.
I also tried to deserialize directly when reading from kafka but I'm not sure what should be the TypeInformation to be returned for getProducedType() method.
public static class ProtoDeserializer implements DeserializationSchema{
#Override
public TypeInformation getProducedType() {
// TODO Auto-generated method stub
return PrimitiveArrayTypeInfo.BYTE_PRIMITIVE_ARRAY_TYPE_INFO;
}
Appreciate all the help. Thanks.
java.lang.RuntimeException: The field protected com.google.protobuf.UnknownFieldSet com.google.protobuf.GeneratedMessage.unknownFields is already contained in the hierarchy of the class com.google.protobuf.GeneratedMessage.Please use unique field names through your classes hierarchy
at org.apache.flink.api.java.typeutils.TypeExtractor.getAllDeclaredFields(TypeExtractor.java:1594)
at org.apache.flink.api.java.typeutils.TypeExtractor.analyzePojo(TypeExtractor.java:1515)
at org.apache.flink.api.java.typeutils.TypeExtractor.privateGetForClass(TypeExtractor.java:1412)
at org.apache.flink.api.java.typeutils.TypeExtractor.privateGetForClass(TypeExtractor.java:1319)
at org.apache.flink.api.java.typeutils.TypeExtractor.createTypeInfoWithTypeHierarchy(TypeExtractor.java:609)
at org.apache.flink.api.java.typeutils.TypeExtractor.privateCreateTypeInfo(TypeExtractor.java:437)
at org.apache.flink.api.java.typeutils.TypeExtractor.getUnaryOperatorReturnType(TypeExtractor.java:306)
at org.apache.flink.api.java.typeutils.TypeExtractor.getFlatMapReturnTypes(TypeExtractor.java:133)
at org.apache.flink.streaming.api.datastream.DataStream.flatMap(DataStream.java:529)
Code:
FlinkKafkaConsumer09<byte[]> kafkaConsumer = new FlinkKafkaConsumer09<>("testArr",new ByteDes(),p);
DataStream<byte[]> input = env.addSource(kafkaConsumer);
DataStream<PBAddress> protoData = input.map(new RichMapFunction<byte[], PBAddress>() {
#Override
public PBAddress map(byte[] value) throws Exception {
PBAddress addr = PBAddress.parseFrom(value);
return addr;
}
});
Maybe you should try this follow:
env.getConfig().registerTypeWithKryoSerializer(PBAddress. class,ProtobufSerializer.class);
or
env.getConfig().registerTypeWithKryoSerializer(PBAddress. class,PBAddressSerializer.class);
public class PBAddressSerializer extends Serializer<Message> {
final private Map<Class,Method> hashMap = new HashMap<Class, Method>();
protected Method getParse(Class cls) throws NoSuchMethodException {
Method method = hashMap.get(cls);
if (method == null) {
method = cls.getMethod("parseFrom",new Class[]{byte[].class});
hashMap.put(cls,method);
}
return method;
}
#Override
public void write(Kryo kryo, Output output, Message message) {
byte[] ser = message.toByteArray();
output.writeInt(ser.length,true);
output.writeBytes(ser);
}
#Override
public Message read(Kryo kryo, Input input, Class<Message> pbClass) {
try {
int size = input.readInt(true);
byte[] barr = new byte[size];
input.read(barr);
return (Message) getParse(pbClass).invoke(null,barr);
} catch (Exception e) {
throw new RuntimeException("Could not create " + pbClass, e);
}
}
}
try this:
public class ProtoDeserializer implements DeserializationSchema<PBAddress> {
#Override
public TypeInformation<PBAddress> getProducedType() {
return TypeInformation.of(PBAddress.class);
}
https://issues.apache.org/jira/browse/FLINK-11333 is the JIRA ticket tracking the issue of implementing first-class support for Protobuf types with evolvable schema. You'll see that there was a pull request quite some time ago, which hasn't been merged. I believe the problem was that there is no support there for handling state migration in cases where Protobuf was previously being used by registering it with Kryo.
Meanwhile, the Stateful Functions project (statefun is a new API that runs on top of Flink) is based entirely on Protobuf, and it includes support for using Protobuf with Flink: https://github.com/apache/flink-statefun/tree/master/statefun-flink/statefun-flink-common/src/main/java/org/apache/flink/statefun/flink/common/protobuf. (The entry point to that package is ProtobufTypeInformation.java.) I suggest exploring this package (which includes nothing statefun specific); however, it doesn't concern itself with migrations from Kryo either.
I'm trying to map a function across a JavaRDD in spark, and I keep getting NotSerializableError on the map call.
public class SparkPrunedSet extends AbstractSparkSet {
private final ColumnPruner pruner;
public SparkPrunedSet(#JsonProperty("parent") SparkSet parent, #JsonProperty("pruner") ColumnPruner pruner) {
super(parent);
this.pruner = pruner;
}
public JavaRDD<Record> getRdd(SparkContext context) {
JavaRDD<Record> rdd = getParent().getRdd(context);
Function<Record, Record> mappingFunction = makeRecordTransformer(pruner);
//The line below throws the error
JavaRDD<Record> mappedRdd = rdd.map(mappingFunction);
return mappedRdd;
}
private Function<Record, Record> makeRecordTransformer() {
return new Function<Record, Record>() {
private static final long serialVersionUID = 1L;
#Override
public Record call(Record record) throws Exception {
// Obviously i'd like to do something more useful in here, but this is enough
// to throw the error
return record;
}
};
}
}
When it runs, I get:
java.io.NotSerializableException: com.package.SparkPrunedSet
Record is an interface that implements serializable, and MapRecord is an implementation of it. Similar code to this exists and works in the codebase, except it's using rdd.filter instead. I've read through most of the other stack overflow entries on this, and none of them seem to help. I thought it might have to do with troubles serializing SparkPrunedSet (although I don't understand why it would even need to do this), so I set all of the fields on it to transient, but that didn't help either. Does anyone have any ideas?
The Function you are creating for the transformation is, in fact, an (anonymous) inner class of SparkPrunedSet. Therefore every instance of that function has an implicit reference to the SparkPrunedSet object that created it.
Therefore, serialization of it will require serialization of SparkPrunedSet.
I'm trying to implement the Externalizable interface to store the data using the LWUIT-IO's storage. This worked great for simple objects that are composed of Strings, booleans and ints.
However, I have an object that is composed of these types, but also of a Vector of the above mentioned Externalizable object. This seem to mess up the process and I get nothing when I try to retrieve the object from storage.
I assumed it was like the Serializable interface and that the Externalizable objects inside the main object are automatically handled. I'm not sure if this is true, or why it's failing.
The object inside the object is:
public class Song implements Externalizable{
String name = "examplesongname";
public void externalize(DataOutputStream out) throws IOException {
out.writeUTF(name);
}
public void internalize(int version, DataInputStream in) throws IOException {
name = in.readUTF();
}
public String getObjectId() {
return "pat.objects.Song";
}
public int getVersion() {
return 1;
}
}
The containing object is as follows:
public class Playlist implements Externalizable{
String name = "exampleplaylistname";
Vector songs = new Vector();
public void externalize(DataOutputStream out) throws IOException {
out.writeUTF(name);
out.write(songs.size());
Enumeration allItems = songs.elements();
while(allItems.hasMoreElements()){
Externalizable nextItem = (Externalizable) allItems.nextElement();
nextItem.externalize(out);
}
}
public void internalize(int version, DataInputStream in) throws IOException {
name = in.readUTF();
int size = in.readInt();
songs= new Vector();
for(int currentIndex = 0; currentIndex < size; currentIndex++){
Object nextItem = new Object();
((Externalizable)nextItem).internalize(version, in);
songs.addElement(nextItem);
}
}
}
public String getObjectId() {
return "pat.objects.Playlist";
}
public int getVersion() {
return 1;
}
}
What am I doing wrong or missing that is making the Playlist (containing object) fail to be stored while if I try to store the first one by itself it works?
Please note that the overriding methods are different that normal Java since this is the LWUIT version of Externalizable interface.
You need to use Util.register(...) to register these classes as externalizable when your app starts up.
Also the call directly to externalize isn't correct. You should use Util.writeObject/readObject to write another externalizable object (with its own version number). You can then avoid the loop over the vector which would be redundant and just write the whole vector.
I would also suggest using Util.readUTF/writeUTF which support null strings as well.
As a sidenote, I'd suggest migrating to Codename one since LWUIT is no longer maintained by anyone. Also Steve Hannah has a nice writeup on externalization is Codename One which is pretty similar to LWUIT (although it now supports Lists and Maps): http://www.shannah.ca/blog/?p=234
Consider the following servlet code:
public class AddDevice extends JsonServlet {
private static final long serialVersionUID = 1L;
#Override
protected void doGet(final JsonServletRequest request,
final JsonServletResponse response) throws ServletException,
IOException {
try {
final DeviceEntity device = new DeviceEntity();
device.type =
FleetManagerDatabaseHelper
.deviceTypesAccessor()
.queryForId(Integer.valueOf(
request.getParameter(DeviceTypeEntity._ID)));
device.sn = request.getParameter(DeviceEntity._SN);
device.status = Long.valueOf(0);
FleetManagerDatabaseHelper.devicesAccessor().create(device);
}
catch (final SQLException e) {
throw new ServletException("device already exists");
}
}
}
This code depends on the DeviceEntity and on the FleetManagerDatabaseHelper classes.
Now, I would like to write a test for it checking that the created entity is filled with the correct type, sn and status values.
For this purpose I could create a FleetManagerDatabaseHelperMockup class.
How would you apply Google Guice (or something else) here with minimal changes?
Your first step is to design for dependency injection--avoid constructors and static methods, and instead take in instances that you require. It looks like those types are Provider<DeviceEntity>, DevicesAccessor, and DeviceTypesAccessor.
Provider is a very simple Guice interface that provides instances of whatever class is in its type argument via a single no-arg method get(). If you have bound Foo, Guice automatically knows how to bind Provider<Foo>. It is extremely useful if your instances are expensive, or if you need more than one over the lifetime of your servlet (as you do).
After refactoring for dependency injection, your class will look like this:
public class AddDevice extends JsonServlet {
private static final long serialVersionUID = 1L;
private final Provider<DeviceEntity> deviceEntityProvider;
private final DevicesAccessor devicesAccessor;
private final DeviceTypesAccessor deviceTypesAccessor;
#Inject
public AddDevice(Provider<DeviceEntity> deviceEntityProvider,
DevicesAccessor devicesAccessor,
DeviceTypesAccessor deviceTypesAccessor>) {
this.deviceEntityProvider = deviceEntityProvider;
this.devicesAccessor = devicesAccessor;
this.deviceTypesAccessor = deviceTypesAccessor;
}
#Override
protected void doGet(final JsonServletRequest request,
final JsonServletResponse response) throws ServletException,
IOException {
try {
final DeviceEntity device = deviceEntityProvider.get();
device.type = deviceTypesAccessor.queryForId(
Integer.valueOf(request.getParameter(DeviceTypeEntity._ID)));
device.sn = request.getParameter(DeviceEntity._SN)
device.status = Long.valueOf(0);
devicesAccessor.create(device);
} catch (final SQLException e) {
throw new ServletException("device already exists");
}
}
}
At this point, it's extremely easy to write a test by passing in a Provider that keeps track of the instance it returns, along with a mock DevicesAccessor and a mock DeviceTypesAccessor. (I recommend Mockito.) If you write your own Provider interface and remove the #Inject, you don't even need to use Guice; in your tests, you could continue to use that constructor, but you would want to satisfy Java EE with a constructor like:
public AddDevice() {
this(new NewDeviceEntityProvider(),
FleetManagerDatabaseHelper.deviceTypesAccessor(),
FleetManagerDatabaseHelper.devicesAccessor());
}
private class NewDeviceEntityProvider implements Provider<DeviceEntity> {
#Override public DeviceEntity get() {
return new DeviceEntity();
}
}
But if you do want to use Guice to remove that boilerplate, just write a Guice Module. Your module would need to bind DeviceTypesAccessor and DevicesAccessor to the instances that FleetManagerDatabaseHelper would return; Guice would see that DeviceEntity has a no-arg constructor and would be able to inject DeviceEntity and Provider<DeviceEntity> automatically. (Comment if you want me to expand on what the Module would look like.)
Hope this helps!