Related
I'm trying to read from sqs queue in batch mode and write to a local file using Apache beam 2.34.0 and AWS beam SDK v1 which throws Illegal mutation exception.
public class SqsReader {
public void run(String[] args) {
SqsReaderOptions options = PipelineOptionsFactory.fromArgs(args).withValidation().
as(SqsReaderOptions.class);
Pipeline p = this.getPipeline(args);
p.apply(SqsIO.read().withQueueUrl(options.getSourceQueueUrl())
.withMaxNumRecords(options.getNumberOfRecords()))
.apply(ParDo.of(new SqsMessageToJson()))
.apply(TextIO.write()
.to(options.getLocalOutputLocation())
.withNumShards(options.getNumShards()));
p.run().waitUntilFinish();
}
public static void main(String[] args) throws IOException {
new SqsReader().run(args);
}
public static class SqsMessageToJson extends DoFn<Message, String> {
#ProcessElement
public void processElement(ProcessContext c) {
String message = Objects.requireNonNull(c.element()).getBody();
c.output(message);
}
}
}
I'm getting the following exception
Jan 10, 2022 11:37:05 AM org.apache.beam.sdk.util.MutationDetectors$CodedValueMutationDetector verifyUnmodifiedThrowingCheckedExceptions
WARNING: Coder of type class org.apache.beam.sdk.coders.SerializableCoder has a #structuralValue method which does not return true when the encoding of the elements is equal. Element Shard{source=org.apache.beam.sdk.io.aws.sqs.SqsUnboundedSource#5f19451c, maxNumRecords=1, maxReadTime=null}
Coder of type class org.apache.beam.sdk.coders.SerializableCoder has a #structuralValue method which does not return true when the encoding of the elements is equal. Element Shard{source=org.apache.beam.sdk.io.aws.sqs.SqsUnboundedSource#5f19451c, maxNumRecords=1, maxReadTime=null}
Exception in thread "main" org.apache.beam.sdk.util.IllegalMutationException: PTransform SqsIO.Read/Read(SqsUnboundedSource)/Read/ParMultiDo(Read) mutated value ValueWithRecordId{id=[98, 55, 50, 51, 56, 51, 102, 57, 45, 97, 52, 100, 56, 45, 52, 99, 100, 50, 45, 97, 49, 55, 49, 45, 48, 57, 100, 48, 100, 53, 50, 51, 99, 50, 54, 51], value={MessageId: b72383f9-a4d8-4cd2-a171-09d0d523c263,ReceiptHandle: AQEBj2FXnTVQ==,MD5OfBody: 38db8cbd101e4c1cfbf47e31c2aaab75,Body: {"test-key": "test-value"},Attributes: {SentTimestamp=1641794775474},MessageAttributes: {requestTimeMsSinceEpoch={StringValue: 1641794824800,StringListValues: [],BinaryListValues: [],}}}} after it was output (new value was ValueWithRecordId{id=[98, 55, 50, 51, 56, 51, 102, 57, 45, 97, 52, 100, 56, 45, 52, 99, 100, 50, 45, 97, 49, 55, 49, 45, 48, 57, 100, 48, 100, 53, 50, 51, 99, 50, 54, 51], value={MessageId: b72383f9-a4d8-4cd2-a171-09d0d523c263,ReceiptHandle: DeVRF8vQATm1f+rHIvR3eaejlRHksL1R7WE4zDT7lSwdIs9gJCYKXFXnTVQ==,MD5OfBody: 38db8cbd101e4c1cfbf47e31c2aaab75,Body: {"test-key": "test-value"},Attributes: {SentTimestamp=1641794775474},MessageAttributes: {requestTimeMsSinceEpoch={StringValue: 1641794824800,StringListValues: [],BinaryListValues: [],}}}}). Values must not be mutated in any way after being output.
at org.apache.beam.runners.direct.ImmutabilityCheckingBundleFactory$ImmutabilityEnforcingBundle.commit(ImmutabilityCheckingBundleFactory.java:137)
at org.apache.beam.runners.direct.EvaluationContext.commitBundles(EvaluationContext.java:231)
at org.apache.beam.runners.direct.EvaluationContext.handleResult(EvaluationContext.java:163)
at org.apache.beam.runners.direct.QuiescenceDriver$TimerIterableCompletionCallback.handleResult(QuiescenceDriver.java:292)
at org.apache.beam.runners.direct.DirectTransformExecutor.finishBundle(DirectTransformExecutor.java:194)
at org.apache.beam.runners.direct.DirectTransformExecutor.run(DirectTransformExecutor.java:131)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: org.apache.beam.sdk.util.IllegalMutationException: Value ValueWithRecordId{id=[98, 55, 50, 51, 56, 51, 102, 57, 45, 97, 52, 100, 56, 45, 52, 99, 100, 50, 45, 97, 49, 55, 49, 45, 48, 57, 100, 48, 100, 53, 50, 51, 99, 50, 54, 51], value={MessageId: b72383f9-a4d8-4cd2-a171-09d0d523c263,ReceiptHandle: AQEBj2KQ==,MD5OfBody: 38db8cbd101e4c1cfbf47e31c2aaab75,Body: {"test-key": "test-value"},Attributes: {SentTimestamp=1641794775474},MessageAttributes: {requestTimeMsSinceEpoch={StringValue: 1641794824800,StringListValues: [],BinaryListValues: [],}}}} mutated illegally, new value was ValueWithRecordId{id=[98, 55, 50, 51, 56, 51, 102, 57, 45, 97, 52, 100, 56, 45, 52, 99, 100, 50, 45, 97, 49, 55, 49, 45, 48, 57, 100, 48, 100, 53, 50, 51, 99, 50, 54, 51], value={MessageId: b72383f9-a4d8-4cd2-a171-09d0d523c263,ReceiptHandle: AQ==,MD5OfBody: 38db8cbd101e4c1cfbf47e31c2aaab75,Body: {"test-key": "test-value"},Attributes: {SentTimestamp=1641794775474},MessageAttributes: {requestTimeMsSinceEpoch={StringValue: 1641794824800,StringListValues: [],BinaryListValues: [],}}}}. Encoding was rO.
at org.apache.beam.sdk.util.MutationDetectors$CodedValueMutationDetector.illegalMutation(MutationDetectors.java:158)
at org.apache.beam.sdk.util.MutationDetectors$CodedValueMutationDetector.verifyUnmodifiedThrowingCheckedExceptions(MutationDetectors.java:153)
at org.apache.beam.sdk.util.MutationDetectors$CodedValueMutationDetector.verifyUnmodified(MutationDetectors.java:128)
at org.apache.beam.runners.direct.ImmutabilityCheckingBundleFactory$ImmutabilityEnforcingBundle.commit(ImmutabilityCheckingBundleFactory.java:127)
... 10 more
Caused by: org.apache.beam.sdk.util.IllegalMutationException: Value ValueWithRecordId{id=[98, 55, 50, 51, 56, 51, 102, 57, 45, 97, 52, 100, 56, 45, 52, 99, 100, 50, 45, 97, 49, 55, 49, 45, 48, 57, 100, 48, 100, 53, 50, 51, 99, 50, 54, 51], value={MessageId: b72383f9-a4d8-4cd2-a171-09d0d523c263,ReceiptHandle: AQEBj=,MD5OfBody: 38db8cbd101e4c1cfbf47e31c2aaab75,Body: {"test-key": "test-value"},Attributes: {SentTimestamp=1641794775474},MessageAttributes: {requestTimeMsSinceEpoch={StringValue: 1641794824800,StringListValues: [],BinaryListValues: [],}}}} mutated illegally, new value was ValueWithRecordId{id=[98, 55, 50, 51, 56, 51, 102, 57, 45, 97, 52, 100, 56, 45, 52, 99, 100, 50, 45, 97, 49, 55, 49, 45, 48, 57, 100, 48, 100, 53, 50, 51, 99, 50, 54, 51], value={MessageId: b72383f9-a4d8-4cd2-a171-09d0d523c263,ReceiptHandle: AQE==,MD5OfBody: 38db8cbd101e4c1cfbf47e31c2aaab75,Body: {"test-key": "test-value"},Attributes: {SentTimestamp=1641794775474},MessageAttributes: {requestTimeMsSinceEpoch={StringValue: 1641794824800,StringListValues: [],BinaryListValues: [],}}}}. Encoding was rO2Mw.
where as the same code works in apache beam 2.31.0 without any issues. What am I missing here?
This issue seems to be caused by an indeterministic coder (SerializableCoder.of(Message.class)) in combination with using the SQS reader in batch mode. Batch mode is implemented using BoundedReadFromUnboundedSource, which is known to cause issues. Usage of it is rather discouraged.
You can follow BEAM-13631 to follow progress on fixing the SQS message coder.
Currently I can't tell you what changes between 2.31 and 2.34 are triggering the issue. But possibly it might not be changes in the SQS IO itself. I'll keep investigating a bit further and hope to give an update on that later.
For now, I recommend trying a few things:
First, try avoid using batch mode (so neither setting maxNumRecords nor maxReadTime). I'm pretty confident that this fixes your issue.
Since recent versions of Beam, there's a separate module for AWS SDK v2 beam-sdks-java-io-amazon-web-services2 (hence my question above). It uses a custom message class for transfer rather than the AWS SDK one and encoding should be deterministic. However, I noticed a few other bugs on the SDK v2 IOs when starting to look into it recently: retry on invalid receipt handles, SQS clients closed too early.
Please let me know if either one helps.
The I/O is much more complicated in Beam 2.34.0 than 2.31.0. For Beam 2.34.0, the deleteBatch logic filters messages to delete based on the inflight state. However, there are assumptions in the extend logic where the inflight state is modified to exclude messages that are assumed expired or to be expired. These messages are not explicitly requested by the I/O to be deleted from sqs nor dropped by the I/O itself (the I/O could be processing a message that should have been expired to wait for it to be resent).
Filed https://issues.apache.org/jira/browse/BEAM-13627.
Though I'm not sure if pulling the same message again with a new receipt handle within the same bundle would cause the problem of mutation detection because receipt handle is part of the Message hashcode unless there is a hash collision in the mutation detector.
TL;DR: debugging process
The mutation was detected in the SqsUnboundedSource, not caused by any other code in the pipeline.
The code that reports the warning and throws the exception is here.
The only field changed is the Receipt handle. It's documented here that:
If you receive a message more than once, each time you receive it, you get a different receipt handle. You must provide the most recently received receipt handle when you request to delete the message (otherwise, the message might not be deleted).
There is no aws_java_sdk_version change between Beam 2.31.0 and Beam 2.34.0. So AWS SDK shouldn't be the culprit.
There is a significant change between Beam 2.31.0 and Beam 2.34.0 for SqsUnboundedReader.
To receive a message more than once, the message must not have been deleted since the first time received. The deletion logic is invoked in SqsCheckpointMark.
How can I convert the following .net usage of Guid.ToByteArray() to Java?
var g= Guid.Parse("9836f2b9-ba8c-42a6-b884-2e9eed9fb95a");
var ga = g.ToByteArray();
.Net Byte array returned
ga= [185,242,54,152,140,186,166,66,184,132,46,158,237,159,185,90]
Attempt in Java (Doesn't match .Net array)
UUID uuid = UUID.fromString("9836f2b9-ba8c-42a6-b884-2e9eed9fb95a");
ByteBuffer bb = ByteBuffer.wrap(new byte[16]);
bb.putLong(uuid.getMostSignificantBits());
bb.putLong(uuid.getLeastSignificantBits());
byte[] ga= bb.array();
Java Byte array returned
ga= [-72,-124,46,-98,-19,-97,-71,90,-104,54,-14,-71,-70,-116,66,-90]
Updated w/ solution from Guid to Base64 in Java
UUID uuid = UUID.fromString("9836f2b9-ba8c-42a6-b884-2e9eed9fb95a");
ByteBuffer bb = ByteBuffer.wrap(new byte[16]);
bb.putLong(uuid.getMostSignificantBits());
bb.putLong(uuid.getLeastSignificantBits());
byte[] uuid_bytes = bb.array();
byte[] guid_bytes = Arrays.copyOf(uuid_bytes,uuid_bytes.length);
guid_bytes[0] = uuid_bytes[3];
guid_bytes[1] = uuid_bytes[2];
guid_bytes[2] = uuid_bytes[1];
guid_bytes[3] = uuid_bytes[0];
guid_bytes[4] = uuid_bytes[5];
guid_bytes[5] = uuid_bytes[4];
guid_bytes[6] = uuid_bytes[7];
guid_bytes[7] = uuid_bytes[6];
byte[] ga= guid_bytes;
An alternate solution to the problem -
public static byte[] getByteArrayFromUUID(UUID uuid) {
ByteBuffer mostSignificantBitsByteBuffer = ByteBuffer.allocate(Long.BYTES)
.putLong(uuid.getMostSignificantBits());
return ByteBuffer.allocate(Long.BYTES * 2)
.order(ByteOrder.LITTLE_ENDIAN)
.putShort(mostSignificantBitsByteBuffer.getShort(2))
.putShort(mostSignificantBitsByteBuffer.getShort(0))
.putShort(mostSignificantBitsByteBuffer.getShort(4))
.putShort(mostSignificantBitsByteBuffer.getShort(6))
.order(ByteOrder.BIG_ENDIAN)
.putLong(uuid.getLeastSignificantBits())
.array();
}
The near-equivalent class in Java is java.util.UUID. However, as you've noticed, the two do not give the same byte arrays. But if you execute the following and look at the array given by Java versus the array given by .NET:
import java.nio.ByteBuffer;
import java.util.Arrays;
import java.util.UUID;
public class Main {
// expected from your question
private static final int[] EXPECTED_BYTES = {
185, 242, 54, 152, 140, 186, 166, 66, 184, 132, 46, 158, 237, 159, 185, 90
};
public static void main(String[] args) {
UUID uuid = UUID.fromString("9836f2b9-ba8c-42a6-b884-2e9eed9fb95a");
byte[] array = toByteArray(uuid);
System.out.println("EXPECTED: " + Arrays.toString(EXPECTED_BYTES));
System.out.println("ACTUAL : " + Arrays.toString(toUnsignedInts(array)));
}
private static byte[] toByteArray(UUID uuid) {
return ByteBuffer.allocate(16)
.putLong(uuid.getMostSignificantBits())
.putLong(uuid.getLeastSignificantBits())
.array();
}
// for visual purposes only
private static int[] toUnsignedInts(byte[] array) {
int[] result = new int[array.length];
for (int i = 0; i < array.length; i++) {
result[i] = Byte.toUnsignedInt(array[i]);
}
return result;
}
}
And the output:
EXPECTED: [185, 242, 54, 152, 140, 186, 166, 66, 184, 132, 46, 158, 237, 159, 185, 90]
ACTUAL : [152, 54, 242, 185, 186, 140, 66, 166, 184, 132, 46, 158, 237, 159, 185, 90]
You'll see the arrays are almost equal, it's just the order of some bytes don't match. The last eight bytes (i.e. the least significant bits) all match, but the first four bytes are reversed, the next two bytes are reversed, and so are the next two bytes. To see it visually:
EXPECTED: [185, 242, 54, 152, 140, 186, 166, 66, 184, 132, 46, 158, 237, 159, 185, 90]
ACTUAL : [152, 54, 242, 185, 186, 140, 66, 166, 184, 132, 46, 158, 237, 159, 185, 90]
|---------------| |------| |-----|
I don't know enough to explain why this difference exists, but this comment on an answer to a question you linked to says:
See also [Universally unique identifier - Wikipedia] "Many systems encode the UUID entirely in a big-endian format." "Other systems, notably Microsoft's marshalling of UUIDs in their COM/OLE libraries, use a mixed-endian format, whereby the first three components of the UUID are little-endian, and the last two are big-endian." – Denis Dec 20 '19 at 13:06
The answer that comment is on gives a solution to your problem, which you've included in your question. That solution simply swaps bytes around to get the desired effect. Here's another solution that doesn't involve creating a copy array:
private static byte[] toByteArray(UUID uuid) {
long mostSigBits = uuid.getMostSignificantBits();
return ByteBuffer.allocate(16)
.order(ByteOrder.LITTLE_ENDIAN)
.putInt((int) (mostSigBits >> 32))
.putShort((short) (((int) mostSigBits) >> 16))
.putShort((short) mostSigBits)
.order(ByteOrder.BIG_ENDIAN)
.putLong(uuid.getLeastSignificantBits())
.array();
}
Note: I'm not very comfortable with bit-shifting, so there may be a more succinct way of accomplishing the above that I couldn't think of.
Which gives the following output:
EXPECTED: [185, 242, 54, 152, 140, 186, 166, 66, 184, 132, 46, 158, 237, 159, 185, 90]
ACTUAL : [185, 242, 54, 152, 140, 186, 166, 66, 184, 132, 46, 158, 237, 159, 185, 90]
Warning: Unfortunately, I'm not sure you can rely on either workaround giving the correct bytes 100% of the time.
How to read a .NET Guid into a Java UUID
Is there any difference between a GUID and a UUID?
import java.io.UnsupportedEncodingException;
import java.nio.charset.StandardCharsets;
import java.util.Arrays;
public class Java87String {
public static void main(String[] args) throws UnsupportedEncodingException {
// TODO Auto-generated method stub
//byte[] b = {-101, 53, -51, -26, 24, 60, 20, -31, -6, 45, 50, 103, -66, 28, 114, -39, 92, 23, -47, 32, -5, -122, -28, 79, 22, -76, 116, -122, -54, -122};
//byte[] b = {-76, -55, 85, -50, 80, -23, 27, 62, -94, -74, 47, -123, -119, 94, 90, 61, -63, 73, 56, -48, -54, -4, 11, 79};
byte[] b = { -5, -122, -28};
System.out.println("Input Array :" + Arrays.toString(b));
System.out.println("Array Length : " + b.length);
String target = new String(b,StandardCharsets.UTF_8);
System.out.println(Arrays.toString(target.getBytes("UTF-8")));
System.out.println("Final Key :" + target);
}
}
The above code returns the following output in Java 7
Input Array :[-5, -122, -28]
Array Length : 3
[-17, -65, -67]
Final Key :�
The Same code returns the following output in Java 8
Input Array :[-5, -122, -28]
Array Length : 3
[-17, -65, -67, -17, -65, -67, -17, -65, -67]
Final Key :���
Sounds like Java8 is doing the right thing of replacing with the default sequence of [-17, -65, -67].
Why is there a difference in output and Any Known bugs in JDK 1.7 which fixes this issue?
Per the String JavaDoc:
The behavior of this constructor when the given bytes are not valid in the given charset is unspecified. The CharsetDecoder class should be used when more control over the decoding process is required.
I think (-5, -122, -28) is a invalid UTF-8 byte sequence, so the JVM may output anything in this case. If it were a valid one, maybe the different Java versions could show the same output.
Does this specific byte sequence have a meaning? just curious
I've got a string that I'm trying to convert to bytes in order to create an md5 hash in both ObjC and Java. For some reason, the bytes are different between the two languages.
Java
System.out.println(Arrays.toString(
("78b4a02fa139a2944f17b4edc22fb175:8907f3c4861140ad84e20c8e987eeae6").getBytes()));
Output:
[55, 56, 98, 52, 97, 48, 50, 102, 97, 49, 51, 57, 97, 50, 57, 52, 52, 102, 49, 55, 98, 52, 101, 100, 99, 50, 50, 102, 98, 49, 55, 53, 58, 56, 57, 48, 55, 102, 51, 99, 52, 56, 54, 49, 49, 52, 48, 97, 100, 56, 52, 101, 50, 48, 99, 56, 101, 57, 56, 55, 101, 101, 97, 101, 54]
ObjC
NSString *str = #"78b4a02fa139a2944f17b4edc22fb175:8907f3c4861140ad84e20c8e987eeae6";
NSData *bytes = [str dataUsingEncoding:NSISOLatin1StringEncoding allowLossyConversion:NO];
NSLog(#"%#", [bytes description]);
Output:
<37386234 61303266 61313339 61323934 34663137 62346564 63323266 62313735 3a383930 37663363 34383631 31343061 64383465 32306338 65393837 65656165 36>
I've tried using different charsets with no luck and can't think of any other reasons why the bytes would be different. Any ideas? I did notice that all of the byte values are different by some factor of 18 but am not sure what is causing it.
Actually, Java is printing in decimal, byte by byte. Obj C is printing in hex, integer by integer.
Referring this chart:
Dec Hex
55 37
56 38
98 62
...
You'll just have to find a way to output byte by byte in Obj C.
I don't know about Obj C, but if that NSLog function works similar to printf() in C, I'd start with that.
A code snippet from Apple
unsigned char aBuffer[20];
NSString *myString = #"Test string.";
const char *utfString = [myString UTF8String];
NSData *myData = [NSData dataWithBytes: utfString length: strlen(utfString)];
[myData getBytes:aBuffer length:20];
The change in bytes can be due to Hex representation. The above code shows how to convert the string to bytes and store the result in a buffer.
I have a string encoded in Base64:
eJx9xEERACAIBMBKJyKDcTzR_hEsgOxjAcBQFVVNvi3qEsrRnWXwbhHOmzWnctPHPVkPu-4vBQ==
How can I decode it in Scala language?
I tried to use:
val bytes1 = new sun.misc.BASE64Decoder().decodeBuffer(compressed_code_string)
But when I compare the byte array with the correct one that I generated in Python language, there is an error. Here is the command I used in python:
import base64
base64.urlsafe_b64decode(compressed_code_string)
The Byte Array in Scala is:
(120, -100, 125, -60, 65, 17, 0, 32, 8, 4, -64, 74, 39, 34, -125, 113, 60, -47, -2, 17, 44, -128, -20, 99, 1, -64, 80, 21, 85, 77, -66, 45, -22, 18, -54, -47, -99, 101, -16, 110, 17, -50, -101, 53, -89, 114, -45, -57, 61, 89, 15, -69, -2, 47, 5)
And the one generated in python is:
(120, -100, 125, -60, 65, 17, 0, 32, 8, 4, -64, 74, 39, 34, -125, 113, 60, -47, -2, 17, 44, -128, -20, 99, 1, -64, 80, 21, 85, 77, -66, 45, -22, 18, -54, -47, -99, 101, -16, 110, 17, -50, -101, 53, -89, 114, -45, -57, 61, 89, 15, -69, -18, 47, 5)
Note that there is a single difference in the end of the array
In Scala, Encoding a String to Base64 and decoding back to the original String using Java APIs:
import java.util.Base64
import java.nio.charset.StandardCharsets
scala> val bytes = "foo".getBytes(StandardCharsets.UTF_8)
bytes: Array[Byte] = Array(102, 111, 111)
scala> val encoded = Base64.getEncoder().encodeToString(bytes)
encoded: String = Zm9v
scala> val decoded = Base64.getDecoder().decode(encoded)
decoded: Array[Byte] = Array(102, 111, 111)
scala> val str = new String(decoded, StandardCharsets.UTF_8)
str: String = foo
There is unfortunately not just one Base64 encoding. The - character doesn't have the same representation in all encodings. For example, in the MIME encoding, it's not used at all. In the encoding for URLs, it is a value of 62--and this is the one that Python is using. The default sun.misc decoder wants + for 62. If you change the - to +, you get the correct answer (i.e. the Python answer).
In Scala, you can convert the string s to MIME format like so:
s.map{ case '-' => '+'; case '_' => '/'; case c => c }
and then the Java MIME decoder will work.
Both Python and Java are correct in terms of the decoding. They are just using a different RFC for this purpose. Python library is using RFC 3548 and the used java library is using RFC 4648 and RFC 2045.
Changing the hyphen(-) into a plus(+) from your input string will make the both decoded byte data are similar.