My pipeline is as follows:
Firehose -> Lambda (AWS' Java SDK) -> (S3 & Redshift)
An un-encoded (raw) JSON record is submitted to Firehose. It then triggers a Lambda function which transforms it slightly. Firehose then puts the transformed record into an S3 bucket and into Redshift.
For Firehose to add the transformed data to S3, it requires that the data be Base64 encoded (and Firehose decodes it before adding it to S3).
However, I have a URL within the data that, when decoded, = characters are replaced with their equivalent unicode character (\u003d) due to it being the character that Amazon's Base64 decoder uses as padding.
https://www.[snipped].com/...?returnurl\u003dnull\u0026referrer\u003dnull
How can I retain those = characters within the decoded data?
Note: I've tried using Base64.getUrlEncoder(), but AWS only seems to support Base64.getEncoder().
It turns out that HTML escaping was enabled on the JSON library (Gson) that I was using when (de)serializing my Lambda record. To fix it, I just had to disable HTML escaping:
new GsonBuilder().disableHtmlEscaping().create();
Related
I am trying to decrypt files that arrives periodically into our s3 bucket.How can I process if the file size is huge (eg 10GB) ,since the computing resources of Lambda is Limited. Im not sure if it is necessary download the whole file into Lambda and perform the decryption or is there some other way we can chunk the file and process?
Edit :- Processing the file here includes decrypting the file and the parse each row and write it to persistent store like a SQL queue or Database.
You can set the byte-range in the GetObjectRequest to load a specific range of bytes from an S3 object.
The following example comes from the AWS official documentation on S3 GetObject API:
// Get a range of bytes from an object and print the bytes.
GetObjectRequest rangeObjectRequest = new GetObjectRequest(bucketName, key).withRange(0, 9);
objectPortion = s3Client.getObject(rangeObjectRequest);
System.out.println("Printing bytes retrieved.");
displayTextInputStream(objectPortion.getObjectContent());
For more information, you can visit the documentation here:
https://docs.aws.amazon.com/AmazonS3/latest/userguide/download-objects.html
I guess this is a stupid question with an obvious solution, but I don't see it yet. So the problem is: I get an IllegalArgumentException on Android, which says my base64 input is not valid. This input took the following way before:
Upload: PDF file -(Java Base64 encoder) > Java Base64 encoded string -(POST)-> PHP -(INSERT as mediumtext via mysqli query)-> MySQL DB
Download: MySQL record -(SELECT via mysqli query and fetch assoc afterwards)-> PHP vars -(JSON)-> Java as JSON -(Jackson library, maps JSON to object containing String)-> Java Base64 String - (Android Base64 decoder)-> Exception
Is there any failure in my workflow? Communication is done with UTF-8 via HttpUrlConnection.
I was able to solve the problem: During transfer to the server + and / got omitted. After manually replacing them, everything is working now.
I am working with the GCS API, attempting to create a survey with image data.
I am using the NuGet package Google.Apis.ConsumerSurveys.v2 version 1.14.0.564 on the .Net platform. I can create surveys that do not contain image data without problem. However, when I try to create a survey with image data I receive an error from the API.
I have on hand base64 encoded png format image data. My images display properly in an IMG tag on a web page when the src attribute is set to
'data:image/png;base64,<image base64 string>'
I want to send this image data to the API to populate the survey image. My understanding is that I need to set the Data property of the Google.Apis.ConsumerSurveys.v2.Data.SurveyQuestionImage object to a string containing the image data. I have not been successful.
I first decode my base64 string to a byte array:
byte[] bytes = Convert.FromBase64String(<image base64 string>);
I have tried setting the Data property in the SurveyQuestionImage object as:
image.Data = Encoding.Unicode.GetString(bytes);
This results in this error from the API:
Google.Apis.Requests.RequestError Invalid value for ByteString: <the Data string>
I have also tried converting the byte array to a hexadecimal encoded string as:
StringBuilder sb = new StringBuilder(bytes.Length);
foreach (Byte b in bytes)
{
sb.Append(b.ToString("X2"));
}
image.Data = sb.ToString();
This results in the more hopeful error:
Google.Apis.Requests.RequestError Invalid Value supplied to API: image_data was bad. Request Id: 579665c300ff05e6c316a09e600001737e3430322d747269616c320001707573682d30372d32322d72313000010112 [400] Errors [ Message[Invalid Value supplied to API: image_data was bad. Request Id: 579665c300ff05e6c316a09e600001737e3430322d747269616c320001707573682d30372d32322d72313000010112] Location[ - ] Reason[INVALID_VALUE] Domain[global] ]
Does anyone know the correct format for the Data property of the Google.Apis.ConsumerSurveys.v2.Data.SurveyQuestionImage object?
The data needs to be base64 encoded and also "urlsafe" or "websafe" depending on what language you are using. (python and java, respectively)
In other words, you'll need to first base64 encode then:
Web safe encoding uses '-' instead of '+', '_' instead of '/'
Hope this helps!
For c# users, check out this technique for making websafe b64:
How to achieve Base64 URL safe encoding in C#?
For .net users, look at the comments in this question:
Converting string to web-safe Base64 format
And also this link for more info about .net specific options for encoding:
http://www.codeproject.com/Tips/76650/Base-base-url-base-url-and-z-base-encoding
And to specifically answer the original poster, try this for converting your byte array to a string.
public static string ToBase64ForUrlString(byte[] input)
{
StringBuilder result = new StringBuilder(Convert.ToBase64String(input).TrimEnd('='));
result.Replace('+', '-');
result.Replace('/', '_');
return result.ToString();
}
I use similar code as its shown here in the question.
Java and AppStore receipt verification
But I still end up getting
{"status":21002, "exception":"java.lang.IllegalArgumentException"}
Can it be a problem at Base64 encoding?. Do I have to convert the base64 encoded string into hex or something else?.
What i post is similar to following
{"receipt-data" : "eyJzaWduYXR1cmUiOiJBbjNJVER0VVNmZWNhaGMxR.....
The problem was at Base64 encoding inside Java. When I do the encoding inside IOS and use that as the request from server without any encoding in Java, then it worked.
I had a similar problem and was receiving the java.lang.IllegalArgumentException from Apple when trying to validate a receipt on my server. The problem was that my base64 encoding logic was inserting lines breaks into the encoded string. Once I updated my code to ensure no new line breaks were being inserted into the encoded string, I was able to successfully verify my receipts against Apple's servers.
I am streaming an image to Magento, and encoding an image using android.util.Base64 using either of:
Base64.encodeToString(content, Base64.CRLF)
Base64.encodeToString(content, Base64.DEFAULT)
But I always receive fault:
The image contents is not valid base64 data
Working: I found that the data had to be encoded twice, one time using
Base64 and another encoding using custom Library
Try removing data node from your base64 code for image.
e.g. if you have data like data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAVQAAABXCAYAA...
then remove data node. It should look like below and pass it to Magento.
iVBORw0KGgoAAAANSUhEUgAAAVQAAABXCAYAA...