We Keep Coding

Kafka Connect S3 sink connector with custom Partitioner strange behavior

I plan to use a custom Field and TimeBased partitioner to partition my data in s3 as follow:
/part_<field_name>=<field_value>/part_date=YYYY-MM-dd/part_hour=HH/....parquet.
My Partitioner works fine, everything is as expected in my S3 bucket.
The problem is linked to the performance of the sink
I have 400kB/s/broker = ~1.2MB/s in my input topic and the sink works with spikes and commit a small number of records.
If I use the classic TimeBasedPartitioner, enter image description here
So my problem seems to be in my custom partitioner. Here is the code:
package test;
import ...;
public final class FieldAndTimeBasedPartitioner<T> extends TimeBasedPartitioner<T> {
private static final Logger log = LoggerFactory.getLogger(FieldAndTimeBasedPartitioner.class);
private static final String FIELD_SUFFIX = "part_";
private static final String FIELD_SEP = "=";
private long partitionDurationMs;
private DateTimeFormatter formatter;
private TimestampExtractor timestampExtractor;
private PartitionFieldExtractor partitionFieldExtractor;
protected void init(long partitionDurationMs, String pathFormat, Locale locale, DateTimeZone timeZone, Map<String, Object> config) {
this.delim = (String)config.get("directory.delim");
this.partitionDurationMs = partitionDurationMs;
try {
this.formatter = getDateTimeFormatter(pathFormat, timeZone).withLocale(locale);
this.timestampExtractor = this.newTimestampExtractor((String)config.get("timestamp.extractor"));
this.timestampExtractor.configure(config);
this.partitionFieldExtractor = new PartitionFieldExtractor((String)config.get("partition.field"));
} catch (IllegalArgumentException e) {
ConfigException ce = new ConfigException("path.format", pathFormat, e.getMessage());
ce.initCause(e);
throw ce;
}
}
private static DateTimeFormatter getDateTimeFormatter(String str, DateTimeZone timeZone) {
return DateTimeFormat.forPattern(str).withZone(timeZone);
}
public static long getPartition(long timeGranularityMs, long timestamp, DateTimeZone timeZone) {
long adjustedTimestamp = timeZone.convertUTCToLocal(timestamp);
long partitionedTime = adjustedTimestamp / timeGranularityMs * timeGranularityMs;
return timeZone.convertLocalToUTC(partitionedTime, false);
}
public String encodePartition(SinkRecord sinkRecord, long nowInMillis) {
final Long timestamp = this.timestampExtractor.extract(sinkRecord, nowInMillis);
final String partitionField = this.partitionFieldExtractor.extract(sinkRecord);
return this.encodedPartitionForFieldAndTime(sinkRecord, timestamp, partitionField);
}
public String encodePartition(SinkRecord sinkRecord) {
final Long timestamp = this.timestampExtractor.extract(sinkRecord);
final String partitionFieldValue = this.partitionFieldExtractor.extract(sinkRecord);
return encodedPartitionForFieldAndTime(sinkRecord, timestamp, partitionFieldValue);
}
private String encodedPartitionForFieldAndTime(SinkRecord sinkRecord, Long timestamp, String partitionField) {
if (timestamp == null) {
String msg = "Unable to determine timestamp using timestamp.extractor " + this.timestampExtractor.getClass().getName() + " for record: " + sinkRecord;
log.error(msg);
throw new ConnectException(msg);
} else if (partitionField == null) {
String msg = "Unable to determine partition field using partition.field '" + partitionField + "' for record: " + sinkRecord;
log.error(msg);
throw new ConnectException(msg);
} else {
DateTime recordTime = new DateTime(getPartition(this.partitionDurationMs, timestamp.longValue(), this.formatter.getZone()));
return this.FIELD_SUFFIX
+ config.get("partition.field")
+ this.FIELD_SEP
+ partitionField
+ this.delim
+ recordTime.toString(this.formatter);
}
}
static class PartitionFieldExtractor {
private final String fieldName;
PartitionFieldExtractor(String fieldName) {
this.fieldName = fieldName;
}
String extract(ConnectRecord<?> record) {
Object value = record.value();
if (value instanceof Struct) {
Struct struct = (Struct)value;
return (String) struct.get(fieldName);
} else {
FieldAndTimeBasedPartitioner.log.error("Value is not of Struct !");
throw new PartitionException("Error encoding partition.");
}
}
}
public long getPartitionDurationMs() {
return partitionDurationMs;
}
public TimestampExtractor getTimestampExtractor() {
return timestampExtractor;
}
}
It's more or less a merge of FieldPartitioner and TimeBasedPartitioner.
Any clue on why I have suck a bad performance on while sinking messages ?
While partitioning using field in the record, deserialize and extract data from the message can cause this issue ?
As I have around 80 different fields values, can it be a memory issue as it will maintain 80 times more buffers in the heap ?
Thanks for your help.

FYI, the problem was the partitioner itself. My partitioner needed to decode the entire message and get the info.
As I have a lot of messages, it takes time to handle all these events.

jsoup to get div elements with classes

I am new to Jsoup parsing and I want to get the list of all the companies on this page: https://angel.co/companies?company_types[]=Startup
Now, a way to do this is actually to inspect the page with the div tags relevant to what I need.
However, when I call the method :
Document doc = Jsoup.connect("https://angel.co/companies?company_types[]=Startup").get();
System.out.println(doc.html());
Firstly I cannot even find those DIV tags in my consol html output, (the ones which are supposed to give a list of the companies)
Secondly, even if I did find it, how can I find a certain Div element with class name :
div class=" dc59 frw44 _a _jm"
Pardon the jargon, I have no idea how to go through this.

The data are not embedded in the page but they are retrieved using subsequent API calls :
a POST https://angel.co/company_filters/search_data to get an ids array & a token named hexdigest
a GET https://angel.co/companies/startups to retrieve company data using the output from the previous request
The above is repeated for each page (thus a new token & a list of ids are needed for each page). This process can be seen using Chrome dev console in Network tabs.
The first POST request gives JSON output but the second request (GET) gives HTML data in a property of a JSON object.
The following extracts the company filter :
private static CompanyFilter getCompanyFilter(final String filter, final int page) throws IOException {
String response = Jsoup.connect("https://angel.co/company_filters/search_data")
.header("Content-Type", "application/x-www-form-urlencoded;charset=UTF-8")
.header("X-Requested-With", "XMLHttpRequest")
.data("filter_data[company_types][]=", filter)
.data("sort", "signal")
.data("page", String.valueOf(page))
.userAgent("Mozilla")
.ignoreContentType(true)
.post().body().text();
GsonBuilder gsonBuilder = new GsonBuilder();
Gson gson = gsonBuilder.create();
return gson.fromJson(response, CompanyFilter.class);
}
Then the following extracts companies :
private static List<Company> getCompanies(final CompanyFilter companyFilter) throws IOException {
List<Company> companies = new ArrayList<>();
URLConnection urlConn = new URL("https://angel.co/companies/startups?" + companyFilter.buildRequest()).openConnection();
urlConn.setRequestProperty("User-Agent", "Mozilla");
urlConn.connect();
BufferedReader reader = new BufferedReader(new InputStreamReader(urlConn.getInputStream(), "UTF-8"));
HtmlContainer htmlObj = new Gson().fromJson(reader, HtmlContainer.class);
Element doc = Jsoup.parse(htmlObj.getHtml());
Elements data = doc.select("div[data-_tn]");
if (data.size() > 0) {
for (int i = 2; i < data.size(); i++) {
companies.add(new Company(data.get(i).select("a").first().attr("title"),
data.get(i).select("a").first().attr("href"),
data.get(i).select("div.pitch").first().text()));
}
} else {
System.out.println("no data");
}
return companies;
}
The main function :
public static void main(String[] args) throws IOException {
int pageCount = 1;
List<Company> companies = new ArrayList<>();
for (int i = 0; i < 10; i++) {
System.out.println("get page n°" + pageCount);
CompanyFilter companyFilter = getCompanyFilter("Startup", pageCount);
pageCount++;
System.out.println("digest : " + companyFilter.getDigest());
System.out.println("count : " + companyFilter.getTotalCount());
System.out.println("array size : " + companyFilter.getIds().size());
System.out.println("page : " + companyFilter.getpage());
companies.addAll(getCompanies(companyFilter));
if (companies.size() == 0) {
break;
} else {
System.out.println("size : " + companies.size());
}
}
}
Company, CompanyFilter & HtmlContainer are model class :
class CompanyFilter {
#SerializedName("ids")
private List<Integer> mIds;
#SerializedName("hexdigest")
private String mDigest;
#SerializedName("total")
private String mTotalCount;
#SerializedName("page")
private int mPage;
#SerializedName("sort")
private String mSort;
#SerializedName("new")
private boolean mNew;
public List<Integer> getIds() {
return mIds;
}
public String getDigest() {
return mDigest;
}
public String getTotalCount() {
return mTotalCount;
}
public int getpage() {
return mPage;
}
private String buildRequest() {
String out = "total=" + mTotalCount + "&";
out += "sort=" + mSort + "&";
out += "page=" + mPage + "&";
out += "new=" + mNew + "&";
for (int i = 0; i < mIds.size(); i++) {
out += "ids[]=" + mIds.get(i) + "&";
}
out += "hexdigest=" + mDigest + "&";
return out;
}
}
private static class Company {
private String mLink;
private String mName;
private String mDescription;
public Company(String name, String link, String description) {
mLink = link;
mName = name;
mDescription = description;
}
public String getLink() {
return mLink;
}
public String getName() {
return mName;
}
public String getDescription() {
return mDescription;
}
}
private static class HtmlContainer {
#SerializedName("html")
private String mHtml;
public String getHtml() {
return mHtml;
}
}
The full code is also available here

Issue with toString() Implementation in Android

I have a slightly more advanced class I am trying to write a toString() for
in order to accomplish what I am trying to do I need to be able to change the assignment of certain variables when doing toString().
TO make it simple I am going to remove a bunch of stuff except what allows it to work.
public enum PacketElementType {
NONE((byte)0, "None"),
BYTE((byte)1, "Byte"),
SHORT((byte)2, "Short"),
INT((byte)3, "Int"),
LONG((byte)4, "Long"),
FLOAT((byte)5, "Float"),
STRING((byte)6, "String"),
BIN((byte)7, "Bin");
private final byte typeValue;
private final String typeName;
PacketElementType(byte type, String name)
{
this.typeValue = type;
this.typeName = name;
}
public String getTypeName() {
return typeName;
}
public byte getTypeValue() {
return typeValue;
}
}
public class Packet {
private final int DEFAULT_SIZE = 1024 * 2;
private final int ADD_SIZE = 1024;
private byte[] _buffer = new byte[1];
private int _ptr = 0;
private int _bodyStart = 0;
private int _elements, _bodyLen = 0;
private int op;
private long id;
public Packet(int op, long id) {
setOp(op);
setId(id);
_buffer = new byte[DEFAULT_SIZE];
}
public int getOp() {
return op;
}
public void setOp(int op) {
this.op = op;
}
public long getId() {
return id;
}
public void setId(long id) {
this.id = id;
}
public PacketElementType peek() {
int pie = _ptr;
if (pie + 2 > _buffer.length)
return PacketElementType.NONE;
return PacketElementType.values()[_buffer[_ptr]];
}
protected Packet putSimple(PacketElementType type, byte... val) {
int len = val.length + 1;
this.ensureSize(len);
_buffer[++_ptr] = type.getTypeValue();
System.arraycopy(val, 0, _buffer, _ptr, val.length);
_ptr += val.length;
_elements++;
_bodyLen += len;
return this;
}
public Packet putByte(byte val) {
return this.putSimple(PacketElementType.BYTE, val);
}
public Packet putByte(boolean val) {
return this.putByte(val ? (byte) 1 : (byte) 0);
}
public byte getByte() throws Exception {
if (this.peek() != PacketElementType.BYTE)
throw new Exception("Expected Byte, got " + this.peek().getTypeName() + ".");
_ptr += 1;
return _buffer[++_ptr];
}
protected void ensureSize(int required) {
if (_ptr + required >= _buffer.length) {
byte[] b = new byte[_buffer.length + Math.max(ADD_SIZE, required * 2)];
System.arraycopy(_buffer, 0, b, 0, _buffer.length);
_buffer = b;
}
}
private boolean isValidType(PacketElementType type)
{
return (type.getTypeValue() >= PacketElementType.BYTE.getTypeValue() && type.getTypeValue() <= PacketElementType.BIN.getTypeValue());
}
protected String toStringHack()
{
StringBuilder result = new StringBuilder();
int prevPtu = _ptr;
_ptr = _bodyStart;
try {
result.append(String.format("Op: %1$08d %3$s, Id: %2$016d\r\n", this.getOp(), this.getId(), Op.getName(this.getOp())));
} catch (IllegalAccessException e) {
e.printStackTrace();
return result.append("Failed to convert packet to string").toString();
}
PacketElementType type;
for (int i = 1; (this.isValidType(type = this.peek()) && _ptr < _buffer.length); ++i)
{
if (type == PacketElementType.BYTE)
{
byte data = 0;
try {
data = getByte();
} catch (Exception e) {
e.printStackTrace();
result.append("Failed to parse element at position ").append(i);
continue;
}
result.append(String.format("%1&03d [%2$s] Byte : %3$s", i, String.format("%1$016d", data), data));
}
}
return result.toString();
}
//TODO: toString
#Override
public String toString()
{
return toStringHack();
}
}
public class Op {
public class Msgr
{
}
public static String getName(int op) throws IllegalAccessException {
for (Field field : Op.class.getFields())
{
if ((int)field.get(null) == op)
return field.getName();
}
for (Field field : Op.Msgr.class.getFields())
{
if ((int)field.get(null) == op)
return field.getName();
}
return "?";
}
}
[2
When debugging, _ptr won't set in toString(), when not debugging, _ptr won't set in putSimple().
I'm so close to pulling my hair out, please and thanks, if you could help me I would really be glad! Please and thank you again!
To test for this bug please review the following example:
Packet p = new Packet(1, 10001).putByte(true);
Toast.makeText(this, p.toString(), Toast.LENGTH_LONG).show();
for me I throw this inside the built in test class first, and then tried it in the onCreate from the main activity.
toString() will only return the Op and Id because _ptr is at , peek() will attempt to read the byte starting at that position instead of at 0 where it would find our 1 byte.
Edit
It seems like... _ptr = _bodyStart; is being seen as something other than an assignment, is this possible?

Result you see is ok - debugger shows you those variables before evaluation. Ad a line after this one (like a log or smth.) and set breakpoint on it.

so it turns out I was missing just one small tiny little details..... Not tiny at all, I apologize for not seeing this earlier. toString() would fail because of a malformed String.format() call as well as the failure to set ptr back to it's original value after toString() was completed.
result.append(String.format("%1&03d [%2$s] Byte : %3$s", i, String.format("%1$016d", data), data));
Should have been (where right after %1 I had an & instead of a $)
String hello = String.format("%1$03d [%2$s] Byte : %3$d\r\n", i, StringUtils.leftPad(String.format("%02X", data), 16, '.'), (int) data);
and just before returning the string, I needed to do the following
ptr = prevPtu;
with that, the following happens:

generating a number between a range using json

How can we generate a number between a range using Json.
Like we have to generate a number between 0 to 50, how can we perform this in Java using a Json.
This is my Json Data
{
"rand": {
"type': "number",
"minimum": 0,
"exclusiveMinimum": false,
"maximum": 50,
"exclusiveMaximum": true
}
}
This is what I have tried in Java
public class JavaApplication1 {
public static void main(String[] args) {
try {
for (int i=0;i<5;i++)
{
FileInputStream fileInputStream = new FileInputStream("C://users/user/Desktop/V.xls");
HSSFWorkbook workbook = new HSSFWorkbook(fileInputStream);
HSSFSheet worksheet = workbook.getSheet("POI Worksheet");
HSSFRow row1 = worksheet.getRow(0);
String e1Val = cellE1.getStringCellValue();
HSSFCell cellF1 = row1.getCell((short) 5);
System.out.println("E1: " + e1Val);
JSONObject obj = new JSONObject();
obj.put("value", e1Val);
System.out.print(obj + "\n");
Map<String,Object> c_data = mapper.readValue(e1Val, Map.class);
System.out.println(a);
}
} catch (FileNotFoundException e) {
} catch (IOException e) {
}
}
}
Json Data is stored in excel sheet, from there I am reading it in Java program

Get a Json-reader like GSON.
Read in the JSON to an equivalent Object like
public class rand{
private String type;
private int minimum;
private boolean exclusiveMinimum;
private int maximum;
private boolean exclusiveMaximum;
//this standard-constructor is needed for the JsonReader
public rand(){
}
//Getter for all Values
}
and after reading in your JSON you can access your Data via your getter-methods

I think that Jackson may be of help here.
I suggest that you create a data model in Java that reflects the JSON. This can along the lines of:
// This is the root object. It contains the input data (RandomizerInput) and a
// generate-function that is used for generating new random ints.
public class RandomData {
private RandomizerInput input;
#JsonCreator
public RandomData(#JsonProperty("rand") final RandomizerInput input) {
this.input = input;
}
#JsonProperty("rand")
public RandomizerInput getInput() {
return input;
}
#JsonProperty("generated")
public int generateRandomNumber() {
int max = input.isExclusiveMaximum()
? input.getMaximum() - 1 : input.getMaximum();
int min = input.isExclusiveMinimum()
? input.getMinimum() + 1 : input.getMinimum();
return new Random().nextInt((max - min) + 1) + min;
}
}
// This is the input data (pretty much what is described in the question).
public class RandomizerInput {
private final boolean exclusiveMaximum;
private final boolean exclusiveMinimum;
private final int maximum;
private final int minimum;
private final String type;
#JsonCreator
public RandomizerInput(
#JsonProperty("type") final String type,
#JsonProperty("minimum") final int minimum,
#JsonProperty("exclusiveMinimum") final boolean exclusiveMinimum,
#JsonProperty("maximum") final int maximum,
#JsonProperty("exclusiveMaximum") final boolean exclusiveMaximum) {
this.type = type; // Not really used...
this.minimum = minimum;
this.exclusiveMinimum = exclusiveMinimum;
this.maximum = maximum;
this.exclusiveMaximum = exclusiveMaximum;
}
public int getMaximum() {
return maximum;
}
public int getMinimum() {
return minimum;
}
public String getType() {
return type;
}
public boolean isExclusiveMaximum() {
return exclusiveMaximum;
}
public boolean isExclusiveMinimum() {
return exclusiveMinimum;
}
}
To use these classes the ObjectMapper from Jackson can be used like this:
public static void main(String... args) throws IOException {
String json =
"{ " +
"\"rand\": { " +
"\"type\": \"number\", " +
"\"minimum\": 0, " +
"\"exclusiveMinimum\": false, " +
"\"maximum\": 50, " +
"\"exclusiveMaximum\": true " +
"} " +
"}";
// Create the mapper
ObjectMapper mapper = new ObjectMapper();
// Convert JSON to POJO
final RandomData randomData = mapper.readValue(json, RandomData.class);
// Either you can get the random this way...
final int random = randomData.generateRandomNumber();
// Or, you can serialize the whole thing as JSON....
String str = mapper.writeValueAsString(randomData);
// Output is:
// {"rand":{"type":"number","minimum":0,"exclusiveMinimum":false,"maximum":50,"exclusiveMaximum":true},"generated":21}
System.out.println(str);
}
The actual generation of a random number is based on this SO question.

How to filter events in Jersey SSE?

When i was looking examples of jersey sse i have found one example sse-item-store-webapp in jersey example folder. It is very simple app that has one input and one button. You type some text, click the button and other people get changes.
#Path("items")
public class ItemStoreResource {
private static final ReentrantReadWriteLock storeLock = new ReentrantReadWriteLock();
private static final LinkedList<String> itemStore = new LinkedList<String>();
private static final SseBroadcaster broadcaster = new SseBroadcaster();
private static volatile long reconnectDelay = 0;
#GET
#Produces(MediaType.TEXT_PLAIN)
public String listItems() {
try {
storeLock.readLock().lock();
return itemStore.toString();
} finally {
storeLock.readLock().unlock();
}
}
#GET
#Path("events")
#Produces(SseFeature.SERVER_SENT_EVENTS)
public EventOutput itemEvents(#HeaderParam(SseFeature.LAST_EVENT_ID_HEADER) #DefaultValue("-1") int lastEventId) {
final EventOutput eventOutput = new EventOutput();
if (lastEventId >= 0) {
LOGGER.info("Received last event id :" + lastEventId);
// decide the reconnect handling strategy based on current reconnect delay value.
final long delay = reconnectDelay;
if (delay > 0) {
LOGGER.info("Non-zero reconnect delay [" + delay + "] - responding with HTTP 503.");
throw new ServiceUnavailableException(delay);
} else {
LOGGER.info("Zero reconnect delay - reconnecting.");
replayMissedEvents(lastEventId, eventOutput);
}
}
if (!broadcaster.add(eventOutput)) {
LOGGER.severe("!!! Unable to add new event output to the broadcaster !!!");
// let's try to force a 5s delayed client reconnect attempt
throw new ServiceUnavailableException(5L);
}
return eventOutput;
}
private void replayMissedEvents(final int lastEventId, final EventOutput eventOutput) {
try {
storeLock.readLock().lock();
final int firstUnreceived = lastEventId + 1;
final int missingCount = itemStore.size() - firstUnreceived;
if (missingCount > 0) {
LOGGER.info("Replaying events - starting with id " + firstUnreceived);
final ListIterator<String> it = itemStore.subList(firstUnreceived, itemStore.size()).listIterator();
while (it.hasNext()) {
eventOutput.write(createItemEvent(it.nextIndex() + firstUnreceived, it.next()));
}
} else {
LOGGER.info("No events to replay.");
}
} catch (IOException ex) {
throw new InternalServerErrorException("Error replaying missed events", ex);
} finally {
storeLock.readLock().unlock();
}
}
#POST
public void addItem(#FormParam("name") String name) {
// Ignore if the request was sent without name parameter.
if (name == null) {
return;
}
final int eventId;
try {
storeLock.writeLock().lock();
eventId = itemStore.size();
itemStore.add(name);
// Broadcasting an un-named event with the name of the newly added item in data
broadcaster.broadcast(createItemEvent(eventId, name));
// Broadcasting a named "size" event with the current size of the items collection in data
broadcaster.broadcast(new OutboundEvent.Builder().name("size").data(Integer.class, eventId + 1).build());
} finally {
storeLock.writeLock().unlock();
}
}
private OutboundEvent createItemEvent(final int eventId, final String name) {
Logger.getLogger(ItemStoreResource.class.getName()).info("Creating event id [" + eventId + "] name [" + name + "]");
return new OutboundEvent.Builder().id("" + eventId).data(String.class, name).build();
}
}
For example, if i have a chat rooms i don't understand how to implement that using SSE becouse every client connects to /items/events and if someone post new message to some chat broadcaster will broadcast this message to all signed events however i want broadcast events only for some chat.
Who works with Jersey SSE could you advise how to implement that ?

try to use smth like a map with chat room ids to SseBroadcast object, then you could subscribe all users from certain room to broadcaster. You may use that as for tet-a-tet conversations or team conversations.
Sample below:
private static final Map<Long, SseBroadcaster> ROOM_SSE_BROADCASTER = new ConcurrentHashMap<>();
#GET
#Path("/updatestate/{roomId}/{userId}")
#Produces(SseFeature.SERVER_SENT_EVENTS)
public EventOutput updateState(#PathParam("roomId") Long roomId, #PathParam("userId") Long userId) {
EventOutput eo = new EventOutput();
ROOM_SSE_BROADCASTER.get(roomId).add(eo);
return eo;
}
public static void updateRoom(Long roomId) {
ROOM_SSE_BROADCASTER.get(roomId).broadcast(buildEvent());
}
public static void registerRoom(Long roomId) {
ROOM_SSE_BROADCASTER.put(roomId, new SseBroadcaster());
}
private static OutboundEvent buildEvent() {
OutboundEvent.Builder builder = new OutboundEvent.Builder();
OutboundEvent event = builder.data(String.class, "update").build();
return event;
}