I have a bot which analyses chat command and provides a small database. When I tell the bot it should write the content of the database to the irc, it gets kicked because of excess flooding. How can I prevent this?
(I'm using freenode.net)
PS: Here is my code:
for (User u : users.values()) {
ps.println("PRIVMSG " + CHANNEL + " :" + u.toString());
}
ps is a PrintStream which writes to Socket.getOutputStream().
CHANNEL is the name of my channel. users.size() is ~ 60.
Obviously a client can't change a servers thresholds for detecting flooding and disconnecting clients which do so. The whole idea of flood protection would be pointless, otherwise.
So, you must limit your output to such a level, that it will not trigger flooding kick. Add a (configurable) sleep between ps.println calls, and set the sleep duration so that you don't get kicked.
Work-around would be to use a pastebin-like service, and just send URL and summary/title text to IRC in one line. Using these services is a standard procedure for any IRC paste longer than 3 lines these days anyway. Various "pastebin" services have REST APIs (example), so this should be rather easy way to go.
Related
As I wrote in title we need in project notify or execute method of some thread by another. This implementation is part of long polling. In following text describe and show my implementation.
So requirements are that:
UserX send request from client to server (poll action) immediately when he got response from previous. In service is executed spring async method where thread immediately check cache if there are some new data in database. I know that cache is usually used for methods where for specific input is expected specific output. This is not that case, because I use cache to reduce database calls and output of my method is always different. So cache help me store notification if I should check database or not. This checking is running in while loop which end when thread find notification to read database in cache or time expired.
Assume that UserX thread (poll action) is currently in while loop and checking cache.
In that moment UserY (push action) send some data to server, data are stored in database in separated thread, and also in cache is stored userId of recipient.
So when UserX is checking cache he found id of recipient (id of recipient == his id in this case), and then break loop and fetch these data.
So in my implementation I use google guava cache which provide manually write.
private static Cache<Long, Long> cache = CacheBuilder.newBuilder()
.maximumSize(100)
.expireAfterWrite(5, TimeUnit.MINUTES)
.build();
In create method I store id of user which should read these data.
public void create(Data data) {
dataRepository.save(data);
cache.save(data.getRecipient(), null);
System.out.println("SAVED " + userId + " in " + Thread.currentThread().getName());
}
and here is method of polling data:
#Async
public CompletableFuture<List<Data>> pollData(Long previousMessageId, Long userId) throws InterruptedException {
// check db at first, if there are new data no need go to loop and waiting
List<Data> data = findRecent(dataId, userId));
data not found so jump to loop for some time
if (data.size() == 0) {
short c = 0;
while (c < 100) {
// check if some new data added or not, if yes break loop
if (cache.getIfPresent(userId) != null) {
break;
}
c++;
Thread.sleep(1000);
System.out.println("SEQUENCE: " + c + " in " + Thread.currentThread().getName());
}
// check database on the end of loop or after break from loop
data = findRecent(dataId, userId);
}
// clear data for that recipient and return result
cache.clear(userId);
return CompletableFuture.completedFuture(data);
}
After User X got response he send poll request again and whole process is repeated.
Can you tell me if is this application design for long polling in java (spring) is correct or exists some better way? Key point is that when user call poll request, this request should be holded for new data for some time and not response immediately. This solution which I show above works, but question is if it will be works also for many users (1000+). I worry about it because of pausing threads which should make slower another requests when no threads will be available in pool. Thanks in advice for your effort.
Check Web Sockets. Spring supports it from version 4 on wards. It doesn't require client to initiate a polling, instead server pushes the data to client in real time.
Check the below:
https://spring.io/guides/gs/messaging-stomp-websocket/
http://www.baeldung.com/websockets-spring
Note - web sockets open a persistent connection between client and server and thus may result in more resource usage in case of large number of users. So, if you are not looking for real time updates and is fine with some delay then polling might be a better approach. Also, not all browsers support web sockets.
Web Sockets vs Interval Polling
Longpolling vs Websockets
In what situations would AJAX long/short polling be preferred over HTML5 WebSockets?
In your current approach, if you are having a concern with large number of threads running on server for multiple users then you can trigger the polling from front-end every time instead. This way only short lived request threads will be triggered from UI looking for any update in the cache. If there is an update, another call can be made to retrieve the data. However don't hit the server every other second as you are doing otherwise you will have high CPU utilization and user request threads may also suffer. You should do some optimization on your timing.
Instead of hitting the cache after a delay of 1 sec for 100 times, you can apply an intelligent algorithm by analyzing the pattern of cache/DB update over a period of time.
By knowing the pattern, you can trigger the polling in an exponential back off manner to hit the cache when the update is most likely expected. This way you will be hitting the cache less frequently and more accurately.
I'm working with Pupil Labs, a huge open source for eye/pupil tracking. The entire code is written in Python. The so-called Pupil Remote is based on ZeroMQ.
If I start running the Filter Messages everything is fine. For my purposes I need to "translate" Filter Messages into Java because I created an Android app, which should call a client, which has the purpose to serve as the python client.
Here's what I've done so far:
import android.annotation.SuppressLint;
import org.zeromq.ZMQ;
import java.nio.charset.Charset;
import static java.lang.Thread.sleep;
public class ZeroMQClient {
#SuppressLint("NewApi")
public static void requestGazeData() {
ZMQ.Context context = ZMQ.context(1);
ZMQ.Socket subscriber = context.socket(ZMQ.SUB);
System.out.println("Connecting to server...");
subscriber.connect("tcp://xxx.x.x.x:50020");
System.out.println("Connected");
String gaze = "gaze";
subscriber.subscribe(gaze.getBytes(Charset.forName("UTF-8")));
while (true) {
String msg = subscriber.recvStr();
System.out.println(msg);
subscriber.close();
context.term();
}
}
}
Now as you can expect, why I'm asking you, nothing happens, I don't receive any data from the Pupil Labs server. I oriented myself on this post, but unfortunately, it didn't work out for me. Also the IP-Address and port are the same as on the server. It works neither locally nor remotely.
Happy about any answer, since I stuck at this.
Due to the correct establishment in terms of my implementation the actual issue was the firewall, which just blocked the connection. By posting my solution I'm hopefully able to help future visitors of this question.
The final solution, after having debugged the root-cause issue is below
Happy about having answer, you have to set a subscription Policy:
ZeroMQ expects each SUB-side to first explicitly say, what this SUB-side wants to receive from PUB ( Yes, what it to subscribes to ).
Like your mailbox will never get newspapers in, without first subscribing to any. :o)
So setup an empty string "" in the subscriber and you are done:
// String filterPermitANY = ""; // WAS AN EXAMPLE TO TEST
// subscriber.subscribe( filterPermitANY.getBytes() );// IF PUB.send()-s ANY
String gaze = "gaze"; // WAS ON TOPIC
subscriber.subscribe( gaze.getBytes() ); //
Voilá.
Having zero-warranty what python version is running on the opposite side, tweaking may take place for string-representation matching...
( Also recommended to setup LINGER to 1, that prevents from hanging terminations
and preferably it is the best time to turn the process
into using a non-blocking .poll() + .recv( ..., ZMQ_DONTWAIT ) in a soft-realtime maintained event-loop )
[ 1 ] We have got confirmed the Android/ZeroMQ side is working fine
if the PUB-side was mocked by a plain python-PUB infinite-sender and the Android-SUB was subscribed to String filterPermitANY ="";
This makes the above claim "It's an issue from the android side" actually void if not misleading.
[ 2 ] Next comes the question why it still does not work?
And the answer is: because the above designed code does not follow the published principles, how to connect and use the Pupil Labs API.
A careful reader will notice that the Pupil Labs API is not connected by the SUB-side ( be it an Android or python or whatever else implementation of such a peer ) on a port :50020, but on another port, which is first asked about via another dialogue, held over an REQ/REP-formal communication archetype ( lines 13/14/15+19 ).
Epilogue
Knocking on a wrong door will never make the intended interview happen.
One first has to ask onto which door to knock next, so as to get the Pupil Labs API into the game.
I'm using Jetty 9.3.5 and I would like to know what is the proper way to handle unreliable connections when sending websocket messages, specifically: I noticed cases when a websocket connection does not close normally so, even though the client side is down, it takes a lot of time until onClose() is triggered on the server (for ex. a user closes the laptop lid and puts it in standby - it can take 1-2 hours until the close event is received on the server side).
Thus, because the client is still registered, the server keeps sending messages that begin to build up. This becomes an issue when sending a large number of messages.
I've tested sending byte messages with:
Session.getRemote().sendBytes(ByteBuffer, WriteCallback)
Session.getRemote().sendBytesByFuture(ByteBuffer);
To simulate the connection down on one side (ie. user puts laptop in standby), on Linux, I assigned an IP address to eth0 interface, started sending the messages and then brought it down:
ifconfig eth0 192.168.1.1
ifconfig eth0 up
--- start sending messages (simple incremented numbers) and connect using Chrome browser and print them ---
ifconfig eth0 down
This way: the messages were still being sent by Jetty, the Chrome client did not receive them, the onCllose or onError was not triggered on server-side
My questions regarding Jetty are:
Is there a way to clear queued messages that were not delivered?
I've tried, but with no luck:
Session.getRemote().flush();
Can a max number of queued messages be set?
I've tried:
WebSocketServletFactory.getPolicy().setMaxBinaryMessageBufferSize(1)
Can I detect if the client does not receive the message? (or if the connection is in abnormal state let's say)
I've tried:
session.getRemote().sendBytes(bb, new WriteCallback() {
#Override
public void writeSuccess() {
//print success }
#Override
public void writeFailed(Throwable arg0) {
//print fail
}
});
But this prints success even though the messages are not received.
I also tried to use, but couldn't find a solution:
factory.getPolicy().setIdleTimeout(...);
factory.getPolicy().setAsyncWriteTimeout(3000);
sendPing()
Thanks in advance!
Unfortunately, the WebSocket protocol, being a message passing protocol isn't really designed for this level of nuance between messages.
The first message MUST complete before you can even think of sending the next message. So if you have a message in process, then there is no way to safely cancel that message.
At best, an API could exist to truncate that message with a CONTINUATION / empty payload / fin=true.
But even then the remote endpoint wouldn't know that you canceled the message, it would just see a partial message.
Detecting connectivity issues is best handled with either OS level events (like Android's Connectivity intents), or via periodic websocket PING (which inserts itself in front of the line for outgoing websocket frames.
However, even with PING, if your outgoing websocket frame is in-progress, even the PING cannot be sent until that websocket frame is done sending.
RemoteEndpoint.flush() will attempt to flush any pending messages (and frames), not clear out pending messages (or frames).
As for detecting if client got the message, you'll need to implement some sort of message ACK into your own layer to verify that, the protocol has no such concept. (Some libs/apis built on top of websocket have implemented message ACK in that layer. The cometd message ack extension comes to mind as a real world example)
What sort of situation are you attempting to solve for?
Perhaps using the RemoteEndpoint.sendPartialString(String, boolean) or RemoteEndpoint.sendPartialBytes(ByteBuffer, boolean) to send smaller frames of the whole message could be useful to you. However, the other side might not have an API that can read those partial frames (eg: Javascript in a browser).
I´ve been looking for a good book or article about this topic but didnt find much. I didnt find a good example - piece of code - for a specific scenario. Like clients/server conversation.
In my application´s protocol they have to send/recieve messages. Like:
Server want to send a file to a client
Client can accpet or no,
if he accepts, server will send bytes over the same connection/socket.
The rest of my application all uses blocking methods, server has a method
Heres what I did:
Server method:
public synchronized void sendFile(File file)
{
//send messsage asking if I can send a file
//block on read, waiting for client responde
//if client answers yes, start sending the bytes
//else return
}
Client methods:
public void reciveCommand()
{
//read/listen for a command from socket
//if is a send file command handleSendFileCommand();
//after the return of handleSendFileCommand() listen for another command
}
public void handleSendFileCommand()
{
//get the file server want to send
//check if it already has the file
//if it already has, then send a command to the socket saying it already has and return
//else send a command saying server can send the file
//create a FileInputStream, recive bytes and then return method
}
I am 100% sure this is wrong because, there is no way server and clients would talk bidirecional, I mean, when server wants to send a command to a server, they have to follow an order of commands until that conversation is finished, only then, they can send/recive another sequence of commands. Thats why I made all methods that send requests synchronized
It didnt took me a lot of time to realize I need to study about design patterns for that kind of application...
I read about Chain of Responsibility design pattern but I dont get it how can I use it or another good design pattern in that situation.
I hope someone can help me with some code example-like.
Thanks in advance
synchronized keyword in Java means something completely different - it marks a method or a code block as a critical section that only single thread can execute at a time. You don't need it here.
Then, a TCP connection is bi-directional on the byte-stream level. The synchronization between the server and a client is driven by the messages exchanged. Think of a client (same pretty much applies to the server) as a state machine. Some types of messages are acceptable in the current state, some are not, some switch the node into different state.
Since you are looking into design patterns, the State pattern is very applicable here.
Salesforce can send up to 100 requests inside 1 SOAP message. While sending this type of Bulk Ooutbound message request my PHP script finishes executing but SF fails to accept the ACK used to clear the message queue on the Salesforce side of things. Looking at the Outbound message log (monitoring) I see all the messages in a pending state with the Delivery Failure Reason "java.net.SocketTimeoutException: Read timed out". If my script has finished execution, why do I get this error?
I have tried these methods to increase the execution time on my server as I have no access on the Salesforce side:
set_time_limit(0); // in the script
max_execution_time = 360 ; Maximum execution time of each script, in seconds
max_input_time = 360 ; Maximum amount of time each script may spend parsing request data
memory_limit = 32M ; Maximum amount of memory a script may consume
I used the high settings just for testing.
Any thoughts as to why this is failing the ACK delivery back to Salesforce?
Here is some of the code:
This is how I accept and send the ACK file for the imcoming SOAP request
$data = 'php://input';
$content = file_get_contents($data);
if($content) {
respond('true');
} else {
respond('false');
}
The respond function
function respond($tf) {
$ACK = <<<ACK
<?xml version = "1.0" encoding = "utf-8"?>
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<soapenv:Body>
<notifications xmlns="http://soap.sforce.com/2005/09/outbound">
<Ack>$tf</Ack>
</notifications>
</soapenv:Body>
</soapenv:Envelope>
ACK;
print trim($ACK);
}
These are in a generic script that I include into the script that uses the data for a specific workflow. I can process about 25 requests (That are in 1 SOAP response) but once I go over that I get the timeout error in the Salesforce queue. for 50 requests is usually takes my PHP script 86.77 seconds.
Could it be Apache? PHP?
I have also tested just accepting the 100 request SOAP response and just accepting and sending the ACK the queue clears out, so I know it's on my side of things.
I show no errors in the apache log, the script runs fine.
I did find some info on the Salesforce site but still no luck. Here is the link.
Also I'm using the PHP Toolkit 11 (From Salesforce).
Other forum with good SF help
Thanks for any insight into this,
--Phill
UPDATE:
If I receive the incoming message and print the response, should this happen first regardless if I do anything else after? Or does it wait for my process to finish and then print the response?
UPDATE #2:
okay I think I have the problem:
PHP uses the single thread processing approach and will not send back the ACK file until the thread has completed it's processing. Is there a way to make this a mutli thread process?
Thread #1 - accept the incoming SOAP request and send back the ACK
Thread #2 - Process the SOAP request
I know I could break it up into like a DB table or flat file, but is there a way to accomplish this without doing that?
I'm going to try to close the socket after the ACK submission and continue the processing, cross my fingers it will work.
Sounds like the outbound message is hitting the timeout. Other users have reported timeouts as low as 10 seconds (see forum link below). The sandbox instance that I use (cs1) is timing out after about 1 minute, from my testing. It's possible that the timeout is an organization or instance level setting that Salesforce controls.
Two things you could try:
Open a support ticket with
Salesforce to see if they can
increase the timeout value for
outbound messages. From my
experience, there are lot of
settings that they can modify on the
organization level - this might be
one of them.
Offload processing of your data, so
that the ACK is sent immediately
back to Salesforce. Then the actual
processing of your data will take
place asynchronously. ie. Message
queue, separate thread, etc.
Some other resources that might be helpful:
related Salesforce forum discussion
Outbound messaging documentation
I think they timeout the thing waiting for Your script to end.
There is a way You could try to fix this.
Output the envelope with ack message at the beginning and then flush the thing so that their server gets it before You end processing. No threading, just plain priorities rethinking :)
read this for best info on flushing content
Are you 100% sure that Salesforce will wait the amount of time your scripts need too run? 80 seconds seem like a loong time too me.
If all requests failed I would guess that Salesforce expects you to set the Content-Type header appropriately, but this does not seem to be the case.
I don't know about Salesforce, but if you want to make some multithreading with PHP you should take a look at this code example and more precisely to pcntl_fork().
N.B: pcntl is not enabled by default and won't work on Windows platforms.
So what I've done is:
Accept all incoming OBM's, parse them into a DB
When this is done kick of a process that runs in the background (Actually I send it to the background so the script can end)
Send ACK file back
By just accepting the raw data, parsing into fields and inserting it into a DB is fairly quick. Then I issue a Linux Command Line command that also send the processing script to run in the background. Then I send the ACK file to SF and the script ends within the allotted time. It is cumbersome to split the script process into two separate stages but it works.