Imagine a webapp which (sometimes) takes long time to respond to some HTTP (POST/GET/etc) request - how would You find such a request on server side?
So far I've used tomcat AccessLogValve to see the "completed" requests, but that doesn't let me to see the "in-progress" (stuck) ones :(
For example:
with netstat I'm able to identify long-lived sockets which could give me a count of currently-stuck requests (not URIs though), but HTTP keep-alives invalidate this approach
I could stackdump the app-server (kill -3 <server_pid>) multiple times and guess which threads are running long and reverse-engineer the URIs - not a smart way either
I could inject a router/proxy in front of web-app server (substitute hostnames, clone certs) which would show me the currently-running calls - not a simple approach
I could fall into just running tcpdump continously and parsing the traffic to keep list of currently-running URIs, but what to do with httpS then?
the closest I found is tomcat7's StuckThreadDetectionValve which'd periodically reports long-running calls, but it outputs the stacktrace (not URI) and doesn't provide "live" data (e.g. only polls periodically, floods the logs and lets to see the state of 1-60 seconds ago, but not "now")
Maybe I'm just missing/overlooking one of vital/core/basic tomcat features? or maybe weblogic (or any other app-server) has something robust to offer for this?
I'm kind of lost without such simple and essential feature.
Help? Please?
OK - creation of my own Valve was a proper and easy approach, sharing below. Apache did rework AccessLogValve multiple times, but all revisions follow same concept:
invoke(...) method just uses getNext().invoke(request,response) to invoke a chain of remaining valves and the actual handler/executor
log(...) method is invoked after above is complete
So we only need to:
also invoke log(...) before the getNext().invoke(request,response)
modify log(...) to distinguish "before" and "after" invocations
Easiest way would've been:
#Override
public void invoke(Request request, Response response) throws IOException, ServletException {
log(request, response, -1); // negative time indicates "before"
super.invoke(request, response);
}
But tomcat_6.0.16 code wasn't well-extendable, so I prefixed log-messages (in a hard-coded manner) with Thread.getName() and "before"/"after" indicator. Also I preferred to use reflection to access private AccessLogValve.getDate() :
package org.apache.catalina.valves;
import java.io.IOException;
import java.lang.reflect.Method;
import java.util.Date;
import javax.servlet.ServletException;
import org.apache.catalina.connector.Request;
import org.apache.catalina.connector.Response;
public class PreAccessLogValve extends AccessLogValve {
#Override
public void invoke(Request request, Response response) throws IOException, ServletException {
long timeStart = System.currentTimeMillis();
log(request, response, -timeStart); // negative time indicates "before" request
getNext().invoke(request, response);
log(request, response, System.currentTimeMillis() - timeStart); // actual (positive) - "after"
}
public void log(Request request, Response response, long time) {
if (started && getEnabled() && null != logElements && (null == condition || null == request.getRequest().getAttribute(condition))) {
StringBuffer result = new StringBuffer();
try {
Date date = (Date) methodGetDate.invoke(this);
for (int i = 0; i < logElements.length; i++) {
logElements[i].addElement(result, date, request, response, time);
}
} catch (Throwable t) { t.printStackTrace(); }
log(Thread.currentThread().getName() + (time<0?" > ":" < ") + result.toString());
}
}
private static final Method methodGetDate;
static {
Method m = null;
try {
m = AccessLogValve.class.getDeclaredMethod("getDate");
m.setAccessible(true);
} catch (Throwable t) { t.printStackTrace(); }
methodGetDate = m;
}
}
compiled above code with catalina.jar + servlet-api.jar and produced new catalina-my.jar, which was placed into tomcat/lib folder. After that - I've modified server.xml to have:
<Valve className="org.apache.catalina.valves.PreAccessLogValve"
directory="/tmp" prefix="test." suffix=".txt"
pattern="%a %t %m %U %s %b %D" resolveHosts="false" buffered="false"/>
Here's the sample output:
http-8007-exec-1 > 10.61.105.105 [18/Jan/2014:05:54:14 +0000] POST /admin/0$en_US/secure/enduser/search.do 200 - -1390024454470
http-8007-exec-5 > 10.61.105.105 [18/Jan/2014:05:54:17 +0000] GET /admin/0$en_US/secure/enduser/search.do 200 - -1390024457300
http-8007-exec-5 < 10.61.105.105 [18/Jan/2014:05:54:17 +0000] GET /admin/0$en_US/secure/enduser/search.do 200 13933 44
http-8007-exec-3 > 10.61.105.105 [18/Jan/2014:05:54:17 +0000] GET /admin/html/main.js 200 - -1390024457490
http-8007-exec-3 < 10.61.105.105 [18/Jan/2014:05:54:17 +0000] GET /admin/html/main.js 200 3750 0
http-8007-exec-5 > 10.61.105.105 [18/Jan/2014:05:54:17 +0000] GET /admin/images/layout/logo.gif 200 - -1390024457497
http-8007-exec-5 < 10.61.105.105 [18/Jan/2014:05:54:17 +0000] GET /admin/images/layout/logo.gif 200 1996 0
http-8007-exec-1 < 10.61.105.105 [18/Jan/2014:05:54:24 +0000] POST /admin/0$en_US/secure/enduser/search.do 200 13308 10209
This way all "in-progress" URIs can be easily retrieved at any moment:
[root#serv1 tomcat]# awk '{if(">"==$2){if($1 in S)print S[$1];S[$1]=$0}else delete S[$1]}END{for(i in S)print S[i]}' test
http-8007-exec-4 > 10.61.105.105 [18/Jan/2014:06:13:20 +0000] GET /admin/images/1x1blank.gif 200 - -13
http-8007-exec-2 > 10.61.105.105 [18/Jan/2014:06:13:16 +0000] POST /admin/servlet/handlersvr 200 - -13
Unfortunately, there is not a simple way to get a list of the in-flight HTTP requests that are taking a long time. As you mention, taking several thread dumps a few seconds apart will tell you which threads are performing the HTTP operations slowly (because the thread stack will be identical in each one that is waiting for the response). But, it doesn't tell you much more than that unless you can follow the code back to a static piece of code with the URL. But, you can take the thread dumps and identify the thread IDs, then take a heap dump and find those threads in the heap dump. While not straight-forward and definitely not simple, you can get the URL that is being used, how long it has been waiting, etc.
Related
I have written an application on Android which realises sending simply requests (using Volley) to the server. The server is stood up on the NodeMCU (ESP8266) microcontroller, written in Lua. The problem is, that after sending the request, application not always is able to print the response. If the address is e.g. "http://www.google.com" it correctly sends request and receive and display response, but if it is the address from the code below - it correctly sends request (the server reacts) but does not (?) receive response (does not display it, displays: "That didn't work!"). Do you have any ideas, how can I fix it and be able to print the response?
Android (part responsible for sending requests):
buttonSynchro.setOnClickListener(new View.OnClickListener() {
#Override
public void onClick(View view) {
// Instantiate the RequestQueue.
String url = "http://192.168.1.12/";
// Request a string response from the provided URL.
StringRequest stringRequest = new StringRequest(Request.Method.GET, url,
new Response.Listener<String>() {
#Override
public void onResponse(String response) {
// Display the first 500 characters of the response string.
testTextView.setText("Response is: "+ response.substring(0,500));
}
}, new Response.ErrorListener() {
#Override
public void onErrorResponse(VolleyError error) {
testTextView.setText("That didn't work!");
}
});
// Add the request to the RequestQueue.
RequestQueue queue = Volley.newRequestQueue(SettingsActivity.this);
queue.add(stringRequest);
}
});
NodeMCU, Lua:
station_cfg={}
station_cfg.ssid="Dom"
station_cfg.pwd="lalala"
wifi.sta.config(station_cfg)
function receive(conn, request)
print(request)
print()
local buf = "";
buf = buf.."<!doctype html><html>";
buf = buf.."<h1> ESP8266 Web Server</h1>";
buf = buf.."</html>";
conn:send(buf);
conn:on("sent", function(sck) sck:close() end);
collectgarbage();
end
function connection(conn)
conn:on("receive", receive)
end
srv=net.createServer(net.TCP, 30)
srv:listen(80, connection)
The code by nPn works in some user agents (Chrome/Firfox/curl/wget on macOS) but not in others (Safari on macOS & iOS, Firefox Klar on iOS). That likely is due to missing HTTP headers.
I advise you stick to the example we have in our documentation at https://nodemcu.readthedocs.io/en/latest/en/modules/net/#netsocketsend.
srv = net.createServer(net.TCP)
function receiver(sck, data)
print(data)
print()
-- if you're sending back HTML over HTTP you'll want something like this instead
local response = {"HTTP/1.0 200 OK\r\nServer: NodeMCU on ESP8266\r\nContent-Type: text/html\r\n\r\n"}
response[#response + 1] = "<!doctype html><html>"
response[#response + 1] = "<h1> ESP8266 Web Server</h1>"
response[#response + 1] = "</html>"
-- sends and removes the first element from the 'response' table
local function send(localSocket)
if #response > 0 then
localSocket:send(table.remove(response, 1))
else
localSocket:close()
response = nil
end
end
-- triggers the send() function again once the first chunk of data was sent
sck:on("sent", send)
send(sck)
end
srv:listen(80, function(conn)
conn:on("receive", receiver)
end)
Also, your code (and nPn's for that matter) makes assumptions about WiFi being available where it shouldn't.
wifi.sta.config(station_cfg) (with auto-connect=true) and wifi.stat.connect are asynchronous and thus non-blocking - as are many other NodeMCU APIs. Hence, you should put the above code into a function and only call it once the device is connected to the AP and got an IP. You do that by e.g. registering a callback for the STA_GOT_IP event with the WiFi event monitor. You'll find a very elaborate example of a boot sequence that listens to all WiFi events at https://nodemcu.readthedocs.io/en/latest/en/upload/#initlua. For starters you may want to trim this and only listen for got-IP.
Based on your comment above and the link you posted showing the traceback, your android app is crashing in the onResponse() method because you are asking for a substring longer than the actual string length.
You can fix this in a number of ways, but one would be to make the ending index be the minimum of the length of the response and 500 (which I assume is the max you can take in your TextView?). You can try changing
testTextView.setText("Response is: "+ response.substring(0,500));
to
testTextView.setText("Response is: "+ response.substring(0, Math.min(response.length(), n)));
or whatever other way you think is more appropriate to limit the length of the response that does not cause the IndexOutOfBoundsException
See the substring method here
public String substring(int beginIndex,
int endIndex)
Returns a new string that is a substring of this string. The substring
begins at the specified beginIndex and extends to the character at
index endIndex - 1. Thus the length of the substring is
endIndex-beginIndex.
Examples:
"hamburger".substring(4, 8) returns "urge"
"smiles".substring(1, 5) returns "mile"
Parameters:
beginIndex - the beginning index, inclusive.
endIndex - the ending index, exclusive. Returns:
the specified substring. Throws:
IndexOutOfBoundsException - if the beginIndex is negative, or endIndex is larger than the length of this String object, or
beginIndex is larger than endIndex.
I am not a Lua expert, but I think you are registering your "sent" callback after you send the response.
I think you should move it into the connection function:
station_cfg={}
station_cfg.ssid="Dom"
station_cfg.pwd="lalala"
wifi.sta.config(station_cfg)
function receive(conn, request)
print(request)
print()
local buf = "";
buf = buf.."<!doctype html><html>";
buf = buf.."<h1> ESP8266 Web Server</h1>";
buf = buf.."</html>";
conn:send(buf);
collectgarbage();
end
function connection(conn)
conn:on("receive", receive)
conn:on("sent", function(sck) sck:close() end);
end
srv=net.createServer(net.TCP, 30)
srv:listen(80, connection)
I have tried to implement range request video playback on a system that has webservlet and UI, that sends the range request from chrome starting with bytes :0- to the backend dataserver. Now I have been sending the full stream as I was under the impression the jetty server handles the response range. I see it work for the first and second request but then fails as the next request has a range that is less than what the previous range was.
(Request 1) Range:bytes=0-
(Response 1) Accept-Ranges:bytes
Content-Length:6748748
Content-Range:bytes 0-10005/6748748
(Request 2) Range:bytes=6717440-
(Response 2) Accept-Ranges:bytes
Content-Length:6748748
Content-Range:bytes 6717440-6718465/6748748
(Request 3) Range:bytes=3932160-
(Response 3) Accept-Ranges:bytes
Content-Length:6748748
Content-Range:bytes 3932160-3933185/6748748
(Request 4) Range:bytes=5701632-
(Response 4) Fails -
Can anyone make sense of this? With short videos this does not occur, so is there some timeout issue but then why is the chrome request with a smaller range? This is what I specify in the headers but again do not explicitly send the bytes as requested as I thought jetty handles it.
if(inputStream != null) {
if (parameters.containsKey("Range")) {
String range =parameters.get("Range").toString();
String[] ranges = range.split("=")[1].split("-");
final int from = Integer.parseInt(ranges[0]);
if(parameters.containsKey("Content-Length")) {
int sLength = (int) parameters.get("Content-Length");
int to = 10005 + from;
if (to >= sLength) {
to = (int) (sLength - 1);
}
if (ranges.length == 2) {
to = Integer.parseInt(ranges[1]);
}
final String responseRange = String.format("bytes %d-%d/%d", from, to, sLength);
parameters.put("Responserange", responseRange);
}
}
EDIT:
My logs in the dataserver side show the following consistently with each request being handled:
java.nio.channels.ClosedChannelException,
Added additional stack trace
java.nio.channels.ClosedChannelException
at
org.eclipse.jetty.util.IteratingCallback.close(IteratingCallback.java:427)
at org.eclipse.jetty.server.HttpConnection.onClose(HttpConnection.java:489)
at org.eclipse.jetty.io.ssl.SslConnection.onClose(SslConnection.java:217)
We're having some trouble trying to implement a Pool of SftpConnections for our application.
We're currently using SSHJ (Schmizz) as the transport library, and facing an issue we simply cannot simulate in our development environment (but the error keeps showing randomly in production, sometimes after three days, sometimes after just 10 minutes).
The problem is, when trying to send a file via SFTP, the thread gets locked in the init method from schmizz' TransportImpl class:
#Override
public void init(String remoteHost, int remotePort, InputStream in, OutputStream out)
throws TransportException {
connInfo = new ConnInfo(remoteHost, remotePort, in, out);
try {
if (config.isWaitForServerIdentBeforeSendingClientIdent()) {
receiveServerIdent();
sendClientIdent();
} else {
sendClientIdent();
receiveServerIdent();
}
log.info("Server identity string: {}", serverID);
} catch (IOException e) {
throw new TransportException(e);
}
reader.start();
}
isWaitForServerIdentBeforeSendingClientIdent is FALSE for us, so first of all the client (we) send our identification, as appears in logs:
"Client identity String: blabla"
Then it's turn for the receiveServerIdent:
private void receiveServerIdent() throws IOException
{
final Buffer.PlainBuffer buf = new Buffer.PlainBuffer();
while ((serverID = readIdentification(buf)).isEmpty()) {
int b = connInfo.in.read();
if (b == -1)
throw new TransportException("Server closed connection during identification exchange");
buf.putByte((byte) b);
}
}
The thread never gets the control back, as the server never replies with its identity. Seems like the code is stuck in this While loop. No timeouts, or SSH exceptions are thrown, my client just keeps waiting forever, and the thread gets deadlocked.
This is the readIdentification method's impl:
private String readIdentification(Buffer.PlainBuffer buffer)
throws IOException {
String ident = new IdentificationStringParser(buffer, loggerFactory).parseIdentificationString();
if (ident.isEmpty()) {
return ident;
}
if (!ident.startsWith("SSH-2.0-") && !ident.startsWith("SSH-1.99-"))
throw new TransportException(DisconnectReason.PROTOCOL_VERSION_NOT_SUPPORTED,
"Server does not support SSHv2, identified as: " + ident);
return ident;
}
Seems like ConnectionInfo's inputstream never gets data to read, as if the server closed the connection (even if, as said earlier, no exception is thrown).
I've tried to simulate this error by saturating the negotiation, closing sockets while connecting, using conntrack to kill established connections while the handshake is being made, but with no luck at all, so any help would be HIGHLY appreciated.
: )
I bet following code creates a problem:
String ident = new IdentificationStringParser(buffer, loggerFactory).parseIdentificationString();
if (ident.isEmpty()) {
return ident;
}
If the IdentificationStringParser.parseIdentificationString() returns empty string, it will be returned to the caller method. The caller method will keep calling the while ((serverID = readIdentification(buf)).isEmpty()) since the string is always empty. The only way to break the loop would be if call to int b = connInfo.in.read(); returns -1... but if server keeps sending the data (or resending the data) this condition is never met.
If this is the case I would add some kind of artificial way to detect this like:
private String readIdentification(Buffer.PlainBuffer buffer, AtomicInteger numberOfAttempts)
throws IOException {
String ident = new IdentificationStringParser(buffer, loggerFactory).parseIdentificationString();
numberOfAttempts.incrementAndGet();
if (ident.isEmpty() && numberOfAttempts.intValue() < 1000) { // 1000
return ident;
} else if (numberOfAttempts.intValue() >= 1000) {
throw new TransportException("To many attempts to read the server ident").
}
if (!ident.startsWith("SSH-2.0-") && !ident.startsWith("SSH-1.99-"))
throw new TransportException(DisconnectReason.PROTOCOL_VERSION_NOT_SUPPORTED,
"Server does not support SSHv2, identified as: " + ident);
return ident;
}
This way you would at least confirm that this is the case and can dig further why .parseIdentificationString() returns empty string.
Faced a similar issue where we would see:
INFO [net.schmizz.sshj.transport.TransportImpl : pool-6-thread-2] - Client identity string: blablabla
INFO [net.schmizz.sshj.transport.TransportImpl : pool-6-thread-2] - Server identity string: blablabla
But on some occasions, there were no server response.
Our service would typically wake up and transfer several files simultaneously, one file per connection / thread.
The issue was in the sshd server config, we increased maxStartups from default value 10
(we noticed the problems started shortly after batch sizes increased to above 10)
Default in /etc/ssh/sshd_config:
MaxStartups 10:30:100
Changed to:
MaxStartups 30:30:100
MaxStartups
Specifies the maximum number of concurrent unauthenticated connections to the SSH daemon. Additional connections will be dropped until authentication succeeds or the LoginGraceTime expires for a connection. The default is 10:30:100. Alternatively, random early drop can be enabled by specifying the three colon separated values start:rate:full (e.g. "10:30:60"). sshd will refuse connection attempts with a probability of rate/100 (30%) if there are currently start (10) unauthenticated connections. The probability increases linearly and all connection attempts are refused if the number of unauthenticated connections reaches full (60).
If you cannot control the server, you might have to find a way to limit your concurrent connection attempts in your client code instead.
We have a problem. Our customers are complaining that they are getting duplicate emails in their in-box. Some days up to 5 or 6 instances of the exact same email at the exact same time. We don't understand why. The code has been re-written at least once but the problem persists.
I'll try to explain this... but it's a bit complicated :O(
Every night (early morning) we want to send our users a daily report containing usage stats. So we have a cron job:
<cron>
<url>/redacted/report/url</url>
<description>Send out daily reports to active subscribers</description>
<schedule>every 2 hours</schedule>
</cron>
The cron job hits the servlet get method:
protected void doGet(HttpServletRequest req, HttpServletResponse resp) throws ServletException, IOException {
AccountFilter filter = AccountFilter.forWebSafeName(req.getParameter("filter"));
createTasks(filter, null);
}
Which calls the createTasks method with a null cursor:
private void createTasks(AccountFilter accountFilter, String cursor) {
try {
PagedResults<Account> pagedAccounts = accountRepository.getAccounts(accountFilter.getFilter(), 50, cursor);
createTaskBatch(pagedAccounts);
// If there are still more results in cursor, then send cursor back to this servlet's doPost method so we don't hit the request time limit
if (pagedAccounts.getCursor() != null) {
getQueue(QUEUE_NAME).add(withUrl(WORKER_URL).param(CURSOR_KEY, pagedAccounts.getCursor()).param(FILTER_KEY, accountFilter.getWebSafeName()));
}
} catch(Exception ex) {
logger.log(Level.WARNING, "Problem creating daily report task batch for filter " + accountFilter.getWebSafeName(), ex);
}
}
which grabs 50 accounts and iterates over them creating new queued jobs for the emails that should be sent at this time. There is code to explcitely check the last report sent timestamp and update the timestamp BEFORE creating the new queued task. This should err on the side of not sending the report rather than sending duplicates:
private void createTaskBatch(PagedResults<Account> pagedAccounts) {
// GAE datastore query might return duplicate results?!
List<Account> list = pagedAccounts.getResults();
Set<Account> noDuplicates = new HashSet<>(list);
int dups = list.size() - noDuplicates.size();
if ( dups > 0 ){
logger.warning ("Accounts paged results contained " + dups + " duplicates!");
}
for (Account account : noDuplicates) {
try {
if (lastReportSentOver12HoursAgo(account)) {
List<Parent> parents = parentRepository.getVerifiedParentsForAccount(account.getId());
if (eitherParentSubscribed(parents)) {
List<AccountUser> users = accountUserRepository.listUsers(account.getId());
List<Device> devices = getUserDevices(account, users);
if (!devices.isEmpty()) {
DateTimeZone tz = getMostCommonTimezone(devices);
if ( null == tz ){
logger.warning("No timezone found for account: " + account.getId() );
}
else{
// Send early in the morning as the report contains the previous day's stats
if (now(tz).getHourOfDay() < 7) {
// mark sent now because queue might not be processed for a while
// and the next cursor set might contain some of the same accounts
accountRepository.markReportSent(account.getId(), now());
getQueue(QUEUE_NAME).add(withUrl(DailyReportServlet.WORKER_URL).param(DailyReportServlet.ACCOUNT_ID, account.getId()).param(DailyReportServlet.COMMON_TIMEZONE, tz.getID()));
}
}
}
}
}
} catch(Exception ex) {
logger.log(Level.WARNING, "Problem creating daily report task for " + account.getId(), ex);
}
}
}
The servlet POST method takes care of handling the follow up pages of results via the cursor method:
public void doPost(HttpServletRequest req, HttpServletResponse resp) throws IOException {
AccountFilter accountFilter = AccountFilter.forWebSafeName(req.getParameter(FILTER_KEY));
logger.log(Level.INFO, "doPost hit from task queue with filter " + accountFilter.getWebSafeName());
String cursor = req.getParameter(CURSOR_KEY);
createTasks(accountFilter, cursor);
}
There is another servlet that handles each report task and it just creates the email contents and calls send on the com.sendgrid.SendGrid class.
The eventual consistency in Datastore seems a likely candidate but that should be resolved within a few seconds and I don't see how that would account for both the number of customers complaining and the number of duplicates that some customers see.
Help! Any ideas? Are we being dumb somewhere?
UPDATED
For clarity... the email send task queue ends up in this method which does catch exceptions and reports them back to us. We don't see an exception for the duplicate cases:
private void sendReport(Account account, DateTimeZone tz) throws IOException, EntityNotFoundException {
try {
boolean sent = false;
Map<String, Object> root = buildEmailData(account, tz);
for (Parent parent : parentRepository.getVerifiedParentsForAccount(account.getId())) {
if (parent.getEmailPreferences().isSubscribedReports()) {
emailBuilder.send(account, parent, root, "report", EmailSender.NOTIFICATION);
sent = true;
}
}
if ( sent ){
accountRepository.markReportSent(account.getId(), now());
}
} catch (Exception ex) {
String message = "Problem building report email for account " + account.getId();
logger.log(Level.WARNING, message, ex);;
new TeamNotificationEvent( message + " : exception: " + ex.getMessage()).fire();
throw new IOException(message, ex);
}
}
UPDATE 2 AFTER ADDING EXTRA DEBUG LOGGING
I see two POSTS in at the same time to the same task queue with the same cursor:
09:35:08.397 2015-04-30 200 0 B 3.78s /ws/notification/daily-report-task-creator
0.1.0.2 - - [30/Apr/2015:01:35:08 -0700] "POST /ws/notification/daily-report-task-creator HTTP/1.1" 200 0 "http://screentimelabs.appspot.com/ws/notification/daily-report-task-creator" "AppEngine-Google; (+http://code.google.com/appengine)" "screentimelabs.appspot.com" ms=3782 cpu_ms=662 queue_name=dailyReports task_name=8168414365365326983 instance=00c61b117c33a909790f0d1882657e04f40b2c7e app_engine_release=1.9.20
09:35:04.618 com.screentime.service.taskqueue.reports.DailyReportTaskCreatorServlet createTasks: createTasks called for filter: ACTIVE with cursor: E-ABAIICO2oQc35zY3JlZW50aW1lbGFic3InCxIHQWNjb3VudCIaamFybW8ua2Fya2thaW5lbkBnbWFpbC5jb20MiAIAFA
09:35:08.432 2015-04-30 200 0 B 8.84s /ws/notification/daily-report-task-creator
0.1.0.2 - - [30/Apr/2015:01:35:08 -0700] "POST /ws/notification/daily-report-task-creator HTTP/1.1" 200 0 "http://screentimelabs.appspot.com/ws/notification/daily-report-task-creator" "AppEngine-Google; (+http://code.google.com/appengine)" "screentimelabs.appspot.com" ms=8837 cpu_ms=1348 queue_name=dailyReports task_name=50170612326424582061 instance=00c61b117c2bffe8de313e96fea8aeb813f4b20f app_engine_release=1.9.20 trace_id=7e5c0348382e66cf4e2c6ba400529fb7
09:34:59.608 com.screentime.service.taskqueue.reports.DailyReportTaskCreatorServlet createTasks: createTasks called for filter: ACTIVE with cursor: E-ABAIICO2oQc35zY3JlZW50aW1lbGFic3InCxIHQWNjb3VudCIaamFybW8ua2Fya2thaW5lbkBnbWFpbC5jb20MiAIAFA
Searching for 1 particular account id I see these requests:
09:35:08.397 2015-04-30 200 0 B 3.78s /ws/notification/daily-report-task-creator
09:35:08.432 2015-04-30 200 0 B 8.84s /ws/notification/daily-report-task-creator
09:35:08.443 2015-04-30 200 0 B 6.73s /ws/notification/daily-report-task-creator
09:35:10.541 2015-04-30 200 0 B 4.03s /ws/notification/daily-report-task-creator
09:35:10.690 2015-04-30 200 0 B 11.09s /ws/notification/daily-report-task-creator
09:35:13.678 2015-04-30 200 0 B 862ms /ws/notification/daily-report-worker
09:35:13.829 2015-04-30 500 0 B 1.21s /ws/notification/daily-report-worker
09:35:14.677 2015-04-30 200 0 B 1.56s /ws/notification/daily-report-worker
09:35:14.961 2015-04-30 200 0 B 346ms /ws/notification/daily-report-worker
Some have repeated cursor values.
I will make a guess because i dont see the task queue code. Its likely that you are not handling errors correctly in the task queue. If a task queue finishes with an error, gae will re-queue it. thus if some emails were already sent, the task will still run again. you need a way to remember what you already processed in the task queue so a retry wont reprocess those.
I have configured the analog local phone with cisco adapter, so I can make any outbound call from SIP phone. But I can't achieve this by AMI which calls to outbound channel through trunk then plays prompt.
manager.conf:
[asteriskjava]
secret = asteriskjava
deny = 0.0.0.0/0.0.0.0
permit = 127.0.0.1/255.255.255.0
read = all
write = all
extensions.conf:
[bulk]
exten => 8,1,Playback(thank-you-cooperation)
exten => h,1,Hangup
source code:
public class HelloManager
{
private ManagerConnection managerConnection;
public HelloManager() throws IOException
{
ManagerConnectionFactory factory = new ManagerConnectionFactory(
"localhost", "asteriskjava", "asteriskjava");
this.managerConnection = factory.createManagerConnection();
}
public void run() throws IOException, AuthenticationFailedException,
TimeoutException
{
OriginateAction originateAction;
ManagerResponse originateResponse;
originateAction = new OriginateAction();
originateAction.setChannel("SIP/405/7000000");
originateAction.setContext("bulk");
originateAction.setExten("8");
originateAction.setPriority(new Integer(1));
originateAction.setAsync(true);
// connect to Asterisk and log in
managerConnection.login();
// send the originate action and wait for a maximum of 30 seconds for Asterisk
// to send a reply
originateResponse = managerConnection.sendAction(originateAction, 30000);
// print out whether the originate succeeded or not
System.out.println("---" + originateResponse.getResponse());
// and finally log off and disconnect
managerConnection.logoff();
}
}
Where 405 is the UserID of CISCO adapter for outgoing calls, 7000000 is a sample cell phone number.
Here is the logs:
== Manager 'asteriskjava' logged on from 127.0.0.1
== Manager 'asteriskjava' logged off from 127.0.0.1
== Using SIP RTP CoS mark 5
> Channel SIP/405-0000000c was answered.
-- Executing [8#bulk:1] Playback("SIP/405-0000000c", "thank-you-cooperation") in new stack
-- <SIP/405-0000000c> Playing 'thank-you-cooperation.gsm' (language 'en')
-- Auto fallthrough, channel 'SIP/405-0000000c' status is 'UNKNOWN'
-- Executing [h#bulk:1] Hangup("SIP/405-0000000c", "") in new stack
== Spawn extension (bulk, h, 1) exited non-zero on 'SIP/405-0000000c'
I think SIP/405 is answering, executing Playback then hangs up, not redirecting to sample number.
Any suggestions?
EDIT: How can I configure my cisco adapter in order to redirect outgoing calls, not to answer and make the bridge?
You have configure ring, answer and busy recognition on your ATA.
Asterisk work as you requested as far as i can see from your trace.
If adapter not calling, you have check with your adapater settings. For example it can be calling in tone, why you line expect it is pulse.
Also can be incorrect adapter type for your task. For calling out via PSTN line you need FXO adapter,not FXS.