We Keep Coding

Hystrix circuitBreaker.sleepWindowInMilliseconds not working

I'm having a spring boot application which is calling iteratively a mockserver instance via a hystrix command, with a fallback method.
The mockserver is configured to allways respond with status code 500. When running without having circuitBreaker.sleepWindowInMilliseconds, everything works fine, the call is done to the mockserver and then the fallback method is invoked.
After configuring circuitBreaker.sleepWindowInMilliseconds value to 5 minutes or so, I would expect that during 5 minutes no calls are done to the mockserver all the calls being directed to the fallback method, but that's not the case.
It looks like the circuitBreaker.sleepWindowInMilliseconds configuration is ignored.
For instance if I reconfigure the mockservice to reply with status code 200 while the iteration is still running, it would imediately print the "mockservice response",without waiting 5 minutes.
in the spring boot main application class:
#RequestMapping("/iterate")
public void iterate() {
for (int i = 1; i<100; i++ ) {
try {
System.out.println(bookService.readingMockService());
Thread.sleep(3000);
} catch (Exception e) {
System.out.println(e.getMessage());
}
}
}
in the spring boot service :
#HystrixCommand(groupKey = "ReadingMockService", commandKey = "ReadingMockService", threadPoolKey = "ReadingMockService", fallbackMethod = "reliableMock", commandProperties = {
#HystrixProperty(name ="circuitBreaker.sleepWindowInMilliseconds", value = "300000") })
public String readingMockService() {
URI uri = URI.create("http://localhost:1080/askmock");
return this.restTemplate.getForObject(uri, String.class);
}
also the mock server is running on the same machine, being configured like :
new MockServerClient("127.0.0.1", 1080).reset();
new MockServerClient("127.0.0.1", 1080)
.when(request("/askmock"))
.respond(response()
.withStatusCode(500)
.withBody("mockservice response")
.applyDelay());

Found the problem :
This property (...circuitBreaker.sleepWindowInMilliseconds ) works together with another one (...circuitBreaker.requestVolumeThreshold ).
If not specifically set this defaults to 20, meaning that first hystrix will try to connect 20 times the usual way and only afterwards the sleepWindowInMilliseconds will get activated and will go to fallback only.
Also the circuit break opens only if the percentage of failed calls exceeds circuitBreaker.errorThresholdPercentage
and in the same time the total number of failed calls exceeds circuitBreaker.requestVolumeThreshold, all within a window of metrics.rollingStats.timeInMilliseconds

From the docs:
https://github.com/Netflix/Hystrix/wiki/configuration#circuitBreaker.sleepWindowInMilliseconds
and by looking at the source code:
https://github.com/Netflix/Hystrix/blob/master/hystrix-core/src/main/java/com/netflix/hystrix/HystrixCommandProperties.java
Using
#HystrixProperty(name="hystrix.command.ReadingMockService.circuitBreaker.sleepWindowInMilliseconds"
should work.

SSH Server Identification never received - Handshake Deadlock [SSHJ]

We're having some trouble trying to implement a Pool of SftpConnections for our application.
We're currently using SSHJ (Schmizz) as the transport library, and facing an issue we simply cannot simulate in our development environment (but the error keeps showing randomly in production, sometimes after three days, sometimes after just 10 minutes).
The problem is, when trying to send a file via SFTP, the thread gets locked in the init method from schmizz' TransportImpl class:
#Override
public void init(String remoteHost, int remotePort, InputStream in, OutputStream out)
throws TransportException {
connInfo = new ConnInfo(remoteHost, remotePort, in, out);
try {
if (config.isWaitForServerIdentBeforeSendingClientIdent()) {
receiveServerIdent();
sendClientIdent();
} else {
sendClientIdent();
receiveServerIdent();
}
log.info("Server identity string: {}", serverID);
} catch (IOException e) {
throw new TransportException(e);
}
reader.start();
}
isWaitForServerIdentBeforeSendingClientIdent is FALSE for us, so first of all the client (we) send our identification, as appears in logs:
"Client identity String: blabla"
Then it's turn for the receiveServerIdent:
private void receiveServerIdent() throws IOException
{
final Buffer.PlainBuffer buf = new Buffer.PlainBuffer();
while ((serverID = readIdentification(buf)).isEmpty()) {
int b = connInfo.in.read();
if (b == -1)
throw new TransportException("Server closed connection during identification exchange");
buf.putByte((byte) b);
}
}
The thread never gets the control back, as the server never replies with its identity. Seems like the code is stuck in this While loop. No timeouts, or SSH exceptions are thrown, my client just keeps waiting forever, and the thread gets deadlocked.
This is the readIdentification method's impl:
private String readIdentification(Buffer.PlainBuffer buffer)
throws IOException {
String ident = new IdentificationStringParser(buffer, loggerFactory).parseIdentificationString();
if (ident.isEmpty()) {
return ident;
}
if (!ident.startsWith("SSH-2.0-") && !ident.startsWith("SSH-1.99-"))
throw new TransportException(DisconnectReason.PROTOCOL_VERSION_NOT_SUPPORTED,
"Server does not support SSHv2, identified as: " + ident);
return ident;
}
Seems like ConnectionInfo's inputstream never gets data to read, as if the server closed the connection (even if, as said earlier, no exception is thrown).
I've tried to simulate this error by saturating the negotiation, closing sockets while connecting, using conntrack to kill established connections while the handshake is being made, but with no luck at all, so any help would be HIGHLY appreciated.
: )

I bet following code creates a problem:
String ident = new IdentificationStringParser(buffer, loggerFactory).parseIdentificationString();
if (ident.isEmpty()) {
return ident;
}
If the IdentificationStringParser.parseIdentificationString() returns empty string, it will be returned to the caller method. The caller method will keep calling the while ((serverID = readIdentification(buf)).isEmpty()) since the string is always empty. The only way to break the loop would be if call to int b = connInfo.in.read(); returns -1... but if server keeps sending the data (or resending the data) this condition is never met.
If this is the case I would add some kind of artificial way to detect this like:
private String readIdentification(Buffer.PlainBuffer buffer, AtomicInteger numberOfAttempts)
throws IOException {
String ident = new IdentificationStringParser(buffer, loggerFactory).parseIdentificationString();
numberOfAttempts.incrementAndGet();
if (ident.isEmpty() && numberOfAttempts.intValue() < 1000) { // 1000
return ident;
} else if (numberOfAttempts.intValue() >= 1000) {
throw new TransportException("To many attempts to read the server ident").
}
if (!ident.startsWith("SSH-2.0-") && !ident.startsWith("SSH-1.99-"))
throw new TransportException(DisconnectReason.PROTOCOL_VERSION_NOT_SUPPORTED,
"Server does not support SSHv2, identified as: " + ident);
return ident;
}
This way you would at least confirm that this is the case and can dig further why .parseIdentificationString() returns empty string.

Faced a similar issue where we would see:
INFO [net.schmizz.sshj.transport.TransportImpl : pool-6-thread-2] - Client identity string: blablabla
INFO [net.schmizz.sshj.transport.TransportImpl : pool-6-thread-2] - Server identity string: blablabla
But on some occasions, there were no server response.
Our service would typically wake up and transfer several files simultaneously, one file per connection / thread.
The issue was in the sshd server config, we increased maxStartups from default value 10
(we noticed the problems started shortly after batch sizes increased to above 10)
Default in /etc/ssh/sshd_config:
MaxStartups 10:30:100
Changed to:
MaxStartups 30:30:100
MaxStartups
Specifies the maximum number of concurrent unauthenticated connections to the SSH daemon. Additional connections will be dropped until authentication succeeds or the LoginGraceTime expires for a connection. The default is 10:30:100. Alternatively, random early drop can be enabled by specifying the three colon separated values start:rate:full (e.g. "10:30:60"). sshd will refuse connection attempts with a probability of rate/100 (30%) if there are currently start (10) unauthenticated connections. The probability increases linearly and all connection attempts are refused if the number of unauthenticated connections reaches full (60).
If you cannot control the server, you might have to find a way to limit your concurrent connection attempts in your client code instead.

How to use JCo connection without creating *.JcoDestination file

I'm trying to connect to SAP ECC 6.0 using JCo. I'm following this tutorial. However, there is a Note saying:
For this example the destination configuration is stored in a file that is called by the program. In practice you should avoid this for security reasons.
And that is reasonable and understood. But, there is no explenation how to set up secure destination provider.
I found solution in this thread that created custom implementation of DestinationDataProvider and that works on my local machine. But when I deploy it on Portal I get an error saying that there is already registered DestinationDataProvider.
So my question is:
How to store destination data in SAP Java EE application?
Here is my code to further clarify what I'm trying to do.
public static void main(String... args) throws JCoException {
CustomDestinationProviderMap provider = new CustomDestinationProviderMap();
com.sap.conn.jco.ext.Environment.registerDestinationDataProvider(provider);
Properties connectProperties = new Properties();
connectProperties.setProperty(DestinationDataProvider.JCO_ASHOST, "host.sap.my.domain.com");
connectProperties.setProperty(DestinationDataProvider.JCO_SYSNR, "00");
connectProperties.setProperty(DestinationDataProvider.JCO_CLIENT, "100");
connectProperties.setProperty(DestinationDataProvider.JCO_USER, "user");
connectProperties.setProperty(DestinationDataProvider.JCO_PASSWD, "password");
connectProperties.setProperty(DestinationDataProvider.JCO_LANG, "en");
provider.addDestination(DESTINATION_NAME1, connectProperties);
connect();
}
public static void connect() throws JCoException {
String FUNCTION_NAME = "BAPI_EMPLOYEE_GETDATA";
JCoDestination destination = JCoDestinationManager.getDestination(DESTINATION_NAME1);
JCoContext.begin(destination);
JCoFunction function = destination.getRepository().getFunction(FUNCTION_NAME);
if (function == null) {
throw new RuntimeException(FUNCTION_NAME + " not found in SAP.");
}
//function.getImportParameterList().setValue("EMPLOYEE_ID", "48");
function.getImportParameterList().setValue("FSTNAME_M", "ANAKIN");
function.getImportParameterList().setValue("LASTNAME_M", "SKYWALKER");
try {
function.execute(destination);
} catch (AbapException e) {
System.out.println(e.toString());
return;
}
JCoTable table = function.getTableParameterList().getTable("PERSONAL_DATA");
for (int i = 0; i < table.getNumRows(); i++) {
table.setRow(i);
System.out.println(table.getString("PERNO") + '\t' + table.getString("FIRSTNAME") + '\t' + table.getString("LAST_NAME")
+'\t' + table.getString("BIRTHDATE")+'\t' + table.getString("GENDER"));
}
JCoContext.end(destination);
}

Ok, so I got this up and going and thought I'd share my research.
You need to add your own destination in Portal. To achieve that you need to go to NetWeaver Administrator, located at: host:port/nwa. So it'll be something like sapportal.your.domain.com:50000/nwa.
Then you go to Configuration-> Infrastructure-> Destinations and add your destination there. You can leave empty most of the fields like Message Server. The important part is Destination name as it is how you will retrieve it and destination type which should be set to RFC Destination in my case. Try pinging your newly created destination to check if its up and going.
Finally you should be able to get destination by simply calling: JCoDestination destination = JCoDestinationManager.getDestination(DESTINATION_NAME); as it is added to your Portal environment and managed from there.

Take a look at the CustomDestinationDataProvider in the JCo examples of the Jco connector download. The important parts are:
static class MyDestinationDataProvider implements DestinationDataProvider
...
com.sap.conn.jco.ext.Environment.registerDestinationDataProvider(new MyDestinationDataProvider());
Then you can simply do:
instance = JCoDestinationManager.getDestination(DESTINATION_NAME);
Btw. you may also want to check out http://hibersap.org/ as they provide nice ways to store the config as well.

More info about save in GORM

I have a simple block of code, that return false
PromoDebit promoDebit = new PromoDebit();
promoDebit.promoCode=promoCode;
promoDebit.userId=userId;
promoDebit.countUsages=countUsages;
promoDebit.endDate=endDate;
promoDebit.startDate=startDate;
promoDebit.status=1;
promoDebit.calcValue=Float.parseFloat(p.getProperty("promoPercent"));
if(promoDebit.save(flush: true)){
log.info "GOOD!"
} else {
log.info "BAD!"
}
How can I get more info about GORM.save fail? Stdrout log show nothning even if I turn on logSql in DataSource.groovy

First ,make sure that your Log4j is configured correctly.
Here http://grails.org/doc/2.3.1/guide/conf.html#logging
And ,If you wan to know more detail info on GORM.save fail like
PromoDebit promoDebit = new PromoDebit();
promoDebit.promoCode=promoCode;
promoDebit.userId=userId;
promoDebit.countUsages=countUsages;
promoDebit.endDate=endDate;
promoDebit.startDate=startDate;
promoDebit.status=1;
promoDebit.calcValue=Float.parseFloat(p.getProperty("promoPercent"));
Then ,
if (!promoDebit.save()) {
log.warn myDomainObj.errors.allErrors.join(' \n')
//each error is an instance of org.springframework.validation.FieldError
}
And , I like this one
if (!promoDebit.save()) {
promoDebit.errors.each {
println it
}
}
If you want to throw an exception for EVERY domain classes, then simply set grails.gorm.failOnError to true in grails-app/conf/Config.groovy
or Simply
promoDebit.save(failOnError:true)
Cheers!

log.warn "error occurred by saving: $promoDebit.errors"
you will see the validation failures in the log. If some SQL-constraint gets broken you will get a full-blown Exception like DataIntegrityException

What is the best solution to perform scheduled tasks in Java, whatever the OS?

I'd like to generate alarms on my Java desktop application :
alarms set with a specific date/time which can be in 5 minutes or 5 months
I need to be able to create a SWT application when the alarm is triggered
I need this to be able to work on any OS. The software users will likely have Windows (90% of them), and the rest Mac OS (including me)
the software license must allow me to use it in a commercial program, without requiring to open source it (hence, no GPL)
I cannot require the users to install Cygwin, so the implementation needs to be native to Windows and Unix
I am developing using Java, Eclipse, SWT and my application is deployed from my server using Java Web Start. I'm using Mac OS X.6 for developing.
I think I have a few options:
Run my application at startup, and handle everything myself;
Use a system service.
Use the cron table on Unix, and Scheduled Tasks on Windows
Run at startup
I don't really like this solution, I'm hoping for something more elegant.
Refs: I would like to run my Java program on System Startup on Mac OS/Windows. How can I do this?
System service
If I run it as a system service, I can benefit from this, because the OS will ensure that my software:
is always running
doesn't have/need a GUI
restarts on failure
I've researched some resources that I can use:
run4j — CPL — runs on Windows only, seems like a valid candidate
jsvc — Apache 2.0 — Unix only, seems like a valid candidate
Java Service Wrapper — Various — I cannot afford paid licenses, and the free one is a GPL. Hence, I don't want to/can't use this
My questions in the system service options are:
Are there other options?
Is my planned implementation correct:
at the application startup, check for existence of the service
if it is not installed:
escalate the user to install the service (root on Unix, UAC on Windows)
if the host OS is Windows, use run4j to register the service
if the host OS is Unix, use jsvc to register the service
if it is not running, start it
Thus, at the first run, the application will install the service and start it. When the application closes the service is still running and won't need the application ever again, except if it is unregistered.
However, I think I still miss the "run on startup" feature.
Am I right? Am I missing something?
cron / Task Scheduler
On Unix, I can easily use the cron table without needing the application to escalate the user to root. I don't need to handle restarts, system date changes, etc. Seems nice.
On Windows, I can use the Task Scheduler, even in command-line using At or SchTasks. This seems nice, but I need this to be compatible from XP up to 7, and I can't easily test this.
So what would you do? Did I miss something? Do you have any advice that could help me pick the best and most elegant solution?

Bicou: Great that you shared your solution!
Note that the "schtasks.exe" has some localization issues, if you want to create a daily trigger with it, on an English Windows you'd have to use "daily", on a German one (for example) you'd have to use "täglich" instead.
To resolve this issue I've implemented the call to schtasks.exe with the /xml-option, providing a temporary xml-file which I create by template.
The easiest way to create such a template is to create a task "by hand" and use the "export"-function in the task management GUI tool.

Of the available options you have listed, IMHO Option 3 is better.
As you are looking only for an external trigger to execute the application, CRON or Scheduled tasks are better solutions than other options you have listed. By this way, you remove a complexity from your application and also your application need not be running always. It could be triggered externally and when the execution is over, your application will stop. Hence, unnecessary resource consumption is avoided.

Here's what I ended up implementing:
public class AlarmManager {
public static final String ALARM_CLI_FORMAT = "startalarm:";
public static SupportedOS currentOS = SupportedOS.UNSUPPORTED_OS;
public enum SupportedOS {
UNSUPPORTED_OS,
MAC_OS,
WINDOWS,
}
public AlarmManager() {
final String osName = System.getProperty("os.name");
if (osName == null) {
L.e("Unable to retrieve OS!");
} else if ("Mac OS X".equals(osName)) {
currentOS = SupportedOS.MAC_OS;
} else if (osName.contains("Windows")) {
currentOS = SupportedOS.WINDOWS;
} else {
L.e("Unsupported OS: "+osName);
}
}
/**
* Windows only: name of the scheduled task
*/
private String getAlarmName(final long alarmId) {
return new StringBuilder("My_Alarm_").append(alarmId).toString();
}
/**
* Gets the command line to trigger an alarm
* #param alarmId
* #return
*/
private String getAlarmCommandLine(final long alarmId) {
return new StringBuilder("javaws -open ").append(ALARM_CLI_FORMAT).append(alarmId).append(" ").append(G.JNLP_URL).toString();
}
/**
* Adds an alarm to the system list of scheduled tasks
* #param when
*/
public void createAlarm(final Calendar when) {
// Create alarm
// ... stuff here
final long alarmId = 42;
// Schedule alarm
String[] commandLine;
Process child;
final String alarmCL = getAlarmCommandLine(alarmId);
try {
switch (currentOS) {
case MAC_OS:
final String cron = new SimpleDateFormat("mm HH d M '*' ").format(when.getTime()) + alarmCL;
commandLine = new String[] {
"/bin/sh", "-c",
"crontab -l | (cat; echo \"" + cron + "\") | crontab"
};
child = Runtime.getRuntime().exec(commandLine);
break;
case WINDOWS:
commandLine = new String[] {
"schtasks",
"/Create",
"/ST "+when.get(Calendar.HOUR_OF_DAY) + ":" + when.get(Calendar.MINUTE),
"/SC ONCE",
"/SD "+new SimpleDateFormat("dd/MM/yyyy").format(when.getTime()), // careful with locale here! dd/MM/yyyy or MM/dd/yyyy? I'm French! :)
"/TR \""+alarmCL+"\"",
"/TN \""+getAlarmName(alarmId)+"\"",
"/F",
};
L.d("create command: "+Util.join(commandLine, " "));
child = Runtime.getRuntime().exec(commandLine);
break;
}
} catch (final IOException e) {
L.e("Unable to schedule alarm #"+alarmId, e);
return;
}
L.i("Created alarm #"+alarmId);
}
/**
* Removes an alarm from the system list of scheduled tasks
* #param alarmId
*/
public void removeAlarm(final long alarmId) {
L.i("Removing alarm #"+alarmId);
String[] commandLine;
Process child;
try {
switch (currentOS) {
case MAC_OS:
commandLine = new String[] {
"/bin/sh", "-c",
"crontab -l | (grep -v \""+ALARM_CLI_FORMAT+"\") | crontab"
};
child = Runtime.getRuntime().exec(commandLine);
break;
case WINDOWS:
commandLine = new String[] {
"schtasks",
"/Delete",
"/TN \""+getAlarmName(alarmId)+"\"",
"/F",
};
child = Runtime.getRuntime().exec(commandLine);
break;
}
} catch (final IOException e) {
L.e("Unable to remove alarm #"+alarmId, e);
}
}
public void triggerAlarm(final long alarmId) {
// Do stuff
//...
L.i("Hi! I'm alarm #"+alarmId);
// Remove alarm
removeAlarm(alarmId);
}
}
Usage is simple. Schedule a new alarm using:
final AlarmManager m = new AlarmManager();
final Calendar cal = new GregorianCalendar();
cal.add(Calendar.MINUTE, 1);
m.createAlarm(cal);
Trigger an alarm like this:
public static void main(final String[] args) {
if (args.length >= 2 && args[1] != null && args[1].contains(AlarmManager.ALARM_CLI_FORMAT)) {
try {
final long alarmId = Long.parseLong(args[1].replace(AlarmManager.ALARM_CLI_FORMAT, ""));
final AlarmManager m = new AlarmManager();
m.triggerAlarm(alarmId);
} catch (final NumberFormatException e) {
L.e("Unable to parse alarm !", e);
}
}
}
Tested on Mac OS X.6 and Windows Vista. The class L is an helper to System.out.println and G holds my global constants (here, my JNLP file on my server used to launch my application).

You can also try using Quartz http://quartz-scheduler.org/ . It has a CRON like syntax to schedule jobs.

I believe your scenario is correct. Since services are system specific things, IMHO you should not user a generic package to cover them all, but have a specific mechanism for every system.