We Keep Coding

What is the reason for the error ‘high-bound for max-jOPS is measured to be 0’ when run specjbb2015

used the default args
RAM： 126G
CPU： 4way-16Core-2.5Hz
Run with no JVM_OPTS, SPEC_OPTS ......
Here is the log:
64s: (validation=PASSED) Property settings are COMPLIANT.
64s:
64s: Searching for high-bound for max-jOPS possible for this system
64s: Searching from base IR = 100
64s: rampup IR = 0 [reconfig (term) (init) ] ... (rIR:aIR:PR = 0:0:0) (tPR = 0) [OK]
118s: rampup IR = 10 ... (rIR:aIR:PR = 10:13:13) (tPR = 1148) [OK]
123s: rampup IR = 20 ... (rIR:aIR:PR = 20:17:17) (tPR = 1346) [OK]
126s: rampup IR = 30 .... (rIR:aIR:PR = 30:29:29) (tPR = 2305) [OK]
131s: rampup IR = 40 .??????????????????????????????????????????????????????????????????????????????????. (rIR:aIR:PR = 40:0:0) (tPR = 25) [IR is under limit] [PR is under limit]
215s: Re-initializing (term) (init) ... OK
262s: high-bound for max-jOPS is measured to be 0
262s:
262s: Failed to continue on zero high-bound
262s:
262s: (term... (rIR:aIR:PR = 0:0:0) (tPR = 0) [OK]
265s: ) Tests are skipped.
265s:
265s: Generating level 0 report from ./specjbb2015-C-20210311-00001.data.gz

Error loading PKCS keystore with multiple card readers

I am running into a weird problem when reading CAC card, in a multi terminal setup, from Nth (Except 1st) terminal.
Environment - JDK8_u211, Destop without any card reader, Internal card reader for laptop, USB card reader, CAC\PIV card
USE-CASE Tried
When the CAC card is plugged into internal slot of Laptop, everything works fine with and withou USB card terminal connected to laptop.
When the CAC card is plugged into the USB reader and connected to the Desktop, everything works fine.
When the CAC card is plugged into the USB reader and connected to the Laptop, the PKCS store instantiation fails, as the config uses slot=1
When the CAC card is plugged into the USB reader and connected to the Laptop, if I hard code slot=4 in the config, everything works fine.
Problem: Unable to determine the correct slot to use within the config file. Somehow number of terminals is not adding up to determine the slot at runtime.
By enabling JDK debug logs, i could verify that internally "C_GetSlotList" returns 0,4 where as I have only 2 terminals, hence using index 0 or 1.
Does anyone know how to determine the correlation between terminal vs slot or is there something really wrong in the way i am trying to read the card ?
Please advise.
Sample code
public class MultipleTerminals {
private static String DLL_PATH = "C:\\opensc-pkcs11.dll";
public static void main(String[] args) throws Exception {
MyTerminal custTerminalObj = getSmartCardTerminal();
CardTerminal terminal = custTerminalObj.getTerminal();
Integer terminalIdx = custTerminalObj.getIndex();
System.out.println("Using Terminal.Name: " + terminal.getName());
System.out.println("Using Terminal.isCardPresent: " + ((CardTerminal)terminal).isCardPresent());
if (terminal.isCardPresent()) { // duplicate check
registerProvider(4);
KeyStore keyStore = createKeyStore();
printCertificate(keyStore);
}
}
public static MyTerminal getSmartCardTerminal() {
System.out.println("---------------------------------------------------");
MyTerminal terminal = null;
int terminalIdx = -1;
try {
TerminalFactory factory = TerminalFactory.getDefault();
if (factory != null) {
List<CardTerminal> allTerminals = factory.terminals().list();
for(CardTerminal term : allTerminals) {
System.out.println("Terminal Name: " + term.getName());
System.out.println("isCardPresent: " + term.isCardPresent());
System.out.println("----------");
}
for(CardTerminal term : allTerminals) {
terminalIdx++;
if(term.isCardPresent()) {
terminal = new MyTerminal();
terminal.setIndex(terminalIdx);
terminal.setTerminal(term);
System.out.println("Using Terminal Idx: " + terminalIdx);
}
}
}
} catch (CardException e1) {
terminal = null;
System.out.println("SmartCardHelper.getSmartCardTerminal --> "
+ "Exception while accessing smart-card terminal: "
+ e1.getMessage());
e1.printStackTrace();
}
return terminal;
}
public static void registerProvider(Integer idx) {
System.out.println("---------------------------------------------------");
String ext = "attributes(*,*,*)=\n{\nCKA_TOKEN=true\nCKA_LOCAL=true\n}";
String OPENSC_DEFAULT_PROVIDER_NAME = "PKCS#11";
String OPENSC_PKCS11_DLL = DLL_PATH;
String configString = "name = "
+ OPENSC_DEFAULT_PROVIDER_NAME.replace(' ', '_') + "\n"
+ "library = "
+ OPENSC_PKCS11_DLL + "\n slot = " + idx + " "
+ "\n attributes = compatibility \n" + ext;
System.out.println("\tConfigString: " + configString);
ByteArrayInputStream is = new ByteArrayInputStream(
configString.getBytes(Charset.defaultCharset()));
Provider cardSecurityProvider = new sun.security.pkcs11.SunPKCS11(
is);
displayProviders();
Security.addProvider(cardSecurityProvider);
displayProviders();
System.out.println("==================");
}
public static KeyStore createKeyStore() throws KeyStoreException {
KeyStore.CallbackHandlerProtection callbackHandler = new KeyStore.CallbackHandlerProtection(
new PIVCardPinHandler());
KeyStore.Builder builder = KeyStore.Builder.newInstance("PKCS11", null,
callbackHandler);
return builder.getKeyStore();
}
public static void displayProviders() {
System.out.println("---------------------------------------------------");
Provider[] objs = Security.getProviders();
System.out.println("Provider Count: " + objs.length);
String type = "KeyStore";
String algorithm="PKCS11";
ProviderList list = Providers.getProviderList();
Provider provider = list.getProvider("SunPKCS11-PKCS");
System.out.println("Provider[SunPKCS11-PKCS]: " + provider);
if(provider != null) {
Set<Service> svcs = provider.getServices();
System.out.println("Service count: " + svcs.size());
}
Service firstService = list.getService(type, algorithm);
System.out.println("Type[KeyStore], Algo[PKCS11], Service: " + firstService);
}
public static void printCertificate(KeyStore keyStore)
throws KeyStoreException {
for (String alias : Collections.list(keyStore.aliases())) {
System.out.println("Alias: " + alias);
}
}
}
class MyTerminal{
private Integer index;
private CardTerminal terminal;
public MyTerminal() {
}
public Integer getIndex() {
return index;
}
public void setIndex(Integer index) {
this.index = index;
}
public CardTerminal getTerminal() {
return terminal;
}
public void setTerminal(CardTerminal terminal) {
this.terminal = terminal;
}
}
* Output TestCase-1 (WORKING)*
---------------------------------------------------
Terminal Name: Alcor Micro USB Smart Card Reader 0
isCardPresent: true
----------
Terminal Name: Identive SCR33xx v2.0 USB SC Reader 0
isCardPresent: false
----------
Using Terminal Idx: 0
Using Terminal.Name: Alcor Micro USB Smart Card Reader 0
Using Terminal.isCardPresent: true
---------------------------------------------------
ConfigString:
name = PKCS#11
library = C:\opensc-pkcs11.dll
slot = 0
attributes = compatibility
attributes(*,*,*)=
{
CKA_TOKEN=true
CKA_LOCAL=true
}
SunPKCS11 loading ---DummyConfig-1---
sunpkcs11: Initializing PKCS#11 library C:\opensc-pkcs11.dll
Information for provider SunPKCS11-PKCS
Library info:
cryptokiVersion: 2.20
manufacturerID: DUMMY_VALUE
flags: 0
libraryDescription: DUMMY_VALUE
libraryVersion: 0.19
All slots: 0, 4
Slots with tokens: 0, 4
Slot info for slot 0:
slotDescription: Alcor Micro USB Smart Card Reader 0
manufacturerID: DUMMY_VALUE
flags: CKF_TOKEN_PRESENT | CKF_REMOVABLE_DEVICE | CKF_HW_SLOT
hardwareVersion: 0.00
firmwareVersion: 0.00
Token info for token in slot 0:
label: DUMMY_VALUE
manufacturerID: DUMMY_VALUE
model: PKCS#15 emulated
serialNumber: DUMMY_VALUE
flags: CKF_RNG | CKF_LOGIN_REQUIRED | CKF_USER_PIN_INITIALIZED | CKF_TOKEN_INITIALIZED
ulMaxSessionCount: CK_EFFECTIVELY_INFINITE
ulSessionCount: 0
ulMaxRwSessionCount: CK_EFFECTIVELY_INFINITE
ulRwSessionCount: 0
ulMaxPinLen: 8
ulMinPinLen: 4
ulTotalPublicMemory: CK_UNAVAILABLE_INFORMATION
ulFreePublicMemory: CK_UNAVAILABLE_INFORMATION
ulTotalPrivateMemory: CK_UNAVAILABLE_INFORMATION
ulFreePrivateMemory: CK_UNAVAILABLE_INFORMATION
hardwareVersion: 0.00
firmwareVersion: 0.00
utcTime:
---------------------------------------------------
Provider Count: 10
Provider[SunPKCS11-PKCS]: null
Type[KeyStore], Algo[PKCS11], Service: null
---------------------------------------------------
Provider Count: 11
Provider[SunPKCS11-PKCS]: SunPKCS11-PKCS version 1.8
Service count: 26
Type[KeyStore], Algo[PKCS11], Service: SunPKCS11-PKCS: KeyStore.PKCS11 -> sun.security.pkcs11.P11KeyStore
aliases: [PKCS11-PKCS]
(KeyStore)
==================
sunpkcs11: caller passed NULL pin
* Output TestCase-3 (NOT WORKING)*
---------------------------------------------------
Terminal Name: Alcor Micro USB Smart Card Reader 0
isCardPresent: false
----------
Terminal Name: Identive SCR33xx v2.0 USB SC Reader 0
isCardPresent: true
----------
Using Terminal Idx: 1
Using Terminal.Name: Identive SCR33xx v2.0 USB SC Reader 0
Using Terminal.isCardPresent: true
---------------------------------------------------
ConfigString:
name = PKCS#11
library = C:\opensc-pkcs11.dll
slot = 1
attributes = compatibility
attributes(*,*,*)=
{
CKA_TOKEN=true
CKA_LOCAL=true
}
SunPKCS11 loading ---DummyConfig-1---
sunpkcs11: Initializing PKCS#11 library C:\opensc-pkcs11.dll
Information for provider SunPKCS11-PKCS
Library info:
cryptokiVersion: 2.20
manufacturerID: DUMMY_VALUE
flags: 0
libraryDescription: DUMMY_VALUE
libraryVersion: 0.19
All slots: 0, 4
Slots with tokens: 0, 4
---------------------------------------------------
Provider Count: 10
Provider[SunPKCS11-PKCS]: null
Type[KeyStore], Algo[PKCS11], Service: null
---------------------------------------------------
Provider Count: 11
Provider[SunPKCS11-PKCS]: SunPKCS11-PKCS version 1.8
Service count: 0
Type[KeyStore], Algo[PKCS11], Service: null
==================
Exception in thread "main" java.security.KeyStoreException: KeyStore instantiation failed
at java.security.KeyStore$Builder$2.getKeyStore(KeyStore.java:1967)
Caused by: java.security.KeyStoreException: PKCS11 not found
at java.security.KeyStore.getInstance(KeyStore.java:851)
at java.security.KeyStore$Builder$2$1.run(KeyStore.java:1923)
at java.security.KeyStore$Builder$2$1.run(KeyStore.java:1918)
at java.security.AccessController.doPrivileged(Native Method)
at java.security.KeyStore$Builder$2.getKeyStore(KeyStore.java:1964)
... 2 more
Caused by: java.security.NoSuchAlgorithmException: PKCS11 KeyStore not available
at sun.security.jca.GetInstance.getInstance(GetInstance.java:159)
at java.security.Security.getImpl(Security.java:695)
at java.security.KeyStore.getInstance(KeyStore.java:848)
... 6 more
* Output TestCase-4 (WORKING)*
---------------------------------------------------
Terminal Name: Alcor Micro USB Smart Card Reader 0
isCardPresent: false
----------
Terminal Name: Identive SCR33xx v2.0 USB SC Reader 0
isCardPresent: true
----------
Using Terminal Idx: 1
Using Terminal.Name: Identive SCR33xx v2.0 USB SC Reader 0
Using Terminal.isCardPresent: true
---------------------------------------------------
ConfigString:
name = PKCS#11
library = C:\opensc-pkcs11.dll
slot = 4
attributes = compatibility
attributes(*,*,*)=
{
CKA_TOKEN=true
CKA_LOCAL=true
}
SunPKCS11 loading ---DummyConfig-1---
sunpkcs11: Initializing PKCS#11 library C:\opensc-pkcs11.dll
Information for provider SunPKCS11-PKCS
Library info:
cryptokiVersion: DUMMY_VALUE
manufacturerID: DUMMY_VALUE
flags: 0
libraryDescription: DUMMY_VALUE
libraryVersion: DUMMY_VALUE
All slots: 0, 4
Slots with tokens: 0, 4
Slot info for slot 4:
slotDescription: DUMMY_VALUE
manufacturerID: DUMMY_VALUE
flags: CKF_TOKEN_PRESENT | CKF_REMOVABLE_DEVICE | CKF_HW_SLOT
hardwareVersion: 0.00
firmwareVersion: 0.00
Token info for token in slot 4:
label: DUMMY_LABEL
manufacturerID: DUMMY_VALUE
model: PKCS#15 emulated
serialNumber: DUMMY_VALUE
flags: CKF_RNG | CKF_LOGIN_REQUIRED | CKF_USER_PIN_INITIALIZED | CKF_TOKEN_INITIALIZED
ulMaxSessionCount: CK_EFFECTIVELY_INFINITE
ulSessionCount: 0
ulMaxRwSessionCount: CK_EFFECTIVELY_INFINITE
ulRwSessionCount: 0
ulMaxPinLen: 8
ulMinPinLen: 4
ulTotalPublicMemory: CK_UNAVAILABLE_INFORMATION
ulFreePublicMemory: CK_UNAVAILABLE_INFORMATION
ulTotalPrivateMemory: CK_UNAVAILABLE_INFORMATION
ulFreePrivateMemory: CK_UNAVAILABLE_INFORMATION
hardwareVersion: 0.00
firmwareVersion: 0.00
utcTime:
---------------------------------------------------
Provider Count: 10
Provider[SunPKCS11-PKCS]: null
Type[KeyStore], Algo[PKCS11], Service: null
---------------------------------------------------
Provider Count: 11
Provider[SunPKCS11-PKCS]: SunPKCS11-PKCS version 1.8
Service count: 26
Type[KeyStore], Algo[PKCS11], Service: SunPKCS11-PKCS: KeyStore.PKCS11 -> sun.security.pkcs11.P11KeyStore
aliases: [PKCS11-PKCS]
(KeyStore)
==================
sunpkcs11: caller passed NULL pin

You are getting confused between slot id and slot index. They are two different attributes.
The C_GetSlotList gives you an array of slot id's not slot indexes. In your case 0 and 4 are slot id's, not slot indexes. The slot index is the index of the slot id in the array.
In your getSmartCardTerminal() method where you are identifying the terminalIdx you are identifying the index, but in your registerProvider(Integer idx) method you are mapping the index as the id, here:
OPENSC_PKCS11_DLL + "\n slot = " + idx + " "
When you are using the slot index you should be using slotListIndex and when you are using the slot id, you should be using slot.
So to fix your problem change it to:
OPENSC_PKCS11_DLL + "\n slotListIndex = " + idx + " "
Documentation here.

CryptoJS encrypt HMACSha256 different than Java

I'm trying to convert this code of CryptoJS to Kotlin:
const hash = CryptoJS.HmacSHA256(message, key);
const signature = CryptoJS.enc.Hex.stringify(hash);
That's the kotlin code equivalent to above snippet:
private fun generateSignature(key: String, payload: String): String {
val algorithm = "HmacSHA256"
return Mac.getInstance(algorithm)
.apply { init(SecretKeySpec(key.toByteArray(), algorithm)) }
.run { doFinal(payload.toByteArray()) }
.let { HexUtils.toHexString(it) }
}
But it is not working at all. They generate different results. CryptoJS generates an array of bytes that has 8 positions, the Java code generates an array of bytes that has 32 positions.
I don't know what Im doing wrong. I need to make my Kotlin code work exactly as the javascript one.
Update: I can't change the Javascript way. I have to do the exactly same thing in Kotlin
Update2: Here is a test where the JS code and the Kotlin code generates different results.
Input:
key = 's21fk4vb-5415-46c7-aade-303dcf432bb4'
message = 'POST,/wallets/3323461f96-bdf3-4e03-bc93-7da1fb27aee7/withdraw/,1573148023809,{"amount":"1.0","bank":{"bank":"789","agency":"456","account":"12378","accountDigit":"6","name":"joao","taxId":"33206913098","holderType":"personal"}}'
Results with JS code:
Result of encrypt in bytes:
{sigBytes: 32, words: [8]}
sigBytes: 32
words: [8]
0: 2102759135
1: -196086391
2: -2099697915
3: -1620551271
4: 2463524
5: 1757965357
6: -1039993965
7: -1798822705
Bytes to Hex:
7d558edff44ff58982d927059f6859990025972468c86c2dc202f39394c824cf
Results with Kotlin code:
Result of encrypt in bytes:
{byte[32]#1126}
0 = 82
1 = -110
2 = -100
3 = -128
4 = -63
5 = 22
6 = -103
7 = -31
8 = 83
9 = -125
10 = -72
11 = 109
12 = -91
13 = -69
14 = 54
15 = -41
16 = 27
17 = -107
18 = -60
19 = -110
20 = -57
21 = -29
22 = -20
23 = -32
24 = -66
25 = 88
26 = 87
27 = -50
28 = -47
29 = -18
30 = -96
31 = 25
Bytes to Hex:
52929c80c11699e15383b86da5bb36d71b95c492c7e3ece0be5857ced1eea019

No SHA-256 hash can have only 8 byte positions. The output, as the name suggests, should be 256 bits or 32 bytes. What I suspect to happen is that the input of stringify is already presumed to be bytes, while CryptoJS functions return a WordArray of 32 bit words. As 8 * 32 = 256 this seems reasonable.
So I presume you can simply fix this by using a function on the WordArray instead, for instance hash.toString('hex').

Golang and apache AB

I have a system with HTTP POST requests and it runs with Spring 5 (standalone tomcat). In short it looks like this:
client (Apache AB) ----> micro service (java or golang) --> RabbitMQ --> Core(spring + tomcat).
The thing is, when I use my Java (Spring) service, it is ok. AB shows this output:
ab -n 1000 -k -s 2 -c 10 -s 60 -p test2.sh -A 113:113 -T 'application/json' https://127.0.0.1:8449/SecureChat/chat/v1/rest-message/send
This is ApacheBench, Version 2.3 <$Revision: 1807734 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking 127.0.0.1 (be patient)
Completed 100 requests
...
Completed 1000 requests
Finished 1000 requests
Server Software:
Server Hostname: 127.0.0.1
Server Port: 8449
SSL/TLS Protocol: TLSv1.2,ECDHE-RSA-AES256-GCM-SHA384,2048,256
Document Path: /rest-message/send
Document Length: 39 bytes
Concurrency Level: 10
Time taken for tests: 434.853 seconds
Complete requests: 1000
Failed requests: 0
Keep-Alive requests: 0
Total transferred: 498000 bytes
Total body sent: 393000
HTML transferred: 39000 bytes
Requests per second: 2.30 [#/sec] (mean)
Time per request: 4348.528 [ms] (mean)
Time per request: 434.853 [ms] (mean, across all concurrent
requests)
Transfer rate: 1.12 [Kbytes/sec] received
0.88 kb/s sent
2.00 kb/s total
Connection Times (ms)
min mean[+/-sd] median max
Connect: 4 14 7.6 17 53
Processing: 1110 4317 437.2 4285 8383
Waiting: 1107 4314 437.2 4282 8377
Total: 1126 4332 436.8 4300 8403
That is through TLS.
But when I try to use my Golang service I get timeout:
Benchmarking 127.0.0.1 (be patient)...apr_pollset_poll: The timeout specified has expired (70007)
Total of 92 requests completed
And this output:
ab -n 100 -k -s 2 -c 10 -s 60 -p test2.sh -T 'application/json' http://127.0.0.1:8089/
This is ApacheBench, Version 2.3 <$Revision: 1807734 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking 127.0.0.1 (be patient)...^C
Server Software:
Server Hostname: 127.0.0.1
Server Port: 8089
Document Path: /
Document Length: 39 bytes
Concurrency Level: 10
Time taken for tests: 145.734 seconds
Complete requests: 92
Failed requests: 1
(Connect: 0, Receive: 0, Length: 1, Exceptions: 0)
Keep-Alive requests: 91
Total transferred: 16380 bytes
Total body sent: 32200
HTML transferred: 3549 bytes
Requests per second: 0.63 [#/sec] (mean)
Time per request: 15840.663 [ms] (mean)
Time per request: 1584.066 [ms] (mean, across all concurrent requests)
Transfer rate: 0.11 [Kbytes/sec] received
0.22 kb/s sent
0.33 kb/s total
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.0 0 0
Processing: 1229 1494 1955.9 1262 20000
Waiting: 1229 1291 143.8 1262 2212
Total: 1229 1494 1955.9 1262 20000
That is through plane tcp.
I guess I have some mistakes in my code. I made it in one file
func initAmqp(rabbitUrl string) {
var err error
conn, err = amqp.Dial(rabbitUrl)
failOnError(err, "Failed to connect to RabbitMQ")
}
func main() {
err := gcfg.ReadFileInto(&cfg, "config.gcfg")
if err != nil {
log.Fatal(err);
}
PrintConfig(cfg)
if cfg.Section_rabbit.RabbitUrl != "" {
initAmqp(cfg.Section_rabbit.RabbitUrl);
}
mux := http.NewServeMux();
mux.Handle("/", NewLimitHandler(1000, newTestHandler()))
server := http.Server {
Addr: cfg.Section_basic.Port,
Handler: mux,
ReadTimeout: 20 * time.Second,
WriteTimeout: 20 * time.Second,
}
defer conn.Close();
log.Println(server.ListenAndServe());
}
func NewLimitHandler(maxConns int, handler http.Handler) http.Handler {
h := &limitHandler{
connc: make(chan struct{}, maxConns),
handler: handler,
}
for i := 0; i < maxConns; i++ {
h.connc <- struct{}{}
}
return h
}
func newTestHandler() http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
handler(w, r);
})
}
func handler(w http.ResponseWriter, r *http.Request) {
if b, err := ioutil.ReadAll(r.Body); err == nil {
fmt.Println("message is ", string(b));
res := publishMessages(string(b))
w.Write([]byte(res))
w.WriteHeader(http.StatusOK)
counter ++;
}else {
w.WriteHeader(http.StatusInternalServerError)
w.Write([]byte("500 - Something bad happened!"))
}
}
func publishMessages(payload string) string {
ch, err := conn.Channel()
failOnError(err, "Failed to open a channel")
q, err = ch.QueueDeclare(
"", // name
false, // durable
false, // delete when unused
true, // exclusive
false, // noWait
nil, // arguments
)
failOnError(err, "Failed to declare a queue")
msgs, err := ch.Consume(
q.Name, // queue
"", // consumer
true, // auto-ack
false, // exclusive
false, // no-local
false, // no-wait
nil, // args
)
failOnError(err, "Failed to register a consumer")
corrId := randomString(32)
log.Println("corrId ", corrId)
err = ch.Publish(
"", // exchange
cfg.Section_rabbit.RabbitQeue, // routing key
false, // mandatory
false, // immediate
amqp.Publishing{
DeliveryMode: amqp.Transient,
ContentType: "application/json",
CorrelationId: corrId,
Body: []byte(payload),
Timestamp: time.Now(),
ReplyTo: q.Name,
})
failOnError(err, "Failed to Publish on RabbitMQ")
defer ch.Close();
result := "";
for d := range msgs {
if corrId == d.CorrelationId {
failOnError(err, "Failed to convert body to integer")
log.Println("result = ", string(d.Body))
return string(d.Body);
}else {
log.Println("waiting for result = ")
}
}
return result;
}
Can someone help?
EDIT
here are my variables
type limitHandler struct {
connc chan struct{}
handler http.Handler
}
var conn *amqp.Connection
var q amqp.Queue
EDIT 2
func (h *limitHandler) ServeHTTP(w http.ResponseWriter, req *http.Request) {
select {
case <-h.connc:
fmt.Println("ServeHTTP");
h.handler.ServeHTTP(w, req)
h.connc <- struct{}{}
default:
http.Error(w, "503 too busy", http.StatusServiceUnavailable)
}
}
EDIT 3
func failOnError(err error, msg string) {
if err != nil {
log.Fatalf("%s: %s", msg, err)
panic(fmt.Sprintf("%s: %s", msg, err))
}
}

Implement a java UDF and call it from pyspark

I need to create a UDF to be used in pyspark python which uses a java object for its internal calculations.
If it were a simple python I would do something like:
def f(x):
return 7
fudf = pyspark.sql.functions.udf(f,pyspark.sql.types.IntegerType())
and call it using:
df = sqlContext.range(0,5)
df2 = df.withColumn("a",fudf(df.id)).show()
However, the implementation of the function I need is in java and not in python. I need to wrap it somehow so I can call it in a similar way from python.
My first try was to do implement the java object, then wrap it in python in pyspark and convert that to UDF. That failed with serialization error.
Java code:
package com.test1.test2;
public class TestClass1 {
Integer internalVal;
public TestClass1(Integer val1) {
internalVal = val1;
}
public Integer do_something(Integer val) {
return internalVal;
}
}
pyspark code:
from py4j.java_gateway import java_import
from pyspark.sql.functions import udf
from pyspark.sql.types import IntegerType
java_import(sc._gateway.jvm, "com.test1.test2.TestClass1")
a = sc._gateway.jvm.com.test1.test2.TestClass1(7)
audf = udf(a,IntegerType())
error:
---------------------------------------------------------------------------
Py4JError Traceback (most recent call last)
<ipython-input-2-9756772ab14f> in <module>()
4 java_import(sc._gateway.jvm, "com.test1.test2.TestClass1")
5 a = sc._gateway.jvm.com.test1.test2.TestClass1(7)
----> 6 audf = udf(a,IntegerType())
/usr/local/spark/python/pyspark/sql/functions.py in udf(f, returnType)
1595 [Row(slen=5), Row(slen=3)]
1596 """
-> 1597 return UserDefinedFunction(f, returnType)
1598
1599 blacklist = ['map', 'since', 'ignore_unicode_prefix']
/usr/local/spark/python/pyspark/sql/functions.py in __init__(self, func, returnType, name)
1556 self.returnType = returnType
1557 self._broadcast = None
-> 1558 self._judf = self._create_judf(name)
1559
1560 def _create_judf(self, name):
/usr/local/spark/python/pyspark/sql/functions.py in _create_judf(self, name)
1565 command = (func, None, ser, ser)
1566 sc = SparkContext.getOrCreate()
-> 1567 pickled_command, broadcast_vars, env, includes = _prepare_for_python_RDD(sc, command, self)
1568 ctx = SQLContext.getOrCreate(sc)
1569 jdt = ctx._ssql_ctx.parseDataType(self.returnType.json())
/usr/local/spark/python/pyspark/rdd.py in _prepare_for_python_RDD(sc, command, obj)
2297 # the serialized command will be compressed by broadcast
2298 ser = CloudPickleSerializer()
-> 2299 pickled_command = ser.dumps(command)
2300 if len(pickled_command) > (1 << 20): # 1M
2301 # The broadcast will have same life cycle as created PythonRDD
/usr/local/spark/python/pyspark/serializers.py in dumps(self, obj)
426
427 def dumps(self, obj):
--> 428 return cloudpickle.dumps(obj, 2)
429
430
/usr/local/spark/python/pyspark/cloudpickle.py in dumps(obj, protocol)
644
645 cp = CloudPickler(file,protocol)
--> 646 cp.dump(obj)
647
648 return file.getvalue()
/usr/local/spark/python/pyspark/cloudpickle.py in dump(self, obj)
105 self.inject_addons()
106 try:
--> 107 return Pickler.dump(self, obj)
108 except RuntimeError as e:
109 if 'recursion' in e.args[0]:
/home/mendea3/anaconda2/lib/python2.7/pickle.pyc in dump(self, obj)
222 if self.proto >= 2:
223 self.write(PROTO + chr(self.proto))
--> 224 self.save(obj)
225 self.write(STOP)
226
/home/mendea3/anaconda2/lib/python2.7/pickle.pyc in save(self, obj)
284 f = self.dispatch.get(t)
285 if f:
--> 286 f(self, obj) # Call unbound method with explicit self
287 return
288
/home/mendea3/anaconda2/lib/python2.7/pickle.pyc in save_tuple(self, obj)
566 write(MARK)
567 for element in obj:
--> 568 save(element)
569
570 if id(obj) in memo:
/home/mendea3/anaconda2/lib/python2.7/pickle.pyc in save(self, obj)
284 f = self.dispatch.get(t)
285 if f:
--> 286 f(self, obj) # Call unbound method with explicit self
287 return
288
/usr/local/spark/python/pyspark/cloudpickle.py in save_function(self, obj, name)
191 if islambda(obj) or obj.__code__.co_filename == '<stdin>' or themodule is None:
192 #print("save global", islambda(obj), obj.__code__.co_filename, modname, themodule)
--> 193 self.save_function_tuple(obj)
194 return
195 else:
/usr/local/spark/python/pyspark/cloudpickle.py in save_function_tuple(self, func)
234 # create a skeleton function object and memoize it
235 save(_make_skel_func)
--> 236 save((code, closure, base_globals))
237 write(pickle.REDUCE)
238 self.memoize(func)
/home/mendea3/anaconda2/lib/python2.7/pickle.pyc in save(self, obj)
284 f = self.dispatch.get(t)
285 if f:
--> 286 f(self, obj) # Call unbound method with explicit self
287 return
288
/home/mendea3/anaconda2/lib/python2.7/pickle.pyc in save_tuple(self, obj)
552 if n <= 3 and proto >= 2:
553 for element in obj:
--> 554 save(element)
555 # Subtle. Same as in the big comment below.
556 if id(obj) in memo:
/home/mendea3/anaconda2/lib/python2.7/pickle.pyc in save(self, obj)
284 f = self.dispatch.get(t)
285 if f:
--> 286 f(self, obj) # Call unbound method with explicit self
287 return
288
/home/mendea3/anaconda2/lib/python2.7/pickle.pyc in save_list(self, obj)
604
605 self.memoize(obj)
--> 606 self._batch_appends(iter(obj))
607
608 dispatch[ListType] = save_list
/home/mendea3/anaconda2/lib/python2.7/pickle.pyc in _batch_appends(self, items)
637 write(MARK)
638 for x in tmp:
--> 639 save(x)
640 write(APPENDS)
641 elif n:
/home/mendea3/anaconda2/lib/python2.7/pickle.pyc in save(self, obj)
304 reduce = getattr(obj, "__reduce_ex__", None)
305 if reduce:
--> 306 rv = reduce(self.proto)
307 else:
308 reduce = getattr(obj, "__reduce__", None)
/usr/local/spark/python/lib/py4j-0.9-src.zip/py4j/java_gateway.py in __call__(self, *args)
811 answer = self.gateway_client.send_command(command)
812 return_value = get_return_value(
--> 813 answer, self.gateway_client, self.target_id, self.name)
814
815 for temp_arg in temp_args:
/usr/local/spark/python/pyspark/sql/utils.py in deco(*a, **kw)
43 def deco(*a, **kw):
44 try:
---> 45 return f(*a, **kw)
46 except py4j.protocol.Py4JJavaError as e:
47 s = e.java_exception.toString()
/usr/local/spark/python/lib/py4j-0.9-src.zip/py4j/protocol.py in get_return_value(answer, gateway_client, target_id, name)
310 raise Py4JError(
311 "An error occurred while calling {0}{1}{2}. Trace:\n{3}\n".
--> 312 format(target_id, ".", name, value))
313 else:
314 raise Py4JError(
Py4JError: An error occurred while calling o18.__getnewargs__. Trace:
py4j.Py4JException: Method __getnewargs__([]) does not exist
at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:335)
at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:344)
at py4j.Gateway.invoke(Gateway.java:252)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:133)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.GatewayConnection.run(GatewayConnection.java:209)
at java.lang.Thread.run(Thread.java:745)
EDIT: I also tried to make the java class serializable but to no avail.
My second attempt was to define the UDF in java to begin with but that failed as I am not sure how to correctly wrap it:
java code:
package com.test1.test2;
import org.apache.spark.sql.api.java.UDF1;
public class TestClassUdf implements UDF1<Integer, Integer> {
Integer retval;
public TestClassUdf(Integer val) {
retval = val;
}
#Override
public Integer call(Integer arg0) throws Exception {
return retval;
}
}
but how would I use it?
I tried:
from py4j.java_gateway import java_import
java_import(sc._gateway.jvm, "com.test1.test2.TestClassUdf")
a = sc._gateway.jvm.com.test1.test2.TestClassUdf(7)
dfint = sqlContext.range(0,15)
df = dfint.withColumn("a",a(dfint.id))
but I get:
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-5-514811090b5f> in <module>()
3 a = sc._gateway.jvm.com.test1.test2.TestClassUdf(7)
4 dfint = sqlContext.range(0,15)
----> 5 df = dfint.withColumn("a",a(dfint.id))
TypeError: 'JavaObject' object is not callable
and I tried to use a.call instead of a:
df = dfint.withColumn("a",a.call(dfint.id))
but got:
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
in ()
3 a = sc._gateway.jvm.com.test1.test2.TestClassUdf(7)
4 dfint = sqlContext.range(0,15)
----> 5 df = dfint.withColumn("a",a.call(dfint.id))
/usr/local/spark/python/lib/py4j-0.9-src.zip/py4j/java_gateway.py in __call__(self, *args)
796 def __call__(self, *args):
797 if self.converters is not None and len(self.converters) > 0:
--> 798 (new_args, temp_args) = self._get_args(args)
799 else:
800 new_args = args
/usr/local/spark/python/lib/py4j-0.9-src.zip/py4j/java_gateway.py in _get_args(self, args)
783 for converter in self.gateway_client.converters:
784 if converter.can_convert(arg):
--> 785 temp_arg = converter.convert(arg, self.gateway_client)
786 temp_args.append(temp_arg)
787 new_args.append(temp_arg)
/usr/local/spark/python/lib/py4j-0.9-src.zip/py4j/java_collections.py in convert(self, object, gateway_client)
510 HashMap = JavaClass("java.util.HashMap", gateway_client)
511 java_map = HashMap()
--> 512 for key in object.keys():
513 java_map[key] = object[key]
514 return java_map
TypeError: 'Column' object is not callable
Any help would be appriciated.

I got this working with the help of another question (and answer) of your own about UDAFs.
Spark provides a udf() method for wrapping Scala FunctionN, so we can wrap the Java function in Scala and use that. Your Java method needs to be static or on a class that implements Serializable.
package com.example
import org.apache.spark.sql.UserDefinedFunction
import org.apache.spark.sql.functions.udf
class MyUdf extends Serializable {
def getUdf: UserDefinedFunction = udf(() => MyJavaClass.MyJavaMethod())
}
Usage in PySpark:
def my_udf():
from pyspark.sql.column import Column, _to_java_column, _to_seq
pcls = "com.example.MyUdf"
jc = sc._jvm.java.lang.Thread.currentThread() \
.getContextClassLoader().loadClass(pcls).newInstance().getUdf().apply
return Column(jc(_to_seq(sc, [], _to_java_column)))
rdd1 = sc.parallelize([{'c1': 'a'}, {'c1': 'b'}, {'c1': 'c'}])
df1 = rdd1.toDF()
df2 = df1.withColumn('mycol', my_udf())
As with the UDAF in your other question and answer, we can pass columns into it with return Column(jc(_to_seq(sc, ["col1", "col2"], _to_java_column)))

In lines with https://dzone.com/articles/pyspark-java-udf-integration-1 you could define UDF1 with in Java using
public class AddNumber implements UDF1<Long, Long> {
#Override
public Long call(Long num) throws Exception {
return (num + 5);
}
}
And then after adding the jar to your pyspark with --package <your-jar>
you can use it in pyspark as:
from pyspark.sql import functions as F
from pyspark.sql.types import LongType
>>> df = spark.createDataFrame([float(i) for i in range(100)], FloatType()).toDF("a")
>>> spark.udf.registerJavaFunction("addNumber", "com.example.spark.AddNumber", LongType())
>>> df.withColumn("b", F.expr("addNumber(a)")).show(5)
+---+---+
| a| b|
+---+---+
|0.0| 5|
|1.0| 6|
|2.0| 7|
|3.0| 8|
|4.0| 8|
+---+---+
only showing top 5 rows