How to encode nested Python Protobuf - java

Been stumped on this for a while and pulling what is left of my hair out.
Sending non-nested Protobufs from Python to Java and Java to Python without
an issue with WebSockets. My problem is sending a nested version over a WebSocket. I believe my issue is on
the Python encoding side.
Your guidance is appreciated.
.proto file
message Response {
// Reflect back to caller
required string service_name = 1;
// Reflect back to caller
required string method_name = 2;
// Who is responding
required string client_id = 3;
// Status Code
required StatusCd status_cd = 4;
// RPC response proto
optional bytes response_proto = 5;
// Was callback invoked
optional bool callback = 6 [default = false];
// Error, if any
optional string error = 7;
//optional string response_desc = 6;
}
message HeartbeatResult {
required string service = 1;
required string timestamp = 2;
required float status_cd = 3;
required string status_summary = 4;
}
A Heartbeat result is supposed to get sent in the reponse_proto field
of the Response Protobuf. I am able to do this in Java to Java but Python
to Java is not working.
I've included two variations of the python code. Neither of which works.
def GetHeartbeat(self):
print "GetHeartbeat called"
import time
ts = time.time()
import datetime
st = datetime.datetime.fromtimestamp(ts).strftime('%Y-%m-%d %H:%M:%S')
heartbeatResult = rpc_pb2.HeartbeatResult()
heartbeatResult.service = "ALERT_SERVICE"
heartbeatResult.timestamp = st
heartbeatResult.status_cd = rpc_pb2.OK
heartbeatResult.status_summary = "OK"
response = rpc_pb2.Response()
response.service_name = ""
response.method_name = "SendHeartbeatResult"
response.client_id = "ALERT_SERVICE"
response.status_cd = rpc_pb2.OK
response.response_proto = str(heartbeatResult).encode('utf-8')
self.sendMessage(response.SerializeToString())
print "GetHeartbeat finished"
def GetHeartbeat2(self):
print "GetHeartbeat called"
import time
ts = time.time()
import datetime
st = datetime.datetime.fromtimestamp(ts).strftime('%Y-%m-%d %H:%M:%S')
heartbeatResult = rpc_pb2.HeartbeatResult()
heartbeatResult.service = "ALERT_SERVICE"
heartbeatResult.timestamp = st
heartbeatResult.status_cd = rpc_pb2.OK
heartbeatResult.status_summary = "OK"
response = rpc_pb2.Response()
response.service_name = ""
response.method_name = "SendHeartbeatResult"
response.client_id = "ALERT_SERVICE"
response.status_cd = rpc_pb2.OK
response.response_proto = heartbeatResult.SerializeToString()
self.sendMessage(response.SerializeToString())
print "GetHeartbeat finished"
Errors on the Java server side are:
(GetHeartbeat) Protocol message end-group tag did not match expected tag
and
(GetHeartbeat2)
Message: [org.java_websocket.exceptions.InvalidDataException: java.nio.charset.MalformedInputException: Input length = 1
at org.java_websocket.util.Charsetfunctions.stringUtf8(Charsetfunctions.java:80)
at org.java_websocket.WebSocketImpl.deliverMessage(WebSocketImpl.java:561)
at org.java_websocket.WebSocketImpl.decodeFrames(WebSocketImpl.java:328)
at org.java_websocket.WebSocketImpl.decode(WebSocketImpl.java:149)
at org.java_websocket.server.WebSocketServer$WebSocketWorker.run(WebSocketServer.java:593)
Caused by: java.nio.charset.MalformedInputException: Input length = 1
at java.nio.charset.CoderResult.throwException(CoderResult.java:277)
at java.nio.charset.CharsetDecoder.decode(CharsetDecoder.java:798)
at org.java_websocket.util.Charsetfunctions.stringUtf8(Charsetfunctions.java:77)

Solution
Also posted this question on protobuf group
Credit to Christopher Head and Ilia Mirkin for providing input on the google group
https://groups.google.com/forum/#!topic/protobuf/Cp7zWiWok9I
response.response_proto = base64.b64encode(heartbeatResult.SerializeToString())
self.sendMessage(response.SerializeToString())
FYI, Ilia also suggested base64 encoding the entire message but this seems to be working at the moment.

Related

CreateProcessAsUserW error code 6 Invalid Handle JNA

I'm using the JNA to call the Windows API. I want to start a process (doesn't matter which) as a specific user. The two API calls I use are:
LogonUserW
CreateProcessAsUserW
LogonUserW succeeds, but CreateProcessAsUserW fails with Error 6. According to the Windows System Error Codes Doc, this corresponds to "ERROR_INVALID_HANDLE".
As far as I can tell, the only handle I pass in is the user handle. I don't see what could be wrong with that. According to the LogonUserW doc,
In most cases, the returned handle is a primary token that you can use in calls to the CreateProcessAsUser function. However, if you specify the LOGON32_LOGON_NETWORK flag, LogonUser returns an impersonation token that you cannot use in CreateProcessAsUser unless you call DuplicateTokenEx to convert it to a primary token.
However, I don't use LOGON32_LOGON_NETWORK.
Some of the struct parameters have handles, but I either pass NULL or they are populated by the API call instead of by me.
Here's the meat of my code:
final PointerByReference userPrimaryToken =
new PointerByReference();
System.out.printf(
"ptr.peer = %d\n",
Pointer.nativeValue(userPrimaryToken.getValue())
);
final boolean logonOk = MyWinBase.INSTANCE.LogonUserW(
toCString(<my-username>), // hidden
toCString("ANT"),
toCString(<my-password>), // hidden
/* This logon type is intended for batch servers, where
processes may be executing on behalf of a user without their
direct intervention. This type is also for higher
performance servers that process many plaintext
authentication attempts at a time, such as mail or web
servers.*/
WinBase.LOGON32_LOGON_BATCH,
WinBase.LOGON32_PROVIDER_DEFAULT,
userPrimaryToken
);
System.out.printf("ok = %b\n", logonOk);
System.out.printf(
"ptr.peer = %d\n",
Pointer.nativeValue(userPrimaryToken.getValue())
);
final STARTUPINFOW.ByReference startupInfoW =
new STARTUPINFOW.ByReference();
startupInfoW.cb = startupInfoW.size();
startupInfoW.lpReserved = Pointer.NULL;
startupInfoW.lpDesktop = Pointer.NULL;
startupInfoW.lpTitle = Pointer.NULL;
startupInfoW.dwFlags
= startupInfoW.dwX = startupInfoW.dwY
= startupInfoW.dwXSize = startupInfoW.dwYSize
= startupInfoW.dwXCountChars = startupInfoW.dwYCountChars
= startupInfoW.dwFillAttribute
= startupInfoW.wShowWindow
= 0;
startupInfoW.cbReserved2 = 0;
startupInfoW.lpReserved2 = Pointer.NULL;
startupInfoW.hStdInput = startupInfoW.hStdOutput
= startupInfoW.hStdError
= Pointer.NULL;
final PROCESS_INFORMATION.ByReference processInformation =
new PROCESS_INFORMATION.ByReference();
processInformation.hProcess = processInformation.hThread
= Pointer.NULL;
processInformation.dwProcessId = processInformation.dwThreadId
= 0;
final boolean createProcessOk = MyProcessThreadsApi.INSTANCE
.CreateProcessAsUserW(
userPrimaryToken.getPointer(),
toCString("C:\\Windows\\System32\\cmd.exe"),
// execute and terminate
toCString("/c whoami > whoami.txt"),
Pointer.NULL,
Pointer.NULL,
false,
WinBase.CREATE_UNICODE_ENVIRONMENT,
new PointerByReference(),
Pointer.NULL,
startupInfoW,
processInformation
);
System.out.printf("ok = %b\n", createProcessOk);
System.out.printf(
"dwProcessId = %d\n", processInformation.dwProcessId
);
System.out.printf(
"last err code = %d\n",
ErrHandlingApi.INSTANCE.GetLastError()
);
Here's my output:
ptr.peer = 0
ok = true
ptr.peer = 1040
ok = false
dwProcessId = 0
last err code = 6
Any suggestions?
Looking at this piece of code:
final PointerByReference userPrimaryToken = ...;
Online documentation says it represents a pointer to pointer, C notation void**
https://java-native-access.github.io/jna/4.2.1/com/sun/jna/ptr/PointerByReference.html
On documentation for LogonUser it expects a PHALDLE pointer to a HANDLE, which is similar to a pointer to pointer, since HANDLE is similar to pointer (it is declared as typedef void *HANDLE;).
https://learn.microsoft.com/en-us/windows/win32/api/winbase/nf-winbase-logonuserw
BOOL LogonUserW(
....
DWORD dwLogonProvider,
PHANDLE phToken
);
But in documentation to CreateProcessAsUser is specified that this function accepts a HANDLE, not a PHANDLE
https://learn.microsoft.com/en-us/windows/win32/api/processthreadsapi/nf-processthreadsapi-createprocessasuserw
BOOL CreateProcessAsUserW(
HANDLE hToken,
LPCWSTR lpApplicationName,
....
);
So I would expect you to pass rather getValue than getPointer. By using getPointer you get the pointer itself, which is most probably the pointer to pointer in your case. I don't know JNA, but expectances are from knowledge of WinAPI
final boolean createProcessOk = MyProcessThreadsApi.INSTANCE
.CreateProcessAsUserW(
userPrimaryToken.getValue(),
....
);

Microservice communication with python in Asynchronous manner

I want to trigger Python script from my Spring boot microservices im Asynchronous manner, SO that my Microservice will be notified once the execution of python script completes.Can any one suggest the best approach for this? appreciated if any one provide some reference to sample code.
Thanks in advance!!!
Thanks,
Sudheer
here is very good Example for something like that:
Source: http://code.activestate.com/recipes/511454-an-asynchronous-http-server-with-cgi-support/
import SimpleAsyncServer
# =============================================================
# An implementation of the HTTP protocol, supporting persistent
# connections and CGI
# =============================================================
import sys
import os
import traceback
import datetime
import mimetypes
import urlparse
import urllib
import cStringIO
class HTTP(SimpleAsyncServer.ClientHandler):
# parameters to override if necessary
root = os.getcwd() # the directory to serve files from
cgi_directories = ['/cgi-bin'] # subdirectories for cgi scripts
logging = True # print logging info for each request ?
blocksize = 2 << 16 # size of blocks to read from files and send
def request_complete(self):
"""In the HTTP protocol, a request is complete if the "end of headers"
sequence ('\r\n\r\n') has been received
If the request is POST, stores the request body in a StringIO before
returning True"""
terminator = self.incoming.find('\r\n\r\n')
if terminator == -1:
return False
lines = self.incoming[:terminator].split('\r\n')
self.requestline = lines[0]
try:
self.method,self.url,self.protocol = lines[0].strip().split()
except:
self.method = None # indicates bad request
return True
# put request headers in a dictionary
self.headers = {}
for line in lines[1:]:
k,v = line.split(':',1)
self.headers[k.lower().strip()] = v.strip()
# persistent connection
close_conn = self.headers.get("connection","")
if (self.protocol == "HTTP/1.1"
and close_conn.lower() == "keep-alive"):
self.close_when_done = False
# parse the url
scheme,netloc,path,params,query,fragment = urlparse.urlparse(self.url)
self.path,self.rest = path,(params,query,fragment)
if self.method == 'POST':
# for POST requests, read the request body
# its length must be specified in the content-length header
content_length = int(self.headers.get('content-length',0))
body = self.incoming[terminator+4:]
# request is incomplete if not all message body received
if len(body)<content_length:
return False
f_body = cStringIO.StringIO(body)
f_body.seek(0)
sys.stdin = f_body # compatibility with CGI
return True
def make_response(self):
"""Build the response : a list of strings or files"""
if self.method is None: # bad request
return self.error_resp(400,'Bad request : %s' %self.requestline)
resp_headers, resp_body, resp_file = '','',None
if not self.method in ['GET','POST','HEAD']:
return self.err_resp(501,'Unsupported method (%s)' %self.method)
else:
file_name = self.file_name = self.translate_path()
if not os.path.exists(file_name):
return self.err_resp(404,'File not found')
elif self.managed():
response = self.mngt_method()
else:
ext = os.path.splitext(file_name)[1]
c_type = mimetypes.types_map.get(ext,'text/plain')
resp_line = "%s 200 Ok\r\n" %self.protocol
size = os.stat(file_name).st_size
resp_headers = "Content-Type: %s\r\n" %c_type
resp_headers += "Content-Length: %s\r\n" %size
resp_headers += '\r\n'
if self.method == "HEAD":
resp_string = resp_line + resp_headers
elif size > HTTP.blocksize:
resp_string = resp_line + resp_headers
resp_file = open(file_name,'rb')
else:
resp_string = resp_line + resp_headers + \
open(file_name,'rb').read()
response = [resp_string]
if resp_file:
response.append(resp_file)
self.log(200)
return response
def translate_path(self):
"""Translate URL path into a path in the file system"""
return os.path.join(HTTP.root,*self.path.split('/'))
def managed(self):
"""Test if the request can be processed by a specific method
If so, set self.mngt_method to the method used
This implementation tests if the script is in a cgi directory"""
if self.is_cgi():
self.mngt_method = self.run_cgi
return True
return False
def is_cgi(self):
"""Test if url in a cgi directory"""
for path in self.cgi_directories:
if self.path.startswith(path):
rest = self.path[len(path):]
if not rest or rest.startswith('/'):
return True
return False
def run_cgi(self):
# set CGI environment variables
self.make_cgi_env()
# redirect print statements to a cStringIO
save_stdout = sys.stdout
sys.stdout = cStringIO.StringIO()
# run the script
try:
execfile(self.file_name)
except:
sys.stdout = cStringIO.StringIO()
sys.stdout.write("Content-type:text/plain\r\n\r\n")
traceback.print_exc(file=sys.stdout)
response = sys.stdout.getvalue()
if self.method == "HEAD":
# for HEAD request, don't send message body even if the script
# returns one (RFC 3875)
head_lines = []
for line in response.split('\n'):
if not line:
break
head_lines.append(line)
response = '\n'.join(head_lines)
sys.stdout = save_stdout # restore sys.stdout
# close connection in case there is no content-length header
self.close_when_done = True
resp_line = "%s 200 Ok\r\n" %self.protocol
return [resp_line + response]
def make_cgi_env(self):
"""Set CGI environment variables"""
env = {}
env['SERVER_SOFTWARE'] = "AsyncServer"
env['SERVER_NAME'] = "AsyncServer"
env['GATEWAY_INTERFACE'] = 'CGI/1.1'
env['DOCUMENT_ROOT'] = HTTP.root
env['SERVER_PROTOCOL'] = "HTTP/1.1"
env['SERVER_PORT'] = str(self.server.port)
env['REQUEST_METHOD'] = self.method
env['REQUEST_URI'] = self.url
env['PATH_TRANSLATED'] = self.translate_path()
env['SCRIPT_NAME'] = self.path
env['PATH_INFO'] = urlparse.urlunparse(("","","",self.rest[0],"",""))
env['QUERY_STRING'] = self.rest[1]
if not self.host == self.client_address[0]:
env['REMOTE_HOST'] = self.host
env['REMOTE_ADDR'] = self.client_address[0]
env['CONTENT_LENGTH'] = str(self.headers.get('content-length',''))
for k in ['USER_AGENT','COOKIE','ACCEPT','ACCEPT_CHARSET',
'ACCEPT_ENCODING','ACCEPT_LANGUAGE','CONNECTION']:
hdr = k.lower().replace("_","-")
env['HTTP_%s' %k.upper()] = str(self.headers.get(hdr,''))
os.environ.update(env)
def err_resp(self,code,msg):
"""Return an error message"""
resp_line = "%s %s %s\r\n" %(self.protocol,code,msg)
self.close_when_done = True
self.log(code)
return [resp_line]
def log(self,code):
"""Write a trace of the request on stderr"""
if HTTP.logging:
date_str = datetime.datetime.now().strftime('[%d/%b/%Y %H:%M:%S]')
sys.stderr.write('%s - - %s "%s" %s\n' %(self.host,
date_str,self.requestline,code))
if __name__=="__main__":
# launch the server on the specified port
port = 8081
print "Asynchronous HTTP server running on port %s" %port
print "Press Ctrl+C to stop"
server = SimpleAsyncServer.Server('', port)
HTTP.logging = False
try:
SimpleAsyncServer.loop(server,HTTP)
except KeyboardInterrupt:
for s in server.client_handlers:
server.close_client(s)
print 'Ctrl+C pressed. Closing'
One way of doing asynchronous execution in Python would be to use the celery framework. It's really simple to install and setup (basically just pip install), and the semantic for calling a method asynchronously is super clean.
If you are not too bound to Spring you could even switch to the pymacaron microservice framework (basically a flask app that uses swagger to spawn a REST API). Pymacaron (http://pymacaron.com/) supports asynchronous calls by default, via the pymacaron-async module. See the examples here: http://pymacaron.com/async.html

How To Decode Protocol Buffer String in .Java file

I've searched all around for name to call this, but i can't find any....
Its in a .java file sent to me by a friend(He Thought i would decode it straightaway), without knowing i'm also a noob of this....
This is the String data i want to decode without compiling the Java file.
String[] descriptorData = {
"\n0com/google/javascript/jscomp/function_" +
"info.proto\022\006jscomp\"\277\002\n\026FunctionInformati" +
"onMap\0223\n\005entry\030\001 \003(\n2$.jscomp.FunctionIn" +
"formationMap.Entry\0225\n\006module\030e \003(\n2%.jsc" +
"omp.FunctionInformationMap.Module\032\207\001\n\005En" +
"try\022\n\n\002id\030\002 \002(\005\022\023\n\013source_name\030\003 \002(\t\022\023\n\013" +
"line_number\030\004 \002(\005\022\023\n\013module_name\030\005 \002(\t\022\014" +
"\n\004size\030\006 \002(\005\022\014\n\004name\030\007 \002(\t\022\027\n\017compiled_s" +
"ource\030\010 \002(\t\032/\n\006Module\022\014\n\004name\030f \002(\t\022\027\n\017c" +
"ompiled_source\030g \002(\tB \n\034com.google.javas","cript.jscompP\001"
What you see there is if I'm not mistaken a piece of auto generated code that describes
message FunctionInformationMap {
repeated group Entry = 1 {
required int32 id = 2;
required string source_name = 3;
required int32 line_number = 4;
required string module_name = 5;
required int32 size = 6;
required string name = 7;
required string compiled_source = 8;
}
}
It's here
https://github.com/google/closure-compiler/blob/master/src/com/google/javascript/jscomp/function_info.proto
and the generated code here can be found for example here
https://code.google.com/p/closure-compiler/source/browse/gen/com/google/javascript/jscomp/FunctionInfo.java?name=v20140407
Ps: I just googled "com/google/javascript/jscomp/function_info.proto" but you can actually reverse the process. Hints here for example https://www.sysdream.com/reverse-engineering-protobuf-apps
I Simply solved this by using the System.Out.PrintIn to print out the string data to a TXT file...
That's It..
Thanks...

How to use Any protobuf type in Java?

The code below doesn't work. Is it the best way to use Any? I'm trying to read from/write to Any and identify the type stored by Any.
My .proto:
syntax = "proto3";
import "google/protobuf/any.proto";
message CommandListPrinters {
}
message Commands {
int32 id = 1;
repeated google.protobuf.Any command = 2;
}
my.java:
CommandListPrinters commandListPrinters = CommandListPrinters.newBuilder().build();
Any any = Any.newBuilder()
.setValue(commandListPrinters.toByteString()).build();
Commands.Builder commandsBuilder = Commands.newBuilder().setId(0);
commandsBuilder.addCommand(any);
Commands commands = commandsBuilder.build();
//
byte [] ba = commands.toByteArray();
Commands cmds2 = Commands.parseFrom(ba);
for (Any any2 : cmds2.getCommandList()) {
Descriptor fe = any2.getDescriptorForType();
// This IF is FALSE;
if (fe.equals(CommandListPrinters.getDescriptor()) ) {
CommandListPrinters cmdLR = CommandListPrinters.parseFrom(any2.getValue());
}
}
I'm new to stackoverflow, so I can't comment, but I was wondering:
why does your CommandListPrinters message has no fields?
Calling
CommandListPrinters.getDescriptor()
makes me think CommandListPrinters should have a field descriptor. Also the Any proto doesn't have a field named descriptor_for_type either. See https://github.com/google/protobuf/blob/master/src/google/protobuf/any.proto
So
Descriptor fe = any2.getDescriptorForType();
doesn't makes much sense either.

Getting output parameter value set by VBScript (WMI) method in java via JACOB

Am trying to convert a VBScript to java using JACOB - Java COM bridge library.
'Create' method in VBScript accepts a [out] param in it's method and it sets it upon method execution and i couldn't figure out how to retrieve it back via JACOB.
VBScript in question:
Function CreateProcess(strComputer, strCommand)
Dim objWMIService, objProcess
Set objWMIService = GetObject("winmgmts:" & "{impersonationLevel=impersonate}!\\" & strComputer & "\root\cimv2")
Set objProcess = objWMIService.Get("Win32_Process")
errReturn = objProcess.Create (strCommand, Null, Null, intProcessID)
Set objWMIService = Nothing
Set objProcess = Nothing
CreateProcess = intProcessID
End Function
intProcessID is [out] param set after method execution. (Create API contract)
Converted java code(incomplete and modified slightly for demonstration):
public static void createProcess() {
String host = "localhost";
String connectStr = String
.format("winmgmts:{impersonationLevel=impersonate}!\\\\%s\\root\\CIMV2",
host);
ActiveXComponent axWMI = new ActiveXComponent(connectStr);
Variant vCollection = axWMI.invoke("get", new Variant("Win32_Process"));
Dispatch d = vCollection.toDispatch();
Integer processId = null;
int result = Dispatch.call(d, "Create", "notepad.exe", null, null, processId)
.toInt();
System.out.println("Result:" + result);
// WORKS FINE until here i.e. notepad launches properly, however processId still seems to be null. Following commented code is wrong - doesn't work
//Variant v = Dispatch.get(d, "processId"); // even ProcessId doesn't work
//int pId = v.getInt();
//System.out.println("process id:"
// + pId);
// what is the right way to get the process ID set by 'Create' method?
}
Would be great if you could provide some pointers or relevant code. Ask me more if needed. Thanks in advance.
Replacing
Integer processId = null;
with
Variant processId = new Variant(0, true);
should solve the problem. You should then have process ID of the notepad.exe process in the processId variant, and it can be fetched by
processId.getIntRef()

Categories