How to process MultiLine input log file in Spark using Java - java

I am new to Spark and It seems very confusing to me. I had gone through the spark documentation for Java API But couldn't figure out the way to solve my problem.
I have to process a logfile in spark-Java and have very little time left for the same. Below is the log file that contains the device records(device id, decription, ip address, status) span over multiple lines.
It also contains some other log information which I am not bothered about.
How can I get the device information log from this huge log file.
Any help is much appreciated.
Input Log Data :
!
!
!
device AGHK75
description "Optical Line Terminal"
ip address 1.11.111.12/10
status "FAILED"
!
device AGHK78
description "Optical Line Terminal"
ip address 1.11.111.12/10
status "ACTIVE"
!
!
context local
!
no ip domain-lookup
!
interface IPA1_A2P_1_OAM
description To_A2P_1_OAM
ip address 1.11.111.12/10
propagate qos from ip class-map ip-to-pd
!
interface IPA1_OAM_loopback loopback
description SE1200_IPA-1_OAM_loopback
ip address 1.11.111.12/10
ip source-address telnet snmp ssh radius tacacs+ syslog dhcp-server tftp ftp icmp-dest-unreachable icmp-time-exceed netop flow-ip
What I have done so far is:
Java Code
JavaRDD<String> logData = sc.textFile("logFile").cache();
List<String> deviceRDD = logData.filter(new Function<String, Boolean>() {
Boolean check=false;
public Boolean call(String s) {
if(s.contains("device") ||(check == true && ( s.contains("description") || s.contains("ip address"))))
check=true;
else if(check==true && s.contains("status")){
check=false;
return true;
}
else
check=false;
return check; }
}).collect();
Current Output :
device AGHK75
description "Optical Line Terminal"
ip address 1.11.111.12/10
status "FAILED"
device AGHK78
description "Optical Line Terminal"
ip address 1.11.111.12/10
status "ACTIVE"
Expected Output is:
AGHK75,"Optical Line Terminal",1.11.111.12/10,"FAILED"
AGHK78,"Optical Line Terminal",1.11.111.12/10,"ACTIVE"

You can use sc.wholeTextFiles("logFile") for getting the data as key,value pair of where key will be the file name and value as data in it.
Then you can use some string operation for splitting of the data as per the start and end delimiter of single log data with "!" and do a filter first for checking if the first word is device and then do a flatMap on it which will make it as singleLog text RDD.
and then get the data from it using the map.
Please try it and let me know whether if this logic is working for you.
Added code in Spark Scala:
val ipData = sc.wholeTextFiles("abc.log")
val ipSingleLog = ipData.flatMap(x=>x._2.split("!")).filter(x=>x.trim.startsWith("device"))
val logData = ipSingleLog.map(x=>{
val rowData = x.split("\n")
var device = ""
var description = ""
var ipAddress = ""
var status = ""
for (data <- rowData){
if(data.trim().startsWith("device")){
device = data.split("device")(1)
}else if(data.trim().startsWith("description")){
description = data.split("description")(1)
}else if(data.trim().startsWith("ip address")){
ipAddress = data.split("ip address")(1)
}else if(data.trim().startsWith("status")){
status = data.split("status")(1)
}
}
(device,description,ipAddress,status)
})
logData.foreach(println)

Spark will take each line as a separate item with sc.textFile. You can get it to split on a different char using sc.hadoopConfiguration().set("textinputformat.record.delimiter", "!").
#Test
public void test() throws ParseException, IOException {
hadoop.write("/test.txt", "line 1\nline 2\n!\nline 3\nline 4");
JavaSparkContext sc = spark.getContext();
sc.hadoopConfiguration().set("textinputformat.record.delimiter", "!");
System.out.println(sc.textFile(hadoop.getMfs().getUri() + "/test.txt").collect());
assertThat(sc.textFile(hadoop.getMfs().getUri() + "/test.txt").count(), is(2L));
}

I believe the only correct way that works everywhere is
Configuration hadoopConf = new Configuration();
hadoopConf.set("textinputformat.record.delimiter", "delimiter");
JavaPairRDD<LongWritable, Text> input = jsc.newAPIHadoopFile(path,
TextInputFormat.class, LongWritable.class, Text.class, hadoopConf);
There are issues in hadoop related code. Depending on size of the input file it produces additional records: MAPREDUCE-6549,MAPREDUCE-5948. It certainly works starting with 2.7.2.
Even though as mlk suggests using spark context would perfectly work, it'll fail in case you try to read another file with different delimiter using the same spark context. By default the delimiter is new line symbol and it'll be changed as soon as this option is applied.
The reason is that spark context shares hadoopConfiguration object and it's hard to reason, where exactly this value is going to be needed. As a workaround the one might materialize RDD and cache it, but it's still might happen that the same RDD would be recomputed.
Given way would work everywhere, because every time it uses new Configuration.

Related

Get drive letters of USB 3.0 devices (Java under Windows)

My Java program needs to get a list of the drive letters of connected USB devices, but only those that support USB 3.0 (both the device and the USB port it is plugged into, so that it works with high speed).
Currently I try to use WMI through a PowerShell command my Java programm executes.
I already found this: Powershell: Grab USB Drive letter. But it would also list USB 2.0 devices.
Regarding version detection I found this: How to check the version of the available USB ports? - The PowerShell command I tried is Get-WmiObject Win32_USBHub. This brings up several problems. First: It lists far more stuff than only USB drives (I think also all the USB hubs of my PC). Second: Even though there is a field USBVersion for all items in the list it is always empty.
Update
The essence of my research over the last days is, that there are 2 realms of information I need to connect.
Drives / Logical Drives
Drive Letter
BusType (is equal to "USB" for my matter)
USB devices
Vendor ID and Product ID (VID&PID)
bcdUSB (value within the usb device descriptor, indicating USB Version)
For a given drive letter I need to find the bcdUSB value. But I haven't found a way to get the drive corresponding to a USB device.
What I tried so far
WMI over PowerShell
Relevant commands I found are
Get-Disk // Get BusType
gwmi Win32_LogicalDisk // Get drive letter
// Those make the connection between disk and logical disk
gwmi Win32_LogicalDiskToPartition
gwmi Win32_LogicalDiskToPartition
Even though I get the BusType I couldn't make a connection to bcdUSB
usb4java (Link)
I only get information from the USB device realm here. I can load devices and see ther VID&PID and the bcdUSB value, but no way to map this to drives and drive letters.
lsusb via Cygwin
According to this post the linux command is easier to handle than WMI. So I tried to use it under Windows. But I like usb4java I only got VID&PID + bcdUSB, not the mount point (drive letter).
Searching the Windows Registry
I did a few string searchs in the Windows registry. No success.
Reading Windows Event log
I thought about ovserving Windows events to detect what Drive and what USB device connect at the same time. I didn't even find events when plugging in a USB stick.
Maybe this is what you are looking for:
Find Windows Drive Letter of a removable disk from USB VID/PID
At least someone marked the answer as working... :-)
Since the suggested Link solves this problem for C# not Java and leaves out one step, I'll post my final code here.
Summary
In Java
Use USB4Java to find all connected USB devices with bcdUSB=0x0300
Get Vendor ID and Product ID (VID&PID) for that devices
Via Powershell (with jPowerShell)
Get PnPEntity for given VID&PID
Get related USB Controller
Find associator of that USB Controller that is associated with a disk drive
Get that Disk drive
Get related disk partition
Get related logical disk -> LogicalDisk.DeviceID = Drive Letter
Code
Java class:
class UsbDetector {
private PowerShell shell;
#PostConstruct
private void init() {
shell = com.profesorfalken.jpowershell.PowerShell.openSession();
}
#OnDestroy
private void onShutdownHook() {
shell.close();
}
/**
* Get drive letters of USB 3.0 devices.
*/
public List<String> getDriveLettersForUsb3Devices() throws IOException, UsbException {
List<UsbDevice> devicesUSB3 = getAllUsb3Devices();
ImmutableList.Builder<String> driveLetterList = ImmutableList.builder();
for (UsbDevice device : devicesUSB3) {
String vidAndPid = getVidAndPid(device);
String powerShellScript = buildScript(vidAndPid);
String driveLetter = executeOnPowerShell(powerShellScript);
driveLetterList.add(driveLetter);
}
return driveLetterList.build();
}
private String executeOnPowerShell(String powerShellScript) {
InputStream psScriptStream = new ByteArrayInputStream(powerShellScript.getBytes());
BufferedReader psScriptReader = new BufferedReader(new InputStreamReader(psScriptStream));
PowerShellResponse response = shell.executeScript(psScriptReader);
return response.getCommandOutput();
}
private String buildScript(String vidAndPid) throws IOException {
InputStream psScriptStream =
getClass().getClassLoader().getResourceAsStream("GetUsbDrives.ps1");
String psScript = IOUtil.toString(psScriptStream);
psScript = String.format("$input=\"%s\"", vidAndPid) + "\n" + psScript;
return psScript;
}
/**
* The Vendor ID and Product ID are necessary to find the device via WMI.
*/
private String getVidAndPid(UsbDevice device) {
short vendorId = device.getUsbDeviceDescriptor().idVendor();
short productId = device.getUsbDeviceDescriptor().idProduct();
String vendorIdHexString = String.format("%04x", vendorId).toUpperCase();
String productIdHexString = String.format("%04x", productId).toUpperCase();
String vidAndPid = String.format("VID_%s&PID_%s", vendorIdHexString, productIdHexString);
return vidAndPid;
}
/**
* From all Usb devices find those with USB 3.0. The value bcdUsb is a hexadecimal coded number
* telling us the USB version.
*/
private List<UsbDevice> getAllUsb3Devices() throws UsbException {
List<UsbDevice> devicesUSB3 = Lists.newArrayList();
UsbServices services = new org.usb4java.javax.Services();
UsbHub hub = services.getRootUsbHub();
List<UsbDevice> devices = getAllUsbDevices(hub);
for (UsbDevice device : devices) {
UsbDeviceDescriptor descriptor = device.getUsbDeviceDescriptor();
short bcdUsb = descriptor.bcdUSB();
String bcdDecoded = DescriptorUtils.decodeBCD(bcdUsb);
if (Objects.equal(bcdDecoded, "3.00")) {
devicesUSB3.add(device);
}
}
return devicesUSB3;
}
/**
* UsbHubs can either mount UsbDevices or further UsbHubs. This method searches through the tree
* of UsbHubs for UsbDevices and returns them as list.
*/
private List<UsbDevice> getAllUsbDevices(UsbHub hub) {
List<UsbDevice> devices = Lists.newArrayList();
List<UsbDevice> attachedDevices = hub.getAttachedUsbDevices();
for (UsbDevice device : attachedDevices) {
if (device instanceof UsbHub) {
List<UsbDevice> subdevices = getAllUsbDevices((UsbHub) device);
devices.addAll(subdevices);
} else {
devices.add(device);
}
}
return devices;
}
}
PowerShell script:
# $input = "VID_XXXX&PID_XXXX (this line is added in Java Code)
# For given VID and PID of a USB device we search for
# the corresponding logical disk to get the drive letter.
# The chain of objects is:
# PnPEntity (PnP = Plug and Play)
# -> USBController
# -> Some associator of USBController that has a related disk drive
# -> diskDrive
# -> diskPartition
# -> logicalDisk
# Find PnPEntity for given VID and PID
$usbPnPEntity = (gwmi Win32_PnPEntity | where DeviceID -match $input)
# Get USB Controller related to PnP Entity
$usbController = $usbPnPEntity.getRelated("Win32_USBController")
$usbControllerID = $usbController.DeviceID
# Find objects associated with the USB Controller
$query = "ASSOCIATORS OF {Win32_USBController.DeviceID='$usbControllerID'}"
$associators = ([wmisearcher]$query).get()
# Search through associators
foreach ($associator in $associators) {
# Find associator that is related to a disk Drive
$assoDeviceID = $associator.DeviceID
$diskDrive = (gwmi win32_diskdrive | where PNPDeviceID -eq $assoDeviceID)
if($diskDrive){
# Get logical Disk related to the disk drive
$logicalDisk = $diskDrive.getRelated("Win32_DiskPartition").getRelated("Win32_LogicalDisk")
# Print device ID which is the drive letter (e.g. "C:")
$logicalDisk.DeviceID
break
}
}
Maven dependencies:
<dependency>
<groupId>org.usb4java</groupId>
<artifactId>usb4java-javax</artifactId>
<version>1.3.0</version>
</dependency>
<dependency>
<groupId>com.profesorfalken</groupId>
<artifactId>jPowerShell</artifactId>
<version>3.1.1</version>
</dependency>

How to use the pyjnius read and write arguments for Bluetooth getInputStream and getOutputStream?

I have been fumbling with getting data from a bluetooth RS232 converter using the python for android jnius library. I had thought that it would be as simple as the PySerial library, but as I am still new to java the implementation is quite different. I created the bluetooth connection quite easily, but as soon as I try to read or write data I get a jnius.jnius.JavaException that No methods matching your arguments, and that the methods available for read are: '()I', '([B)I', '([BII)I' and for write are '(I)V', '([B)V', '([BII)V'. I tried finding this in the developer android docs as well as DuckDuckGoing it but with no clarity. I also tried the BufferedReader example (Here) using the readLine() method, but I am constantly getting an error: JavaException: JVM exception occurred: Attempt to invoke virtual method 'int java.io.InputStream.read(byte[], int, int)' on a null object reference.
Can someone please point me to the documentation to understand the above read and write arguments?
Also, please help me to understand why the read, readLine() and write objects does not return any data?
Java libraries I call:
BluetoothAdapter = autoclass('android.bluetooth.BluetoothAdapter')
BluetoothDevice = autoclass('android.bluetooth.BluetoothDevice')
BluetoothSocket = autoclass('android.bluetooth.BluetoothSocket')
InputStreamReader = autoclass('java.io.InputStreamReader')
BufferedReader = autoclass('java.io.BufferedReader')
UUID = autoclass('java.util.UUID')
System = autoclass('java.lang.System')
Connecting code (got this from Github and above link):
def get_socket_stream(self, name):
paired_devices = BluetoothAdapter.getDefaultAdapter().getBondedDevices().toArray()
self.rfsocket == None
for device in paired_devices:
if device.getName() == name:
if device.bluetoothEnabled:
if not device.connected:
self.rfsocket = device.createInsecureRfcommSocketToServiceRecord(
UUID.fromString("00001101-0000-1000-8000-00805f9b34fb"))
self.reader = InputStreamReader(self.rfsocket.getInputStream(), 'LATIN-1')
recv_stream = BufferedReader(self.reader)
send_stream = self.rfsocket.getOutputStream()
break
print(self.rfsocket.getRemoteDevice().getName())
if self.rfsocket.port <= 0:
self.rfsocket = device.createRfcommSocket(1)
if not self.rfsocket.connected:
print('port two: ',self.rfsocket.port)
self.rfsocket.connect()
Read and write code (source: same as above link):
def GetBSerial(self):
self.messager('Trying Serial')
self.recv_stream, self.send_stream = self.get_socket_stream(devicename)
if not self.rfsocket.connected and self.recv_stream == None:
print("Get paired device failed")
else:
print('Initiating trigger')
self.weight_ticker()
def GetInput(self):
self.send_stream.write("Hallo\r\n")
self.send_stream.flush
if self.rfsocket.connected and self.recv_stream != None:
if self.weigh_tme > 0:
while self.recv_stream.ready != None:
try:
self.scale_output = self.recv_stream.readLine()
except jnius.jnius.JavaException as e:
print("JavaException: ", e, self.rfsocket.connected)
except ValueError as e:
print("Misc error: ", e)
try:
self.show_input(self.scale_output)
except ValueError:
pass
Update:
So I finally got the Input using the readLine() method to not return an error and also return the string. I cleaned things up a bit, but the code does not differ much. Main thing though is I checked whether the device != None and if rfsocket.connected: before creating getInputStream within my eventloop, so as not to re-create the socket object. Have to test more to see where was the main problem. Still do not no what the arguments are of the read and write method. The readLine() method returns the string intermitantly or not at all and my eventloop seems to not work with the readLine() method.
Update on update:
The event loop works again. My bad, I did not call the trigger object correctly. The readLine() method has a strange pattern in that on the first read it gives me JavaException: JVM exception occurred: Attempt to invoke virtual method 'int java.io.InputStream.available()' on a null object reference, subsequent reads gives me pieces of the expected string or an empty string. I had similar string bits and pieces returned when I received the data via a hardline using PySerial. The solution was resetting the input buffer. Is there something similar in the above java libraries?
Finally cracked the exceptions:
Yes, this is exciting! After many hours I had noticed that I could not get the Input anymore only exceptions. I tried the BufferedInputStream library and got the same result, there was no more intermittant read. Why? So I re-Apk'd the main file of last night and boom intermittant Input read once again.
The reason was if I create the java objects when the rfsocket Bluetooth object is not connected to the specified port, Null objects was initialised on a different port which for some reason was not seen by the if blocks self.recv_stream is not None and self.recv_stream != None. Probably because they are not Null objects but Null for the subsequent port 1 socket connection I specified.
The readline() work as is in my example, the read() takes three arguments bytes(), int offset, int len(bytes() which is not clear from the jnius.jnius.exception hieroglyphics' message. Still figuring out the write method. One thing that you can specify in the BufferedReader method is a 2nd argument for the chunk size you want to read or in java speak defaultCharBufferSize.
So I am posting my answers as I have solved them all.
To connect to bluetooth I built on the GitHub example by following the suggestions on the android developers site. I had to set my own socket explicitly before creating the getOutputStream and getInputStream java objects otherwise the ports of the connection and objects will not be the same. You only need to call the GetBSerial() to connect
def get_socket_stream(self, name):
defaultCharBufferSize = 8192
try:
blueAdapt = BluetoothAdapter.getDefaultAdapter()
if self.rfsocket is not None:
if self.rfsocket.connected:
reader = InputStreamReader(self.rfsocket.getInputStream(), getEncode)
recv_stream = BufferedReader(reader)
send_stream = self.rfsocket.getOutputStream()
else:
self.rfsocket = self.device.createRfcommSocketToServiceRecord(UUID.fromString(getUuid))
if self.get_port_connect():
reader = InputStreamReader(self.rfsocket.getInputStream(), getEncode)
recv_stream = BufferedReader(reader, defaultCharBufferSize)
send_stream = self.rfsocket.getOutputStream()
else:
if blueAdapt is not None:
if blueAdapt.isEnabled():
paired_devices = blueAdapt.getBondedDevices().toArray()
self.rfsocket = None
for self.device in paired_devices:
if self.device.getName() == name:
if self.device.bluetoothEnabled:
self.rfsocket = self.device.createRfcommSocketToServiceRecord(
UUID.fromString(getUuid))
if self.rfsocket is not None:
if self.get_port_connect(): #connect and set the port before creating java objects
reader = InputStreamReader(self.rfsocket.getInputStream(), getEncode)
recv_stream = BufferedReader(reader, defaultCharBufferSize)
send_stream = self.rfsocket.getOutputStream()
break
else:
self.ids.bluet_info.text = '[b]Bluetooth not enabled[/b]'
if recv_stream is not None and send_stream is not None:
return recv_stream, send_stream
else:
return False, False
except UnboundLocalError as e:
return False, False
except TypeError as e:
return False, False
def get_port_connect(self):
try:
if self.rfsocket.port <= 0:
self.rfsocket = self.device.createRfcommSocket(1) #set the port explicitly
if not self.rfsocket.connected:
self.rfsocket.connect()
else:
if not self.rfsocket.connected:
self.rfsocket.connect()
if self.rfsocket.connected:
self.ids.bluet_info.text = '[b]Connected[/b]'
return True
except jnius.jnius.JavaException as e:
self.ids.bluet_info.text = '[b]Cannot connect to socket[/b]'
def GetBSerial(self):
try:
getDevname = self.the.config.get('bluetoothsettings', 'stringbluetdevname')
self.recv_stream, self.send_stream = self.get_socket_stream(getDevname)
except jnius.jnius.JavaException as e:
self.ids.bluet_info.text = '[b]Not Connected[/b]'
I used the readLine() method, but to use the read() method, there are two ways to build a string. Either externally (I only tried this one) or in an Array.
Import:
CharBuilder = autoclass('java.lang.Character')
Externally:
if self.recv_stream.ready() != None:
r = self.recv_stream.read()
theChar = CharBuilder.toChars(r) #gives a tuple of which the first element is a character
self.read += theChar[0]
You have to play around with building your string to know where the string must start.
The first thing about the write() method is it takes a bytes object. So you build a bytearray() and use it as an argument. Example using ESC/POS printer initialise command and a string:
i = [27,64] #ASCII escape integer and at sign integer
pre = bytearray(i)
cmd = 'Hello You\n'.encode('UTF-8')
#extend bytearray
pre.extend(cmd)
self.send_stream.write(pre)
self.send_stream.flush()
Could not figure out how to create a bytearray integer and string in one go, yet.

Using JZMQ with EPGM Transport Is Not Sending or Receiving Data

I'm experimenting with java flavored zmq to test the benefits of using PGM over TCP in my project. So I changed the weather example, from the zmq guide, to use the epgm transport.
Everything compiles and runs, but nothing is being sent or received. If I change the transport back to TCP, the server receives the messages sent from the client and I get the console output I'm expecting.
So, what are the requirements for using PGM? I changed the string, that I'm passing to the bind and connect methods, to follow the zmq api for zmq_pgm: "transport://interface;multicast address:port". That didn't work. I get and invalid argument error whenever I attempt to use this format. So, I simplified it by dropping the interface and semicolon which "works", but I'm not getting any results.
I haven't been able to find a jzmq example that uses pgm/epgm and the api documentation for the java binding does not define the appropriate string format for an endpoint passed to bind or connect. So what am I missing here? Do I have to use different hosts for the client and the server?
One thing of note is that I'm running my code on a VirtualBox VM (Ubuntu 14.04/OSX Mavericks host). I'm not sure if that has anything to do with the issue I'm currently facing.
Server:
public class wuserver {
public static void main (String[] args) throws Exception {
// Prepare our context and publisher
ZMQ.Context context = ZMQ.context(1);
ZMQ.Socket publisher = context.socket(ZMQ.PUB);
publisher.bind("epgm://xx.x.x.xx:5556");
publisher.bind("ipc://weather");
// Initialize random number generator
Random srandom = new Random(System.currentTimeMillis());
while (!Thread.currentThread ().isInterrupted ()) {
// Get values that will fool the boss
int zipcode, temperature, relhumidity;
zipcode = 10000 + srandom.nextInt(10000) ;
temperature = srandom.nextInt(215) - 80 + 1;
relhumidity = srandom.nextInt(50) + 10 + 1;
// Send message to all subscribers
String update = String.format("%05d %d %d", zipcode, temperature, relhumidity);
publisher.send(update, 0);
}
publisher.close ();
context.term ();
}
}
Client:
public class wuclient {
public static void main (String[] args) {
ZMQ.Context context = ZMQ.context(1);
// Socket to talk to server
System.out.println("Collecting updates from weather server");
ZMQ.Socket subscriber = context.socket(ZMQ.SUB);
//subscriber.connect("tcp://localhost:5556");
subscriber.connect("epgm://xx.x.x.xx:5556");
// Subscribe to zipcode, default is NYC, 10001
String filter = (args.length > 0) ? args[0] : "10001 ";
subscriber.subscribe(filter.getBytes());
// Process 100 updates
int update_nbr;
long total_temp = 0;
for (update_nbr = 0; update_nbr < 100; update_nbr++) {
// Use trim to remove the tailing '0' character
String string = subscriber.recvStr(0).trim();
StringTokenizer sscanf = new StringTokenizer(string, " ");
int zipcode = Integer.valueOf(sscanf.nextToken());
int temperature = Integer.valueOf(sscanf.nextToken());
int relhumidity = Integer.valueOf(sscanf.nextToken());
total_temp += temperature;
}
System.out.println("Average temperature for zipcode '"
+ filter + "' was " + (int) (total_temp / update_nbr));
subscriber.close();
context.term();
}
}
There are a couple possibilities:
You need to make sure ZMQ is compiled with the --with-pgm option: see here - but this doesn't appear to be your issue if you're not seeing "protocol not supported"
Using raw pgm requires root privileges because it requires the ability to create raw sockets... but epgm doesn't require that, so it shouldn't be your issue either (I only bring it up because you use the term "pgm/epgm", and you should be aware that they are not equally available in all situations)
What actually appears to be the problem in your case is that pgm/epgm requires support along the network path. In theory, it requires support out to your router, so your application can send a single message and have your router send out multiple messages to each client, but if your server is aware enough, it can probably send out multiple messages immediately and bypass this router support. The problem is, as you correctly guessed, trying to do this all on one host is not supported.
So, you need different hosts for client and server.
Another bit to be aware of is that some virtualization environments--RHEV/Ovirt and libvirt/KVM with the mac_filter option enabled come to mind-- that, by default, neuter one's abilities via (eb|ip)tables to utilize mcast between guests. With libvirt, of course, the solution is to simply set the option to '0' and restart libvirtd. RHEV/Ovirt require a custom plugin.
At any rate, I would suggest putting a sniffer on the network devices on each system you are using and watching to be sure traffic that is exiting the one host is actually visible on the other.

Java ProgramCall.run hangs

Busy trying to Call RPG function from Java and got this example from JamesA. But now I am having trouble, here is my code:
AS400 system = new AS400("MachineName");
ProgramCall program = new ProgramCall(system);
try
{
// Initialise the name of the program to run.
String programName = "/QSYS.LIB/LIBNAME.LIB/FUNNAME.PGM";
// Set up the 3 parameters.
ProgramParameter[] parameterList = new ProgramParameter[2];
// First parameter is to input a name.
AS400Text OperationsItemId = new AS400Text(20);
parameterList[0] = new ProgramParameter(OperationsItemId.toBytes("TestID"));
AS400Text CaseMarkingValue = new AS400Text(20);
parameterList[1] = new ProgramParameter(CaseMarkingValue.toBytes("TestData"));
// Set the program name and parameter list.
program.setProgram(programName, parameterList);
// Run the program.
if (program.run() != true)
{
// Report failure.
System.out.println("Program failed!");
// Show the messages.
AS400Message[] messagelist = program.getMessageList();
for (int i = 0; i < messagelist.length; ++i)
{
// Show each message.
System.out.println(messagelist[i]);
}
}
// Else no error, get output data.
else
{
AS400Text text = new AS400Text(50);
System.out.println(text.toObject(parameterList[1].getOutputData()));
System.out.println(text.toObject(parameterList[2].getOutputData()));
}
}
catch (Exception e)
{
//System.out.println("Program " + program.getProgram() + " issued an exception!");
e.printStackTrace();
}
// Done with the system.
system.disconnectAllServices();
The application Hangs at this lineif (program.run() != true), and I wait for about 10 minutes and then I terminate the application.
Any idea what I am doing wrong?
Edit
Here is the message on the job log:
Client request - run program QSYS/QWCRTVCA.
Client request - run program LIBNAME/FUNNAME.
File P6CASEL2 in library *LIBL not found or inline data file missing.
Error message CPF4101 appeared during OPEN.
Cannot resolve to object YOBPSSR. Type and Subtype X'0201' Authority
FUNNAME insert a row into table P6CASEPF through a view called P6CASEL2. P6CASEL2 is in a different library lets say LIBNAME2. Is there away to maybe set the JobDescription?
Are you sure FUNNAME.PGM is terminating and not hung with a MSGW? Check QSYSOPR for any messages.
Class ProgramCall:
NOTE: When the program runs within the host server job, the library list will be the initial library list specified in the job description in the user profile.
So I saw that my problem is that my library list is not setup, and for some reason, the user we are using, does not have a Job Description. So to over come this I added the following code before calling the program.run()
CommandCall command = new CommandCall(system);
command.run("ADDLIBLE LIB(LIBNAME)");
command.run("ADDLIBLE LIB(LIBNAME2)");
This simply add this LIBNAME, and LIBNAME2 to the user's library list.
Oh yes, the problem is Library list not set ... take a look at this discussion on Midrange.com, there are different work-around ...
http://archive.midrange.com/java400-l/200909/msg00032.html
...
Depe

How do you tell whether a string is an IP or a hostname

So you have a String that is retrieved from an admin web UI (so it is definitely a String). How can you find out whether this string is an IP address or a hostname in Java?
Update: I think I didn't make myself clear, I was more asking if there is anything in the Java SDK that I can use to distinguish between IPs and hostnames? Sorry for the confusion and thanks for everybody who took/will take the time to answer this.
You can use a regular expression with this pattern:
\b(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\b
That will tell you if it's an IPv4 address.
Do we get to make the assumption that it is one or the other, and not something completely different? If so, I'd probably use a regex to see if it matched the "dotted quad" format.
You can see if the string matches the number.number.number.number format, for example:
\b\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\b
will match anything from 0 - 999.
Anything else you can have it default to hostname.
URI validator = new URI(yourString);
That code will validate the IP address or Hostname. (It throws a malformed URI Exception if the string is invalid)
If you are trying to distinguish the two..then I miss read your question.
You can use a security manager with the InetAddress.getByName(addr) call.
If the addr is not a dotted quad, getByName will attempt to perform a connect to do the name lookup, which the security manager can capture as a checkConnect(addr, -1) call, resulting in a thrown SecurityException that you can catch.
You can use System.setSecurityManager() if you're running fully privileged to insert your custom security manager before the getByName call is made.
It is not as simple as it may appear, there are some ambiguities around characters like hyphens, underscore, and square brackets '-', '_', '[]'.
The Java SDK is has some limitations in this area. When using InetAddress.getByName it will go out onto the network to do a DNS name resolution and resolve the address, which is expensive and unnecessary if all you want is to detect host vs address. Also, if an address is written in a slightly different but valid format (common in IPv6) doing a string comparison on the results of InetAddress.getByName will not work.
The IPAddress Java library will do it. The javadoc is available at the link. Disclaimer: I am the project manager.
static void check(HostName host) {
try {
host.validate();
if(host.isAddress()) {
System.out.println("address: " + host.asAddress());
} else {
System.out.println("host name: " + host);
}
} catch(HostNameException e) {
System.out.println(e.getMessage());
}
}
public static void main(String[] args) {
HostName host = new HostName("1.2.3.4");
check(host);
host = new HostName("1.2.a.4");
check(host);
host = new HostName("::1");
check(host);
host = new HostName("[::1]");
check(host);
host = new HostName("1.2.?.4");
check(host);
}
Output:
address: 1.2.3.4
host name: 1.2.a.4
address: ::1
address: ::1
1.2.?.4 Host error: invalid character at index 4
Couldn't you just to a regexp match on it?
Use InetAddress#getAllByName(String hostOrIp) - if hostOrIp is an IP-address the result is an array with single InetAddress and it's .getHostAddress() returns the same string as hostOrIp.
import java.net.InetAddress;
import java.net.UnknownHostException;
import java.util.Arrays;
public class IPvsHostTest {
private static final org.slf4j.Logger LOG = org.slf4j.LoggerFactory.getLogger(IPvsHostTest.class);
#org.junit.Test
public void checkHostValidity() {
Arrays.asList("10.10.10.10", "google.com").forEach( hostname -> isHost(hostname));
}
private void isHost(String ip){
try {
InetAddress[] ips = InetAddress.getAllByName(ip);
LOG.info("IP-addresses for {}", ip);
Arrays.asList(ips).forEach( ia -> {
LOG.info(ia.getHostAddress());
});
} catch (UnknownHostException e) {
LOG.error("Invalid hostname", e);
}
}
}
The output:
IP-addresses for 10.10.10.10
10.10.10.10
IP-addresses for google.com
64.233.164.100
64.233.164.138
64.233.164.139
64.233.164.113
64.233.164.102
64.233.164.101
This code still performs the DNS lookup if a host name is specified, but at least it skips the reverse lookup that may be performed with other approaches:
...
isDottedQuad("1.2.3.4");
isDottedQuad("google.com");
...
boolean isDottedQuad(String hostOrIP) throws UnknownHostException {
InetAddress inet = InetAddress.getByName(hostOrIP);
boolean b = inet.toString().startsWith("/");
System.out.println("Is " + hostOrIP + " dotted quad? " + b + " (" + inet.toString() + ")");
return b;
}
It generates this output:
Is 1.2.3.4 dotted quad? true (/1.2.3.4)
Is google.com dotted quad? false (google.com/172.217.12.238)
Do you think we can expect the toString() behavior to change anytime soon?

Categories