Get drive letters of USB 3.0 devices (Java under Windows) - java

My Java program needs to get a list of the drive letters of connected USB devices, but only those that support USB 3.0 (both the device and the USB port it is plugged into, so that it works with high speed).
Currently I try to use WMI through a PowerShell command my Java programm executes.
I already found this: Powershell: Grab USB Drive letter. But it would also list USB 2.0 devices.
Regarding version detection I found this: How to check the version of the available USB ports? - The PowerShell command I tried is Get-WmiObject Win32_USBHub. This brings up several problems. First: It lists far more stuff than only USB drives (I think also all the USB hubs of my PC). Second: Even though there is a field USBVersion for all items in the list it is always empty.
Update
The essence of my research over the last days is, that there are 2 realms of information I need to connect.
Drives / Logical Drives
Drive Letter
BusType (is equal to "USB" for my matter)
USB devices
Vendor ID and Product ID (VID&PID)
bcdUSB (value within the usb device descriptor, indicating USB Version)
For a given drive letter I need to find the bcdUSB value. But I haven't found a way to get the drive corresponding to a USB device.
What I tried so far
WMI over PowerShell
Relevant commands I found are
Get-Disk // Get BusType
gwmi Win32_LogicalDisk // Get drive letter
// Those make the connection between disk and logical disk
gwmi Win32_LogicalDiskToPartition
gwmi Win32_LogicalDiskToPartition
Even though I get the BusType I couldn't make a connection to bcdUSB
usb4java (Link)
I only get information from the USB device realm here. I can load devices and see ther VID&PID and the bcdUSB value, but no way to map this to drives and drive letters.
lsusb via Cygwin
According to this post the linux command is easier to handle than WMI. So I tried to use it under Windows. But I like usb4java I only got VID&PID + bcdUSB, not the mount point (drive letter).
Searching the Windows Registry
I did a few string searchs in the Windows registry. No success.
Reading Windows Event log
I thought about ovserving Windows events to detect what Drive and what USB device connect at the same time. I didn't even find events when plugging in a USB stick.

Maybe this is what you are looking for:
Find Windows Drive Letter of a removable disk from USB VID/PID
At least someone marked the answer as working... :-)

Since the suggested Link solves this problem for C# not Java and leaves out one step, I'll post my final code here.
Summary
In Java
Use USB4Java to find all connected USB devices with bcdUSB=0x0300
Get Vendor ID and Product ID (VID&PID) for that devices
Via Powershell (with jPowerShell)
Get PnPEntity for given VID&PID
Get related USB Controller
Find associator of that USB Controller that is associated with a disk drive
Get that Disk drive
Get related disk partition
Get related logical disk -> LogicalDisk.DeviceID = Drive Letter
Code
Java class:
class UsbDetector {
private PowerShell shell;
#PostConstruct
private void init() {
shell = com.profesorfalken.jpowershell.PowerShell.openSession();
}
#OnDestroy
private void onShutdownHook() {
shell.close();
}
/**
* Get drive letters of USB 3.0 devices.
*/
public List<String> getDriveLettersForUsb3Devices() throws IOException, UsbException {
List<UsbDevice> devicesUSB3 = getAllUsb3Devices();
ImmutableList.Builder<String> driveLetterList = ImmutableList.builder();
for (UsbDevice device : devicesUSB3) {
String vidAndPid = getVidAndPid(device);
String powerShellScript = buildScript(vidAndPid);
String driveLetter = executeOnPowerShell(powerShellScript);
driveLetterList.add(driveLetter);
}
return driveLetterList.build();
}
private String executeOnPowerShell(String powerShellScript) {
InputStream psScriptStream = new ByteArrayInputStream(powerShellScript.getBytes());
BufferedReader psScriptReader = new BufferedReader(new InputStreamReader(psScriptStream));
PowerShellResponse response = shell.executeScript(psScriptReader);
return response.getCommandOutput();
}
private String buildScript(String vidAndPid) throws IOException {
InputStream psScriptStream =
getClass().getClassLoader().getResourceAsStream("GetUsbDrives.ps1");
String psScript = IOUtil.toString(psScriptStream);
psScript = String.format("$input=\"%s\"", vidAndPid) + "\n" + psScript;
return psScript;
}
/**
* The Vendor ID and Product ID are necessary to find the device via WMI.
*/
private String getVidAndPid(UsbDevice device) {
short vendorId = device.getUsbDeviceDescriptor().idVendor();
short productId = device.getUsbDeviceDescriptor().idProduct();
String vendorIdHexString = String.format("%04x", vendorId).toUpperCase();
String productIdHexString = String.format("%04x", productId).toUpperCase();
String vidAndPid = String.format("VID_%s&PID_%s", vendorIdHexString, productIdHexString);
return vidAndPid;
}
/**
* From all Usb devices find those with USB 3.0. The value bcdUsb is a hexadecimal coded number
* telling us the USB version.
*/
private List<UsbDevice> getAllUsb3Devices() throws UsbException {
List<UsbDevice> devicesUSB3 = Lists.newArrayList();
UsbServices services = new org.usb4java.javax.Services();
UsbHub hub = services.getRootUsbHub();
List<UsbDevice> devices = getAllUsbDevices(hub);
for (UsbDevice device : devices) {
UsbDeviceDescriptor descriptor = device.getUsbDeviceDescriptor();
short bcdUsb = descriptor.bcdUSB();
String bcdDecoded = DescriptorUtils.decodeBCD(bcdUsb);
if (Objects.equal(bcdDecoded, "3.00")) {
devicesUSB3.add(device);
}
}
return devicesUSB3;
}
/**
* UsbHubs can either mount UsbDevices or further UsbHubs. This method searches through the tree
* of UsbHubs for UsbDevices and returns them as list.
*/
private List<UsbDevice> getAllUsbDevices(UsbHub hub) {
List<UsbDevice> devices = Lists.newArrayList();
List<UsbDevice> attachedDevices = hub.getAttachedUsbDevices();
for (UsbDevice device : attachedDevices) {
if (device instanceof UsbHub) {
List<UsbDevice> subdevices = getAllUsbDevices((UsbHub) device);
devices.addAll(subdevices);
} else {
devices.add(device);
}
}
return devices;
}
}
PowerShell script:
# $input = "VID_XXXX&PID_XXXX (this line is added in Java Code)
# For given VID and PID of a USB device we search for
# the corresponding logical disk to get the drive letter.
# The chain of objects is:
# PnPEntity (PnP = Plug and Play)
# -> USBController
# -> Some associator of USBController that has a related disk drive
# -> diskDrive
# -> diskPartition
# -> logicalDisk
# Find PnPEntity for given VID and PID
$usbPnPEntity = (gwmi Win32_PnPEntity | where DeviceID -match $input)
# Get USB Controller related to PnP Entity
$usbController = $usbPnPEntity.getRelated("Win32_USBController")
$usbControllerID = $usbController.DeviceID
# Find objects associated with the USB Controller
$query = "ASSOCIATORS OF {Win32_USBController.DeviceID='$usbControllerID'}"
$associators = ([wmisearcher]$query).get()
# Search through associators
foreach ($associator in $associators) {
# Find associator that is related to a disk Drive
$assoDeviceID = $associator.DeviceID
$diskDrive = (gwmi win32_diskdrive | where PNPDeviceID -eq $assoDeviceID)
if($diskDrive){
# Get logical Disk related to the disk drive
$logicalDisk = $diskDrive.getRelated("Win32_DiskPartition").getRelated("Win32_LogicalDisk")
# Print device ID which is the drive letter (e.g. "C:")
$logicalDisk.DeviceID
break
}
}
Maven dependencies:
<dependency>
<groupId>org.usb4java</groupId>
<artifactId>usb4java-javax</artifactId>
<version>1.3.0</version>
</dependency>
<dependency>
<groupId>com.profesorfalken</groupId>
<artifactId>jPowerShell</artifactId>
<version>3.1.1</version>
</dependency>

Related

Printing to TSC printer from java application

I am developing a java application (which will run on linux desktop) to print shipping labels using TSC TTP-244 Pro printer. This printer supports TSPL2 commands.
I am using USB connection and started writing some simple tests using usb4java high-level API in-order to communicate to this printer. I am able to successfully query for the printer status/state <ESC>?! or <ESC>?S (<ESC> here is ASCII 27) with out any issues but unable to issue PRINT command.
Below is my code.
#Test
public void printTest() throws UsbException
{
final UsbServices services = UsbHostManager.getUsbServices();
UsbDevice printerUsbDevice = findDevice(services.getRootUsbHub(), 0x1234, 0x1734);
UsbConfiguration configuration = device.getActiveUsbConfiguration();
UsbInterface iface = configuration.getUsbInterface((byte) 1);
iface.claim();
try
{
UsbEndpoint inEndpoint = iface.getUsbEndpoint(0x01);
UsbPipe pipe = inEndpoint.getUsbPipe();
UsbEndpoint outEndpoint = iface.getUsbEndpoint(0x82);
UsbPipe pipe2 = outEndpoint.getUsbPipe();
pipe2.open();
pipe.open();
pipe.syncSubmit(27 + "!?".getBytes("US-ASCII"));
pipe.close();
pipe2.open();
byte[] statusResponse = pipe2.syncSubmit(new byte[1]);
pipe2.close();
System.out.println(new String(statusResponse, "US-ASCII")); // Here status got is "00" if ok otherwise getting error code
pipe.open();
pipe.syncSubmit("SIZE 57 mm,37 mm");
pipe.syncSubmit("GAP 3 mm,0 mm");
pipe.syncSubmit("DIRECTION 1");
pipe.syncSubmit("CLS");
pipe.syncSubmit("TEXT 10,10 "3",0,1,1, "Test printing");
pipe.syncSubmit("PRINT 1");
pipe.close();
// at this pint of time, printer is not printing anything instead just idle
}
finally
{
iface.release();
}
}
private UsbDevice findDevice(UsbHub hub, short vendorId, short productId)
{
for (UsbDevice device : (List<UsbDevice>) hub.getAttachedUsbDevices())
{
UsbDeviceDescriptor desc = device.getUsbDeviceDescriptor();
if (desc.idVendor() == vendorId && desc.idProduct() == productId) return device;
if (device.isUsbHub())
{
device = findDevice((UsbHub) device, vendorId, productId);
if (device != null) return device;
}
}
return null;
}
Is my usb communication correct ?
Does this USB communication with the TSC printer works without installing any printer driver (on linux) ?
Yes, your communication is correct.
Yes, USB communication on linux can work directly, without drivers.
If printer is not accepting some command, double check that command, maybe there should be some control sum, or something else you have missed? Study how exactly this command should be structured.
I have done a project using serial port to communicate with printer, the details of a command are very important. The status command could have no control sum and is very simple, that's why it work out-of-the-box, but with more sophisticated command you need to read the documentation and debug.
There is also possibility, that the printer is using differnt USB endpoint for other than "status" communication.

Get Device Descriptor of SerialPort using jssc

How to get the Device Descriptor of a SerialPort using jssc ( java-simple-serial-connector)?
The getPortName() method gives port name (e.g. COM2), but descriptor would be more helpful.
If it's necessary to patch this opensource API to get the Device Descriptor how can it be done?
Look at gohai/java-simple-serial-connector, the SerialPortList class has a getPortProperties(String portName) method to get the port properties, unfortunately is not implemented yet for windows but is easy to do an implementation and recompile again to make it work.
I hope this helps.
With the help from zamanillo his input I can now answer this myself.
it is not possible to do it with jssc 2.8.0.
There are actions underway to extend jssc but I don't know the release schedule nor drive to get this fixed.
The extension is the method
SerialPortList->getPortProperties(String portName)
Implementations are available in modified versions of jssc 2.8.0 for linux and mac. Windows implementations are harder (to find).
Here is the code for linux and mac taken from https://github.com/gohai/java-simple-serial-connector/blob/processing/src/cpp/_nix_based/jssc.cpp
JNIEXPORT jobjectArray JNICALL Java_jssc_SerialNativeInterface_getPortProperties
(JNIEnv *env, jclass cls, jstring portName) {
const char* portNameChar = (const char*)env->GetStringUTFChars(portName, NULL);
jclass stringClass = env->FindClass("Ljava/lang/String;");
jobjectArray ret = env->NewObjectArray(5, stringClass, NULL);
#ifdef __APPLE__
// this code is based on QtSerialPort
CFMutableDictionaryRef matching = IOServiceMatching(kIOSerialBSDServiceValue);
io_iterator_t iter = 0;
kern_return_t kr = IOServiceGetMatchingServices(kIOMasterPortDefault, matching, &iter);
if (kr != kIOReturnSuccess) {
env->ReleaseStringUTFChars(portName, portNameChar);
return ret;
}
io_registry_entry_t service;
while ((service = IOIteratorNext(iter))) {
// compare portName against cu and tty devices
bool found = false;
CFTypeRef cu = 0;
cu = IORegistryEntrySearchCFProperty(service, kIOServicePlane, CFSTR(kIOCalloutDeviceKey), kCFAllocatorDefault, 0);
if (cu) {
char buffer[MAXPATHLEN];
CFStringGetCString(CFStringRef(cu), buffer, sizeof(buffer), kCFStringEncodingUTF8);
//fprintf(stdout, "getPortProperties: %s\n", buffer);
//fflush(stdout);
if (strcmp(portNameChar, buffer) == 0) {
found = true;
}
CFRelease(cu);
}
CFTypeRef tty = 0;
tty = IORegistryEntrySearchCFProperty(service, kIOServicePlane, CFSTR(kIODialinDeviceKey), kCFAllocatorDefault, 0);
if (tty) {
char buffer[MAXPATHLEN];
CFStringGetCString(CFStringRef(tty), buffer, sizeof(buffer), kCFStringEncodingUTF8);
//fprintf(stdout, "getPortProperties: %s\n", buffer);
//fflush(stdout);
if (strcmp(portNameChar, buffer) == 0) {
found = true;
}
CFRelease(tty);
}
if (!found) {
// not port we're looking for
//fprintf(stderr, "getPortProperties: %s not found", portNameChar);
//fflush(stderr);
IOObjectRelease(service);
continue;
}
io_registry_entry_t entry = service;
do {
int val = 0;
char buffer[255];
CFTypeRef idProduct = 0;
idProduct = IORegistryEntrySearchCFProperty(entry, kIOServicePlane, CFSTR(kUSBProductID), kCFAllocatorDefault, 0);
if (idProduct && !env->GetObjectArrayElement(ret, 0)) {
CFNumberGetValue(CFNumberRef(idProduct), kCFNumberIntType, &val);
sprintf(buffer, "%04x", val);
jstring tmp = env->NewStringUTF(buffer);
env->SetObjectArrayElement(ret, 0, tmp);
env->DeleteLocalRef(tmp);
CFRelease(idProduct);
}
CFTypeRef idVendor = 0;
idVendor = IORegistryEntrySearchCFProperty(entry, kIOServicePlane, CFSTR(kUSBVendorID), kCFAllocatorDefault, 0);
if (idVendor && !env->GetObjectArrayElement(ret, 1)) {
CFNumberGetValue(CFNumberRef(idVendor), kCFNumberIntType, &val);
sprintf(buffer, "%04x", val);
jstring tmp = env->NewStringUTF(buffer);
env->SetObjectArrayElement(ret, 1, tmp);
env->DeleteLocalRef(tmp);
CFRelease(idVendor);
}
CFTypeRef manufacturer = 0;
manufacturer = IORegistryEntrySearchCFProperty(entry, kIOServicePlane, CFSTR(kUSBVendorString), kCFAllocatorDefault, 0);
if (manufacturer && !env->GetObjectArrayElement(ret, 2)) {
CFStringGetCString(CFStringRef(manufacturer), buffer, sizeof(buffer), kCFStringEncodingUTF8);
jstring tmp = env->NewStringUTF(buffer);
env->SetObjectArrayElement(ret, 2, tmp);
env->DeleteLocalRef(tmp);
CFRelease(manufacturer);
}
CFTypeRef product = 0;
product = IORegistryEntrySearchCFProperty(entry, kIOServicePlane, CFSTR(kUSBProductString), kCFAllocatorDefault, 0);
if (product && !env->GetObjectArrayElement(ret, 3)) {
CFStringGetCString(CFStringRef(product), buffer, sizeof(buffer), kCFStringEncodingUTF8);
jstring tmp = env->NewStringUTF(buffer);
env->SetObjectArrayElement(ret, 3, tmp);
env->DeleteLocalRef(tmp);
CFRelease(product);
}
CFTypeRef serial = 0;
serial = IORegistryEntrySearchCFProperty(entry, kIOServicePlane, CFSTR(kUSBSerialNumberString), kCFAllocatorDefault, 0);
if (serial && !env->GetObjectArrayElement(ret, 4)) {
CFStringGetCString(CFStringRef(serial), buffer, sizeof(buffer), kCFStringEncodingUTF8);
jstring tmp = env->NewStringUTF(buffer);
env->SetObjectArrayElement(ret, 4, tmp);
env->DeleteLocalRef(tmp);
CFRelease(serial);
}
kr = IORegistryEntryGetParentEntry(entry, kIOServicePlane, &entry);
} while (kr == kIOReturnSuccess);
IOObjectRelease(entry);
IOObjectRelease(service);
}
IOObjectRelease(iter);
#endif // __APPLE__
env->ReleaseStringUTFChars(portName, portNameChar);
return ret;
}
For windows I only found a solution in processing where a custom dll is used.
The Device Descriptor is the root of the descriptor tree and contains basic device information. The unique numbers, idVendor and idProduct, identify the connected device. The Windows operating system uses these numbers to determine which device driver to load.
idVendor is the number assigned to each company producing USB-based devices. The USB Implementers’ Forum is responsible for administering the assignment of Vendor IDs.
The idProduct is another 16-bit field containing a number assigned by the manufacturer to identify a specific product.
From jantje's question,
What I need is what windows 10 shows as names in the connected
devices?
Ans: Open the "Settings app" and click on Devices. Clicking on Devices will open a tab where you can adjust the settings for all your printers, connected devices, Bluetooth devices, mouse & touchpad, typing settings and autoplay settings. This connected devices tab shows the hardware connected to your PC. Click on Add a device and your PC will automatically scan for the connected devices. The Bluetooth tab is simple with simple settings for connecting a device to your PC via Bluetooth. Click on Bluetooth button and the device will automatically start scanning for any Bluetooth device in the range.
If there is any kind of problem for showing devices or gives not available, then we require following jobs.
Scan the hardware and devices to find out the driver problems.
Fix the corrupted, damaged, or incompatible Bluetooth driver.
Update to the latest Bluetooth driver for Windows 10.
Download and install the best-matched driver in Windows 10.
Top 2. Remove and Add the Bluetooth Devices
Top 3. Allow Bluetooth Devices to Connect to This Computer
Can’t connect to computer from your Bluetooth enabled device? Make sure you have allowed the Bluetooth devices to connect to your computer. Try the steps below:
1. Go to Control Panel. Click “Hardware and Sound” and “Bluetooth Devices”.
2. Click the Options tab.
3. Make sure the “Allow Bluetooth devices to connect to this computer” check box is selected.
How to Connect Bluetooth Devices to Your Windows 10 Computer?
To connect a Bluetooth enabled mobile phone:
Go to the Start button and select “Settings”, “Devices” and
“Bluetooth”. Click “Add”.
Alternatively, go to Control Panel, click “Hardware and Sound” and
“Bluetooth Devices”. Click “Add”.
Resource Link:
Top 3 Ways to Fix Bluetooth Not Available after Windows 10 Update
Issue
http://windows.microsoft.com/en-gb/windows-10/getstarted-connect-to-bluetooth-devices
Fix network connection issues
There are 7 ways to fix network connected issues:
Update the network adapter driver
Roll back the network adapter driver
Run the Network troubleshooter followed by networking commands
Temporarily turn off firewalls
Temporarily turn off any antivirus or malware-prevention software
Uninstall the network adapter driver and restart
Reinstall network devices using netcfg –d command
UPDATE:
How does your answer link to jssc?
I am checking for your issue. I got that in windows 7 it works fine and in windows 10, there are some hardware related issue which causes problem.
The developers are working on this area. But it is not fixed yet. The issue#63 and issue#85 will clarify you.
/**
* Get serial port names in Windows
*
* #since 2.3.0
*/
private static String[] getWindowsPortNames(Pattern pattern, Comparator<String> comparator) {
String[] portNames = serialInterface.getSerialPortNames();
if(portNames == null){
return new String[]{};
}
TreeSet<String> ports = new TreeSet<String>(comparator);
for(String portName : portNames){
if(pattern.matcher(portName).find()){
ports.add(portName);
}
}
return ports.toArray(new String[ports.size()]);
}
Here I got that, they are using some patterns for various OS.
OS_LINUX: Pattern.compile("(ttyS|ttyUSB|ttyACM|ttyAMA|rfcomm|ttyO)[0-9]{1,3}");
OS_SOLARIS: Pattern.compile("[0-9]*|[a-z]*");
OS_MAC_OS_X: Pattern.compile("tty.(serial|usbserial|usbmodem).*");
OS_WINDOWS: Pattern.compile("");
Suggestions:
use only official and latests drivers.
jSSC-2.8.0 Release version (24.01.2014)
Fixes: Important! Fixed bug with port handles potential leakage.
This version contains native libs for Windows(x86, x86-64), Linux(x86,
x86-64, ARM soft & hard float), Solaris(x86, x86-64), Mac OS X(x86,
x86-64, PPC, PPC64). All native libs contains in the jssc.jar file and
you don't need manage native libs manually.
Resource Link:
https://github.com/tcr/node-jssc/blob/master/jSSC-2.6.0-Release/src/src/jssc/SerialPortList.java

How to process MultiLine input log file in Spark using Java

I am new to Spark and It seems very confusing to me. I had gone through the spark documentation for Java API But couldn't figure out the way to solve my problem.
I have to process a logfile in spark-Java and have very little time left for the same. Below is the log file that contains the device records(device id, decription, ip address, status) span over multiple lines.
It also contains some other log information which I am not bothered about.
How can I get the device information log from this huge log file.
Any help is much appreciated.
Input Log Data :
!
!
!
device AGHK75
description "Optical Line Terminal"
ip address 1.11.111.12/10
status "FAILED"
!
device AGHK78
description "Optical Line Terminal"
ip address 1.11.111.12/10
status "ACTIVE"
!
!
context local
!
no ip domain-lookup
!
interface IPA1_A2P_1_OAM
description To_A2P_1_OAM
ip address 1.11.111.12/10
propagate qos from ip class-map ip-to-pd
!
interface IPA1_OAM_loopback loopback
description SE1200_IPA-1_OAM_loopback
ip address 1.11.111.12/10
ip source-address telnet snmp ssh radius tacacs+ syslog dhcp-server tftp ftp icmp-dest-unreachable icmp-time-exceed netop flow-ip
What I have done so far is:
Java Code
JavaRDD<String> logData = sc.textFile("logFile").cache();
List<String> deviceRDD = logData.filter(new Function<String, Boolean>() {
Boolean check=false;
public Boolean call(String s) {
if(s.contains("device") ||(check == true && ( s.contains("description") || s.contains("ip address"))))
check=true;
else if(check==true && s.contains("status")){
check=false;
return true;
}
else
check=false;
return check; }
}).collect();
Current Output :
device AGHK75
description "Optical Line Terminal"
ip address 1.11.111.12/10
status "FAILED"
device AGHK78
description "Optical Line Terminal"
ip address 1.11.111.12/10
status "ACTIVE"
Expected Output is:
AGHK75,"Optical Line Terminal",1.11.111.12/10,"FAILED"
AGHK78,"Optical Line Terminal",1.11.111.12/10,"ACTIVE"
You can use sc.wholeTextFiles("logFile") for getting the data as key,value pair of where key will be the file name and value as data in it.
Then you can use some string operation for splitting of the data as per the start and end delimiter of single log data with "!" and do a filter first for checking if the first word is device and then do a flatMap on it which will make it as singleLog text RDD.
and then get the data from it using the map.
Please try it and let me know whether if this logic is working for you.
Added code in Spark Scala:
val ipData = sc.wholeTextFiles("abc.log")
val ipSingleLog = ipData.flatMap(x=>x._2.split("!")).filter(x=>x.trim.startsWith("device"))
val logData = ipSingleLog.map(x=>{
val rowData = x.split("\n")
var device = ""
var description = ""
var ipAddress = ""
var status = ""
for (data <- rowData){
if(data.trim().startsWith("device")){
device = data.split("device")(1)
}else if(data.trim().startsWith("description")){
description = data.split("description")(1)
}else if(data.trim().startsWith("ip address")){
ipAddress = data.split("ip address")(1)
}else if(data.trim().startsWith("status")){
status = data.split("status")(1)
}
}
(device,description,ipAddress,status)
})
logData.foreach(println)
Spark will take each line as a separate item with sc.textFile. You can get it to split on a different char using sc.hadoopConfiguration().set("textinputformat.record.delimiter", "!").
#Test
public void test() throws ParseException, IOException {
hadoop.write("/test.txt", "line 1\nline 2\n!\nline 3\nline 4");
JavaSparkContext sc = spark.getContext();
sc.hadoopConfiguration().set("textinputformat.record.delimiter", "!");
System.out.println(sc.textFile(hadoop.getMfs().getUri() + "/test.txt").collect());
assertThat(sc.textFile(hadoop.getMfs().getUri() + "/test.txt").count(), is(2L));
}
I believe the only correct way that works everywhere is
Configuration hadoopConf = new Configuration();
hadoopConf.set("textinputformat.record.delimiter", "delimiter");
JavaPairRDD<LongWritable, Text> input = jsc.newAPIHadoopFile(path,
TextInputFormat.class, LongWritable.class, Text.class, hadoopConf);
There are issues in hadoop related code. Depending on size of the input file it produces additional records: MAPREDUCE-6549,MAPREDUCE-5948. It certainly works starting with 2.7.2.
Even though as mlk suggests using spark context would perfectly work, it'll fail in case you try to read another file with different delimiter using the same spark context. By default the delimiter is new line symbol and it'll be changed as soon as this option is applied.
The reason is that spark context shares hadoopConfiguration object and it's hard to reason, where exactly this value is going to be needed. As a workaround the one might materialize RDD and cache it, but it's still might happen that the same RDD would be recomputed.
Given way would work everywhere, because every time it uses new Configuration.

Using JZMQ with EPGM Transport Is Not Sending or Receiving Data

I'm experimenting with java flavored zmq to test the benefits of using PGM over TCP in my project. So I changed the weather example, from the zmq guide, to use the epgm transport.
Everything compiles and runs, but nothing is being sent or received. If I change the transport back to TCP, the server receives the messages sent from the client and I get the console output I'm expecting.
So, what are the requirements for using PGM? I changed the string, that I'm passing to the bind and connect methods, to follow the zmq api for zmq_pgm: "transport://interface;multicast address:port". That didn't work. I get and invalid argument error whenever I attempt to use this format. So, I simplified it by dropping the interface and semicolon which "works", but I'm not getting any results.
I haven't been able to find a jzmq example that uses pgm/epgm and the api documentation for the java binding does not define the appropriate string format for an endpoint passed to bind or connect. So what am I missing here? Do I have to use different hosts for the client and the server?
One thing of note is that I'm running my code on a VirtualBox VM (Ubuntu 14.04/OSX Mavericks host). I'm not sure if that has anything to do with the issue I'm currently facing.
Server:
public class wuserver {
public static void main (String[] args) throws Exception {
// Prepare our context and publisher
ZMQ.Context context = ZMQ.context(1);
ZMQ.Socket publisher = context.socket(ZMQ.PUB);
publisher.bind("epgm://xx.x.x.xx:5556");
publisher.bind("ipc://weather");
// Initialize random number generator
Random srandom = new Random(System.currentTimeMillis());
while (!Thread.currentThread ().isInterrupted ()) {
// Get values that will fool the boss
int zipcode, temperature, relhumidity;
zipcode = 10000 + srandom.nextInt(10000) ;
temperature = srandom.nextInt(215) - 80 + 1;
relhumidity = srandom.nextInt(50) + 10 + 1;
// Send message to all subscribers
String update = String.format("%05d %d %d", zipcode, temperature, relhumidity);
publisher.send(update, 0);
}
publisher.close ();
context.term ();
}
}
Client:
public class wuclient {
public static void main (String[] args) {
ZMQ.Context context = ZMQ.context(1);
// Socket to talk to server
System.out.println("Collecting updates from weather server");
ZMQ.Socket subscriber = context.socket(ZMQ.SUB);
//subscriber.connect("tcp://localhost:5556");
subscriber.connect("epgm://xx.x.x.xx:5556");
// Subscribe to zipcode, default is NYC, 10001
String filter = (args.length > 0) ? args[0] : "10001 ";
subscriber.subscribe(filter.getBytes());
// Process 100 updates
int update_nbr;
long total_temp = 0;
for (update_nbr = 0; update_nbr < 100; update_nbr++) {
// Use trim to remove the tailing '0' character
String string = subscriber.recvStr(0).trim();
StringTokenizer sscanf = new StringTokenizer(string, " ");
int zipcode = Integer.valueOf(sscanf.nextToken());
int temperature = Integer.valueOf(sscanf.nextToken());
int relhumidity = Integer.valueOf(sscanf.nextToken());
total_temp += temperature;
}
System.out.println("Average temperature for zipcode '"
+ filter + "' was " + (int) (total_temp / update_nbr));
subscriber.close();
context.term();
}
}
There are a couple possibilities:
You need to make sure ZMQ is compiled with the --with-pgm option: see here - but this doesn't appear to be your issue if you're not seeing "protocol not supported"
Using raw pgm requires root privileges because it requires the ability to create raw sockets... but epgm doesn't require that, so it shouldn't be your issue either (I only bring it up because you use the term "pgm/epgm", and you should be aware that they are not equally available in all situations)
What actually appears to be the problem in your case is that pgm/epgm requires support along the network path. In theory, it requires support out to your router, so your application can send a single message and have your router send out multiple messages to each client, but if your server is aware enough, it can probably send out multiple messages immediately and bypass this router support. The problem is, as you correctly guessed, trying to do this all on one host is not supported.
So, you need different hosts for client and server.
Another bit to be aware of is that some virtualization environments--RHEV/Ovirt and libvirt/KVM with the mac_filter option enabled come to mind-- that, by default, neuter one's abilities via (eb|ip)tables to utilize mcast between guests. With libvirt, of course, the solution is to simply set the option to '0' and restart libvirtd. RHEV/Ovirt require a custom plugin.
At any rate, I would suggest putting a sniffer on the network devices on each system you are using and watching to be sure traffic that is exiting the one host is actually visible on the other.

app works in simulator but not in device

I have this classes for my app:
NewsApp.java
ScreenApp.java
Item.java
ui/TableList.java
The app retrieve a list of links from a webservice (.net), I use KSoap Library (as Reference project).
I use JDE 4.5 for develop, because with Eclipse I cant use the method "setRowHeight(index, int)" of ListField class, then I need use JDE 4.5
Ok, I compile the app (F7 key), and run in simulator (F5 key).
In simulator, go to the icon app, and try to open... nothing happends... the app not open... are strange... no error message (ScreenApp.java line 57)... but... if I few more minutes... I see the error message (ScreenApp.java line 57)... I think maybe is because the app try connect...
Later... I think is because not exists a internet connection in simulator (I see EDGE in the top of simulator... is strange), I stop de simulator, open MDS, and run simulator again (F5 key), and now works... the list show correctly... and I can open the links in the blackberry browser.
Now... I put all compiled files in same directory, create a ALX file:
NewsApp.alx
And install this app on device, the installation works ok, I go to the list of applications on device (8520), Open the app and I see the connection message (ScreenApp.java line 57);
I dont understand why ? in this phone (8520) I have EDGE connection with my carrier, I have the WIFI active... I can browse in any page (default browser)... but my app cant retrieve information from webservice... :(
Anybody help me please ?
You need to use Different connection parameter at the end of the url when application run on the device.
For ex. in case of wifi, you need to append ;interface=wifi" at the end of the URL.
Detail code is: you need to call getConnectionString() to get the connection sufix according to device network. I hope this will solve your problem.
/**
* #return connection string
*/
static String getConnectionString()
{
// This code is based on the connection code developed by Mike Nelson of AccelGolf.
// http://blog.accelgolf.com/2009/05/22/blackberry-cross-carrier-and-cross-network-http-connection
String connectionString = null;
// Simulator behavior is controlled by the USE_MDS_IN_SIMULATOR variable.
if(DeviceInfo.isSimulator())
{
// logMessage("Device is a simulator and USE_MDS_IN_SIMULATOR is true");
connectionString = ";deviceside=true";
}
// Wifi is the preferred transmission method
else if(WLANInfo.getWLANState() == WLANInfo.WLAN_STATE_CONNECTED)
{
// logMessage("Device is connected via Wifi.");
connectionString = ";interface=wifi";
}
// Is the carrier network the only way to connect?
else if((CoverageInfo.getCoverageStatus() & CoverageInfo.COVERAGE_DIRECT) == CoverageInfo.COVERAGE_DIRECT)
{
//logMessage("Carrier coverage.");
String carrierUid = getCarrierBIBSUid();
if(carrierUid == null)
{
// Has carrier coverage, but not BIBS. So use the carrier's TCP network
// logMessage("No Uid");
connectionString = ";deviceside=true";
}
else
{
// otherwise, use the Uid to construct a valid carrier BIBS request
// logMessage("uid is: " + carrierUid);
connectionString = ";deviceside=false;connectionUID="+carrierUid + ";ConnectionType=mds-public";
}
}
// Check for an MDS connection instead (BlackBerry Enterprise Server)
else if((CoverageInfo.getCoverageStatus() & CoverageInfo.COVERAGE_MDS) == CoverageInfo.COVERAGE_MDS)
{
// logMessage("MDS coverage found");
connectionString = ";deviceside=false";
}
// If there is no connection available abort to avoid bugging the user unnecssarily.
else if(CoverageInfo.getCoverageStatus() == CoverageInfo.COVERAGE_NONE)
{
//logMessage("There is no available connection.");
}
// In theory, all bases are covered so this shouldn't be reachable.
else
{
//logMessage("no other options found, assuming device.");
connectionString = ";deviceside=true";
}
return connectionString;
}
/**
* Looks through the phone's service book for a carrier provided BIBS network
* #return The uid used to connect to that network.
*/
private static String getCarrierBIBSUid()
{
ServiceRecord[] records = ServiceBook.getSB().getRecords();
int currentRecord;
for(currentRecord = 0; currentRecord < records.length; currentRecord++)
{
if(records[currentRecord].getCid().toLowerCase().equals("ippp"))
{
if(records[currentRecord].getName().toLowerCase().indexOf("bibs") >= 0)
{
return records[currentRecord].getUid();
}
}
}
return null;
}

Categories