We Keep Coding

Need help in converting EBCDIC to Hexadecimal

I am writing an hive UDF to convert the EBCDIC character to Hexadecimal.
Ebcdic characters are present in hive table.Currently I am able to convert it, bit it is ignoring few characters while conversion.
Example:
This is the EBCDIC value stored in table:
AGNSAñA¦ûÃÃÂõÂjÂq  à ()
Converted hexadecimal:
c1c7d5e2000a5cd4f6ef99187d07067203a0200258dd9736009f000000800017112400000000001000084008403c000000000000000080
What I want as output:
c1c7d5e200010a5cd4f6ef99187d0706720103a0200258dd9736009f000000800017112400000000001000084008403c000000000000000080
It is ignoring to convert the below EBCDIC characters:
01 - It is start of heading
10 - It is a escape
15 - New line.
Below is the code I have tried so far:
public class EbcdicToHex extends UDF {
public String evaluate(String edata) throws UnsupportedEncodingException {
byte[] ebcdiResult = getEBCDICRawData(edata);
String hexResult = getHexData(ebcdiResult);
return hexResult;
}
public byte[] getEBCDICRawData (String edata) throws UnsupportedEncodingException {
byte[] result = null;
String ebcdic_encoding = "IBM-037";
result = edata.getBytes(ebcdic_encoding);
return result;
}
public String getHexData(byte[] result){
String output = asHex(result);
return output;
}
public static String asHex(byte[] buf) {
char[] HEX_CHARS = "0123456789abcdef".toCharArray();
char[] chars = new char[2 * buf.length];
for (int i = 0; i < buf.length; ++i) {
chars[2 * i] = HEX_CHARS[(buf[i] & 0xF0) >>> 4];
chars[2 * i + 1] = HEX_CHARS[buf[i] & 0x0F];
}
return new String(chars);
}
}
While converting, its ignoring few EBCDIC characters. How to make them also converted to hexadecimal?

I think the problem lies elsewhere, I created a small testcase where I create a String based on those 3 bytes you claim to be ignored, but in my output they do seem to be converted correctly:
private void run(String[] args) throws Exception {
byte[] bytes = new byte[] {0x01, 0x10, 0x15};
String str = new String(bytes, "IBM-037");
byte[] result = getEBCDICRawData(str);
for(byte b : result) {
System.out.print(Integer.toString(( b & 0xff ) + 0x100, 16).substring(1) + " ");
}
System.out.println();
System.out.println(evaluate(str));
}
Output:
01 10 15
011015
Based on this it seems both your getEBCDICRawData and evaluate method seem to be working correctly and makes me believe your String value may already be incorrect to start with. Could it be the String is already missing those characters? Or perhaps a long shot, but maybe the charset is incorrect? There are different EBCDIC charsets, so maybe the String is composed using a different one? Although I doubt this would make much difference for the 01, 10 and 15 bytes.
As a final remark, but probably unrelated to your problem, I usually prefer to use the encode/decode functions on the charset object to do such conversions:
String charset = "IBM-037";
Charset cs = Charset.forName(charset);
ByteBuffer bb = cs.encode(str);
CharBuffer cb = cs.decode(bb);

Appending bytes

i am developing an Android application and I am having some issues with adding bytes to compose a data packet. I have created a method to get the MAC Address of the client and the return value is a string (MACAdd).
private String getMACAddress() {
WifiManager manager = (WifiManager) getSystemService(Context.WIFI_SERVICE);
WifiInfo info = manager.getConnectionInfo();
MACAdd= info.getMacAddress();
return MACAdd;
}
I have declared MACAddrs as a static String and have done the following assignment in createPackect() method.
MACAddrs = getMACAddress();
byte [] MACAddres = MACAddrs.getBytes();
In the same createPacket() method, I have declared following data as well.
byte Node = (byte)0x3039;
byte Cost = (byte)0x1;
byte MaxMsgSize = (byte) 0x100000;
byte KeepAlivePiriod = (byte)0x2710;
byte NetworkID = (byte)0x1;
I need to add these bytes into a single array to compose the data packet.
byte[] BroadcastPacket = new byte[Node + Cost + SerialNumber + MaxMsgSize + KeepAlivePiriod + NetworkID + MACAddres + IPAddress];
as I am adding two different byte types, I am having a error. Please advise me how to solve this issue......

A convenience method to concatenate byte arrays might help:
byte[] concatArrays(byte[] ... arrays) {
int totalLength = 0;
for (byte[] arr : arrays) {
totalLength += arr.length;
}
byte[] result = new byte[totalLength];
int start = 0;
for (byte[] arr : arrays) {
System.arraycopy(arr, 0, result, start, arr.length);
start += arr.length;
}
return result;
}
Using this, you could create byte[] BroadcastPacket like this:
byte[] BroadcastPacket = concatArrays(
new byte[Node + Cost + SerialNumber + MaxMsgSize + KeepAlivePiriod + NetworkID],
MACAddrs.getBytes(),
IPAddrs.getBytes()
);
Btw, you should use camelCase for variable names.
And you don't need to cast such values to byte:
byte cost = (byte) 0x1;
This works just as well:
byte cost = 0x1;
If the value you're trying to assign to a byte doesn't actually fit into a byte,
casting to byte you are losing precision, for example here:
byte b3 = (byte) 0x100000;
If you don't mind the precision loss by design, that's fine, otherwise this can be a nasty bug.

Parsing binary data in java received over TCP socket server

I have a socket server which keeps listening to incoming requests. The data received will be in the form of binary array of bytes.
Data format is something like this.
2321902321221200AA
Whereas 1 byte is data begin
4 bits is version
4 bits is data return type
5 bytes are product code
2 bytes data length
The question is, how to parse the data and segregate the parameters.
Thanks in advance!!

Try java.io.DataInputStream:
DataInputStream dis = new DataInputStream(in);
byte b = dis.readByte();
int version = (b >> 4) & 0xF;
int returnType = b & 0xF;
byte[] productCode = new byte[5];
dis.readFully(productCode);
int len = dis.readShort() & 0xFFFF;

if use the java binary block parser then code will look like
class Parsed {
#Bin byte begin;
#Bin(type = BinType.BIT) int version;
#Bin(type = BinType.BIT) int returnType;
#Bin byte [] productCode;
#Bin(type = BinType.USHORT) int dataLength;
}
final Parsed parsed = JBBPParser.prepare("byte begin; bit:4 version; bit:4 returnType; byte [5] productCode; ushort dataLength;")
.parse(new byte[]{0x23,0x21,(byte)0x90,0x23,0x21,0x22,0x12,0x00,(byte)0xAA})
.mapTo(Parsed.class);
assertEquals(0x23, parsed.begin);
assertEquals(0x01, parsed.version);
assertEquals(0x02, parsed.returnType);
assertArrayEquals(new byte[]{(byte)0x90,0x23,0x21,0x22,0x12}, parsed.productCode);
assertEquals(0x00AA,parsed.dataLength);

try {
char [] cbuf = new char[16];
char databegin = cbuf[0];
char [] version = Arrays.copyOfRange(cbuf, 1, 6)
char [] product_typep = Arrays.copyOfRange(cbuf, 7, 12)
char []data_lendth = Arrays.copyOfRange(cbuf, 13, 15)
} catch(Error e){
System.out.println(e);
}

byte [] data = receiveData ();
int dataBegin = data [0]; // Once field is 1-byte, it is simple!
int version = data [1] & 0x0F; // Use shift (>>>) and binary "and" (&)
int returnCode = // to extract value of fields that are
(data [1] >>> 4) & 0x0F; // smaller than one byte
byte [] productCode = // Copy fixed-size portions of data
new byte [] { // into separate arrays using hardcode
data [2], data [3], // (as here), or System.arrayCopy
data [4], data [5], // in case field occupies quite
data [6]}; // a many bytes.
int dataLength = // Use shift (<<) binary or (|) to
(data [7] & 0xFF) | // Combine several bytes into one integer
((data [8] & 0xFF) << 8); // We assume little-endian encoding here

I would got for some king of package reader:
class Record {
.....
Record static fromBytes(byte[] bytes) {
// here use ByteBuffer or DataInputStream to extract filds
......
}
}
Record readNextRecord(InputStream in) {
int len = in.read() && 0xFF;
byte[] data = new byte[len];
in.read(data);
return Record.fromBytes(data)
}
{
InputStream in = ....;
Record r readNextRecord(in);
process (r);
}
Of course you need to add error handling. In general, for something which should run reliable, I will suggest to use NIO framework like Grizzly or Netty.

You might get the data via the ByteArrayOutputStream
And then parse the bytes by applying masks (mainly AND, OR).
Take a look at This question
Hope this helps

String and byte[] issue java

I am converting a String to byte[] and then again byte[] to String in java using the getbytes() and String constructor,
String message = "Hello World";
byte[] message1 = message.getbytes();
using PipedInput/OutputStream I send this to another thread, where,
byte[] getit = new byte[1000];
pipedinputstream.read(getit);
print(new String(getit));
This last print result in 1000 to be printed... I want the actual string length. How can i do that?

When reading the String, you need to get the number of bytes read, and give the length to your String:
byte[] getit = new byte[1000];
int readed = pipedinputstream.read(getit);
print(new String(getit, 0, readed));
Note that if your String is longer than 1000 bytes, it will be truncated.

You are ignoring the number of bytes read. Do it as below:
byte[] getit = new byte[1000];
int bytesRead = pipedinputstream.read(getit);
print(new String(getit, 0, bytesRead).length());

public String getText (byte[] arr)
{
StringBuilder sb = new StringBuilder (arr.length);
for (byte b: arr)
if (b != 32)
sb.append ((char) b);
return sb.toString ();
}
not so clean, but should work.

I am converting a String to byte[] and then again byte[] to String
Why? The round trip is guaranteed not to work. String is not a container for binary data. Don't do this. Stop beating your head against the wall: the pain will stop after a while.

How to convert a Java String to an ASCII byte array?

How to convert a Java String to an ASCII byte array?

Using the getBytes method, giving it the appropriate Charset (or Charset name).
Example:
String s = "Hello, there.";
byte[] b = s.getBytes(StandardCharsets.US_ASCII);
If more control is required (such as throwing an exception when a character outside the 7 bit US-ASCII is encountered) then CharsetDecoder can be used:
private static byte[] strictStringToBytes(String s, Charset charset) throws CharacterCodingException {
ByteBuffer x = charset.newEncoder().onMalformedInput(CodingErrorAction.REPORT).encode(CharBuffer.wrap(s));
byte[] b = new byte[x.remaining()];
x.get(b);
return b;
}
Before Java 7 it is possible to use: byte[] b = s.getBytes("US-ASCII");. The enum StandardCharsets, the encoder as well as the specialized getBytes(Charset) methods have been introduced in Java 7.

If you are a guava user there is a handy Charsets class:
String s = "Hello, world!";
byte[] b = s.getBytes(Charsets.US_ASCII);
Apart from not hard-coding arbitrary charset name in your source code it has a much bigger advantage: Charsets.US_ASCII is of Charset type (not String) so you avoid checked UnsupportedEncodingException thrown only from String.getBytes(String), but not from String.getBytes(Charset).
In Java 7 there is equivalent StandardCharsets class.

There is only one character wrong in the code you tried:
Charset characterSet = Charset.forName("US-ASCII");
String string = "Wazzup";
byte[] bytes = String.getBytes(characterSet);
^
Notice the upper case "String". This tries to invoke a static method on the string class, which does not exist. Instead you need to invoke the method on your string instance:
byte[] bytes = string.getBytes(characterSet);

The problem with other proposed solutions is that they will either drop characters that cannot be directly mapped to ASCII, or replace them with a marker character like ?.
You might desire to have for example accented characters converted to that same character without the accent. There are a couple of tricks to do this (including building a static mapping table yourself or leveraging existing 'normalization' defined for unicode), but those methods are far from complete.
Your best bet is using the junidecode library, which cannot be complete either but incorporates a lot of experience in the most sane way of transliterating Unicode to ASCII.

String s = "ASCII Text";
byte[] bytes = s.getBytes("US-ASCII");

If you happen to need this in Android and want to make it work with anything older than FroYo, you can also use EncodingUtils.getAsciiBytes():
byte[] bytes = EncodingUtils.getAsciiBytes("ASCII Text");

In my string I have Thai characters (TIS620 encoded) and German umlauts. The answer from agiles put me on the right path. Instead of .getBytes() I use now
int len = mString.length(); // Length of the string
byte[] dataset = new byte[len];
for (int i = 0; i < len; ++i) {
char c = mString.charAt(i);
dataset[i]= (byte) c;
}

Convert string to ascii values.
String test = "ABCD";
for ( int i = 0; i < test.length(); ++i ) {
char c = test.charAt( i );
int j = (int) c;
System.out.println(j);
}

I found the solution. Actually Base64 class is not available in Android. Link is given below for more information.
byte[] byteArray;
byteArray= json.getBytes(StandardCharsets.US_ASCII);
String encoded=Base64.encodeBytes(byteArray);
userLogin(encoded);
Here is the link for Base64 class: http://androidcodemonkey.blogspot.com/2010/03/how-to-base64-encode-decode-android.html

To convert String to ASCII byte array:
String s1 = "Hello World!";
byte[] byteArray = s1.getBytes(StandardCharsets.US_ASCII);
// Now byteArray is [72, 101, 108, 108, 111, 32, 87, 111, 114, 108, 100, 33]
To convert ASCII byte array to String:
String s2 = new String(byteArray, StandardCharsets.US_ASCII));

Try this:
/**
* #(#)demo1.java
*
*
* #author
* #version 1.00 2012/8/30
*/
import java.util.*;
public class demo1
{
Scanner s=new Scanner(System.in);
String str;
int key;
void getdata()
{
System.out.println ("plase enter a string");
str=s.next();
System.out.println ("plase enter a key");
key=s.nextInt();
}
void display()
{
char a;
int j;
for ( int i = 0; i < str.length(); ++i )
{
char c = str.charAt( i );
j = (int) c + key;
a= (char) j;
System.out.print(a);
}
public static void main(String[] args)
{
demo1 obj=new demo1();
obj.getdata();
obj.display();
}
}
}