We Keep Coding

Converting a string of hexadecimal valiues into an array of bytes using Scanner in Java

I would like to convert a string with the form "AA BB CC" or "0xAA 0xBB 0xCC" to an array of bytes. Reading the documentation of Scanner looked promising.
I assumed that hasNextByte() and getNextByte() would do the job but in fact not byte seems to be detected. The code is pretty simple:
byte[] bytesFromString(String value) {
List<Byte> list = new ArrayList<>();
Scanner scan = new Scanner(value);
while(scan.hasNextByte(16)) {
list.add(scan.nextByte(16));
}
byte[] bytes = new byte[list.size()];
for(int i = 0; i < list.size(); i++){
bytes[i] = list.get(i);
}
return bytes;
}
I always receive an empty array as output: hasNextByte(16) never detects the byte.
Any particular reason why it is not working?

The problem is that your values (AA BB and CC) are out of range for byte, since byte is signed.
A possible workaround is to read the values as short and then cast to byte:
while (scan.hasNextShort(16)) {
list.add((byte) scan.nextShort(16));
}
Printing the results, you will see that the values are negative:
String input = "AA BB CC";
byte[] result = bytesFromString(input);
System.out.println(Arrays.toString(result)); // [-86, -69, -52]
Showing the overflow.

How does encoding/decoding bytes work in Java?

Little background: I'm doing cryptopals challenges and I finished https://cryptopals.com/sets/1/challenges/1 but realized I didn't learn what I guess is meant to be learned (or coded).
I'm using the Apache Commons Codec library for Hex and Base64 encoding/decoding. The goal is to decode the hex string and re-encode it to Base64. The "hint" at the bottom of the page says "Always operate on raw bytes, never on encoded strings. Only use hex and base64 for pretty-printing."
Here's my answer...
private static Hex forHex = new Hex();
private static Base64 forBase64 = new Base64();
public static byte[] hexDecode(String hex) throws DecoderException {
byte[] rawBytes = forHex.decode(hex.getBytes());
return rawBytes;
}
public static byte[] encodeB64(byte[] bytes) {
byte[] base64Bytes = forBase64.encode(bytes);
return base64Bytes;
}
public static void main(String[] args) throws DecoderException {
String hex = "49276d206b696c6c696e6720796f757220627261696e206c696b65206120706f69736f6e6f7573206d757368726f6f6d";
//decode hex String to byte[]
byte[] myHexDecoded = hexDecode(hex);
String myHexDecodedString = new String(myHexDecoded);
//Lyrics from Queen's "Under Pressure"
System.out.println(myHexDecodedString);
//encode myHexDecoded to Base64 encoded byte[]
byte[] myHexEncoded = encodeB64(myHexDecoded);
String myB64String = new String(myHexEncoded);
//"pretty printing" of base64
System.out.println(myB64String);
}
...but I feel like I cheated. I didn't learn how to decode bytes that were encoded as hex, and I didn't learn how to encode "pure" bytes to Base64, I just learned how to use a library to do something for me.
If I were to take a String in Java then get its bytes, how would I encode those bytes into hex? For example, the following code snip turns "Hello" (which is readable English) to the byte value of each character:
String s = "Hello";
char[] sChar = s.toCharArray();
byte[] sByte = new byte[sChar.length]
for(int i = 0; i < sChar.length; i++) {
sByte[i] = (byte) sChar[i];
System.out.println("sByte[" + i + "] = " +sByte[i]);
}
which yields sByte[0] = 72, sByte[1] = 101, sByte[2] = 108, sByte[3] = 108, sByte[4] = 111
Lets use 'o' as an example - I am guessing its decimal version is 111 - do I just take its decimal version and change that to its hex version?
If so, to decode, do I just take the the characters in the hex String 2 at a time, decompose them to decimal values, then convert to ASCII? Will it always be ASCII?

to decode, do I just take the the characters in the hex String 2 at a time, decompose them to decimal values, then convert to ASCII? Will it always be ASCII?
No. You take the characters 2 at a time, transform the character '0' to the numeric value 0, the character '1' to the numeric value 1, ..., the character 'a' (or 'A', depending on which encoding you want to support) to the numeric value 10, ..., the character 'f' or 'F' to the numeric value 15.
Then you multiply the first numeric value by 16, and you add it to the second numeric value to get the unsigned integer value of your byte. Then you transform that unsigned integer value to a signed byte.
ASCII has nothing to do with this algorithm.
To see how it's done in practice, since commons-codec is open-source, you can just look at its implementation.

Convert String to/from byte array without encoding

I have a byte array read over a network connection that I need to transform into a String without any encoding, that is, simply by treating each byte as the low end of a character and leaving the high end zero. I also need to do the converse where I know that the high end of the character will always be zero.
Searching the web yields several similar questions that have all got responses indicating that the original data source must be changed. This is not an option so please don't suggest it.
This is trivial in C but Java appears to require me to write a conversion routine of my own that is likely to be very inefficient. Is there an easy way that I have missed?

No, you aren't missing anything. There is no easy way to do that because String and char are for text. You apparently don't want to handle your data as text—which would make complete sense if it isn't text. You could do it the hard way that you propose.
An alternative is to assume a character encoding that allows arbitrary sequences of arbitrary byte values (0-255). ISO-8859-1 or IBM437 both qualify. (Windows-1252 only has 251 codepoints. UTF-8 doesn't allow arbitrary sequences.) If you use ISO-8859-1, the resulting string will be the same as your hard way.
As for efficiency, the most efficient way to handle an array of bytes is to keep it as an array of bytes.

This will convert a byte array to a String while only filling the upper 8 bits.
public static String stringFromBytes(byte byteData[]) {
char charData[] = new char[byteData.length];
for(int i = 0; i < charData.length; i++) {
charData[i] = (char) (((int) byteData[i]) & 0xFF);
}
return new String(charData);
}
The efficiency should be quite good. Like Ben Thurley said, if performance is really such an issue don't convert to a String in the first place but work with the byte array instead.

Here is a sample code which will convert String to byte array and back to String without encoding.
public class Test
{
public static void main(String[] args)
{
Test t = new Test();
t.Test();
}
public void Test()
{
String input = "Hèllo world";
byte[] inputBytes = GetBytes(input);
String output = GetString(inputBytes);
System.out.println(output);
}
public byte[] GetBytes(String str)
{
char[] chars = str.toCharArray();
byte[] bytes = new byte[chars.length * 2];
for (int i = 0; i < chars.length; i++)
{
bytes[i * 2] = (byte) (chars[i] >> 8);
bytes[i * 2 + 1] = (byte) chars[i];
}
return bytes;
}
public String GetString(byte[] bytes)
{
char[] chars = new char[bytes.length / 2];
char[] chars2 = new char[bytes.length / 2];
for (int i = 0; i < chars2.length; i++)
chars2[i] = (char) ((bytes[i * 2] << 8) + (bytes[i * 2 + 1] & 0xFF));
return new String(chars2);
}
}

Using deprecated constructor String(byte[] ascii, int hibyte)
String string = new String(byteArray, 0);

String is already encoded as Unicode/UTF-16. UTF-16 means that it can take up to 2 string "characters"(char) to make one displayable character. What you really want is to use is:
byte[] bytes = System.Text.Encoding.Unicode.GetBytes(myString);
to convert a String to an array of bytes. This does exactly what you did above except it is 10 times faster in performance. If you would like to cut the transmission data nearly in half, I would recommend converting it to UTF8 (ASCII is a subset of UTF8) - the format the internet uses 90% of the time, by calling:
byte[] bytes = Encoding.UTF8.GetBytes(myString);
To convert back to a string use:
String myString = Encoding.Unicode.GetString(bytes);
or
String myString = Encoding.UTF8.GetString(bytes);

How can i convert character 0s and 1s read from a txt File to byte?

I want to read a txt file containing 0s and 1s , and convert them to bytes but first convert to binary. e.g textFile contains "00101001". Every character is 1 byte , after reading from file i get 8 bytes. Now i want to append them to a String and convert to byte. I should get 1 byte from 8 bytes.Is that right?
is there is a better option?.
What i have did so far:
public byte[] convertBytes(String File1, int length) throws IOException
{
byte BytesfromFile[]=readInput(File1,length); // readinput() returns (length*8) bytes,contains 48 or 49s.
StringBuilder stringBuilder = new StringBuilder(); //Stringbuilder append chars to build String
byte convertedBytes[]=new byte[BytesfromFile.length/8]; //size of byte array init
int stringToInt;
byte intToByte;
for(int index=0,bitcounter=0,indexCoverted=0;index<BytesfromFile.length;index++,++bitcounter)
{
stringBuilder.append((char) BytesfromFile[index]); //append 48 or 49 casted to char
if(bitcounter==8)
{ String Binary = stringBuilder.toString(); // if 8 bits appended, toString
stringToInt = Integer.parseInt(Binary, 2); // convert Binary String to Int
intToByte = (byte)stringToInt; // cast Int to byte
convertedBytes[indexCoverted++]=intToByte;
bitcounter=0;
stringBuilder.setLength(0); //set length to 0, to append next 8 bit.
}
}
return convertedBytes;
}

You can use the Byte.parseByte method with a radix of 2:
byte b = Byte.parseByte(str, 2);
You can alternatively use the BigInteger(String val,int radix) constructor to quickly do the conversion on a long String of binary digits:
String input = "1010010010010101010010010101001";
byte[] arr = new BigInteger(input, 2).toByteArray();
// Result:
// [82, 74, -92, -87]

Convert int to hex byte value

I want to read block wise data of NFC tag. For which the command is a byte array and it needs block number.
public static byte[] readSingleBlockCmd(int blockNo) {
byte[] cmd = new byte[3];
cmd[0] = (byte) 0x02;//flag
cmd[1] = (byte) 0x23;//cmd
cmd[2]= (byte)blockNo;
return cmd;
}
How can I change the int blockNo to its hexadecimal value , which can be cast to byte .I want the byte value and not an byte []
I have gone through the following links
Convert integer into byte array (Java)
How to autoconvert hexcode to use it as byte[] in Java?
Java integer to byte array
Thanks!

Converting an integer (in decimal) to hex can be done with the following line:
String hex = Integer.toHexString(blockNo);
Then to convert it to byte, you can use
Byte.parseByte(hex,16);
But if all you wanted to do is convert the parameter to bytes:
Byte.parseByte(blockNo);
would work too I guess. Correct me if I'm wrong.