I am trying to divide txt files into ArrayList of strings and so far it works, but first words in the file always starts with (int)'65279' and I can't even copy this character here. Also, in GUI it looks like second letter of word is missing but at the same time it works in console. Other words are as they should be. I am using UTF-8 format .txt files. How can I change format in netBeans and GUI made in this IDE?
U+FEFF is the byte order mark. It's used to indicate the character encoding/endianness (to you can easily tell the difference between big and little-endian UTF-16, for example).
If it's causing you a problem, the simplest thing is just to strip it:
if (text.startsWith("\ufeff")) {
text = text.substring(1);
}
Related
I have an array of strings that I need to save into a txt file.I'm only allowed to make max 64kb files so I need to know when to stop putting strings into the file.
Is there some method that having an array of strings,i can find out how big the file will be without creating the file ?
Is the file going to be ASCII-encoded? If so, every character you write will be 1 byte. Add up the string lengths as you go, and if the total number of characters goes greater than 64k, you know to stop. Don't forget to include newlines between strings, in case you're doing that.
Java brings with it a library to input and output data named NIO. I imagine that you should know about how to use it. If you do not know how to use NIO, look at the following links to learn more:
http://en.wikipedia.org/wiki/New_I/O
https://blogs.oracle.com/slc/entry/javanio_vs_javaio
http://docs.oracle.com/javase/tutorial/essential/io/fileio.html
We all know that all data types are just bytes in the end. With characters, we have the same thing, with a little more detail. The characters (letters, numbers, symbols and so on.) in the World are mapped to a table named Unicode, and using some character encoding algorithms you can get a certain number of bytes when you come to save text to a file. How I'd spend hours talking about it for you, I suggest you take a look at the following links to understand more about character encoding:
http://www.w3schools.com/tags/ref_charactersets.asp
https://stackoverflow.com/questions/3049090/character-sets-explained-for-dummies
https://www.w3.org/International/questions/qa-what-is-encoding.en
http://unicode-table.com/en/
http://en.wikipedia.org/wiki/Character_encoding
By using Charset, CharsetEncoder and CharsetDecoder, you can choose a specific character encoding to save your text, depending on, the final size of your file may vary. With the use of UTF-8 (the 8 here means bits), you will end up saving each character in your file with 1 byte each. With UTF-16 (16 here means bits), you will save each character with 2 bytes. This means that as you use a encoding, you end up with a certain number of bytes for each character saved. On the following link you can find the actual encodings supported by the current Java API:
http://docs.oracle.com/javase/7/docs/technotes/guides/intl/encoding.doc.html
With the NIO library, you do not need to actually save a file to know your size. If you just make the use of ByteBuffer, you may already know the final size of your file without even saving it.
Any questions, please comment.
I'm a starter python programmer and I used to know a bit java in the past.
I have some text files (in Turkish) and corresponding xml files which contains offset numbers of
the connectives in the text. For example
-<Conn>
-<Span>
<Text>ama</Text>
<BeginOffset>281</BeginOffset>
<EndOffset>284</EndOffset>
</Span>
</Conn>
this says that there is an 'ama' at the 281 offset in the txt file. But when I read this file with python,
'ama' is at the 301. byte or it is the 272. character in the file. As far as I know, java application doesn't mention any encoding while reading txt files. And I tried to read files with unicode, UTF8 etc...
I need to find a way from these offsets to correct positions in the files. my guess, problem is due to Turkish characters (which may takes different numbers of bytes in different encodings) but I coudn't figure it out.
Any suggestion will be very very good for me.
thanks
Edit:
I used following code in python3.3:
f = open(path, encoding='utf-8')
text = f.read()
text[272:275] # returns 'ama' but it should be text[281:284]
ibbyte = text.encode(encoding='utf-8')
inbytes[292:295] # returns 'ama' but this is also incorrect
as #Gene says it's end-of-line markers. since the java application written in windows, it counts each '\n' as 2 bytes. But python counts them as 1 byte. I count '\n' until the offset number and substract it from the given offset number.
thank you very much for your insightful comments
I have a text file, in which I am writing 3 things
Eg < int,int,char> for each word.
Now, I am reading the file such that I consider a block of 3.1st one I always consider an integer, 2nd one also as integer and the 3rd one as character .There is no problem when the integer is from 0-9 but when it exceeds like 10,100 then my program doesn't work for the obvious reasons.
Like there is no problem when I have to read this
11a here <1=int,1=int,a=char>
but when something like this comes, I face problem
152a here <15=int,2=int,a=char>
I have put the whole text file in a string.Now, how how do I read the characters that I no longer face the above mentioned problem
Some more info: My text file contains characters like this
11a22d33f1234f
Given your current description of the problem, there is no way to determine if an entry such as
152a
corresponds to (15,2,a) or (1,52,a).
Why don't you write to the file with some delimiter between elements, and then split() around the delimiter when reading back in from the file?
your text file has improper format then
how do you want to differ "1 11 a" and "11 1 a" e.g.
cant you use csv or something like that?
I have to read the name of some files and put them in a list as a string. Its not so hard I just have some Problems with some characters like ä,ö,ü ... they are always as a '?' in my string.
Whats the Problem? Well the encoding. Ok this should be easy... thats what i thought. So I tried to use functions like:
new String(insert.getBytes("UTF-8")
or
new String(insert.getBytes("ISO-8859-1"), "UTF-8")
because the most of the files are ISO-8859-1
Its not helping. This is my code:
...
File[] fileList = dir.listFiles();
String insert;
for(File f : fileList) {
...
insert=f.getName().substring(0,f.getName().length()-4);
insert=insert.charAt(0)+insert.substring(1,insert.length()).toLowerCase().replaceFirst("([0-9]*(_s?(i)?(_dat)?)*$)", "").replaceFirst("_", " ");
...
System.out.println("test UTF8: " + new String(insert.getBytes("UTF-8"))); //not helping
System.out.println("test ISO , UTF8: " + new String(insert.getBytes("ISO-8859-1"), "UTF-8")); //not helping
...
names.add(insert);
}
At the end there are a lot of strings with '?' characters in my list.
How to fix the problem? And whats the best way if there are not only ISO-8859-1 files? (lets say there are a lot of unknown encoded files)
Thank You!
Given the extended comments back and forth under the question, it now looks like this is either a font problem or (perhaps more likely) a filename encoding problem.
I asked Lissy to run the following command to let us figure out what the problem is. If she is sure that the filename contain "ä" in them, but that character does not appear when she ls the filename, then this command will tell us whether this is a font or encoding problem.
touch filenäme
ls filen*me
If this shows "filenäme" in the output of ls then we know the problem is with the creation/copy of the files onto this system. This could happen if the program which created the files didn't realize what the filesystem encoding was or was too stupid to do the right thing. The convmv program will probably be the best way to fix this.
convmv -f ENCODING -t utf8 -r .
The question is what is the proper encoding. Possibilities include UTF-16, cp850, or perhaps iso8859-1. convmv --list will show you the list of currently known (to your system) encodings. Since the listed command above only shows you what it might do, it is safe to run several times with different encodings until you find one which works for all files.
If this is a font problem, we'll have to look into that
Unexpected question marks, spalts, etc in a String are a sign that something somewhere doesn't recognize a particular character when converting from one character set to another.
In your case, the problem could be occurring in a couple of places:
It could be occurring when your Java program is reading the file names from the directory (in the dir.listFiles() call).
It could be happening when you print the characters to the console stream.
In either case, the root cause is most likely a mismatch between what Java thinks the locale settings should be and the settings that the operating system and/or command shell are using.
As an experiment, try to list a directory containing the problematic file names from the command line. Do you see question marks or other splats there?
A second experiment to perform is to modify your Java program to dump one of the problem Strings as a sequence of numbers representing the character codes for each of the characters. Do you see the character codes for an ASCII / Unicode '?'.
The encoding of the content of the file name has nothing to do with the encoding of the file name itself.
You should get correct results from System.out.println(insert)
If you don't, it means that the shell has a different character encoding that the default character encoding for your system (this rarely happens; it would usually be the result of an explicit command to switch encodings in the shell).
If the file names are displayed correctly when you list the directory in the shell, I would expect them to be displayed correctly without specifying an encoding in your Java program.
If the shell is incapable of displaying the character (it is substituting the replacement character 0xFFFD (�) for these unprintable characters), there's nothing you can do from your Java application to change that. You need to change the terminal character encoding, install the right fonts, etc.; that is a operating system issue, not a Java issue.
At the same time, even if your terminal can't display the correct results, the Java program should be handling the character encodings correctly without your intervention.
The library behind the File API is figuring out the correct character encoding for your system and doing the necessary decoding into characters. Likewise, the database driver should negotiate with the database to determine the correct encoding, and do any necessary encoding into bytes on behalf of your application.
In a comment you wrote:
#mdrg: well, theres a Problem. I have to read the name of the files and then put them into a database. And there are a lot of '?' , that shouldnt be... – Lissy 27 mins ago
My guess is that the column you're inserting the filenames into specifies US-ASCII as the encoding and replaces characters outside that range with a replacement character, which in your case is the question mark.
So you have to find out the encoding for the column in your database table where you store the filenames. Various products have various syntaxes for retrieving that information.
In Java 1.6 you can use System.console() instead of System.out.println() to display accentuated characters to console.
public class Test {
public static void main(String args[]){
String s = "caractères français : à é \u00e9"; // Unicode for "é"
System.console().writer().println(s);
}
}
and the output is
C:\temp>java Test
caractères français : à é é
I'm trying to read mail in my outbox which usually contains one attached pdf file. If the pdf file name contains English characters, the function below works fine. But if the file name contains any non-English character (for example, filename1(chinesecharacter).pdf) my function is not able to read it. Can anybody tell me what changes I need to make in my function?
Just simply check the ASCII (or Unicode?) values against the range(s) of values with English characters. Every character corresponds to a number in its character set.
Or you could create an array of all English characters, and check it against that. There may also be an API function in Java.
This line indicates you might have a problem decoding non-ISO 8859 character sets, e.g. UTF-8, due to the weak handling of RFC2822 encoded file names:
if(fileName.startsWith("=?iso-8859"))
{
String strFolder = strFolderName.substring(strFolderName.lastIndexOf("/")+1,
strFolderName.length());
fileName = strFolder + ".pdf";
}
http://en.wikipedia.org/wiki/MIME#Encoded-Word