I'm trying to add extra training data to my nl-personTest.bin file with OpenNLP.
Now is my problem that when I run my code to add the extra trainingsdata it removes the already existing data and only add my new data.
How can I just add extra trainingsdata instead of replacing it?
I did use the following code, (got it from Open NLP NER is not properly trained)
public class TrainNames
{
public static void main(String[] args)
{
train("nl", "person", "namen.txt", "nl-ner-personTest.bin");
}
public static String train(String lang, String entity,InputStreamFactory inputStream, FileOutputStream modelStream) {
Charset charset = Charset.forName("UTF-8");
TokenNameFinderModel model = null;
ObjectStream<NameSample> sampleStream = null;
try {
ObjectStream<String> lineStream = new PlainTextByLineStream(inputStream, charset);
sampleStream = new NameSampleDataStream(lineStream);
TokenNameFinderFactory nameFinderFactory = new TokenNameFinderFactory();
model = NameFinderME.train("nl", "person", sampleStream, TrainingParameters.defaultParams(),
nameFinderFactory);
} catch (FileNotFoundException fio) {
} catch (IOException io) {
} finally {
try {
sampleStream.close();
} catch (IOException io) {
}
}
BufferedOutputStream modelOut = null;
try {
modelOut = new BufferedOutputStream(modelStream);
model.serialize(modelOut);
} catch (IOException io) {
} finally {
if (modelOut != null) {
try {
modelOut.close();
} catch (IOException io) {
}
}
}
return "Something goes wrong with training module.";
}
public static String train(String lang, String entity, String taggedCoprusFile,
String modelFile) {
try {
InputStreamFactory inputStream = new InputStreamFactory() {
FileInputStream fileInputStream = new FileInputStream("namen.txt");
public InputStream createInputStream() throws IOException {
return fileInputStream;
}
};
return train(lang, entity, inputStream,
new FileOutputStream(modelFile));
} catch (Exception e) {
e.printStackTrace();
}
return "Something goes wrong with training module.";
} }
Anyone any ideas to solve this problem?
Because If I want to have an accurate trainingset I need to have at least 15K
sentences says the documation.
I think that OpenNLP does not support to expand existing binary NLP models.
If you have all training data available, collect them all and then train them at once. You can use SequenceInputStream. I modified your example to use another InputStreamFactory
public String train(String lang, String entity, InputStreamFactory inputStream, FileOutputStream modelStream) {
// ....
try {
ObjectStream<String> lineStream = new PlainTextByLineStream(trainingDataInputStreamFactory(Arrays.asList(
new File("trainingdata1.txt"),
new File("trainingdata2.txt"),
new File("trainingdata3.txt")
)), charset);
// ...
}
// ...
}
private InputStreamFactory trainingDataInputStreamFactory(List<File> trainingFiles) {
return new InputStreamFactory() {
#Override
public InputStream createInputStream() throws IOException {
List<InputStream> inputStreams = trainingFiles.stream()
.map(f -> {
try {
return new FileInputStream(f);
} catch (FileNotFoundException e) {
e.printStackTrace();
return null;
}
})
.filter(Objects::nonNull)
.collect(Collectors.toList());
return new SequenceInputStream(new Vector<>(inputStreams).elements());
}
};
}
Related
I am using aws sdk v4 authentication for peformaing s3 operations like upload ,download etc. Now my problem i.e. I want to show progress bar for running operations and the internet giving me solutions using TransferManager but I am using s3 client so please suggest me how can I use progress bar ?
Here is my sample code for upload file.
public static String putS3Object(S3Client s3Client,
String bucketName,
String objectKey,
String objectPath) {
try {
Map<String, String> metadata = new HashMap<>();
metadata.put("myVal", "test");
PutObjectRequest putOb = PutObjectRequest.builder()
.bucket(bucketName)
.key(objectKey).metadata(metadata)
.build();
PutObjectResponse response = s3Client.putObject(putOb,
RequestBody.fromBytes(getObjectFile(objectPath)));
return response.eTag();
} catch (S3Exception e) {
System.err.println(e.getMessage());
System.exit(1);
}
return "";
}
private static byte[] getObjectFile(String filePath) {
FileInputStream fileInputStream = null;
byte[] bytesArray = null;
try {
File file = new File(filePath);
bytesArray = new byte[(int) file.length()];
fileInputStream = new FileInputStream(file);
fileInputStream.read(bytesArray);
} catch (IOException e) {
e.printStackTrace();
} finally {
if (fileInputStream != null) {
try {
fileInputStream.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
return bytesArray;
}
I'm trying to save user settings to a file, from where I can read the later. But I cant get it to work properly. I've tried reading up on this, but I'm still having problems.
Map<String, String> userSettings = new HashMap<>();
public void updateUserSettings(){
userSettings.clear();
userSettings.put("item0", item0);
userSettings.put("item1", item1);
userSettings.put("item2", item2);
userSettings.put("item3", item3);
userSettings.put("item4", item4);
userSettings.put("item5", item5);
userSettings.put("item6", item6);
userSettings.put("item7", item7);
userSettings.put("i0", Float.toString(i0));
userSettings.put("i1", Float.toString(i1));
userSettings.put("i2", Float.toString(i2));
userSettings.put("i3", Float.toString(i3));
userSettings.put("i4", Float.toString(i4));
userSettings.put("i5", Float.toString(i5));
userSettings.put("i6", Float.toString(i6));
userSettings.put("i7", Float.toString(i7));
userSettings.put("huvudMaskin", huvudMaskin);
userSettings.put("minorMaskin1", minorMaskin1);
userSettings.put("minorMaskin2", minorMaskin2);
userSettings.put("maskinTid", Float.toString(maskinTid));
writeSettings();
}
public void writeSettings() {
try
{
FileOutputStream fos = context.openFileOutput("test.ser", Context.MODE_PRIVATE);
ObjectOutputStream oos = new ObjectOutputStream(fos);
oos.writeObject(userSettings);
oos.close();
} catch (IOException e) {
e.printStackTrace();
}
}
public void readSetttings() {
try
{
FileInputStream fileInputStream = new FileInputStream(context.getFilesDir()+"test.ser");
ObjectInputStream objectInputStream = new ObjectInputStream(fileInputStream);
Map myHashMap = (Map)objectInputStream.readObject();
userSettings = null;
userSettings = myHashMap;
}
catch(ClassNotFoundException | IOException | ClassCastException e) {
e.printStackTrace();
}
executeSettings();
}
I have both read and write rights to the app.
Im not getting anything out of this. I've checked the hashmap, and it works as intended. I have also tried a lot of different approaches, and the only thing I managed to get working was saving strings to a .txt file.
private String subFolder = "/userdata";
private String file = "test.ser";
public void writeSettings() {
File cacheDir = null;
File appDirectory = null;
if (android.os.Environment.getExternalStorageState().
equals(android.os.Environment.MEDIA_MOUNTED)) {
cacheDir = getApplicationContext().getExternalCacheDir();
appDirectory = new File(cacheDir + subFolder);
} else {
cacheDir = getApplicationContext().getCacheDir();
String BaseFolder = cacheDir.getAbsolutePath();
appDirectory = new File(BaseFolder + subFolder);
}
if (appDirectory != null && !appDirectory.exists()) {
appDirectory.mkdirs();
}
File fileName = new File(appDirectory, file);
FileOutputStream fos = null;
ObjectOutputStream out = null;
try {
fos = new FileOutputStream(fileName);
out = new ObjectOutputStream(fos);
out.writeObject(userSettings);
} catch (IOException ex) {
ex.printStackTrace();
} catch (Exception e) {
e.printStackTrace();
} finally {
try {
if (fos != null)
fos.flush();
fos.close();
if (out != null)
out.flush();
out.close();
} catch (Exception e) {
}
}
}
public void readSetttings() {
File cacheDir = null;
File appDirectory = null;
if (android.os.Environment.getExternalStorageState().
equals(android.os.Environment.MEDIA_MOUNTED)) {
cacheDir = getApplicationContext().getExternalCacheDir();
appDirectory = new File(cacheDir + subFolder);
} else {
cacheDir = getApplicationContext().getCacheDir();
String BaseFolder = cacheDir.getAbsolutePath();
appDirectory = new File(BaseFolder + subFolder);
}
if (appDirectory != null && !appDirectory.exists()) return; // File does not exist
File fileName = new File(appDirectory, file);
FileInputStream fis = null;
ObjectInputStream in = null;
try {
fis = new FileInputStream(fileName);
in = new ObjectInputStream(fis);
Map<String, String> myHashMap = (Map<String, String> ) in.readObject();
userSettings = myHashMap;
System.out.println("count of hash map::"+userSettings.size() + " " + userSettings);
} catch (FileNotFoundException e) {
e.printStackTrace();
} catch (StreamCorruptedException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
} catch (ClassNotFoundException e) {
e.printStackTrace();
} catch (Exception e) {
e.printStackTrace();
}finally {
try {
if(fis != null) {
fis.close();
}
if(in != null) {
in.close();
}
} catch (IOException e) {
e.printStackTrace();
}
}
}
Your problem is very simple: you are using two different file names when writing the data resp. reading it.
FileOutputStream fos = context.openFileOutput("test.ser", Context.MODE_PRIVATE);
vs.
FileInputStream fileInputStream = new FileInputStream(context.getFilesDir()+"test.ser");
And, most likely, your reading code did throw an IOException at you, telling you something about trying to open a file that doesn't exist.
Thus, the real take-away/answer here: read those exception messages very carefully. Typically, they tell you exactly what the problem is!
Change these lines :
public void readSetttings(){
String path=context.getFilesDir() + File.seprator + "test.ser";
if(! new File(path).exists() ){
//throw NullPointerException ;
//return;
/*
*you can choose one of these
*pay attention : when choose NullPointerException you shold add throws Exceptions on your method
*/
}
try{
FileInputStream fileInputStream =context.openFileInput("test.ser");
ObjectInputStream objectInputStream = new ObjectInputStream(fileInputStream);
Map myHashMap = (Map)objectInputStream.readObject();
userSettings = myHashMap;
}catch(ClassNotFoundException | IOException | ClassCastException e) {
e.printStackTrace();
}
executeSettings();
}
If it is only primitives that you want to store then you should be using SharedPreferences which Android provides out of the box.
public static final String PREFS = "usersettings";
#Override
protected void onCreate(Bundle b){
.....
// read user settings on start
SharedPreferences settings = getSharedPreferences(PREFS, 0);
int someId = settings.getInteger("someId", 0);
setSomeId(id);
}
#Override
protected void onStop(){
.....
SharedPreferences settings = getSharedPreferences(PREFS, 0);
SharedPreferences.Editor editor = settings.edit();
editor.putInteger("someId", mSomeId);
// commit changes on exit
editor.commit();
}
How can I save an ArrayList to a file?
What am I doing wrong?
I have used this SO question to help me with Serializable objects.:
how to serialize ArrayList on android
and I used this SO question on how to write an array list:
Java - How Can I Write My ArrayList to a file, and Read (load) that file to the original ArrayList?
However when I attempt to write the to the file I get the error:
java.io.NotSerializableException: at
java.io.ObjectOutputStream.writeObject at
com.mycompany.MyClass.saveData
Here is MyClass that attempts to save the file
private ArrayList < MyCustomObject > arrayList;
private File dataFile;
private String FILE_NAME = "FILE_DATA.dat";
public void init(final Context context) {
this.appContext = context;
dataFile = new File(appContext.getFilesDir(), FILE_NAME);
if (dataFile.exists()) {
loadData();
} else {
try {
dataFile.createNewFile();
} catch (IOException e) {
e.printStackTrace();
}
arrayList = new ArrayList < MyCustomObject > ();
saveData();
}
}
private void saveData() {
FileOutputStream fos = null;
try {
fos = new FileOutputStream(dataFile);
} catch (FileNotFoundException e) {
e.printStackTrace();
} finally {
if (fos != null) {
ObjectOutputStream oos = null;
try {
oos = new ObjectOutputStream(fos);
if (oos != null) {
oos.writeObject(arrayList);
}
assert oos != null;
oos.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
}
private void loadData() {
FileInputStream fis = null;
try {
fis = new FileInputStream(dataFile);
} catch (FileNotFoundException e) {
e.printStackTrace();
} finally {
if (fis != null) {
ObjectInputStream ois = null;
try {
ois = new ObjectInputStream(fis);
if (ois != null) {
try {
arrayList = (ArrayList < MyCustomObject > ) ois.readObject();
} catch (ClassNotFoundException e) {
e.printStackTrace();
}
} else {
}
assert ois != null;
ois.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
}
Here is MyCustomObject
public class MyCustomObject implements Serializable {
public String myname = "";
public String someOtherItem = "";
public int aNumber = 0;
public MyCustomObject getCustomObject() {
return this;
}
}
Replace this method
public MyCustomObject MyCustomObject() {
return this;
}
in your MyCustomObject class and your code should work fine. Use something like
public MyCustomObject getMyCustomObject() {
return this;
}
Because the way you name your method is conflicting wit the default constructor that java creates for MyCustomObject class when you do not provide a constructor yourself. I assume that you are using this method to be able to add an instance of MyCustomObject to your array list: you don't really need such a method but with the proper naming you can still use it.
You should also put sample datas in your ArrayList before saving it to the disk by calling the saveData() method.
Here is an illustration from your code that works. I am not sure what your Context object is exactly but your are using it to get access to the file path, so to get things going I just used a particular file path.
public class MyClass {
private ArrayList < MyCustomObject > arrayList;
private File dataFile;
private String FILE_NAME = "FILE_DATA.dat";
public void init(final Context context) {
dataFile = new File("C:\\lompo\\file1.txt");
if (dataFile.exists()) {
loadData();
} else {
try {
dataFile.createNewFile();
} catch (IOException e) {
e.printStackTrace();
}
arrayList = new ArrayList < MyCustomObject > ();
MyCustomObject obj1 = new MyCustomObject();
obj1.aNumber = 125;
obj1.myname = "HIS NAME";
arrayList.add(obj1);
saveData();
}
}
public static void main(String[] args) {
MyClass myClazz = new MyClass();
myClazz.init(null);
System.out.println("Arraylist has " + myClazz.arrayList.size() + " elements");
}
private void saveData() {
FileOutputStream fos = null;
try {
fos = new FileOutputStream(dataFile);
} catch (FileNotFoundException e) {
e.printStackTrace();
} finally {
if (fos != null) {
ObjectOutputStream oos = null;
try {
oos = new ObjectOutputStream(fos);
if (oos != null) {
oos.writeObject(arrayList);
}
assert oos != null;
oos.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
}
private void loadData() {
FileInputStream fis = null;
try {
fis = new FileInputStream(dataFile);
} catch (FileNotFoundException e) {
e.printStackTrace();
} finally {
if (fis != null) {
ObjectInputStream ois = null;
try {
ois = new ObjectInputStream(fis);
if (ois != null) {
try {
arrayList = (ArrayList < MyCustomObject > ) ois.readObject();
} catch (ClassNotFoundException e) {
e.printStackTrace();
}
} else {
}
assert ois != null;
ois.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
}
}
As you can see in the code, the first run of the main method will save the file on disk with an arrayList populated by one object. The second run reads from the file and then I printed the number of elements and the infos that I have saved before: the picture illustrates the result
I have been trying to create a class called TextFileReaderWriter I want to use the getters and setters to read and write to a text file in such a way that I can call the class and the method from anywhere in the program by simply using setfileContents(somestring) and somestring = getfileContents() something like this
example:
TextFileReaderWriter trw = new TextFileReaderWriter();
trw.setfileContents(somestring); //this would write 'somestring' to the text file.
String somestring = trw.getfileContents(); //this would return 'somestring' from the text file.
Here's what I have so far but it writes nothing to the file:
public class TextFileReaderWriter extends Activity{
String fileContents;
Context context;
String TAG = "MYTAG";
public TextFileReaderWriter(String fileContents, Context context) {
this.fileContents = fileContents;
this.context = context;
}
public String getFileContents() {
return fileContents;
}
public void setFileContents(String fileContents) {
this.fileContents = fileContents;
FileOutputStream fos = null;
try {
fos = context.openFileOutput("UserInputStore", Context.MODE_PRIVATE);
} catch (FileNotFoundException e) {
e.printStackTrace();
}
OutputStreamWriter osw = new OutputStreamWriter(fos);
try {
osw.write(fileContents);
Log.d(TAG, fileContents);
} catch (IOException e) {
e.printStackTrace();
}
}
}
You don't need the OutputStreamWriter--FileOutputStreamwill do the trick just fine.
//what you had before
FileOutputStream fos = null;
try {
fos = context.openFileOutput(filename, Context.MODE_PRIVATE);
} catch (FileNotFoundException e) {
e.printStackTrace();
}
//use just the file output stream to write the data
//data here is a String
if (fos != null) {
try {
fos.write(data.getBytes());
fos.close();
} catch (IOException e) {
e.printStackTrace();
}
}
Method to save data on disk :
protected static void saveDataOnDisk(String data) {
ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream();
try {
ObjectOutput objectOutput = new ObjectOutputStream(byteArrayOutputStream);
objectOutput.writeObject(data);
byte[] buffer = byteArrayOutputStream.toByteArray();
File loginDataFile = (new File(filePath)); // file path where you want to write your data
loginDataFile.createNewFile();
FileOutputStream fileOutputStream = new FileOutputStream(loginDataFile);
fileOutputStream.write(buffer);
fileOutputStream.close();
objectOutput.flush();
objectOutput.close();
byteArrayOutputStream.flush();
byteArrayOutputStream.close();
Log.i(“SAVE”, ”———————-DONE SAVING”);
} catch(IOException ioe) {
Log.i(“SAVE”, “———serializeObject|”+ioe);
}
}
Method to fetch data from disk:
private static Object getDataFromDisk() {
try {
FileInputStream fileInputeStream = new FileInputStream(FilePath);
ObjectInputStream objectInputStream = new ObjectInputStream(fileInputeStream);
Object data = (Object) objectInputStream.readObject();
objectInputStream.close();
fileInputeStream.close();
return dataModel;
} catch (Exception error) {
Log.i(“FETCH”, ”—-getDataFromDisk———ERROR while reading|” + error);
}
return null;
}
while downloading the object from amazon s3 i cant able to download it to diffrent folder rather then the file uploaded path...why it is happening like this may be it is metaData problem....please post your valuable comments Thanks in advance..below am posting the code for upload and download
public void AmazonUpload(String fileObj) throws IOException {
try {
this.key = fileObj;
try {
if (this.key == null) {
} else {
if (readFile(this.key) != null) {
// this.key="1";
this.putObjResult = this.amzObj.putObject(new PutObjectRequest(this.bucketName, this.key, readFile(this.key)));
}
}
} catch (AmazonServiceException ae) {
System.out.println(ae.getMessage());
}
} catch (AmazonServiceException ex) {
Logger.getLogger(AthinioCloudMigration.class.getName()).log(Level.SEVERE, null, ex);
}
}
public void AmazonDownload(String dirName, String xmlFilename, String amazonid) throws ParserConfigurationException, SAXException, TransformerException, IOException {
String cloudid;
cloudid = amazonid;
this.comm = new CommonResources(xmlFilename);
this.RequestFiles=new ArrayList();
try {
this.RequestFiles = this.comm.getXML(cloudid);
if (this.RequestFiles != null) {
int len = this.RequestFiles.size();
System.out.println(len);
for (int index = 0; index < len; index++) {
this.CRobj = (CommonResources) this.RequestFiles.get(index);
if (cloudid.equals(this.CRobj.getCloudID())) {
this.newFile = new File(dirName + this.CRobj.getFileName().concat(".rec"));
System.out.println(newFile);
newFile.createNewFile();
this.metaData = this.amzObj.getObject(new GetObjectRequest(this.bucketName, (dirName + this.CRobj.getFileName())), this.newFile);
System.out.println(metaData);
java.io.File tmp = new java.io.File(dirName + this.CRobj.getFileName());
System.out.println(tmp);
tmp.delete();
76,23 87%
Since in Amazon S3 there is no folder structure you receive everything as a object.
For Example: In your bucket you store file in a folder like structure but when you request fro the objects from S3 you will receve your file like folder1/folder2/demo.txt.
So try this one, Get the InputStream for the S3 for your file like amazonS3.getObject(bucket, "folder1/folder2/demo.txt").getObjectContent();. After getting your InputStream pass your File Name, Download location and InputStream to the below method. If you use Java 7 use FileSystems.getDefault().getPath(fullPathWithFileName).getFileName().toString() to get file name from your object name.
public void saveFile(String uploadFileName, String path, InputStream inputStream) throws Exception {
DataOutputStream dos = null;
OutputStream out = null;
try {
File newDirectory = new File(path);
if (!newDirectory.exists()) {
newDirectory.mkdirs();
}
File uploadedFile = new File(path, uploadFileName);
out = new FileOutputStream(uploadedFile);
byte[] fileAsBytes = new byte[inputStream.available()];
inputStream.read(fileAsBytes);
dos = new DataOutputStream(out);
dos.write(fileAsBytes);
} catch (IOException io) {
io.printStackTrace();
} catch (Exception e) {
e.printStackTrace();
} finally {
try {
if (out != null) {
out.close();
}
if (dos != null) {
dos.close();
}
} catch (IOException e) {
e.printStackTrace();
}
}
}