We Keep Coding

Informix db clob with utf-8 characters stores incomplete data

I am facing a problem with my Informix db while storing CLOB data with UTF-8 charset data.
The regular varchar datatype with UTF-8 characters is storing without any issue; however, when I store the data as a CLOB and data (in my case XML) contains utf-8 characters, only truncated XML is stored.
Example Characters: german umlaut
Informix db character encoding is set to utf-8.
Below is my hbm file
<?xml version="1.0"?>
<!DOCTYPE hibernate-mapping PUBLIC
"-//Hibernate/Hibernate Mapping DTD 3.0//EN"
"http://hibernate.sourceforge.net/hibernate-mapping-3.0.dtd">
<hibernate-mapping package="com.test.adapter.persistence.bean">
<class name="MediaLayoutConfigPojo" table="medialayoutconfig">
<id name="dbId" type="int" column="id">
<generator class="native" />
</id>
<property name="layoutxml" type="clob" column="layoutxml" />
<property name="name" type="string" column="name" />
<property name="description" type="string" column="description" />
<property name="layouttype" type="string" column="layouttype" />
</class>
</hibernate-mapping>
SQL driver class is
<property name="connection.driver_class">com.informix.jdbc.IfxDriver</property>
<property name="dialect">org.hibernate.dialect.InformixDialect</property>
I am using the ifxjdbc-3.50.JC9.jar JDBC driver (unfortunately, I'm stuck with it; upgrading it may cause additional breakages since it's a legacy application).
Hibernate version used is: hibernate-core-3.6.3.Final.jar
I went through this link https://hibernate.atlassian.net/browse/HHH-6127. Not sure If I am hitting the same issue in driver class.
My table definition and sample data I am trying store and what exactly stored in given below:
Here is my table definition:
Column name Type Nulls
id serial no
layouttype varchar(32,0) no
name lvarchar(256) no
description lvarchar(512) yes
layoutxml clob no
This the data I am trying to insert into CLOB layoutxml:
<MediaPropertiesLayout><name>Система</name><description></description><header><entry><showInPopOver/><displayName>Система</displayName><mediaProperty>callVariable1</mediaProperty><uiEditable>false</uiEditable></entry></header><column></column><column></column><uri></uri></MediaPropertiesLayout>
Actual data stored in layoutxml (CLOB)
<MediaPropertiesLayout><name>Система</name><description></description><header><entry><showInPopOver/><displayName>Система</displayName><mediaProperty>callVariable1</mediaProperty><uiEditable>false</uiEditable></entry></header><column></column><column></column><uri>
i.e. Closing tags for uri and MediaPropertiesLayout is truncated. This will vary depending upon the UTF-8 characters I use. If I use larger string instead of "Система", then truncation will be at different place.
Any pointer will be great help.

Are you sure you are passing a correct UTF8 value?
Usually when you get truncation is because the encoding is wrong.
I tried inserting the UTF8 string 'Füße' (46 C3BC C39F 65) with a simple Java console app and it behaves as expected.
D:\Infx\ids1410>chcp 65001
Active code page: 65001
D:\Infx\ids1410>java com.informix.jdbc.Version
IBM Informix JDBC Driver Version 3.50.JC9X3
D:\Infx\ids1410>javac -encoding utf8 clob.java
D:\Infx\ids1410>od -tcx1 clob.txt
0000000000 F e
46 C3 BC C3 9F 65
0000000006
D:\Infx\ids1410>java -Dfile.encoding=UTF-8 clob
--Füße--Füße--
D:\Infx\ids1410>oncheck -pp utf8:t1 256
addr stamp chksum nslots flag type frptr frcnt next prev
2:4566 36224780 acf0 1 1 DATA 106 3982 0 0
slot ptr len flg
1 24 82 0
slot 1:
0: 6 46 c3 bc c3 9f 65 0 0 0 1 0 0 0 d9 c8 .FC<C.e.......YH
16: b7 a6 3 0 0 0 3 0 0 0 3 0 0 0 a1 f8 7&............!x
32: 10 61 0 0 0 0 1 0 0 0 0 0 0 0 0 0 .a..............
48: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ................
64: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ................
80: 0 0 ................
D:\Infx\ids1410>dbaccess utf8 -
Database selected.
> unload to t1.unl select * from t1;
1 row(s) unloaded.
> !od -x t1.unl
0000000000 C346 C3BC 659F 307C 362C 632C 6F6C 3162
0000000020 3031 2E66 6138 7C31 000A
0000000031
> !cat t1.unl
Füße|0,6,clob110f.8a1|
> !od -x clob110f.8a1
0000000000 C346 C3BC 659F
0000000006
> !cat clob110f.8a1
Füße>
>
I don't have 3.50.JC9 at hand, but 3.50.JC8 and 3.50.JC9X3 did the same so I'm sure the 'vanilla' 3.50.JC9 will work too.
Maybe you can try the console app with your xml data (make sure it is in UTF8) and see if it still gets corrupted or not.
// ----- clob.java -----
import java.sql.*;
import java.io.*;
import com.informix.jdbc.*;
public class clob {
public static void main(String[] args) {
Connection con = null;
try {
InputStreamReader isr = new InputStreamReader( System.in );
BufferedReader stdin = new BufferedReader( isr );
Class.forName("com.informix.jdbc.IfxDriver");
con = DriverManager.getConnection("jdbc:informix-sqli://420ito:9091/utf8:INFORMIXSERVER=ids1410;user=informix;password=mypassw;DB_LOCALE=en_US.utf8;SQLIDEBUG=pp");
System.out.println ("");
Statement stm1 = con.createStatement();
try {
stm1.executeUpdate ("drop table t1;");
} catch (Exception ex) {}
stm1.executeUpdate ("create table t1 (c1 varchar(20), c2 clob);");
PreparedStatement ps = con.prepareStatement("insert into t1 values ('Füße',?);");
FileInputStream fis = new FileInputStream("clob.txt");
BufferedInputStream dataStream = new BufferedInputStream(fis);
ps.setAsciiStream(1, dataStream, dataStream.available());
ps.execute();
ps.close();
PreparedStatement ps2 = con.prepareStatement("select * from t1;");
IfxResultSet resultSet = (IfxResultSet) ps2.executeQuery();
while (resultSet.next()) {
System.out.format("--%s--%s--\n",resultSet.getString(1),resultSet.getString(2));
try {
} catch ( Exception ex ) {};
}
resultSet.releaseBlob();
ps2.close();
((IfxSqliConnect) con).releaseBlob();
} catch (Exception ex) {
ex.printStackTrace();
if (ex instanceof SQLException) {
SQLException sqlex = (SQLException) ex;
System.out.println ("Message: " + sqlex.getMessage());
}
}
}
}
// ----- clob.java -----
If it works the issue maybe in how Hibernate handles that data.

Issue still exists in creating Clob in Hibernate. Somehow, Hibernate not able to calculate the string length based on UTF-8 characters. When you have these kind of special character, you need layoutXML.getBytes("utf-8").length instead layoutXML.length(). This difference is actually causing the truncation.
However, I skipped the entire conversion overhead all together, just by declaring 'layoutxml' as String in POJO instead of java.sql.Clob. For this I upgraded the hibernate version to 5.6.1.Final.
With 3.5.2-Final hibernate verison, even this approach doesn’t work.
Note: Declaring CLOB entries as Strings in POJO has its own disadvantages related to performance.

Apache Commons Imaging (EXIF): Setting tags/tags missing

I'm using the Apache Commons Imaging library (Java 8, you can find my code here) and I've come across a few problems with tags:
If I open the image info of e.g. this .jpg file with Win 10, there are "Origin" tags, e.g. "Authors" and "Date acquired":
You can find a list of EXIF tags here and it includes the "Authors" one and also an additional "Artist" one. The "Artist" tag seemed to have existed in the library at one point (source) but the apidocs don't list it anymore and ExifTagConstants.TIFF_TAG_ARTIST doesn't exist. Same thing with the "GPSAltitude" tag: It should exist according to the EXIF list but I can't seem to find it in the library.
I tried to use the "Maker" tag instead:
final TiffOutputDirectory exifDir = outputSet.getOrCreateExifDirectory();
exifDir.removeField(ExifTagConstants.EXIF_TAG_MAKER_NOTE);
exifDir.add(ExifTagConstants.EXIF_TAG_MAKER_NOTE, "Test Maker");
But ExifTagConstants.EXIF_TAG_MAKER_NOTE's type of TagInfoUndefineds doesn't seem to be valid for exifDir.add.
I also tried to add the date:
exifDir.removeField(ExifTagConstants.EXIF_TAG_DATE_TIME_DIGITIZED);
exifDir.add(ExifTagConstants.EXIF_TAG_DATE_TIME_DIGITIZED, "1970/01/01");
This just throws an exception:
org.apache.commons.imaging.ImageWriteException: Tag expects 20 byte(s), not 1
The only tag I've managed to successfully write so far is ExifTagConstants.EXIF_TAG_USER_COMMENT.
How do I use/write these tags (author/artist, date, altitude,...)?
Edit:
I managed to find two of the tags:
exifDir.add(MicrosoftTagConstants.EXIF_TAG_XPAUTHOR, "Test Author");//Author
exifDir.add(GpsTagConstants.GPS_TAG_GPS_ALTITUDE, new RationalNumber(3, 1));//Altitude
... but they aren't written into the file.
exifDir.add(ExifTagConstants.EXIF_TAG_USER_COMMENT, "my comment");
works, so I know it's actually writing tags, it just doesn't work for the two above. Any idea what's wrong?

As #haraldK already mentioned: GPS data isn't part of the actual EFIX directory, which is also the case in the "Apache Commons Imaging" library.
So instead of writing the altitude with
double someDouble = 123.123456789;
int alt = (int) Math.round(someDouble*1000); //round to 3 decimal places
final TiffOutputDirectory exifDir = outputSet.getOrCreateExifDirectory();
exifDir.removeField(GpsTagConstants.GPS_TAG_GPS_ALTITUDE);
exifDir.add(GpsTagConstants.GPS_TAG_GPS_ALTITUDE, new RationalNumber(alt, 1000));
use:
final TiffOutputDirectory gpsDir = outputSet.getOrCreateGPSDirectory();
gpsDir.removeField(GpsTagConstants.GPS_TAG_GPS_ALTITUDE);
gpsDir.add(GpsTagConstants.GPS_TAG_GPS_ALTITUDE, new RationalNumber(alt, 1000));
This will write "123.123" into the "Altitude" field that can be viewed through the Windows Explorer (right-click on image -> Properties -> Details).
As for the other tags:
final TiffOutputDirectory exifDir = outputSet.getOrCreateExifDirectory();
final TiffOutputDirectory gpsDir = outputSet.getOrCreateGPSDirectory();
final TiffOutputDirectory rootDir = outputSet.getOrCreateRootDirectory();
final TiffOutputDirectory intDir = outputSet.getInteroperabilityDirectory(); //Not sure what this one is used for
//Writing into the "Authors" field
rootDir.removeField(MicrosoftTagConstants.EXIF_TAG_XPAUTHOR);
rootDir.add(MicrosoftTagConstants.EXIF_TAG_XPAUTHOR, "Me");
//Writing into the "Program Name" field
rootDir.removeField(ExifTagConstants.EXIF_TAG_SOFTWARE);
rootDir.add(ExifTagConstants.EXIF_TAG_SOFTWARE, "My App");
//Writing into the "Date taken" field
exifDir.removeField(ExifTagConstants.EXIF_TAG_DATE_TIME_ORIGINAL);
exifDir.add(ExifTagConstants.EXIF_TAG_DATE_TIME_ORIGINAL, "1970:01:01 12:34:56");
//Writing into the "Digitized Date"
exifDir.removeField(ExifTagConstants.EXIF_TAG_DATE_TIME_DIGITIZED);
exifDir.add(ExifTagConstants.EXIF_TAG_DATE_TIME_DIGITIZED, "1970:01:01 12:34:56");
//Not visible in "Properties" (use image editing software to see it)
//Writing the GPS time stamp
gpsDir.removeField(GpsTagConstants.GPS_TAG_GPS_DATE_STAMP);
gpsDir.add(GpsTagConstants.GPS_TAG_GPS_DATE_STAMP, "1970:01:01");
//Apparently only writes year & not visible in "Properties" (use image editing software to see it)
I haven't found the right tag for "Date Acquired" yet, so if someone knows the exact one, please comment on this answer.

Old way to go :
You can looking at a specification of EXIF metadatas and you can implement some custom parsing to get them with How to extract EXIF metadatas from JPEG files.
Recent way to go :
You can read EXIF datas with that ImageData class which is working very well
Now if you want to modify/update the images metadatas, you can use the documentation in which you will find the following method :
/**
48 * This example illustrates how to add/update EXIF metadata in a JPEG file.
49 *
50 * #param jpegImageFile
51 * A source image file.
52 * #param dst
53 * The output file.
54 * #throws IOException
55 * #throws ImageReadException
56 * #throws ImageWriteException
57 */
58 public void changeExifMetadata(final File jpegImageFile, final File dst)
59 throws IOException, ImageReadException, ImageWriteException {
60
61 try (FileOutputStream fos = new FileOutputStream(dst);
62 OutputStream os = new BufferedOutputStream(fos);) {
63
64 TiffOutputSet outputSet = null;
65
66 // note that metadata might be null if no metadata is found.
67 final ImageMetadata metadata = Imaging.getMetadata(jpegImageFile);
68 final JpegImageMetadata jpegMetadata = (JpegImageMetadata) metadata;
69 if (null != jpegMetadata) {
70 // note that exif might be null if no Exif metadata is found.
71 final TiffImageMetadata exif = jpegMetadata.getExif();
72
73 if (null != exif) {
74 // TiffImageMetadata class is immutable (read-only).
75 // TiffOutputSet class represents the Exif data to write.
76 //
77 // Usually, we want to update existing Exif metadata by
78 // changing
79 // the values of a few fields, or adding a field.
80 // In these cases, it is easiest to use getOutputSet() to
81 // start with a "copy" of the fields read from the image.
82 outputSet = exif.getOutputSet();
83 }
84 }
85
86 // if file does not contain any exif metadata, we create an empty
87 // set of exif metadata. Otherwise, we keep all of the other
88 // existing tags.
89 if (null == outputSet) {
90 outputSet = new TiffOutputSet();
91 }
92
93 {
94 // Example of how to add a field/tag to the output set.
95 //
96 // Note that you should first remove the field/tag if it already
97 // exists in this directory, or you may end up with duplicate
98 // tags. See above.
99 //
100 // Certain fields/tags are expected in certain Exif directories;
101 // Others can occur in more than one directory (and often have a
102 // different meaning in different directories).
103 //
104 // TagInfo constants often contain a description of what
105 // directories are associated with a given tag.
106 //
107 final TiffOutputDirectory exifDirectory = outputSet.getOrCreateExifDirectory();
108 // make sure to remove old value if present (this method will
109 // not fail if the tag does not exist).
110 exifDirectory.removeField(ExifTagConstants.EXIF_TAG_APERTURE_VALUE);
111 exifDirectory.add(ExifTagConstants.EXIF_TAG_APERTURE_VALUE,
112 new RationalNumber(3, 10));
113 }
114
115 {
116 // Example of how to add/update GPS info to output set.
117
118 // New York City
119 final double longitude = -74.0; // 74 degrees W (in Degrees East)
120 final double latitude = 40 + 43 / 60.0; // 40 degrees N (in Degrees
121 // North)
122
123 outputSet.setGPSInDegrees(longitude, latitude);
124 }
125
126 // printTagValue(jpegMetadata, TiffConstants.TIFF_TAG_DATE_TIME);
127
128 new ExifRewriter().updateExifMetadataLossless(jpegImageFile, os,
129 outputSet);
130 }
131 }
I have just found a usefull example for your use case I think. You will know now how to find them, I am sure it will be easier to modify them :
#Override
public Exif getExif(Photo photo) throws ServiceException {
File file = new File(photo.getPath());
String exposure = "not available";
double aperture = 0.0;
double focalLength = 0.0;
int iso = 0;
boolean flash = false;
String make = "not available";
String model = "not available";
double altitude = 0.0;
try {
final ImageMetadata metadata = Imaging.getMetadata(file);
final JpegImageMetadata jpegMetadata = (JpegImageMetadata) metadata;
if (jpegMetadata.findEXIFValueWithExactMatch(ExifTagConstants.EXIF_TAG_EXPOSURE_TIME)
!= null) {
exposure = jpegMetadata
.findEXIFValueWithExactMatch(ExifTagConstants.EXIF_TAG_EXPOSURE_TIME)
.getValueDescription().split(" ")[0];
}
if (jpegMetadata.findEXIFValueWithExactMatch(ExifTagConstants.EXIF_TAG_APERTURE_VALUE)
!= null) {
aperture = jpegMetadata
.findEXIFValueWithExactMatch(ExifTagConstants.EXIF_TAG_APERTURE_VALUE)
.getDoubleValue();
}
if (jpegMetadata.findEXIFValueWithExactMatch(ExifTagConstants.EXIF_TAG_FOCAL_LENGTH)
!= null) {
focalLength = jpegMetadata
.findEXIFValueWithExactMatch(ExifTagConstants.EXIF_TAG_FOCAL_LENGTH)
.getDoubleValue();
}
if (jpegMetadata.findEXIFValueWithExactMatch(ExifTagConstants.EXIF_TAG_ISO) != null) {
iso = jpegMetadata.findEXIFValueWithExactMatch(ExifTagConstants.EXIF_TAG_ISO)
.getIntValue();
}
if (jpegMetadata.findEXIFValueWithExactMatch(ExifTagConstants.EXIF_TAG_FLASH) != null) {
flash = jpegMetadata.findEXIFValueWithExactMatch(ExifTagConstants.EXIF_TAG_FLASH)
.getIntValue() != 0;
}
if (jpegMetadata.findEXIFValueWithExactMatch(TiffTagConstants.TIFF_TAG_MAKE) != null) {
make = jpegMetadata.findEXIFValueWithExactMatch(TiffTagConstants.TIFF_TAG_MAKE)
.getValueDescription();
}
if (jpegMetadata.findEXIFValueWithExactMatch(TiffTagConstants.TIFF_TAG_MODEL) != null) {
model = jpegMetadata.findEXIFValueWithExactMatch(TiffTagConstants.TIFF_TAG_MODEL)
.getValueDescription();
}
if (jpegMetadata.findEXIFValueWithExactMatch(GpsTagConstants.GPS_TAG_GPS_ALTITUDE)
!= null) {
altitude = jpegMetadata
.findEXIFValueWithExactMatch(GpsTagConstants.GPS_TAG_GPS_ALTITUDE)
.getDoubleValue();
}
return new Exif(photo.getId(), exposure, aperture, focalLength, iso, flash, make, model,
altitude);
} catch (IOException | ImageReadException e) {
throw new ServiceException(e.getMessage(), e);
}
}

I found the Artist Tag:
exifDirectory.add(TiffTagConstants.TIFF_TAG_ARTIST, "JANE DOUGH");
Haven't tested whether this will work though.
EDIT:
It works well.
Online Exif Viewer Image
The image shows that the Artist Tag works.

Chronicle Consumer not reading records correctly?

I am using chronicle-queue (5.16.13) to write and read json values to a chronicle file.
To write objects I use the following in a loop
try (final DocumentContext dc = appender.writingDocument()) {
dc.wire().write(() -> "msg").text("Hallo asdf");
System.out.println("your data was store to index="+ dc.index());
return true;
} catch (Exception e) {
logger.warn("Unable to store value to chronicle", e);
return false;
}
and to read items I do the following call in a loop
DocumentContext documentContext;
do {
documentContext = tailer.readingDocument();
currentOffset = documentContext.index();
System.out.println("Current offset: " + currentOffset);
} while (!documentContext.isData());
What I observe is that the variable currentOffsetdoes not change and after some time (depending on the payload size, it seems) the loop goes infinite and the current offset has crazy values.
The output (shortened) for the first loop is
Writing 0
your data was store to index=76385993359360
Writing 1
your data was store to index=76385993359361
Writing 2
your data was store to index=76385993359362
Writing 3
your data was store to index=76385993359363
Writing 4
your data was store to index=76385993359364
Writing 5
your data was store to index=76385993359365
Writing 6
your data was store to index=76385993359366
Writing 7
your data was store to index=76385993359367
Writing 8
your data was store to index=76385993359368
Writing 9
your data was store to index=76385993359369
Writing 10
your data was store to index=76385993359370
Writing 11
your data was store to index=76385993359371
Writing 12
your data was store to index=76385993359372
Writing 13
your data was store to index=76385993359373
Writing 14
your data was store to index=76385993359374
Writing 15
your data was store to index=76385993359375
Writing 16
your data was store to index=76385993359376
Writing 17
your data was store to index=76385993359377
Writing 18
your data was store to index=76385993359378
Writing 19
your data was store to index=76385993359379
Writing 20
your data was store to index=76385993359380
Writing 21
your data was store to index=76385993359381
Writing 22
your data was store to index=76385993359382
Writing 23
your data was store to index=76385993359383
Writing 24
your data was store to index=76385993359384
Writing 25
your data was store to index=76385993359385
Writing 26
your data was store to index=76385993359386
And for the second loop
Reading 0
Current offset: 76385993359360
Reading 1
Current offset: 76385993359360
Reading 2
Current offset: 76385993359360
Reading 3
Current offset: 76385993359360
Reading 4
Current offset: 76385993359360
Reading 5
Current offset: 76385993359360
Reading 6
Current offset: 76385993359360
Reading 7
Current offset: 76385993359360
Reading 8
Current offset: 76385993359360
Reading 9
Current offset: 76385993359360
Reading 10
Current offset: 76385993359360
Reading 11
Current offset: 76385993359360
Reading 12
Current offset: 76385993359360
Reading 13
Current offset: 76385993359360
Reading 14
Current offset: 76385993359360
Reading 15
Current offset: 76385993359360
Reading 16
Current offset: 76385993359360
Reading 17
Current offset: 76385993359360
Reading 18
Current offset: 76385993359360
Reading 19
Current offset: 76385993359360
Reading 20
Current offset: 76385993359360
Reading 21
Current offset: 76385993359360
Reading 22
Current offset: 76385993359360
Reading 23
Current offset: 76385993359360
Reading 24
Current offset: 76385993359360
Reading 25
Current offset: -9223372036854775808
Am I doing something completely wrong?
Can then anyobody hint me to the correct usage?
Thanks a lot!
Edit: Added minimal working example
The following unit test fails for me.
#Test
public void fails() throws Exception {
String basePath = System.getProperty("java.io.tmpdir");
String path = Files.createTempDirectory(Paths.get(basePath), "chronicle-")
.toAbsolutePath()
.toString();
logger.info("Using temp path '{}'", path);
SingleChronicleQueue chronicleQueue = SingleChronicleQueueBuilder
.single()
.path(path)
.build();
// Create Appender
ExcerptAppender appender = chronicleQueue.acquireAppender();
// Create Tailer
ExcerptTailer tailer = chronicleQueue.createTailer();
tailer.toStart();
int numberOfRecords = 10;
// Write
for (int i = 0; i <= numberOfRecords; i++) {
System.out.println("Writing " + i);
try (final DocumentContext dc = appender.writingDocument()) {
dc.wire().write(() -> "msg").text("Hello World!");
System.out.println("your data was store to index=" + dc.index());
} catch (Exception e) {
logger.warn("Unable to store value to chronicle", e);
}
}
// Read
for (int i = 0; i <= numberOfRecords; i++) {
System.out.println("Reading " + i);
DocumentContext documentContext = tailer.readingDocument();
long currentOffset = documentContext.index();
System.out.println("Current offset: " + currentOffset);
Wire wire = documentContext.wire();
if (wire != null) {
String msg = wire
.read("msg")
.text();
}
}
chronicleQueue.close();
}
Output is
Writing 0
your data was store to index=76385993359360
Writing 1
your data was store to index=76385993359361
Writing 2
your data was store to index=76385993359362
Writing 3
your data was store to index=76385993359363
Writing 4
your data was store to index=76385993359364
Writing 5
your data was store to index=76385993359365
Writing 6
your data was store to index=76385993359366
Writing 7
your data was store to index=76385993359367
Writing 8
your data was store to index=76385993359368
Writing 9
your data was store to index=76385993359369
Writing 10
your data was store to index=76385993359370
Reading 0
Current offset: 76385993359360
Reading 1
Current offset: 76385993359360
Reading 2
Current offset: 76385993359360
Reading 3
Current offset: 76385993359360
Reading 4
Current offset: -9223372036854775808
Reading 5
Current offset: -9223372036854775808
Reading 6
Current offset: -9223372036854775808
Reading 7
Current offset: -9223372036854775808
Reading 8
Current offset: -9223372036854775808
Reading 9
Current offset: -9223372036854775808
Reading 10
Current offset: -9223372036854775808

Using DocumentContext is intended to be one of the lower level interfaces and not to everyone's taste. I favour using the MethodReader/MethodWriter approach unless you have a reason to work at the lower level.
#Test
public void works() {
String path = OS.TMP + "/chronicle-" + System.nanoTime();
System.out.println("Using temp path " + path);
try (SingleChronicleQueue queue = SingleChronicleQueueBuilder
.single()
.path(path)
.build()) {
ExcerptAppender appender = queue.acquireAppender();
Messager messager = appender.methodWriter(Messager.class);
int numberOfRecords = 10;
// Write
for (int i = 0; i <= numberOfRecords; i++) {
System.out.print("Writing " + i);
messager.msg("Hello World!");
System.out.println(", your data was stored at index=" + appender.lastIndexAppended());
}
ExcerptTailer tailer = queue.createTailer();
MethodReader reader = tailer.methodReader((Messager) msg -> {
System.out.println("Current offset: " + tailer.index()
+ " msg: " + msg);
});
// Read
while (reader.readOne()) {
// busy wait.
}
}
}
This prints
Using temp path C:\Users\peter\AppData\Local\Temp\/chronicle-412979753710181
[main] DEBUG net.openhft.chronicle.bytes.MappedFile - Allocation of 0 chunk in C:\Users\peter\AppData\Local\Temp\chronicle-412979753710181\metadata.cq4t took 15.418 ms.
[main] DEBUG net.openhft.chronicle.bytes.MappedFile - Allocation of 0 chunk in C:\Users\peter\AppData\Local\Temp\chronicle-412979753710181\20181226.cq4 took 25.061 ms.
Writing 0, your data was stored at index=76841259892736
Writing 1, your data was stored at index=76841259892737
Writing 2, your data was stored at index=76841259892738
Writing 3, your data was stored at index=76841259892739
Writing 4, your data was stored at index=76841259892740
Writing 5, your data was stored at index=76841259892741
Writing 6, your data was stored at index=76841259892742
Writing 7, your data was stored at index=76841259892743
Writing 8, your data was stored at index=76841259892744
Writing 9, your data was stored at index=76841259892745
Writing 10, your data was stored at index=76841259892746
Current offset: 76841259892736 msg: Hello World!
Current offset: 76841259892737 msg: Hello World!
Current offset: 76841259892738 msg: Hello World!
Current offset: 76841259892739 msg: Hello World!
Current offset: 76841259892740 msg: Hello World!
Current offset: 76841259892741 msg: Hello World!
Current offset: 76841259892742 msg: Hello World!
Current offset: 76841259892743 msg: Hello World!
Current offset: 76841259892744 msg: Hello World!
Current offset: 76841259892745 msg: Hello World!
Current offset: 76841259892746 msg: Hello World!
[main] DEBUG net.openhft.chronicle.queue.impl.single.SingleChronicleQueueBuilder - File released C:\Users\peter\AppData\Local\Temp\chronicle-412979753710181\20181226.cq4
[main] DEBUG net.openhft.chronicle.queue.impl.single.SingleChronicleQueueBuilder - File released C:\Users\peter\AppData\Local\Temp\chronicle-412979753710181\20181226.cq4
NOTE: This writes the same data as the original post.
An advantage of using this interface approach is that you can implement your business component entirely using interfaces of methods with DTOs and not use Chronicle (or at transport) at all. This simplifies testing business logic as you remove the transport from the tests.

I found the answer myself using the Suggestion from #PeterLawrey and wrapping the Document Context in try-with-resources. This solves the issue.
See the corrected snippet below
#Test
public void works() throws Exception {
String basePath = System.getProperty("java.io.tmpdir");
String path = Files.createTempDirectory(Paths.get(basePath), "chronicle-")
.toAbsolutePath()
.toString();
logger.info("Using temp path '{}'", path);
SingleChronicleQueue chronicleQueue = SingleChronicleQueueBuilder
.single()
.path(path)
.build();
// Create Appender
ExcerptAppender appender = chronicleQueue.acquireAppender();
// Create Tailer
ExcerptTailer tailer = chronicleQueue.createTailer();
tailer.toStart();
int numberOfRecords = 10;
// Write
for (int i = 0; i <= numberOfRecords; i++) {
System.out.println("Writing " + i);
try (final DocumentContext dc = appender.writingDocument()) {
dc.wire().write(() -> "msg").text("Hello World!");
System.out.println("your data was store to index=" + dc.index());
} catch (Exception e) {
logger.warn("Unable to store value to chronicle", e);
}
}
// Read
for (int i = 0; i <= numberOfRecords; i++) {
System.out.println("Reading " + i);
try (DocumentContext documentContext = tailer.readingDocument()) {
long currentOffset = documentContext.index();
System.out.println("Current offset: " + currentOffset);
Wire wire = documentContext.wire();
if (wire != null) {
String msg = wire
.read("msg")
.text();
}
}
}
chronicleQueue.close();
}
which produces the expected output
Writing 0
your data was store to index=76385993359360
Writing 1
your data was store to index=76385993359361
Writing 2
your data was store to index=76385993359362
Writing 3
your data was store to index=76385993359363
Writing 4
your data was store to index=76385993359364
Writing 5
your data was store to index=76385993359365
Writing 6
your data was store to index=76385993359366
Writing 7
your data was store to index=76385993359367
Writing 8
your data was store to index=76385993359368
Writing 9
your data was store to index=76385993359369
Writing 10
your data was store to index=76385993359370
Reading 0
Current offset: 76385993359360
Reading 1
Current offset: 76385993359361
Reading 2
Current offset: 76385993359362
Reading 3
Current offset: 76385993359363
Reading 4
Current offset: 76385993359364
Reading 5
Current offset: 76385993359365
Reading 6
Current offset: 76385993359366
Reading 7
Current offset: 76385993359367
Reading 8
Current offset: 76385993359368
Reading 9
Current offset: 76385993359369
Reading 10
Current offset: 76385993359370
Hope this helps someone else.

searching and replacing content from one file to another file using a keyword using python or java

Locked. There are disputes about this question’s content being resolved at this time. It is not currently accepting new answers or interactions.
I have 2 files which contains some data like this!!
File 1 contains:
/begin MENT AE0DAQ0O41 ""
ECU_ADDRESS 0x8111DSCC
ECU_ADDRESS_EXTENSION 0x0
/begin IF_DATA CAN_EXT
120
LINK_MAP "AE0DAQ0O41" 0x8111DSCC 0x0 0 0x2 1 0x2F 0x1
DISPLAY 0 0 655
/end IF_DATA
SYMBOL_LINK "AE0DAQ0O41" 0
/end MENT
File 2 contains:
name value line keyword
.data 80008114+000005 AE0DAQ0O43
.data 80008116+000005 AE0DAQ0O41
.data 80008118+000005 EA0DAQ0O45
.data 8000811a+000005 AE0DAF0O89
Now what we need to do is take a keyword AE0DAQ0O41 and need to search in the next file.
It has some value before the keyword, so we need to take that value 80008116 and need to replace it in
ECU_ADDRESS 0x8111DSCC and also LINK_MAP AE0DAQ0O41 0x8111DSCC 0x0 0 0x2 1 0x2F 0x1 for (0x8111DSCC it needs to be 0x80008116) and save it to FILE 1.
FILE 1 is to be saved like this :
/begin MENT AE0DAQ0O41 ""
ECU_ADDRESS 0x80008116
ECU_ADDRESS_EXTENSION 0x0
/begin IF_DATA CAN_EXT
120
LINK_MAP "AE0DAQ0O41" 0x80008116 0x0 0 0x2 1 0x2F 0x1
DISPLAY 0 0 655
/end IF_DATA
SYMBOL_LINK "AE0DAQ0O41" 0
/end MENT
How do we do that ??? because it has multiple lines like this ????
Thanks in advance!!!!!!!

If you consider your File 2 as a tab seperated value file then you could read the File 1 line by line and then compare the keyword in the file1 with each line in file2.
When you get a match then write another file with the new inputs

Quick and dirty solution:
(Assuming that the inputs are both text files...)
The code creates a dictionary by mining the second file.
The first file is processed line by line and written to the output file after the required modifications.
This is certainly not the best way to go about it.
If you know the exact format of the files, you can optimize the code to run a lot faster.
fout = open('output.txt' , 'w')
beg, ecu, lnk = '/begin','ECU_ADDRESS', 'LINK_MAP'
keyVal = dict()
with open('file2.txt') as f2:
for line in f2:
b = line.split(' ')
newK, newV = b[-1].replace('\n','') , b[-2].split('+')[0]
keyVal[newK] = newV
with open('file1.txt') as f1:
value,keyword = '', ''
for line in f1:
a = line.split(' ')
loc = 0
if beg in a and 'MENT' in a:
keyword = a[a.index(beg)+2]
value = '0x'+keyVal.get(keyword,keyword)
elif ecu in a:
loc = a.index(ecu) + 1
elif lnk in a: loc = a.index(lnk) + 2
else : loc = 0
if loc != 0:
a[loc] = value
a = ' '.join(a)
fout.writelines(a)
fout.close()

Search & display results [java]

I run a small online gaming community and deal with a database of accounts.
The setup is this:
Folder named Accounts
Inside the Accounts directory, there is 200,000+ text files organized by player name. Access to this folder manually is a pain because of the needed RAM to get in and search files. I find this very inconvenient.
I access this directory to send password reminders or for highscores on who has been playing the longest.
Here is an example of an account file. This file is named Falcon.txt
[ACCOUNT]
character-username = Falcon
character-password = falconpassword
[INFO]
character-coordx = 3252
character-coordy = 3432
character-active = yes
character-ismember = 1
character-messages = 5
character-lastconnection = [removed]
character-lastlogin = 2009-11-29
character-energy = 100
character-gametime = 193
character-gamecount = 183
[EQUIPMENT]
character-equip = 0 4724 0
character-equip = 1 1052 0
character-equip = 2 6585 0
character-equip = 3 4151 0
character-equip = 4 4720 0
character-equip = 5 1215 0
character-equip = 6 -1 0
character-equip = 7 4722 0
character-equip = 8 -1 0
character-equip = 9 775 0
character-equip = 10 1837 0
character-equip = 11 -1 0
character-equip = 12 6735 0
character-equip = 13 -1 0
[APPEARANCE]
character-look = 0 1
character-look = 1 1
character-look = 2 2
character-look = 3 3
character-look = 4 5
character-look = 5 2
[STATS]
character-skill = 0 1 0
character-skill = 1 1 0
character-skill = 2 1 0
character-skill = 3 1 0
character-skill = 4 1 0
character-skill = 5 1 0
character-skill = 6 1 0
character-skill = 7 1 0
character-skill = 8 1 0
character-skill = 9 1 0
character-skill = 10 1 0
character-skill = 11 1 0
character-skill = 12 1 0
character-skill = 13 1 0
character-skill = 14 1 0
character-skill = 15 1 0
character-skill = 16 1 0
character-skill = 17 1 0
character-skill = 18 1 0
character-skill = 19 1 0
character-skill = 20 1 0
[ITEMS]
[BANK]
[FRIENDS]
[IGNORES]
[END]
There is a huge database of these and search through the directory in the files for values.
Values I mean by item ID's or IP addresses to find and track other accounts.
However I have a new problem and my development for this is crashing.
As you can see in the file the lines are organized by tabs.
character-equip = 0 4724 1
If I put the value 4724 in my search application, I want it to print out the value 1 tab to the right of the found search result. I want it to print out the value for the found results only, not extra results.
So the search could look like this:
1 "Enter item to find:"
2 "Enter item to find: 4724"
3 "Account Falcon.txt has 1!"
press any key to continue...
Or if there was more quantity of that equipped item
character-equip = 5 1239 102
1. "Enter item to find:"
2. "Enter item to find: 1239"
3. "Account Falcon2.txt has 102!"
press any key to continue...
I simply want to input an item ID, and have it display the value after the found value. The white space is a tab. I have tried doing this and the only successful way of getting any result is to put a tab in between the search term. So if I want to find item 1239 id type this in the cmd line:
Enter item to find:<tab>1239<tab>
It would then search for that and will display the accounts with that item in it. However I still have to individually open up the accounts to find out the quantity of that item. I want the search results to display the quantity of the item if the value is found. However if the value is a quantity and it trys to search one tab over, I want it to either skip it or say zero.
This is what I mean.
character-equip = 0 1024 1239
Enter item to find: 1239
If it hits this account I want to make the search results display a zero if it cannot tab over or view any values in the white space. So it will display as null or zero
Account Falcon3.txt has null!
or
Account Falcon3.txt has 0!
I've attempted to do this but I am unsure how to achieve this.
Here is my code.
import java.io.*;
import java.util.*;
public class ItemDatabase {
public static void main(String args[]) {
System.out.print("Enter item to find: ");
Scanner sc = new Scanner(System.in);
find(sc.nextLine());
}
public static void find(String delim) {
File dir = new File("accounts");
if (dir.exists()) {
String read;
try {
File files[] = dir.listFiles();
for (int i = 0; i < files.length; i++) {
File loaded = files[i];
if (loaded.getName().endsWith(".txt")) {
BufferedReader in = new BufferedReader(new FileReader(loaded));
StringBuffer load = new StringBuffer();
while ((read = in.readLine()) != null) {
load.append(read + "\n");
}
String delimiter[] = new String(load).split(delim);
if(delimiter.length > 1) {
System.out.println("Account " + loaded.getName() + "has " + delimiter[1] + "!");
}
}
}
} catch (Exception e) {
e.printStackTrace();
}
} else {
System.out.println("error: dir wasn't found!");
}
}
}
Thanks guys I hope you can help me.

This is simply crying out for a database. If your entire back end is running in a single java process I'd recommend going with something like Apache Derby or H2.
If you'd rather not move to a database, and the main problem is the act of listing all the entries in a single directory, could you split it into a heirarchy. Your Falcon account could then be located in F/FA/Falcon.txt - each directory would then contain a more manageable number of files.

Aside from the need for a database, you could probably implement your solution more intuitively and easily using commandline utilities such as find, grep, etc. or a text-processing language such as perl.
Maybe
grep '[0-9]+\t[0-9]+\t1239' Account_Falcon3.txt
Would return
character-equip = 0 1024 1239
You could then easily see that the value is 0 for that item.
I cannot emphasize enough the need to not write a Java program to do this - you won't do as good a job as the authors of the standard shell utilities. Let's face it, the fact that you are asking this question indicates that you are a newb! :) (We are all newbs depending on the topic).