We Keep Coding

Migration from dcm4che2 to dcm4che3

I have used below mentioned API of dcm4che2 from this repository http://www.dcm4che.org/maven2/dcm4che/ in my java project.
dcm4che-core-2.0.29.jar
org.dcm4che2.data.DicomObject
org.dcm4che2.io.StopTagInputHandler
org.dcm4che2.data.BasicDicomObject
org.dcm4che2.data.UIDDictionary
org.dcm4che2.data.DicomElement
org.dcm4che2.data.SimpleDcmElement
org.dcm4che2.net.service.StorageCommitmentService
org.dcm4che2.util.CloseUtils
dcm4che-net-2.0.29.jar
org.dcm4che2.net.CommandUtils
org.dcm4che2.net.ConfigurationException
org.dcm4che2.net.NetworkApplicationEntity
org.dcm4che2.net.NetworkConnection
org.dcm4che2.net.NewThreadExecutor
org.dcm4che3.net.service.StorageService
org.dcm4che3.net.service.VerificationService
Currently i want to migrate to dcm4che3 but, above listed API is not found in dcm4che3 which i have downloaded from this repository http://sourceforge.net/projects/dcm4che/files/dcm4che3/
Could you please guide me for alternate approach?

As you have already observed, the BasicDicomObject is history -- alongside quite a few others.
The new "Dicom object" is Attributes -- an object is a collection of attributes.
Therefore, you create Attributes, populate them with the tags you need for RQ-behaviour (C-FIND, etc) and what you get in return is another Attributes object from which you pull the tags you want.
In my opinion, dcm4che 2.x was vague on the subject of dealing with individual value representations. dcm4che 3.x is quite a bit clearer.
The migration demands a rewrite of your code regarding how you query and how you treat individual tags. On the other hand, dcm4che 3.x makes the new code less convoluted.
On request, I have added the initial setup of a connection to some service class provider (SCP):
// Based on org.dcm4che:dcm4che-core:5.25.0 and org.dcm4che:dcm4che-net:5.25.0
import org.dcm4che3.data.*;
import org.dcm4che3.net.*;
import org.dcm4che3.net.pdu.AAssociateRQ;
import org.dcm4che3.net.pdu.PresentationContext;
import org.dcm4che3.net.pdu.RoleSelection;
import org.dcm4che3.net.pdu.UserIdentityRQ;
// Client side representation of the connection. As a client, I will
// not be listening for incoming traffic (but I could choose to do so
// if I need to transfer data via MOVE)
Connection local = new Connection();
local.setHostname("client.on.network.com");
local.setPort(Connection.NOT_LISTENING);
// Remote side representation of the connection
Connection remote = new Connection();
remote.setHostname("pacs.on.network.com");
remote.setPort(4100);
remote.setTlsProtocols(local.getTlsProtocols());
remote.setTlsCipherSuites(local.getTlsCipherSuites());
// Calling application entity
ApplicationEntity ae = new ApplicationEntity("MeAsAServiceClassUser".toUpperCase());
ae.setAETitle("MeAsAServiceClassUser");
ae.addConnection(local); // on which we may not be listening
ae.setAssociationInitiator(true);
ae.setAssociationAcceptor(false);
// Device
Device device = new Device("MeAsAServiceClassUser".toLowerCase());
device.addConnection(local);
device.addApplicationEntity(ae);
// Configure association
AAssociateRQ rq = new AAssociateRQ();
rq.setCallingAET("MeAsAServiceClassUser");
rq.setCalledAET("NameThatIdentifiesTheProvider"); // e.g. "GEPACS"
rq.setImplVersionName("MY-SCU-1.0"); // Max 16 chars
// Credentials (if appropriate)
String username = "username";
String passcode = "so secret";
if (null != username && username.length() > 0 && null != passcode && passcode.length() > 0) {
rq.setUserIdentityRQ(UserIdentityRQ.usernamePasscode(username, passcode.toCharArray(), true));
}
Example, pinging the PACS (using the setup above):
String[] TRANSFER_SYNTAX_CHAIN = {
UID.ExplicitVRLittleEndian,
UID.ImplicitVRLittleEndian
};
// Define transfer capabilities for verification SOP class
ae.addTransferCapability(
new TransferCapability(null,
/* SOP Class */ UID.Verification,
/* Role */ TransferCapability.Role.SCU,
/* Transfer syntax */ TRANSFER_SYNTAX_CHAIN)
);
// Setup presentation context
rq.addPresentationContext(
new PresentationContext(
rq.getNumberOfPresentationContexts() * 2 + 1,
/* abstract syntax */ UID.Verification,
/* transfer syntax */ TRANSFER_SYNTAX_CHAIN
)
);
rq.addRoleSelection(new RoleSelection(UID.Verification, /* is SCU? */ true, /* is SCP? */ false));
try {
// 1) Open a connection to the SCP
Association association = ae.connect(local, remote, rq);
// 2) PING!
DimseRSP rsp = association.cecho();
rsp.next(); // Consume reply, which may fail
// Still here? Success!
// 3) Close the connection to the SCP
if (as.isReadyForDataTransfer()) {
as.waitForOutstandingRSP();
as.release();
}
} catch (Throwable ignore) {
// Failure
}
Another example, retrieving studies from a PACS given accession numbers; setting up the query and handling the result:
String modality = null; // e.g. "OT"
String accessionNumber = "1234567890";
//--------------------------------------------------------
// HERE follows setup of a query, using an Attributes object
//--------------------------------------------------------
Attributes query = new Attributes();
// Indicate character set
{
int tag = Tag.SpecificCharacterSet;
VR vr = ElementDictionary.vrOf(tag, query.getPrivateCreator(tag));
query.setString(tag, vr, "ISO_IR 100");
}
// Study level query
{
int tag = Tag.QueryRetrieveLevel;
VR vr = ElementDictionary.vrOf(tag, query.getPrivateCreator(tag));
query.setString(tag, vr, "STUDY");
}
// Accession number
{
int tag = Tag.AccessionNumber;
VR vr = ElementDictionary.vrOf(tag, query.getPrivateCreator(tag));
query.setString(tag, vr, accessionNumber);
}
// Optionally filter on modality in study if 'modality' is provided,
// otherwise retrieve modality
{
int tag = Tag.ModalitiesInStudy;
VR vr = ElementDictionary.vrOf(tag, query.getPrivateCreator(tag));
if (null != modality && modality.length() > 0) {
query.setString(tag, vr, modality);
} else {
query.setNull(tag, vr);
}
}
// We are interested in study instance UID
{
int tag = Tag.StudyInstanceUID;
VR vr = ElementDictionary.vrOf(tag, query.getPrivateCreator(tag));
query.setNull(tag, vr);
}
// Do the actual query, needing an AppliationEntity (ae),
// a local (local) and remote (remote) Connection, and
// an AAssociateRQ (rq) set up earlier.
try {
// 1) Open a connection to the SCP
Association as = ae.connect(local, remote, rq);
// 2) Query
int priority = 0x0002; // low for the sake of demo :)
as.cfind(UID.StudyRootQueryRetrieveInformationModelFind, priority, query, null,
new DimseRSPHandler(as.nextMessageID()) {
#Override
public void onDimseRSP(Association assoc, Attributes cmd,
Attributes response) {
super.onDimseRSP(assoc, cmd, response);
int status = cmd.getInt(Tag.Status, -1);
if (Status.isPending(status)) {
//--------------------------------------------------------
// HERE follows handling of the response, which
// is just another Attributes object
//--------------------------------------------------------
String studyInstanceUID = response.getString(Tag.StudyInstanceUID);
// etc...
}
}
});
// 3) Close the connection to the SCP
if (as.isReadyForDataTransfer()) {
as.waitForOutstandingRSP();
as.release();
}
}
catch (Exception e) {
// Failure
}
More on this at https://github.com/FrodeRanders/dicom-tools

Why does this env object keep growing in size ?

I have been working on a web crawler for some time now, the idea is simple, I have a SQL table containing a list of websites, I have many threads fetching the first website from the table and deleting it, then crawling it ( in a heap like manner).
Code is a bit too long so I'm gonna try and delete some parts of it :
while(true){
if(!stopped){
System.gc();
Statement stmt;
String scanned = "scanned";
if (!scan)scanned = "crawled";
Connection connection = null;
try {
connection = Utils.getConnection();
} catch (Exception e1) {
connection.close();
e1.printStackTrace();
}
String name;
stmt = connection.createStatement();
ResultSet rs = null;
boolean next;
do {
rs = stmt.executeQuery("select url from websites where "+scanned+" = -1");
next = rs.next();
} while (next && Utils.inBlackList(rs.getString(1)));
if(next){
name = rs.getString(1);
stmt.executeUpdate("UPDATE websites SET "+scanned+" = 1 where url = '"+Utils.stripDomainName(name)+"'");
String backup_name = name;
name = Utils.checkUrl(name);
System.out.println(scanned + " of the website : " + name +" just started by the Thread : " + num);
// And here is the important part, I think
CrawlConfig config = new CrawlConfig();
String ts = Utils.getTime();
SecureRandom random = new SecureRandom();
String SessionId = new BigInteger(130, random).toString(32);
String crawlStorageFolder = "tmp/temp_storageadmin"+SessionId;
config.setCrawlStorageFolder(crawlStorageFolder);
config.setPolitenessDelay(Main.POLITENESS_DELAY);
config.setMaxDepthOfCrawling(Main.MAX_DEPTH_OF_CRAWLING);
config.setMaxPagesToFetch(Main.MAX_PAGES_TO_FETCH);
config.setResumableCrawling(Main.RESUMABLE_CRAWLING);
int numberOfCrawlers = Main.NUMBER_OF_CRAWLERS;
PageFetcher pageFetcher = new PageFetcher(config);
RobotstxtConfig robotstxtConfig = new RobotstxtConfig();
RobotstxtServer robotstxtServer = new RobotstxtServer(robotstxtConfig, pageFetcher);
try {
controller = new CrawlerController(config, pageFetcher, robotstxtServer);
controller.addSeed(name);
controller.setSeeed(name);
controller.setTimestamp(ts);
controller.setSessiiid("admin"+num+scan);
//Main.crawls.addCrawl("admin"+num+scan, new Crawl(name,"admin"+num+scan,ts));
stmt.executeUpdate("DELETE FROM tempCrawl WHERE SessionID = '"+"admin"+num+scan+"'");
if (!scan){
// Main.crawls.getCrawl("admin"+num+scan).setCrawl(true);
stmt.executeUpdate("INSERT INTO tempCrawl (SessionID, url, ts, done, crawledpages, misspelled, english, proper, scan, crawl )"
+ " VALUES ( '"+"admin"+num+scan+"', '"+name+"', '"+ts+"', false, 0, 0, true, false, "+false+" , "+true+" )");
}else{
//Main.crawls.getCrawl("admin"+num+scan).setScan(true);
stmt.executeUpdate("INSERT INTO tempCrawl (SessionID, url, ts, done, crawledpages, misspelled, english, proper, scan, crawl )"
+ " VALUES ( '"+"admin"+num+scan+"', '"+name+"', '"+ts+"', false, 0, 0, true, false, "+true+" , "+false+" )");
}
connection.close();
controller.start_auto(Crawler.class, numberOfCrawlers, false, scan,num);
} catch(Exception e){
rs.close();
connection.close();
e.printStackTrace();
}
}else{
rs.close();
connection.close();
}
//CrawlerController.start_auto(scan, num);
if (stopping){
stopped = true;
stopping = false;
}
}}
} catch (Exception e) {
e.printStackTrace();
}
As you can see, each time I'm creating a crawlerController, and crawling a website and so on.
The problem here is that jvm memory heap keeps increasing in size considerably. After profiling the application using yourKit Java profiler I located the memory leak in the following lines of code :
yourKit profiling screenshot
Now this is the exact line from where the memory leak starts, this env variable seems to take up too much space, and keeps increasing after each operation, whereas the operations are independent.
Environment env = new Environment(envHome, envConfig);
I don't really know what this variable does, and how I could fix it, one more thing, I did alter the crawlController source code, I thought that might be relevant.

Assuming that you are using crawler4j as crawling-framework.
Everytime you create a crawl controller you instantiate a new frontier, which is shared between the crawler threads to manage the queue of URLs to crawl. Moreover, a so called 'docIdServer' is created, which has the responsiblity to manage if an incoming URL (e.g. website) has already been processed in this crawl.
This frontier and the docIdServer are based on an in-memory database, in which the environment is responsible for caching, locking, logging and transaction. For that reason, this variable will grow over time.
If you set resumable crawling to true, the database will operate in file-mode and there it will grow slower.

Microsoft Excel as SQL Database

I have followed the instructions provided in this website from content 5.3 and the code works fine.
My plan is to make a jar file (containing an interface/GUI), distribute that jar file to users, and then have them all read/write data should be from one excel file. When I place the excel file in local drive it works, but when I place the file in network folder/server, the java creates a problem:
java.exe has encountered a problem and needs to close. We are sorry
for the inconvenience.
or
Java Result: -1073741811
Any suggestions? Thank you
public class TestIntoExcel
{
public String s;
public double number;
public Date d;
public void display()throws ClassNotFoundException, SQLException
{
Class.forName("oracle.jdbc.driver.OracleDriver");
Connection writeConnection = DriverManager.getConnection
("jdbc:odbc:usersavedataODBC");
writeConnection.setReadOnly(false);
Statement writeStatement = writeConnection.createStatement();
writeStatement.executeUpdate("CREATE TABLE TEST_INSERT(COL1 INT,COL2 VARCHAR(10),COL3 DATE)");
PreparedStatement writeStatement2 =
writeConnection.prepareStatement("INSERT INTO TEST_INSERT(COL1,COL2,COL3)VALUES(?,?,?)");
for(int i = 0; i<3;i++)
{
writeStatement2.setDouble(1, i);
writeStatement2.setString(2, "Row" + i);
writeStatement2.setDate(3, new java.sql.Date(new Date().getTime()));
writeStatement2.execute();
}
String query = "select *from[TEST_INSERT]";
ResultSet rset = writeStatement.executeQuery(query);
//System.out.println(rset);
while(rset.next())
{
number = rset.getDouble("COL1");
s = rset.getString("COL2");
d = rset.getDate("COL3");
System.out.println(number+"\n"+s+"\n"+d);
}
writeStatement.close();
writeStatement2.close();
writeConnection.close();

Read a image from a longblob MySQL very fast

As the title says, I wanna retrive images (stored in longblob) very fast when the jslider is moved. I have 360 cases, and it is work with no error, but the problem consist in the lag/delay of each case/image when the jslider is moved. I tested this idea with images retrived from local machine and it works very fast/clean. I know that the problem can be from the internet connection, but trust me I have at least 3-4 MB/s download/upload.
Few extra notes:
table engine: MyISAM
column: longblob with every image ~170-200 kb - .png
//calling jslider and setup
jslider1 = new javax.swing.JSlider(); //my jslider
jslider1.setMajorTickSpacing(10);
jslider1.setMaximum(360);
jslider1.setMinorTickSpacing(5);
jslider1.setOrientation(javax.swing.JSlider.VERTICAL);
//and my change event
private void jslider1StateChanged(javax.swing.event.ChangeEvent evt) {
int x = jslider1.getValue();
switch (x) {
case 1:
try {
String sql = "select imga from test where deg ='" + x + "'";
pst = conn.prepareStatement(sql);
rs = pst.executeQuery();
if (rs.next()) {
byte[] imagedata = rs.getBytes("imga");
format = new ImageIcon(imagedata);
jLabel1.setIcon(format); //where I put my image
}
} catch (Exception e) {
e.printStackTrace();
}
break;
case 2:
//[...] //I have 360 cases
}
} // close state changed

Why do you want to save the images in a longblob anyway? Can't you store them within the Jar itself. If you still want to retrieve from Database longblob consider using Caching to load your longblobs from the database and cache it in the local system using a Initializing Asynchronous Thread when your Swing application starts up.

reading huge data from database and writing into xml Java

I have huge data billions of records in tables what is the best way to read it in plain Java and write it in XML file?
Thanks

If by best you mean fastest - I would consider using native database tools to dump the files as this will be way faster than using JDBC.

Java (+Hibernate?) will slow the process down unnecessarily. Easier to do sqlplus script and spool formatted fields into your xml file.

On Toad you can right click a table and click export to xml. on the commercial version I think you can export all tables but I'm not sure

Another possibility (working with all db with a JDBC driver) would be to use Apache Cocoon. There are actually two ways: XSP ((alone or and with ESQL). Both technos are really quick to develop.
XSP alone example. Think of XSP as a little bit like JSP but generating XML instead of HTML. From a DB for instance.
<?xml version="1.0"?>
<xsp:page language="java" xmlns:xsp="http://apache.org/xsp"
xmlns:esql="http://apache.org/cocoon/SQL/v2"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://apache.org/cocoon/SQL/v2 xsd/esql.xsd"
space="strip">
<xsp:structure>
<xsp:include>java.sql.Connection</xsp:include>
<xsp:include>java.sql.DriverManager</xsp:include>
<xsp:include>java.sql.PreparedStatement</xsp:include>
<xsp:include>java.sql.SQLException</xsp:include>
<xsp:include>java.sql.ResultSet</xsp:include>
</xsp:structure>
<xsp:logic><![CDATA[
private static final String connectionString =
"jdbc:mysql://localhost/mandarin?user=mandarin&password=mandarin" ;
private Connection conn = null ;
private PreparedStatement pstmt = null ;
private void openDatabase() {
try {
DriverManager.registerDriver(new com.mysql.jdbc.Driver());
conn = DriverManager.getConnection (connectionString);
pstmt = conn.prepareStatement(
"select " +
" count(*) as cardinality " +
" from " +
" unihan50 u " +
" where " +
" unicode_id >= ? and " +
" unicode_id <= ? " ) ;
} catch (SQLException e) {
e.printStackTrace();
}
}
private int getRangeCardinality ( int lowerBound, int upperBound ) {
int cnt = 0 ;
try {
cnt = 2 ;
pstmt.setInt ( 1, lowerBound ) ;
pstmt.setInt ( 2, upperBound ) ;
boolean sts = pstmt.execute () ;
if ( sts ) {
ResultSet rs = pstmt.getResultSet();
if (rs != null && rs.next() ) {
cnt = rs.getInt ( "cardinality" ) ;
}
}
} catch (SQLException e) {
e.printStackTrace();
}
return cnt ;
}
private void closeDatabase() {
try {
pstmt.close () ;
} catch (SQLException e) {
e.printStackTrace();
}
try {
conn.close () ;
} catch (SQLException e) {
e.printStackTrace();
}
}
]]>
</xsp:logic>
<ranges>
<xsp:logic><![CDATA[
openDatabase() ;
for ( int i = 0; i < 16 ; i++ ) {
int from = i * 0x1000 ;
int to = i * 0x1000 + 0x0fff ;
]]>
<range>
<from>0x<xsp:expr>Integer.toString(from, 16)</xsp:expr></from>
<to>0x<xsp:expr>Integer.toString(to, 16)</xsp:expr></to>
<count><xsp:expr>getRangeCardinality ( from, to )</xsp:expr></count>
</range>
}
closeDatabase () ;
</xsp:logic>
</ranges>
</xsp:page>
XSP is even more straightforward coupled with ESQL. Here is sample
<?xml version="1.0" encoding="UTF-8"?>
<xsp:page language="java" xmlns:xsp="http://apache.org/xsp"
xmlns:esql="http://apache.org/cocoon/SQL/v2"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:xsp-request="http://apache.org/xsp/request/2.0"
xsi:schemaLocation="http://apache.org/cocoon/SQL/v2 xsd/esql.xsd"
space="strip">
<keys>
<esql:connection>
<esql:pool>mandarinMySQL</esql:pool>
<esql:execute-query>
<esql:query><![CDATA[
select
unicode_id,
kMandarin,
...
from
unihan50_unified
where
add_strokes = 0
order by
radical
]]>
</esql:query>
<esql:results>
<esql:row-results><key><esql:get-columns /></key></esql:row-results>
</esql:results>
</esql:execute-query>
</esql:connection>
</keys>
</xsp:page>

I'll be using database inbuild procedure (e.g. XML path) to get data already converted in xml format.
Now there are 2 ways to write in the file:
1. If you have to have Java interface (JDBC) to retrieve data (due to business req) then I'll simply read this data and write in a File (No XML Parser involvement unless you need to verify the output).
2. If you do not have Java restriction then I'll simply write a Stored Procedure which will dump XML data in a file.
Update to comment:
Workflow for fastest retrieval:
Create Stored Procedure which will retrieve data and dump into a file.
Call this SP through Java (as you said you need it)
Either SP can return you the file name or you can create SP which will take file name so you can dynamically manage the output location.
I have not used Oracle for a very long time but I hope this link can help you to kickstart.

If the DB is Oracle, then you can simply use JDBC with a SQLX query. This will generate your result set directly as XML fragments on the server much faster than if you'd do it on your own on the client side. SQLX has been available since 8.1.7 as project Aurora and since 9i in standard as XMLDB.
Here is a simple example.
select XMLelement ("Process",
XMLelement( "number", p.p_ka_id, '.', p_id ),
XMLElement( "name", p.p_name ),
XMLElement ( "processGroup", pg.pg_name ) )
from
PMP_processes p,
PMP_process_groups pg
where
condition ;
In addition to XMLelement, SQLX has XMLattribute, XMLforest, XMLaggregate... which allows you any resulting tree.

Use StAX to write the xml, not DOM.

You can query to the database and retrieve all data into a RESULTSET and use the following code to start off a root element.
DocumentBuilderFactory documentBuilderFactory = DocumentBuilderFactory.newInstance();
DocumentBuilder documentBuilder = documentBuilderFactory.newDocumentBuilder();
Document document = documentBuilder.newDocument();
Element Element_root = document.createElement("rootElement");
Thereafter you can add on as many as child elements using
Element Element_childnode = document.createElement("childnode");//create child node
Element_childnode.appendChild(document.createTextNode("Enter the value of text here"));//add data to child node
Element_root.appendChild(Element_childnode);//close the child node
Do not forget to close the opened node close the root at the end WITHOUT FAIL
Use this to close root.
document.appendChild(Element_causelist);
At the end if you have a XSD validate it your xml against it.....googling the validation online will provide good results.... http://tools.decisionsoft.com/schemaValidate/
NOTE : TIME !!! It will take time when data is huge nos...
But I think this is one and the most easiest way of doing it....Taking in consideration the data, I think one should run the program during down time when there is less traffic....
Hope this helps....Good Luck Gauls....

public class someclassname{
public static String somemethodname(){
String sql;
sql="SELECT * from yourdatabase.yourtable ";
return sql;
}
public static String anothermethodname(){
/*this is another method which is used to excute another query simultaneously*/
String sql;
sql="SELECT * from youdatabase.yourtable2";
return sql;
}
private static void saveasxml(String sql,String targetFile) throws SQLException, XMLStreamException, IOException{
int i,count;
FileOutputStream fos;
try{
Class.forName("com.mysql.jdbc.Driver");
Connection con=DriverManager.getConnection("jdbc:mysql://yourdomain:yourport/yourdatabase","username","password");
Statement stmt=con.createStatement();
ResultSet rs=stmt.executeQuery(sql);
ResultSetMetaData rsmd=rs.getMetaData();
count=rsmd.getColumnCount();
XMLOutputFactory outputFactory = XMLOutputFactory.newFactory();
fos=new FileOutputStream(targetFile);
XMLStreamWriter writer = outputFactory.createXMLStreamWriter(fos);
writer.writeStartDocument();
writer.writeCharacters("\n");
writer.writeStartElement("maintag line");
writer.writeCharacters("\n");
while(rs.next()){
writer.writeCharacters("\t");
writer.writeStartElement("foreveyrow-tagline");
writer.writeCharacters("\n\t");
for(i=1;i<count+1;i++){
writer.writeCharacters("\t");
writer.writeStartElement("Field"+i);
writer.writeCharacters(rs.getString(i));
writer.writeEndElement();
writer.writeCharacters("\n\t");
}
writer.writeEndElement();
writer.writeCharacters("\n");
}
writer.writeEndElement();
writer.writeEndDocument();
writer.close();
}catch(ClassNotFoundException | SQLException e){
}
}
public static void main(String args[]) throws Exception{
saveasxml(somemethodname(), " file location-path");
saveasxml(anothermethodname(), "file location path");
}
}

Thanks all for replying , so far i have managed to get a solution based on using threads and use multiple selects instead of one single complex sql joins (i hate SQL complex ones) life should be simple :) so i didn't waste too much time writing them i am using new threads for each select statements.
any better solution in POJO probabaly using spring is also fine
Thanks
gauls