Here is the problem:
for your reference:
database entries 1,2 and 3 are made using jython 2.2.1 using jdbc1.2.
database entry 4 is made using vb the old to be replace program using odbc.
We have found that if I copy and paste both jython and vb MailBody entries to wordpad directly from that SQL Server Enterprise Manager software it outputs the format perfectly with correct line returns. if I compare the bytes of each file with a hex editor or KDiff3 they are binary identically the same.
There is a 3rd party program which consumes this data. Sadly that 3rd party program reads the data and for entries 1 to 3 it displays the data without line returns. though for entry 4 it correctly formats the text. As futher proof we can see in the picture, the data in the database is displayed differently.
Somehow the line returns are preserved in the database for the vb entries but the jython entries they are overlooked. if I click on the 'MailBody' field of entry 4 i can press down i can see the rest of the email. Whereas the data for jython is displayed in one row.
What gives, what am i missing, and how do I handle this?
Here is a snippet of the code where I actually send it to the database.
EDIT: FYI: please disregard the discrepancies in the 'Processed' column, it is irrelevant.
EDIT: what i want to do is make the jython program input the data in the same way as the vb program. So that the 3rd party program will come along and correctly display the data.
so what it will look like is every entry in 'MailBody' will display "This is a testing only!" then next line "etc etc" so if I was to do a screendump all entries would resemble database entry 4.
SOLVED
add _force_CRLF to the mix:
def _force_CRLF(self, data):
'''Make sure data uses CRLF for line termination.
Nicked the regex from smtplib.quotedata. '''
print data
newdata = re.sub(r'(?:\r\n|\n|\r(?!\n))', "\r\n", data)
print newdata
return newdata
def _execute_insert(self):
try:
self._stmt=self._con.prepareStatement(\
"INSERT INTO EmailHdr (EntryID, MailSubject, MailFrom, MailTo, MailReceive, MailSent, AttachNo, MailBody)\
VALUES (?, ?, ?, ?, ?, ?, ?, cast(? as varchar (" + str(BODY_FIELD_DATABASE) + ")))")
self._stmt.setString(1,self._emailEntryId)
self._stmt.setString(2,self._subject)
self._stmt.setString(3,self._fromWho)
self._stmt.setString(4,self._toWho)
self._stmt.setString(5,self._format_date(self._emailRecv))
self._stmt.setString(6,self._format_date(self._emailSent))
self._stmt.setString(7,str(self._attachmentCount))
self._stmt.setString(8,self._force_CRLF(self._format_email_body()))
self._stmt.execute()
self._prepare_inserting_attachment_data()
self._insert_attachment_data()
except:
raise
def _format_email_body(self):
if not self._emailBody:
return "could not extract email body"
if len(self._emailBody) > BODY_TRUNCATE_LENGTH:
return self._clean_body(self._emailBody[:BODY_TRUNCATE_LENGTH])
else:
return self._clean_body(self._emailBody)
def _clean_body(self,dirty):
'''this method simply deletes any occurrence of an '=20' that plagues my output after much testing this is not related to the line return issue, even if i comment it out I still have the problem.'''
dirty=str(dirty)
dirty=dirty.replace(r"=20","")
return r"%s"%dirty
I suggest to add a debug output to your program, dumping character codes before insertion in DB. There are chances that Jython replace CrLf pair with single character and doesn't restore it when written to DB.
You should look at the quopri module (and others regarding email) so you don't have to use dirty tricks as _clean_body
Related
Runnning sql query to export the contents to CSV file i notice that certain columns do not get displayed properly.in my cast date timestamp is not properly displayed in cell
Code as below :
COPY (select hostname as "Host Name",devicetype as "Device Type",platform as "Model",Ipaddress as "IP Address",swversion as "Software Version",configuredTime as "Configured",activeTime as "Active",cluster as "Clusters",location as "Location",macaddress as "Mac Address",devicepool as "Device Pool" from (select getmodelinfo.ipaddress,max(getmodelinfo.hostname) as
hostname, max(getmodelinfo.macaddress) as macaddress,
max(getmodelinfo.devicetype) as devicetype,
max(getmodelinfo.platform) as platform,
max(getmodelinfo.swversion) as swversion, min(getmodelinfo.day_end_date) as configuredtime,
max(getmodelinfo.active_end_date) as activetime,
max(getmodelinfo.ucmclustername)
as cluster, max(getmodelinfo.ucmlocation) as location,
max(getmodelinfo.ucmdevicepool) as devicepool from (select
pcwh_inv.uniquedeviceid,pcwh_inv.ipaddress,
pcwh_inv.endpointmodel, pcwh_inv.hostname, pcwh_inv.devicetype, pcwh_inv.platform,
pcwh_inv.macaddress, pcwh_inv.version as
swversion,pcwh_inv.deployed_day_end_date as day_end_date,
pcwh_inv.lastupdated_day_end_date as active_end_date,
pcwh_inv.ucmclustername,pcwh_inv.ucmlocation,pcwh_inv.ucmdevicepool
from pcwh_inventory_20160410 pcwh_inv,
(select pcwh_inv.uniquedeviceid, max(pcwh_inv.lastupdated_day_end_date) as
lasttime from pcwh_inventory_20160410 pcwh_inv where pcwh_inv.ipaddress
notnull group by
pcwh_inv.uniquedeviceid)gettime where pcwh_inv.endpointmodel = 'SX' and
pcwh_inv.uniquedeviceid = gettime.uniquedeviceid and
pcwh_inv.lastupdated_day_end_date = gettime.lasttime and pcwh_inv.mgmtstatus not in ('Deleted')
)getmodelinfo group by ipaddress) M) TO '/opt/emms/emsam/export/raj2.csv' DELIMITER ',' CSV HEADER
Please find below screenshot of the result I get on running query without exporting to csv , on export to csv (image below) , image of cell content when i click on particular row content it shows correctly though(bottom most image)
It seems to me an issue with formatting csv file could you please let me know how I could do this from query level to display exact contents as in db column?
I believe nothing is wrong with your code, but that Excel is treating that column as something other than a date, this happens often. It can be resolved by selecting all of the affected cells, right clicking, and changing the format to text through the subsequent menus.
If this CSV is going to be manipulated further by another application, I don't think you will have an issue, as the data is seemingly fine.
If for whatever reason you need it to be correctly displayed in Excel dynamically, you may need to look into something like this.. Java - Excel Cell Type
The problem is straight forward there is no error with CSV format.Actually the data entries done by you are in the wrong way u can't have a space in a data entry that will cause a problem.
So in the 6th and 7th column u can see there is a space between the data entries so due to which problem is taking place.
To prevent that just add a new column for date.
It will solve all the problems and the data will be entered in your database.
I am trying to execute a whole directory of .SQL files in Java.
I'm struggling to execute a stored procedure. I have found this so far (the most helpful) including a dead link, sadly. I have downloaded liquibase also, but I cannot figure out how I should use it for this purpose.
In my current code, I split the files including procedures into different statements:
(Statements split in a Vector[String] and executed in a for loop)
Example:
//File f;
//Statement st;
Vector<String> vProcedure = getProcedureStatements(f, Charset.defaultCharset(), "//");
for (Iterator<String> itr = vProcedure.iterator(); itr.hasNext();)
st.execute(itr.next());
System.out.println(f.getName() + " - done executing.");
The Vector contains the four elements (see SQL-Code #SPLIT x).
DROP PROCEDURE IF EXISTS `Add_Position`; #SPLIT 1
DELIMITER // #SPLIT 2
CREATE PROCEDURE `Add_Position`
(
IN iO_ID INT,
IN iCID INT,
IN iWID INT,
IN iA INT
)
BEGIN
#statements;
END
// #SPLIT 3
DELIMITER ; #SPLIT 4
Result when trying to execute #SPLIT 2:
com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'DELIMITER //' at line 1
Q: Could anyone tell me if there's a exteral library I could use, or how liquibase does work ? I can't get it to work the JDBC-way.
The statement DELIMITER \\ is not actually SQL - it is a command that is interpreted by the mysql command-line tool (and possibly also their GUI tool), but if you pass it straight to the execution engine, you will get this syntax error.
Unfortunately, Liquibase uses ; as the default statement separator, so it is difficult to get this sort of thing to work using SQL files and Liquibase. I have put in a pull request to allow setting the delimiter from the Liquibase command line - see https://github.com/liquibase/liquibase/pull/361.
Can any one suggest how to insert two lines in a single cell? I.e., I need to enter the first line and then the second line should start from new line in the single column cell. Suppose I have a column defined as varchar(100), I need to store the string core java,j2EE will service request as,
core java
j2EE will service request
in a single colomn.
When I retrieve back from the database and displayed on a JSP page it should display in two lines.
I am trying to retrieve Japanese content from database and display using jsp. Is Japanese content (which is in utf-8 format)causing any the problem so that <br /> tag is not parsing as line break.It is coming as string <br /> when I display it on screen.
A newline character ('\n')is still a character, so there's no problem inserting it:
Connection conn = ...;
PreparedStatement ps =
conn.prepareStatement
("INSERT INTO my_table VALUES (?)");
ps.setString (1, "j2EE will service request\nin a single coloumn.");
ps.executeUpdate();
Notes:
Different platforms have different line separators, so depending on how exactly this data is going to be consumed, using System.getProperty("line.separator") may be more appropriate.
For clarity's sake, again, this code omits resource management (e.g., closing the statement) and error-handling code.
I was doing this code, and something weird has happened.
I make a bulk insert from an external file. But the result it's just fragmented, or maybe corrupted.
cnx=factoryInstace.getConnection();
pstmt = cnx.prepareStatement("DELETE FROM TEMPCELULAR");
pstmt.executeUpdate();
pstmt = cnx.prepareStatement("EXECUTE BLOCK AS BEGIN if (exists(select 1 from rdb$relations where rdb$relation_name = 'EXT_TAB')) then execute statement 'DROP TABLE EXT_TAB;'; END");
pstmt.executeUpdate();
pstmt = cnx.prepareStatement("CREATE TABLE EXT_TAB EXTERNAL '"+txtarchivoProcesar.getText()+"'(CELULAR varchar(11))");
pstmt.executeUpdate();
pstmt = cnx.prepareStatement("INSERT INTO TEMPCELULAR (CELULAR)SELECT CELULAR FROM EXT_TAB");
pstmt.executeUpdate();
pstmt = cnx.prepareStatement("SELECT CELULAR FROM TEMPCELULAR");
ResultSet rs=pstmt.executeQuery();
while(rs.next()){
System.out.println("::"+rs.getString(1));
}
And now, all of a sudden the rows on my table look like this:
::c#gmail.com
::abc2#gmail.
::m
abc3#gma
::.com
abc4#
::ail.com
ab
::#gmail.com
::bc6#gmail.c
::abc7#gmai
::com
abc8#g
::il.com
abc
::gmail.com
::c10#gmail.c
::
The blank spaces between results were not made by me. This is the result as it is.
Source file for external table:
abc#gmail.com
abc2#gmail.com
abc3#gmail.com
abc4#gmail.com
abc5#gmail.com
abc6#gmail.com
abc7#gmail.com
abc8#gmail.com
abc9#gmail.com
abc10#gmail.com
sneciosup#hotmail.com
¿What's wrong with my code?
I've haven't seen this wack results in years.
The database is created of the users pc on the first run. Hence, while in production every time I run the program.
Any help, will be appreciated.
The external table file in Firebird is not just a plaintext file, it is a fixed width format with special requirements to the content and layout. See the Interbase 6.0 Data Definition Guide, page 107-111 (available for download from http://www.firebirdsql.org/en/reference-manuals/ ) or The Firebird Book by Helen Borrie page 281-287.
The problems I see right now are:
you declare the column in the external table to be VARCHAR(11),
while the shortest emailaddress in your file is 13 characters, the
longest is 21 characters, so Firebird will never be able to read a
full emailaddress from the file
you don't specify a separate column for your linebreaks, so your
linebreaks will simply be part of the data which is read
you have declared the column as VARCHAR, this requires the records
to have a very specific format, where the first two bytes declare
the actual data length followed by a string of that length (and even then it
only reads upto the declared length of the column). Either make sure
you follow the requirements for VARCHAR columns, or simply use the
CHAR datatype and make sure you pad the column with spaces upto the
declared length.
I am not 100% sure, but your default database characterset may also be involved in how the data is read.
I have .Data file given in the above format . I am writing a program in java that will take the values from the .data file and put it in the buffer. MY java program is connected to Mysql(windows) via JDBC. So I need to read the values from the file given in the above format and put it the buffer like
Insert Into building values ("--", "---",----)
In this way, i store these values and jdbc will populate the database tables on Mysql(windows). Please tell me teh best way.
Check out the answers to this question for reading file lines and splitting them into chunks. I know the question says Groovy: but most answers are Java. Then insert the values you retrieved via JDBC.
Actually, since your data file is obviously CSV, you could also use a CSV libary like OpenCSV to read the values.
The data is in CSV format, so use a CSV library to parse the file and then just add some JDBC code to insert this into database.
Or just call MySQL CSV import command from Java:
try {
// Execute a command with arguments
String command = "mysqlimport [options] db_name textfile1 [textfile2 ...]";
Process child = Runtime.getRuntime().exec(command);
} catch (IOException e) {
}
This is the fourth question for the same task... If your data file is well formatted like in the example you provided, then you don't have to split the line into values:
Source: "AAH196","Austin","TX","Virginia Beach","VA"
Target: INSERT INTO BUILDING VALUES("AAH196","Austin","TX","Virginia Beach","VA");
<=> "INSERT INTO BUILDING VALUES(" + Source + ");"
Just take a complete row from you csv file and concatenate a SQL expression.
(see my answer to question 1 of 4 - BTW, if SQL INJECTION is a potential problem, splitting a line of values is not a solution too)
you can bind your csv with java beans using opencsv.
http://opencsv.sourceforge.net/
you can make these beans persistent using an ORM framework, like Hibernate, Cayenne or with JPA which're based on annotations and map your fields to tables easily without creating any sql statement.
This would be a perfect job for Groovy. Here's a gist with a small skeleton script to build upon.