Runnning sql query to export the contents to CSV file i notice that certain columns do not get displayed properly.in my cast date timestamp is not properly displayed in cell
Code as below :
COPY (select hostname as "Host Name",devicetype as "Device Type",platform as "Model",Ipaddress as "IP Address",swversion as "Software Version",configuredTime as "Configured",activeTime as "Active",cluster as "Clusters",location as "Location",macaddress as "Mac Address",devicepool as "Device Pool" from (select getmodelinfo.ipaddress,max(getmodelinfo.hostname) as
hostname, max(getmodelinfo.macaddress) as macaddress,
max(getmodelinfo.devicetype) as devicetype,
max(getmodelinfo.platform) as platform,
max(getmodelinfo.swversion) as swversion, min(getmodelinfo.day_end_date) as configuredtime,
max(getmodelinfo.active_end_date) as activetime,
max(getmodelinfo.ucmclustername)
as cluster, max(getmodelinfo.ucmlocation) as location,
max(getmodelinfo.ucmdevicepool) as devicepool from (select
pcwh_inv.uniquedeviceid,pcwh_inv.ipaddress,
pcwh_inv.endpointmodel, pcwh_inv.hostname, pcwh_inv.devicetype, pcwh_inv.platform,
pcwh_inv.macaddress, pcwh_inv.version as
swversion,pcwh_inv.deployed_day_end_date as day_end_date,
pcwh_inv.lastupdated_day_end_date as active_end_date,
pcwh_inv.ucmclustername,pcwh_inv.ucmlocation,pcwh_inv.ucmdevicepool
from pcwh_inventory_20160410 pcwh_inv,
(select pcwh_inv.uniquedeviceid, max(pcwh_inv.lastupdated_day_end_date) as
lasttime from pcwh_inventory_20160410 pcwh_inv where pcwh_inv.ipaddress
notnull group by
pcwh_inv.uniquedeviceid)gettime where pcwh_inv.endpointmodel = 'SX' and
pcwh_inv.uniquedeviceid = gettime.uniquedeviceid and
pcwh_inv.lastupdated_day_end_date = gettime.lasttime and pcwh_inv.mgmtstatus not in ('Deleted')
)getmodelinfo group by ipaddress) M) TO '/opt/emms/emsam/export/raj2.csv' DELIMITER ',' CSV HEADER
Please find below screenshot of the result I get on running query without exporting to csv , on export to csv (image below) , image of cell content when i click on particular row content it shows correctly though(bottom most image)
It seems to me an issue with formatting csv file could you please let me know how I could do this from query level to display exact contents as in db column?
I believe nothing is wrong with your code, but that Excel is treating that column as something other than a date, this happens often. It can be resolved by selecting all of the affected cells, right clicking, and changing the format to text through the subsequent menus.
If this CSV is going to be manipulated further by another application, I don't think you will have an issue, as the data is seemingly fine.
If for whatever reason you need it to be correctly displayed in Excel dynamically, you may need to look into something like this.. Java - Excel Cell Type
The problem is straight forward there is no error with CSV format.Actually the data entries done by you are in the wrong way u can't have a space in a data entry that will cause a problem.
So in the 6th and 7th column u can see there is a space between the data entries so due to which problem is taking place.
To prevent that just add a new column for date.
It will solve all the problems and the data will be entered in your database.
Related
I use the following command to import data from a .csv file into a MySQL database table like so:
String loadQuery = "LOAD DATA LOCAL INFILE '" + file + "' INTO TABLE source_data_android_cell FIELDS TERMINATED BY ','" + "ENCLOSED BY '\"'"
+ " LINES TERMINATED BY '\n' " + "IGNORE 1 LINES(.....)" +"SET test_date = STR_TO_DATE(#var1, '%d/%m/%Y %k:%i')";
However, as one of the columns in the sourcefile contains a really screwy data which is: viva Y31L.RastaMod䋢_Version the program refuses to import the data into MySQL and keeps throwing this error:
java.sql.SQLException: Invalid utf8 character string: 'viva
Y31L.RastaMod'
I searched up on this but cant really understand what exactly the error was, other than that the INPUT format of this string "viva Y31L.RastaMod䋢_Version" was wrong and didn't fit the utf8 format used in the MySQL database?
However, I already did the following which is SET NAMES UTF8MB4 in my MySQL db, since it was suggested in other questions that UTF8MB4 was more flexible in accepting weird characters.
I explored this further by manually inserting that weird data into MySQL database table in the Command Prompt, which worked fine. In fact, the table displayed almost the full entry: viva Y31L.RastaMod?ã¢_Version. But if I ran my program from the IDE the file gets rejected.
Would appreciate any explanations.
Second minor question related to the import process of csv file into mySQL:
I noticed that I couldn't import a copy of the same file into the MySQL database. Errors thrown included that the data was a duplicate. Is that because MySQL rejects duplicate column data? But when I changed all the data of one column leaving the rest the same in that copied file, it gets imported correctly. Why is that so?
I don't think this immediate error has to do with the destination of the data not being able to cope with UTF-8 characters, but rather the way you are using LOAD DATA. You can try specifying the character set which should be used when loading the data. Consider the following LOAD DATA command, which is what you had originally but slightly modified:
LOAD DATA LOCAL INFILE path/to/file INTO TABLE source_data_android_cell
CHARACTER SET utf8
FIELDS TERMINATED BY ','
ENCLOSED BY '"'
LINES TERMINATED BY '\n'
IGNORE 1 LINES(.....)
SET test_date = STR_TO_DATE(#var1, '%d/%m/%Y %k:%i')
This being said, you should also make sure that the target table uses a character set which supports the data you are trying to load into it.
Currently I'm trying to import an SQL-INSERT dump from a postgres database into my Derby development/testing database using the Eclipse's Data Tools SQL scratchpad. The export created a lot of data that looked like the following:
CREATE TABLE mytable ( testfield BLOB );
INSERT INTO mytable ( testfield ) VALUES ('\x0123456789ABCDEF');
Executing it in Eclispe's SQL Scratchpad results in (translated from german):
Columns of type 'BLOB' shall not contain values of type 'CHAR'.
The problem seems, that the PostgreSQL admin tool exported BLOB data in a format like '\x0123456789ABCDEF' which is not recognized by Derby (Embedded).
Changing this to X'0123456789ABCDEF' or simply '0123456789ABCDEF'did also not work.
The only thing that worked was CAST (X'123456789ABCDEF' AS BLOB), but I'm not yet sure, if this results in the correct binary data when read back in Java and if the X'0123456789ABCDEF'is 100% portable.
CAST (...whatever... AS BLOB) doesn't work in java DB / Apache DERBY!
One must use the built-in system procedure
SYSCS_UTIL.SYSCS_IMPORT_DATA_LOBS_FROM_EXTFILE. I do not think there is any other way. For instance:
CALL SYSCS_UTIL.SYSCS_IMPORT_DATA_LOBS_FROM_EXTFILE (
'MYSCHEMA', 'MYTABLE',
'MY_KEY, MY_VARCHAR, MY_INT, MY_BLOB_DATA',
'1,3,4,2',
'c:\tmp\import.txt', ',' , '"' , 'UTF-8',
0);
where the referenced "import.txt" file will be CSV like (as specified by the ',' and '"' arguments above) and contain as 2nd field (I scrambled the CSV field order versus DB column names order on purpose to illustrate) a file name that contains the binary data in proper for the BLOB's. For instance, "import.txt" is like:
"A001","c:\tmp\blobimport.dat.0.55/","TEST",123
where the supplied BLOB data file name bears the convention "filepath.offset.length/"
Actually, you can first export your table with
CALL SYSCS_UTIL.SYSCS_EXPORT_TABLE_LOBS_TO_EXTFILE(
'MYSCHEMA', 'MYTABLE', 'c:\tmp\export.txt', ',' ,'"',
'UTF-8', 'c:\tmp\blobexport.dat');
to generate sample files with the syntax to reuse on import.
I have a serious problem. My app retrieves text from an online database, the problem is that if I put a text in the database that contains the apostrophe in the site displays correctly in the application fails me while synchronizing with the database. I also enabled the magic quote gpc but nothing. How can I fix?
First, disable magic_quote_gpc, as the are deprecated/removed
Secondly, look at mysqli_real_escape_string() when sending data to database (which I assumed is your 'synchronizing with the database' is).
You have to escape the apostroph like this:
INSERT INTO my_table (col1, col2) VALUES (123, 'Hello there''s');
use $user_input = mysql_real_esape_string($user_input)
I am working with a 3rd product called JPOS and it has an XMLPackager whereby I get a string from this packager that contains a record in an XML format such as:
<MACHINE><B000>STRING_VALUE</B000><B002>STRING_VALUE</B002><B003>STRING_VALUE</B003><B004>STRING_VALUE</B004><B007>STRING_VALUE</B007><B011>STRING_VALUE</B011><B012>STRING_VALUE</B012><B013>STRING_VALUE</B013><B015>STRING_VALUE</B015><B018>STRING_VALUE</B018><B028>STRING_VALUE</B028><B032>STRING_VALUE</B032><B035>STRING_VALUE</B035><B037>STRING_VALUE</B037><B039>STRING_VALUE</B039><B041>STRING_VALUE</B041><B043>STRING_VALUE</B043><B048>STRING_VALUE</B048><B049>STRING_VALUE</B049><B058>STRING_VALUE</B058><B061>STRING_VALUE</B061><B063>STRING_VALUE</B063><B127>STRING_VALUE</B127></MACHINE>
I have a SQL server table that contains a column for each of the listed. Not that it matters but I could potentially have thru defined with specific STRING_VALUEs. I'm not sure what is the best way to go about this in Java. My understanding is that SQL Server can take an XML string (not document) and do an insert. Is it best to parse each value and then put into a list that populate each value into? This is the first time I've used an XML file and therefore trying to get some help/direction.
Thanks.
Sorry, one of my colleagues was able to help and provide a quick answer. I'll try it from my Java code and it looks like it should work great. Thanks anyway.
Here is the SP that she created whereby I can pass in my XML string and bit value:
CREATE PROCEDURE [dbo].[sbssp_InsertArchivedMessages]
(
#doc varchar(max),
#fromTo bit
)
AS
BEGIN
DECLARE #idoc int, #lastId int
EXEC sp_xml_preparedocument #idoc OUTPUT, #doc
INSERT INTO [dbo].[tblArchivedMessages]
SELECT * FROM OPENXML(#idoc, '/MACHINE', 2) WITH [dbo].[tblArchivedMessages]
SET #lastId = (SELECT IDENT_CURRENT('tblArchivedMessages'))
UPDATE [dbo].[tblArchivedMessages]
SET FromToMach = #fromTo
WHERE ID = #lastId
END
GO
Regards.
Here is the problem:
for your reference:
database entries 1,2 and 3 are made using jython 2.2.1 using jdbc1.2.
database entry 4 is made using vb the old to be replace program using odbc.
We have found that if I copy and paste both jython and vb MailBody entries to wordpad directly from that SQL Server Enterprise Manager software it outputs the format perfectly with correct line returns. if I compare the bytes of each file with a hex editor or KDiff3 they are binary identically the same.
There is a 3rd party program which consumes this data. Sadly that 3rd party program reads the data and for entries 1 to 3 it displays the data without line returns. though for entry 4 it correctly formats the text. As futher proof we can see in the picture, the data in the database is displayed differently.
Somehow the line returns are preserved in the database for the vb entries but the jython entries they are overlooked. if I click on the 'MailBody' field of entry 4 i can press down i can see the rest of the email. Whereas the data for jython is displayed in one row.
What gives, what am i missing, and how do I handle this?
Here is a snippet of the code where I actually send it to the database.
EDIT: FYI: please disregard the discrepancies in the 'Processed' column, it is irrelevant.
EDIT: what i want to do is make the jython program input the data in the same way as the vb program. So that the 3rd party program will come along and correctly display the data.
so what it will look like is every entry in 'MailBody' will display "This is a testing only!" then next line "etc etc" so if I was to do a screendump all entries would resemble database entry 4.
SOLVED
add _force_CRLF to the mix:
def _force_CRLF(self, data):
'''Make sure data uses CRLF for line termination.
Nicked the regex from smtplib.quotedata. '''
print data
newdata = re.sub(r'(?:\r\n|\n|\r(?!\n))', "\r\n", data)
print newdata
return newdata
def _execute_insert(self):
try:
self._stmt=self._con.prepareStatement(\
"INSERT INTO EmailHdr (EntryID, MailSubject, MailFrom, MailTo, MailReceive, MailSent, AttachNo, MailBody)\
VALUES (?, ?, ?, ?, ?, ?, ?, cast(? as varchar (" + str(BODY_FIELD_DATABASE) + ")))")
self._stmt.setString(1,self._emailEntryId)
self._stmt.setString(2,self._subject)
self._stmt.setString(3,self._fromWho)
self._stmt.setString(4,self._toWho)
self._stmt.setString(5,self._format_date(self._emailRecv))
self._stmt.setString(6,self._format_date(self._emailSent))
self._stmt.setString(7,str(self._attachmentCount))
self._stmt.setString(8,self._force_CRLF(self._format_email_body()))
self._stmt.execute()
self._prepare_inserting_attachment_data()
self._insert_attachment_data()
except:
raise
def _format_email_body(self):
if not self._emailBody:
return "could not extract email body"
if len(self._emailBody) > BODY_TRUNCATE_LENGTH:
return self._clean_body(self._emailBody[:BODY_TRUNCATE_LENGTH])
else:
return self._clean_body(self._emailBody)
def _clean_body(self,dirty):
'''this method simply deletes any occurrence of an '=20' that plagues my output after much testing this is not related to the line return issue, even if i comment it out I still have the problem.'''
dirty=str(dirty)
dirty=dirty.replace(r"=20","")
return r"%s"%dirty
I suggest to add a debug output to your program, dumping character codes before insertion in DB. There are chances that Jython replace CrLf pair with single character and doesn't restore it when written to DB.
You should look at the quopri module (and others regarding email) so you don't have to use dirty tricks as _clean_body