I am trying to insert string-values larger than 5500 characters into a MSSQL 2008 database. I get the error
String or binary data would be truncated.
when I try to insert these values even though the data type of the column is declared nvarchar(max). Is there a cap on the string-size that I can enter? If so, what would be a solution to this problem?
EDIT
When i manually enter the data via mgmt studio it works. However, I'm performing the updates via JDBC-driver & prepared statements.
My query is of the form:
UPDATE table SET columnX = value1 WHERE columnX = value2;
I add this statement to a batch, and once every 1000 statements I execute them.
value1 in this case contains a large amount of characters.
columnX is definitely defined as nvarchar(max)
Microsoft encourages us to use the SQLServerPreparedStatement to modify large-value-types like varchar(max).
http://msdn.microsoft.com/en-us/library/ms378813.aspx
You could also use a stored procedure to solve the problem, that is what I did, when I had similar problems with JDBC/Pentaho.
You could write a stored procedure on the sql-server that fulfills the task. And just call that stored procedure from your Java-code.
I hope that helps!
Related
I am doing a project about querying relational data of PostgreSQL on Matlab. I have followed this example to connect Matlab and PostgreSQL.
% Add jar file to classpath (ensure it is present in your current dir)
javaclasspath('postgresql-9.0-801.jdbc4.jar');
% Username and password you chose when installing postgres
props=java.util.Properties;
props.setProperty('user', '<your_postgres_username>');
props.setProperty('password', '<your_postgres_password>');
% Create the database connection (port 5432 is the default postgres chooses
% on installation)
driver=org.postgresql.Driver;
url = 'jdbc:postgresql://<yourhost>:<yourport>/<yourdb>';
conn=driver.connect(url, props);
% A test query
sql='select * from <table>'; % Gets all records
ps=conn.prepareStatement(sql);
rs=ps.executeQuery();
% Read the results into an array of result structs
count=0;
result=struct;
while rs.next()
count=count+1;
result(count).var1=char(rs.getString(2));
result(count).var2=char(rs.getString(3));
...
end
I am able to get the column name from the PgResultSet by using the ResultSetMetaData
rsmd = rs.getMetaData();
columnsNumber = rsmd.getColumnCount();
name1 = rsmd.getColumnName(1);
name2 = rsmd.getColumnName(2);
but I am unable to set it to the result of the query
while rs.next()
count=count+1;
result(count).var1=char(rs.getString(2));
result(count).var2=char(rs.getString(3));
...
end
The var1 and var2 are shown as the name of the columns on the struct data result and when I assign the variable name1 and name2, it still shows "name1" and "name2" in replace for "var1" and "var2" on the struct result instead of the column name of the PostgreSQL that I have set in variable name1 and name2.
Thank you in advanced!
What concerns your code, it should be only slightly fixed to do what you want. The corrected code is as follows:
while rs.next()
count=count+1;
result(count).(name1)=char(rs.getString(2));
result(count).(name2)=char(rs.getString(3));
...
end
The matter was in dynamic field references, i.e. when you reference some field by its name put in a string variable, this is done by the notation .(). For more details on dynamic field referencing please see, for instance, a post "Use Dynamic Field References" by Loren Shure on the blog "Loren on the Art of Matlab".
But I'd like to note that, as far as I know, using JDBC as a connector to PostgreSQL is reasonable only for the case
when you have to import/export a rather limited volume of data of only scalar types (scalar numerics, logicals, strings, timestamps and so on). If you have to deal with more complex types (such as arrays) or the volume of data is rather big (about 1Gb), JDBC is no longer efficient (not to say in certain cases almost impossible to use). IMHO for such situations there is a more efficient and convenient way to solve both your problems (retrieving column names and importing data in the form of Matlab structures). Namely, you can use PgMex for these purposes. Your code above can be transformed in such a way (we assume that all the parameters below marked by <> signs are properly filled and the corresponding table exists in the respective database):
% Create the database connection
dbConn=com.allied.pgmex.pgmexec('connect',[...
'host=<yourhost> dbname=<yourdb> port=<yourport> '...
'user=<your_postgres_username> password=<your_postgres_password>']);
% A test query
sql='select * from <table>'; % Gets all records
pgResult=com.allied.pgmex.pgmexec('exec',dbConn,sql); % Perform this test query
To get column names you need to execute the following code:
nFields=com.allied.pgmex.pgmexec('nFields',pgResult);
fieldNameCVec=cell(nFields,1);
for iField=1:nFields
fieldNameCVec{iField}=com.allied.pgmex.pgmexec('fName',pgResult,iField-1);
end
Finally, to put these results in a structure you need to do as follows:
% Read the results
outCVec=cell(nFields,1);
fieldSpecStr='%<field_type_1> %<field_type_2> ...';
inpCVec=num2cell(0:nFields-1);
[outCVec{:}]=com.allied.pgmex.pgmexec('getf',pgResult,...
fieldSpecStr,inpCVec{:});
% Get only values ignoring NULLs (if NULLS are not to be ignored, the code
% should be improved by taking into accout not only valueVec, but
% also isNullVec and isValueNull being indicators of NULLs)
fieldValCVec=cellfun(#(SFieldValInfo)SFieldValInfo.valueVec,...
outCVec,'UniformOutput',false);
SResult=struct(transpose([fieldNameCVec(:) fieldValCVec(:)]));
What concerns the format of input and output arguments for the command
getf (including fieldSpecStr) please see its documentation on PgMex website.
It should be noted, that values of fields for all tuples are retrieved
for one call of the command (instead of a cycle you use in your own code)
and that it is done very fast (approximately 3.5 times
faster rather than the same operation done through Matlab Database Toolbox
working via a direct JDBC connection as mentioned at the end of
"Performance comparison of PostgreSQL connectors in Matlab" article).
And you do not need to convert these values somehow, all is done in Matlab-friendly and native way (in a form of matrices,
multi-dimensional arrays, structures and arbitrary other Matlab formats).
All values of each particular field are put by the code above into
the respective field of SResult structure simply as an array with a size along the first dimension coinciding with the number of tuples
retrieved. If you like to have results for each separate tuple in a separate
structure, you can use the following code:
fieldValCVec=cellfun(#(valueVec)num2cell(valueVec,[2 ndims(valueVec)]),...
fieldValCVec,'UniformOutput',false);
tupleFieldValCMat=transpose(horzcat(fieldValCVec{:}));
SResultCVec=cellfun(#(tupleFieldValCVec)struct(...
transpose([fieldNameCVec(:) tupleFieldValCVec(:)])),...
num2cell(tupleFieldValCMat,1),'UniformOutput',false);
SResultVec=vertcat(SResultCVec{:});
EDIT: Academic licenses for PgMex are free of charge.
I am having trouble puting those single quotes for ASCII/Timestamp columns and not puting for other types like Int, Decimal, Boolean etc.
The data comes from another db/table, which is a sql.
I have all the column data as string. I don't want to format each column data to check null values and then decide to put quote or not.
Is it possible to pass in insert data value without giving single quotes, using prepared statement or whatever.
If you don't want to write a loader that uses prepared statements (via the CQL driver...which is a good idea), I can think of one other way. To import without using single quotes, you should be able to accomplish this with the COPY FROM CQL3 command (setting the QUOTE parameter to an empty string). If you can dump your RDBMS data to a csv file, you should be able to insert those values into Cassandra like this:
COPY myColumnFamily (colname1,colname2,colname3)
FROM '/home/myUser/rdbmsdata.csv' WITH QUOTE='';
Check out the documentation on the COPY command for more information. Examples can be found here.
EDIT:
I also read the above question and assumed that you did not want a prepared statement-based answer. Since that's obviously not the case, I thought I'd also provide one here (using DataStax's Java CQL driver). Note that this answer is based on my column family and column names from my example above, and assumes that col1 is the (only) primary key.
PreparedStatement statement = session.prepare(
"UPDATE myKeyspace.myColumnFamily " +
"SET col2=?, col3=? " +
"WHERE col1=?");
BoundStatement boundStatement = statement.bind(
strCol2, strCol3, strCol1);
session.execute(boundStatement);
This solution does not require you to encapsulate your string data in single quotes, and has a few added benefits over your String.ReplaceAll:
Allows you to insert values containing single quotes.
Escapes your values, protecting you from CQL-Injection (the lesser-known relative of SQL-Injection).
In CQL, both UPDATE and INSERT add a record if it does not exist and update it if it does (effectively known as an "UPSERT"). Using an UPDATE over an INSERT supports counter columns (if your schema ends up using them).
Prepared statements are faster, because they allow Cassandra to only have to parse the query once, and then re-run that same query with different values.
For more information, check out DataStax's documentation on using prepared statements with the Java Driver.
Finally did it using String.format clubbed with replace
String.format("INSERT INTO xyz_zx(A,B,C,D) VALUES('%s','%s',%s,%s);",(Object[])Strings).replaceAll("'null'","null");
So I am querying a MySql database from my java application and I am trying to use a query,
Select count(*) from table where `NUMERIC`='1'
to count the rows from a database. When I run this query it works fine, and I get a 1 returned (I am using a test db with 12 records, Numeric has values 1-12 so this makes sense). However I wanted to try to break this and do some error handling. I changed my query to
Select count(*) from table where `Numeric`='1adjfa'
I expected this to return 0, however it still returns 1. In fact, as long as I have 1 at the beginning of the value it will work, if I change the value to just 'adjfa' than it returns 0. I have confirmed this through both my Java App and the MySQL workbench. Any ideas as to why this returns 1, even with the junk at the end of it?
Two different data types can not be compared. Instead one of the two needs to be cast/coerced to the same data type as the other.
In your case you're not doing the coercion, so the DB Engine is doing an implicit coercion.
Based on data-type-order-of-precedence, the database engine chooses the string to be coerced to a numeric.
The value '1adjfa' therefore becomes a 1, and then your comparison is being made.
This results is your query effectively being:
Select count(*) from table where `Numeric` = 1
You should either not be comparing numerics and strings, or do the coercion yourself, for example...
Select count(*) from table where CAST(`Numeric` AS VARCHAR(32)) = '1adjfa'
In terms of breaking the query, I'm hoping that in your application you're actually using parameterised queries. This will allow you to define the data-type of the parameter, and your application should throw the error if the wrong data-type is supplied.
Numeric has a number data type. To make the comparision to 1adjfa the DB engine tries to convert it also to a number which results in 1 and the rest gets cut off.
I'm converting (or trying to) an Ms AccessDB into derby.
When I extract the data from certain varchar / text / memo field from access they are filled with apostrophe, and mathematical symbols (percent, less than etc), and possible foreign characters
I need to keep these and I test for them so as I can use an 'escape sequence' to ensure they get put into the database.
However for now I am unable to get the data into the DB without it failing on these fields. When the SQL fails I output the SQL string, and cut and past it into ij. Then I modify just the first record, and it is always these characters that cause me grief.
I've tried to modify the strings by surrounding with "double quote marks" but that just gives a different error (stating that it has 'enounterd """ at line1 column x' which is always the first occurance of the double quote).
I haven't found a setting in derby to alter the behaviour for strings, yet. Is there one?
I have also tried to set the SQL statment to a preparedStatement then use the {call preparedStatement} again this fails also. I can't use the {escape "escape char} in a normal statment as derby just says incorrect syntax at me.
How do others manage to get user content with strange characters into a field in derby?
Do I need to change my field into a CLOB or something other than varchar / long Varchar?
Are my problems being caused by using the wrong characteset (eg iso rather UTF-8), how do I tell what it is, how to change it?
Below is a sample of the SQL insert that fails when I send it to derby (via my JAVA 'programme')
insert into S1.SORTIEDESSAI (OBS, DATEDUSORTIE, CONTREINDIC, FIN,
PDEVU, REFUS, INVDECISN, ADMIN, MOTIF_DE_LA_SORTIE, NOMVALIDEE,
DATEVALIDEE) values ('"0001/0001"' , '2007-07-15' , false , true ,
'"null"' , '"null"' , '"null"' , '"null"' , '"2. FIN DE L’ESSAI"' ,
'"DR SIMON"' , '2011-04-19' )
Note:
Actually I look at the above and notice that the order of columns names isn't good? It was OK yesterday, not sure why it would have changed? something to do with Access returning the column names in a random order from the resultSetMetaData, which would be a surprise.
for now I recomend any further answers to hold off whilst I sort this problem out, OK solved that problem, do I need to set another question about this behaviour....
Back to the main thread...
Ok as you can see on my SQL statement I have wrapped any varchar fields in double quotes. This always fails (even directly through ij). help help help...
I'm not quite sure what your question is, but in general you can input these characters by using a PreparedStatement of the form: INSERT INTO tablename (columnname) values (?), and then using the PreparedStatement.setString() method to supply your character data for that column.
I have a field with varchar(100) in mysql, I want to store first 100 characters because my data length is 200 characters(ignore last 100 character).I doesn't want to change my source code. Which is possible in MS-Access and MS Server but I want to do this in mysql.
I am applying this in java with hibernate, means I am not writing insertion code for this. Here I am just using save() method and its throwing "Large data".
I have got Exception-
Caused by: java.sql.BatchUpdateException: Data truncation: Data too long for column 'FBUrl' at row 1
at com.mysql.jdbc.PreparedStatement.executeBatchSerially(PreparedStatement.java:1527)
at com.mysql.jdbc.PreparedStatement.executeBatch(PreparedStatement.java:1065)
at org.hibernate.jdbc.BatchingBatcher.doExecuteBatch(BatchingBatcher.java:58)
at org.hibernate.jdbc.AbstractBatcher.executeBatch(AbstractBatcher.java:195)
glb.chatmeter.exception.AdException: Could not Save Facebook page Data.
Note: An hour after the below, the question was edited with a substantial change. This answer answered the question as it was originally, but doesn't address the edited version.
You can use substring:
INSERT INTO MyTable (Myfield) values (SUBSTRING('long string', 1, 100))
The pos parameter starts at 1 (oddly), and it's okay if the len parameter is larger than the length of what you're actually inserting.
You can use SUBSTRING() to trim inserts:
INSERT INTO table (column) VALUES (SUBSTRING("your data...", 1, 100))
try this
SELECT INSERT('your string', 0, 100, '');
REFERENCE
The only way to do it without changing the source code is to fiddle with the configuration of the MySQL server. More specifically, the sql_mode variable:
http://dev.mysql.com/doc/refman/5.1/en/server-sql-mode.html
I believe you have to set STRICT_TRANS_TABLES:
For STRICT_TRANS_TABLES, MySQL
converts an invalid value to the
closest valid value for the column and
insert the adjusted value. If a value
is missing, MySQL inserts the implicit
default value for the column data
type. In either case, MySQL generates
a warning rather than an error and
continues processing the statement.
Implicit defaults are described in
Section 10.1.4, “Data Type Default
Values”.
However, it's important to note that this setting will affect many other things. The only reason I see not to change the source code is that you don't have access to it, and in such case you can probably just enlarge the DB column.