The short:
I installed MySQL 8.0.27 on a newish MacBook running OSX 11.6 (BigSur).
I use binary logging and mysqlbinlog to generate updates for older mysql 5.7 installs on separate machines.
The problem is that the mysqlbinlog-generated SQL for updates made through JDBC includes the statement
SET ##session.character_set_client=255,##session.collation_connection=255,##session.collation_server=83/!/;
which the older mysqls do not handle.
Following advice from http://minsql.com/mysql8/MySQL-8.0-to-5.7-backward-replication/, I set up the mysql8 such that mods made manually using the mysql client do not generate that line, instead setting the character_set_client=33 (utf8).
But when mods are made by the application using JDBC (mysql-connector/J v 8.0.24) that offending statement returns.
The JDBC connection URL includes the characterEncoding=UTF-8 option.
My question is: Is there some way I can get mysqlbinlog to not set the character_set_client=255 when mods are made through JDBC through some combination of mysql setup, jdbc connection configuration, and/or mysqlbinlog options?
More:
Yes, I know down-version replication (which, technically, this is not) is not officially supported. This system does non-real-time database synchronization between mostly offline clients.
The my.cnf is:
[mysqld]
port=63306
server-id=0
log-bin=lolo-bin.log
log-bin-index=bin-log.index
max_binlog_size=100M
binlog_format=mixed
binlog-do-db=database
character_set_server=utf8
collation_server=utf8_bin
init-connect='SET NAMES utf8 COLLATE utf8_bin'
[client]
# doesn't work with mysqlbinlog
# default-character-set=utf8
JDBC connection URL is:
jdbc:mysql://localhost:63306/database?characterEncoding=utf8
I have added the --set-charset=UTF8 mysqlbinlog option to no avail.
Partial mysqlbinlog SQL output (with set character_set_client at end):
# The proper term is pseudo_replica_mode, but we use this compatibility alias
# to make the statement usable on server versions 8.0.24 and older.
/*!50530 SET ##SESSION.PSEUDO_SLAVE_MODE=1*/;
/*!50003 SET #OLD_COMPLETION_TYPE=##COMPLETION_TYPE,COMPLETION_TYPE=0*/;
/*!40101 SET #OLD_CHARACTER_SET_CLIENT=##CHARACTER_SET_CLIENT */;
/*!40101 SET #OLD_CHARACTER_SET_RESULTS=##CHARACTER_SET_RESULTS */;
/*!40101 SET #OLD_COLLATION_CONNECTION=##COLLATION_CONNECTION */;
/*!40101 SET NAMES UTF8 */;
DELIMITER /*!*/;
# at 4
#211101 10:04:33 server id 0 end_log_pos 125 CRC32 0xdd1b24c3 Start: binlog v 4, server v 8.0.27 created 211101 10:04:33 at startup
# Warning: this binlog is either in use or was not closed properly.
ROLLBACK/*!*/;
BINLOG '
IR6AYQ8AAAAAeQAAAH0AAAABAAQAOC4wLjI3AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAhHoBhEwANAAgAAAAABAAEAAAAYQAEGggAAAAICAgCAAAACgoKKioAEjQA
CigBwyQb3Q==
'/*!*/;
# at 125
#211101 10:04:33 server id 0 end_log_pos 156 CRC32 0x79526201 Previous-GTIDs
# [empty]
# at 156
#211101 10:05:08 server id 0 end_log_pos 235 CRC32 0xcbe86045 Anonymous_GTID last_committed=0 sequence_number=1 rbr_only=no original_committed_timestamp=1635786308071091 immediate_commit_timestamp=1635786308071091 transaction_length=674
# original_commit_timestamp=1635786308071091 (2021-11-01 10:05:08.071091 PDT)
# immediate_commit_timestamp=1635786308071091 (2021-11-01 10:05:08.071091 PDT)
/*!80001 SET ##session.original_commit_timestamp=1635786308071091*//*!*/;
/*!80014 SET ##session.original_server_version=80027*//*!*/;
/*!80014 SET ##session.immediate_server_version=80027*//*!*/;
SET ##SESSION.GTID_NEXT= 'ANONYMOUS'/*!*/;
# at 235
#211101 10:05:08 server id 0 end_log_pos 330 CRC32 0x38ea85f7 Query thread_id=7818 exec_time=0 error_code=0
SET TIMESTAMP=1635786308/*!*/;
SET ##session.pseudo_thread_id=7818/*!*/;
SET ##session.foreign_key_checks=1, ##session.sql_auto_is_null=0, ##session.unique_checks=1, ##session.autocommit=1/*!*/;
SET ##session.sql_mode=1168113696/*!*/;
SET ##session.auto_increment_increment=1, ##session.auto_increment_offset=1/*!*/;
/*!\C utf8mb4 *//*!*/;
SET ##session.character_set_client=255,##session.collation_connection=255,##session.collation_server=83/*!*/;
SET ##session.lc_time_names=0/*!*/;
SET ##session.collation_database=DEFAULT/*!*/;
/*!80011 SET ##session.default_collation_for_utf8mb4=255*//*!*/;
BEGIN
/*!*/;
# at 330
#211101 10:05:08 server id 0 end_log_pos 799 CRC32 0xfd49bf12 Query thread_id=7818 exec_time=0 error_code=0
thanks,
Linus
When upgrading data from 5.7 to 8.0, a non-trivial script must be run the handle the differences in this major upgrade.
Review the steps you used.
I tried to insert some special character via java into oracle table and then retrieve it again--assuming my encoding will work.
Below is the code which i tried.
String s=new String("yesterday"+"\u2019"+"s");
...
statement.executeUpdate("INSERT into test1 values ('"+s+"')");
ResultSet rs=statement.executeQuery("select * from test1");
while (rs.next()) {
System.out.println(new String(rs.getString(1).getBytes("UTF-8"),"UTF-8"));
}
...
Now, when I try to see output via commandline execution it displays special character always: yesterday’s
My question is: why even after using encoding, it is not showing expected result. i.e. yesterday’s. Is above mentioned code is not correct or some modification is required?
P.S.: In eclipse, the code might result yesterday’s, but if executed via command line , it shows yesterday’s
I am using :
-- JDK1.6
-- Oracle : 11.1.0.6.0
-- NLS_Database_Parameters: NLS_CHARACTERSET WE8MSWIN1252
--Windows
Edit:
\u2019 : this is RIGHT SINGLE QUOTATION MARK & I am looking for this character only.
Check the java property "file.encoding" when you run on the commandline, it may be set to something other than "UTF-8" causing the text to display incorrectly when you output on the commandline.
Here is an illustration of what I suggested in a comment (change the character set of your client). Straight from my SQL*Plus:
SQL> select unistr('\2019') from dual;
U
-
Æ
SQL> $chcp 1252
Active code page: 1252
SQL> select unistr('\2019') from dual;
U
-
’
If this works for you, you may want to add $chcp 1252 to your [g]login.sql.
The problem is that the character encoding for the apostrophe is \u0027
I ran this in the command line:
public class Yesterday{
public static void main(String[] args) {
String s = new String("yesterday" + "\u0027" +"s");
System.out.println(s);
}
}
it resulted in:
yesterday's
I get following link from the google http://translate.google.com/translate_a/t?client=t&hl=en&sl=auto&tl=fa&multires=1&prev=btn&ssel=0&tsel=3&uptl=fa&alttl=en&sc=1&text=hello which contain some persian alphabets, so I wanna save it into mysql database with following code :
pageurl = new URL("http://translate.google.com/translate_a/t?client=t&hl=en&sl=auto&tl=fa&multires=1&prev=btn&ssel=0&tsel=3&uptl=fa&alttl=en&sc=1&text=of");
t = pageurl.openConnection();
t.setRequestProperty("User-Agent", "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.4; en-US; rv:1.9.2.2) Gecko/20100316 Firefox/3.6.2");
in = new BufferedReader(new InputStreamReader(t.getInputStream()));
preparedStatement2 = con.prepareStatement("update `en_db` set `meaning` = ? where `id` = ?");
preparedStatement2.setString(1, in.readLine());
preparedStatement2.setInt(2, id);
preparedStatement2.executeUpdate();
in.close();
but it will save something wrong in database,like [[["??","of","",""]],[["preposition",["??","?? ????","?? ???","?? ????","?? ???","?","?? ????","?? ????","?? ???","??????"],[["??",["of","from","in","by"]],["?? ????",["of"]],["?? ???",["on behalf of","of","for"]],["?? ????",["about","on","concerning","of","toward","in re"]],["?? ???",["of","with"]],["?",["of"]],["?? ????",["of"]],["?? ????",["of"]],["?? ???",["of"]],["??????",["by","via","per","of","with"]]]]],"en",,[["??",[5],0,0,1000,0,1,0]],[["of",4,,,""],["of",5,[["??",1000,0,0],["?? ??",0,0,0],["??????? ??",0,0,0],["?? ??",0,0,0]],[[0,2]],"of"]],,,,6]
if I print it by System.out.println it will show [[["از","of","",""]],[["preposition",["از","از لحاظ","از طرف","در باره","در جهت","ز","از مبدا","از منشا","در سوی","بوسیله"],[["از",["of","from","in","by"]],["از لحاظ",["of"]],["از طرف",["on behalf of","of","for"]],["در باره",["about","on","concerning","of","toward","in re"]],["در جهت",["of","with"]],["ز",["of"]],["از مبدا",["of"]],["از منشا",["of"]],["در سوی",["of"]],["بوسیله",["by","via","per","of","with"]]]]],"en",,[["از",[5],0,0,1000,0,1,0]],[["of",4,,,""],["of",5,[["از",1000,0,0],["ای از",0,0,0],["استفاده از",0,0,0],["را از",0,0,0]],[[0,2]],"of"]],,,,16]
How should I solve it?
All of the other answers, and: your database connection URL should be something like:
jdbc:mysql://localhost/mydatabase?useUnicode=true&characterEncoding=UTF-8
This ensures that the driver communicates in UTF-8 too.
Important
in = new BufferedReader(new InputStreamReader(t.getInputStream(), "UTF-8"));
I have seen in Stack Overflow answers to google Translate that the header was given with a language to receive the correct encoding, but everything is already fine.
You table fields are defined with a character set which does not support Persian characters (most probably, Latin1)
You need to convert them into a character set which supports them:
ALTER TABLE en_db MODIFY meaning VARCHAR(100) CHARACTER SET UTF8;
(for each field individually), or
ALTER TABLE en_db CONVERT TO CHARACTER SET UTF8;
(for all fields).
Check how your MySql DB is configured.
Take a look on the following article: http://dev.mysql.com/doc/refman/5.0/en/charset-unicode.html
The encoding can be configured either globally (per DB) or per table:
CREATE TABLE `mytable` (
.................
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
I´ve been having this problem for a long time, I´ve searched the internet many times for the solution, tried lots of them but not found an adequate solution.
I really don´t know what to do so if you could please help me I´d be very thankful.
(Sorry for my poor english).
Question: How can I solve the charset incompatibility between the input archive and a MYSql table?
Problem: When importing the archive from on my computer the information appears in my database, but some chars as ('ã', 'ç', 'á', etc..) are shown as ?.
Aditional information
I'm using MYSql, my version and variable status are:
MySQL VERSION : 5.5.10
HOST : localhost
USER : root
PORT : 3306
SERVER DEFAULT CHARSET : utf8
character_set_client : utf8
character_set_connection : utf8
character_set_database : utf8
character_set_filesystem : BINARY
character_set_results : utf8
character_set_server : utf8
character_set_system : utf8
collation_connection : utf8_general_ci
collation_database : utf8_general_ci
collation_server : utf8_general_ci
completion_type : NO_CHAIN
concurrent_insert : AUTO
The query that´s being used is:
LOAD DATA LOCAL INFILE 'xxxxx/file.txt'
INTO TABLE xxxxTable
FIELDS TERMINATED BY ';'
LINES TERMINATED BY ' '
IGNORE 1 LINES
( status_ordenar,numero,newstatus,rede,data_emissao,inicio,termino,tempo_indisp
, cli_afet,qtd_cli_afet,cod_encerr,uf_ofensor,localidades,clientes_afetados
, especificacao,equipamentos,area_ofens,descricao_encerr,criticidade,cod_erro
, observacao,id_falha_perc,id_falha_conf,nba,solucao,falhapercebida,falhaconfirmada
, resp_i,resp_f,resp_ue,pre_handover,falha_identificada,report_netcool,tipo_falha
, num_notificacao,equip_afetados,descricao)
About the file being imported:
I´ve opened the file with open office whith 3 charsets:
UTF8 - Gave me strange chars in place of the 'ç', 'ã', etc...
ISO-8859-1 - OK.
WIN-1252 - OK.
ASCII/US - OK.
Already tested: I´ve tested some charsets in my database: latin1, utf-8, ascii, but all of them gave me the same result (? instead of 'á', 'ç' etc).
Extra: I'm using Java with Java JDBC to generate and send the query.
file.txt is saved in ISO-8859-1 or Windows-1252 (these two are very similar), and being interpreted as UTF-8 by MySQL. These are incompatible.
How can I tell?
See point 3.: the file displays correctly when interpreted as ISO-8859-1 or Windows-1252.
See point 1.: character_set_database : utf8
Solution: either convert the file to UTF-8, or tell MySQL to interpret it as ISO-8859-1 or Windows-1252.
Background: the characters you provide (ã etc.) are single-byte values in windows-1252, and these bytes are illegal values in UTF-8, thus yielding the '?'s (unicode replacement characters).
Snippet from MySQL docs:
LOAD DATA INFILE Syntax
The character set indicated by the character_set_database system variable is used to interpret the information in the file.
Saved your characters with standard Windows Notepad as UTF-8 file (Notepad++ is also OK).
Exact file content:
'ã', 'ç', 'á'
MySQL version: 5.5.22
Database charset: utf8
Database collation: utf8_general_ci
CREATE TABLE `abc` (
`qwe` text
) ENGINE=InnoDB DEFAULT CHARSET=utf8
Imported data with command
LOAD DATA LOCAL INFILE 'C:/test/utf8.txt'
INTO TABLE abc
FIELDS TERMINATED BY ';'
LINES TERMINATED BY ' '
IGNORE 1 LINES
( qwe)
Result (displayed in SQLyog):
So, first - you should check original file with reliable editor (notepad, notepad++). If file corrupted, then you should take another file.
Second - if file is OK, add you Java code for sending data to MySql into question.