On Windows - where is a mqji.properties for me to use? - java

My application is a stand alone Java app, that customers download, install and run. It uses MQ to communicate with a host, which has been working for years. Neither myself nor customers have anything MQ installed on their Windows machines; we include and use com.ibm.mq.jar to do the work.
Apparently though, MQ needs a mqji.properties file in the classpath to prevent this :
Unable to load message catalog - mqji
com.ibm.mq.MQException: Message catalog not found
So my question is : Where do I get one??

Here is a copy of my mqji.properties file.
Copy and paste into a text editor.
Save as mqji.properties
Put the directory that this file is in into your CLASSPATH.
# mqjiEn_US.properties
# Messages produced by the Websphere MQ Java interface classes
# (shared by bindings and by client)
# Messages beginning with MQJI are explanations for exceptions
# Messages beginning with MQJE are errors
#
# From class MQDistributionList
#
1=MQJI001: Queue manager object was null.
2=MQJI002: Not connected to a queue manager.
3=MQJI003: No object records supplied.
4=MQJI004: No response records supplied.
5=MQJI005: Not enough response records supplied.
6=MQJI006: An object or response record was null.
7=MQJI007: openResponse parameter was null.
8=MQJI008: Null message passed to put.
9=MQJI009: Null put message options passed to put.
10=MQJI010: Number of message trackers and response records do not match.
11=MQJI011: The distribution list has been closed.
#
# From class MQEnvironment
#
12=Websphere MQ Client for Java v5.3
#
# From class MQGetMessageOptions
#
13=MQJI013: Unsupported version number - {0}
14=MQJI014: Insufficient data received from queue manager.
15=MQJI015: Erroneous eyecatcher : {0}
#
# From class MQManagedObject
#
16=MQJI016: Object has been closed.
17=MQJI017: No selectors specified.
#
# From class MQMD
#
18=MQJI015: Erroneous eyecatcher : {0}
19=MQJI018: Array copy error in MQMD
#
# From class MQMessage
#
20=MQJI019: Malformed UTF string in MQMessage::readLine
21=MQJI020: Unsupported codeset : {0}
22=MQJI021: String index error occurred during codeset conversion
23=MQJI015: Erroneous eyecatcher : {0}
#
# From class MQOD
#
24=MQJI022: Unmatched number of object and response records
#
# From class MQPutMessageOptions
#
25=MQJI023: Null MQMessageTracker object supplied
#
# From class MQMessageTracker
#
26=MQJI024: Array copy error in MQMessageTracker
#
# From class MQQueue
#
27=MQJI025: Null MQMessage passed to get
28=MQJI026: Null MQGetMessageOptions passed to get
29=MQJI027: The queue has been closed
30=MQJI028: Null MQMessage passed to put
31=MQJI029: Null MQPutMessageOptions passed to put
#
# From class MQQueueManager
#
32=MQJI030: The queue manager does not support distribution lists.
#
# From class MQS390FloatSupport
#
33=MQJI031: Number outside of range for double precision S/390 Float
#
# From class MQS390PackedDecimalSupport
#
34=MQJI032: Invalid sign nibble in packed decimal
35=MQJI033: Packed Decimal digit outside of range 0-9
36=MQJI034: Outside of range for short packed decimal (+/-999)
37=MQJI035: Outside of range for integer packed decimal (+/-9999999)
38=MQJI036: Outside of range for long packed decimal (0+/-999999999999999)
#
# From class MQException
#
39=MQJE001: Completion Code {0}, Reason {1}
40=MQJE001: An MQException occurred: Completion Code {0}, Reason {1}\n{2}
#
# Messages produced by client only classes...
#
41=MQJI037: Error occurred during Websphere MQ API call - reason code {0}
42=MQJI038: Unexpected internal error during string index processing
43=MQJE002: Socket output stream was null
44=MQJE003: IO error transmitting message buffer
45=MQJE004: Socket input stream was null
46=MQJE005: TSH eyecatcher not found. Eyecatcher was {0}
47=MQJE006: Internal error during array copy
48=MQJE007: IO error reading message data
49=MQJE008: IOException whilst sending status message
50=MQJE009: Failed to build API header
51=MQJE010: Unknown host: {0}
52=MQJE011: Socket connection attempt refused
53=MQJE012: Security error - cannot connect to host {0}
54=MQJE013: Error accessing socket streams
55=MQJE014: Control Point rejected connection
56=MQJE015: Error connecting to Control Point
57=MQJE016: MQ queue manager closed channel immediately during connect\n\Closure reason = {0}
58=MQJE017: MQ queue manager sent status error {0} during connect
59=MQJE018: Protocol error - unexpected segment type received
60=MQJE019: Error creating initial data segment
61=MQJE020: CCSID not supported by queue manager
62=MQJE021: Encoding not supported by queue manager
63=MQJE022: FAP level not supported by queue manager
64=MQJE023: Negotiation failed on maximum messages per batch
65=MQJE024: Sequence wrap value not supported by queue manager
66=MQJE025: Channel closed after two initial changes. Closure reason {0}
67=MQJE026: Queue manager sent status error {0} during connect
68=MQJE027: Queue manager security exit rejected connection with error code {0}
69=MQJE028: Channel closed during security exchanges
70=MQJE029: Unexpected message type sent by queue manager
71=MQJE030: IOException during security flows
72=MQJE031: Security exit closed the channel
73=MQJE032: Queue manager security exit rejected connection with reason code {0}
74=MQJE033: A required security flow was not sent by the queue manager
75=MQJE034: Unexpected message type sent by queue manager
76=MQJE035: Negotiated maximum transmission size is too small
77=MQJE036: Queue manager rejected connection attempt
78=MQJE037: Remote queue manager closed the connection
79=MQJE038: Unexpected segment type {0} received
80=MQJE039: IOException whilst building connection data stream
81=MQJE040: Channel closed by exit
82=MQJE041: Unsupported version number - (0)
83=MQJE042: Erroneous eyecatcher: {0}
84=MQJE043: Insuffucient data received from queue manager
85=MQJE044: Array copy error in MQMD
86=MQJE045: Malformed UTF string
87=MQJE046: Unsupported codeset : {0}
88=MQJE047: String index error occurred during codeset conversion
89=MQJE048: Invalid sign nibble in packed decimal
90=MQJE049: Packed Decimal digit outside of range 0-9
91=MQJE050: Outside of range for short packed decimal (+/-999)
92=MQJE051: Outside of range for integer packed decimal (+/-9999999)
93=MQJE052: Outside of range for long packed decimal (0+/-999999999999999)
94=Websphere MQ Bindings for Java v5.3
95=MQJE053: The Websphere MQ Bindings for Java library could not be loaded
96=MQJE054: The queue manager does not support distribution lists
97=MQJE055: The queue manager does not support version 2 Websphere MQ API structures
98=MQJE056: Initial negotiation failure
99=MQJE057: Channel closed during security exchanges
100=MQJE058: Invalid number of object or response records
101=MQJE059: String index error
102=MQJE060: Could not find class {0}
103=MQJE061: Could not find field {0}
104=MQJE062: Could not find method {0}
#
# Messages used by MQManagedConnectionJ11, ManagedConnectionFactories
# and MQManagedConnectionMetaData
#
105=MQJI039: Invalid ConnectionRequestInfo object
106=MQJI040: MQManagedConnection already destroyed
107=MQJI041: Method {0} is not supported by Websphere MQ Classes for Java
108=MQJI042: MQManagedConnection is not reusable
#
# Messages from MQManagedConnectionMetaData
#
109=IBM Websphere MQ
110=Command Level {0}
#
# Messages from MQXAi
#
111=Security manager prevented access to native methods library for MQ XA support.
112=Failed to load native methods library for MQ XA support.
#
# Messages from MQXAResource
#
113=xa_open failed
114=XA operation failed, see errorCode
115=XAResource closed
#
# Messages produced by com.ibm.mq.MQMsg2
#
116=MQJE063: Unsupported character set {0}
117=MQJE064: Unsupported version of MQMD structure {0}
118=MQJE065: Inconsistent internal state detected
# message when server doesn't support client XA
119=client connection not XA enabled
#
# Messages produced via SSL options on a client connection
#
120=MQJE066: {0} provided as an unsupported object type
121=MQJE067: Peer name {0} did not match requested name {1}
122=MQJE068: Server certificate has been revoked
123=MQJE069: Unable to contact CertStore
124=MQJE070: SSL Protocol error: Channel not configured for SSL?

The correct answer here is that the customers need to download the actual WMQ client and install it with your application and that your application should be distributed without the WMQ client jars. This is true for a variety of reasons:
You are not authorized to redistribute the vendor's code. When WMQ client or server code is bundled in an OEM license, it is the full install and not just a few jar files.
One of the reasons for installing the full client is to get support. The full client install is free and includes a number of diagnostic facilities. The only reason not to require a full install is convenience and if you are charging money for the app any convenience is far outweighed by potential liability.
Maintenance. If the customer installs the WMQ client then they can apply periodic maintenance from IBM and know that it is a complete and intact distribution. Unless your company is prepared to guarantee that the components they choose to redistribute are a valid installation, you are better off using the official install. Given the nature of the question is asking this community what the valid non-vendor installation is, I don't see how your company can stand behind this configuration of IBM's code.
Regulatory compliance. If any of your customers are subject to PCI, SOX, HIPAA or government regulations, certification cannot be achieved under this configuration. For example, to be PCI compliant the deployed application must have applied Fix Pack 7.0.1.3 quite some time ago. Without the full install, it is impossible to apply the fix pack or prove to an auditor it has been installed based on the jars.
Security. If you are not using SSL or an exit than it is possible for your customers to impersonate one another or the WMQ administrative account. In order to be using SSL you would have to have installed the IBM JCCE provider which comes in GSKit, which is part of the full client install and contains native compiled C code ot available in jar files. Based on the description of the app, I'm guessing any of your customers can remotely, anonymously administer your QMgr and execute arbitrary OS code. Not good.
So the correct answer is for you and your clients to go download and install the full WMQ client (SupportPac MQC7) and configure the app to run on that. Anything less places your clients and your company at risk.
And if you have any doubt about the security exposures I mentioned, please refer to the IMPACT Hardening WebSphere MQ presentation and the WebSphere MQ Hands-on Security Lab posted here.

Related

JMS message listener invoker failed, Cause: Identifier contains invalid JMS identifier character '-': 'x-request-id'

I'm working with JMS and queues (Azure queues) for the first time. I'm required to make a queue where Rubi server would write some data and Java would read it from queue and will do further executions.
This process is working fine locally on my machine. I've created a REST endpoint which is writing data in the queue and once data is written in the queue, the listener would take over and read the data and execute.
When we deploy it to Azure the error I can see in logs which is not letting the Queues start is
Setup of JMS message listener invoker failed for destination 'queue' - trying to recover. Cause: Identifier contains invalid JMS identifier character '-': 'x-request-id'
Zipkin is also present on the Azure server as a distributed tracing system and I guess this x-request-id is related to Zipkin which is creating the problem. I've searched Google for the issue but couldn't understand why its happening.
Following is detailed error message:
[36mc.m.s.l.NextGenRequestLoggingFilter [0;39m [2m:[0;39m
Before request [uri=/services/deal-service/api/v2/deals/ack;headers=
[x-request-id:"2d8d86d7-4fbf-9db6-8e95-28813f21a85c",
x-envoy-internal:"true", x-b3-parentspanid:"a209cdc649b0b890", content-
length:"575", x-forwarded-proto:"http", postman-token:"ad074595-
76a5-474b-9711-7e071b12b3b0", x-b3-sampled:"1", x-forwarded-
for:"10.244.2.1", accept:"*/*",
authorization: "some-token-YJc4tg--34jPRziJNSACqNQ", x-b3-
traceid:"6b40ff22781be67ba209cdc649b0b890", x-b3-
spanid:"702684ddb62cfe6b",
host:"portal-gateway.52.228.65.225.nip.io",
cache-control:"no-cache", accept-encoding:"gzip, deflate, br",
user-agent:"PostmanRuntime/7.22.0",
Content-Type:"application/xml;charset=UTF-8"]]
2020-02-18T15:19:34.197666458Z [2m2020-02-18 15:19:34.197[0;39m .
[32mDEBUG
[,6b40ff22781be67ba209cdc649b0b890,702684ddb62cfe6b,true][0;39m .
[35m9[0;39m [2m---[0;39m [2m[ XNIO-1 task-15][0;39m
Section 3.5.1 of the JMS 2 specification states this about message properties:
Property names must obey the rules for a message selector identifier. See
Section 3.8 “Message selection” for more information.
In regards to identifiers, section 3.8.1.1 states, in part:
An identifier is an unlimited-length character sequence that must begin with a Java identifier start character; all following characters must be Java identifier part characters. An identifier start character is any character for which the method Character.isJavaIdentifierStart returns true. This includes '_' and '$'. An identifier part character is any character for which the method Character.isJavaIdentifierPart returns true.
If you pass the character - into either Character.isJavaIdentifierStart or Character.isJavaIdentifierPart the return value is false. In other words, the - character in the name of a message property violates the JMS specification and therefore will cause an error.
From the error message its obvious that you are using qpid JMS client for communication through queues.
qpid client won’t allow any keys which violates java variable naming convention e.g. you won’t be able to send x-request-id in a queue’s header
which qpid jms client is consuming as it’ll throw error.
You need to take care of istio/zipkin to not to add certain headers (id you don’t need them actually) with the queue when its trying to communicate on azure bus.
So you have to disable the istio/zipkin libraries to intercept the request for queues so that request to/from queue can be made without headers. This will fix the issue.
Details of error (Java stack trace) would be really useful here.
By error message I assume, you are using qpid JMS client, that is performing check of message properties' names. These names can contain only characters, that are valid Java identifier characters.
In string 'queue-name' there is a '-' character, that is not Java identifier. To fix, you need to change 'queue-name' into something with valid characters, for example 'queue_name' (with underscore), or 'queueName' (camel case).

IBM MQ call failed with compcode '2' ('MQCC_FAILED') reason '2035' ('MQRC_NOT_AUTHORIZED')

I'm having a hard time trying to connect to my institution's MQ v9.
I was provided connectivity info by the MQ team:-
String hostName = '...'
int port = ...
String queueManager = '...'
String channel = '...'
String userId = 'ABC123'
String password = '...'
Given the following code...
JmsConnectionFactory cf = JmsFactoryFactory.
getInstance(WMQConstants.WMQ_PROVIDER).
createConnectionFactory()
cf.setStringProperty(WMQConstants.WMQ_HOST_NAME, hostName)
cf.setIntProperty(WMQConstants.WMQ_PORT, port)
cf.setStringProperty(WMQConstants.WMQ_CHANNEL, channel)
cf.setIntProperty(WMQConstants.WMQ_CONNECTION_MODE, WMQConstants.WMQ_CM_CLIENT)
cf.setStringProperty(WMQConstants.WMQ_QUEUE_MANAGER, queueManager)
cf.setStringProperty(WMQConstants.USERID, userId)
cf.setStringProperty(WMQConstants.PASSWORD, password)
// tried with both `true` and `false`... same error
cf.setBooleanProperty(WMQConstants.USER_AUTHENTICATION_MQCSP, true)
Connection connection = cf.createConnection()
connection.start()
connection.close()
... I'm getting this error:-
Exception in thread "main" com.ibm.msg.client.jms.DetailedJMSSecurityException:
JMSWMQ2013: The security authentication was not valid
that was supplied for queue manager '...' with connection
mode 'Client' and host name '...'.
Please check if the supplied username and password
are correct on the queue manager to which you are
connecting.
Caused by: com.ibm.mq.MQException: JMSCMQ0001: IBM
MQ call failed with compcode '2' ('MQCC_FAILED') reason
'2035' ('MQRC_NOT_AUTHORIZED').
I was told by the MQ team that the log has something like this:-
----- amqzfuca.c : 4527 -------------------------------------------------------
04/17/2019 10:32:20 AM - Process(10468.40757) User(...) Program(...)
Host(...) Installation(Installation1)
VRMF(9.1.0.1) QMgr(...)
Time(2019-04-17T15:32:20.542Z)
RemoteHost(...)
CommentInsert1(...)
CommentInsert2(...)
CommentInsert3(CLNTUSER(XYZ) ADDRESS(...))
AMQ9777E: Channel was blocked
EXPLANATION:
The inbound channel '...' was blocked from address '...'
because the active values of the channel matched a record
configured with USERSRC(NOACCESS). The active values of the channel were
'CLNTUSER(XYZ) ADDRESS(...)'.
... and it fails because it's using the wrong credential to connect.
While I passed in a different credential (user ID: ABC123), the MQ log sees the user ID I used to log into my machine (user ID: XYZ).
Why does the credential I explicitly passed in get omitted? How do I fix this?
I'm using this dependency:
<dependency>
<groupId>com.ibm.mq</groupId>
<artifactId>com.ibm.mq.allclient</artifactId>
<version>9.1.2.0</version>
</dependency>
I'm not using IBM JRE... more precisely, I'm using Oracle JDK 1.8 on my Mac, if that helps.
Thank you.
UPDATE 2019-04-22
I was able to get more accurate logs from MQ team now since I'm trying too many things at the same time.
If I set USER_AUTHENTICATION_MQCSP to true, then my machine's user ID (XYZ) is passed in.
If I set USER_AUTHENTICATION_MQCSP to false, then I'm getting different error message now:-
04/22/2019 01:19:49 PM - Process(1147099.9759) User(...) Program(...)
Host(rofesb911a) Installation(Installation1)
VRMF(9.1.0.1) QMgr(...)
Time(2019-04-22T18:19:49.323Z)
RemoteHost(...)
CommentInsert1(wa03598)
CommentInsert2(REQUIRED)
CommentInsert3(MCAUSER(ABC123) CLNTUSER(ABC123) ADDRESS(...))
AMQ9790I: The failed authentication check was caused by a CHLAUTH record with
CHCKCLNT(REQUIRED).
EXPLANATION:
The user ID 'ABC123' and its password were checked because the inbound
connection matched a channel authentication record with CHCKCLNT(REQUIRED).
The active values of the channel were 'MCAUSER(ABC123) CLNTUSER(ABC123)
ADDRESS(...)'. The MATCH(RUNCHECK) mode of the DISPLAY CHLAUTH
MQSC command can be used to identify the relevant CHLAUTH record.
The good news is it's seeing the correct user ID (ABC123), but I was told the password is invalid. I don't believe it was a password problem because I was able to use that same credential to access other protected web services.
Your MQ team have given you the credentials to use (i.e. user id and password) so I would assume that they have turned on user id and password checking on the queue manager.
Queue Manager is not using ADOPTCTX(YES)
ADOPTCTX(YES) is a setting on the queue manager that indicates that once user id and password have been verified as being correct, the user id (in your case "ABC123") should then be used for all further security checks (e.g. am I allowed to use this queue). If this setting is NO, then after the password validation is complete, it will actually use the client machine logged on user id which is also sent up to the queue manager (in your case "XYZ"). It seems likely that this is the case on your queue manager.
USER_AUTHENTICATION_MQCSP mode
There are actually two ways that a user id and password can be sent from a Java client application to the queue manager.
The first (decades old) was in use prior to IBM MQ V8, and used a restricted length flow (12 chars max each field) that was originally there for DOS SNA clients to send the two fields to the QMgr. This flow across the network was also the way the client logged on user id got sent to the queue manager, and so only one could be sent.
The newer one, added in MQ V6, and made more useful with password checking natively in the queue manager in MQ V8, was the MQCSP structure, which allowed for variable length user id and password fields. This was a different network flow from the one that sends the client logged on user id, and so both are sent to the queue manager.
When you set USER_AUTHENTICATION_MQCSP to true, you were telling the Java client to use the second mode. This gives the opportunity to be tripped up by the ADOPTCTX(NO) setting. If you set it to false, the only user id that makes it to the queue manager is ABC123 (in your example) and will likely give you a different, perhaps successful result.
Try your application with USER_AUTHENTICATION_MQCSP set to false, and when it works, advise your MQ team that they should use ADOPTCTX(YES) which is also now the default value, then you can switch back to USER_AUTHENTICATION_MQCSP set to true.

Default thread pool in tomcat of a spring boot application

In a spring boot application or in generally, does tomcat has a default thread pool configured?
If we do not configure anything, the tomcat will initiate new threads for each request and the thread gets destroy once the request finish?
And if a thread pool configured, particular thread would serve many requests when ever container pick that thread from pool?
Here is the configs of the embed Tomcat in springboot
server.tomcat.accept-count=100 # Maximum queue length for incoming connection requests when all possible request processing threads are in use.
server.tomcat.accesslog.buffered=true # Whether to buffer output such that it is flushed only periodically.
server.tomcat.accesslog.directory=logs # Directory in which log files are created. Can be absolute or relative to the Tomcat base dir.
server.tomcat.accesslog.enabled=false # Enable access log.
server.tomcat.accesslog.file-date-format=.yyyy-MM-dd # Date format to place in the log file name.
server.tomcat.accesslog.pattern=common # Format pattern for access logs.
server.tomcat.accesslog.prefix=access_log # Log file name prefix.
server.tomcat.accesslog.rename-on-rotate=false # Whether to defer inclusion of the date stamp in the file name until rotate time.
server.tomcat.accesslog.request-attributes-enabled=false # Set request attributes for the IP address, Hostname, protocol, and port used for the request.
server.tomcat.accesslog.rotate=true # Whether to enable access log rotation.
server.tomcat.accesslog.suffix=.log # Log file name suffix.
server.tomcat.additional-tld-skip-patterns= # Comma-separated list of additional patterns that match jars to ignore for TLD scanning.
server.tomcat.background-processor-delay=10s # Delay between the invocation of backgroundProcess methods. If a duration suffix is not specified, seconds will be used.
server.tomcat.basedir= # Tomcat base directory. If not specified, a temporary directory is used.
server.tomcat.internal-proxies=10\\.\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}|\\
192\\.168\\.\\d{1,3}\\.\\d{1,3}|\\
169\\.254\\.\\d{1,3}\\.\\d{1,3}|\\
127\\.\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}|\\
172\\.1[6-9]{1}\\.\\d{1,3}\\.\\d{1,3}|\\
172\\.2[0-9]{1}\\.\\d{1,3}\\.\\d{1,3}|\\
172\\.3[0-1]{1}\\.\\d{1,3}\\.\\d{1,3}\\
0:0:0:0:0:0:0:1\\
::1 # Regular expression that matches proxies that are to be trusted.
server.tomcat.max-connections=10000 # Maximum number of connections that the server accepts and processes at any given time.
server.tomcat.max-http-header-size=0 # Maximum size in bytes of the HTTP message header.
server.tomcat.max-http-post-size=2097152 # Maximum size in bytes of the HTTP post content.
server.tomcat.max-threads=200 # Maximum amount of worker threads.
server.tomcat.min-spare-threads=10 # Minimum amount of worker threads.
server.tomcat.port-header=X-Forwarded-Port # Name of the HTTP header used to override the original port value.
server.tomcat.protocol-header= # Header that holds the incoming protocol, usually named "X-Forwarded-Proto".
server.tomcat.protocol-header-https-value=https # Value of the protocol header indicating whether the incoming request uses SSL.
server.tomcat.redirect-context-root=true # Whether requests to the context root should be redirected by appending a / to the path.
server.tomcat.remote-ip-header= # Name of the HTTP header from which the remote IP is extracted. For instance, `X-FORWARDED-FOR`.
server.tomcat.resource.cache-ttl= # Time-to-live of the static resource cache.
server.tomcat.uri-encoding=UTF-8 # Character encoding to use to decode the URI.
server.tomcat.use-relative-redirects= # Whether HTTP 1.1 and later location headers generated by a call to sendRedirect will use relative or absolute redirects.
As you can see for the default value, the minimum amount of worker threads is 10, and the maximum amount of worker threads is 200, and the maximum queue length for incoming connection requests when all possible request processing threads are in use is 100.
Yes spring boot uses Embeded tomcat server, you can modify some of its configs in application.yml or application.properties By default it has 200 threads spring-docs
# EMBEDDED SERVER CONFIGURATION (ServerProperties)
server.port=8080
server.address= # bind to a specific NIC
server.session-timeout= # session timeout in seconds
server.context-path= # the context path, defaults to '/'
server.servlet-path= # the servlet path, defaults to '/'
server.tomcat.access-log-pattern= # log pattern of the access log
server.tomcat.access-log-enabled=false # is access logging enabled
server.tomcat.protocol-header=x-forwarded-proto # ssl forward headers
server.tomcat.remote-ip-header=x-forwarded-for
server.tomcat.basedir=/tmp # base dir (usually not needed, defaults to tmp)
server.tomcat.background-processor-delay=30; # in seconds
server.tomcat.max-threads = 0 # number of threads in protocol handler
server.tomcat.uri-encoding = UTF-8 # character encoding to use for URL decoding

Can we append message of Gmail to Yahoo using Java Mail API

We want to write a program using Java Mail-1.5 that can append the message of Gmail to Yahoo account. Is this possible with Java Mail API? What we are trying is that we have downloaded eml of Gmail message at our system, now we want to append these messages to one Yahoo account. On trying this we are getting this errors -
javax.mail.MessagingException: A4 NO [SERVERBUG] LIST Server error - Please try again later;
nested exception is: com.sun.mail.iap.CommandFailedException: A4 NO [SERVERBUG] LIST Server error - Please try again later
at com.sun.mail.iap.Protocol.handleResult(Protocol.java:351)
at com.sun.mail.imap.protocol.IMAPProtocol.doList(IMAPProtocol.java:1100)
at com.sun.mail.imap.protocol.IMAPProtocol.list(IMAPProtocol.java:1046)
at com.sun.mail.imap.IMAPFolder$1.doCommand(IMAPFolder.java:518)
at com.sun.mail.imap.IMAPFolder.doProtocolCommand(IMAPFolder.java:2991)
at com.sun.mail.imap.IMAPFolder.doCommand(IMAPFolder.java:2931)
... 8 more
Sometime this process producing similar errors as some are below listed -
A33 NO Invalid "flag-list" : Contains unsupported flag or other invalid item.
A18 BAD [CLIENTBUG] APPEND Command arguments invalid
A19 BAD [CLIENTBUG] APPEND Command arguments invalid
A21 NO [SERVERBUG] LIST Server error - Please try again later
A12 BAD [CLIENTBUG] APPEND Command arguments invalid
.............
............
Is this possible to append message of Gmail (eml) to Yahoo account using Java Mail API??
Thanks,
Neelam Sharma
This is really a question about what the Yahoo mail server supports. As you've discovered, the Yahoo mail server sometimes fails operations due to a "server bug". It may take a few retries to get the operation to work.
The APPEND failure seems to be due to the server limiting what flags you can set when you append the message. The debug output will show you what flags it's trying to set. You might need to clear those flags in the message before appending the message.
See the JavaMail populate.java demo program, and in particular the dontPreserveFlags option in that program.

RabbitMQ AMQP.BasicProperties.Builder values

In the RabbitMQ/AMQP Java client, you can create an AMQP.BasicProperties.Builder, and use it to build() an instance of AMQP.BasicProperties. This built properties instance can then be used for all sorts of important things. There are lots of "builder"-style methods available on this builder class:
BasicProperties.Builder propsBuilder = new BasicProperties.Builder();
propsBuilder
.appId(???)
.clusterId(???)
.contentEncoding(???)
.contentType(???)
.correlationId(???)
.deliveryMode(2)
.expiration(???)
.headers(???)
.messageId(???)
.priority(???)
.replyTo(???)
.timestamp(???)
.type(???)
.userId(???);
I'm looking for what fields these builer methods help "build-up", and most importantly, what valid values exist for each field. For instance, what is a clusterId, and what are its valid values? What is type, and what are its valid values? Etc.
I have spent all morning scouring:
The Java client documentation; and
The Javadocs; and
The RabbitMQ full reference guide; and
The AMQP specification
In all these docs, I cannot find clear definitions (besides some vague explanation of what priority, contentEncoding and deliveryMode are) of what each of these fields are, and what their valid values are. Does anybody know? More importantly, does anybody know where these are even documented? Thanks in advance!
Usually I use very simple approach to memorize something. I will provide all details below, but here is a simple picture of BasicProperties field and values. I've also tried to properly highlight queue/server and application context.
If you want me to enhance it a bit - just drop a small comment. What I really want is to provide some visual key and simplify understanding.
High-level description (source 1, source 2):
Please note Clust ID has been deprecated, so I will exclude it.
Application ID - Identifier of the application that produced the message.
Context: application use
Value: Can be any string.
Content Encoding - Message content encoding
Context: application use
Value: MIME content encoding (e.g. gzip)
Content Type - Message content type
Context: application use
Value: MIME content type (e.g. application/json)
Correlation ID - Message correlated to this one, e.g. what request this message is a reply to. Applications are encouraged to use this attribute instead of putting this information into the message payload.
Context: application use
Value: any value
Delivery mode - Should the message be persisted to disk?
Context: queue implementation use
Value: non-persistent (1) or persistent (2)
Expiration - Expiration time after which the message will be deleted. The value of the expiration field describes the TTL period in milliseconds. Please see details below.
Context: queue implementation use
Headers - Arbitrary application-specific message headers.
Context: application use
Message ID - Message identifier as a string. If applications need to identify messages, it is recommended that they use this attribute instead of putting it into the message payload.
Context: application use
Value: any value
Priority - Message priority.
Context: queue implementation use
Values: 0 to 9
ReplyTo - Queue name other apps should send the response to. Commonly used to name a reply queue (or any other identifier that helps a consumer application to direct its response). Applications are encouraged to use this attribute instead of putting this information into the message payload.
Context: application use
Value: any value
Time-stamp - Timestamp of the moment when message was sent.
Context: application use
Value: Seconds since the Epoch.
Type - Message type, e.g. what type of event or command this message represents. Recommended to be used by applications instead of including this information into the message payload.
Context: application use
Value: Can be any string.
User ID - Optional user ID. Verified by RabbitMQ against the actual connection username.
Context: queue implementation use
Value: Should be authenticated user.
BTW, I've finally managed to review latest sever code (rabbitmq-server-3.1.5), there is an example in rabbit_stomp_test_util.erl:
content_type = <<"text/plain">>,
content_encoding = <<"UTF-8">>,
delivery_mode = 2,
priority = 1,
correlation_id = <<"123">>,
reply_to = <<"something">>,
expiration = <<"my-expiration">>,
message_id = <<"M123">>,
timestamp = 123456,
type = <<"freshly-squeezed">>,
user_id = <<"joe">>,
app_id = <<"joe's app">>,
headers = [{<<"str">>, longstr, <<"foo">>},
{<<"int">>, longstr, <<"123">>}]
Good to know somebody wants to know all the details. Because it is much better to use well-known message attributes when possible instead of placing information in the message body. BTW, basic message properties are far from being clear and useful. I would say it is better to use a custom one.
Good example (source)
Update - Expiration field
Important note: expiration belongs to queue context. So message might be dropped by the servers.
README says the following:
expiration is a shortstr; since RabbitMQ will expect this to be
an encoded string, we translate a ttl to the string representation
of its integer value.
Sources:
Additional source 1
Additional source 2
At time of writing:
The latest AMQP standard is AMQP 1.0 OASIS Standard.
The latest version of RabbitMQ is 3.1.5 (server and client), which claims to support AMQP 0.9.1 (pdf and XML schemas zipped).
RabbitMQ provides it's own description of the protocol as XML schema including extensions (i.e. non-standard), plus XML schema without extensions (which is identical to the schema linked via (2)) and pdf doc.
In this answer:
links in (3) are the primary source of detail
(2) pdf doc is used as secondary detail if (3) is inadequate
The source code (java client, erlang server) is used as tertiary detail if (2) is inadequate.
(1) is generally not used - the protocol and schema have been (fairly) significantly evolved for/by OASIS and should apply to future versions of RabbitMQ, but do not apply now. The two exceptions where (1) was used was for textual descriptions of contentType and contentEncoding - which is safe, because these are standard fields with good descriptions in AMQP 1.0.
The following text is paraphrased from these sources by me to make a little more concise or clear.
content-type (AMQP XML type="shortstr"; java type="String"): Optional. The RFC-2046 MIME type for the message’s application-data section (body). Can contain a charset parameter defining the character encoding used: e.g., ’text/plain; charset=“utf-8”’. Where the content type is unknown the content-type SHOULD NOT be set, allowing the recipient to determine the actual type. Where the section is known to be truly opaque binary data, the content-type SHOULD be set to application/octet-stream.
content-encoding (AMQP XML type="shortstr"; java type="String"): Optional. When present, describes additional content encodings applied to the application-data, and thus what decoding mechanisms need to be applied in order to obtain the media-type referenced by the content-type header field. Primarily used to allow a document to be compressed without losing the identity of its underlying content type. A modifier to the content-type, interpreted as per section 3.5 of RFC 2616. Valid content-encodings are registered at IANA. Implementations SHOULD NOT use the compress encoding, except as to remain compatible with messages originally sent with other protocols, e.g. HTTP or SMTP. Implementations SHOULD NOT specify multiple content-encoding values except as to be compatible with messages originally sent with other protocols, e.g. HTTP or SMTP.
headers (AMQP XML type="table"; java type="Map"): Optional. An application-specified list of header parameters and their values. These may be setup for application-only use. Additionally, it is possible to create queues with "Header Exchange Type" - when the queue is created, it is given a series of header property names to match, each with optional values to be matched, so that routing to this queue occurs via header-matching.
deliveryMode (RabbitMQ XML type="octet"; java type="Integer"): 1 (non-persistent) or 2 (persistent). Only works for queues that implement persistence. A persistent message is held securely on disk and guaranteed to be delivered
even if there is a serious network failure, server crash, overflow etc.
priority (AMQP XML type="octet"; java type="Integer"): The relative message priority (0 to 9). A high priority message is [MAY BE?? - GB] sent ahead of lower priority messages waiting in the same message queue. When messages must be discarded in order to maintain a specific service quality level the server will first discard low-priority messages. Only works for queues that implement priorities.
correlation-id (AMQP XML type="octet"; java type="String"): Optional. For application use, no formal (RabbitMQ) behaviour. A client-specific id that can be used to mark or identify messages between clients.
replyTo (AMQP XML type="shortstr"; java type="String"): Optional. For application use, no formal (RabbitMQ) behaviour but may hold the name of a private response queue, when used in request messages. The address of the node to send replies to.
expiration (AMQP XML type="shortstr"; java type="String"): Optional. RabbitMQ AMQP 0.9.1 schema from (3) states "For implementation use, no formal behaviour". The AMQP 0.9.1 schema pdf from (2) states an absolute time when this message is considered to be expired. However, both these descriptions must be ignored because this TTL link and the client/server code indicate the following is true. From the client, expiration is only be populated via custom application initialisation of BasicProperties. At the server, this is used to determine TTL from the point the message is received at the server, prior to queuing. The server selects TTL as the minimum of (1) message TTL (client BasicProperties expiration as a relative time in milliseconds) and (2) queue TTL (configured x-message-ttl in milliseconds). Format: string quoted integer representing number of milliseconds; time of expiry from message being received at server.
message-id (AMQP XML type="shortstr"; java type="String"): Optional. For application use, no formal (RabbitMQ) behaviour. If set, the message producer should set it to a globally unique value. In future (AMQP 1.0), a broker MAY discard a message as a duplicate if the value of the message-id matches that of a previously received message sent to the same node.
timestamp (AMQP XML type="timestamp"; java type="java.util.Date"): Optional. For application use, no formal (RabbitMQ) behaviour. An absolute time when this message was created.
type (AMQP XML type="shortstr"; java type="String"): Optional. For application use, no formal (RabbitMQ) behaviour. [Describes the message as being of / belonging to an application-specific "type" or "form" or "business transaction" - GB]
userId (AMQP XML type="shortstr"; java type="String"): Optional. XML Schema states "For application use, no formal (RabbitMQ) behaviour" - but I believe this has changed in the latest release (read on). If set, the client sets this value as identity of the user responsible for producing the message. From RabbitMQ: If this property is set by a publisher, its value must be equal to the name of the user used to open the connection (i.e. validation occurs to ensure it is the connected/authenticated user). If the user-id property is not set, the publisher's identity remains private.
appId (RabbitMQ XML type="shortstr"; java type="String"): Optional. For application use, no formal (RabbitMQ) behaviour. The creating application id. Can be populated by producers and read by consumers. (Looking at R-MQ server code, this is not used at all by the server, although the "webmachine-wrapper" plugin provides a script and matching templates to create a webmachine - where an admin can provide an appId to the script.)
cluster Id (RabbitMQ XML type="N/A"; java type="String"): Deprecated in AMQP 0.9.1 - i.e. not used. In previous versions, was the intra-cluster routing identifier, for use by cluster applications, which should not be used by client applications (i.e. not populated). However, this has been deprecated and removed from the current schema and is not used by R-MQ server code.
As you can see above, the vast majority of these properties do not have enumerated / constrained / recommended values because they are "application use only" and are not used by RabbitMQ. So you have an easy job. You're free to write/read values that are useful to your application - as long as they match datatype and compile :). ContentType and contentEncoding are as per standard HTTP use. DeliveryMode and priority are constrained numbers.
Note: Useful, but simple constants for AMQP.BasicProperties are available in class MessageProperties.
Cheers :)
UPDATE TO POST:
With many thanks to Renat (see comments), have looked at erlang server code in rabbit_amqqueue_process.erl and documentation at RabbitMQ TTL Extensions to AMQP. Message expiration (time-to-live) can be specified
per queue via:
Map<String, Object> args = new HashMap<String, Object>();
args.put("x-message-ttl", 60000);
channel.queueDeclare("myqueue", false, false, false, args);
or per message via:
byte[] messageBodyBytes = "Hello, world!".getBytes();
AMQP.BasicProperties properties = new AMQP.BasicProperties();
properties.setExpiration("60000");
channel.basicPublish("my-exchange", "routing-key", properties, messageBodyBytes);
Here, the ttl/expiration is in millisecs, so 60 sec in each case.
Have updated above definition of expiration to reflect this.
The AMQP spec defines a generic, extensible model for properties.
AMQP properties are somewhat similar in concept to HTTP headers, in that they represent metadata about the messages in question. Just as in HTTP, they are framed separately to the message payload. But they are basically a key/value map.
Some brokers like RabbitMQ will interpret certain message properties like expiration to add extra vendor-specific value (in that case, enforcing a TTL).
But in the end, AMQP properties are just a big bunch of key/value pairs that get safely sent along with each message, should you choose to do so. Your AMQP broker's documentation will tell you which ones they interpret specially and how to send your own ones.
All that being said, if you're asking this question in the first place then you probably don't need to worry about them at all. You will be successfully able to send messages without having to worry about setting any message properties at all.

Categories