We Keep Coding

Getting the junk values in the URL - RESTAssured

I'm automating my GET API with RESTAssured, while I see the URI just before hitting the API, it is just normal, but when I add the query parameters, it adds junk values. I don't understand why adding query params does so, even when I print query params, it gives me proper value.
Here is my code snippet.
System.out.println("QUERY PARAMS"+queryParams);
System.out.println(">>>>"+uri);
Response response = RestAssured.given().config(RestAssured.config().sslConfig(this.getSslConfig())).filter(new AllureRestAssured()).contentType(appJsonContentType)
.headers(requestHeader)
.queryParams(queryParams)
.when()
.log()
.all().get(uri);
And here is the logs -
QUERY PARAMS{amount=[2000.0], currency=[USD], date=[2022-04-01T00:00:00Z], id=[413275]}
>>>>https://www.example.com/amount/21345699
Request method: GET
Request URI: https://www.example.com/amount/21345699?amount=2000.0&currency=USD&date=2022-04-01T00%3A00%3A00Z&id=413275
I'm not sure why I'm getting this %3A in my URL. I tried
.all.get(URLEncoder.encode(uri, StandardCharsets.UTF_8));
but it adds some localhost:8080 ahead of the URL. Can someone please help?

These are URL encoded characters, as not all the character can be written directly to URL.
%3A is URL encoded sign ":" which I assume is separating fields of hour:minute:second in your parameters

How to suppress url encoding with spring boot

I have created a GET/POST API using Spring boot which has a http url parameter say refid. Now this parameter is already encoded before invoking GET/POST request
e.g. http://localhost:8080/users/TESTFNkJXiQAH%2FJBKxigBx
But, when I deploy this through Spring Boot, the encoded refid is encoded again and the refid changes. i.e. it becomes:
http://localhost:8080/users/TESTFNkJXiQAH%252FJBKxigBx
I want to suppress this 2nd encoding by Spring boot. Can anyone advise here?

Don't know if you are still having this problem or you found out why it's happening, but because I was trying to explain to someone the phenomenon, I looked if there is already a good explanation. But since you also ask and I didn't find any, here is my answer.
So you encode your refid
TESTFNkJXiQAH%2FJBKxigBx
before you send it through the url, which then you give into a browser. Now this is only the encoded refid. When you call it through a URL directly you have to encode it again, according to the HTML URL encoding standards. That’s why the double escape. Also read this. E.g. so if your refid looks like this
test%123
and you encode it you turn it into
test%25123
now if you also want to pass it through a url on the browser you'd have to encode it again.
test%2525123
But if a service A is using this service and service A encodes this refid properly then you wont have this problem. It's happening only because you are trying to call this api endpoint through the browser.
Of course I take for granted that you are doing this:
String decoded = URLDecoder.decode(refid, "UTF-8");
in your controller

Pass the decoded URL in first place instead of doing inconvenient things to stop double encoding.
You get already decoded field in rest controller.
Example if you pass www.xyz.com?name=nilesh%20salpe
you will get value of param name as "nilesh salpe" and not "nilesh%20salpe"

This is a basic example of URLDecoder:
#RequestMapping(value = "/users/{refId}", method = GET)
public void yourMethod(#PathVariable("refId") String refId) {
// This is what you get in Spring Boot
String encoded = refId; //"TESTFNkJXiQAH%252FJBKxigBx"
String decoded = URLDecoder.decode(encoded, "UTF-8");
System.out.println(decoded);
// Result TESTFNkJXiQAH%2FJBKxigBx
}

Url encoding issue with Jersey Client

I need to make a service call such as this:
http://myservice.com/path?var1=value1&var2=value2
The issue I have is value1 and value2 ends up getting encoded, and this makes the service call fail. For example, value1 is something like "a=b&b=c;2&&="... it contains special characters, basically.
I am guessing that this is an issue for the service to fix - to properly handle decoding encoded characters, which I do not think it is currently doing.
Here is a sample of how I am making these requests:
WebTarget target = client.target("http://test.com")
.path("path1")
.queryParam("var1", var1);
Builder builder = target.request();
...
What's puzzling to me is that if I make the same request just using Chrome, everything works. So that makes me to believe that I should have some way with the Jersey API of "disabling" the encoding.

Only way I have found so far to use "raw" Url is to use URI.
So call like this
URI uri = URI.create("http://localhost/~Common~0#/edit?vadf&&sfs&&fdsfd=fs&fsd");
WebTarget target = client.target(uri);
You get request url
1 > GET http://localhost/~Common~0#/edit?vadf&&sfs&&fdsfd=fs&fsd
Everything else I tried resulted in encoding special characters.

Server returned HTTP response code: 400

I am reading data from a webservice. The issue if I put the link on the browser it works fine. When I run like this give me error. I am suspecting is it due to the way how I send my parameters. My paramater list has this dID=1,5,7,11,14,18,26&FromDate=18 Sep 2012 00:00 am&ToDate=18 Sep 2012 10:00 am. Do I need to do some encoding here?
URL xmlURLDM = new URL(urlDM);
InputStream xml2 = xmlURLDM.openStream();
I get this error
java.io.IOException: Server returned HTTP response code: 400 for URL:
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1612)
at java.net.URL.openStream(URL.java:1035)
at xmlreader.main(xmlreader.java:172)

You do need encoding, most likley it is the spaces in your URL that is causing the trouble.
Use Javas built in url-encoding. eg:
String encoded = URLEncoder.encode(myUrl, "UTF-8");
...
call web service with encoded as URL
There can be other reasons for the status code being 400, but this encoding issue is probably your first stumbling block.

The Documentation of URL says,
The URL class does not itself encode or decode any URL components
according to the escaping mechanism defined in RFC2396. It is the
responsibility of the caller to encode any fields, which need to be
escaped prior to calling URL, and also to decode any escaped fields,
that are returned from URL. Furthermore, because URL has no knowledge
of URL escaping, it does not recognise equivalence between the encoded
or decoded form of the same URL.
So please use URLEncoder.encode() before you invoke URL()

Charset filter causing issue in parsing UTF-8 characters

I am using Spring MVC's charset filter. This is the URL that I use to invoke my servlet from my applet
http://192.168.0.67/MyServlet?p1=団
As you can see, the parameter has a unicode character 団. So I use
URLEncoder.encode("団", "UTF-8");
and now my URL becomes
http://192.168.0.67/MyServlet?p1=%E5%9B%A3
However, from the servlet, calling
request.getParameter("p1");
already return some gibberish that cannot be decoded with URLDecoder. BTW, invoking
URLDecoder.decode("%E5%9B%A3", "UTF-8");
does give me the original unicode character. It's just that the servlet has garbled the parameter before it can even be decoded. Does anyone know why? request.getParameter() doesn't decode parameter with UTF-8?

The Spring MVC's charset filter will only set the request body encoding, not the request URI encoding. You need to set the charset for the URI encoding in the servletcontainer configuration. Lot of servletcontainers default to ISO-8859-1 to decode the URI. It's unclear what servletcontainer you're using, so here's just an example for Tomcat: edit the <Connector> entry of /conf/server.xml to add URIEncoding="UTF-8":
<Connector ... URIEncoding="UTF-8">
If you can't edit the server's configuration for some reason (e.g. 3rd party hosting and such), then you should consider to use POST instead of GET:
String query = "p1=" + URLEncoder.encode("団", "UTF-8");
URLConnection connection = new URL(getCodeBase(), "MyServlet").openConnection();
connection.setDoOutput(true); // This sets request method to POST.
connection.getOutputStream().write(query.getBytes("UTF-8"));
// ...
This way you can in doPost() use ServletRequest#setCharacterEncoding() to tell the Servlet API what charset to use to parse the request body (or just rely on the Spring MVC's charset filter from doing this job):
request.setCharacterEncoding("UTF-8");
String p1 = request.getParameter("p1"); // You don't need to decode yourself!
// ...
See also:
Unicode - How to get the characters right?