We Keep Coding

Postman request encoding [closed]

Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 7 years ago.
Improve this question
I'm writing my own HTTP server. I'm using Java Socket for it. I read request from InputStream next way
val input = BufferedReader(InputStreamReader(socket.inputStream, "UTF-8"))
When I receive requests from curl or browser all is good. But when I receive request from Postman I get something like this:
��g�������#���=��g�������#���, ��9� ��3���5�/�=��9� ��3���5�/�,
I tried to use others encodings for InputStreamReader such as UTF-16 and ASCII. Which encoding is used for Postman requests and how I can read it on my server?
UPDATE: Sorry, this is my failure. I used HTTPS when making requests.

If the first two bytes received are ASCII 31 (0x1F) and 139 (0x8B) you are receiving a GZIP stream and for some reason convinced the client that you supported GZIP Content-Encoding (not the same thing as the charset encoding).
The data looks more binary than character encoding. Gzip, deflate, SSL, or other reason for binary data is what should be looked at.
You may not be doing content negotiation correctly and therefore receiving binary gzip or deflate. Or using HTTPS vs. HTTP. Or uploading an image. Something but not text.

NOTE: my other answer is more direct, this is related HTTP server libraries for use in Kotlin, in case others read this question and want to know alternatives. Not sure if this is an XY problem and if the author is not aware of other options.
Other ways if you do not want to build an HTTP library, but instead use one:
Raw HTTP frameworks:
Anything written for Java, and specifically:
Undertow.io
Vert.x 3 with Vert.x-web
Netty - very low level
Somewhat more than raw HTTP, but not a MVC or REST framework:
Wasabi - Kotlin specific
And for fuller REST or Web frameworks:
KTOR - Kotlin specific
Kovert - Kotlin specific, REST but adding views (Disclaimer: I'm the author)
Spark Java - Java, works nicely in Kotlin
Vert.x Nubes - Java, to work on top of Vert.x
Kikaha - Java, to work on top of Undertow

Android Studio how to connect my project with mySQL db (news feed app) [closed]

Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 7 years ago.
Improve this question
I have searched a lot for the answer but could not find it. Please help me understand how to connect my android app to mySQL.

Assuming you mean to have your Android application talk remotely to a server hosting your news feed source (articles, etc.) then you might want to have a look at creating a RESTful service (a basic example of how to achieve this can be found here).
You'll want to be familiar with:
SQL (http://www.w3schools.com/sql/)
JSON (http://www.w3schools.com/json/)
RESTful services
(http://www.drdobbs.com/web-development/restful-web-services-a-tutorial/240169069)
Basically what would happen is whenever your activity needs to display news feed results to the user, it can query your RESTful backend (via an http POST/GET request) and wait for your web service to respond. Your web service will listen for HTTP requests matching a specific URL, for example, mynewsfeed.com/get_articles, and upon getting this request, your web service will issue an SQL query against whatever database contains your articles and return the results in an HTTP response.
I'd personally recommend sending data back and for between the server/client using JSON (unless you have a reason for using another format.) This way, when your Android application gets the response from the server, it can parse the JSON data for various attributes contained in your article (title, author, content, etc.)
Google's GSON library has a neat way of converting JSON data into a Java object, so you don't have to worry about manually parsing each JSON field out yourself: https://sites.google.com/site/gson/gson-user-guide.
As for how to actually build the web service, there are tons of frameworks out there. I've personally used Flask (http://flask.pocoo.org/docs/0.10/tutorial/) in the past and find it very lightweight, straightforward and supported by pretty decent documentation. You'll also need somewhere to host the webservice and store the SQL database (for this I pay ~$30 a year for a simple hosting service and a domain.)
Hope this points you in the right direction (there is a lot of material here.) I'm happy to provide clarification if what I've posted isn't clear.

Java: http request, print headers and parse json

I have no knowledge of Java, but a collaborator is writing code and attempts to parse a JSON response that the server that I am writing is providing. I also created this question yesterday trying to see if there was a problem on my end. However, the problem still remains.
So, can someone give me some simple code of what needs to be written in Java in order to make an http request to a server (say http://www.google.com) print the headers of the response and moreover parse a json object that is returned? Or at least, which functions should I be using and which libraries I should include in the source code. I want to believe that I will figure it out after that point.
I am totally clueless how to do that and what sort of libraries I need in java. Honestly, I do not know java.

Your first step would be looking into the Servlet specification of the Java language. I would recommend the Google GSON library for parsing JSON.

Scrape data from HTML pages using Java, output to database [closed]

Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 7 years ago.
Improve this question
I need to know how to create a scraper (in Java) to gather data from HTML pages and output to a database...do not have a clue where to start so any information you can give me on this would be great. Also, you can't be too basic or simple here...thanks :)

First you need to get familiar with a HTML DOM parser in Java like JTidy. This will help you to extract the stuff you want from a HTML file. Once you have the essential stuff, you can use JDBC to put in the database.
It might be tempting to use regular expression for this job. But don't. HTML is not a regular language so regex are not the way to go.

I am running a scraper using JSoup I'm a noob yet found it to be very intuitive and easy to work with. It is also capable of parsing a wide range or sources html, XML, RSS, etc.
I experimented with htmlunit with little to no success.

i successfully used lobo browser API in a project that scraped HTML pages. the lobo browser project offers a browser but you can also use the API behind it very easily. it will also execute javascript and if that javascript manipulates the DOM, then that will also be reflected in the DOM when you investigate the DOM. so, in short, the API allows you mimic a browser, you can also work with cookies and stuff.
now for getting the data out of the HTML, i would first transform the HTML to valid XHTML. you can use jtidy for this. since XHTML is valid XML, you can use XPath to retrieve the data you want very easily. if you try to write code that parses the data from the raw HTML, your code will become a mess quickly. therefore i'd use XPath.
Once you have the data, you can insert it into a DB with JDBC or maybe use Hibernate if you want to avoid writing too much SQL

A HUGE percentage of websites are build on malformed HTML code. It is essential that you use something like HtmlCleaner to clean up the source code that you want to parse.
Then you can successfully use XPath to extract Nodes and Regex to parse specific part of the strings you extracted from the page.
At least this is the technique I used.
You can use the xHtml that is returned from HtmlCleaner as a sort of Interface between your Application and the remote Page you're trying to parse. You should test against this and in the case the remote page changes you just have to extract the new xHtml cleaned by HtmlCleaner, re-adapt the XPath Queries to extract what you need and re-test your Application code against the new Interface.
In the case you want to create a MultiThreaded 'scraper' be aware that HtmlCleaner is not Thread Safe (refer my post here).
This post can give you an idea of how to parse a correctly formatted xHtml using XPath.
Good Luck! ;)
note: at the time I implemented my Scraper, HtmlCleaner did a better job in normalizing the pages I wanted to parse. In some cases jTidy was failing in doing the same job so I'd suggest you to give it a try

Using JTidy you can scrap data from HTML. Then yoou can use JDBC.

Easiest frameworks to implement Java REST web services [closed]

Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
Questions asking us to recommend or find a tool, library or favorite off-site resource are off-topic for Stack Overflow as they tend to attract opinionated answers and spam. Instead, describe the problem and what has been done so far to solve it.
Closed 9 years ago.
Improve this question
What are the best frameworks for implementing both client and server REST frameworks in Java? I've been struggling a little to find an easy to use solution.
Update: Both Jersey and Restlet seem like good options. We'll probably use Restlet but we'll experiment with both.

Jersey is really easy for both. To write web services, you use annotations:
#Path("/helloworld")
public class HelloWorldResource {
// The Java method will process HTTP GET requests
#GET
// The Java method will produce content identified by the MIME Media
// type "text/plain"
#Produces("text/plain")
public String helloWorld() {
// Return some cliched textual content
return "Hello World";
}
}
For a client:
Client client = Client.create();
WebResource webResource = client.resource("http://localhost:8080/helloworld");
String s = webResource.get(String.class);
System.out.println(s); // prints Hello World

Restlet sounds like it should provide what you're looking for:
Support for client and server (in a relatively symmetric api)
Smart url binding
mime type understanding (given accepted mime types, it will ask your resources for their representation in that type)
Supports JAX-RS annotations (just like Jersey)

Take a look at dropwizard too.

Restlet also support annotations in its 2.0 version, both on the client and server-side. The JAX-RS API is also supported as an extension.
Here is a simple example for server-side:
public class HelloWorldResource extends ServerResource {
#Get
public String represent() {
return "hello, world";
}
}
On the client-side:
// Outputting the content of a Web page
new ClientResource("http://www.restlet.org").get().write(System.out);
For further documentation, check this page.

There's JBoss' new RESTEasy library. It appears to be under rapid development since its initial launch. I've no idea if it's any good; it's on my 'check it out' list.

I haven't used it personally but some teams that I work with are using Spring 3 MVC. REST in Spring 3: #MVC looks like a good blog post overview. The RESTful features include "URI Templates", "Content Negotiation", "HTTP Method Conversion", "ETag support" and more.
Edit: Also, see this question: Can anyone recommend a Java web framework that is based on MVC and supports REST ?

You could take a look at the CXF JAX-RS implementation. For complete list of its features check the CXF web site for JAX-RS.
The community behind the project seems to be very active (July 2013). An indication of that is the number of messages per day in the CXF mailing lists.

I can recommend Apache wink, a new framework still in incubation mode, but very mature and high quality.
http://incubator.apache.org/wink/
It implements the JAX-RS specification, it has both client & server framework for REST development.
Apache is standing behind this project - that's always a good sign (and a good license :-) )
What I love most about this framework is the intuitive integration with Spring, it's very useful if you want your framework to be easily configured and extended.

UPDATE: Xydra Restless is not longer maintained +++ If your are using Goolge AppEngine before they release a "reserve instance" feature, you might consider Xydra Restless which has few features but loads fast.

My favourite is Spring MVC, you have support for both, client and server side... And you have Android support too =)
For example, you can see a example of Spring Android here