We Keep Coding

Read large data from database using JdbcTemplate and expose via api?

I have a requirement to read a large data set from a postgres database which needs to be accessible via a rest api endpoint. The client consuming the data will then need to transform the data into csv format(might need to support json and xml later on).
On the server side we are using Spring Boot v2.1.6.RELEASE and spring-jdbc v5.1.8.RELEASE.
I tried using paging and loop through all the pages and store the result into a list and return the list but resulted in OutOfMemory error as the data set does not fit into memory.
Streaming the large data set looks like a good way to handle memory limits.
Is there any way that I can just return a Stream of all the database entities and also have the rest api return the same to the client? How would the client deserialize this stream?
Are there any other alternatives other than this?

If your data is so huge that it doesn't fit into memory - I'm thinking gigabytes or more - then it's probably too big to reasonably provide as single HTTP response. You will hold the connection open for a very long time. If you have a problem mid-way through, the client will need to start all over at the beginning, potentially minutes ago.
A more user-friendly API would introduce pagination. Your caller could specify a page size and the index of the page to fetch as part of their request
For example
/my-api/some-collection?size=100&page=50
This would represent fetching 100 items, starting from the 5000th (5000 - 5100)
Perhaps you could place some reasonable constraints on the page size based on what you are able to load into memory at one time.

How to send large json data to the spring controller

How can i send large volume of json data to a spring controller. Say, i have large json data of about 100k or 1000k records which i need to send to my rest controller in spring or springboot then what is the best/most efficient approach to solve the problem.
I know that the data can be sent using request body but i think sending such a large volume of data in the request body of a REST api is not efficient. I may be wrong here, please correct me if i am.
And the data needs to be stored in the database as quickly as possible. So, i need a quick and reliable approach to the problem.

There are two parts to your problem.
1. How to receive such a huge volume: If there is a huge volume of data being received, its generally a good idea to save(from the input stream of the response) it locally as a file and process that data asynchronously.(Make sure you set an appropriately high read timeout, else the data stream might be interrupted ) .
2. How do you process such a huge file: With big files, memory footprint needs to be minimal. For XML , SaxParsers are a golden standard . I found this library which is very similar to sax parsing, but for Json
http://rapidjson.org/md_doc_sax.html

You can use reactive approach and streaming the data.
With Spring, use MediaType.APPLICATION_STREAM_JSON_VALUE producer and Flux as return type.
On the client side, subscribe to the stream and process the data, or you can use Spring Batch to save the data to the Database.

Best way to query multiple data from webservice and mysql?

I have an android app which uses java rest api, which gets the data from mysql database.
After I open the app, I download a list of some items. Then, I make another 5 requests to rest api to get other connected resources, so:
First I download a list of shops.
Then, having these shops ids, I make 5 http requests to rest api and it fetches (from mysql) lists of photos (not actual images, just urls and ids), ratings, opening hours and 2 more things
This makes 6 api (and db) calls and it takes a long time.
Is there a better way to do it?

You might rewrite the multiple queries into a single JOIN and fetch them in a single network round trip.
Be sure you understand where the time is being spent and why. Your queries might be slow if you have WHERE clauses that don't use indexes. Run EXPLAIN PLAN and see if there's a TABLE SCAN being executed. If you see one, you can immediately improve performance by adding appropriate indexes on the columns in the WHERE clauses.

Why don't yout retrieve all those data in the first call? the response of restapi can be json or xml.
REQUEST: GET /shop/list
RESPONSE XML:
<shops>
<shop name="shop1" img="../../img1.jpg" ></shop>
<shop name="shop2" img="../../img2.jpg" ></shop>
<shop name="shop3" img="../../img3.jpg" ></shop>
</shops>
RESPONSE JSON:
[{name:"shop1",img:"../../img1.jpg"},
{name:"shop2",img:"../../img2.jpg"},
{name:"shop3",img:"../../img3.jpg"}]
You could handle these responses in your Java/Android application

How to fetch data dynamically from web server?

I have a Mongodb database that contains a Poll Collection.
The Poll collection has a number of Poll documents. This could be a large number of documents.
I am using Java Servlet for serving HTTP requests.
How can I implement a feed kind of retrieval mechanism at the server side?
For e.g., In the first request, I want to retrieve 1 to 10, documents, then 11 to 20 and so on...
As there is a scroll in the view, i want to get the data from server and send to client.
Does Mongodb provide a way to do this?

I think what you are looking for is a pagination. You could use the limit and skip methods with your find query.
First request
db.Poll.find().skip(0).limit(10)
Second request
db.Poll.find().skip(10).limit(10)
...
...
Note: You should also be sorting your find with some field.
db.Poll.find().skip(10).limit(10).sort({_id:-1})
For more info on the cursor methods you could look here: http://docs.mongodb.org/manual/reference/method/js-cursor/

How to Fetch specific numbers of emails either using imap or pop3 in java?

I have to fetch fix number emails using imap or pop3 protocol and java mail service . is there any way to do it. Thanks in advance.

Well in this case, fetch command can be used:
FETCH 2:4 (FLAGS BODY[HEADER.FIELDS (To)])
This command will fetch messages with UID 2 to 4 and fetch required information.
In case of JavaMail API I guess this kind of method is not expose, so firstly do a fast fetch to get meta data, and then iterate over the required number of messages fetching each one at a time.
There's a way to do fast fetch using fetchprofile.
I don't remember exactly as I tried it long time back. But this should be a head start.

First, use the Folder.getMessages method to get Message objects for the set of messages you want to fetch. The getMessages method does not fetch any message data.
Next, depending on what you're doing, you might want to use the Folder.fetch method to prefetch some message metadata (e.g., headers). This is more useful with IMAP than with POP3. Note that the fetch method can be applied to the array of Message objects you got in the first step.
Finally, iterate over the Message objects you got in the first step and access the data you need for each message. This is when the message data will actually be fetched from the server.