I import more than one hundred data use eclipse in the computer and no problem. But when I deploy on application server it will out of memory, and I copy the local tomcat.
what is this problem？
The date more than millions than this in another function, and just less than few field. it can export normal.
int totalSize = behaviorPortraitService.querySiletUserPortraitCountMonth(sietUserPortraitForm.getParams());
int pageNum = 0;
if (totalSize % 1000000 == 0) {
pageNum = totalSize / 1000000;
}else{
pageNum = totalSize / 1000000 + 1;
}
write.append(" year/month IMSI phone number The date of open card"
+ enter);
outSTr = response.getOutputStream(); // establish
buff = new BufferedOutputStream(outSTr);
// Loop total number of pages，Get the number of pages per page
for (int i = 1; i <= pageNum; i++) {
// According to the number of pages to get PageResult object
list = behaviorPortraitService.exportSiletUserPortraitMonth(sietUserPortraitForm.getParams(),i,100000);
// circulate List collection to write
for (int j = 0; j < list.size(); j++) {
// Recycling collection，Write data to the TXT
write.append(list.get(j).getBegin_endtime());
write.append(" " + list.get(j).getImsi());
write.append(" " + list.get(j).getMsisdn());
write.append(" "+ list.get(j).getSilencedays());
write.append(" " + list.get(j).getStarttime()
+ enter);
}
// After writing a page object，empty List，wipe cache
list.clear();
System.gc();
}
buff.write(write.toString().getBytes("UTF-8"));
Error:
org.springframework.web.util.NestedServletException: Handler processing failed; nested exception is java.lang.OutOfMemoryError: Java heap space
org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:839)
org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:719)
org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:644)
org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:549)
javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
com.trajectory.manager.controller.filter.SessionPrivaligeFilter.doFilter(SessionPrivaligeFilter.java:79)
org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:88)
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76)
root cause
java.lang.OutOfMemoryError: Java heap space
java.lang.StringCoding$StringEncoder.encode(StringCoding.java:232)
java.lang.StringCoding.encode(StringCoding.java:272)
java.lang.String.getBytes(String.java:946)
com.trajectory.behaviorportrait.controller.ExportSiletUserPortraitController.writeWeekTxtWeek(ExportSiletUserPortraitController.java:209)
com.trajectory.behaviorportrait.controller.ExportSiletUserPortraitController.handleRequestInternal(ExportSiletUserPortraitController.java:69)
org.springframework.web.servlet.mvc.AbstractController.handleRequest(AbstractController.java:153)
org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:48)
org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:790)
org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:719)
org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:644)
org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:549)
javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
com.trajectory.manager.controller.filter.SessionPrivaligeFilter.doFilter(SessionPrivaligeFilter.java:79)
org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:88)
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76)
You are creating a very large String which you are converting to a very large byte[].
Instead of doing either of these I suggest writing the data progressively to the output stream. i.e. don't use write.append just use buf.write for binary or if you want to write text, use
PrintWriter pw = new PrintWriter(new OutputStreamWriter(
new BufferedOutputStream(socket.getOuptutStream()), "UTF-8"));
pw.print( some text here );
pw.print( some more ext here );
pw.println( the end of the line );
pw.close(); // when finished.
This doesn't use much memory at all.
Related
I am using a java persistence Queue named BigQueue, It stores the data in the disk, bigQueue.gc() is used to clear the used disk space. The big queue.gc() is not clearing the used disk space. The disk memory is continuously increasing.
IBigQueue bigQueue = new BigQueueImpl("/home/test/BigQueueNew", "demo1");
for (int i = 0; i < 10000; i++) {
ManagedObject mo = new ManagedObject();
mo.setName("Aravind " + i);
bigQueue.enqueue(serialize(mo));
}
while (!bigQueue.isEmpty()) {
ManagedObject mo = (ManagedObject) deserialize(bigQueue.dequeue());
System.out.println("Key Dqueue ME");
}
bigQueue.close();
// bigQueue.removeAll(); bigQueue.gc();; System.out.println("Big Queue is " + bigQueue.isEmpty() +" Size is "+bigQueue.size());
In case someone is looking at this as well.
If you are using Java 11 on ubuntu, this could be a known issue. Refer to the link below.
Unless it is fixed at the source, you could download the source and fix it yourself.
https://github.com/bulldog2011/bigqueue/issues/39
I am trying to get all the average CPU usage for EC2s in a specific region. I tried the following:
Request one instance statistics per request, which works but very slow.
I tried to remove the dimensions from the request but I got an empty response, although cli returns all instances available (Statistics for up to 35 instances over a span of 24 hours).
I tried set 10 dimensions per request but I got this Exception:
2016-01-24 23:13:30.925 java[20057:31669898] GL_FRAMEBUFFER_INCOMPLETE_ATTACHMENT_EXT java.lang.RuntimeException: Requested texture dimensions (62206x154) require dimensions (0x154) that exceed maximum texture size (16384)
at com.sun.prism.es2.ES2RTTexture.create(ES2RTTexture.java:220)
at com.sun.prism.es2.ES2ResourceFactory.createRTTexture(ES2ResourceFactory.java:158)
at com.sun.prism.es2.ES2SwapChain.createGraphics(ES2SwapChain.java:210)
at com.sun.prism.es2.ES2SwapChain.createGraphics(ES2SwapChain.java:40)
at com.sun.javafx.tk.quantum.PresentingPainter.run(PresentingPainter.java:87)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at com.sun.javafx.tk.RenderJob.run(RenderJob.java:58)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at com.sun.javafx.tk.quantum.QuantumRenderer$PipelineRunnable.run(QuantumRenderer.java:125)
at java.lang.Thread.run(Thread.java:745)
code:
private List<Datapoint> monitorInstance(AmazonCloudWatchClient cloudWatchClient, List<Dimension> dimensions) {
List<Datapoint> datapoints = new ArrayList<>();
try {
int dimensionsLastIndex = 0;
for (int dimensionsIndex = 0; dimensionsIndex < dimensions.size(); dimensionsIndex = dimensionsLastIndex + 1) {
dimensionsLastIndex = (dimensionsIndex + 10) <= dimensions.size() ? dimensionsIndex + 9 : dimensions.size() - 1;
long offsetInMilliseconds = 1000 * 60 * 60 * 24;
GetMetricStatisticsRequest request = new GetMetricStatisticsRequest()
.withStartTime(new Date(new Date().getTime() - offsetInMilliseconds))
.withNamespace("AWS/EC2")
.withPeriod(60 * 60)
.withDimensions(dimensions.subList(dimensionsIndex, dimensionsLastIndex))
.withMetricName("CPUUtilization")
.withStatistics("Average")
.withEndTime(new Date());
GetMetricStatisticsResult getMetricStatisticsResult = cloudWatchClient.getMetricStatistics(request);
datapoints.addAll(getMetricStatisticsResult.getDatapoints());
}
return datapoints;
} catch (AmazonServiceException e) {
error(e.getMessage(), stackTraceToString(e));
}
return new ArrayList<>();
}
Although I want to fix this Exception, I wonder if there is a simple way to get the stats for all instances in one request.
I've noticed that jstack doesn't work when code is in a tight loop.
For example in this code:
private static void testAllocationNotOnHeap(){
long time = System.currentTimeMillis();
long result =0;
for (int j = 0; j < 20_000; j++) {
for (int i = 0; i < 100_000_000; i++) {
result += new TestObject(5,5).getResult();
}
}
System.out.println("Allocation not on heap took "
+ (System.currentTimeMillis()-time)
+ " with result " + result);
}
If you run jstack you get this message:
Unable to open socket file: target process not responding or HotSpot VM not loaded
The -F option can be used when the target process is not responding
As soon as you allow the code to 'breathe' e.g. by add a System.out every few thousand iterations it all works fine.
Note this is the same if you want to run jconsole or flight recorder from jmc.
I'd like to understand why this is and if there's anything I can do about it as I'd like to profile a tight loop.
I want to get the size of an SAP BusinessObjects report through Java code. My main concern is the size of the data we fetch through query, e.g., select si_id,si_name, si_size from ci_infoobjects where si_kind='webi'
How to get the size of particular report?
As Hardik Patel stated in the comments once you have a IInfoObject:
IFiles ifiles = object.getFiles();
IFile boFile = null;
long reportSize=0;
for (int k=0; k<ifiles.size(); k++)
{
boFile = (IFile) ifiles.get(k);
System.out.println("Size : " + boFile.getSize());
reportSize += boFile.getSize();
}
System.out.println("Size : " + reportSize);
I'm using spring ws on the server side with a AxiomSoapmessageFactory:
<bean id="messageFactory" class="org.springframework.ws.soap.axiom.AxiomSoapMessageFactory">
<property name="payloadCaching" value="true"/>
<!-- Need to figure out the appropriate directory -->
<property name="attachmentCacheDir" value="..."/>
</bean>
This works fine when the services are ingesting a message. However, when the services attempt to serve up a large message I get Java out of memory exception. Here is the stack trace:
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:2786)
at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:94)
at org.apache.axiom.mime.impl.axiom.MultipartWriterImpl$PartOutputStream.write(MultipartWriterImpl.java:42)
at javax.activation.DataHandler.writeTo(DataHandler.java:294)
at org.apache.axiom.mime.impl.axiom.MultipartWriterImpl.writePart(MultipartWriterImpl.java:116)
at org.apache.axiom.om.impl.OMMultipartWriter.writePart(OMMultipartWriter.java:136)
at org.apache.axiom.om.impl.MIMEOutputUtils.writeSOAPWithAttachmentsMessage(MIMEOutputUtils.java:269)
at org.springframework.ws.soap.axiom.AxiomSoapMessage.writeSwAMessage(AxiomSoapMessage.java:298)
at org.springframework.ws.soap.axiom.AxiomSoapMessage.writeTo(AxiomSoapMessage.java:248)
at org.springframework.ws.server.MessageDispatcher.getMessageContent(MessageDispatcher.java:192)
at org.springframework.ws.server.MessageDispatcher.receive(MessageDispatcher.java:174)
at org.springframework.ws.transport.support.WebServiceMessageReceiverObjectSupport.handleConnection(WebServiceMessageReceiverObjectSupport.java:88)
at org.springframework.ws.transport.http.WebServiceMessageReceiverHandlerAdapter.handle(WebServiceMessageReceiverHandlerAdapter.java:57)
at org.springframework.ws.transport.http.ErrorAwareWebServiceMessageReceiverHandlerAdapter.handle(ErrorAwareWebServiceMessageReceiverHandlerAdapter.java:42)
at org.springframework.ws.transport.http.MessageDispatcherServlet2.doService(MessageDispatcherServlet2.java:148)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:571)
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:511)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:857)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
at java.lang.Thread.run(Thread.java:680)
The problem seems to be that the services are attempting to read the whole attachment into memory prior to writing it out. Is there a way to stream the attachment?
I'm running this in a Tomcat instance on OSX 10.6.7 java --version =
java version "1.6.0_24"
Java(TM) SE Runtime Environment (build 1.6.0_24-b07-334-10M3326)
Java HotSpot(TM) 64-Bit Server VM (build 19.1-b02-334, mixed mode)
Have you tried changing the following parameters to allow for more memory?
-Xms512m
-Xmx1024m
-XX:PermSize=256m
I believe you need to set the attachmentCaching property to true. The default is false.
Truns out this was caused by turning message tracing on. When message tracing is on the MessageSender (at least in spring-ws 1.5.9) tries to copy the entire message into a ByteArrayOutputString and then converts the whole thing into a string. The offending functions are MessageSender.recieve and MessageSender.getContent
public void receive(MessageContext messageContext) throws Exception {
// Let's keep a reference to the request content as it came in, it might be changed by interceptors in dispatch()
String requestContent = "";
if (receivedMessageTracingLogger.isTraceEnabled()) {
requestContent = getMessageContent(messageContext.getRequest());
receivedMessageTracingLogger.trace("Received request [" + requestContent + "]");
}
else if (receivedMessageTracingLogger.isDebugEnabled()) {
receivedMessageTracingLogger.debug("Received request [" + messageContext.getRequest() + "]");
}
dispatch(messageContext);
if (messageContext.hasResponse()) {
WebServiceMessage response = messageContext.getResponse();
if (sentMessageTracingLogger.isTraceEnabled()) {
String responseContent = getMessageContent(response);
sentMessageTracingLogger.trace("Sent response [" + responseContent + "] for request [" +
requestContent + "]");
}
else if (sentMessageTracingLogger.isDebugEnabled()) {
sentMessageTracingLogger.debug("Sent response [" + response + "] for request [" +
messageContext.getRequest() + "]");
}
}
else if (sentMessageTracingLogger.isDebugEnabled()) {
sentMessageTracingLogger
.debug("MessageDispatcher with name '" + beanName + "' sends no response for request [" +
messageContext.getRequest() + "]");
}
private String getMessageContent(WebServiceMessage message) throws IOException {
ByteArrayOutputStream bos = new ByteArrayOutputStream();
message.writeTo(bos);
return bos.toString("UTF-8");
}