Opticks consuming large amount of memory

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Opticks consuming large amount of memory

Hotka, Michael

Hello…

 

I have an Opticks Plugin that plays Matlab processed video files. When I run the Plugin, I notice that the size of the Opticks executable in memory, as seen by the task manager, grows and grows in size until it consumes all the memory allocated for the process and the PC crashes.

 

I have convinced myself, thru the debugger, that the memory is allocated and hung onto when my Plugin calls the engPutVariable when writing the image frame to Matlab and then holds onto a smaller amount of memory on the engPutVariable when writing the associated MetaData to Matlab.

 

The plug uses mxArrays to pass both kinds of information into the engPutVariable call, and I have seen the mxDestroyArray function being called in my Plug immediately after returning from the engPutVariable call.

 

If I “clear” all processing knowledge from my Plugin with a button that is available, my GUI is cleared of information, but the current size of the Opticks executable remains what it was. If I start processing a new image file, the size of the Opticks executable increases from this previous value.

 

I suspect that somewhere deep in the Opticks side of this Matlab interface, this memory is being saved, or the memory is allocated to a pointer, transferred to Matlab and the next call to engPutVariable orphans this memory and the garbage collector is not finding it.

 

What can I do in my Plugin to force the release of these chunks of memory so the Opticks executable grows to a certain reasonable value and maintains that value no matter how many image frames are processed?

 

Thank you for your help on this.

 

Mike



This message and any enclosures are intended only for the addressee. Please
notify the sender by email if you are not the intended recipient. If you are
not the intended recipient, you may not use, copy, disclose, or distribute this
message or its contents or enclosures to any other person and any such actions
may be unlawful. Ball reserves the right to monitor and review all messages
and enclosures sent to or from this email address.

------------------------------------------------------------------------------
One dashboard for servers and applications across Physical-Virtual-Cloud
Widest out-of-the-box monitoring support with 50+ applications
Performance metrics, stats and reports that give you Actionable Insights
Deep dive visibility with transaction tracing using APM Insight.
http://ad.doubleclick.net/ddm/clk/290420510;117567292;y
_______________________________________________
Opticks-devs mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/opticks-devs
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Opticks consuming large amount of memory

John Tobe
Do you have any opticks raster elements that aren't being cleaned up? Could you share a sample of code that reproduces the problem?


From: Hotka, Michael [[hidden email]]
Sent: Friday, May 08, 2015 9:27 AM
To: [hidden email]
Cc: Grant, David A
Subject: [Opticks-devs] Opticks consuming large amount of memory

Hello…

 

I have an Opticks Plugin that plays Matlab processed video files. When I run the Plugin, I notice that the size of the Opticks executable in memory, as seen by the task manager, grows and grows in size until it consumes all the memory allocated for the process and the PC crashes.

 

I have convinced myself, thru the debugger, that the memory is allocated and hung onto when my Plugin calls the engPutVariable when writing the image frame to Matlab and then holds onto a smaller amount of memory on the engPutVariable when writing the associated MetaData to Matlab.

 

The plug uses mxArrays to pass both kinds of information into the engPutVariable call, and I have seen the mxDestroyArray function being called in my Plug immediately after returning from the engPutVariable call.

 

If I “clear” all processing knowledge from my Plugin with a button that is available, my GUI is cleared of information, but the current size of the Opticks executable remains what it was. If I start processing a new image file, the size of the Opticks executable increases from this previous value.

 

I suspect that somewhere deep in the Opticks side of this Matlab interface, this memory is being saved, or the memory is allocated to a pointer, transferred to Matlab and the next call to engPutVariable orphans this memory and the garbage collector is not finding it.

 

What can I do in my Plugin to force the release of these chunks of memory so the Opticks executable grows to a certain reasonable value and maintains that value no matter how many image frames are processed?

 

Thank you for your help on this.

 

Mike



This message and any enclosures are intended only for the addressee. Please
notify the sender by email if you are not the intended recipient. If you are
not the intended recipient, you may not use, copy, disclose, or distribute this
message or its contents or enclosures to any other person and any such actions
may be unlawful. Ball reserves the right to monitor and review all messages
and enclosures sent to or from this email address.
<script id="F5_watermark"></script>


This message and any enclosures are intended only for the addressee. Please
notify the sender by email if you are not the intended recipient. If you are
not the intended recipient, you may not use, copy, disclose, or distribute this
message or its contents or enclosures to any other person and any such actions
may be unlawful. Ball reserves the right to monitor and review all messages
and enclosures sent to or from this email address.

------------------------------------------------------------------------------
One dashboard for servers and applications across Physical-Virtual-Cloud
Widest out-of-the-box monitoring support with 50+ applications
Performance metrics, stats and reports that give you Actionable Insights
Deep dive visibility with transaction tracing using APM Insight.
http://ad.doubleclick.net/ddm/clk/290420510;117567292;y
_______________________________________________
Opticks-devs mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/opticks-devs
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Opticks consuming large amount of memory

tclarke
Administrator
In reply to this post by Hotka, Michael

The various objects should document their behavior re: RAII, etc. If you're asking about the DataAccessor/RasterPager piece, that should be in the API docs (in the class references and I believe in one of the howto sections).

 

P.S. I'm moving this discussion to the dev mailing list as that's the appropriate place for general Opticks API questions/discussions.


From: Hotka, Michael
Sent: Thursday, June 11, 2015 11:36 AM
To: Tobe, John
Cc: Clarke, Trevor; Goffena, Robert; Couvutsakis, Jason
Subject: RE: Opticks consuming large amount of memory

Hello…

 

Is there documentation I can read about how Opticks handles memory management?

 

Mike

 

From: Tobe, John
Sent: Thursday, June 11, 2015 8:36 AM
To: Hotka, Michael
Cc: Clarke, Trevor; Goffena, Robert; Couvutsakis, Jason
Subject: RE: Opticks consuming large amount of memory

 

You shouldn't have to manually decrement the reference count, at least in normal circumstances.  The DataAccessor should clean itself up after it goes out of scope, resulting in the pager class releasing the memory for the page.  The pager may be reading more than the requested amount of memory to avoid repeatedly going back to disc and hurting performance, but it looks like you're processing a frame at a time so that's probably not the case.  Trevor, Jason, and Bob may have some ideas of what's going on.


From: Hotka, Michael
Sent: Wednesday, June 10, 2015 11:18 PM
To: Tobe, John
Subject: RE: Opticks consuming large amount of memory

Hello…

 

Now that the mxCalloc issue is fixed per your suggestion, I am onto the next part of my code that appears to hang onto large amounts of memory after exiting the routine they reside in.

 

The code:

   // Get the RasterElement

   const RasterElement* pElement = getRasterElement();

   if (pElement == NULL)

   {

      return (false);

   }

   // Create a data request

   FactoryResource<DataRequest> pRequest;

   if (pRequest.get() == NULL)

   {

      return (false);

   }

   DimensionDescriptor currBand = mpLayer->getDisplayedBand(GRAY);

   pRequest->setBands(currBand, currBand, 1);

   pRequest->setWritable(false);

 

   // Request an accessor

   DataAccessor pAcc = pElement->getDataAccessor(pRequest.release());

 

The DataAccessor pAcc seems to grap and hold onto about 150MB of memory. I added the call at the end of the routine:

 

pAcc->decrementRefCount();

 

which calls the DataAccessorImpl int decrementRefCount routine and not the DataAccessor.h private void decrementRefCount, which calls the mpDeleter function to release the memory allocated for the DataAccessor class.

 

How do I call the private void decrementRefCount and not the inline int decrementRefCount?

 

Thank you for all your help on my memory issues.

 

Mike



This message and any enclosures are intended only for the addressee. Please
notify the sender by email if you are not the intended recipient. If you are
not the intended recipient, you may not use, copy, disclose, or distribute this
message or its contents or enclosures to any other person and any such actions
may be unlawful. Ball reserves the right to monitor and review all messages
and enclosures sent to or from this email address.

------------------------------------------------------------------------------

_______________________________________________
Opticks-devs mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/opticks-devs
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Opticks consuming large amount of memory

Dustan Adkins
My first guess would be the pager is storing the data in a cache. Which pager are you using?

Also what interleave is your data? If it is BIP or BIL then concurrentBands may be getting ignored. If it is BSQ then his will read and cache the entire band. I would try changing interleave on import and see if the memory usage stays the same.

On Jun 11, 2015, at 1:07 PM, Clarke, Trevor <[hidden email]> wrote:

The various objects should document their behavior re: RAII, etc. If you're asking about the DataAccessor/RasterPager piece, that should be in the API docs (in the class references and I believe in one of the howto sections).

 

P.S. I'm moving this discussion to the dev mailing list as that's the appropriate place for general Opticks API questions/discussions.


From: Hotka, Michael
Sent: Thursday, June 11, 2015 11:36 AM
To: Tobe, John
Cc: Clarke, Trevor; Goffena, Robert; Couvutsakis, Jason
Subject: RE: Opticks consuming large amount of memory

Hello…

 

Is there documentation I can read about how Opticks handles memory management?

 

Mike

 

From: Tobe, John
Sent: Thursday, June 11, 2015 8:36 AM
To: Hotka, Michael
Cc: Clarke, Trevor; Goffena, Robert; Couvutsakis, Jason
Subject: RE: Opticks consuming large amount of memory

 

You shouldn't have to manually decrement the reference count, at least in normal circumstances.  The DataAccessor should clean itself up after it goes out of scope, resulting in the pager class releasing the memory for the page.  The pager may be reading more than the requested amount of memory to avoid repeatedly going back to disc and hurting performance, but it looks like you're processing a frame at a time so that's probably not the case.  Trevor, Jason, and Bob may have some ideas of what's going on.


From: Hotka, Michael
Sent: Wednesday, June 10, 2015 11:18 PM
To: Tobe, John
Subject: RE: Opticks consuming large amount of memory

Hello…

 

Now that the mxCalloc issue is fixed per your suggestion, I am onto the next part of my code that appears to hang onto large amounts of memory after exiting the routine they reside in.

 

The code:

   // Get the RasterElement

   const RasterElement* pElement = getRasterElement();

   if (pElement == NULL)

   {

      return (false);

   }

   // Create a data request

   FactoryResource<DataRequest> pRequest;

   if (pRequest.get() == NULL)

   {

      return (false);

   }

   DimensionDescriptor currBand = mpLayer->getDisplayedBand(GRAY);

   pRequest->setBands(currBand, currBand, 1);

   pRequest->setWritable(false);

 

   // Request an accessor

   DataAccessor pAcc = pElement->getDataAccessor(pRequest.release());

 

The DataAccessor pAcc seems to grap and hold onto about 150MB of memory. I added the call at the end of the routine:

 

pAcc->decrementRefCount();

 

which calls the DataAccessorImpl int decrementRefCount routine and not the DataAccessor.h private void decrementRefCount, which calls the mpDeleter function to release the memory allocated for the DataAccessor class.

 

How do I call the private void decrementRefCount and not the inline int decrementRefCount?

 

Thank you for all your help on my memory issues.

 

Mike



This message and any enclosures are intended only for the addressee. Please
notify the sender by email if you are not the intended recipient. If you are
not the intended recipient, you may not use, copy, disclose, or distribute this
message or its contents or enclosures to any other person and any such actions
may be unlawful. Ball reserves the right to monitor and review all messages
and enclosures sent to or from this email address.
------------------------------------------------------------------------------
_______________________________________________
Opticks-devs mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/opticks-devs

------------------------------------------------------------------------------

_______________________________________________
Opticks-devs mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/opticks-devs

------------------------------------------------------------------------------

_______________________________________________
Opticks-devs mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/opticks-devs
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Opticks consuming large amount of memory

Hotka, Michael

Hello…

 

Thank you for these tips.

 

I will get back with you when I find the answers to your great questions.

 

Mike

 

From: Dustan Adkins [mailto:[hidden email]]
Sent: Thursday, June 11, 2015 11:24 AM
To: [hidden email]
Subject: Re: [Opticks-devs] Opticks consuming large amount of memory

 

My first guess would be the pager is storing the data in a cache. Which pager are you using?

 

Also what interleave is your data? If it is BIP or BIL then concurrentBands may be getting ignored. If it is BSQ then his will read and cache the entire band. I would try changing interleave on import and see if the memory usage stays the same.


On Jun 11, 2015, at 1:07 PM, Clarke, Trevor <[hidden email]> wrote:

The various objects should document their behavior re: RAII, etc. If you're asking about the DataAccessor/RasterPager piece, that should be in the API docs (in the class references and I believe in one of the howto sections).

 

P.S. I'm moving this discussion to the dev mailing list as that's the appropriate place for general Opticks API questions/discussions.


From: Hotka, Michael
Sent: Thursday, June 11, 2015 11:36 AM
To: Tobe, John
Cc: Clarke, Trevor; Goffena, Robert; Couvutsakis, Jason
Subject: RE: Opticks consuming large amount of memory

Hello…

 

Is there documentation I can read about how Opticks handles memory management?

 

Mike

 

From: Tobe, John
Sent: Thursday, June 11, 2015 8:36 AM
To: Hotka, Michael
Cc: Clarke, Trevor; Goffena, Robert; Couvutsakis, Jason
Subject: RE: Opticks consuming large amount of memory

 

You shouldn't have to manually decrement the reference count, at least in normal circumstances.  The DataAccessor should clean itself up after it goes out of scope, resulting in the pager class releasing the memory for the page.  The pager may be reading more than the requested amount of memory to avoid repeatedly going back to disc and hurting performance, but it looks like you're processing a frame at a time so that's probably not the case.  Trevor, Jason, and Bob may have some ideas of what's going on.


From: Hotka, Michael
Sent: Wednesday, June 10, 2015 11:18 PM
To: Tobe, John
Subject: RE: Opticks consuming large amount of memory

Hello…

 

Now that the mxCalloc issue is fixed per your suggestion, I am onto the next part of my code that appears to hang onto large amounts of memory after exiting the routine they reside in.

 

The code:

   // Get the RasterElement

   const RasterElement* pElement = getRasterElement();

   if (pElement == NULL)

   {

      return (false);

   }

   // Create a data request

   FactoryResource<DataRequest> pRequest;

   if (pRequest.get() == NULL)

   {

      return (false);

   }

   DimensionDescriptor currBand = mpLayer->getDisplayedBand(GRAY);

   pRequest->setBands(currBand, currBand, 1);

   pRequest->setWritable(false);

 

   // Request an accessor

   DataAccessor pAcc = pElement->getDataAccessor(pRequest.release());

 

The DataAccessor pAcc seems to grap and hold onto about 150MB of memory. I added the call at the end of the routine:

 

pAcc->decrementRefCount();

 

which calls the DataAccessorImpl int decrementRefCount routine and not the DataAccessor.h private void decrementRefCount, which calls the mpDeleter function to release the memory allocated for the DataAccessor class.

 

How do I call the private void decrementRefCount and not the inline int decrementRefCount?

 

Thank you for all your help on my memory issues.

 

Mike



This message and any enclosures are intended only for the addressee. Please
notify the sender by email if you are not the intended recipient. If you are
not the intended recipient, you may not use, copy, disclose, or distribute this
message or its contents or enclosures to any other person and any such actions
may be unlawful. Ball reserves the right to monitor and review all messages
and enclosures sent to or from this email address.

------------------------------------------------------------------------------

_______________________________________________
Opticks-devs mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/opticks-devs



This message and any enclosures are intended only for the addressee. Please
notify the sender by email if you are not the intended recipient. If you are
not the intended recipient, you may not use, copy, disclose, or distribute this
message or its contents or enclosures to any other person and any such actions
may be unlawful. Ball reserves the right to monitor and review all messages
and enclosures sent to or from this email address.

------------------------------------------------------------------------------

_______________________________________________
Opticks-devs mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/opticks-devs
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Opticks consuming large amount of memory

John Tobe
In reply to this post by tclarke
I think you need to make your own Page class that inherits from RasterPage and not CachedPage.  Should be fairly simple since it would be specific to your data.


From: Hotka, Michael
Sent: Sunday, June 14, 2015 9:49 PM
To: Goffena, Robert; Clarke, Trevor; Tobe, John
Cc: Couvutsakis, Jason
Subject: RE: Opticks consuming large amount of memory

Hello…

 

I don’t think I am having a “memory leak” issue anymore, but have stumbled into “normal operation” of grabbing and holding onto memory by Opticks.

 

Dustin Adkin’s comment about paging and storing data in cache was spot on. This is exactly what is happening. I have attached the pager we use to get image data into Opticks. At line 23, about 655 Kb are grabbed by the Array Resource. By line 28, this data has been copied into the CachedPage::UnitPtr object and is now part of Opticks Cache memory. I read that caching image data is useful if you wish to make a movie. We don’t have that requirement. Just play the date thru once from beginning to end.

 

Is there a way to tell Opticks not to keep the cached image data once it has been displayed on the display window?

 

If there is not a way to inhibit Opticks from keeping all this data, in the attached routine, would I keep knowledge of the previous cached image’s memory and “reuse” this cached memory for the next image frame?

 

Or is there another way to feed just one frame of data to Opticks and not have it cache anything?

 

Thank you all in advance for any suggestions you might give me.

 

Mike

 

From: Goffena, Robert
Sent: Friday, June 12, 2015 5:08 PM
To: Clarke, Trevor; Hotka, Michael; Tobe, John; [hidden email]
Cc: Couvutsakis, Jason
Subject: RE: Opticks consuming large amount of memory

 

Are you still having this problem?  Can you share more of the code (the code which calls the code which you have provided below)?

 

Robert Goffena

Ball Aerospace & Technologies Corp.

2875 Presidential Dr. Fairborn, OH 45324-6269

Phone:  (937) 320-4096

Fax:  (937) 429-1687

Email:  [hidden email]

 

From: Clarke, Trevor
Sent: Thursday, June 11, 2015 1:03 PM
To: Hotka, Michael; Tobe, John; [hidden email]
Cc: Goffena, Robert; Couvutsakis, Jason
Subject: RE: Opticks consuming large amount of memory

 

The various objects should document their behavior re: RAII, etc. If you're asking about the DataAccessor/RasterPager piece, that should be in the API docs (in the class references and I believe in one of the howto sections).

 

P.S. I'm moving this discussion to the dev mailing list as that's the appropriate place for general Opticks API questions/discussions.


From: Hotka, Michael
Sent: Thursday, June 11, 2015 11:36 AM
To: Tobe, John
Cc: Clarke, Trevor; Goffena, Robert; Couvutsakis, Jason
Subject: RE: Opticks consuming large amount of memory

Hello…

 

Is there documentation I can read about how Opticks handles memory management?

 

Mike

 

From: Tobe, John
Sent: Thursday, June 11, 2015 8:36 AM
To: Hotka, Michael
Cc: Clarke, Trevor; Goffena, Robert; Couvutsakis, Jason
Subject: RE: Opticks consuming large amount of memory

 

You shouldn't have to manually decrement the reference count, at least in normal circumstances.  The DataAccessor should clean itself up after it goes out of scope, resulting in the pager class releasing the memory for the page.  The pager may be reading more than the requested amount of memory to avoid repeatedly going back to disc and hurting performance, but it looks like you're processing a frame at a time so that's probably not the case.  Trevor, Jason, and Bob may have some ideas of what's going on.


From: Hotka, Michael
Sent: Wednesday, June 10, 2015 11:18 PM
To: Tobe, John
Subject: RE: Opticks consuming large amount of memory

Hello…

 

Now that the mxCalloc issue is fixed per your suggestion, I am onto the next part of my code that appears to hang onto large amounts of memory after exiting the routine they reside in.

 

The code:

   // Get the RasterElement

   const RasterElement* pElement = getRasterElement();

   if (pElement == NULL)

   {

      return (false);

   }

   // Create a data request

   FactoryResource<DataRequest> pRequest;

   if (pRequest.get() == NULL)

   {

      return (false);

   }

   DimensionDescriptor currBand = mpLayer->getDisplayedBand(GRAY);

   pRequest->setBands(currBand, currBand, 1);

   pRequest->setWritable(false);

 

   // Request an accessor

   DataAccessor pAcc = pElement->getDataAccessor(pRequest.release());

 

The DataAccessor pAcc seems to grap and hold onto about 150MB of memory. I added the call at the end of the routine:

 

pAcc->decrementRefCount();

 

which calls the DataAccessorImpl int decrementRefCount routine and not the DataAccessor.h private void decrementRefCount, which calls the mpDeleter function to release the memory allocated for the DataAccessor class.

 

How do I call the private void decrementRefCount and not the inline int decrementRefCount?

 

Thank you for all your help on my memory issues.

 

Mike

<script id="F5_watermark"></script>


This message and any enclosures are intended only for the addressee. Please
notify the sender by email if you are not the intended recipient. If you are
not the intended recipient, you may not use, copy, disclose, or distribute this
message or its contents or enclosures to any other person and any such actions
may be unlawful. Ball reserves the right to monitor and review all messages
and enclosures sent to or from this email address.

------------------------------------------------------------------------------

_______________________________________________
Opticks-devs mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/opticks-devs
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Opticks consuming large amount of memory

Dustan Adkins
I didn't get the attachment, but I think John is on the right track. You might also be able to subclass Pager instead of CachedPager. There may also be a member variable on CachedPager you can change. I think either would work. 

On Jun 15, 2015, at 9:52 AM, Tobe, John <[hidden email]> wrote:

I think you need to make your own Page class that inherits from RasterPage and not CachedPage.  Should be fairly simple since it would be specific to your data.


From: Hotka, Michael
Sent: Sunday, June 14, 2015 9:49 PM
To: Goffena, Robert; Clarke, Trevor; Tobe, John
Cc: Couvutsakis, Jason
Subject: RE: Opticks consuming large amount of memory

Hello…

 

I don’t think I am having a “memory leak” issue anymore, but have stumbled into “normal operation” of grabbing and holding onto memory by Opticks.

 

Dustin Adkin’s comment about paging and storing data in cache was spot on. This is exactly what is happening. I have attached the pager we use to get image data into Opticks. At line 23, about 655 Kb are grabbed by the Array Resource. By line 28, this data has been copied into the CachedPage::UnitPtr object and is now part of Opticks Cache memory. I read that caching image data is useful if you wish to make a movie. We don’t have that requirement. Just play the date thru once from beginning to end.

 

Is there a way to tell Opticks not to keep the cached image data once it has been displayed on the display window?

 

If there is not a way to inhibit Opticks from keeping all this data, in the attached routine, would I keep knowledge of the previous cached image’s memory and “reuse” this cached memory for the next image frame?

 

Or is there another way to feed just one frame of data to Opticks and not have it cache anything?

 

Thank you all in advance for any suggestions you might give me.

 

Mike

 

From: Goffena, Robert
Sent: Friday, June 12, 2015 5:08 PM
To: Clarke, Trevor; Hotka, Michael; Tobe, John; [hidden email]
Cc: Couvutsakis, Jason
Subject: RE: Opticks consuming large amount of memory

 

Are you still having this problem?  Can you share more of the code (the code which calls the code which you have provided below)?

 

Robert Goffena

Ball Aerospace & Technologies Corp.

2875 Presidential Dr. Fairborn, OH 45324-6269

Phone:  (937) 320-4096

Fax:  (937) 429-1687

Email:  [hidden email]

 

From: Clarke, Trevor
Sent: Thursday, June 11, 2015 1:03 PM
To: Hotka, Michael; Tobe, John; [hidden email]
Cc: Goffena, Robert; Couvutsakis, Jason
Subject: RE: Opticks consuming large amount of memory

 

The various objects should document their behavior re: RAII, etc. If you're asking about the DataAccessor/RasterPager piece, that should be in the API docs (in the class references and I believe in one of the howto sections).

 

P.S. I'm moving this discussion to the dev mailing list as that's the appropriate place for general Opticks API questions/discussions.


From: Hotka, Michael
Sent: Thursday, June 11, 2015 11:36 AM
To: Tobe, John
Cc: Clarke, Trevor; Goffena, Robert; Couvutsakis, Jason
Subject: RE: Opticks consuming large amount of memory

Hello…

 

Is there documentation I can read about how Opticks handles memory management?

 

Mike

 

From: Tobe, John
Sent: Thursday, June 11, 2015 8:36 AM
To: Hotka, Michael
Cc: Clarke, Trevor; Goffena, Robert; Couvutsakis, Jason
Subject: RE: Opticks consuming large amount of memory

 

You shouldn't have to manually decrement the reference count, at least in normal circumstances.  The DataAccessor should clean itself up after it goes out of scope, resulting in the pager class releasing the memory for the page.  The pager may be reading more than the requested amount of memory to avoid repeatedly going back to disc and hurting performance, but it looks like you're processing a frame at a time so that's probably not the case.  Trevor, Jason, and Bob may have some ideas of what's going on.


From: Hotka, Michael
Sent: Wednesday, June 10, 2015 11:18 PM
To: Tobe, John
Subject: RE: Opticks consuming large amount of memory

Hello…

 

Now that the mxCalloc issue is fixed per your suggestion, I am onto the next part of my code that appears to hang onto large amounts of memory after exiting the routine they reside in.

 

The code:

   // Get the RasterElement

   const RasterElement* pElement = getRasterElement();

   if (pElement == NULL)

   {

      return (false);

   }

   // Create a data request

   FactoryResource<DataRequest> pRequest;

   if (pRequest.get() == NULL)

   {

      return (false);

   }

   DimensionDescriptor currBand = mpLayer->getDisplayedBand(GRAY);

   pRequest->setBands(currBand, currBand, 1);

   pRequest->setWritable(false);

 

   // Request an accessor

   DataAccessor pAcc = pElement->getDataAccessor(pRequest.release());

 

The DataAccessor pAcc seems to grap and hold onto about 150MB of memory. I added the call at the end of the routine:

 

pAcc->decrementRefCount();

 

which calls the DataAccessorImpl int decrementRefCount routine and not the DataAccessor.h private void decrementRefCount, which calls the mpDeleter function to release the memory allocated for the DataAccessor class.

 

How do I call the private void decrementRefCount and not the inline int decrementRefCount?

 

Thank you for all your help on my memory issues.

 

Mike

<script id="F5_watermark"></script>


This message and any enclosures are intended only for the addressee. Please
notify the sender by email if you are not the intended recipient. If you are
not the intended recipient, you may not use, copy, disclose, or distribute this
message or its contents or enclosures to any other person and any such actions
may be unlawful. Ball reserves the right to monitor and review all messages
and enclosures sent to or from this email address.
------------------------------------------------------------------------------
_______________________________________________
Opticks-devs mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/opticks-devs

------------------------------------------------------------------------------

_______________________________________________
Opticks-devs mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/opticks-devs

------------------------------------------------------------------------------

_______________________________________________
Opticks-devs mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/opticks-devs
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Opticks consuming large amount of memory

goffena
I think John, Dustan, Trevor are sending you on the correct path.

Out of curiosity, I searched through Opticks code base for releasePage, and noticed the following calling releasePage. Almost looks like if you created a BSQ, 1-concurrent-band/frame DataAccessor, and looped through the frames, that releasePage may not get called.  Setting number of concurrent rows to be less than number of rows in frame, might workaround that.  Still think John/Dustan/Trevor have the correct answer though.

DataAccessorImpl::updateIfNeeded
   if (mCurrentRow >= mConcurrentRows)
   {
      mpRasterElement->incrementDataAccessor
   }

RasterElementImp::incrementDataAccessor
   da.mpRasterPager->releasePage()

DataAccessor::~DataAccessor
   mpRasterPager->releasePage

CachedPager::releasePage
   delete pPage;

You have mentioned you only want to play through the data once.  Will each pixel only get accessed once though?  Or will you play through each pixel once, and then have algorithms access the pixel multiple times?

Bob Goffena



--------------------------------------------
On Mon, 6/15/15, Dustan Adkins <[hidden email]> wrote:

 Subject: Re: [Opticks-devs] Opticks consuming large amount of memory
 To: "[hidden email]" <[hidden email]>
 Date: Monday, June 15, 2015, 11:05 AM
 
 I
 didn't get the attachment, but I think John is on the
 right track. You might also be able to subclass Pager
 instead of CachedPager. There may also be a member variable
 on CachedPager you can change. I think either would
 work. 
 On Jun 15,
 2015, at 9:52 AM, Tobe, John <[hidden email]>
 wrote:
 
 
 
 
 #yiv8128718542 #yiv8128718542  _filtered #yiv8128718542
 {font-family:Calibri;}
  _filtered #yiv8128718542 {font-family:Tahoma;}
  _filtered #yiv8128718542 {font-family:Consolas;}
 #yiv8128718542 p.yiv8128718542MsoNormal, #yiv8128718542
 li.yiv8128718542MsoNormal, #yiv8128718542
 div.yiv8128718542MsoNormal
  {margin:0in;margin-bottom:.0001pt;font-size:12.0pt;}
 #yiv8128718542 a:link, #yiv8128718542
 span.yiv8128718542MsoHyperlink
  {color:blue;text-decoration:underline;}
 #yiv8128718542 a:visited, #yiv8128718542
 span.yiv8128718542MsoHyperlinkFollowed
  {color:purple;text-decoration:underline;}
 #yiv8128718542 p
  {margin:0in;margin-bottom:.0001pt;font-size:12.0pt;}
 #yiv8128718542 pre
  {margin:0in;margin-bottom:.0001pt;font-size:10.0pt;}
 #yiv8128718542 p.yiv8128718542MsoAcetate, #yiv8128718542
 li.yiv8128718542MsoAcetate, #yiv8128718542
 div.yiv8128718542MsoAcetate
  {margin:0in;margin-bottom:.0001pt;font-size:8.0pt;}
 #yiv8128718542 span.yiv8128718542HTMLPreformattedChar
  {font-family:Consolas;}
 #yiv8128718542 span.yiv8128718542BalloonTextChar
  {}
 #yiv8128718542 p.yiv8128718542msochpdefault, #yiv8128718542
 li.yiv8128718542msochpdefault, #yiv8128718542
 div.yiv8128718542msochpdefault
  {margin:0in;margin-bottom:.0001pt;font-size:10.0pt;}
 #yiv8128718542 span.yiv8128718542htmlpreformattedchar0
  {font-family:Consolas;}
 #yiv8128718542 span.yiv8128718542balloontextchar0
  {}
 #yiv8128718542 span.yiv8128718542balloontextchar00
  {}
 #yiv8128718542 span.yiv8128718542balloontextchar000
  {}
 #yiv8128718542 span.yiv8128718542emailstyle19
  {color:#1F497D;}
 #yiv8128718542 span.yiv8128718542emailstyle20
  {color:#1F497D;}
 #yiv8128718542 span.yiv8128718542emailstyle21
  {color:#1F497D;}
 #yiv8128718542 span.yiv8128718542emailstyle22
  {color:#1F497D;}
 #yiv8128718542 span.yiv8128718542emailstyle23
  {color:#1F497D;}
 #yiv8128718542 span.yiv8128718542emailstyle24
  {color:#1F497D;}
 #yiv8128718542 span.yiv8128718542emailstyle28
  {color:#1F497D;}
 #yiv8128718542 span.yiv8128718542htmlpreformattedchar00
  {}
 #yiv8128718542 span.yiv8128718542emailstyle33
  {color:#1F497D;}
 #yiv8128718542 span.yiv8128718542EmailStyle36
  {color:#1F497D;}
 #yiv8128718542 span.yiv8128718542EmailStyle37
  {color:#1F497D;}
 #yiv8128718542 .yiv8128718542MsoChpDefault
  {font-size:10.0pt;}
  _filtered #yiv8128718542 {margin:1.0in 1.0in 1.0in 1.0in;}
 #yiv8128718542
 I
 think you need to make your own Page class that inherits
 from RasterPage and not CachedPage.  Should be fairly
 simple since it would be specific to your data.
 
 
 
 
 
 From: Hotka, Michael
 
 Sent: Sunday, June 14, 2015 9:49 PM
 
 To: Goffena, Robert; Clarke, Trevor; Tobe, John
 
 Cc: Couvutsakis, Jason
 
 Subject: RE: Opticks consuming large amount of
 memory
 
 
 
 
 
 
 
 Hello…
  
 I don’t think I am having a
 “memory leak” issue anymore, but have stumbled into
 “normal operation” of grabbing and holding onto memory
 by Opticks.
  
 Dustin Adkin’s comment about
 paging and storing data in cache was spot on. This is
 exactly what is happening. I have attached the pager we use
 to get image
  data into Opticks. At line 23, about 655 Kb are grabbed by
 the Array Resource. By line 28, this data has been copied
 into the CachedPage::UnitPtr object and is now part of
 Opticks Cache memory. I read that caching image data is
 useful if you wish to make a
  movie. We don’t have that requirement. Just play the date
 thru once from beginning to end.
  
 Is there a way to tell Opticks not
 to keep the cached image data once it has been displayed on
 the display window?
  
 If there is not a way to inhibit
 Opticks from keeping all this data, in the attached routine,
 would I keep knowledge of the previous cached image’s
 memory
  and “reuse” this cached memory for the next image
 frame?
  
 Or is there another way to feed
 just one frame of data to Opticks and not have it cache
 anything?
  
 Thank you all in advance for any
 suggestions you might give me.
  
 Mike
  
 
 
 From: Goffena, Robert
 
 
 Sent: Friday, June 12, 2015 5:08 PM
 
 To: Clarke, Trevor; Hotka, Michael; Tobe, John; [hidden email]
 
 Cc: Couvutsakis, Jason
 
 Subject: RE: Opticks consuming large amount of
 memory
 
 
  
 Are you still having this
 problem?  Can you share more of the code (the code which
 calls the code which you have provided below)?
  
 
 Robert Goffena
 Ball Aerospace & Technologies
 Corp.
 2875 Presidential Dr. Fairborn, OH
 45324-6269
 Phone:  (937) 320-4096
 Fax:  (937) 429-1687
 Email: 
 [hidden email]
 
  
 
 
 From: Clarke, Trevor
 
 
 Sent: Thursday, June 11, 2015 1:03 PM
 
 To: Hotka, Michael; Tobe, John;
 [hidden email]
 
 Cc: Goffena, Robert; Couvutsakis, Jason
 
 Subject: RE: Opticks consuming large amount of
 memory
 
 
  
 
 The various objects
 should document their behavior re: RAII, etc. If you're
 asking about the DataAccessor/RasterPager piece, that should
 be in the API docs (in the class references
  and I believe in one of the howto sections).
  
 P.S. I'm moving this
 discussion to the dev mailing list as that's the
 appropriate place for general Opticks API
 questions/discussions.
 
 
 
 
 
 From: Hotka, Michael
 
 Sent: Thursday, June 11, 2015 11:36 AM
 
 To: Tobe, John
 
 Cc: Clarke, Trevor; Goffena, Robert; Couvutsakis,
 Jason
 
 Subject: RE: Opticks consuming large amount of
 memory
 
 
 
 Hello…
  
 Is there documentation I can read
 about how Opticks handles memory management?
  
 Mike
  
 
 
 From: Tobe, John
 
 
 Sent: Thursday, June 11, 2015 8:36 AM
 
 To: Hotka, Michael
 
 Cc: Clarke, Trevor; Goffena, Robert; Couvutsakis,
 Jason
 
 Subject: RE: Opticks consuming large amount of
 memory
 
 
  
 
 You shouldn't have to manually
 decrement the reference count, at least in normal
 circumstances.  The DataAccessor should clean itself up
 after it goes out of
  scope, resulting in the pager class releasing the memory
 for the page.  The pager may be reading more than the
 requested amount of memory to avoid repeatedly going back to
 disc and hurting performance, but it looks like you're
 processing a frame at a time
  so that's probably not the case.  Trevor, Jason, and
 Bob may have some ideas of what's going on.
 
 
 
 
 
 From: Hotka, Michael
 
 Sent: Wednesday, June 10, 2015 11:18 PM
 
 To: Tobe, John
 
 Subject: RE: Opticks consuming large amount of
 memory
 
 
 
 Hello…
  
 Now that the mxCalloc issue is
 fixed per your suggestion, I am onto the next part of my
 code that appears to hang onto large amounts of memory after
 exiting
  the routine they reside in.
  
 The code:
    //
 Get
 the
 RasterElement
   
 const
 RasterElement*
 pElement
 =
 getRasterElement();
   
 if
 (pElement
 ==
 NULL)
   
 {
       return
 (false);
   
 }
   
 //
 Create
 a
 data
 request
   
 FactoryResource<DataRequest>
 pRequest;
   
 if
 (pRequest.get()
 ==
 NULL)
   
 {
      
 return
 (false);
   
 }
   
 DimensionDescriptor
 currBand
 =
 mpLayer->getDisplayedBand(GRAY);
   
 pRequest->setBands(currBand,
 currBand,
 1);
   
 pRequest->setWritable(false);
  
    //
 Request
 an accessor
    DataAccessor
 pAcc =
 pElement->getDataAccessor(pRequest.release());
  
 The DataAccessor pAcc seems to grap
 and hold onto about 150MB of memory. I added the call at the
 end of the routine:
  
 pAcc->decrementRefCount();
  
 which calls the DataAccessorImpl int
 decrementRefCount routine and not the DataAccessor.h private
 void decrementRefCount, which calls the mpDeleter function
 to release the memory allocated for the DataAccessor
  class.
  
 How do I call the private void
 decrementRefCount and not the inline int
 decrementRefCount?
  
 Thank you for all your help on my
 memory issues.
  
 Mike
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 This message and any enclosures are intended only for the
 addressee.  Please
 
 notify the sender by email if you are not the intended
 recipient.  If you are
 
 not the intended recipient, you may not use, copy, disclose,
 or distribute this
 
 message or its contents or enclosures to any other person
 and any such actions
 
 may be unlawful.  Ball reserves the right to monitor and
 review all messages
 
 and enclosures sent to or from this email address.
 
 
 
 ------------------------------------------------------------------------------
 _______________________________________________
 Opticks-devs mailing list
 [hidden email]
 https://lists.sourceforge.net/lists/listinfo/opticks-devs
 
 -----Inline Attachment Follows-----
 
 ------------------------------------------------------------------------------
 
 -----Inline Attachment Follows-----
 
 _______________________________________________
 Opticks-devs mailing list
 [hidden email]
 https://lists.sourceforge.net/lists/listinfo/opticks-devs
 
 -----Inline Attachment Follows-----
 
 ------------------------------------------------------------------------------
 
 -----Inline Attachment Follows-----
 
 _______________________________________________
 Opticks-devs mailing list
 [hidden email]
 https://lists.sourceforge.net/lists/listinfo/opticks-devs
 

------------------------------------------------------------------------------
_______________________________________________
Opticks-devs mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/opticks-devs
Loading...