We have begun rolling out our product which makes use of yours (version 8.0.0.211) for drawing a map. We now have 3 different customers reporting OutOfMemoryException, all with the same terminal call stack included below. The maps are not really very involved, just a collection of shape file layers and a few InMemoryFeatureLayer instances, and it does include an implementation of your min-map to provide the user location context when zoomed. I’ve run our application in a memory profiler, and nothing obvious pops. Obviously the map may simply be the likely victim when memory runs short, but it does seem unusual that it happens with this call stack each and every time. Do you have any open issues that may relate to this?
System.OutOfMemoryException: Out of memory.
at System.Drawing.Graphics.CheckErrorStatus(Int32 status)
at System.Drawing.Graphics.DrawImage(Image image, Int32 x, Int32 y)
at System.Drawing.Graphics.DrawImageUnscaled(Image image, Int32 x, Int32 y)
at ThinkGeo.MapSuite.DesktopEdition.WinformsMap./xM=(Graphics lxc=, RectangleShape mBc=)
at ThinkGeo.MapSuite.DesktopEdition.WinformsMap.sBM=(RectangleShape Wxc=, RectangleShape XBc=)
at ThinkGeo.MapSuite.DesktopEdition.WinformsMap.9xM=(RectangleShape lRc=)
at ThinkGeo.MapSuite.DesktopEdition.WinformsMap.8xM=(Int32 kRc=, RectangleShape khc=)
at ThinkGeo.MapSuite.DesktopEdition.WinformsMap.Refresh(RectangleShape extent, IEnumerable`1 overlays)
at ThinkGeo.MapSuite.DesktopEdition.WinformsMap.Refresh(RectangleShape extent, Overlay overlay)
at ThinkGeo.MapSuite.DesktopEdition.WinformsMap.Refresh(Overlay overlay)
OutOfMemoryException
Hi Russ,
Like the exception message shows, out of memory is thrown from system dll, we cannot look into it. But our developers will double check refresh related code for try whether we can solve it.
From code it looks it thrown when you are trying to render background bitmap. If your code set special background layer please let me know, which should be helpful message.
Regards,
Don
We do have a BackgroundLayer on the mini-map. It’s just used to provide the background color. Our background image is just provided by the first layer which is a GdiPlusRasterLayer. With your obfuscation, it’s not clear what exactly is involved in producing the error. Any additional information would be appreciated.
We now have this on 4 sites, all resulting from the same terminal call stack. When I run our product in a memory profiler with data/state backups from these sites, performing operations that I’m told lead up to the OOM, it runs along at a relatively steady state of memory usage about where and how I would expect it to perform if there were no leaks. There was no hint of constantly climbing memory consumption, and nothing obvious in object counts etc. And at least one occurrence is confirmed to have happened within an hour of starting the process, so it should be pretty obvious in the memory profiler.
And I just noticed, at least one of the sites that experienced this actually does not use the background image, so the Raster Layer is not added. Other than the background layer of the mini-map, which just has a brush applied, there is nothing like a "background bitmap" in use.
Hi Russ,
I am sorry we still hadn’t reproduced that local.
Does this exception is frequently thrown or only occasionally been thrown when memory usage is heavy?
If possible, could you please try to disable mini map and see whether it will been thrown again. I am not sure whether it’s caused by mini map, but we can only try to locate where is the problem because the call stack information don’t contains more information.
Regards,
Don
We have not reproduced this in house, and the mini map is not configurable on client sites. We would need a custom build to be deployed, and as I’m sure you are aware, that requires considerable bureaucratic effort. Without stronger reasoning, I don’t see that happening soon.
Could you at least provide me with a de-obfuscated version of that call stack? Maybe then I would better understand what’s being done, and come up with ways to investigate.
We just got a new OOM report. Again, this one has almost the same call stack, but not exactly. The only difference appears to be the line "ThinkGeo.MapSuite.DesktopEdition.WinformsMap.shM=" just before calling into the BCL. Hopefully you can provide the non-obfuscated names, and maybe we can figure out what is causing this.
System.OutOfMemoryException: Out of memory.
at System.Drawing.Graphics.CheckErrorStatus(Int32 status)
at System.Drawing.Graphics.DrawImage(Image image, Int32 x, Int32 y)
at System.Drawing.Graphics.DrawImageUnscaled(Image image, Int32 x, Int32 y)
at ThinkGeo.MapSuite.DesktopEdition.WinformsMap.shM=(IEnumerable`1 XRc=, RectangleShape Xhc=)
at ThinkGeo.MapSuite.DesktopEdition.WinformsMap.sBM=(RectangleShape Wxc=, RectangleShape XBc=)
at ThinkGeo.MapSuite.DesktopEdition.WinformsMap.9xM=(RectangleShape lRc=)
at ThinkGeo.MapSuite.DesktopEdition.WinformsMap.8xM=(Int32 kRc=, RectangleShape khc=)
at ThinkGeo.MapSuite.DesktopEdition.WinformsMap.Refresh(RectangleShape extent, IEnumerable`1 overlays)
at ThinkGeo.MapSuite.DesktopEdition.WinformsMap.Refresh(RectangleShape extent, Overlay overlay)
at ThinkGeo.MapSuite.DesktopEdition.WinformsMap.Refresh(Overlay overlay)
Hi Russ,
I am sorry we cannot get a non-obfuscated version dll. And in fact, from your call stack information, our developer can locate where is the exception thrown, and from the information, it looks the non-obfuscated names is not helpful.
It looks this time it is thrown when we draw other overlay after we complete draw background layer. It call Graphics.DrawImageUnscaled function and get this exception.
I want to know whether you met same issue in original version dll? And if possible please let us know your dll version.
Because from the code compare, it looks we don’t changed related code for long time. But our developer still will look into code to find possible resource not release issue.
BTW, do have you the record of memory usage status before get this exception? I want to make sure whether it’s a memory leak.
Regards,
Don
I’m not asking for a non-obfuscated DLL, I’m asking for a clear non-obfuscated version of the call stack I sent you. That way I can see what is happening, and maybe guess at a course of action. At this point all I know is that it ALWAYS happens in “Refresh”. And we now have more sights reporting the same problem always with a call stack as provided.
This issue has only recently surfaced. The original version use for our old map was from before the massive rewrite, and never had this problem. Up until recently, we had not experienced it on the new map component. At this time don’t have anything to indicate a common theme, or anything that changed leading up to this.
Our current version of MapSuiteCore is 8.0.0.211.
Up to this point, it does not appear to be a memory leak. I’ve run some of the reproducing site’s data and map backups in a memory profiler, and there is not indication of excessive memory consumption over a long period of intense interaction. I’ve also deployed tools to the problem sites that can take periodic snapshots of system and process resources. The machines have been rebooted, the program started, and a baseline snapshot established. Once they have a repro, we’ll get a snapshot before the dismiss the exception notification. I have not yet had a post crash report, but at this point I don’t expect it to be a real resource issue. I more suspect an invalid image is being passed that causes a false OOM to be generated.
Hi Russ,
The /xM= is the function DrawBackgroundBitmap and the shM= is the function DrawOverlaysForSingleThread. It looks you are not using multiply tile mode. The other obfuscated function is not helpful.
We viewed the latest change of it, we only remove a type cast last year, this change have no effect on any logic, I think if this is the reason of why you get the OOM, that should be a false exception just like you said. Maybe in original code it won’t log it but after we modify the code, it will thrown exception.
I will try to roll the change back, please get the version 9.0.43.0 for test.
Any update please let me know.
Regards,
Don
I somehow missed this post, sorry for not getting back. What is the current product release version, and will it include this change?
In the meantime I’ve updated to 9.0.0.58. But this has a number of breaking changes, and I can find no suggestions on workarounds. Do you have a document that covers this? Specifically obsolete functionality like:
InMemoryFeatureLayer.Columns: I’m using your TrackInteractiveOverlay, so do not construct the TrackShapeLayer in my code.
AreaStyles.CreateSimpleAreaStyle: I use this to create styles with properties required.
LineStyles.CreateSimpleLineStyle: Likewise.
AreaStyles.CreateHatchStyle: And so on.
I would expect there to be a document somewhere that covers breaking changes, and replacement options. You message in the Obsolete attribute seems completely useless. It may have “better looking styles”, but seems to provide no feature for creating custom styles.
BTW, just to be complete, we have the same problem with PointStyle.
And we have a similar issue with TrackShapeLayer.Columns in a class derived from you TrackInteractiveOverlay. It marks Columns obsolete and insists we pass in columns through the ctor, but we don’t construct that layer instance, you do.
Hi Russ,
We only roll back the change in development version, please try 9.0.58.0 but not 9.0.0.58.
Please just ignore the "obsolete" information in 9.0, the APIs maybe will be modified or removed in 10.0 but not actually will be.
Regards,
Don
When will it make it to the production build? We are very late in a release cycle, and this seemingly random OOM coming out of the TG map component Refresh is the only thing keeping us from closing this Release. And there is no consistent Repro, so I can’t just test the dev build to see if it fixes things. We’ve got sites that sometimes reproduce in a day or two, but may take a week, and others that usually don’t crash for a week or so. Over time leading up to the crash, we don’t see a rise in resources (memory, handles, GDI, etc) and there is no evidence of a “real OOM” when it does crash, but always inside the ThinkGeo Map Refresh method while calling one of the DrawImage routines. Either we’ve missed something, or everything points to a “fake” OOM.
Hi Russ,
In fact we won’t moved any changes to release branch before it’s proved safe and stable, but for this change it’s very small change and it only include a force type cast, so I have moved it to release branch.
Important, it won’t meant moved this change to release can solve the OOM issue. Please let us know whether that’s helpful.
Please get the change from 9.0.0.65 or higher version.
Regards,
Don
Thanks, I’ll be looking for that version. I understand about that protocol for moving to production. I’m still not sure where this OOM is coming from, and hopefully this is a magic bullet. It repro’s fairly rarely, and indications are that there should be no OOM condition (system resource dumps, and profilers, and perfmon, etc). But it always manifests with almost identical “Refresh” call stacks (varying with one of 2 last calls before calling BCL). I don’t have any answers yet, but I’m ready for this to go away.
Hi Russ,
Thanks for your keep research on it. The out of memory related issue is very hard to locate and solve.
If this works for you please let us know.
Regards,
Don
We did not experience the OOM after updating to 9.0.0.66. Not ever getting any clear information about the specific cause of the problem, we were hesitant to declare it fixed, but eventually due to release pressures we did release it hoping for the best. Unfortunately, just as we feared, we got it again. This call stack also starts with Refresh, but terminates in a different obfuscated named method, but based on the arguments, I would say it’s the same call stack from the original report.
Please make this a priority. This has already consumed huge resources and delayed our release by months. All information gathered from resource monitoring at the point of crash, profiling, and many other types of investigation indicate that this should not be an actual OOM. And even if it was, the odds that it would ALWAYS manifest from a call to Refresh on your component would seem very unlikely.
System.OutOfMemoryException: Out of memory.
at System.Drawing.Graphics.CheckErrorStatus(Int32 status)
at System.Drawing.Graphics.DrawImage(Image image, Int32 x, Int32 y)
at System.Drawing.Graphics.DrawImageUnscaled(Image image, Int32 x, Int32 y)
at ThinkGeo.MapSuite.DesktopEdition.WinformsMap.lxQ=(Graphics graphics, RectangleShape extent)
at ThinkGeo.MapSuite.DesktopEdition.WinformsMap.dBQ=(RectangleShape drawingExtent, RectangleShape extent)
at ThinkGeo.MapSuite.DesktopEdition.WinformsMap.lBQ=(RectangleShape extent)
at ThinkGeo.MapSuite.DesktopEdition.WinformsMap.khQ=(Int32 delayInterval, RectangleShape extent)
at ThinkGeo.MapSuite.DesktopEdition.WinformsMap.Refresh(RectangleShape extent, IEnumerable`1 overlays)
at ThinkGeo.MapSuite.DesktopEdition.WinformsMap.Refresh(RectangleShape extent, Overlay overlay)
at ThinkGeo.MapSuite.DesktopEdition.WinformsMap.Refresh(Overlay overlay)
Russ,
What the platform did you compile your project, X64, X86 or “Any CPU”? If “Any CPU”, can you let me know the system threw the OOM exception is 64bit or 32bit?
Each 32bit process has a maximum of 2GB available as Virtual Address Space (VAS), this 2GB is the ceiling for the code pages as well as Data pages, like everything, even it has a huge amount of memory physically. I used to have this issue when my 32bit project seems taking much less memory than 2G, but still throw the OOM exception. I know you have a pretty big project and that draw image method might be the last straw.
So if it’s X86, please switch to “Any CPU” and have another try. I will check the source code anyway.
Ben
All Systems are W764, and the application is compiled x86 due to a dependency on a 3rd party component. So we cannot switch to “Any CPU”.
However, I don’t think that would matter. I am aware of the system limitations based on Platform. At the time of the OOM exception, the process hosting your map component was using a 272,044K total memory allocated, and total allocated process handles were less than 6500. Neither anywhere near where you would expect to be anywhere near an OOM.