I somehow missed this post, sorry for not getting back. What is the current product release version, and will it include this change?
In the meantime I’ve updated to 9.0.0.58. But this has a number of breaking changes, and I can find no suggestions on workarounds. Do you have a document that covers this? Specifically obsolete functionality like:
InMemoryFeatureLayer.Columns: I’m using your TrackInteractiveOverlay, so do not construct the TrackShapeLayer in my code.
AreaStyles.CreateSimpleAreaStyle: I use this to create styles with properties required.
LineStyles.CreateSimpleLineStyle: Likewise.
AreaStyles.CreateHatchStyle: And so on.
I would expect there to be a document somewhere that covers breaking changes, and replacement options. You message in the Obsolete attribute seems completely useless. It may have “better looking styles”, but seems to provide no feature for creating custom styles.
OutOfMemoryException
BTW, just to be complete, we have the same problem with PointStyle.
And we have a similar issue with TrackShapeLayer.Columns in a class derived from you TrackInteractiveOverlay. It marks Columns obsolete and insists we pass in columns through the ctor, but we don’t construct that layer instance, you do.
Hi Russ,
We only roll back the change in development version, please try 9.0.58.0 but not 9.0.0.58.
Please just ignore the "obsolete" information in 9.0, the APIs maybe will be modified or removed in 10.0 but not actually will be.
Regards,
Don
When will it make it to the production build? We are very late in a release cycle, and this seemingly random OOM coming out of the TG map component Refresh is the only thing keeping us from closing this Release. And there is no consistent Repro, so I can’t just test the dev build to see if it fixes things. We’ve got sites that sometimes reproduce in a day or two, but may take a week, and others that usually don’t crash for a week or so. Over time leading up to the crash, we don’t see a rise in resources (memory, handles, GDI, etc) and there is no evidence of a “real OOM” when it does crash, but always inside the ThinkGeo Map Refresh method while calling one of the DrawImage routines. Either we’ve missed something, or everything points to a “fake” OOM.
Hi Russ,
In fact we won’t moved any changes to release branch before it’s proved safe and stable, but for this change it’s very small change and it only include a force type cast, so I have moved it to release branch.
Important, it won’t meant moved this change to release can solve the OOM issue. Please let us know whether that’s helpful.
Please get the change from 9.0.0.65 or higher version.
Regards,
Don
Thanks, I’ll be looking for that version. I understand about that protocol for moving to production. I’m still not sure where this OOM is coming from, and hopefully this is a magic bullet. It repro’s fairly rarely, and indications are that there should be no OOM condition (system resource dumps, and profilers, and perfmon, etc). But it always manifests with almost identical “Refresh” call stacks (varying with one of 2 last calls before calling BCL). I don’t have any answers yet, but I’m ready for this to go away.
Hi Russ,
Thanks for your keep research on it. The out of memory related issue is very hard to locate and solve.
If this works for you please let us know.
Regards,
Don
We did not experience the OOM after updating to 9.0.0.66. Not ever getting any clear information about the specific cause of the problem, we were hesitant to declare it fixed, but eventually due to release pressures we did release it hoping for the best. Unfortunately, just as we feared, we got it again. This call stack also starts with Refresh, but terminates in a different obfuscated named method, but based on the arguments, I would say it’s the same call stack from the original report.
Please make this a priority. This has already consumed huge resources and delayed our release by months. All information gathered from resource monitoring at the point of crash, profiling, and many other types of investigation indicate that this should not be an actual OOM. And even if it was, the odds that it would ALWAYS manifest from a call to Refresh on your component would seem very unlikely.
System.OutOfMemoryException: Out of memory.
at System.Drawing.Graphics.CheckErrorStatus(Int32 status)
at System.Drawing.Graphics.DrawImage(Image image, Int32 x, Int32 y)
at System.Drawing.Graphics.DrawImageUnscaled(Image image, Int32 x, Int32 y)
at ThinkGeo.MapSuite.DesktopEdition.WinformsMap.lxQ=(Graphics graphics, RectangleShape extent)
at ThinkGeo.MapSuite.DesktopEdition.WinformsMap.dBQ=(RectangleShape drawingExtent, RectangleShape extent)
at ThinkGeo.MapSuite.DesktopEdition.WinformsMap.lBQ=(RectangleShape extent)
at ThinkGeo.MapSuite.DesktopEdition.WinformsMap.khQ=(Int32 delayInterval, RectangleShape extent)
at ThinkGeo.MapSuite.DesktopEdition.WinformsMap.Refresh(RectangleShape extent, IEnumerable`1 overlays)
at ThinkGeo.MapSuite.DesktopEdition.WinformsMap.Refresh(RectangleShape extent, Overlay overlay)
at ThinkGeo.MapSuite.DesktopEdition.WinformsMap.Refresh(Overlay overlay)
Russ,
What the platform did you compile your project, X64, X86 or “Any CPU”? If “Any CPU”, can you let me know the system threw the OOM exception is 64bit or 32bit?
Each 32bit process has a maximum of 2GB available as Virtual Address Space (VAS), this 2GB is the ceiling for the code pages as well as Data pages, like everything, even it has a huge amount of memory physically. I used to have this issue when my 32bit project seems taking much less memory than 2G, but still throw the OOM exception. I know you have a pretty big project and that draw image method might be the last straw.
So if it’s X86, please switch to “Any CPU” and have another try. I will check the source code anyway.
Ben
All Systems are W764, and the application is compiled x86 due to a dependency on a 3rd party component. So we cannot switch to “Any CPU”.
However, I don’t think that would matter. I am aware of the system limitations based on Platform. At the time of the OOM exception, the process hosting your map component was using a 272,044K total memory allocated, and total allocated process handles were less than 6500. Neither anywhere near where you would expect to be anywhere near an OOM.
Since we thought it went away with the 9.0.0.66 update, after not seeing it for 2 weeks on sites that had reproduced roughly once a week, the decision was made to go ahead with the long delayed roll out. It’s only been a short time, and we’ve gotten dozens of reports now. And they all continue to look like this (taken from a report I got just minutes ago)…
System.OutOfMemoryException: Out of memory.
at System.Drawing.Graphics.CheckErrorStatus(Int32 status)
at System.Drawing.Graphics.DrawImage(Image image, Int32 x, Int32 y)
at System.Drawing.Graphics.DrawImageUnscaled(Image image, Int32 x, Int32 y)
at ThinkGeo.MapSuite.DesktopEdition.WinformsMap.dRQ=(IEnumerable`1 drawingOverlays, RectangleShape extent)
at ThinkGeo.MapSuite.DesktopEdition.WinformsMap.dBQ=(RectangleShape drawingExtent, RectangleShape extent)
at ThinkGeo.MapSuite.DesktopEdition.WinformsMap.lBQ=(RectangleShape extent)
at ThinkGeo.MapSuite.DesktopEdition.WinformsMap.khQ=(Int32 delayInterval, RectangleShape extent)
at ThinkGeo.MapSuite.DesktopEdition.WinformsMap.Refresh(RectangleShape extent, IEnumerable`1 overlays)
at ThinkGeo.MapSuite.DesktopEdition.WinformsMap.Refresh(RectangleShape extent, Overlay overlay)
at ThinkGeo.MapSuite.DesktopEdition.WinformsMap.Refresh(Overlay overlay)
As you know, we had to do a clean sheet rewrite of our map that got stranded years ago when Think Geo did a clean sheet rewrite of your map components (version 3.0 IIRC). I wrote a lot of code in a short time and I am not at all averse to the idea that I may have in some way contributed to this. But as far as I can tell there is no actual OOM condition. I’ve examined the various memory pools, and memory totals. Handle counts. Everything I could dream up and it all looks well within expected norms. And nothing I have tried has given me any insight into why this is occurring, much less how to fix it. I also have no idea what was changed in the 66 build, or why it made repro much less likely. But we NEED a solution of some sort very soon, and with every single occurrence being that call stack (with minor variation in the last 2 calls), the focus is squarely on your map control at this point.
Russ,
Here is what I found:
1. As you confirmed the memory looks good, so this must be a known Microsoft issue that OOM doesn’t really mean Out of Memory. It might because of an invalid image or inappropriate operation, many people were complaining about this:
stackoverflow.com/questions/6506089/system-drawing-out-of-memory-exception
stackoverflow.com/questions/3665206/why-is-my-c-sharp-paint-method-running-out-of-memory
2, I checked the code and all the exceptions was thrown in the following method, I think the issue might be from the bitmap it passed in but I’m not sure. I checked the code changes but didn’t find any clue that would bring in this issue, or partly fix it. Giving we didn’t get any OOM exception report from other users, I’m thinking it’s some certain code/data in the project made this code throw the exception.
bufferedGraphics.Graphics.DrawImageUnscaled(Overlay.ScreenBitmapTile.Bitmap, x, y)
I think the best way is to create a customized assembly with whole bunch of debugging trace, using that assembly in your application, let exception throw and collect all the traced information. That would help us find out which overlay got this exception, what bitmap and what kind of graphics that is. How does that sound? it might take a day or two to finish that customized assembly.
Ben
I agree on all points. I’ve actually seen an OOM before that was thrown (in MS code) due to an assumption that a null pointer from an internal allocation method must indicate an OOM condition. But it was actually a handle limit that was hit. Which is why I have already setup for dumps of handle counts (process, GDI, etc) at the sites experiencing the problems. But it could also be a malformed bitmap if it’s internal metadata and structure are not correct. Something along the lines of what happened in C/C++ if you didn’t null terminate a string.
Anyway, thanks for the update. I’m putting things in place so that we can do a custom build with your instrumented build and deploy to key problem sites. We’ve already done such with my instrumented builds trying to identify related patterns, but like the resource dumps, mini-dumps, and profiling sessions, the results didn’t really provide any further insight. Hopefully you will have more luck.
Russ, just sent you the customized assembly through email, please run the new assembly, let the exception thrown and send me back the trace information.
This has the same problem as last time a private assembly was sent to me.
First off, it looks like you guys change the public interface frequently even without an update in major/minor versions. In this case, you have removed a previous virtual function for DrawCore(ThinkGeo.MapSuite.Core.GeoCanvas). I had overloaded this, so that makes this change an undocumented breaking change. So either you will need to do the same for a build matching our version, or I will have to take time to make changes to address your breaking changes, which then could invalidate or change the characteristic of the test. In my opinion, this undocumented breaking changes frequently surfacing from build to build is really something that should be addressed. It makes us VERY reluctant to update to your newer builds. At the very least they should be documented, including suggestions on how the now missing functionality can be replicated after the change.
Also, once we got past this on the last similar attempt, there were 2 subsequent errors that had to be resolved and wasted more time. First, all our assemblies are signed (“strong named”), so dependent assemblies must be strong named. I took a peak at that manifest of this single DLL, and confirmed that the assembly you sent is not strong named, so it would not have worked anyway.
It has also become apparent in past attempts to try a custom binary that the “public interface change breaks referencing consumers” also adversely affects your internal dependencies. And because of that, you can’t give me a single binary to try. Every time this has been attempted, there were missing public entry point errors, and days of wasted delays. And of course there are the strong name references that also present a problem for single binary distribution (not insurmountable, but avoidable).
So, to get this to work, I’ll need a complete set of matching Desktop related binaries just as if I had gotten them from a standard deployment. And they will need to be signed (strong named) or the loader will refuse to load them in our project. And if you provide an update build based on current code, please also include instructions for addressing the missing DrawCore. That will save me having to sort it out on my own, potentially risking introducing new unknowns in the process.
Also, please be aware that we won’t be able to put this out for general release. That release and the resulting large number of users is what generated the sudden series of reports. Our beta testers, the only place we are going to reasonably be able to test this have not reported a single OOM since the last “maybe it will help” update to build 9.0.0.66. I have no idea why that build seemed to help, but it basically killed our already infrequent repro (maybe a day, probably 4 to 6, or maybe 2 weeks). The only thing we were fairly confident in was that at sites that repro, every time it was in less than 2 weeks, usually a lot less. That’s what caused management to accept the risk of releasing the product based only on running for 3 weeks with no repro. Unfortunately, my pessimism in warning about the risk proved well founded when the reports started flooding in.
I just wanted to include that for background, and to let you know that even when we get a buildable set, putting this out there may not get a fast response. Hopefully useful information can be logged without the need for an actual OOM triggering the logging. If that’s all there is, it may take a while. So please consider what might be useful to collect and study in the intervening time.
I have no clue about the breaking change you mentioned. The version
I sent to you had an identical API with 9.0.0.0, (I’m pretty sure with 9.0.0.66
as well). What exactly is that DrawCore(ThinkGeo.MapSuite.Core.GeoCanvas)
method, which class is it in? I don’t
believe we would have got rid of any major methods like that.
We totally understand the importance of avoiding breaking
change, we had a strict check to avoid breaking changes between 2 public releases.
Whenever removing an API, we would mark it as obsolete in one public version and
remove it in the next. For example we are now in Map Suite 9.0, to get rid of
an API today I would mark it obsolete but still keep it in Map Suite 10.0, and
actually remove it in Map Suite 11.0. All
the API changes between the major releases have been logged here. wiki.thinkgeo.com/wiki/Map_Suite_Desktop_Edition_Release_Change_Log
The dev branch daily build might have breaking change but
release branch should not. All the APIs in release branch should be compatible (only
new APIs, no remove or edit) with the major release. Can you give me one example of any breaking changes
you see in release branch?
Non strong named APIs are more popular among our customers, that’s
why we provide non strong named assemblies unless addressed.
I sent you a single dll because I don’t want to bring in
more uncertainties from other assemblies, I believed the fewer changes the
better. I tested just replacing that single
assembly works with the other “old” assemblies.
So please show me more info about the
breaking change issue you got (DrawCore) , I’m very interested to see why you
have so many breaking change issues in the update, which are not supposed to be
there at all.
Sorry, I expected that would be enough without further context. It’s from TrackInteractiveOverlay. I have a class that derives from your TrackInteractiveOverlay, and overrides DrawCore. When I build with 9.0.0.66, there is no error. When I build with your updated 9.0.0.100 DesktopEdition.DLL, I get a compile error “no suitable method found to override” for that method.
And I’ve run into this before. We also had problems with our previous update of your library where events didn’t fire in the same conditions, the discussion is on this board. Other cases, as you say, with marking obsolete. But some of those were far to broad in scope allowing no alternative for some functionality. Anyway, we’ve found that updating is something we do only as necessary, and if possible early in a product cycle. Good words to live by regardless, but we’ve been bitten on this a few times.
And I agree on introducing the fewest changes. But I’m guessing when you tested it working with old assemblies, that was not a signed scenario. Again, to introduce least change, I can’t use it without being strong named, and that means I need a cohesive build. The most convenient for me would be your changes in the source base for 66.
I’m surprised you didn’t see it. it’s in the source code, it can be seen through a reflection tool, I can create a sub class and override that method in a test project. Here below is that assembly in ILSpy. Can you check again if you were using the correct assembly? 
We had a strict API test to avoid API breaking change in every release. I agree with you that sometime we might have feature breaking change, like an event was raised in a previous version but not in a newer version. We tried to avoid that but I admitted that happened sometimes, as there is not a tool to detect it and our UnitTest doesn’t cover all the APIs/Scenarios.
The assembly I sent to you was based on 9.0.0.100, as we are reluctantly to make changes in release branch, this one should have few changes with 9.0.0.66. Let me create a strong name version and have another try.
Interesting, so it is.
But it was not me that was looking for it, this is a compiler error I get when I substitute the old 66 DesktopEdition.dll with your new 100 version. I switch back, the error goes away. Just tried it again, same result. The compiler is not happy, and the last time such a thing happened with a single DLL update it only broke again due to references between your assemblies after I figured out how to restructure my code for the change. So when it happened on this version, I didn’t look further. At this point, I’m not sure why it’s giving that error since the method is obviously there. And to offer further confusion, if I remove the old overload and see what intellisense says, it provides the same signature again, and then the compiler complains the same way again. So VS “sees” it, but the compiler does not. I wonder if it’s somehow a invalid/inaccurate response to the lack of strong naming in the new assembly?
So, just swapping in/out the 66/100 assemblies causes it to come and go, with no other changes. But using several different reflection options, they all show the overload signature remains intact. Wow, not sure what to say on that one, except the compiler is still offering the same error…
Edit Update:
On a hunch, I decided to “fix” that error by simply removing override and, as expected, then got a complaint about hiding the method in the base class. Rename to get past that, and then everything went off the rails. Line after line of “does not contain”, “cannot convert”, etc. Then the final error, “Assembly generation failed – Referenced assembly ‘DesktopEdition’ does not have a strong name”.
Something about the replacement of that assembly with a non-strong named assembly appears to have somehow triggered the fist singular error that completely errored out the compiler, preventing it from continuing, and blocked the deluge of errors including the real root cause, lack of strong name. I don’t see any other explanation than that the compiler choked in an unexpected way due to the substitution…