Since we thought it went away with the 9.0.0.66 update, after not seeing it for 2 weeks on sites that had reproduced roughly once a week, the decision was made to go ahead with the long delayed roll out. It’s only been a short time, and we’ve gotten dozens of reports now. And they all continue to look like this (taken from a report I got just minutes ago)…
System.OutOfMemoryException: Out of memory.
at System.Drawing.Graphics.CheckErrorStatus(Int32 status)
at System.Drawing.Graphics.DrawImage(Image image, Int32 x, Int32 y)
at System.Drawing.Graphics.DrawImageUnscaled(Image image, Int32 x, Int32 y)
at ThinkGeo.MapSuite.DesktopEdition.WinformsMap.dRQ=(IEnumerable`1 drawingOverlays, RectangleShape extent)
at ThinkGeo.MapSuite.DesktopEdition.WinformsMap.dBQ=(RectangleShape drawingExtent, RectangleShape extent)
at ThinkGeo.MapSuite.DesktopEdition.WinformsMap.lBQ=(RectangleShape extent)
at ThinkGeo.MapSuite.DesktopEdition.WinformsMap.khQ=(Int32 delayInterval, RectangleShape extent)
at ThinkGeo.MapSuite.DesktopEdition.WinformsMap.Refresh(RectangleShape extent, IEnumerable`1 overlays)
at ThinkGeo.MapSuite.DesktopEdition.WinformsMap.Refresh(RectangleShape extent, Overlay overlay)
at ThinkGeo.MapSuite.DesktopEdition.WinformsMap.Refresh(Overlay overlay)
As you know, we had to do a clean sheet rewrite of our map that got stranded years ago when Think Geo did a clean sheet rewrite of your map components (version 3.0 IIRC). I wrote a lot of code in a short time and I am not at all averse to the idea that I may have in some way contributed to this. But as far as I can tell there is no actual OOM condition. I’ve examined the various memory pools, and memory totals. Handle counts. Everything I could dream up and it all looks well within expected norms. And nothing I have tried has given me any insight into why this is occurring, much less how to fix it. I also have no idea what was changed in the 66 build, or why it made repro much less likely. But we NEED a solution of some sort very soon, and with every single occurrence being that call stack (with minor variation in the last 2 calls), the focus is squarely on your map control at this point.
OutOfMemoryException
Russ,
Here is what I found:
1. As you confirmed the memory looks good, so this must be a known Microsoft issue that OOM doesn’t really mean Out of Memory. It might because of an invalid image or inappropriate operation, many people were complaining about this:
stackoverflow.com/questions/6506089/system-drawing-out-of-memory-exception
stackoverflow.com/questions/3665206/why-is-my-c-sharp-paint-method-running-out-of-memory
2, I checked the code and all the exceptions was thrown in the following method, I think the issue might be from the bitmap it passed in but I’m not sure. I checked the code changes but didn’t find any clue that would bring in this issue, or partly fix it. Giving we didn’t get any OOM exception report from other users, I’m thinking it’s some certain code/data in the project made this code throw the exception.
bufferedGraphics.Graphics.DrawImageUnscaled(Overlay.ScreenBitmapTile.Bitmap, x, y)
I think the best way is to create a customized assembly with whole bunch of debugging trace, using that assembly in your application, let exception throw and collect all the traced information. That would help us find out which overlay got this exception, what bitmap and what kind of graphics that is. How does that sound? it might take a day or two to finish that customized assembly.
Ben
I agree on all points. I’ve actually seen an OOM before that was thrown (in MS code) due to an assumption that a null pointer from an internal allocation method must indicate an OOM condition. But it was actually a handle limit that was hit. Which is why I have already setup for dumps of handle counts (process, GDI, etc) at the sites experiencing the problems. But it could also be a malformed bitmap if it’s internal metadata and structure are not correct. Something along the lines of what happened in C/C++ if you didn’t null terminate a string.
Anyway, thanks for the update. I’m putting things in place so that we can do a custom build with your instrumented build and deploy to key problem sites. We’ve already done such with my instrumented builds trying to identify related patterns, but like the resource dumps, mini-dumps, and profiling sessions, the results didn’t really provide any further insight. Hopefully you will have more luck.
Russ, just sent you the customized assembly through email, please run the new assembly, let the exception thrown and send me back the trace information.
This has the same problem as last time a private assembly was sent to me.
First off, it looks like you guys change the public interface frequently even without an update in major/minor versions. In this case, you have removed a previous virtual function for DrawCore(ThinkGeo.MapSuite.Core.GeoCanvas). I had overloaded this, so that makes this change an undocumented breaking change. So either you will need to do the same for a build matching our version, or I will have to take time to make changes to address your breaking changes, which then could invalidate or change the characteristic of the test. In my opinion, this undocumented breaking changes frequently surfacing from build to build is really something that should be addressed. It makes us VERY reluctant to update to your newer builds. At the very least they should be documented, including suggestions on how the now missing functionality can be replicated after the change.
Also, once we got past this on the last similar attempt, there were 2 subsequent errors that had to be resolved and wasted more time. First, all our assemblies are signed (“strong named”), so dependent assemblies must be strong named. I took a peak at that manifest of this single DLL, and confirmed that the assembly you sent is not strong named, so it would not have worked anyway.
It has also become apparent in past attempts to try a custom binary that the “public interface change breaks referencing consumers” also adversely affects your internal dependencies. And because of that, you can’t give me a single binary to try. Every time this has been attempted, there were missing public entry point errors, and days of wasted delays. And of course there are the strong name references that also present a problem for single binary distribution (not insurmountable, but avoidable).
So, to get this to work, I’ll need a complete set of matching Desktop related binaries just as if I had gotten them from a standard deployment. And they will need to be signed (strong named) or the loader will refuse to load them in our project. And if you provide an update build based on current code, please also include instructions for addressing the missing DrawCore. That will save me having to sort it out on my own, potentially risking introducing new unknowns in the process.
Also, please be aware that we won’t be able to put this out for general release. That release and the resulting large number of users is what generated the sudden series of reports. Our beta testers, the only place we are going to reasonably be able to test this have not reported a single OOM since the last “maybe it will help” update to build 9.0.0.66. I have no idea why that build seemed to help, but it basically killed our already infrequent repro (maybe a day, probably 4 to 6, or maybe 2 weeks). The only thing we were fairly confident in was that at sites that repro, every time it was in less than 2 weeks, usually a lot less. That’s what caused management to accept the risk of releasing the product based only on running for 3 weeks with no repro. Unfortunately, my pessimism in warning about the risk proved well founded when the reports started flooding in.
I just wanted to include that for background, and to let you know that even when we get a buildable set, putting this out there may not get a fast response. Hopefully useful information can be logged without the need for an actual OOM triggering the logging. If that’s all there is, it may take a while. So please consider what might be useful to collect and study in the intervening time.
I have no clue about the breaking change you mentioned. The version
I sent to you had an identical API with 9.0.0.0, (I’m pretty sure with 9.0.0.66
as well). What exactly is that DrawCore(ThinkGeo.MapSuite.Core.GeoCanvas)
method, which class is it in? I don’t
believe we would have got rid of any major methods like that.
We totally understand the importance of avoiding breaking
change, we had a strict check to avoid breaking changes between 2 public releases.
Whenever removing an API, we would mark it as obsolete in one public version and
remove it in the next. For example we are now in Map Suite 9.0, to get rid of
an API today I would mark it obsolete but still keep it in Map Suite 10.0, and
actually remove it in Map Suite 11.0. All
the API changes between the major releases have been logged here. wiki.thinkgeo.com/wiki/Map_Suite_Desktop_Edition_Release_Change_Log
The dev branch daily build might have breaking change but
release branch should not. All the APIs in release branch should be compatible (only
new APIs, no remove or edit) with the major release. Can you give me one example of any breaking changes
you see in release branch?
Non strong named APIs are more popular among our customers, that’s
why we provide non strong named assemblies unless addressed.
I sent you a single dll because I don’t want to bring in
more uncertainties from other assemblies, I believed the fewer changes the
better. I tested just replacing that single
assembly works with the other “old” assemblies.
So please show me more info about the
breaking change issue you got (DrawCore) , I’m very interested to see why you
have so many breaking change issues in the update, which are not supposed to be
there at all.
Sorry, I expected that would be enough without further context. It’s from TrackInteractiveOverlay. I have a class that derives from your TrackInteractiveOverlay, and overrides DrawCore. When I build with 9.0.0.66, there is no error. When I build with your updated 9.0.0.100 DesktopEdition.DLL, I get a compile error “no suitable method found to override” for that method.
And I’ve run into this before. We also had problems with our previous update of your library where events didn’t fire in the same conditions, the discussion is on this board. Other cases, as you say, with marking obsolete. But some of those were far to broad in scope allowing no alternative for some functionality. Anyway, we’ve found that updating is something we do only as necessary, and if possible early in a product cycle. Good words to live by regardless, but we’ve been bitten on this a few times.
And I agree on introducing the fewest changes. But I’m guessing when you tested it working with old assemblies, that was not a signed scenario. Again, to introduce least change, I can’t use it without being strong named, and that means I need a cohesive build. The most convenient for me would be your changes in the source base for 66.
I’m surprised you didn’t see it. it’s in the source code, it can be seen through a reflection tool, I can create a sub class and override that method in a test project. Here below is that assembly in ILSpy. Can you check again if you were using the correct assembly? 
We had a strict API test to avoid API breaking change in every release. I agree with you that sometime we might have feature breaking change, like an event was raised in a previous version but not in a newer version. We tried to avoid that but I admitted that happened sometimes, as there is not a tool to detect it and our UnitTest doesn’t cover all the APIs/Scenarios.
The assembly I sent to you was based on 9.0.0.100, as we are reluctantly to make changes in release branch, this one should have few changes with 9.0.0.66. Let me create a strong name version and have another try.
Interesting, so it is.
But it was not me that was looking for it, this is a compiler error I get when I substitute the old 66 DesktopEdition.dll with your new 100 version. I switch back, the error goes away. Just tried it again, same result. The compiler is not happy, and the last time such a thing happened with a single DLL update it only broke again due to references between your assemblies after I figured out how to restructure my code for the change. So when it happened on this version, I didn’t look further. At this point, I’m not sure why it’s giving that error since the method is obviously there. And to offer further confusion, if I remove the old overload and see what intellisense says, it provides the same signature again, and then the compiler complains the same way again. So VS “sees” it, but the compiler does not. I wonder if it’s somehow a invalid/inaccurate response to the lack of strong naming in the new assembly?
So, just swapping in/out the 66/100 assemblies causes it to come and go, with no other changes. But using several different reflection options, they all show the overload signature remains intact. Wow, not sure what to say on that one, except the compiler is still offering the same error…
Edit Update:
On a hunch, I decided to “fix” that error by simply removing override and, as expected, then got a complaint about hiding the method in the base class. Rename to get past that, and then everything went off the rails. Line after line of “does not contain”, “cannot convert”, etc. Then the final error, “Assembly generation failed – Referenced assembly ‘DesktopEdition’ does not have a strong name”.
Something about the replacement of that assembly with a non-strong named assembly appears to have somehow triggered the fist singular error that completely errored out the compiler, preventing it from continuing, and blocked the deluge of errors including the real root cause, lack of strong name. I don’t see any other explanation than that the compiler choked in an unexpected way due to the substitution…
As you said, it might be an invalid response to the lack of strong name. I’ve sent you the new strong named DesktopEdition.dll through email. I attached its dependencies as well even just replacing single DesktopEdition.dll works in my test. Again I suggest to only replace one assembly to avoid unnecessary uncertainties. Let me know what it turns out. Please be aware the exception is still supposed to be thrown with the new assembly, only debug info will be dumped to the log file.
In the same way that an OOM is sometimes reported when that is not the real cause, I believe that VS was unhappy with the lack of strong name, DrawCore error was it’s imprecise way of complaining about that problem. In any case, replacing only the DE.DLL works just fine in this strong named version. I’ll move forward with the test, and thank you for your help.
It will only log to the log file after OOM is thrown, I catch and rethrow the same exception but add dumping code in. I also fixed a potential issue (A BufferedGraphicsContext object is created in the SizeChanged event but the old object was not disposed, I added the Dispose() to the old object) which might lead to your issue. All the changes are based on source 9.0.0.100.
So if the exception can never be thrown, the above change might do the trick. Otherwise send me the generated log and I can dig more.
Unfortunately, we just had a new report of this problem. It was confirmed as running with the 9.5.0.100 version build.
Here is the call stack.
System.OutOfMemoryException: Out of memory.
at ThinkGeo.MapSuite.DesktopEdition.WinformsMap.LBQ=(Bitmap bitmap, Int32 x, Int32 y, Overlay overlay, String id)
at ThinkGeo.MapSuite.DesktopEdition.WinformsMap.TxQ=(Graphics graphics, RectangleShape extent)
at ThinkGeo.MapSuite.DesktopEdition.WinformsMap.KxQ=(RectangleShape drawingExtent, RectangleShape extent)
at ThinkGeo.MapSuite.DesktopEdition.WinformsMap.TBQ=(RectangleShape extent)
at ThinkGeo.MapSuite.DesktopEdition.WinformsMap.ShQ=(Int32 delayInterval, RectangleShape extent)
at ThinkGeo.MapSuite.DesktopEdition.WinformsMap.Refresh(RectangleShape extent, IEnumerable`1 overlays)
at ThinkGeo.MapSuite.DesktopEdition.WinformsMap.Refresh(RectangleShape extent, Overlay overlay)
at ThinkGeo.MapSuite.DesktopEdition.WinformsMap.Refresh(Overlay overlay)
So it is the first time you got this OOM in the last 18 months?
We have the latest Map Suite 10.0 released which has many bug fixes and new features, and we would highly recommend you to upgrade to that version. Here shows how to upgrade from 9.0 to 10.0. http://wiki.thinkgeo.com/wiki/map_suite_10_upgrade_guide
We are already working on moving to your v10 for our next release. I’ve been following your migration document, but ran into what I believe to be the last few remaining problems with adapting to the structural changes. My question can be seen at:
But we just rolled out a release, and there is no way we are going to adopt such a major update for that release. In fact, the risk imposed by the major structural changes in v10 is why we specifically decided to keep the build we thought fixed this problem for the release we just rolled out.
And I have confirmed the site reporting the problem is running our new release, and has the correct custom ThinkGeo build.
Oh, and in answer to your other question, yes, it’s been over a year since the last report. This is one of the worst sites from our last discussions on the matter, and my first expectation was that it had somehow gotten rolled back to the old version. But no, it’s running the 9.5.0.100 version.
Any ideas on how to proceed? Does the new call stack provide any new insights? The signatures appear different than before (and by more than obfuscation), but that could just be the effect of the changes that were made to fix it last time. I don’t suppose the enhanced logging remains in the *.100 version?
Russ,
This OOM issue if hard to recreate , remember last time I created a specific version for you and it took days to recreate it? I believe this has to be done this time as well. So please either send us the log file, or we need to create a new version for you with loggers. Check my email in 2015 and you can find the specific version I sent you with loggers, I don’t think your current version 9.5.0.100 has it.
I would recommend you to upgrade to the latest 10.X version, as 9.5.0.100 is a customized version just created for you, it’s isolated in the version line and we cannot make changes based on that. If you want to stick to 9.X, please get the latest version 9.0.0.638 then which has all the changes in 9.5.0.100. However just let you know 9.X has lower priority than 10.X now so it might take longer time for the support team to get into it. Sure you can always contact sales@thinkgeo.com see if there’s a way to expedite it.
Ben
Well, bad news. We just got another identical report from another site.
And as I said, we JUST rolled our a new version, so there is no way we are patching that to v10. In fact, we just abandoned the v10 migration for our next version as well (see my other post regarding). Turns out MapEngine wasn’t even visible in the standard deployment, and I was advised to update NuGet (again). After spending days, finding incomplete docs and samples, and many additional problems, we just can’t risk the impact on our next version schedule (which is already tight).
So, back to our v10 (edit: should have said v9) patch. I was hoping the logging might be in the 9.5.0.100 build with some way to just turn it on. So I guess I’ll dig out that old logger build and see if we can get it on the sites having trouble.
The major rip up on your v10 combined with the OOM issue remaining in the 9.0 custom build puts us between a rock and a hard place for both this AND our next versions.