ThinkGeo.com    |     Documentation    |     Premium Support

Out of Memory for Multipolygonshape

Hi,

I am building some rather large multipolygon shapes and it looks like I hit a limit on how many polygons can be contained. Can you advice if this is expected or a bug? Here’s a snipit that produces the behavior. It does not fail at the same location every time, so I would like to know how I can anticipate the limit so I can break the shapes up accordingly.

MultipolygonShape mpg = new MultipolygonShape();
for (int i = 0; i < 27000000; i++)
{
RingShape ring = new RingShape();
ring.Vertices.Add(new Vertex(0, 0));
ring.Vertices.Add(new Vertex(0, 5));
ring.Vertices.Add(new Vertex(5, 5));
ring.Vertices.Add(new Vertex(5, 0));
PolygonShape pg = new PolygonShape(ring);
mpg.Polygons.Add(pg);
}

Thanks,
Damian

Additional information about the above.

I am adding this layer to an InMemoryFeatureLayer and then using the Draw method to save to a bitmap image.

I got around the polygon memory issue by adding additional features that were subsets of my original polygon set, but when calling the draw command, I get the following error:

Arithmetic operation resulted in an overflow.

Any way around this?

Thanks,
Damian

Hi Damian,

I tested it on my computer whose memory size is 16GB, the limit are the following, if your application is designed to run on the computers with 16GB memory, I think it’s safe to use 16,000,000 as the limit. If you want to run your application to the low-memory machines, you need to reduce it to a small number, like 100,000 or 5,000 to make sure it won’t break.

The reason why it didn’t fail at the same location every time is that the memory was being used dynamically, other apps would occupy memory also.

  1. 16510612
  2. 16258894
  3. 16426681
  4. 16594466
  5. 16678362
  6. 16262379
  7. 16346276
  8. 16597952
  9. 16597950
  10. 16263539

Regarding the “Arithmetic operation resulted in an overflow.” error, could you please post the code snippet or the sample to help us reproduce it?

Thanks
Leo

Hi Leo,

Here’s the code where the arithmetic error happens. Specifically at the Draw function.

Regarding the number of objects, I find it odd that I don’t experience any peaks in memory usage prior to the MultipolygonShape error. I have 64 Gb of Ram on my machine and the failures were very near to your reported 16M with no other “sizeable” programs running. I think it is something else.

        using (Bitmap bitmap = new Bitmap(w, h))
        {
            PlatformGeoCanvas canvas = new PlatformGeoCanvas();
            canvas.BeginDrawing(bitmap, rect, GeographyUnit.Meter);
            imfl.Draw(canvas, new Collection<SimpleCandidate>());
            canvas.EndDrawing();
            bitmap.Save(filename, System.Drawing.Imaging.ImageFormat.Png);
        }

Thanks,
Damian

hi Damian,

I tried your code, it didn’t throw any errors, I was using ThinkGeo.MapSuite 10.6.7. which vesion are you using? and what kind of layer are you using?

Thanks,
Leo

Leo,

Here is the full code minus the part about adding the polygons to the allBins object since that is a data specific process. Essentially, the allBins object has a double value which is used to color code the individual polygons with a class break style. The polygons are all rectangles that create a map. So, you could generically make some code that took a rectangle shape and translated it in rows and columns and randomly assign to values between 0 and 10. The rows times the columns should be greater than 16M. In my specific case, I had 27M cells.

Please look at the memory usage on your machine when you add polygons to the InMemoryFeatureLayer and see if the memory is actually being totally consumed. As I said, mine is not and the maxpg segmentation gets me past the out of memory error…

ConcurrentDictionary<double, MultipolygonShape> allBins = new ConcurrentDictionary<double, MultipolygonShape>();

int maxpg = 15000000;
foreach (var pg in allBins)
{
if (pg.Value.Polygons.Count == 0) continue;

            if (pg.Value.Polygons.Count > maxpg)
            {
                int fc = Convert.ToInt32(Math.Ceiling(pg.Value.Polygons.Count / (float)maxpg));
                for (int i = 0; i < fc; i++)
                {
                    List<PolygonShape> pgs = pg.Value.Polygons.Skip(i * maxpg).Take(maxpg).ToList();
                    MultipolygonShape mpg = new MultipolygonShape(pgs);
                    imfl.InternalFeatures.Add(new Feature(mpg, new List<string>() { offsetName + ":" + pg.Key }));
                }
            }
            else
                imfl.InternalFeatures.Add(new Feature(pg.Value, new List<string>() { offsetName + ":" + pg.Key }));
        }

        Debug.Print(string.Format("Adding bins to imfl took {0:0.0} s", s.Elapsed.Seconds));
        s.Restart();

        imfl.Open();
        RectangleShape rect = imfl.GetBoundingBox();

        // Have to set map size first otherwise it is taking forever if the data are already loaded
        double mapX = rect.LowerRightPoint.X - rect.LowerLeftPoint.X;

        double minBin = model.GridDef.BinX;
        if (model.GridDef.BinY < model.GridDef.BinX)
            minBin = model.GridDef.BinY;

        double ratio = 16 / 9f;
        ratio = rect.Width / rect.Height;

        int w = 5120;
        int h = Convert.ToInt32(w / ratio);

        double pixelsPerBin = minBin * w / mapX;

        while (pixelsPerBin < 4)
        {
            w += 100;
            h = Convert.ToInt32(w / ratio);

            double imageSize = w * h * 4f;
            if (imageSize >= 2.147483648e9)
            {
                w -= 100;
                h = Convert.ToInt32(w / ratio);
                break;
            }

            pixelsPerBin = minBin * w / mapX;
        }

        // Bitmap can't exceed 65535 x 65535
        if (h >= 65535)
        {
            h = 65535;
            w = Convert.ToInt32(h * ratio);
        }

        Debug.Print(string.Format("Building map took {0:0.0} s", s.Elapsed.Seconds));
        s.Restart();

        using (Bitmap bitmap = new Bitmap(w, h))
        {
            PlatformGeoCanvas canvas = new PlatformGeoCanvas();
            canvas.BeginDrawing(bitmap, rect, GeographyUnit.Meter);
            imfl.Draw(canvas, new Collection<SimpleCandidate>());
            canvas.EndDrawing();
            bitmap.Save(filename, System.Drawing.Imaging.ImageFormat.Png);
        }

Regards,
Damian

hi Damian,

Here’s the sample I created according to your code, but regarding the adding logic, I’m not sure if I understand you correctly, Could you please take a look at the sample, correct the code if it’s wrong, and send it to us?

memory_issue.zip (12.5 KB)

Thanks,
Leo

The attachment looks incomplete. It’s just a blank WPF form. Probably easier to create a console app anyway.

Regards,
Damian

Leo,

I have created a project that replicates the error. See attached the project and the error.

I have 64 Gb RAM on my machine and this will use about 30 Gb to complete the build process, but the error is in the Draw.

MemoryIssue.zip (25.8 KB)

Regards,
Damian

Damian,

This issue is recreated and fixed in the latest beta version ThinkGeo.MapSuite 11.0.0-beta154 . The cause of this issue is some intermediate variable exceeds the int limit because of the large size of the polygon.

Please get that version and have a try (you don’t need to update the other packages), Let us know if everything goes well and we will move it over to the release version 10...

Thanks,
Ben

Hi Ben,

Fantastic. It works.

Please let me know when you have it rolled out into v10. Will that be imminent or some days in the future. I have been holding a release based on this issue.

Thanks,
Damian

It can be sometime this week, will keep you posted.

Ben

Hi Damian, the fix is available in ThinkGeo.MapSuite 10.6.20 now.

Thank you. It’s working well.

Regards,
Damian

That’s Great!

Thanks,
Ben