Is there anything that can be done to speed up building indexes for Nautical Charts, while managing their resources?
I’m seeing considerable time, CPU, and memory usage during the process; up to 1600x the source file size in memory used with at least one core maxed out for several minutes as the source file size increases.
I’m aware that file size != complexity but the numbers seem excessive!
I’m testing using a selection of files from the NOAA data set (link @ bottom) and my tests produce these numbers;
US4FL12M.000 (0.25Mb):
Building Index: 4.86s
Peak Memory: 0.23Gb
Output: 0.02Mb
US5FL49M.000 (0.46Mb)
Building Index: 9.29s
Peak Memory: 0.22Gb
Output: 0.03Mb
US4FL18M.000 (0.99Mb)
Building Index: 66.28s
Peak Memory: 0.56Gb
Output: 0.06Mb
US3GC06M.000 (1.42Mb)
Building Index: 227.89s
Peak Memory: 5.87Gb
Output: 0.12Mb
US4FL14M.000 (2.35Mb):
Building Index: 358.72s
Peak Memory: 8.59Gb
Output: 0.16Mb
US5AK4JM.000 (4.85Mb):
Building Index: 669.95s
Peak Memory: 8.00Gb
Output: 0.40Mb
I’ve shared my test program and the maps I’ve been using. If you could shed some light on whether I’m doing something stupid I’d appreciate it. 11 minutes and 8Gb of memory to produce a 400Kb index from a 5Mb file seems wildly excessive - I don’t think I can justify that kind of overhead!
Thanks for the update. Figured it was the hyr file.
I sure hope it can be optimized (tbh I’m struggling to imagine how it takes that long!) As a side note, other products I’ve evaluated are loading and displaying the biggest chart in my set in <5 seconds with negligible memory usage.
We have observably improved the performance of our index building, please see our test result:
US4FL12M.000 (0.25Mb):
Old Building: 4.81s
Now Building: 3.1s
Peek Memory: 0.04GB
US5FL49M.000 (0.47Mb):
Old Building: 10.369s
Now Building: 4.059s
Peek Memory: 0.06GB
US4FL18M.000 (1.04Mb):
Old Building: 69.665s
Now Building: 13.116s
Peek Memory: 0.17GB
US3GC06M.000 (1.49Mb):
Old Building: 181.173s
Now Building: 61.167s
Peek Memory: 0.49GB
US4FL14M.000 (2.40Mb):
Old Building: 339.822s
Now Building: 37.4s
Peek Memory: 0.37GB
US5AK4JM.000 (4.97Mb):
Old Building: 695.392s
Now Building: 230.263s
Peek Memory: 0.84GB
You can get our latest development version (11.0.0-beta011) of ThinkGeo.MapSuite.Layers.NauticalCharts and test it on your side.
Please let me know any further questions.
After we dig into the code further, we have made more improvements in memory and performance for BuildIndex method. Now we can keep the memory stable and under a reasonable/accept level. For performance, we improve some by increasing the usage of CPU. Here is the new test result on our side. The latest version is 11.0.0-beta012.
US4FL12M.000 (0.25Mb):
Time (11.0.0-beta010): 4.81s
Time (11.0.0-beta011): 3.1s
Time (Latest): 2.25s
Maybe you are curious why we have to build the .hyr file, according to the .000 file. Here are the reasons:
• ThinkGeo Nautical Charts SDK strictly follows the specification from IHO https://www.iho.int/iho_pubs/draft_pubs/S-52_e6.1.0/S-52_e6.1.0_and_PresLib_e4.0.0.htm. It’s rich in the symbols and you can easily to switch between different modes, show or hide different features, according to your requirements. To support so many different, complicated but colorful symbols, all are calculated and drawn on the fly, thus performance consumed. So we analysis all features in the *.000 and create the drawing instructions of each feature in BuildIndex method, there it has some geometry processing codes, so a bit slow when building the index, but the good news is that you will experience a pretty good drawing performance when showing them up in the application. You just need to build the index only once. I’m not sure whether you tried the same product from ArcGIS, we have a much better drawing performance.
Thanks for the update, although I note that the times have crept up again - personally I’d be happy to trade off (some) memory usage for faster performance, but of course I’d love both
We created a new version (11.0.0-beta013) with a great performance boost and memory deduction (So you now have both ). Please get it from Nuget and have a try, I’m sure you will love it. Below is our test result:
US4FL12M.000 (0.25Mb):
Time (11.0.0-beta010): 4.81s
Time (11.0.0-beta011): 3.1s
Time (11.0.0-beta012): 2.25s
Time (11.0.0-beta013): 0.75s
). Please get it from Nuget and have a try, I’m sure you will love it. Below is our test result: