ThinkGeo.com    |     Documentation    |     Premium Support

Nautical Chart Indexes

#1

Hi

Is there anything that can be done to speed up building indexes for Nautical Charts, while managing their resources?

I’m seeing considerable time, CPU, and memory usage during the process; up to 1600x the source file size in memory used with at least one core maxed out for several minutes as the source file size increases.

I’m aware that file size != complexity but the numbers seem excessive!

I’m testing using a selection of files from the NOAA data set (link @ bottom) and my tests produce these numbers;

US4FL12M.000 (0.25Mb):
Building Index: 4.86s
Peak Memory: 0.23Gb
Output: 0.02Mb

US5FL49M.000 (0.46Mb)
Building Index: 9.29s
Peak Memory: 0.22Gb
Output: 0.03Mb

US4FL18M.000 (0.99Mb)
Building Index: 66.28s
Peak Memory: 0.56Gb
Output: 0.06Mb

US3GC06M.000 (1.42Mb)
Building Index: 227.89s
Peak Memory: 5.87Gb
Output: 0.12Mb

US4FL14M.000 (2.35Mb):
Building Index: 358.72s
Peak Memory: 8.59Gb
Output: 0.16Mb

US5AK4JM.000 (4.85Mb):
Building Index: 669.95s
Peak Memory: 8.00Gb
Output: 0.40Mb

I’ve shared my test program and the maps I’ve been using. If you could shed some light on whether I’m doing something stupid I’d appreciate it. 11 minutes and 8Gb of memory to produce a 400Kb index from a 5Mb file seems wildly excessive - I don’t think I can justify that kind of overhead!

Test Program & Maps

Cheers

#2

Hi Kev,

Thanks for your sample and data, but when I tried to build index for your data, it throw exception and failed to build.

I don’t know what’s the reason of it, so I need more time to test it and reply you after that.

Regards,

Ethan

#3

Hi Kev,

I found the slow reason is for Nautical Chart data, it asked to generate a S52 index (*.hyr), the time is takes to generate the S52 index.

I am not sure whether this process can get optimized because s52 developer hadn’t replied me.

I will update this post when I get reply about it.

Regards,

Ethan

#4

Hi Ethan

Thanks for the update. Figured it was the hyr file.

I sure hope it can be optimized (tbh I’m struggling to imagine how it takes that long!) As a side note, other products I’ve evaluated are loading and displaying the biggest chart in my set in <5 seconds with negligible memory usage.

#5

Hi Kev,

We have observably improved the performance of our index building, please see our test result:

US4FL12M.000 (0.25Mb):
Old Building: 4.81s
Now Building: 3.1s
Peek Memory: 0.04GB

US5FL49M.000 (0.47Mb):
Old Building: 10.369s
Now Building: 4.059s
Peek Memory: 0.06GB

US4FL18M.000 (1.04Mb):
Old Building: 69.665s
Now Building: 13.116s
Peek Memory: 0.17GB

US3GC06M.000 (1.49Mb):
Old Building: 181.173s
Now Building: 61.167s
Peek Memory: 0.49GB

US4FL14M.000 (2.40Mb):
Old Building: 339.822s
Now Building: 37.4s
Peek Memory: 0.37GB

US5AK4JM.000 (4.97Mb):
Old Building: 695.392s
Now Building: 230.263s
Peek Memory: 0.84GB

You can get our latest development version (11.0.0-beta011) of ThinkGeo.MapSuite.Layers.NauticalCharts and test it on your side.
Please let me know any further questions.

Regards,
Vincent

#6

Hi Kev, just FYI there is still room to improve and we are still on it.

#7

Hi

Thanks for the update, I’ll grab the latest builds and check it out!

Cheers

#8

Hi Kev,

Any update we will let you know.

Regards,

Ethan

#9

Hi Kev,

After we dig into the code further, we have made more improvements in memory and performance for BuildIndex method. Now we can keep the memory stable and under a reasonable/accept level. For performance, we improve some by increasing the usage of CPU. Here is the new test result on our side. The latest version is 11.0.0-beta012.

US4FL12M.000 (0.25Mb):
Time (11.0.0-beta010): 4.81s
Time (11.0.0-beta011): 3.1s
Time (Latest): 2.25s

Peek Memory (11.0.0-beta011): 0.04GB
Peek Memory (Latest): 0.02GB

Peek CPU (Latest): 39%

US5FL49M.000 (0.47Mb):
Time (11.0.0-beta010): 10.369s
Time (11.0.0-beta011): 4.059s
Time (Latest): 4.32s

Peek Memory (11.0.0-beta011): 0.06GB
Peek Memory (Latest): 0.03GB

Peek CPU (Latest): 59%

US4FL18M.000 (1.04Mb):
Time (11.0.0-beta010): 69.665s
Time (11.0.0-beta011): 13.116s
Time (Latest): 42.4s

Peek Memory (11.0.0-beta011): 0.17GB
Peek Memory (Latest): 0.07GB

Peek CPU (Latest): 83%

US3GC06M.000 (1.49Mb):
Time (11.0.0-beta010): 181.173s
Time (11.0.0-beta011): 61.167s
Time (Latest): 57.7s

Peek Memory (11.0.0-beta011): 0.49GB
Peek Memory (Latest): 0.11GB

Peek CPU (Latest): 90%

US4FL14M.000 (2.40Mb):
Time (11.0.0-beta010): 339.822s
Time (11.0.0-beta011): 37.4s
Time (Latest): 122.9s

Peek Memory (11.0.0-beta011): 0.37GB
Peek Memory (Latest): 0.18GB

Peek CPU (Latest): 91%

US5AK4JM.000 (4.97Mb):
Time (11.0.0-beta010): 695.392s
Time (11.0.0-beta011): 230.263s
Time (Latest): 319.5s

Peek Memory (11.0.0-beta011): 0.84GB
Peek Memory (Latest): 0.37GB

Peek CPU (Latest): 94%

Maybe you are curious why we have to build the .hyr file, according to the .000 file. Here are the reasons:
• ThinkGeo Nautical Charts SDK strictly follows the specification from IHO https://www.iho.int/iho_pubs/draft_pubs/S-52_e6.1.0/S-52_e6.1.0_and_PresLib_e4.0.0.htm. It’s rich in the symbols and you can easily to switch between different modes, show or hide different features, according to your requirements. To support so many different, complicated but colorful symbols, all are calculated and drawn on the fly, thus performance consumed. So we analysis all features in the *.000 and create the drawing instructions of each feature in BuildIndex method, there it has some geometry processing codes, so a bit slow when building the index, but the good news is that you will experience a pretty good drawing performance when showing them up in the application. You just need to build the index only once. I’m not sure whether you tried the same product from ArcGIS, we have a much better drawing performance.

Regards,

Vincent

Building index for layers on background thread
#10

Hi

Thanks for the update, although I note that the times have crept up again - personally I’d be happy to trade off (some) memory usage for faster performance, but of course I’d love both :wink:

Cheers

#11

Hi Kev,

Thanks to let us know the performance problem about it before.

Our developer mainly try to solve the speed problem, the memory usage is hard to controlled at the same time.

If we have new enhancement about it we will put the update it.

Regards,

Ethan

#12

Hi Kev,

We created a new version (11.0.0-beta013) with a great performance boost and memory deduction (So you now have both :grinning: ). Please get it from Nuget and have a try, I’m sure you will love it. Below is our test result:

US4FL12M.000 (0.25Mb):
Time (11.0.0-beta010): 4.81s
Time (11.0.0-beta011): 3.1s
Time (11.0.0-beta012): 2.25s
Time (11.0.0-beta013): 0.75s

Peek Memory (11.0.0-beta011): 0.04GB
Peek Memory (11.0.0-beta012): 0.02GB
Peek Memory (11.0.0-beta013): 0.01GB

Peek CPU (11.0.0-beta012): 39%
Peek CPU (11.0.0-beta013): 10%

US5FL49M.000 (0.47Mb):
Time (11.0.0-beta010): 10.369s
Time (11.0.0-beta011): 4.059s
Time (11.0.0-beta012): 4.32s
Time (11.0.0-beta013): 1.39s

Peek Memory (11.0.0-beta011): 0.06GB
Peek Memory (11.0.0-beta012): 0.03GB
Peek Memory (11.0.0-beta013): 0.02GB

Peek CPU (11.0.0-beta012): 59%
Peek CPU (11.0.0-beta013): 19%

US4FL18M.000 (1.04Mb):
Time (11.0.0-beta010): 69.665s
Time (11.0.0-beta011): 13.116s
Time (11.0.0-beta012): 42.4s
Time (11.0.0-beta013): 4.53s

Peek Memory (11.0.0-beta011): 0.17GB
Peek Memory (11.0.0-beta012): 0.07GB
Peek Memory (11.0.0-beta013): 0.08GB

Peek CPU (11.0.0-beta012): 83%
Peek CPU (11.0.0-beta013): 79%

US3GC06M.000 (1.49Mb):
Time (11.0.0-beta010): 181.173s
Time (11.0.0-beta011): 61.167s
Time (11.0.0-beta012): 57.7s
Time (11.0.0-beta013): 15.16s

Peek Memory (11.0.0-beta011): 0.49GB
Peek Memory (11.0.0-beta012): 0.11GB
Peek Memory (11.0.0-beta013): 0.14GB

Peek CPU (11.0.0-beta012): 90%
Peek CPU (11.0.0-beta013): 95%

US4FL14M.000 (2.40Mb):
Time (11.0.0-beta010): 339.822s
Time (11.0.0-beta011): 37.4s
Time (11.0.0-beta012): 122.9s
Time (11.0.0-beta013): 10.18s

Peek Memory (11.0.0-beta011): 0.37GB
Peek Memory (11.0.0-beta012): 0.18GB
Peek Memory (11.0.0-beta013): 0.13GB

Peek CPU (11.0.0-beta012): 91%
Peek CPU (11.0.0-beta013): 96%

US5AK4JM.000 (4.97Mb):
Time (11.0.0-beta010): 695.392s
Time (11.0.0-beta011): 230.263s
Time (11.0.0-beta012): 319.5s
Time (11.0.0-beta013): 43.62s

Peek Memory (11.0.0-beta011): 0.84GB
Peek Memory (11.0.0-beta012): 0.37GB
Peek Memory (11.0.0-beta013): 0.29GB

Peek CPU (11.0.0-beta012): 94%
Peek CPU (11.0.0-beta013): 96%

Ben

#13

Hi

That’s outstanding work :grinning:

Cheers

#14

Hi Kev,

Our developer is glad to hear your praise, and we will keeping to provide better product.

Regards,

Ethan