Shapefile Taking too much time to load in paritcular scenario due to thinkgeo internal method

hrishikesh_konde · September 5, 2019, 9:30am

Hello ,
In our project we are facing a problem like the loading of shape file takes different time to load the same shapefile .We will like to inform you that our project folder which is of 5GB and the shapefile is placed in the same folder .currently we found that when we are loading the shapefile it is taking to much time.we gone to root of it by restricting the access to some folder and we found that the thinkgeo internal method goes through the entire project folder looking for the licence.making it more slower and some times making the application goes into not responding state.
the stack trace for the following is attached .
please let us know what can be done.
Thanks and regards,
Hrishikesh Konde
stacktraceWPF.txt (1.2 KB)

Ethan · September 6, 2019, 3:37am

Hi Hrishikesh_konde,

Please don’t worry about search licence file, if it’s very slow which means your file number in project folder is too big, but I did a quickly test, about 20000 files with many subfolders only need takes about 2 seconds the first time, after that the speed will increase to about 0.2 seconds, I don’t think your project folder will contains so many files, the file size is not important for this system API. You can also did a test in your machine:

        Stopwatch sw = new Stopwatch();
        sw.Start();

        int fileCount = Directory.GetFiles(@"D:\ProjectFolder", "*.*", SearchOption.AllDirectories).Length;

        sw.Stop();

        long timeInSeconds = sw.ElapsedMilliseconds / 1000;

The exception is because you set security for folder, please don’t disable the access permissions for project folder and data folder.

If the load shape file very slow, which should because you hadn’t build index for shape file, pleaes check that:

ShapeFileFeatureSource.BuildIndexFile("YourData", BuildIndexMode.DoNotRebuild);

And you can also open cache for improve performance:

LayerOverlay overlay = new LayerOverlay();
overlay.TileCache = new FileBitmapTileCache("YourCacheFolder", "CacheSubFolder");

After that, please don’t render too many shapes in high zoomlevel, which measn you don’t want to render a country road when you zoom to the state zoom level, so please optimize your render logic, show the shapes in suitable levels.

If that’s still cannot solve your problem, please upload the sample to us, you can also upload your data to our FTP, contact your sales to get the FTP account.

Wish that’s helpful.

Regards,

Ethan

hrishikesh_konde · September 6, 2019, 11:20am

Hello Ethan,
As i told you the file count is to big .just to give you an idea there is about 160270 files is in our project folder.and these are mandatory files and folders .We have done all the optimization required to render the shapefile.we also found that during finding the thinkgeo licence it goes to nth level iteration eg:"C:\Application\Digit\BASE\MERGE\WK28_18\B4\ROOT\DK\DKGEAST\3 " instead of searching just the licence file.we found out this by restricting the file access to the to last file i.e. “3” .during this time span of finding licence the other resource in the project is not accessible i.e.
if we click on button on which the map loads the UI freezes and we are also unable to accesses our project folder,and we SUSPECT that, is there any thing that restrict us doing that may be a lock or something in the internal method of thinkgeo ?
we have already generated index for the both the shapefile .we are just rendering world shape with there political boundaries.
Please suggest us the necessary steps for resolving the problem.
Thanks and Regards,
Hrishikesh Konde.

Ethan · September 9, 2019, 8:22am

Hi Hrishikesh_konde,

I checked our logic, we will check all sub folders in target folders, so if you lock any sub folder for example your “3” folder, it will met exception:
a. The app running current directory.
b. …/ProgramData /ThinkGeo/MapSuite {version}/;
c. …/{User}/AppData/Local/ThinkGeo/MapSuite {version}/;

I hadn’t found a folder with about 160000 files for test, so I created a test environment, it looks the GetFile speed is about 130 ms, you can also test that based on your folder.

     string folderName = @"D:\RandomTest";
     int count = 160000;
    
     // Build files
     for (int i = 0; i < count; i++)
     {
         string filename = Path.Combine(folderName, Path.GetRandomFileName());

         Random r = new Random(Guid.NewGuid().ToString().GetHashCode());
         int size = r.Next(1, 500) * 1024; // size = 1k~500k
         byte[] buffer = new byte[size];
         r.NextBytes(buffer);
         using (System.IO.FileStream fs = new System.IO.FileStream(filename, System.IO.FileMode.Create))
         {
             fs.Write(buffer, 0, size);
         }
     }

    // Test the speed
     Stopwatch sw = new Stopwatch();
     sw.Start();

     int fileCount = Directory.GetFiles(folderName, "*.lic", SearchOption.AllDirectories).Length;

     sw.Stop();

It looks your test data is too big to upload, I think you can did a quickly test in your side to make sure whether the slow is caused by search license file or load shape files:

a. I think the most data should be your data, so move all your data to other folder(won’t be search when looking for license file)
b. Modify the data path in render logic
c. Run your application again

If the speed is fast, which means the slow is caused by search license file, load shape file speed is OK.
If the speed is still be slow, which means the slow is caused by load shape file, but search license file is OK.

Please let us know your test result.

Regards,

Ethan

hrishikesh_konde · September 9, 2019, 9:23am

Hello Ethan,
We have already performed the given test and we found out that if we take out a chunk of data from project folder then the speed of loading is fast .but if we again place that data back in that folder the speed become slow and it freezes the UI and the other resources as well .note that all the data in the project folder is must.
Thanks and Regards,
Hrishikesh Konde

Ethan · September 10, 2019, 3:24am

Hi Hrishikesh_konde,

Thanks for your test, it looks that’s actually because the search license, could you please did a further test to make sure that?

Keep the big number of your files in project folder
Comment all your render logic, just render a very small data in your code

Now run the application, because your code only load a very small data now, so if it still slow, that’s make sure is research license did it.

After that, could you please run this code once and let us know what’s the value of theTime:

Stopwatch sw = new Stopwatch();
sw.Start();

int fileCount = Directory.GetFiles("Put your bin folder here", "*.lic", SearchOption.AllDirectories).Length;

sw.Stop();
var theTime = sw.ElapsedMilliseconds;

We will discuss with our development to see whether any workaround we can found about it.

Regards,

Ethan

hrishikesh_konde · September 10, 2019, 6:16am

Hello Ethan ,
We tested what you said ,we found that the time taken was 141694 mil. sec. for the first time the and it takes 6420 mil. sec. after that .the screen shot for first attempt is attached .

Thanks and Regards,
Hrishikesh Konde.

Ethan · September 10, 2019, 8:06am

Hi Hrishikesh_konde,

Thanks to let us know your test result.

We will discuss about how to solve it, maybe it will need to modify the license related logic, so it won’t be very fast fix, any update I will let you know.

Regards,

Ethan

Ethan · September 11, 2019, 8:46am

Hi Hrishikesh_konde,

After discuss our development team modify the search logic, please download this package which it’s available(Should be tomorrow) in Nuget and let us know whether it works for you:

Regards,

Ethan