Tile Caching Issues

[Dennis]

MapSuite Team,

Recently I implemented tile caching in my WMS Server. During development extensive testing was done with no issues found. Today the WMS Server was deployed to our live production environment and incorrect imagery began to appear on several of our client workstations forcing us to backout caching after several hours of operation.

You may want to note that in nearly three hours of live operations the system had accumulated 71,889 total cached tiles comprising 16GB. A fairly busy system.

In some cases sections of any given map extent would appear black. In some cases the imagery that appeared was at the wrong ZoomLevel even though the SQL Lite and Shape File Feature Layers were correct. Without caching these issues never existed.

I’ve discovered a few interesting things.

First, there are cached tiles that are only 384 bytes in size. They have what appears to only be a header. These might correspond to the imagery on the map that appear black. An example of this tile is attached.

Second, there are cached tiles that appear to have partial data. The normal tiles show a 256x256 pixel area, but these tiles are 256 high with width less than 256. These tile examples are attached. These tiles are also zoomed way out when they should not be. These tiles may also account for black imagery within a map extent.

Third, there are tiles that are of the wrong zoom level. They are zoomed way out. For this example two tiles, of the same identifying file name, are attached. One is zoomed out and one is correct.

Below is a list of the tile example files. All times are EST.

Also shown below are the example tiles in one screen capture. Each individual tile is also attached.

My code to setup caching:

    string TheCachedTileDirectory;
    TimeSpan TheTimeSpan;
    FileBitmapTileCache TheTileCache;

    TheTimeSpan = new TimeSpan(6, 0, 0);

    TheTileCache                = new FileBitmapTileCache();

    TheCachedTileDirectory = string.Format("{0}\\{1}", "C:\\MapCachedTilesWms", "TheWmsServerCache");
    TheTileCache.CacheDirectory = TheCachedTileDirectory;
    TheTileCache.CacheId        = "MyLayerName";
    TheTileCache.ExpirationTime = TheTimeSpan;
    TheTileCache.TileAccessMode = TileAccessMode.ReadAddDelete;

    oWmsLayerActive.MapConfiguration.TileCache = TheTileCache;

I am running WmsServerEdition V9.0.0.1103.

Also please note that the tiles being cached emanate from an external WMS Server (nearmap).

Your thoughts?

Regards,
Dennis

The following cached tiles are under the directory …\2256.99444\6507784
07/11/2018 09:45 PM 262,144 6477528-ThisTileIsCorrect.png
07/11/2018 11:12 AM 262,144 6477528-ThisTileIsNotCorrect.png
07/11/2018 11:12 AM 262,144 6477557-ThisTilePartialSize.png
07/11/2018 11:12 AM 131,072 6477558-ThisTilePartialSize.png
07/11/2018 11:12 AM 53,843 6477559-ThisTilePartialSize.png
07/11/2018 11:12 AM 384 6477560-ThisTileIsEmpty.png

These two tiles have the same file name (6477528.png) but one is at the proper ZoomLevel while the other is zoomed way out. The correct one is from the development server while the one zoomed out is a cached tile on the production server.

This tile here appears white, but appears black in the first image above as well as on the map extent below.

These tiles are example of height 256 pixels, but smaller than 256 width.

This is an example of black areas within the extent and the imagery zoomed way out while the feature layers are at the proper ZoomLevel.

Here’s an example of a map extent that has the correct imagery, blacked-out area and imagery that is zoomed out when it should not be zoomed out.

Here’s another interesting map extent. Notice how on the left, bottom, and right the imagery is correct. The remaining area is not only zoomed too far out it is repeating the imagery in that area. Notice that the buildings are repeating.

Below is how the above area should look.

[Ethan]

Hi Dennis,

Our developer review your tile images, that should be caused by 3 reasons.

The 1 and 2 images, we think it’s because you hadn’t set the snap in your server. In server cache you need to set the snap because if you don’t set it, the entire cached tile image size cannot be controlled.

You should want to set it like this: getMapRequest.ScaleSnapMode = ScaleSnapMode.UpperScale;

The 3 to 5 images should because the TileMatrix hadn’t been set, we don’t know whether you had set custom TileMatrix, if you set it please replace the “590591790” in the sample code:

new FileBitmapTileCache(ConfigurationManager.AppSettings[“OSMWorldMapKitCache”], tileCacheId, TileImageFormat.Jpeg, new MapSuiteTileMatrix(590591790));

The 6 image should because network issue, I remembered you forward the request to other WMS server, so if it met network problem in the progress, the blank image(black) should be generated.

You can write a custom service to remove all images which are smaller than a specified size for solve it.

And because your cached tile images have so many problems, your map shows incorrect.

Wish that’s helpful.

Regards,

Ethan

[Dennis]

hi Ethan,

Thanks for your response.

I wanted to be sure to let you know that these issues are intermittent. Most of the time the imagery is displayed correctly. I never once encountered these issues while testing in development. They surfaced intermittently after live deployment.

Concerning your suggested changes:

1. In what namespace are ScaleSnapMode property and ScaleSnapMode enumeration located? I am using WmsServerEdition V9.0.0.1103 and do not see these in any namespace. Also, in what GetMapCore override should this instruction be placed? In the Handler or in the PlugIn?

2. Indeed the ZoomLevel01 Scale that we use is 147647947. But again, this is an intermittent issue so not sure how populating TileMatrix will make a difference. However, I will make this change. There are other properties in TileMatrix such as TileHeight, TileWidth, BoundingBox, CellHeight, CellWidth. What are your recommended values for these properties?

I added the following line to the application. There is no difference that I noticed. The same cached tiles are created and the map extent renders the same, correctly.

    TheTileCache.TileMatrix     = new MapSuiteTileMatrix(147647947);

One thing nearmap pointed out to me is that when there is no tile caching the tile request size to them is always 1050x1600. However, with caching the tile size requests are all of varying values, such as 1024x1536, 768x1536, 768x1280, 512x1536, and other combinations. Why do these values vary?

What is the design consideration of having cached tile sizes all 256x256 pixels?

3-Without tile caching we have noticed on rare occasions where there might be a section of a given map extent where the imagery is not rendered, but never appears black. Only black when caching is enabled. I have already developed a service to delete tiles that have exceeded their ExpirationTime plus a few hours, so I could modify that to also delete tiles less than a certain size.

Did you determine why in the above imagery examples there are tiles that are repeating? And why are tiles zoomed so far out when they should not be? Again, remember that this does not happen on all tiles. Most of the tiles are correct.

Your thoughts are always appreciated.

Thanks,
Dennis

[Ethan]

Hi Dennis,

Thanks for your update.

If you also cannot reproduce that in your development environment, the reason may be multi-threading.

Based on this thinking, we have to mentioned this again, I think we need your sample to try reproduce it and review code to check where is the problem.

If you cannot build the sample, please review your code to see whether you used many “static” source. If so please try your best to reduce to use it and see whether that’s helpful for solve the problem.

And our developer build a simple sample, you can try to deploy the sample and see whether it can reproduce the same problem.

WmsService.zip (2.5 MB)

Regards,

Ethan

[Dennis]

hi Ethan,

I agree you’re correct about this being a threading issue.

The design of our WmsServer is such that it is configuration driven via web.config. All information concerning layers and anything else is verified once, upon startup, and loaded into collections. The preloaded objects contain MapConfiguration, FileBitmapTileCache, BoundingBox, GeographyUnit, CrsCollection, and other controlling properties.

Upon client request for a WmsRasterLayer the PlugIn returns the preloaded ’static’ information. The information is ‘static’ in the sense that all client requests are using the same copy of the information.

Is this the suspect ’static’ information you referred to in your response?

I emailed support@thinkgeo.com three source files that show the preload of the configuration, the use of the data by the PlugIn, and the WmsHandler. Also included is a text file with a brief description of each source code file.

Am I correct in thinking that instead of the PlugIn returning the preloaded static information it should first clone the static information and then return the cloned value?

Let me know if I’ve provided enough information. I can always send you more source files.

If you feel what I’ve outlined is the issue can you offer how to do a Deep Clone of these?

Thanks,
Dennis
OriStar Mapping, Inc.

[Ethan]

Hi Dennis,

Thanks for your update, we had received your classes.

Our developer read your code, and think the “static clsWmsLayerActiveItem” should have problem.

In the function “GetMapConfigurationCore” of aTheWmsLayerActiveItem, you should want to always create new instance of aTheWmsLayerActiveItem.MapConfiguration instead of use the saved one. Because under multi-thread mode, you cannot make sure the property of it hadn’t get changed when it calculate the tile.

Do you think you can modify it, and please let us know whether it’s helpful.

Regards,

Ethan

[Dennis]

hi Ethan,

Thanks for confirming the issue.

The problem for me is how to provide a new instance of MapConfiguration. It was designed this way to be efficient during the real-time request of imagery since the system is extremely busy. I don’t like the thought of recreating MapConfiguration for each invocation of GetMapConfigurationCore as this is a lot of overhead.

I don’t understand what GetMapConfigurationCore itself has to do with tiling as at this point in the processing doesn’t seem like it even knows what map extent has been requested.

Does MapSuite have any DeepClone code available for use with MapConfiguration that I can put in my application?

The more I ponder this issue I’m wondering might it not be better that MapSuite do a deep copy of MapConfiguration since it wants to change it.

Your thoughts?

Thanks,
Dennis

[Ethan]

Hi Dennis,

I think we don’t have a deep clone for MapConfiguration for now.

And in fact you can move your logic which build the MapConfiguration into GetMapConfigurationCore. You just need to make sure each request have its own MapConfiguration, and see whether that can avoid the cache problem.

In fact build MapConfiguration shouldn’t waste too many source, and you can think it’s just a test, if it’s helpful for solve the cache problem, we can think about how to go on optimize the logic, and reduce the source usage.

Regards,

Ethan

[Dennis]

hi Ethan,

The MapConfiguration that is created is for the layer on the Remote WMS Server so that was why I am concerned about efficiency.

I will change the design such that a new MapConfiguration is created for each request and we can see if that solves the issue.

The product is used in a very busy emergency communications center so I have to be extremely careful about introducing a new release. What is your confidence level that a new MapConfiguration for each request is the solution?

It would be great if this is the solution and then you can optimize the MapSuite design for MapConfiguration.

By the way, caching does provide for better user response and the WMS Server runs 10% more efficiently so I am very motivated to see this work.

Thanks,
Dennis

[Ethan]

Hi Dennis,

If the server you mentioned is so important, I am not sure test on it is a good solution or not. Don’t you have another environment for test it for example a simulated server?

But if only this server can reproduce that, please let us know whether the change is helpful.

Regards,

Ethan

[Dennis]

Hi Ethan,

There is a test server, but this is an intermittent issue and only shows up on the live system. I don’t have a method to simulate live activity. Are you having second doubts as to whether a new MapConfiguration for each request is the solution?

A few days ago I posted the following:

One thing nearmap pointed out to me is that when there is no tile caching the tile request size to them is always 1050x1680. However, with caching the tile size requests are all of varying values, such as 1024x1536, 768x1536, 768x1280, 512x1536, and other combinations. Why do these values vary?

Nearmap also sees many requests for 256x256 and this just doesn’t make sense to us.
During the initial deployment of caching I see that all the cached tiles are 256x256. What is the design consideration of having cached tile sizes all 256x256 pixels? This seems inefficient when the client size is 1050x1680.

Even after setting TileMatrix.TileHeight=1050 and TileMatrix.TileWidth=1680 the cached tiles were still 256x256. How do we get the tiles sizes to be 1050x1680 with caching?

Here’s another concerning thing I’ve noticed – for just one map extent a bit less then 1050x1680 results in eight tile cache directories being created with each directory containing twelve 256x256 pixel tiles. This seems odd to me that there are that many directories/files created for just one map extent. Below are examples.

I’ve experimented with implementing a DeepCopy. Below is code that I am using. It appears to work, but events are not copied. I lose the SendingWebRequest and SentWebRequest events. I can easily add code to set those after the DeepCopy. My question for you is does MapSuite have any events that it sets in MapConfiguration? I want to make sure I’m not losing those.

Can you speak to the above as these have to be considered before caching is deployed?

Thanks,
Dennis

    public static T DeepCopy<T>(T obj)
    {
        T result = default(T);

        using (var memoryStream = new MemoryStream())
        {
            var formatter = new System.Runtime.Serialization.Formatters.Binary.BinaryFormatter();

            formatter.Serialize(memoryStream, obj);

            memoryStream.Seek(0, SeekOrigin.Begin);

            result = (T)formatter.Deserialize(memoryStream);

            memoryStream.Close();
        }

        return result;
    }

[Ethan]

Hi Dennis,

For 256x256, in fact it can be think about it’s the default value of many tile provider, please view the image as below:

The Orange area is your map, the black rectangle is the tile image. You can see when you pan or zoom, your cached image cannot always suitable your map extent, so we need to combine the tile images, and draw the map on current extent. But it also will waste some areas which you can found not included in the orange area, we still need draw them but they won’t be shown on map. So the 256 value is a balance value, you can also modify it to 512 x 512 or 1024 x 1024. But set it to a big value or set is not equal square is not good solution.

And for our cache system, the 2256.99444 is the scale value, the sub folder of it and the image name of it is column number and row number. When you cached more and more areas, the image number will keep increase till all the images in this row get cached.

And it looks the MapConfiguration class don’t have event related with it.

Wish that’s helpful.

Regards,

Ethan

[Dennis]

hi Ethan,

Thanks for your diagram and description of how your tile caching works. You mention that the 256x256 size can be changed, but you don’t mention how to change it. Would you provide a code snippet that shows in what class these properties are set so that I might experiment with the effects of alternate values?

The issue with this 256x256 approach is that MapSuite is sending 256x256 requests to our Remote WMS Server, which is woefully inefficient. It would be much more efficient, from our perspective, if you requested one large tile and then broke the large tile into smaller 256x256 tiles. Is this something that MapSuite supports?

In my last post I had asked “Are you having second doubts as to whether a new MapConfiguration for each request is the solution?”. What is your feeling now?

Thanks,
Dennis

[Ethan]

Hi Dennis,

Our developer double check the process today, if you want to reduce the request number to remote server(not your bridge server), you just need to make your client side pass bigger boundingbox.

The bridge server will handle the boundingbox, expand it to suitable cells, find which parts is not cached local, and then sent request to remote server. After get the result, it will find missed parts tiles, and save them, and return the result to your client side.

So you just need keep the tile size of 256x256 in your bridge server cache, and add the request size from client side.

Please try it.

Regards,

Ethan

[Dennis]

Hi Ethan,

My client side already has a request size of 1050x1680.

The Nearmap server is getting requests from the Local WMS Server for 256x256. In fact I have a spreadsheet from them that is a log of requests. I have emailed the spreadsheet to support @thinkgeo.com. Are the requests sizes what you would be expecting to see?

How is the 256x256 changed on the local WMS Server as you mentioned in your previous post.

Please remember that without caching enabled the request to the Nearmap server is always 1050x1680 corresponding to the request from the client.

Thanks,
Dennis

[Ethan]

Hi Dennis,

Thanks for your update, our developer is working for research deeper about it.

Any update or question I will let you know.

Regards,

Ethan

[Ethan]

Hi Dennis,

Our developer did more test, and we think they are not all the 256 x 256 tile size, but also some other tile size for example 256 x 512, 256 x 1024, 512 x 1024 etc.

The reason of that is because we saved tile local as 256 x 256, but if not all the tiles(256 x 256) included in current extent is cached, our logic will build a new bounding box include all uncached tiles, and sent one request.

That means we still only sent one request when it receive one request.

The only situation it don’t sent request is all tiles include in current extent had been cached.

So please don’t worry about it.

Regards,

Ethan

[Dennis]

hi Ethan,

Your explanation of the requested size makes sense, thanks for that explanation.

I attempted to use the MapConfiguration.Layers[x].CloneDeep() method and found that it works while in the Visual Studio debugger, but does not work when the application is run under IIS. The DeepCopy that I wrote also works in Visual Studio debugger and somewhat works under IIS, but not 100%.

After running in the debugger I was remined that the WmsLayerPlugin.GetMapConfigurationCore is invoked only for the very first map extent request amongst all clients. Once that first request has completed GetMapConfigurationCore is never invoked again. No matter how many clients, after the first client, GetMapConfigurationCore is never invoked again. This is because all the clients use the same layer.

At this stage I don’t believe that creating MapConfiguration for each invocation is the solution.

On the first map extent request GetMapConfigurationCore is typically invoked 2-4 times depending on the requested size. Once that first map extent has been processed GetMapConfigurationCore is never invoked again.

So at most there would only be 2-4 individual MapConfiguration objects created. Do you still feel this is the issue?

What I am going to do as the next step is to create MapConfiguration entirely from the original information (no cloning) for each of the 2-4 invocations of GetMapConfigurationCore.

Have you done any more investigation of WmsServerEdition to determine where this problem might be?

One other item is that the clients have their WmsRasterLayer to use JPEG format. On implementation of caching I did not set FileBitmapTileCache.ImageFormat to JPEG so it defaulted to PNG. This meant that there must have been conversion between JPEG and PNG going on for each request. Could this possibly have caused the issue?

Thanks,
Dennis

[Ethan]

Hi Dennis,

Our developer don’t have more suggestion about it.

We checked the sample code from your email, and our code, it looks this is the only possible reason which will make tile size looks so strange under multi-thread environment. Other part of the code won’t have problem.

And I think convert from JPEG to PNG won’t cause tile cache problem, it only need takes a little more machine source.

Regards,

Ethan

[Dennis]

hi Ethan,

Thank you for the confirmation that I need to use unique MapConfiguration for each invocation of GetMapConfigurationCore. I will implement code that creates a new MapConfiguration from the original information with no cloning.

I have also changed the ImageFormat to JPEG so that it matches what is on the client.

Once this is deployed to production I will let you know the results.

Thanks,
Dennis
OriStar Mapping, Inc.