Slow repaint of screen when using bing maps

[Richard6]

Using Thinkgeo.core 13.0.0-beta341 and Thinkgeo.ui.android 13.0.0-beta302
A good portion of the time the maps are very responsive – however every now and then there is an excessively long wait for anything to be drawn to the screen. Somewhere in the 5-15 second time frame. This gives the user the impression that the application is hung. I have not received any complaints about v10 behaving in this fashion

This is a very sporadic problem – but happens more frequently when other layers are added beyond the bingMapsOverlay. I have duplicated this in the HowDoISample project using the BingMapOverlay selection.

The maps do eventually get drawn-- and all of the tiles are displayed so it’s not timing out. The way that I reproduced this is to rapidly pan/zoom in/zoom out. Exact steps are hard to identify – however I believe that zooming in/out is where this is happening most of the time.

This is the stack at the time that we see this long delay – this stack is from the HowDoISample and is similar to the stack we are receiving in our code when this happens.
0xFFFFFFFFFFFFFFFF in System.Threading.Monitor.Monitor_wait C#
0x31 in System.Threading.Monitor.ObjWait at /Users/builder/jenkins/workspace/archive-mono/2020-02/android/release/mcs/class/corlib/System.Threading/Monitor.cs:85,5 C#
0x11 in System.Threading.Monitor.Wait at /Users/builder/jenkins/workspace/archive-mono/2020-02/android/release/mcs/class/referencesource/mscorlib/system/threading/monitor.cs:218,13 C#
0x3 in System.Threading.Monitor.Wait at /Users/builder/jenkins/workspace/archive-mono/2020-02/android/release/mcs/class/referencesource/mscorlib/system/threading/monitor.cs:228,13 C#
0x109 in System.Threading.ManualResetEventSlim.Wait at /Users/builder/jenkins/workspace/archive-mono/2020-02/android/release/external/corert/src/System.Private.CoreLib/shared/System/Threading/ManualResetEventSlim.cs:607,33 C#
0x30 in System.Threading.Tasks.Task.SpinThenBlockingWait at /Users/builder/jenkins/workspace/archive-mono/2020-02/android/release/external/corert/src/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs:2892,25 C#
0x6C in System.Threading.Tasks.Task.InternalWait at /Users/builder/jenkins/workspace/archive-mono/2020-02/android/release/external/corert/src/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs:2832,21 C#
0x13 in System.Threading.Tasks.Task<System.Net.WebResponse>.GetResultCore at /Users/builder/jenkins/workspace/archive-mono/2020-02/android/release/external/corert/src/System.Private.CoreLib/src/System/Threading/Tasks/Future.cs:490,31 C#
0x11 in System.Threading.Tasks.Task<System.Net.WebResponse>.get_Result at /Users/builder/jenkins/workspace/archive-mono/2020-02/android/release/external/corert/src/System.Private.CoreLib/src/System/Threading/Tasks/Future.cs:466,19 C#
0x6 in System.Net.HttpWebRequest.GetResponse at /Users/builder/jenkins/workspace/archive-mono/2020-02/android/release/mcs/class/System/System.Net/HttpWebRequest.cs:1216,5 C#
0x3C in ThinkGeo.Core.WebBasedLayer.xh8= C#
0x2F in ThinkGeo.Core.XyzTileLayer.yB8= C#
0x68 in ThinkGeo.Core.XyzTileLayer.yR8= C#
0x324 in ThinkGeo.Core.XyzTileLayer.DrawCore C#
0x1D in ThinkGeo.Core.BingMapsLayer.DrawCore C#
0x2F4 in ThinkGeo.Core.Layer.ux8= C#
0xA8 in ThinkGeo.Core.Layer.Draw C#
0x27 in ThinkGeo.UI.Android.LayerTileView. C#
0x10 in System.Threading.Tasks.Task.InnerInvoke at /Users/builder/jenkins/workspace/archive-mono/2020-02/android/release/external/corert/src/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs:2476,17 C#
0x1 in System.Threading.Tasks.Task.Execute at /Users/builder/jenkins/workspace/archive-mono/2020-02/android/release/external/corert/src/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs:2319,17 C#
0x6 in System.Threading.Tasks.Task.ExecutionContextCallback at /Users/builder/jenkins/workspace/archive-mono/2020-02/android/release/external/corert/src/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs:2463,13 C#
0x73 in System.Threading.ExecutionContext.RunInternal at /Users/builder/jenkins/workspace/archive-mono/2020-02/android/release/mcs/class/referencesource/mscorlib/system/threading/executioncontext.cs:968,17 C#
0x4 in System.Threading.ExecutionContext.Run at /Users/builder/jenkins/workspace/archive-mono/2020-02/android/release/mcs/class/referencesource/mscorlib/system/threading/executioncontext.cs:910,13 C#
0x38 in System.Threading.Tasks.Task.ExecuteWithThreadLocal at /Users/builder/jenkins/workspace/archive-mono/2020-02/android/release/mcs/class/corlib/corert/Task.cs:72,6 C#
0x50 in System.Threading.Tasks.Task.ExecuteEntry at /Users/builder/jenkins/workspace/archive-mono/2020-02/android/release/external/corert/src/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs:2385,17 C#
0x2 in System.Threading.Tasks.Task.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem at /Users/builder/jenkins/workspace/archive-mono/2020-02/android/release/external/corert/src/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs:2334,13 C#
0x75 in System.Threading.ThreadPoolWorkQueue.Dispatch at /Users/builder/jenkins/workspace/archive-mono/2020-02/android/release/mcs/class/referencesource/mscorlib/system/threading/threadpool.cs:899,29 C#
0x0 in System.Threading._ThreadPoolWaitCallback.PerformWaitCallback at /Users/builder/jenkins/workspace/archive-mono/2020-02/android/release/mcs/class/referencesource/mscorlib/system/threading/threadpool.cs:1261,13 C#

[Richard6]

Some additional information – here are the threads at the time of the delay. This is from our application – but the thread structure was similar in the howDoISample. Most of the threads had a similar if no identical call stack to the above message

[Kyle_Day]

Hey @Richard6,

So, I finally was able to recreate this on my side. All of those threads are actually waiting on the response back from Bing when calling HttpWebRequest.GetResponse():

I can’t say for sure why this is only happening on Android and not iOS. Both are using the same shared ThinkGeo.Core logic for requesting the image from Bing. My first thought was rate limiting from Bing’s side, but you mentioned that you haven’t gotten complaints that it happens in v10 as well. It’s very clear that it’s coming from the GetResponse() though. When this happens though, the map is still responsive. Just the tiles aren’t loading from the Bing base maps.

What you can do to help alleviate this problem is to create an image cache:

bingMapsOverlay.TileCache = new FileRasterTileCache(APP_CACHE_DIR);

This will store the images on disk (there’s also an InMemory version available) with a duration available to invalidate it. This makes it so that when the user does zoom in/out really quickly, it doesn’t make a web request to Bing if it already exists in the cache.

Thanks,
Kyle

[Richard6]

Looked back at the old version 10 code and it does in fact use the cache – our version 12 code does not. This wouldn’t be so bad but its not just the bing base maps that aren’t loading – Our other feature layers are also not loading when this occurs.

I will add in the tile caching for our Android client again to see if that resolves the issue for our release.

[Kyle_Day]

Hey @Richard6,

A TileCache is pretty common on basemap overlays and also layers that have raster data. So, if some of your layers or GeoTiffs or something like that, it might prove useful to add TileCache’s on them as well. Vector layers usually do not need them unless their Indexes are not performant or not built. That kind of depends on the data.

How many Overlays do you have on the map? Just the BingMapsOverlay and a LayerOverlay? How many layers are in the LayerOverlay?

Thanks,
Kyle

[Richard6]

Actually there are a bunch of overlays – a background overlay which holds only a backgroundLayer, the bingmapsOverlay, a layerOverlay for features, and another layerOverlay that we use when features are searched for. As for the layers outside of the bing maps we use only vector data. Our product is configurable from a server application so the # of layers depends on the project in question. The one I primarily test with has 4 layers – 1 polygon, 2 linear and 1 point layer. Client data can have significantly more layers. I’ve seen a few with 15+ layers – all sqlite layers and they are indexed.

If we turn off the base map display we don’t have this speed issue. As for using the cache – the problem is that clients can generate many project that may or may not have overlapping territory. So when moving about initially in an are they do see these longs delays.

I believe we tried Google basemaps and had similar issues but I am waiting for verification on that issue right now. I have recently set up the application in order to test MapBox and will verify if the problem presents itself there or not.

[Richard6]

I am getting the exact same type of behavior from MapBox and Google. All I am really doing is waiting for initial map display and then rapidly zooming in using pinch gestures. 3-4 time without waiting for map updates in between. Zoom out similar behavior with the long waits. Again this is sporadic – sometimes the speed is exactly as expected.

[Kyle_Day]

Hey @Richard6,

That’s a little surprising that your other Overlays don’t appear until the basemaps come in. That seems atypical. I’ll do some testing and see why that might be happening, but what normally should happen is that the other overlays render as soon as they are done drawing and don’t require waiting on the other overlays to finish. Unless maybe your Overlays’ TileType is in SingleTile mode?

I also wonder if you rollback to ThinkGeo.UI.Android 13.0.0-beta295 if you would run into the same issue. Looking at the recent commits, there was quite an overhaul that occurred within the last two months that might have messed things up as well.

Thanks,
Kyle

[Richard6]

No singletiles mode in use. Rolled back to core beta-294 and android version beta-253 and my initial impression is that the really long delays don’t occur – Or at least not as quickly as I am seeing in the most up to date code. I will give a build to our test group and see what their opinion is on the speed concerns.

[Kyle_Day]

Hey @Richard6,

Ok, let us know if your test group says beta-253 resolves the speed issues. I have a feeling that the changes I mentioned might be responsible, but 3rd party confirmation will help guide us and I can get with the developer that made those changes to help fix it in a future update.

Thanks,
Kyle

[Richard6]

The final verdict is that the most recent code is bit faster than beta-253. The behavior is quite sporadic and has gotten more so at the beginning of the month. Which leads me back to your original thought on potential rate limiting – I cannot exclude that as a possibility. However it appears that when we do get these long delays it also appears to affect the drawing of the feature layers as well. Turning off the base maps and we don’t have any issues at all with displaying of our feature layers.

I have some screen shots from a video of a pinch zoom operation – at the end of the zoom it pixelates the image and we end up sitting looking at the pixelated image for somewhere around 20 seconds or so before features are drawn – once the features have finished drawing the old pixelated feature layer finally goes away and a few seconds later the base maps finally get updated – here are the screen shots from the various stages:
Initial View:

After pinching –

Features starting to draw :

Features fully drawn and base maps starting to be displayed:

Once the features start drawing they get drawn quickly – its the wait up to the point where they start drawing that is our concern – and as I said this doesn’t happen all of the time we just have no established pattern as to when this will occur. Zooming in/Out is where we see this – I am not certain it happens during panning. As mentioned we have not seen this at all with base maps off.

[Richard6]

We are in excess of the yearly limit so we are being throttled somewhat by Bing. We switched to the free tier of Google for test purposes, but with the # of people testing we quickly hit some sort of short term limit there as well. Speed was fine until we started getting timeouts when pulling map tiles and then the behavior was actually quite a bit worse then on Bing. To be clear I am mainly talking about the feature layer updates being impacted as opposed to simply the base maps not getting drawn.

[Kyle_Day]

Hey @Richard6,

Okay, so sounds like you prefer to stick with the latest beta for the drawing performance, and along side your latest response as I was writing this up, it really seems that the vector layers are a key problem now. Sucks that your are getting rate limited, hopefully that isn’t as common outside of stress testing. So I think we just need to figure out why the performance on your vector layers are coming in so slow.

So, strictly speaking for the vector overlay, the transition from the zoomed out position to the zoomed in position, it looks like there are a lot more features being drawn here, quite possibly additional layers are being rendered here (looks like street lamps or something on the side of the road is being added). So, a feeling I have is that maybe these layers are taking a long time to load at this zoom level (about 5 seconds from the video timeline) and possibly others.

This can happen for a number of reasons though. Sometimes this is due to there being too much information on the screen at once, or the index for your sqlite layers needs to be reindexed, or in very rare cases, the hardware isn’t fast enough to keep up. Although these scenarios might not exactly reflect your situation.

Here is something I can suggest: move your line layer into it’s own overlay. This way the line come in super fast, as they already do, while the other overlay draws the rest of the points/polygons. This should get rid of that super thick line while waiting (think of the overlays as a different image being overlayed on top of each other, we don’t update that image until the drawing is done for that overlay).

Another suggestion: on each of your layers, there is a DrawingTime property on it. Use that metric to identify which layer is actually taking a long time to draw on that zoom level. Utilizing that can inform you if you need to reduce the number of features being drawn, using a filter style or value style to reduce the amount at those zoom levels that are most strenuous.

Finally, not to sound like a broken record, adding a tile cache onto your vector overlay will alleviate revisits to that location. Doesn’t help the initial load, but it’s a reliable method for speed later.

Thanks,
Kyle

[Richard6]

We have 70+ people in the field using our product with another 8 or so on the development side. All are using the same API key for Bing at the moment. Fielders are doing inventory type work – the projects they work on are usually of a short timeframe (week or less) and the projects are not generally in overlapping areas – So the number of pulls will not likely drop below the yearly pull of 135,000 Bing allows before throttling.

In the case shown here the point feature are poles and the drawing time for the information that we are displaying is not really our concern – there is a rather heavy load of features being drawn from time to time. Its the 20 seconds when the fingers left the screen until something starting happening on the screen that is the concern. Keep in mind we don’t have these long delays when the basemaps are turned off in our product. The drawing time start to finish of the features is about the same as in the screen shots, however there is no long delay before the screen starts to update with no basemaps. The fielders are currently using TG v10 implementation and they are not seeing these long delays before somethings starts drawing – although they do see a delay before all of the basemaps come in – but that is expected with the throttling going on.

Everytime the user switches to a new project of the application is started up the project gets re-indexed. As for the hardware we are using Samsung Galaxy S7 tablets – although the vide above may have come from a Galaxy S4 Tablet.

As for moving the lines into its own layers we see this happen on projects that contain only point features as well – we found that having more overlays caused a speed hit – although the finding was from version 10 not the latest version.

I was not aware of the drawing time property – I will take a look at that to see if that sheds any light on things. Unfortunately the filters and value styles are something that is ultimately controlled by database configuration information that is ultimately under the control of the project administrators.

As for the tile cache – our data is somewhat live data – the application synchs with the server periodically to pull in new data – when that happens we would have to invalidate portions of the cache. We were actually doing that in TG v10 and pulled the code back out as it was actually slowing things down because of the rather short lived nature of the projects and because of any of the updates that came from the server of that the fielders make – such as changes to a Features label or for that matter adding in additional features.

Keep in mind that we are comparing the behavior of TG v12 to TG v10 and we are not seeing these long delays of nothing happening in v10.

[Richard6]

Something else I should have mentioned again is that although we can reproduce this quite frequently we cannot do so at will. It is rather sporadic, but it happens enough that we cant release this version to our fielders.

[Kyle_Day]

Hey @Richard6,

I dug into the code a bit more and made a change that should alleviate the LayerOverlay from being blocked to draw while waiting on the BingOverlay to get a response from their servers. This should also alleviate other Overlays from blocking each other as well.

The update should be available soon on Nuget. Keep a lookout for ThinkGeo.UI.Android 13.0.0-beta310. That will have the fix I put in place. Once we confirm that the issue is resolved, we can merge that into the main v12 branch.

Thanks,
Kyle

[Richard6]

Unfortunately we are still having some issues – I am occasionally getting “Cannot access a disposed object” object name: ThinkGeo.UI.Android.Layer.TileView. I have seen this occur on zoom in/out and panning. I think it was going to update the bing tiles when it occurred but I am not certain.

Now as for the slow repaint – It is difficult to test as the above issue happens somewhat frequently. The overlays no longer appear to wait for the other to finish. However every now and then I get that slow response – it might just be on the initial zoom in – after that zooming in as responsive as panning. Zooming out is a bit different every now and then I get a white screen and then everything shows up all at once.

This was using Thinkgeo.UI.Android 13.0.0-beta311

[Kyle_Day]

Hey @Richard6,

I think I know what’s going on with the disposed TileView exception you are getting. Let me dig into it and see if I can resolve the issue. I’ll push up another change to Nuget soon and notify you when it’s ready.

Thanks,
Kyle

[Leo_Liu]

hi @Richard6,

Could you try upgrade your ThinkGeo.UI.Android to 13.0.0-beta313(ThinkGeo.UI.XamarinForms 13.0.0-beta204, ThinkGeo.UI.XamarinForms.Android 13.0.0-beta229) and see if the expection still exists?

We added a new property IsDrawingAsync to control if drawing overlays async, in your case, you could set mapView.IsDrawingAsync to true to alleviate other Overlays from blocking each other.

Thanks,
Leo

[Richard6]

I no longer see the exception – also I have not run into the case where there is a long delay while zooming in of the old vector graphics hanging around for a while.

At this point I believe that this resolves the issues I was seeing with the repaint.