As below is our understanding about cache. For WPF, the cache is a group of images, if we have cache for an area, the source data don't need to be read again and render in memory, which will takes so many system source. When the cache existing the map will directly read the images and render the images on map, that's very fast.
And in fact the Google maps or any other maps is using the same logic, they have a static library, which contains all the images. Which means when you pan to any area, the speed is the same because you just need load the same number tile images, but not need to wait render different shapes from data.
And for raster layer, if you open cache, it will use the same logic, which means the tile image will be saved in disk and load when user access it. But if you want to load the raster layer dynamic, the speed cannot be guaranteed, because raster layer have many different types, most of them depend on c++ library to read and load, some of them cannot create index so each time the whole raster image need be load to memory, you can think about this process and know how many system source need be taken.
If you want to provide user better experience, you need avoid directly load the raster layer, you can pre-build some tile images for target area and target zoomlevel, which means you need to build a group of tile images, then when user try to access the area, map can quickly get pre-build images and show them on map.
You can try the utility, just make sure render the target raster layer in which area and zoomlevel, then put the same render logic into the LayerProvider class, and it will simulator user pan to all the target area and build the cache for you.
If this utility cannot works for your scenario, I think we don't have better choice for improve the speed to load raster layer dynamic.
Wish that's helpful.