Hi Simo,
WMS itself does not have a tile matrix, so there is no server-side matrix that your custom RasterXyzTileAsyncLayer needs to match.
In this case, the TileMatrixSet is only the client-side tiling grid. It controls how the custom layer converts z/x/y into a BBOX, and also how the local tile cache is organized. So the TileMatrixSet does not need to match any matrix on the WMS server, but the BBOX generated by the TileMatrixSet must be in the same coordinate system as _wms.Crs. The WMS server will interpret the BBOX values using the CRS in the GetMap request.
By default, RasterXyzTileAsyncLayer uses 256 x 256 tiles, GeographyUnit.Meter, and the standard Web Mercator/Spherical Mercator max extent. So if your WMS CRS is EPSG:3857, the default tile matrix is correct.
If your WMS CRS is another projected CRS using meters, the default tile matrix may still work in many cases because it is still a meter-based client-side grid. However, it may not be the best or cleanest grid for that CRS, and the local cache z/x/y values may look less natural. If the CRS extent is outside the default Web Mercator extent, or if you want the cache/grid to align better with that CRS, you should create a custom TileMatrixSet with a bounding box in that CRS.
To set a custom TileMatrixSet, the key is to use a bounding box and GeographyUnit that match the coordinates used by _wms.Crs. For example, EPSG:4326 would normally use GeographyUnit.DecimalDegree and a decimal-degree extent such as MaxExtents.DecimalDegree. A projected CRS in meters would normally use GeographyUnit.Meter with an extent in that projected coordinate system.
Thanks,
Ben