I think you maybe misunderstand the overlay here. Overlay only exist in server side, because in client side each overlay will be rendered as a OpenLayers layer, and each layer contains about 20 tiles (related with the screen size). So if you have 50~60 server side overlay, that means you have at least 50 x 20 = 1000 tiles need be rendered at the same time. That's why your performance is not well.
So I think the best solution is reduce the overlay number.
And add layer in client side is not helpful, the layer which can be added in client side is something like Google or Bing layer, they get source from internet but not from your server, and in fact you add them via OpenLayers api. The other type layers(server overlay) cannot be added in client side.
In fact the server side whether works related with your data source, if some data takes a lot of source, the server side cache can save the time.
Client cache should be set in overlay level, for example:
Map1.StaticOverlay.ClientCache.CacheId = "Test Cache";
Map1.StaticOverlay.ClientCache.Duration = new TimeSpan(2, 0, 0, 0);
Wish that's helpful.