Should I be cache warming my site? If so how should I go about doing this?
Firstly cache warming the whole site is not a good idea in that it is a lot of effort for not a lot of gain. But cache warming specific portions of the site is a good idea.
Most caches, Varnish included, is designed to serve the most requested assets from cache, not every asset.
So there are algorithms in place in Varnish that will flush least recently requested assets from the cache on occasion. The idea is that is frees up memory to hold more frequently requested assets.
I would recommend the following if you would like to cache warm.
Make sure crawler is based in the same region as expected real world traffic, as we have different cached in different regions.
Give the crawler a list of home/category pages, and maybe the most popular products to crawl.
Then have the crawler crawl those IPs.
Things to keep in mind
- During the crawl there will be extra burden on the origin server, so it might be wise to have an acceptable crawl rate.
- section.io instances can scale in and out with traffic, so a new instance may have a cold cache.
- This will not give 100% cache hit due to many factors such as querystrings, cookies etc.