Problems with large object sets? (no replies)

I’m having trouble with an nginx setup built to serve search engines.

Based on the user agent, all bots are served only from cache. We populate the cache with our own set of spiders so we can control the overall load.

Total cache size is ~450 GB in ~12 million files.

The problem is that about 1/3 of the requests coming in live from the bots are misses, even though the requested page was requested by our spider a mere hour previously.

Configured limits should be safe:

proxy_cache_path /var/www/cache levels=1:2 keys_zone=my-cache:2500m max_size=800000m inactive=800h;

Where should I be looking for why these requests were misses?

Thanks,

— David_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx