Graybox NFS Caching Proxy

1. Graybox NFS Caching Proxy By: Paul Cychosz and Garrett Kolpin

2. What are Graybox Techniques? • First we need to know a little about how the “graybox” works. • Inference by observation • No server or client-side modifications necessary

3. Our Problem • Figure out what the client is caching • Maintain low overlap between client and proxy caches

4. Our Solution • Treat the client as a graybox • Monitor NFS traffic • Infer from NFS traffic the contents of client cache

5. Conclusions Cache overlap decreased by 5% to 50% when using graybox techniques

6. Outline • Approach • Results • Conclusions

7. Approach NFS specifics: • Implemented in kernel as another FS type • Replaces “file_operations” with its own functions for read, write, open, etc. • Uses VFS abstraction VFS RPC / XDR retrans., data wrapping NFS kernel code TCP / UDP

8. Approach Client cache / Replacement • Caches on page level • Not much NFS specific code – Uses global Linux page cache • Not a static size, no upper-bound • Timestamps: “jiffies”, per page.

9. Approach Reads typically > ~85% of traffic NFS / RPC “read” structure • RPC Packets sent out: access to dir, getattr to file, access to file, then read, read, read, … • Client use of getattr to revalidate cached copies • Exploit this, proxy explicitly look for these to determine client cache contents

10. Approach Proxy • Sees: • File handle • Offset • size --becomes our “cache element” • Store in hash table • Cache on reads, run our cache element replacement policy on getattr (proof of cached client copy)

11. Approach NFS Client Proxy Replacement (possible force-out) Read system call getattr Revalidate data hit Cache lookup OR Lookup read miss Read setup OR hit miss Return data Add Data Typical read() Forward req. to server

12. Viewing Client Cache Contents Test Environment: • Disable swap space • Use mincore() to view pages that client has cached • Use “balloon” program to keep client cache contents manageable

13. Approach Difficulties • Logistics: Initial setup, client/server/proxy all on same PC – doesn’t work • Client cache size changes -solved: compare percentages, normalize • Repeated RPC calls not idempotent • Not a significant problem, proxy efficiency degrades linearly with NFS performance if network traffic is bad

14. Approach Difficulties • Ambiguities: • Different FS activities DON’T generate unique RPC patterns • getattr used for a few things besides reads -- Null RPC packets Cannot tell exact blocks client caching, only which files Need second, local replacement policy on proxy!

15. Approach Cache Replacement Global Policies Local Policies • LRU • MRU • Forcing entire file • MRU

16. Outline • Approach • Results • Conclusions

17. What We Tested • Control Experiments • Global LRU, no local force-out • Global MRU, no local force-out • Combinations • Global LRU, whole file force-out • Global LRU, block-level local MRU • Global MRU, whole file force-out • Global MRU, block-level local MRU

18. How We Tested • Used Postmark benchmark • 12 files with sizes randomly selected between 4 and 50 kilobytes • Gives us more of a “real-world” workload

19. Results

20. Results

21. Results

22. Outline • Approach • Results • Conclusions

23. Conclusions • Decrease in overlap by between 5 and 50 percent as compared to control tests • MRU global file replacement is good by itself • MRU global policy with whole-file local policy gives highest cache fill and lowest overlap

24. Questions? Feedback? ? ? ?