Storage Issues
This documentation discusses the challenges and optimal strategies for deduplication storage, particularly focusing on read performance critical for reconstructing original data streams stored across different disks. The goal is to distribute chunk deployment evenly across disks to minimize cross-disk access during data reconstruction, ensuring unique data chunk representation. It also covers VM annotations for guiding memory deduplication tools, proposing methods to include relevant information such as OS/kernel versions and application types for improving deduplication efficiency.
Storage Issues
E N D
Presentation Transcript
Last Week • Deduplication storage • Read performance is critical to reconstruct the original data stream.
Problem • Assume that data streams are stored in different disks. • After deduplication, every chunk is unique. • Decide a chunk deployment such that the number of cross-disk access of every data stream during reconstruction is almost the same. A B C
Examples Ca = 5 Ca = 0 Avg. = 2 Avg. = 2 A A Cb= 4 Cb= 0 Std. = √(8 / 3) Std. = √(14 / 3) B B Cc = 2 Cc = 1 C C
Examples(Cont.) Ca = 2 Ca = 3 Avg. = 2 Avg. = 2 A A Cb= 3 Cb= 3 Std. = √(6 / 3) Std. = √(2 / 3) B B Cc = 0 Cc = 1 C C
Optimal Solution Ca = 2 Avg. = 2 A Cb= 2 Std. = 0 B Cc = 2 C
Another Example A A Ca = k Ca = k/2 Avg. = k/2 Avg. = k/2 … … … … B B Cb= k/2 Cb= k/2 Std. = √(k2 / 12) Std. = 0 C C Cc = k/2 Cc = 0 k k/2
VM annotation • Use annotation to guide memory deduplication tool like KSM. • More aggressive if running the same OS kernel.
VM annotation(Cont.) • Every VM is configured via a XML file in /etc/libvirt/qemu. • We can insert annotations to VM’s configure file. • Export ('dump') the XML of the virtual machine you want to edit. • Edit the XML. • Import ('define') the XML.
The Problem Is… • What should we annotate? • OS/ kernel version. • Purpose/ Running what application • Computation-intensive or I/O intensive
Paper Study • Characterizing Datasets for Data Deduplication in Backup Applications • Tradeoffs in Scalable Data Routing for Deduplication Clusters
