Vulnerability Management at Scale

2. Vulnerability Management at Scale • Alexandre Fiori • Production Engineer

4. Overview The evolution of the Vulnerability Management program at Facebook, the birth of the Production Engineering team to support the program, and the pragmatic approach to build a highly scalable system for an ever-growing environment. What’s this about

5. “Facebook was built on an open source stack. We support and encourage the use and development of open source software and hardware”

6. Vulnerability Management Timeline • …2015: PCI DSS • 2016: Internal scanning tools • 2017: Broader network scanning • 2018: Reboot, build up • 2019: VMaaS

7. 2018

8. New challenge • Scan all infrastructure • Merge scanning technologies • Improve vulnerability matching • Start and track remediation • Manage remediation lifecycle

9. Architecture

10. Mindset • Big data • Scalability and reliability • On-line vs off-line pipelines • Concept validation and XFN • Fast prototype and launch

11. ETL

13. Collectors • Scan asset inventories and print to standard output

14. Processors • Pre-process, scan vulnerabilities, post-process

15. Reporters • Reporters manage escalation lifecycle

16. Inventory Classes • Hosted software • Installed software • Running software • Network scanners • Hardware

17. Vulnerability Database • Public vs Proprietary • Multi-vendor system • General purpose datasets • Specialized for ecosystem • Standard format for product->vulnerability matching

18. Implementation

19. Design Principles • Command line tools do one thing • Communicate over a text interface • Core functionality shared as libraries • Composable code, tools, and pipelines • Rely on well established UNIX conventions

20. Industry Standard Technology • MITRE / NIST / NVD • Common Platform Enumeration • Common Vulnerabilities and Exposures • Common Weakness Enumeration • Common Vulnerability Scoring System

21. Infrastructure • Tools • Services • Data warehouse • Dashboards • Notifications

23. Internal Pipelines • Tupperware • Container images and packages • First-party vs third-party codebase • Bad Binary Hunter and Buck • Attribution from package to service

24. Vulnerability Database Tools • nvdsync and $vendor2nvd • vulndb command line tool • Uses NVD CVE JSON 1.0 format • Manages versioned datasets backed by MySQL • Supports vendor snapshots, custom CVEs, and snoozes • vulndb thrift service • CVE lookup and CPE matching

25. Decoration • CWE and CVSS • Domain and sub-domain • First seen • Backlog vs influx • Owner (on-call ID or UNIX username) • Threat Intelligence

26. Remediation • Starts from CVE inventory • Depends on decoration data • Supports feedback loop (e.g. snoozes) • Understands release cycles per inventory class • Manages escalation and lifecycle

27. Lessons Learned • Normalization, aggregation, and blackholes • Per-customer decision trees are burdensome • Handling delays, XFN work, and fine tuning • Tasks and notification updates are annoying • False positives can compromise credibility

28. 2019: VMaaS

29. Goals • Self-service system for vulnerability scan • Custom aggregation defined by collectors • Configurable providers and thresholds • Tier-based service, starting from bronze • Default dashboards and reports per tier • Common CVE inventory for all customers

30. Progress • XFN partnership with early adopters • Migrated “hosted software” inventory class • Total of 10+ collectors in operation • Customers fixing bogus CVEs in the database • Snoozes effectively helping fine-tune reports

31. Next up • Migrate other inventory classes • Improve data quality and detection speed • Tackle backlog via remediation campaigns • Tackle influx via push-blocking scans • Influence company culture outside security org

32. Thank you

33. Q&A