Optimizing Oracle Exadata with Oracle Support Services [CON7054]

2. Optimizing Oracle Exadata with Oracle Support Services [CON7054] A Client View from KPN Eric Zonneveld Architect Engineered Systems Oracle Advanced Customer Support Jan van Dijken Service Delivery Engineer Oracle Advanced Customer Support October 02, 2014

4. Program Agenda KPN Overview Platinum Services Advanced Monitoring and Resolution Advanced Customer Support Lessons Learned 1 2 3 4 5

5. “We have multiple suppliers who deliver support and updates, but that makes it difficult for us to meet our service levels. We believe Oracle Platinum Services can improve the way we support our environment so that we are able to meet our service levels and, at the same time, lower our costs.” Play video Michel Bosman Lead Architect, KPN

6. KPN Overview Infrastructure, project and organization

7. KPN Overview Project 2010-2011 • Project startup (Mar) • Ordering (Nov) • POC Systems delivered X2-2 quarter (Dec) • Phase I: Initiation and testing (Jan) • Phase II: POC and pilots (Feb) • Phase III: EXA demo (Mar) • Phase IV: DTAP systems delivered X2-2 full (Jun) • First critical application chain live (Nov) 2012-2013 • Platinum Services (Aug) • DTA systems delivered X3-8 full (Jan) • Oracle ACS and AM&R contract for service delivery (IM, PM, CM) (Jan) • EXATEAM created (Jan) • Migration DTA -> X3 (Jun)Separate production and non-production at rack level 2014 • Delivery second production P2 based on 12c (Jan) • Migrate workload P1 to P2 (Apr) • Exalogic (May) • Exalytic (Aug)

8. Infrastructure Haarlem Aalsmeer 40 km KPN Overview Dataguard

9. KPN Overview Total Hardware • 24 compute nodes • 62 storage nodes • 560 cores • 10,8 TB memory • 55,8 TB flash • 1,824 TB storage • 16 IB switches • 6 CISCO switches • 4 KVM Per Site • Storage bandwidth 50GB/s X293GB/s X3 per rack • 1,5M IOPS (read) 1M IOPS (write) per full rack

10. KPN Overview Challenges • Interesting landscape … • Huge Oracle database landscape (>2,000), lots of Oracle middleware and applications • All hardware and OS platforms in 27 datacenters • All versions (e.g. Oracle database 7 and up) • Interesting objectives … • Data center migration and application / chain consolidation • Company strategy is to have vendor / partner take more responsibility • Decrease spend

11. KPN Overview Challenges • Which leads to … • Increasing workloads because of consolidation • Increasing demands for performance, stability and availability • While at the same time … • Aggressive outsource strategy • Continuous re-organization • Challenging indeed

12. KPN Overview How to engage • “Old fashioned” support did not work • Tiered model • Offshore • “New way” of working • DBA centric, end-2-end responsible • 1 team onshore • Strong combination between Oracle (run) and KPN (change)

13. KPN Overview Organization • Oracle Platinum Services • Change management (Patching) • Oracle Advanced Support Cloud Services • Incident management, problem management • Oracle Onsite Support • Innovation, life cycle management (infra upgrade) • KPN ITS Platform Business Applications • Change management, business consulting, backup & recovery

14. KPN Overview Benefits of introducing Exadata • Scalability • Servicing 594 databases (incl. 132 standby databases) • 220TB storage used (net) • 58 applications • Performance • Performance gains ranging from 2x to 100x • Special hardware and backup infra no longer needed • Application scaling up 10x volume without a problem

15. KPN Overview Benefits of introducing Exadata • Availability • Availability increase (no unplanned downtime) • Real Application Clusters • Active dataguard • Maximum availability architecture • Faster life cycle management • Most up-to-date and secure platform • Introduction of 12c at least a year before other platforms

16. KPN Overview Benefits of introducing Exadata • But what about support? • Incident Management SLA’s (notification, diagnostic, resolution) • MTTR • Service request handling • Problem management • Business KPI’s (anything)

17. Platinum Services Make the invisible visible

18. Platinum Services What is it? • Part of hardware support • “Free” but with rules • Limited scope for incident management • Oracle Advanced Support Gateway server • OEM12c, customized with metrics, thresholds and escalation rules • Patching • Change management (in ACS portal)

19. Platinum Services What changed? • Visibility of incidents (not events!) • Visibility of problems • Visibility of (wrong) configuration • RDS in Oracle executable • Visibility of Oracle best practices not being applied • Huge backlog • Moved from >120 open SR’s after engagement to average 15 SR’s currently

20. Oracle Monitoring Advanced Support Gateway • Platinum aids companyDBA’s with the incident management process

21. Oracle Advanced Support Cloud Services Advanced Monitoring and Resolution

22. Oracle Advanced Support Cloud Services What is it? • Extends Platinum Services, adds problem management • Broader scope • More events • Diagnostics • RCA’s • Resolution • Integration with client IM/PM processes (ITIL-based) • Service Now

23. Oracle Advanced Support Cloud Services What changed? • More responsibility delegated to Oracle (IM and PM process) • Shifts focus from KPN DBA’s from running the infrastructure (run) to aid applications and business processes, introduce new functionality (change) • Advanced Support portal

24. Oracle Advanced Support Cloud Services What changed? • Increased service level objectives, e.g. • Resolution • Severity 1: 80% < 1 day, MTTR average < 1 day • Severity 2: 80% < 3 days, MTTR average < 3 days • Availability reporting • Of (by the client) nominated critical database services and related applications, >99.99% • Incident summary reporting, quarterly • Production / non-Production, by date and severity, by category (Exadata software, database, compute node, storage, switch, other)

25. Oracle Onsite Support Customer Integration

26. Oracle Onsite Support What is it? • Acts as the “glue” between Oracle Platinum, AM&R and the client • Life cycle management (upgrades) • Innovation • New functionality, new releases • Best practices, Standardization • Expert troubleshooting • On site, close contact, customer intimacy

27. Lessons Learned 2 years of engagement, what worked and what didn’t?

28. Lessons Learned Room for improvement • Oracle should be more visible in what they do • Frequent face-to-face reporting in a format that appeals to the customer • Client does not want to know the differences within Oracle (Platinum, AM&R and ACS), should be one team and one voice • Incident to problem promotion can be improved • Portal could be improved • Google search and user friendliness • CI tagging

29. Lessons Learned Room for improvement • Continuous learning within AM&R / Platinum • Enhancement of KA’s (knowledge articles) must become second nature • Change management • Integration of Oracle changes into the CM system and processes of the client • Work on “awareness” of customers of patch frequency, hardware maintenance etc. • Communication between support engineers and customer • Do not assume customers read all mail or plough through the portal every day

30. Lessons Learned What worked? • Insight into Exadata significantly improved (exposure of problems) • Co-operation between KPN and Oracle • Service delivery management (TAM) • Co-engineering between Oracle and KPN • Local ACS resources • Complex problem solving

31. Oracle Services Advanced Customer Support and Support + + =