Shibboleth: Transitioning from Test to Production

1. Shibboleth: Transitioning from Test to Production John Ball University at Buffalo john@buffalo.edu

2. UB’s Shib project • Looking for an Auth technology to replace existing infrastructure (mostly Intra campus use) • Shib fit the bill for both Inter and Intra institutional needs • Tight timeframe (Originally Dec ’04) • Wary of getting back in the same place we are now with any single vendor. 2

3. Infrastructure replacement • UB a DCE shop • Looking at Shib (on top of Kerberos and LDAP) to replace AuthN/AuthZ part of DCE • Still trying to figure out some of the other pieces: • Secure RPC • DFS file system replacement 3

4. Political Work • Selling this to the campus • 50/50 mix of Centralized/Decentralized IT • Ken spoke via video conference at open campus session December 2003 • UB involved in I2 since 1999 • CIO on board • Selling this as a solution SUNY-wide 4

5. Progress so far… • Last year started thinking about trying to use Shib to replace the Auth for our Portal as a part of some other architectural changes relating to HA and DR • Planned for this Fall (mid- august) for go live • This meant a May 1 deadline for deciding if we would use Shib? • Our answer – no…not yet. 5

6. Why Not? • Shib 1.2 not quite ready at that time • Still had issues with target stability on Solaris under load • Resources not lined up for this effort over the summer • Marketing and technical (DR/HA) • Maybe too many changes at once • Portal probably not the best place to start… 6

7. The Issues • Shar crashes under load on Solaris Targets (pre 1.2) • Our High Availability Model not 100% proven • Cisco CSS switch • still working out the finer points of the final configuration • Some issues with multiple virtual service pools behind the Cisco CSS interacting • Marketing the changes to the Campus 7

8. The Details • Deploying in a load balanced/HA scenario • Virtualized services • Both Auth and Web application farm • Scary pictures to follow… • 4 Geographic locations • Initially internal application use • Solves future SUNY wide problems as well 8

11. Shib Customization • Wrote some custom code to use an Oracle DB for the SHAR cache • Keeps the state in a HA environment • Looking to contribute this to I2 • Why oracle? 11

12. Performance • Benchmarked current peaks • DCE on Solaris • Apache Web servers • Peaks for our busiest web service ~5500 unique “auths” per hour or 92 per minute • Originally estimated peak Shib capacity to be 1.84 auths per second • with WebISO (Cosign) and Java encryption 12

13. Original Plans • Originally using 4 Sun V120s (Origins) • Originally using Java for SSL • Originally using Shib with Cosign, then we removed it (for performance increase). 13

14. Hardware • UB Historically a Sun shop • Started with 4 Sun V120s • Moved to 4 Sun 280Rs • Dual CPU • Sun Crypto Accelerator cards • Performance still CPU bound • Moved to Linux on 2 “borrowed” Dell 6650s (used the 280s for our LDAP) 14

15. Final Origin Hardware • Purchased 12 Dell 1750s • Dual Xeon 3.2G CPUs • 1 GB RAM • The more CPUs the better! • Plans to deploy 2 Dells per location for production 15

16. Targets Apache on Solaris 8/9 • Initial deployment in our Administrative web farm • Typically a Sunfire 280R in load balanced model behind a Cisco CSS • Hitachi SAN storage • Application Data • User session credentials 16

18. Java • Now using native JCE SSL • Significant CPU savings 18

19. Cosign in, Cosign out, Cosign back in! Running two public key based cookie signing services in serial has some unavoidable overhead. A logon for a user to shibboleth user (with Cosign) has a number of steps in serial: 1. User goes to application 2. User is redirected to shibboleth HS 3. User is redirected to cosign logon page 4. User submits username/password, is redirected back to HS 5. HS makes call back to cosignd 6. User is redirected SHIRE on application server 7. SHIRE makes call back to AA in shibboleth 8. SHIRE redirects user back to application URL in step 1 9. User's browser fetches application data 19

20. Cosign • Doing just shibboleth with basic auth (or just cosign) has almost half the steps. All of these call backs to cosignd and the AA are using SSL as well, requiring some horsepower. • However: You lose High Availability • You cannot maintain session state thru an Origin reboot 20

21. Testing and Tweaking • Testing load using: • Webload (Radview) • JMeter (opensource) • Tweaking and testing • Capacity • Session times • Still no solid numbers yet 21

22. Issues • Not been able to run a Shib 1.1 Target under load on Solaris with any stability – yet… • Getting ready to have a go with new 1.2 code! • Cisco CSS configuration • Still working on a “500” page error about every 500 auths –Tomcat issue? • This may be fixed in a newer version of Tomcat • Kerberos plug-in for LDAP bug, since been fixed 22

23. Next steps • Clean up our environment • Try 1.2 code • Fix CSS issue • Pilot small internal Mail Alias tool application 23

24. Contingencies? • Contingencies if we can not work out Shar stability on Solaris Targets • Move the Shar to another machine? • Probably Linux • Would need to address security since it uses TCP vs Unix sockets • Consider Apache on Linux for the Target application servers • Not a realistic solution for the short term with our Sun web farm infrastructure 24

25. Questions? • john@buffalo.edu 25