Apache Traffic Server Spring Summit 2017
SSL Lazy Loader Plugin Design
Steven Feltner
reveller – IRC
[email protected]
[email protected]
Spring, 2017
Copyright© 2017 GoDaddy Inc. All Rights Reserved.
Problem Statement
• Need to load tens of thousands of SSL Certs
• Certs get updated every few minutes
• Customers purchase new certs, update certs, cancel accounts or revoke certs
• "Statically" configuring through ssl_multicert.config
• Requires minutes to parse and reload
• Often got stuck in a reload circle
• Lengthy startup times
• With such long start up times, upgrades required pulling nodes from rotation
2
Copyright© 2017 GoDaddy Inc. All Rights Reserved.
Design Goals
• Lazy load cert on demand
• First request for https
• Lookup in a domain map to make sure we even need to pursue the effort
• Load cert from redis
• Subsequent requests for same domain go straight to lookup table without the need for redis lookup
• Support for redis sentinels
• Provides high availability for redis clustering
• Provides configuration data about cluster
3
Copyright© 2017 GoDaddy Inc. All Rights Reserved.
Design Features
• Loaded cert TTL
•
•
•
•
"Evicts" cert from memory, freeing memory usage
Offloads infrequently used certs
Refreshes certs already loaded into memory in case of updates/revoked certs
Difficult to balance configuration
• Config time too short - it is not effective
• Config time too long - changes to certs lag behind
• Cert look up blacklist
• Prevents possible DDoS attempts by immediately returning without looking up a cert in redis
• Configurable as time-based or LRU
• Customers can be a PITA when they purchase/renew and then immediately ping their site
• Cust who waited for 3 weeks after his cert expired to renew
4
Copyright© 2017 GoDaddy Inc. All Rights Reserved.
Version 2
• Straight to redis every time
• Uses local redis slave with priority of 0
• Fast sync time
• 200k certs were synced into a new instance in less than 2 secs from main cluster
• Stays current with main cluster but cannot be promoted to master
• Eliminates single point of failure
• If local redis is not available, gets config from redis Sentinel for a read-only node in main cluster
• Once local instance comes back online, all traffic routed to on-box redis
• Reduces overall network traffic by localizing accesses to local redis
• Reload every cert on every request
•
•
•
•
5
No need for cert TTL
Certs are never stored in memory so better memory utilization
Certs never go stale; updates are immediate
Revoked certs are removed from redis immediately
Copyright© 2017 GoDaddy Inc. All Rights Reserved.
Future Features
• Command line messaging into plugin (Thanks, amc!)
• Remove from blacklist
• Get status about a loaded cert based on domain
6
Copyright© 2017 GoDaddy Inc. All Rights Reserved.