Cloud Usage with CloudStack

1. Cloud Usage with CloudStack Sept 2014 Chandra Prakash Mishra

2. In next 45 minutes … • Agenda • CloudStack Usages • Cloudstack configuration to enable Usage Capture • Usage Data Generation Process • Usage Type • Usage Event • Usage Records • How can Access Usage Data • Use Cases

3. UsageRequirements • Meter their cloud's consumption and monetize the cloud offerings by charging for usage • Report on the cloud’s resource usage for monitoring/capacity planning • Existing billing system, just need access to the usage data • Organizations tend to have different billing criteria/charging models and require access to usage data for their specific needs Footer – Apply across document

4. UsageinCloudStacktoday • CloudStack does not offer any native usage monitoring or billing capability • Root Admin can view current System Capacity • Domain Admin and Users can view VM statistics Footer – Apply across document

5. Usage Server • Optional • Separately installed • Provide aggregated usage records • Run at least once per day (Default):1440 • Take data from the events log in cloud DB • “cloud.usage_event” • Create summary usage records in cloud_usage DB • “cloudusage.usage_event” • Access usage records using listUsageRecords • “http://localhost:8096/client/api?command=listUsageRecords&startdate=2013-05-25&enddate=2013-06-05”

6. Usage Server Config

7. Usage Server Config Usage Server • Usage Job will run at PST 00:15 • Generate usage records for 1hrs from 07:15:00 GMT to 08:14:59 GMT • Next exec.time is 01:15 Footer – Apply across document

8. Usage Type

9. CloudStack Event • Usage events are stored in usage_event table. Any resource for which usage has to be tracked needs to log usage events when the the resource is created/allocated and when its deleted/released • Network usage is the only exception which doesn’t use usage events • Once these events are logged usage server will process them and generate Cloudstack usage records Description of various columns in usage_event table: Running VM +--------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | id | type | account_id | created | zone_id | resource_id | resource_name | offering_id | template_id | size | processed | +--------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | 12 | VM.START | 2 | 2011-02-09 11:16:18 | 1 | 5 | i-2-5-KK | 7 | 2 | NULL | 0 | | 15 | VM.STOP | 2 | 2011-02-09 12:00:24 | 1 | 5 | i-2-5-KK | 7 | 2 | NULL | 0 | +--------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ Public IP address +----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | id | type | account_id | created | zone_id | resource_id | resource_name | offering_id | template_id | size | processed | +---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- + | 2 | NET.IPASSIGN | 2 | 2011-02-09 11:16:18 | 1 | 10 | 10.91.30.121 | NULL | NULL | 1 | 0 | | 9 | NET.RELEASE | 2 | 2011-02-09 12:00:24 | 1 | 21 | 10.91.30.121 | NULL | NULL | 0 | 0 | +----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

10. Usage Record Generation • CloudStack events have been logged into cloud.usage_event table • The cloudstack_usage job gets a list of the latest usage events • Inserts these events into cloud_usage.usage_eventstable • Parses cloud_usage.usage_event and populates Helper tables • Helper table data used to populate cloud_usage.cloud_usagetable with aggregation range wise data Footer – Apply across document

11. Network Usage Record Generation • Network usage is processed differently and is not driven by usage events • In Virtual Networking case Iptables rules are set in domR to track public traffic • Management server queries  for Network stats periodically and populates them in user_statistics table account-wise • Cloud.user_statisticsis queried and populated to cloud_usage.user_statistics with latest stats • Network helper entry is created in usage_network table • Network usage for current event is calculated as: • bytesSent= (total bytes sent) – (bytes sent prior to this event);bytesReceived = (total bytes received) – (bytes received prior to this event);

12. Usage Calculation The Usage Record

13. Usage Record Format Virtual Machine Usage Record Format: For running and allocated vm usage Footer – Apply across document

14. Usage Record Format Network Usage Record Format: For more details Please refer this link: https://cloudstack.apache.org/docs/en-US/Apache_CloudStack/4.0.2/html/API_Developers_Guide/usage-record-format.html

15. How we can access usage data Footer – Apply across document

16. CloudStack API • The usage records can be accessed through the CloudStack API • This is best done using the authentication port 8080 and requires both the API Key and Secret Key and for the API request to be signed • CloudStackprovides the following Root Admin API calls and responds with XML and JSON responses • listUsageTypes – lists available Usage Types • listUsageRecords- provides Usage records for a date range • generateUsageRecords- asynchronous usage record generation Footer – Apply across document

17. ListUsageRecords API • The base API request is: • http://<HOST>:<8080>/client/api?command=listUsageRecords&startdate=y • yyy-MM-dd&enddate=yyyy-MM-dd&signature • Optional request parameters: Footer – Apply across document

18. Example response from listUsageRecords • <listusagerecordsresponse> • <count>1816</count> • <usagerecord> • <account>user5</account> • <accountid>10004</accountid> • <domainid>1</domainid> • <zoneid>1</zoneid> • <description>i-3-4-WC running time (ServiceOffering: 1) (Template: 3)</description> • <usage>2.34 Hrs</usage> • <usagetype>1</usagetype> • <rawusage>2.34</rawusage> • <virtualmachineid>4</virtualmachineid> • <name>i-3-4-WC</name> • <offeringid>1</offeringid> • <templateid>3</templateid> • <usageid>245554</usageid> • <type>XenServer</type> • <startdate>2009-09-15T00:00:00-0700</startdate> • <enddate>2009-09-18T16:14:26-0700</enddate> • </usagerecord> • ……… • </listusagerecordsresponse>

19. CloudMonkeyCLI • CloudMonkeyis a CLI tool for CloudStack and can be used as an interactive shell or from within a shell script to call the API • It can output the Usage Data in both JSON and Tabular form to a file or pipe it to another application for further processing • Filtering can be used to limit the result set • It supports argument passing and shell automation • The host, port, apikey and secretkey can be configured on setup Footer – Apply across document

20. CloudMonkeylist usagerecords The command to retrieve Usage Records is: cloudmonkeylist usagerecordsdomainid=7d8707bf-e6ad-11e3-a7e8-f245a5b3ba0f startdate=2014-08-10 enddate=2014-08-11 accountid=efda79de-ff01-4de2-bbe9-56b0ec366a07 Footer – Apply across document

21. SQL • The usage records can be also accessed by directly querying the cloud_usage.cloud_usageMySQL table in CloudStack • Usage Data can be exported in a CSV format from a SELECT query • The SQL query can become complex if the ID fields in the data need to be de-referenced • Typically multiple passes of the query results may be required to extract the required information Footer – Apply across document

22. CloudStackMessage Queue • CloudStack 4.1 started publishing events onto a message queue Uses RabbitMQ as the message broker, but likely to work with other AMQP- based brokers • Use case: A third-party cloud usage solution can subscribe to CloudStack events and generate usage data which can be consumed by their billing software For more details Please refer this link: https://cwiki.apache.org/confluence/display/CLOUDSTACK/EventBus Footer – Apply across document

23. Customer Use Cases Footer – Apply across document

24. MS Excel • MS Excel • MS Excel can be used to analyseCloudStack Usage Records • The usage records are imported from the cloud_usage table into MS Excel via an MySQL ODBC connection • Either PivotTables or reports can be used to present the data • This is convenient and great for integration testing between CloudStack and external systems Footer – Apply across document

25. Splunk • Splunkis an operational intelligence tool that was initially used for collecting and indexing infrastructure logs • We then connected splunkto the CloudStack Usage database and automated the collection and indexing of Usage Data • Setup feature-rich dashboards and the capability to drill down into the Usage Data for BI reporting Footer – Apply across document

26. Amysta Fore more details please refer this link: https://www.youtube.com/watch?v=ZA-VPb7WAy8 http://www.slideshare.net/buildacloud/apaliaamysta-cloud-usage-metering-and-billing Provides usage visibility, cost control and IT billing for Private and Hybrid Clouds Dashboard Integrated in the CloudStack UI Pricing, Alerting, Reporting, Capacity MgmtAmysta consolidates public cloud expenses e.g. AWS and private cloud consumption in one dashboard view Footer – Apply across document

27. SunGard Cloud Portal Footer – Apply across document

28. Thank You!