1. CS 518 - Project Apple OS-X - Project: Grouping Threads by Affinity to a Particular Core based on Cache. 12 December 2008 by Jean-Paul Lauren

2. Problem to be addressed • How canwe group certain processes on one CPU? • {to take advantage of shared cache and optimizations by staying resident on one CPU}

3. Simple Approach • Add a rule to the scheduler, telling it to group based on a tag or token on a process. • Add this tag or token, hardcode it first – then later use a system call to create the value of a processor.

4. Simple Approach • Pretty staraight up approach that avoids directly modifying the scheduler. • Instead, add a rule to the scheduler that takes a token value and groups the processes based on that.

5. Detailed Approach

6. Proposed Solution • The Mach thread policy calls are defined in header /usr/include/mach/thread_policy.h and in /System/Library/Frameworks/System.framework/PrivateHeaders/mach/thread_policy.h as follows:

7. Existing Code - thread_policy.h kern_return_tthread_policy_set( thread_t thread, thread_policy_flavor_t flavor, thread_policy_tpolicy_info, mach_msg_type_number_t count);

8. thread_policy.h kern_return_tthread_policy_get( thread_t thread, thread_policy_flavor_t flavor, thread_policy_tpolicy_info, mach_msg_type_number_t *count); boolean_t *get_default);

9. Modified Code - by adding a new flavor of thread policy: #define THREAD_AFFINITY_POLICY 4 structthread_affinity_policy { integer_taffinity_tag; };

10. by adding a new flavor of thread policy: typedefstructthread_affinity_policythread_affinity_policy_data_t; typedefstructthread_affinity_policy *thread_affinity_policy_t;

11. by adding a new flavor of thread policy: #define THREAD_AFFINITY_POLICY_COUNT ((mach_msg_type_number_t) \ (sizeof (thread_affinity_policy_data_t) / sizeof (integer_t))) #define THREAD_AFFINITY_TAG_NULL 0

12. Tag or Token for grouping • By setting a (non-null) affinity tag for a thread, the thread is placed into the affinity set identified by the "tag". By default, all threads have the THREAD_AFFINITY_NULL affinity. A non-null tag is arbitrary and can convey application-specific information.

13. THREAD_AFFINITY_POLICY • The thread_policy_set (THREAD_AFFINITY_POLICY) call can be made after creating a thread but before starting it running in order to influence its initial placement.

14. Rule places processes according to available CPUs • For example, an application wanting to run 2 threads on separate L2 caches would set the threads with different affinity tags. On a dual core machine, this affinity will effectively be ignored. However, on a 4-core MacPro, the scheduler will try to run threads on separate packages. Similarly, on an 8-core MacPro, the scheduler will try to run these threads on separate dies (which may or may not be in the same physical CPU package).

15. Tag or Token • Threads with default affinity policy will be scheduled more freely on any processor. These threads will be preferentially migrated to run on an idle processor. Threads with affinity tags will tend to remain in place.

16. task_info.h • In addition, the header /usr/include/mach/task_info.h includes the following new information request: structtask_affinity_tag_info { integer_t count; integer_t min; integer_t max; integer_ttask_count; };

17. task_info.h typedefstructtask_affinity_tag_infotask_affinity_tag_info_data_t; typedefstructtask_affinity_tag_info *task_affinity_tag_info_t;

18. task_info.h #define TASK_AFFINITY_TAG_INFO 16 #define TASK_AFFINITY_TAG_INFO_COUNT \ (sizeof(task_affinity_tag_info_data_t) / sizeof(natural_t)) This enables you to obtain the number (count) and range [min .. max] of the affinity tags currently defined for a task.