Feeding the Monster Advanced Data Packaging for Consoles

1. Feeding the MonsterAdvanced Data Packaging for Consoles Bruno ChampouxNicolas Fleury

2. Outline • Next-Generation Loading Problems • LIP: A Loading Solution • Packaging C++ Objects • Demo: LIP Data Viewer • Questions

3. Some things never change…

4. Next-Gen Loading Problem • Processing power up by 10-40X • Memory size up by 8-16X • Optical drive performance up by 1-4X

5. Next-Gen Loading Problem • Xbox 360 • 12X dual-layer DVD drive • Outer edge speed: 15 MB/s • Average seek: 115 ms • PlayStation 3 • Blu-ray performance still unknown • 1.5X is the most likely choice • CAV drive should give 6 to 16 MB/s • Average seek time might be worse than DVD

6. Next-Gen Loading Problem

7. Next-Gen Loading Problem • In order to feed the next-gen data needs, loading will need to be more frequent • Hard drives are optional for PS3 and Xbox 360 • Optical drive performance does not scale with the memory/CPU power increase • Conclusion: • Loading performance must be optimal; any processing other than raw disc transfers must be eliminated

8. Did I Hear “Loading Screen”? • Disruptive • Boring as hell • Non-skippable cutscenes are not better! • Conclusion: • Loading screens must not survive the current generation

9. Background Loading • Game assets are loaded during gameplay • Player immersion is preserved • Solution: • Use blocking I/O in a thread or another processor

10. Background Loading • Requirements: • Cannot be much slower than a loading screen • Must have low CPU overhead • Must not block other streams • Conclusion: • Once again, loading performance must be optimal; any processing other than raw disc transfers must be eliminated

11. Proposing A Solution • Requirements for a next-generation packaging and loading solution: • Large amounts of assets must be loaded at speeds nearing the hardware transfer limit • Background loading must be possible at little CPU cost • Data assets must be streamed in and phased out without causing memory fragmentation

12. Understanding Loading Times • Freeing memory space • Unloading • Defragmenting • Seek time • Read time • Allocations • Parsing • Relocation (pointers, hash ID lookups) • Registration (e.g. physics system)

13. Reducing Loading Times • Always load compressed files • Using N:1 compression will load N times faster • Double-buffering hides decompression time • Plenty of processing power available for decompression on next-gen consoles

14. Reducing Loading Times • Compression algorithm choice • Favor incremental approach • Use an algorithm based on bytes, not bits • Lempel-Ziv family • LZO

15. Reducing Loading Times • Take advantage of spatial and game flow coherence • Batch related data together in one file to save seek time • Place related files next to each other on the disc to minimize seek time

16. Reducing Loading Times • Take advantage of optical disc features • Store frequently accessed data in the outer section of the disc • Store music streams in the middle (prevents full seek) • Store single-use data near the center (videos, cutscenes, engine executable) • Beware of layer switching (0.1 seconds penalty)

17. Reducing Loading Times • Use the “flyweight” design pattern • Geometry instancing • Animation sharing • Favor procedural techniques • Parametric surfaces • Textures (fire, smoke, water)

18. Reducing Loading Times • Always prepare data offline • Eliminate text or intermediate format parsing in the engine • Engine time spent converting or interpreting data is wasted • Load native hardware and middleware formats • Load C++ objects directly

19. Why Load C++ Objects? • More natural way to work with data • Removes any need for parsing or interpreting assets • Creation is inexpensive • Pointer relocation • Hash ID conversion • Object registration

20. Loading C++ Objects • Requires a very smart packaging system • Member pointers • Virtual tables • Base classes • Alignment issues • Endianness

21. Loading Non-C++ Objects • Must be in a format that is ready to use after being read to memory • Texture/normal maps • Havok structures • Audio • Script bytecode • Pretty straightforward

22. Load-In-Place (LIP) • Our solution for packaging and loading game assets • Framework for defining, storing and loading native C++ objects • Dynamic Storage: a self-defragmenting game asset container

23. Load-In-Place: “LIP Item” • 1 LIP item  1 game asset • 1 LIP item  unique hash ID (64-bit) • 32 bits for the type ID and properties • 32 bits for the hashed asset name (CRC-32) • The smallest unit of data that can be • queried • moved by defragmentation • unloaded • Supports both C++ objects and binary blocks

24. Examples of LIP Items • Joint Animation • Character Model • Environment Model Section • Collision Floor Section • Game Object (hero, enemy, trigger, etc.) • Script • Particle Emitter • Texture

25. C++-Based LIP Items • Can be made of any number of C++ objects and arrays • On the disc, all internal pointers are kept relative to the LIP item block • Pointer relocation starts with a placement new on a “relocation constructor” • Internal pointers are relocated automatically through “constructor chaining”

26. Placement “new” Operator • Syntax • new(<address>) <type>; • Calls the constructor but does not allocate memory • Initializes the virtual table • Called once for each LIP item on the main class relocation constructor

27. Relocation Constructors • Required by all classes and structures that • can get loaded by the LIP framework • contain members that require relocation • 3 constructors • Loading relocation constructor • Moving relocation constructor (defragmentation) • Dynamic constructor (optional, can be dummy) • No default constructor!

28. Object Members Relocation • Internal pointer • Must point within the LIP item block • Converted into absolute pointer • External reference (LIP items only) • Stored as a LIP item hash ID • Converted into a pointer in the global asset table entry that points to the referenced LIP item • LIP framework provides wrapper classes with appropriate constructors for all pointer types

29. Relocation Example class GameObject { public: GameObject(const LoadContext& ctx); GameObject(const MoveContext& ctx); GameObject(HASHID id, Script* pScript); protected: lip::RelocPtr<Transfo> mpLocation; lip::LipItemPtr<Script> mpScript; };

30. Relocation Example (cont’d) GameObject::GameObject(const LoadContext& ctx) : mpLocation(ctx), mpScript(ctx) {} GameObject::GameObject(const MoveContext& ctx) : mpLocation(ctx), mpScript(ctx) {} GameObject::GameObject(HASHID id, Script* pScript) : mpLocation(new Transfo), mpScript(pScript) { SetHashId(id); }

31. Relocation Example (cont’d) template<typenameLipItemT> void PlacementNew( lip::LoadContext& loadCtx) { new(loadCtx.pvBaseAddr) LipItemT(loadCtx); } loadCtx.pvBaseAddr = pvLoadMemory; PlacementNew<GameObject>(loadCtx);

32. Relocation Example (cont’d)

33. Load-In-Place: “Load Unit” • Group of LIP items • The smallest unit of data that can be loaded • 1 load unit  1 load command • Number of files is minimized • 1 language-independent file • Models, animations, scripts, environments, … • N language-dependent files • Fonts, in-game text, some textures, audio, … • Load unit files are compressed

34. Load Unit Table • Each LIP item has an entry in the table • Hash ID • Offset to LIP Item

35. Dynamic Storage • Loading process • Load unit files are read and decompressed to available storage memory • Load unit table offsets are relocated • Load unit table entries are merged in the global asset table • A placement new is called for each LIP item • Some LIP item types may require a second initialization pass (e.g. registration)

36. Dynamic Storage • Unloading process • Each LIP item can be removed individually • All LIP items of a load unit can be removed together • Destructors are called on C++ LIP items • Dynamic storage algorithm will defragment the new holes later • Locking • LIP items can be locked • Locked items cannot be moved or unloaded

37. Platform-Specific Issues • GameCube • Special ARAM load unit files • Animations • Collision floors • Small disc  compression • Xbox/Xbox 360 • Special LIP items for DirectX buffers • Vertex, index and texture buffers • 4KB-aligned LIP items (binary blocks) • Buffer headers in separate LIP items (C++ objects)

38. Load-In-Place: Other Uses • Network-based asset editing • LIP items can be transferred from and to our level editor during gameplay • Changes in asset sizes do not matter • Used by Maya exporters to store our intermediate art assets • LIP is much more efficient than parsing XML!

39. Packaging C++ Objects Nicolas Fleury

40. Our Previous Python-Based System • class MyClass(LipObject): x = Member(UInt32) y = Member(UInt32, default=1) p = Member(makeArrayType( makePtrType(SomeClass)))

41. Cool Things with this System • Not too complex to implement. • Python is easy to use. • Introspection support. • A lot of freedom in corresponding C++ classes.

42. Problems with this System • Python and C++ structures must be synchronized. • Exporters must be written, at least partly, in Python. • Validations limited (unless you parse C++ code). • We just invented a Python/C++ hybrid.

43. C++-based system • class MyClass : public MyBaseCls { ... LIP_DECLARE_REGISTER(MyClass); uint32 x;}; • // In .cppLIP_DEFINE_REGISTER(MyClass) { LIP_REGISTER_BASE_CLASS(MyBaseCls); LIP_REGISTER_MEMBER(x);}

44. Consequences • Exporters are now written in C++. • Class content written twice, but synchronization fully validated. • Dummy engine .DLL must be compiled (not a working engine, provides only reflection/introspection). • Need a good build system. • We just added reflection/introspection to C++.

45. Member Registration Information • Name • Offset • Type • Special flags (exposed in level editor, etc.)

46. (Non Empty) Base Class Registration Information • Name • Type • Offset; calculated with: • (size_t)(BaseClassType*)(SubClassType*)1 - 1

47. Member Type Deduction • In IntrospectorBase class:template < typename TypeT, typename MemberT>void RegisterMember( const char* name, MemberT(TypeT::*memberPtr));

48. Member Type Deduction (Arrays) • In IntrospectorBase class:template < typename TypeT, typename MemberTypeT, int sizeT>void RegisterMember( const char* name, MemberT(TypeT::*memberPtr)[sizeT]);

49. Needed Information in Tools • Every class base class (to write their members too). • Every class member. • Every base class offset (to detect missing base class registration). • Every member name, size, type and special flags. • For every type, the necessary alignment and if it is polymorphic.

50. Introspection Classes (Members)