The Tech Behind the Tools of Insomniac Games

1. The Tech Behind the Tools of Insomniac Games Geoff Evans – Insomniac Games

2. Welcome! • An overview of how Insomniac’s tools get current generation games out the door • There are no slick demos of our game engine • This talk is focused tools internals, pipelines, and best practices we have found

4. Outline Areas of improvement: • Revision control • Asset organization • Flexible Asset definition • Efficient Asset Processing

5. Warning: Open Source Ahead • Nocturnal Initiative is Insomniac's Open Source project for tools code & more • Portions available for download under BSD-style license, long term initiative • Wiki: • http://nocturnal.insomniacgames.com • Perforce: • nocturnal.insomniacgames.com:1666

6. Problem • Proprietary revision control system • Implicit file retrieval • Files synced before open • Per-file database overhead • Written files queued for check-in at process exit • Heavy code instrumentation (iopen, iclose) • Damage spreads immediately • All processed data under revision control

7. Solution • Move all revision control to Perforce • Handles branching of binaries efficiently • Label code and data together • Explicit asset retrieval • Users decide when to download • Big speed improvement • Better stability while getting a unit of work done • Remove built data from revision control

8. Branching • We branch all files for milestones • Work continues in the main line during milestone stabilization • Milestone fixes brought back piecemeal • Milestone hacks stay in branch forever

9. Continuous Integration • Custom system implemented in Perl • Populates a ‘Known Good’ label in Perforce • Failure notifies users based on recent checkins(changes between testing and last known good changelist numbers) • Moving forward we intend to sync to ‘Known Good’ by default (code and assets)

10. Problem • Problem: Asset tree organized by engine type • Tools and engine expected numbered assets • Moby 500: Wooden Breakable Crate • Level 9000: My Test Level • Storage location of each asset was fixed or had a natural home in the tree: • /Data/Mobys/Moby500/actor.xml (fixed location) • /Art/Mobys/Moby500/crate.mb (natural home)

11. Problem • Numbered assets are simple to program for • Reference any asset using type and id • Lacked descriptive names, hard to find, hindered re-use of assets • Users tracked assets with pen and paper • Changing the type of an asset lead to complex copy logic that had to copy files around

12. Solution • Tools shouldn’t force asset organization • Users should be able to put files anywhere • Assets are defined in files with a reasonable names (BreakableCrate.entity vs. moby500) • Files should be able to be moved or renamed without a lot of work • Renaming a single file is simple, but references to that file are troublesome

13. Asset Referencing • Historically references are file path strings • Makes renaming content files difficult and error prone • Asset A references B, renaming B to C breaks A • Housecleaning before a sequel game causes pain due to broken references • Typically fixed up by brute force

14. Asset Referencing • Don’t rely on string paths for referencing files • Use a unique IDs to indirect the file location • IDs are assigned and shared via events • Event files are retrieved with asset files • Event files contain simple event logic: • File <foo> was added with id <blah> • File <foo> was moved to <bar> • File <bar> was deleted

15. Event System • Each event is written to a file per-user, per-computer, per-project, per-branch • “geoff-geoff_xp-rcf-devel.event.dat” • Avoid contention for file checkout • Avoid conflict resolution during integrate • EventSystem collates events, sorts by time • Events handled only once

16. File Resolver • File Resolver maps unique ID -> File path • Consumes events from Event System • Populates a local machine SQLite DB • Native formats store only the unique ID • Maya stores ID in dynamic attributes • String paths are fixed up at load time by plug-in • TUID – 64 bit unique ID • Open Source via Nocturnal

17. Asset Referencing • Processed data is output to folders named for the file id of the Asset definition file • Even though assets are renamed, processed data location remains the same • Helps efficiency on the back end of pipeline • It’s possible to reference assets in game by the file ID by hand, but we try to avoid this

18. Here be Dragons • Since databases are built on each computer, it can have tricky machine-specific bugs (revert) • We commit Add and Rename operations immediately to fight missing asset references (dangling TUIDs), data files may be valid stubs • Moving forward we think its possible to store resolver data as meta-data and eliminate collation of Events altogether

19. Problem • Problem: Assets defined by engine type specific data structures • Our assets were defined in terms of our engine, not in terms of the content our engine was consuming • This made changing the engine type arduous since the data could be very different

20. Solution • Asset data should be modular, and that modularity should drive the engine type • Asset definition should serve the content, not specific engine types (wherever possible) • Similar data about Assets should be reusable between engine types

21. Asset Fundamentals • What is an Asset: A discrete unit of content that needs to be processed by one or more builders to be loaded by the game • An Asset itself has only basic properties • An Asset is a collection of Attributes • Asset : public AttributeCollection

22. Assets (C++ Classes)

23. Attributes • Attribute: A block of related information • AnimationAttribute: what animations to use • PhysicsAttribute: physics subsystem info • VisualAttribute: render information • Attributes store the bulk of the complexity • Stored in ‘slots’ using class type id (int32)

24. Attributes (C++ Classes)

25. Asset Example • EntityAsset – content backed placeable object • ArtFileAttribute – reference Maya content file • VisualAttribute – render information

26. Asset Example • LevelAsset – top level chunk of game data • WorldFileAttribute – placed objects in the level • DependenciesAttribute – list of assets to pack

27. Asset Example • ShaderAsset – texture and render properties • ColorMapAttribute – texture, compress settings • NormalMapAttribute – processed differently

28. Attributes • Simplifies sharing structures between Assets • Attributes only exist for features that are configured by content creators (auditing) • Easy to copy and paste between Assets

29. Validating Attributes • If we aren’t careful users could easily create non-sensical Assets • LevelAsset with a TextureMapAttribute? • Shader with a PhysicsAttribute? • Must be able to validate Attributes • Attribute may not work with some Asset classes • Attributes may not work in conjunction

30. Attribute Behavior • Behavior defines default compatibility • Exclusive: ColorMapAttribute • Validated by ShaderAsset, not by LevelAsset • Inclusive: DependenciesAttribute • References to miscellaneous resources • Useful for spawn-only types like weapons, HUD • Loosely related assets like weapons and ammo

31. Attribute Usage • Usage implies Class/Instance relationship • Class – Used in classes only • Instance – Used in instances only • Overridable – Works in both class and instance • Allows us to easily override class settings per instance on an Attribute level

32. Attribute Usage • Our SceneNode class also subclasses AttributeCollection:

33. Attribute Usage • BakedLightingAttribute defines light map dimension and compression settings • Instances of that EntityAsset can override the light map information (if its scaled way up or down)

34. Attribute Sibling Validation • Some Attributes inherently conflict • Attributes can reject potentially invalid siblings • Ex: BakedLightingAttribute will reject the addition of PhysicsAttribute because baked lighting doesn’t work with dynamic objects

35. Engine Type Classification • Assets map to one or more Engine Types • If Asset map to one type, it’s a natural match • ShaderAsset -> ShaderBuilder • LevelAsset -> LevelBuilder • If Asset class maps to more than one • Attributes determine Engine Type • EntityAsset -> ShrubBuilder, TieBuilder, etc…

36. Entity Assets • Shrub – light weight static art • ArtFileAttribute – Maya content file • VisualAttribute – render information Shrub

37. Entity Assets • Tie – more expensive static art • CollisionAttribute – makes it collidable • BakedLightingAttribute – lightmap size Tie

38. Entity Assets • Moby – animated character • PhysicsAttribute – Physics subsystem settings • AnimationAttribute – Animation assets to use Moby

39. Too many Attributes! • Plethora of Attributes is confusing to artists • Keeping Attributes simple is key • Attributes are cumbersome because knowing which Attributes yield each Engine Type its costly to learn

40. Just make it a <Engine Type>! • Sometimes users know what Engine Type they want, and need the ability to conform the Attributes to that Engine Type • Attributes have Enable flag which can be used to disable offending Attributes so data is not lost, just dormant • Changing the type back re-enables the disabled Attributes

41. What Attributes do I need again? • Setting up Assets from scratch can still be non-trivial… basic Attributes are needed to get an Asset up and running • We created a Wizard with simple presets for each Engine type • Moving forward we are going to select creation preset based on export data contents

42. Problem • Problem: .NET was not very productive for generating UI for editing our Assets • We don’t want to give up writing Native code, so .NET in general was not a good fit for us • Marshalling data structures between Native and Managed was time consuming • Windows::Forms::PropertyGrid was hard to deal with to get good custom property editing

43. Solution • Drop .NET and develop our own solutions • Our C++ reflection wasgetting feature-rich • We already had a data-driven properties GUI • Switched to C++ / wxWidgets • Back up and running after 1 man month • Simplified everything

44. Asset Editor UI • Assets and Attributes are loaded into a single tree view • References to other Assets are expanded in-place • Properties displayed for each node in the tree • Property interaction can update information in real-time on the target platform

45. Asset Editor UI

46. What’s under the hood • We generate property GUI for Asset/Attributes from C++ reflection information • Adding a field to an Asset or Attribute shows up automatically • UI-script can add sliders, drop-downs, color pickers, etc…

47. Reflect • Provides class and field level meta-data • All meta-data created at runtime • No compile time parsing required • Adding a field is usually a single line of code • Type Registry assigns type id’s at runtime • Automated persistence, cloning, visitor API • Handles data format changes & legacy files • Open Source via Nocturnal

48. Reflect Classes class Version : public Reflect::ConcreteInheritor<Version, Element> { public: std::stringm_Source; static void EnumerateClass( Reflect::Compositor<Version>& comp ) { comp.AddField( &Version::m_Source, "m_Source" ); } }; void Init() { Reflect::RegisterClass<Version>( "Version" ); }

49. Reflect Enumerations namespace BlendTypes { enumBlendType { Normal, Additive };  static void EnumerateBlendType ( Reflect::Enumeration* info ) { info->AddElement(Normal, "Normal"); info->AddElement(Additive, "Additive"); } } typedefBlendTypes::BlendTypeBlendType; void Init() { Reflect::RegisterEnumeration<BlendType>("BlendType”, &BlendTypes::EnumerateBlendType ); }

50. Inspect • System for data-driven property GUIs • Control class hierarchy • Wrapper classes for wxWidgets UI elements • Can be factory created from UI script • Interpreter class hierarchy • Intimate with data formats • Automates the creation of the UI • Handles callbacks from user interactions • Open Source via Nocturnal