Solution of Sparse Linear Systems

Numerical Simulation CSE245 Solution of Sparse Linear Systems Thanks to: Kin Sou, Deepak Ramaswamy, Michal Rewienski, Jacob White, Shihhsien Kuo and Karen Veroy and especially Luca Daniel

Outline of today’s lecture • Solution of Sparse Linear Systems • Examples of Problems with Sparse Matrices • Struts and joints, resistor grids, 3-D heat flow • Tridiagonal Matrix Factorization • General Sparse Factorization • Fill-in and Reordering • Graph Based Approach • Sparse Matrix Data Structures • Scattering

5 7 9 3 4 6 8 2 1 Applications Sparse Matrices Space Frame Nodal Matrix Space Frame X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X Unknowns : Joint positions Equations : forces = 0

Applications Sparse Matrices Resistor Grid Unknowns : Node Voltages Equations : currents = 0

Applications Sparse Matrices Nodal Formulation Resistor Grid Matrix non-zero locations for 100 x 10 Resistor Grid

Applications Sparse Matrices Nodal Formulation Temperature in a cube Temperature known on surface, determine interior temperature Circuit Model

Outline • Solution of Dense Linear Systems • Solution of Sparse Linear Systems • LU Factorization Reminder. • Example of Problems with Sparse Matrices • Struts and joints, resistor grids, 3-d heat flow • Tridiagonal Matrix Factorization • General Sparse Factorization • Fill-in and Reordering • Graph Based Approach • Sparse Matrix Data Structures • Scattering

Tridiagonal Example Sparse Matrices Nodal Formulation Matrix Form How many operations to do LU factorization? m

Pivot Multiplier Tridiagonal Example Sparse Matrices GE Algorithm For i = 1 to n-1 {“For each Row” For j = i+1 to n {“For each target Row below the source” For k = i+1 to n {“For each Row element beyond Pivot” } } } Order n Operations!

Sparse Matrix Fill-In Sparse Matrices Example Resistor Example Nodal Matrix 0 Symmetric Diagonally Dominant

X X X = fill in where 0 became non-zero Fill-In Sparse Matrices Example Matrix Non zero structure Fill Ins X X X X X= Non zero

X X X X X Fill-In Sparse Matrices Second Example Fill-ins Propagate X X X X X Fill-ins from Step 1 result in Fill-ins in step 2

0 Fill-In Sparse Matrices Reordering Fill-ins 0 No Fill-ins Node Reordering - Can reduce fill-in - Preserves properties (Symmetry, Diagonal Dominance) - Equivalent to swapping rows and columns

Possible Fill-in Locations Fill-in Estimate = (Non zeros in unfactored part of Row -i) (Non zeros in unfactored part of Col -i) Markowitz product Fill-In Sparse Matrices Reordering Where can fill-in occur ? Already Factored Multipliers

Fill-In Sparse Matrices Reordering Markowitz Reordering Greedy Algorithm !

1 2 3 1 0 0 3 2 Fill-In Sparse Matrices Reordering Why only try diagonals ? • Corresponds to node reordering in Nodal formulation • Reduces search cost • Preserves Matrix Properties - Diagonal Dominance - Symmetry

Fill-In Sparse Matrices Pattern of a Filled-in Matrix Very Sparse Very Sparse Dense

Fill-In Sparse Matrices Unfactored Random Matrix Symmetric

Fill-In Sparse Matrices Factored Random Matrix Symmetric

Matrix Graphs Sparse Matrices Construction Structurally Symmetric Matrices and Graphs 1 X X X X X X X 2 X X X X 3 X X X 4 X X X 5 • One Node Per Matrix Row • One Edge Per Off-diagonal Pair

1 2 X X X X 3 X X X 4 X X X X 5 X X X X X X Matrix Graphs Sparse Matrices Markowitz Products Markowitz Products = (Node Degree)2

Matrix Graphs Sparse Matrices Factorization One Step of LU Factorization 1 X X X X X X X X 2 X X X X X 3 X X X 4 X X X X X 5 • Delete the node associated with pivot row • “Tie together” the graph edges

1 2 3 4 5 1 2 3 4 5 Matrix Graphs Sparse Matrices Example Graph Markowitz products ( = Node degree)

5 5 4 3 2 3 4 Graph Matrix Graphs Sparse Matrices Example Swap 2 with 1 X 1 2 2 1 X Eliminate 1

Graphs Sparse Matrices Resistor Grid Example Unknowns : Node Voltages Equations : currents = 0

Matrix Graphs Sparse Matrices Grid Example

What should you pivot for? Sparse Matrices For growth control? Or to avoid fill-ins? A) LU factorization applied to a strictly diagonally dominant matrix will never produce a zero pivot B) The matrix entries produced by LU factorization applied to a strictly diagonally dominant matrix will never increase by more than a factor 2(n-1) [which is anyway the best you can do by pivoting for growth control] Bottom line: if your matrix is strictly diagonally dominant no need for numerical pivot for growth control!

Sparse Factorization Approach • Assume matrix requires NO numerical pivoting. Diagonally dominant or symmetric positive definite. • Pre-Process: • Use Graphs to Determine Matrix Ordering • Many graph manipulation tricks used. • Form Data Structure for Storing Filled-in Matrix • Lots of additional non-zero added • Put numerical values in Data Structure and factor Computation must be organized carefully!

Summary of Sparse Systems • LU Factorization and Diagonal Dominance. • Factor without numerical pivoting • Sparse Matrices • Struts, resistor grids, 3-d heat flow -> O(N) non-zeros • Tridiagonal Matrix Factorization • Factor in O(N) operations • General Sparse Factorization • Markowitz Reordering to minimize fill • Graph Based Approach • Factorization and Fill-in • Useful for estimating Sparse GE complexity

Outline • Solution of Sparse Linear Systems • Example of Problems with Sparse Matrices • Tridiagonal Matrix Factorization • General Sparse Factorization • Fill-in and Reordering • Graph Based Approach • Summary of the Algorithm • Sparse Matrix Data Structures • Scattering and symbolic fill in

Sparse Data Structure Sparse Matrices Vector of row pointers Arrays of Data in a Row Matrix entries Val 11 Val 12 Val 1K 1 Column index Col 11 Col 12 Col 1K Val 21 Val 22 Val 2L Col 21 Col 22 Col 2L Row pointers Column Indices Goal: Never store a 0 Never multiply by 0 Val N1 Val N2 Val Nj N Col N1 Col N2 Col Nj

Sparse Data Structure Sparse Matrices Problem of Misses Eliminating Source Row i from Target row j Row i Row j Must read all the row j entries to find the 3 that match row i

Sparse Data Structure Sparse Matrices Data on Misses More misses than ops! Every Miss is an unneeded Memory Reference!

Sparse Data Structure Sparse Matrices Scattering for Miss Avoidance Row j Use target row approach – Row j is the target. 1) Read all the elements in Row j, and scatter them in an n-length vector 2) Access only the needed elements using array indexing!

Sparse Matrices – Another Data Structure Orthogonal linked list But if fill in occurs, pointer structures change. What do we do? Pre-compute pivoting order and sparse fill in structure symbolically

Summary of Sparse Systems • LU Factorization and Diagonal Dominance. • Factor without numerical pivoting • hard problems (ill-conditioned) • Sparse Matrices • Struts, resistor grids, 3-d heat flow -> O(N) nonzeros • Tridiagonal Matrix Factorization • Factor in O(N) operations • General Sparse Factorization • Markowitz Reordering to minimize fill • Graph Based Approach • Factorization and Fill-in • Useful for estimating Sparse GE complexity • Sparse Data Structures • Scattering and symbolic fill in

Solution of Sparse Linear Systems