Numerical Grid Computations with the OPeNDAP Back End Server (BES)

1. Numerical Grid Computations with the OPeNDAP Back End Server (BES)

2. The Problem • Two part problem; • Compute the so called Stokes vector for a given vector parameter space P using a model F(x,y,λ; P) • Correct P to improve the fit of the theoretical signal to the observed signal using non-linear optimization • The purpose is to determine a vector parameter space for which the model produces the best fit in the Least Square sense. • The complete problem is solved with a Numerical Library Called MERLIN • Very computational Intensive, one complete dataset can take hours on a single node.

3. High Performance Solution • Create multiple versions of the computing node waiting for data to crunch • Create a central place (controller) to browse spectra- polarimetry data inventories and launch inversions upon user’s request. • Let the controller decide how to distribute the problem. • Requires multiple identical processes waiting to be called for duty.

4. Born to Grid • Hyrax BES was designed from the beginning with grid services in mind • Suited for problems that DO NOT REQUIRE coordination of the points after each iteration (see next slide) • Each point is solved individually regardless of the solution to all other points.

5. What this problem is not • Do not think of a problem like the Navier-Stokes equation using a cluster and finite element methods with mesh refinement. • Think Scatter-Gather or Embarrassing Parallel . The Spectral Inversion Problem does not require boundary information to be exchanged in the solution.

6. Y λ X The Dataset: Spectral Images Single element from data cube

7. A slice is nice! Each node gets a Y time λ slice

8. We do not call it scatter-gather for nothing • Each computational node, that is, each BES gets from the controller an OPeNDAP URL specifying the dataset on which to operate and the specific slice using OPeNDAP constraints. • Each computational node returns the solution to the controller as an OPeNDAP data object. • Controller collects all solutions and assembles a complete result as a netCDF data file. • Time to compute solution scales LINEARLY with number of nodes • Grid can be homogeneous or heterogeneous.

9. GRID BES Server 1 BES Server 2 . . . BES Server n HOW SLICING WORKS IN THE INVERSION ENGINE APACHE + MODULE Client Data local or remote using OPeNDAP over HTTP Inversion Engine Each server computes the inversion over spectral images (l,y), all n servers compute the complete set (x,l,y),

10. HYRAX BES RUNNING MERLIN AS A MODULE Remote OPeNDAP server Genetic Algorithm or Neural Network provides Initial Guess CALIBRATED INPUT DATA (e.g. ASP, DLSP, HINODE, ...) Process with Custom Data Filters to Standard Data Engine Format Inversion Engine MERLIN Polarization Reference Frame: +Q along Solar E-W OUTPUT INVERSION RESULTS ANCILLARY DATA: Instrument Profile Dispersion Spectral Bandpass(es) Atomic Data for Spectral Lines Data Source Process

11. Conclusions • Hyrax BES can dynamically load modules that not only do data serving and manipulation, but server side functionality as well. • The use of multiple servers running Hyrax can easily be implemented to linearly improve the performance. This applies to both server side processing as well as load balancing for serving data in demanding systems like the Earth System Grid. • Hyrax Modules offer a simple way to extend the functionality of your server, as well as seamlessly integrate multiple facets of data serving and processing.

12. QUESTIONS?