Obtaining Shape from Scanning Electron Microscope Using Hopfield Neural Network

1. Obtaining Shape from Scanning Electron Microscope Using Hopfield Neural Network Yuji Iwahori1, Haruki Kawanaka1, Shinji Fukui2 and Kenji Funahashi1 1Nagoya Institute of Technology, Japan 2Aichi University of Education, Japan

2. Introduction Shape from Scanning Electron Microscope (SEM) images is the recent topic in computer vision. • The position of a light source and a viewing point are the same under the orthographic projection. • The object stand is rotated to some extent through the observation. Only these conditions can be used to recover the object shape. 2D Image of SEM Recovering 3D Shape

3. Previous Approaches (1) • Photometric Stereo • Estimation using the temporal color space use multiple images under the different light source directions. • Linear Shape from Shading • Photometric Motion the position of viewing point (camera) and light source should be widely located

4. Previous Approaches (2) • Shape from Occluding Boundaries is limited to a simply convex closed curved surface • Shape from Silhouette uses multiple images through 360 degree rotation, is also unavailable to object with local concave shape • Surface Reflectance and Shape from Images Using 90 degree rotation to get the feature points However the rotation angle is limited to SEM

5. New Proposed Approach • Uses optimization with two images observed through the rotation of the object stand • The appropriate initial vector is determined using the Radial Basis Function neural network (RBF-NN) from two images during rotation. • The optimization is introduced using the Hopfield like neural network (HF-NN).

6. Characteristics of SEM Image (1) • Orthographic projection • Rotation angle • Reflectance property i : incident angle, < 70° s ≈ 0.5 • R(i) is normalized to the range of 0 and 1.

7. Characteristics of SEM Image (2) • z= F(x, y) F : height distribution p, q : gradient parameters • l = (0,0,1) : light source direction • : surface normal • cos i = n・l = nz … (3) • From Eq.(1)(2)(3), Cross Section of Reflectance Map(q=0)

8. Rotation Axis on Object Stand • Under the orthographic projection • the gradient of the rotation axis is the same for both images observed during rotation. ex.

9. Estimation of Rotation Axis • Assume Aand Bmove A’and B’during the rotation • Set Aand A’be the same pixel • Then rotation axis is determined so that it becomes perpendicular to the line BB’and passes through the point A.

10. m1 m2 m3 Shape Recovery from Two Images Using Hopfield Neural Network • Hopfield Neural Network (HF-NN) • the mutual connection network • the connection between the neurons are the symmetric • HF-NN can be applied to solve the optimization problem of the energy function

11. Energy Function to be Minimized (p,q,z): unknown variables C1, C2, C3 : the regularization parameters D : the target region of the object • E1 : the smoothness constraint • E2 : the error of the observed image brightness I(x,y) and the reflectance map R(p, q) • E3 : the error of the geometric relation for zand (p, q)

12. I1(x, y) RBF NN nx I2(x, y) nz Initial Vector for Optimization (1) • Radial Basis Function Neural Network (RBF-NN) is introduced to obtain the approximation of gradient p, q • Assume the same pixel (x, y) during the rotation. • The integration of along xdirection results in the height distribution.

13. Initial Vector for Optimization (2) • How to make dataset of RBF-NN A sphere is used to make I1 and I2 using R(p, q), where, R is since a sphere has the whole combination of the surface gradient. • How to use learned RBF-NN The corresponding point of the target object is assumed to be the same during the rotation.

14. Updating Equation using HF-NN • The equation is iteratively used to optimize the energy function, that is, each partial difference becomes 0.

15. Iteration for Optimization • The optimization is applied to each of two images repeatedly. The height z’with the rotation angle is given by • Gradient are also calculated from the height repeatedly during rotation. • C1 is gradually reduces E1 : the smoothness constraint • Optimization is terminated the value of energy function converges in comparison with that of one step before.

16. Experiments (synthesis image) • Rotation angle is 10° • Image size is 64×64 pixels • Rotation axis is along the center of the image. RBF-NN Learning Data : 2000 Learning Epoch : 15 Input Images Initial Height Recovered Height Theoretical Height MSE 1.8961 Maximum Height 10.37

17. Experiments (SEM image) • Rotation angle taken is 10 ° • Rotation axis is set from the known featurepoints A and B Input Images Theoretical Height Initial Height Recovered Height Relaxation Method Theoretical Depth 13.1031 MSE 3.8926

18. Experiments (SEM image) • Rotation angle taken is 10 ° • Rotation axis is set from the known featurepoints A and B Input Images Initial Height Recovered Height

19. Conclusion • A new method is proposed to recover the shape from SEM images. • HF-NN is introduced to solve the optimization problem. The energy function is formulated from two image during rotation. • The initial vector is obtained using RBF-NN.

20. Further works • Getting more accurate result using more images • Treatment of the inter-reflection Thank you