Hidden Gems: UNIX Shared Libraries

1. Hidden Gems:UNIX Shared Libraries Way better than PROBUILD.

2. A Few Words About The Speaker • Tom Bascom, Roaming DBA & Progress User since 1987 • President, DBAppraise, LLC • Remote Database Management Service. • Simplifying the job of Managing and Monitoring The World’s Best Business Applications. • tom@dbappraise.com • VP, White Star Software, LLC • Expert Consulting Services related to all aspects of Progress and OpenEdge. • tom@wss.com

3. What is a UNIX Shared Library? • A collection of executable code and data which is loaded into memory separately from the launch of the main program (_progres). • Progress 4GL has supported Windows DLLs since (at least) version 8. • UNIX DLL support arrived with Progress version 9. • I’ll be using Linux as an example but the principles apply to all UNIX platforms. More Info: http://www.dwheeler.com/program-library/Program-Library-HOWTO/t1.html

4. Why Would You Use One? • To share code from 3rd party, non-Progress, sources. • To leverage proprietary internal libraries. • To gain performance improvements from routines written in faster languages. • To access low-level OS features that are not otherwise available.

5. /lib (and /lib64) /lib> ls *.so* ld-2.4.so libcrypt-2.4.so libnss_dns-2.4.so libresmgr.so.1 ld-linux.so.2 libcrypt.so.1 libnss_dns.so.2 libresmgr.so.1.0.0 ld-lsb.so.2 libc.so.6 libnss_files-2.4.so libresolv-2.4.so ld-lsb.so.3 libdevmapper.so libnss_files.so.2 libresolv.so.2 libacl.so.1 libdevmapper.so.1.02 libnss_hesiod-2.4.so librt-2.4.so libacl.so.1.1.0 libdl-2.4.so libnss_hesiod.so.2 librt.so.1 libanl-2.4.so libdl.so.2 libnss_nis-2.4.so libSegFault.so libanl.so.1 libe2p.so.2 libnss_nisplus-2.4.so libss.so.2 libattr.so.1 libe2p.so.2.3 libnss_nisplus.so.2 libss.so.2.0 libattr.so.1.1.0 libext2fs.so.2 libnss_nis.so.2 libsysfs.so.1 libaudit.so.0 libext2fs.so.2.4 libnss_winbind.so.2 libsysfs.so.1.0.3 libaudit.so.0.0.0 libgcc_s.so.1 libnss_wins.so.2 libthread_db-1.0.so libblkid.so.1 libhistory.so.5 libpamc.so.0 libthread_db.so.1 libblkid.so.1.0 libhistory.so.5.1 libpamc.so.0.81.0 libutil-2.4.so libBrokenLocale-2.4.so libm-2.4.so libpam_misc.so.0 libutil.so.1 libBrokenLocale.so.1 libmemusage.so libpam_misc.so.0.81.2 libuuid.so.1 libbz2.so.1 libm.so.6 libpam.so.0 libuuid.so.1.2 libbz2.so.1.0.0 libncurses.so.5 libpam.so.0.81.5 libwrap.so.0 libc-2.4.so libncurses.so.5.5 libpcprofile.so libwrap.so.0.7.6 libcap.so.1 libnscd.so.1 libpthread-2.4.so libxcrypt.so.1 libcap.so.1.92 libnscd.so.1.0.0 libpthread.so.0 libxcrypt.so.1.2.4 libcidn-2.4.so libnsl-2.4.so libreadline.so.5 libz.so.1 libcidn.so.1 libnsl.so.1 libreadline.so.5.1 libz.so.1.2.3 libcom_err.so.2 libnss_compat-2.4.so libresmgr.so libcom_err.so.2.1 libnss_compat.so.2 libresmgr.so.0.9.8

6. Can You Narrow That Down? • For backwards compatibility there are several versions of many libraries. • Many libraries are “niche” libraries and not generally interesting. • The most interesting (to me) are: • libc, the standard “C” library • libm, the math library • libz, the compression library • libcrypt, the encryption library

7. What Functions are Inside? $ nm -DC /lib/libz.so.1.2.3 | grep -v " _" | grep " T " 00000000000017b0 T adler32 0000000000001a70 T adler32_combine 0000000000001c10 T compress 0000000000001b40 T compress2 0000000000001b20 T compressBound 0000000000001c70 T crc32 0000000000002330 T crc32_combine 0000000000004220 T deflate 00000000000058b0 T deflateBound 0000000000005f70 T deflateCopy …

8. A Few Words of Warning • The slides that you are about to see contain very little in the way of: • Error checking. • Exception handling. • Garbage collection. • And other niceties. • While this may be customary for sample code it is very dangerous when working with real C code. • You have been warned.

9. Calling Functions in libc

10. Problem: No OS-PUTENV() NAME putenv - change or add an environment variable SYNOPSIS #include <stdlib.h> intputenv(char *string); DESCRIPTION The putenv() function adds or changes the value of environment variables. The argument string is of the form name=value. If name does not already exist in the environment, then string is added to the environment. If name does exist, then the value of name in the environment is changed to value. The string pointed to by string becomes part of the environment, so altering the string changes the environment. RETURN VALUE The putenv() function returns zero on success, or non-zero if an error occurs.

11. Calling putenv() define variable x as integer no-undo. procedure putenv external "/lib64/libc.so.6": define input parameter env as character. define return parameter x as long. end. display os-getenv( "XYZZY" ). pause. run putenv( "XYZZY=pflugh", output x ). display os-getenv( "XYZZY" ). os-command value( 'echo "$XYZZY"' ). return.

12. Calling putenv() ┌────────────────────┐ │? │ └────────────────────┘ Press space bar to continue. ┌────────────────────┐ │? pflugh │ └────────────────────┘ Press space bar to continue. pflugh Press space bar to continue.

13. Problem: No Scientific Notation NAME sprintf - formatted string output SYNOPSIS intsprintf( char *str, const char *format, ... ); THE CONVERSION SPECIFIER A character that specifies the type of conversion to be applied. The conversion specifiers and their meanings are: … e,E The double argument is rounded and converted in the style [-]d.ddde±dd where there is one digit before the decimal-point character and the number of digits after it is equal to the precision; if the precision is missing, it is taken as 6; if the precision is zero, no decimal-point character appears. An E conversion uses the letter E (rather than e) to introduce the exponent. The exponent always contains at least two digits; if the value is zero, the exponent is 00.

14. Calling sprintf() define variable x as integer no-undo. define variable c as memptr no-undo. define variable m as memptr no-undo. procedure sprintf external "/lib64/libc.so.6": define input-output parameter fStr as memptr. define input parameter mask as memptr. define input parameter arg as double. define return parameter x as long. end. set-size( c ) = 1024. set-size( m ) = 1024. put-string( m, 1 ) = "%1.4e". run sprintf( input-output c, m, 0.0123, output x ). display get-string( c, 1 ) format “x(20)”. return.

15. Calling sprintf() ┌────────────────────┐ │1.2300e-02 │ └────────────────────┘ Procedure complete. Press space bar to continue.

16. Data Type Mapping

17. Creating Your Own Shared Library

18. Overview • You’ll need some “C” source code. • Compile to object code. • Run the Linker and create a Shared Library. • Test with Progress.

19. Problem: Sub-second PAUSE define variable i as integer no-undo. define variable s as integer no-undo. s = time. do i = 1 to 1000: display i. etime( yes ). do while etime < 500: end. end. display string( time - s, “hh:mm:ss" ). return.

20. What’s Wrong With That? • 8 minutes and 20 seconds of CPU time! $ ps -ef | grep [_]progres tom 21243 27534 1 17:16 pts/0 00:02:30 /usr/pro102a/bin/_progres -1 $ pro –p shortsleep.p ┌──────────────┐ │ i │ │────── ───── │ │ 1,000 08:20 │ └──────────────┘ $ ps -ef | grep [_]progres tom 21243 27534 8 17:16 pts/0 00:10:50 /usr/pro102a/bin/_progres -1

21. Source Code #include <time.h> long shortsleep( unsigned long milliseconds ) { structtimespectmReq; tmReq.tv_sec = (time_t)(milliseconds / 1000) ; tmReq.tv_nsec = (time_t)(milliseconds % 1000) * 1000 * 1000 ; nanosleep( &tmReq, NULL ) ; }

22. Compile to Object Code $ cc –c shortsleep.c -fPIC $ ls shortsleep.cshortleep.o $ file shortleep.o ELF 64-bit LSB relocatable, AMD x86-64, version 1 (SYSV), not stripped

23. Link to Create a Shared Lib $ ld -shared shortsleep.o -o shortsleep.so $ lsshortsleep* shortsleep.cshortsleep.oshortsleep.so $ file shortsleep.so shortsleep.so: ELF 64-bit LSB shared object, AMD x86-64, version 1 (SYSV), not stripped $ nm -DC shortsleep.so shortsleep.so:0000000000100460 A __bss_start shortsleep.so:0000000000100460 A _edata shortsleep.so:0000000000100460 A _end shortsleep.so: U nanosleep shortsleep.so:0000000000000280 T shortsleep

24. Progress 4GL Test Harness define variable i as integer no-undo. define variable x as integer no-undo. define variable s as integer no-undo. procedure shortsleep external "./shortsleep.so": define input parameter ms as long. define return parameter x as long. end. s = time. do i = 1 to 1000: display i. run shortsleep( 500, output x ). end. display string( time - s, “hh:mm:ss" ). return.

25. Test With Progress $ pro –p shortsleep.p Could not open Dynamic Library: ./shortsleep.so (8013) DLL Error : ./shortsleep.so: wrong ELF class: ELFCLASS32 (8014) Press space bar to continue. $ file $DLC/bin/_progres _progres: setuid ELF 64-bit LSB executable, AMD x86-64, version 1 (SYSV), for GNU/Linux 2.4.0, dynamically linked (uses shared libs), for GNU/Linux 2.4.0, not stripped

26. Test Again! • No measurable CPU time! $ ps -ef | grep [_]progres tom 21243 27534 1 17:16 pts/0 00:10:50 /usr/pro102a/bin/_progres -1 $ pro –p shortsleep.p ┌──────────────┐ │ i │ │────── ───── │ │ 1,000 08:20 │ └──────────────┘ $ ps -ef | grep [_]progres tom 21243 27534 8 17:16 pts/0 00:10:50 /usr/pro102a/bin/_progres -1

27. Problem: Regular Expressions NAME regcomp, regexec, regerror, regfree - POSIX regex functions SYNOPSIS #include <sys/types.h> #include <regex.h> intregcomp( regex_t *preg, const char *regex, intcflags ); intregexec( const regex_t *preg, const char *string, size_tnmatch, regmatch_tpmatch[], inteflags ); size_tregerror( interrcode, const regex_t *preg, char *errbuf, size_terrbuf_size ); void regfree( regex_t *preg );

28. Source Code #include <regex.h> long regex_match( char *pattern, char *string ) { regex_txpattern; size_tmatchDepth = 2; regmatch_tpmatch[2]; int offset; if ( regcomp(&xpattern, pattern, REG_EXTENDED|REG_ICASE ) != 0 ) { offset = -2; } else { if ( regexec(&xpattern, string, matchDepth, pmatch, 0 ) != 0 ) { offset = -1; } else { offset = pmatch[1].rm_so; } } regfree( &xpattern ); return offset; }

29. Compile and Link… $ cc –c rx2.c -fPIC $ ld –shared rx2.o –o rx2.so $ file rx2.so rx2.so: ELF 64-bit LSB shared object, AMD x86-64, version 1 (SYSV), not stripped $ nm -DC rx2.so 00000000001004e8 A __bss_start 00000000001004e8 A _edata 00000000001004e8 A _end U regcomp U regexec 0000000000000318 T regex_match U regfree

30. Progress 4GL Test Harness define variable pattern as character no-undo format "x(60)". define variable string as character no-undo view-as editor inner-lines 10 inner-chars 60. define variable x as integer no-undo. procedure regex_match external "./rx2.so": define input parameter pattern as character. define input parameter string as character. define return parameter x as long. end. update pattern skip string skip with side-labels. run regex_match( pattern, string, output x ). display x. return.

31. Test! $ pro –p rx.p ┌──────────────────────────────────────────────────────────┐ │pattern: .*(Fox).* │ │string: The quick brown fox jumped over the lazy dog! │ │ │ │ │ │ │ │ │ │x: 16 │ └──────────────────────────────────────────────────────────┘

32. Shared LibraryChallenges

33. Which Library to Use? • Use a variable? NOT!  • Use a pre-processor. • Use symbolic links. • Wait for 10.2B. procedure putenv external "/lib64/libc.so.6": define input parameter env as character. define return parameter x as long. end.

34. Variable Argument Lists • Isolate different cases in multiple .p files. • Use wrapper libraries. • Wait for 10.2B. intsprintf( char *str, const char *format, ... );

35. Call-Backs? • Not supported . void qsort( void *base, size_tnmemb, size_t size, int(*compar)(const void *, const void *) );

36. Persistence • About 25% faster in crude testing. • RELEASE EXTERNAL “/lib64/libc.so.6”. procedure putenv external "/lib64/libc.so.6" PERSISTENT: define input parameter env as character. define return parameter x as long. end.

37. MEMPTRs define variable x as integer no-undo. define variable c as memptr no-undo. define variable m as memptr no-undo. procedure sprintf external "/lib64/libc.so.6": define input-output parameter fStr as memptr. define input parameter mask as memptr. define input parameter arg as double. define return parameter x as long. end. set-size( c ) = 1024. set-size( m ) = 1024. put-string( m, 1 ) = "%1.4e". run sprintf( input-output c, m, 0.0123, output x ). display get-string( c, 1 ) format “x(20)”. set-size( c ) = 0. set-size( m ) = 0. return.

38. DynamicShared LibraryCall (Special Bonus Slide!)

39. Dynamic Shared Lib Call define variable LIBC as character no-undo initial "/lib64/libc.so.6". function os-putenv returns integer ( input env as character ): define variable hCall as handle no-undo. define variable rVal as integer no-undo. create call hCall. assign hCall:call-name = "putenv" hCall:library = LIBC hCall:call-type = DLL-CALL-TYPE hCall:num-parameters = 1 hCall:return-value-dll-type = "LONG" . hCall:set-parameter( 1, "CHARACTER", "INPUT", env ). hCall:invoke(). rVal = hCall:return-value. delete object hCall. return rVal. end.

40. Conclusion • Share code from 3rd party, non-Progress, sources. • Gain performance improvements from routines written in faster languages. • Access low-level OS features that are not otherwise available. • It beats the heck out of using PROBUILD.

41. Questions?

42. Thank-you!

44. dlopen() and Friends NAME dladdr, dlclose, dlerror, dlopen, dlsym, dlvsym - programming interface to dynamic linking loader SYNOPSIS #include <dlfcn.h> void *dlopen( const char *filename, intflag ); char *dlerror( void ); void *dlsym( void *handle, const char *symbol ); intdlclose( void *handle ); DESCRIPTION The four functions dlopen(), dlsym(), dlclose(), dlerror() implement the interface to the dynamic linking loader.