Flexible Storage Solutions with NeST for Dynamic Resource Management

NeST: Network StorageFlexible Commodity Storage Appliances

NeST: Network Storage • Flexible, commodity based, software-only storage appliances • Building a NeST should be as easy as: Finding a networked machine“Dropping” some software on itSelf-configuring to best utilize machine

NeST resource management • Dynamic user account creation provides storage for migratory grid users • Storage reservations and quota systems intelligent scheduling of data intensive applications • Matchmaking match storage opportunities & storage consumers

Condor NeSTs • Better, smarter checkpoint servers • Checkpoints are just another data file • Data replicated/migrates to different NeSTs • Condor jobs access data from closest NeST • Flexible policy support for managing disk and network resources

New worlds, New problems • Diverse hardware, software platforms - Netapp, EMC advantage • fewer platforms, control over OS - Our approach • Automate configuration to each host system • H/W (Disks): use file system or self-manage • S/W (File System): use either read/write or mmap • Implication: Flexibility at a cost? • Key is design of the software

NeST structure • Modules for communication, transfer & storage Protocol layer • Pluggable protocols allow diverse protocols to be mapped into common control flows Transfer layer • Different concurrency architectures to maximize system throughput on diverse platforms Storage layer • Provides abstract interface to disks

GFTP NeST WiND HTTP NFS Protocol Layer NeST Structure Control Logic Concurrency Architecture Multi-process Multi-threaded Nonblocking Storage Layer Raw disk Local FS RAID Memory

Many Protocols, Single Server • Single point of control - Storage quotas/guarantees can be supported - Bandwidth can be controlled & QoS provided • Single administrative interface • Set policies, Manage user accounts • Maintainable S/W • Shared code base reduces replication, increases maintainability

Protocol layer implementation • Each protocol listens on well-defined port • Central control accepts connections • Protocol layer reads from connection and returns generic request object • Analagous to Linux V-nodes • Add new protocol by writing a couple of methods

Virtual protocol class class nestProtocol { public: virtual nestRequest* receiveRequest( fd_set *ready ) = 0; virtual ssize_t Receive( char *buffer, ssize_t n )= 0; virtual ssize_t Send( char *buffer, ssize_t n ) = 0; virtual bool sendAck( ) = 0; virtual bool sendErrorMessage( int error ) = 0; virtual bool sendDirectoryList( NestFileInfo *list ); virtual bool sendPwd( char *pwd ); virtual bool sendTransferInfo( class nestRequest *req ); virtual bool sendFilesize( int retval, long size) = 0; virtual bool sendQuotaInfo( struct UserUsage* usage ); virtual bool hasSocket() = 0; virtual int getSocket() = 0; };

Virtual protocol class Plus some static methods static int listen( int port ); static nestProtocol* accept( int socket, sockaddr_in *address ); static NestReplyStatus initiateThirdParty( nestClient *client, nestProtocol **connection );

Grid FTP virtual protocol • Good news? • Hard work already done • Already implemented libraries have support for parallel streams, authentication, etc. • Bad news? • Slight mismatch to integrate callback model into synchronous API

Grid FTP - listen // static initialization routine int nestProtocolGridFtp::listen( int port ) { globus_result_t result; globus_ftp_control_server_t *server; globus_module_activate( GLOBUS_FTP_CONTROL_MODULE ); server = (globus_ftp_control_server_t *) Malloc( sizeof( struct globus_ftp_control_server_s ) ); result = globus_ftp_control_server_handle_init( server ); if ( result != GLOBUS_SUCCESS ) { goto ERROR; } short unsigned Port = (short unsigned) port; result = globus_ftp_control_server_listen( server, &Port, listenCallback, NULL ); if ( result != GLOBUS_SUCCESS ) { goto ERROR; } if ( Spipe( pipes ) < 0 ) { goto ERROR; } }

Grid FTP - receiveRequest nestRequest* nestProtocolGridFtp::receiveRequest( fd_set *ready ){ // ignore the fd_set char msg[MESSAGESIZE]; snprintf( msg, MESSAGESIZE, "%s", "Gftp receiveRequest called: " ); nestRequest *req = this->request; if ( req != NULL ) { snprintf( &(msg[strlen(msg)]), MESSAGESIZE - strlen(msg), "%s\n", "Actually have a request to give.\n" ); } else { snprintf( &(msg[strlen(msg)]), MESSAGESIZE - strlen(msg), "%s\n", "No request to give.\n" ); } nest_debug( 1, msg ); return req; }

Grid FTP - sendFilesize bool nestProtocolGridFtp::sendFilesize( int retval, long size){ assert( reqType == FILESIZE_REQUEST ); switch( retval ) { case NEST_SUCCESS: snprintf( buffer, FILEBUFFER, "213 %ld\r\n", size ); break; case NEST_REMOTE_FILE_NOT_FOUND: snprintf( buffer, FILEBUFFER, "550 File not found\r\n" ); break; default: assert( 0 ); } return sendMessage( buffer ); }

Grid FTP - sendMessage bool nestProtocolGridFtp::sendMessage( const void *vptr ) { globus_result_t result; waitIdle(); setWait( true ); result = globus_ftp_control_send_response( handle, (char *)vptr, commandCallback, this ); if ( result != GLOBUS_SUCCESS ) { fprintf( stderr, "globus_ftp_control_send_response error: %s\n", resultToString( result ) ); free( handle ); return false; } return true; }

Grid FTP - commandCallback void nestProtocolGridFtp::commandCallback( void * arg, struct globus_ftp_control_handle_s * handle, globus_object_t * error) { nest_debug( 1, "command_callback called - set wait false\n" ); if( error != GLOBUS_SUCCESS ) { fprintf(stderr,">> command_callback: error %s\n", globus_object_printable_to_string( error ) ); return; } // set the wait flag to false nestProtocolGridFtp *connection = (nestProtocolGridFtp *)arg; connection->setWait( false ); }

Concurrency architecture • Three difficult goals • Low latency • High bandwidth • Multiple simultaneous clients • No single portable solution • Multiple models provide solutions on a range of different platforms • Multi-threaded, multi-process, single process nonblocking

Concurrency architecture Nonblocking Multi-process Multi-threaded Flexible Concurrency • Control logic creates transfer object • Virtual connection from the protocol layer • Virtual connection from the storage layer • Transfer object passed to concurrency layer • Control logic informed when transfer ends

Concurrency Models • Nonblocking model • Performs only non-blocking I/O • Selects on file-descriptors / sockets from each transfer object • Pre-allocated pool of processes • Descriptors are passed over a pipe • Transfer object recreated on other side • Each process does a blocking transfer • Pre-allocated pool of threads • Transfer object enqueued by server • Dequeued by an available thread

Concurrency Models and GFTP • Nonblocking model • Not yet supported. • Grid FTP libraries do not expose socket on which to select. • Pre-allocated pool of processes • Not yet supported. • Haven’t figured out how to send a GridFTP connection over a pipe • Pre-allocated pool of threads • No problem. Fully integrated. • Other models could work . . .

Storage Layer • Three needed areas of flexibility • File systems interfaces • Example: read()/write() or mmap() • Abstract storage models • RAID, JBOD, etc. • Memory storage model also a possibility • Provide file system interface to remote memory • Useful for buffering systems like Kangaroo • Virtual storage model akin to virtual protocol layer • User account administration • Creation and removal • Quotas and guarantees for users and groups

Flexible Storage Solutions with NeST for Dynamic Resource Management