IPSI Belgrade Ltd.

IPSI Belgrade Ltd. ARTSHOP GALLERY vm@ipsi.co.yuoffice@ipsi.co.yuhttp://www.ipsi.co.yu This material was developed with financial help of the WUSA fund of Austria.

Authors Marinkovic Ivan Stojanovski Aleksandar Nikezic Gavro Radakovic Miroslav Skundric Nikola Milutinovic Darko Toskov Ivan Vujovic Ivana Milutinovic Veljko Anucojic Goran

Introduction – IPSI Bgd IPSI Belgrade is a company jointly founded by German and Serbian capital Partners: - IPSI Fraunhofer, Darmstadt, Germany- Telecom Italia Learning Services, Italy- NYU, School of Continuous Professional Studies, USA- Instituto Tecnologico de Durango, Mexico

Introduction – IPSI Bgd - Workspaces of the Future- Environments for Cooperative Working and Learning- Virtual Information and Knowledge Environments- Mobile Interactive Media- Open Adaptive Information Management Systems- Publication Engineering and Technology- Hardware Design and Operating Systems- Networks and WWW

Introduction – IPSI Bgd • Products: • Advanced Virtual Gallery • Semantic Web tutorial and book development • The injection cache, the STS cache, VLSI Detection for Internet/Telephony Interfaces, Genetic Search with Spatial/Temporal Mutations, Browser Acceleration, Technology transfer, Testing the Infrastructure for EBI, Socratenon Distant Web Educating Machine, e-Tourism…

Problem Statement - Creating Web based art gallery with “look and feel” of the real world exhibitions - Visitor moves through the gallery by “walking with options”

Existing Solutions • Musee national des Arts asiatiques http://www.museeguimet.fr/tour-guimet/index.html • Web Server of the Galleria degli Uffizi in Florence http://www.uffizi.firenze.it • The Distributed Interactive Virtual Environment (DIVE) http://www.sics.se/dive/ • The Web3D Repository http://www.web3d.org/vrml/artgal.htm

Proposed Solution - Virtual reality gallery - Advanced search capabilities - Visitor’s criteria based room generation

Why is it better? - Dynamically generated gallery - Content based search engine - User satisfaction

Conditions and Assumptions - PC- Internet connection- Internet Explorer 5.0 or higher - Netscape 7.0 - Cortona VRML plug-in for IE

Analysis and Implementation • - Application is written in ASP.NET using C# as code-behind, and ADO.NET for database access. • - Database server is SQL Server 2000. • Communication with the database is entirely made through XML (using SQLXML3.0 framework). • Queries are made in XPath, while adding, changing and deleting of the records is done through UpdateGrams. • - Application is optimized for Internet Explorer 5.0 or higher, at the 1024x768 screen resolution. Netscape 7.0 or higher is also supported. • - 3D gallery is completely generated on the server side (dynamically) using VRML.

Track 1 Track Requirements: • To stand as the integrative part for the other two tracks • To provide: • User interaction • Database connectivity (database independent) • Search functions (simple and advanced using Track3 output) • Information brokering between artists and buyers • Administration tools • Artworks management tools, etc. • Thin client (3D scene generation on server side)

Track 1 Development Tools: • Application server platform: • Windows XP Professional • IIS 5.1 • MS SQL Server 2000 • Development platform: • ASP.NET • C# as code-behind. • Communication with the underlying database: • XML & XSD using XPath queries (DB independent) • Currently using SQLXML3.0 add-on for ADO.NET

Track 1 Database Design:

Track 1 Administrator Tools: • Separate entry point:http://<server_address>/artshop/admin

Track 1 • Entry point:http://<server_address>/artshop/index.htm Users & Exhibitors:

Track 1 Interesting Details: • Native XML DBMS under development at IPSI Fraunhofer • Practical testing of the XML/XPath database access • Dynamic addition (to the system) of new multimedia types • 3D view of search results

Track 1 Interesting Details: • Application that can connect on the fly to any DBMS which supports XML/XPath is an interesting and possibly useful idea (user just has to set one XML file containing local field mapping, and one XSD to map the database fields to the pre-defined scheme) • Cons: • XPath queries are lot less powerful then standard SQL queries • Inherently, loss of speed (one complex SQL query had to be simulated with couple of XPath queries and additional processing in the code). • For now, SQLXML3.0 does not support complete XPath standard.

Track 1 Errors Made: • Initially, content analysis, picture processing, and adding data to database were completely separated (as specified in the contract), with the idea of later (partial) integration. • Turned out to be a bad idea(required a lot of interventionfrom the ArtShop system administrator when adding artworks).

Track 1 Lessons Learned: • Problem solved by complete integration of forementioned tasks into the one system process which monitors input directory, automatically schedules picture processing and content analysis, and takes care of updating of all necessary fields in all required databases. • With that, we achieved maximum automation, reduced time needed for artwork addition, and reduced amount of data transferred through the Internet (between the administrator’s machine and the application host).

Track 2 Image-Content-Oriented Search Track requirements: • Images used for extracting objects are artistic paintings • Image analyses • Extraction of the features • Create XML file for each image • Fetch the database with the features

Load parameters, input and output directory Open the picture Determine the filter value Put the picture into the reduced matrix Determine histogram Create objects Merge objects into bigger objects Create sorted array of objects Create database objects, prepare them, and put them in XML file and tables in database Track 2 Algorithm:

Create general histogram Sort histogram Remove zero values Remove redundancies Sort histogram Refresh matrix Track 2 Determine histogram:

P8 P8 P3 P8 P3 P5 P6 P5 P4 P5 P4 P8 P3 P8 the first colon the last row the rest of the matrix the last colon the last element in the last row Track 2 Regions for processing matrix:

false Any pixel left in the current region? false true p8 = the current pixel p8.oi == 0 (it doesn’t belong to any object) false true create new object Any neighbour pixel left? false true px = the neighbour pixel p8.oi != px.oi (don’t belong to the same object) true call matrix.unite_pixels method take the next neighbour pixel take the next pixel Creating objects:

Is Picture_Objects list empty? true false q = true; Edge[0] = Picture_Objects[0] false q==true (are there any objects inside Border which colors after translation are the same as the color of Core object) true i < number of objects inside Edge list false true Take ith object Find all neighbour objects, which colors after translation are the same as the color of Core object, and put them into Border i = i+1 Move all pixels from objects inside Edge to Core object Remove objects that are inside Edge from Picture_Objects list Is Border empty true false Move all objects inside Border to Edge q=false; add Core object to big list; Remove pixels belonging to Core object from Picture_Objects Merge objects into bigger objects:

Track 2 Tools used in development: • C# programming language – the chosen tool • Advantage: Includes the best properties from other programming languages (C++, Java, Visual Basic) • Disadvantage: slower processing speed than C++, which is not necessary in this application • C++ - the best alternative tool • Advantage: faster processing speed (unnecessary) • Disadvantage: more complicated code, 50% of all bugs due to use of pointers

Track 2 Original picture:

Track 2 Picture after applyinghistogram values:

Track 2 Picture representedthrough extracted objects:

Index of histogram array Hue Saturation Luminance Description 0 any any <=30 black 1 any any >=lummax white 2 any <20 >30 <30+lum the darkest grey ... … … … … ilmax-1 any <20 >=30+(ilmax-3)*lum <30+(ilmax-2)*lum the lightest grey ilmax <=hue/2 >=240-hue/2 >20 <20+1*sat >30 <30+lum the darkest red with the smallest saturation ilmax+1 <=hue/2 >=240-hue/2 >=20+1*sat < 20+2*sat >30 <30+lum the darkest red with smaller saturation … … … … … ilmax+isat-2 <=hue/2 >=240-hue/2 >=20+(isat-3)*sat < 20 +(isat-2)*sat >30 <30+lum the darkest red with the biggest saturation ilmax+1*(isat-1) <=hue/2 >=240-hue/2 >20 <20+1*sat >=30+lum < 30+2*lum darker red with the smallest saturation ilmax+1*(isat-1)+1 <=hue/2 >=240-hue/2 >=20+1*sat < 20+2*sat >=30+lum < 30+2*lum darker red with smaller saturation … … … … … ilmax+1*(isat-1)+isat-2 <=hue/2 >=240-hue/2 >=20+(isat-3)*sat < 20 +(isat-2)*sat >=30+lum < 30+2*lum darker red with the biggest saturation ilmax+2*(isat-1) <=hue/2 >=240-hue/2 >20 <20+1*sat >=30+2*lum < 30+3*lum dark red with the smallest saturation Track 2 3D HSL space => 1D histogram

ilmax+2*(isat-1)+1 <=hue/2 >=240-hue/2 >=20+1*sat < 20+2*sat >=30+2*lum < 30+3*lum dark red with smaller saturation … … … … … ilmax+(ilmax-3)*(isat-1)+1 <=hue/2 >=240-hue/2 >=20+1*sat < 20+2*sat >=30+(ilmax-3)*lum < 30+(ilmax-2)*lum the brightest red with smaller saturation … … … … … ilmax+(ilmax-3)*(isat-1)+(isat-2) <=hue/2 >=240-hue/2 >=20+(isat-3)*sat < 20 +(isat-2)*sat >=30+(ilmax-3)*lum < 30+(ilmax-2)*lum the brightest red with the biggest saturation ilmax+1*(ilmax-2)*(isat-1) > hue/2 < 3*hue/2 >20 <20+1*sat >30 <30+lum the darkest orange-red with the smallest saturation ilmax+1*(ilmax-2)*(isat-1)+1 > hue/2 < 3*hue/2 >=20+1*sat < 20+2*sat >30 <30+lum the darkest orange-red with smaller saturation … … … … … ilmax+1*(ilmax-2)*(isat-1)+(ilmax-3)*(isat-1)+(isat-2) > hue/2 < 3*hue/2 >=20+(isat-3)*sat < 20 +(isat-2)*sat >=30+(ilmax-3)*lum < 30+(ilmax-2)*lum the brightest orange-red with the biggest saturation ilmax+2*(ilmax-2)*(isat-1) >=3*hue/2 < 5*hue/2 >20 <20+1*sat >30 <30+lum the darkest red-orange with the smallest saturation ilmax+2*(ilmax-2)*(isat-1)+1 >=3*hue/2 < 5*hue/2 >=20+1*sat < 20+2*sat >30 <30+lum the darkest red-orange with smaller saturation … … … … … ilmax+(ihmax-1)*(ilmax-2)*(isat-1)+1 >=(2*ihmax-1)*hue/2 <(2*ihmax+1)*hue/2 >=20+1*sat < 20+2*sat >30 <30+lum the darkest magenta-red with smaller saturation … … … … … ilmax+(ihmax-1)*(ilmax-2)*(isat-1)+(ilmax-3)*(isat-1)+(isat-2) >=(2*ihmax-1)*hue/2 <(2*ihmax+1)*hue/2 >=20+(isat-3)*sat < 20 +(isat-2)*sat >=30+(ilmax-3)*lum < 30+(ilmax-2)*lum the brightest magenta-red with the biggest saturation Track 2 3D HSL space => 1D histogram

Track 2 Lessons learned: • It is impossible to extract objects using only colors as a criterion • It is impossible to extract objects, even using textures, edges, different transformations as criteria • Semantics should be used in segmentation • Colors are the most important features in artistic paintings

Track 3 Track requirements: • Possibility of moving through 3D galleries • Automatic generation of 3D galleries based on user’s query • Manual generation of 3D galleries • User interface for image zooming • Application for image processing

Track 3 Underlying algorithms: • Dynamic creation of gallery • Creation of static galleries • Algorithm for picture zooming • Algorithm for picture processing

Track 3 Creation of galleries: • Validation of the created gallery • Forming VRML files depending on users query • Determining the number of pictures in the gallery • Drawing a 2D floorplan based on the 3D gallery • “Forest fire” algorithm for filling the floorplan with color

Track 3 Picture processing: • Loading image into memory • Clone image into different-size copies • Filtering of copies • Parting of copies

Track 3 Development tools: • C# in .NET Framework for programming image processing • Macromedia Dreamweaver for programming zoom tool • VRML Pad v2.0

Track 3 Flowchart:

Track 3 Creation of galleries: • Making files based on users data • Putting data on server so it can be available for artists • Artist chooses which gallery he/she will be using for exhibition • User can move through 3D world • Selecting the textures for gallery • Selecting the starting position of the user

Track 3 NF filter details: • If the new picture is smaller, every pixel is one pixel of the old picture. • If the new picture is bigger, pixels are calculated based on the pixels surrounding the current.

Track 3 Errors made: • Requests were not precise, so there was a gap at the end of the project between wanted and done • Better results could be done with better using of ASP and XML

Track 3 Lessons learned: • Every member of the team gets a part where his experience is dominating • More planning at the start reduces a lot of work later • Good communication between programmers can save a lot of time

Demo

Future Plans • Improving the existing 3D dynamic gallery • Improving search engine capabilities • Improving feature extraction algorithms and objects recognition

Future Track 1 • 3D Multimedia Showroom Environment: • Implementing a general Web based 3D Multimedia Showroom Environment • Exhibiting various MM data types: images, 3D objects, videos, audio, etc. • Set of MM data types should be extendable

Future Track 2 • MM Object Feature Extraction: • Implementing algorithms and software components for extracting features from MM data types (images, videos, 3D objects), in order to enable content based search • System should be extendable (“plug-in”)

Future Track 3 • Semantic Abstraction of MM Feature Spaces: • Developing methods and SW components which derive mapping from extracted features of MM objects to semantic concepts • Using intelligent classification algorithms (Neural Networks, Fuzzy Classifier) • Developing semantic query engine (answering questions, which could previously only be answered by humans)

Usability • Art Galleries • Museums • Exhibition Fairs

Instead of Conclusion IPSI Belgrade, office@ipsi.co.yuhttp://www.ipsi.co.yu

IPSI Belgrade Ltd.