Advanced Database Crawler and Search Engine Functionality for URL Management and Indexing
This document outlines the algorithms and operations performed by a web crawler and search engine. The crawler retrieves URLs from a database, checking their age and relevance. It extracts essential components from each URL, discards unnecessary strings, and indexes new entries into the database. The search engine processes and inserts keywords, ensuring unique entries while updating existing ones based on the search frequency. The results are displayed in descending order of keyword relevance, maintaining optimal performance and search functionality.
Advanced Database Crawler and Search Engine Functionality for URL Management and Indexing
E N D
Presentation Transcript
CRAWLER & SEARCH ENGINE SyarifahHanum M0507044
Crawling DB Sipder Indexing DB Word Searching DB Cari
CRAWLER • For i=1 to n (Banyaknya URL di DB) If URL > 2 hari then delete • For i=1 to n (Banyaknya URL di DB) • Ambil URL diDataBase • Baca URL per baris j=0 to m (akhirbaris) • Baca URL per karakter k=0 to p (karakterterakhirtiapbaris) • If karakter =href Hasil=simpanbaris • Buangkarakterdepan • Buangkarakterbelakang • Buangtulisan “_files” • Buangtulisan “_index” • If karakterawalhasil >< http hasil2=url+hasil • If hasil2 diakhiridenganico,css,pdf… hasil2 tidakdisimpan • Else Cekapakah hasil2 sudahadadi DB If sudahada update Else insert
indexER • For i=0 to n (n= banyakurldi DB spider) • Ambilalamaturldari spider • fopen=alamaturl • Baca per baris • Explode string dengankarakter “<“ • Gabung karakter2 hasil step sebelumnya • Explode string dengankarakterlainnya (kecualispasi) • Gabung karakter2 hasil step sebelumnya • Replace spasidengan “#?!” • Explode string dengankarakter“#?!“ • Cektabel word • If alamat=“” & karakter[i]=“” • Insert karakter & alamattsb &jum=1. • Else (sudahada) • Jum=jum+1;
SEARCH ENGINE • Insert keyword • Pecah keyword (pemecah = spasi) Simpandiaray • For i=1 to n (banyaknyakatakunci) • Cekketersediaanditabelcari • Jikabelumada insert • Jikasudah jumlah =keyword[‘jum’]+cari[‘jum’]; • Tampilkanhasil (DB cari) order by jum DESC;
