AI Crawler Demo

AI Crawler Priority Explorer

Prioritize newly discovered webpages for crawling using precomputed PageRank scores and a simple quality heuristic. This demo helps simulate how an AI crawler could choose which pages to crawl first.

Priority Score = 0.7 × PageRank + 0.3 × Quality Score
PR
PageRank — 70% Higher PageRank means the page is more authoritative and well connected in the web graph.
Q
Quality — 30% Pages ending in .edu or .gov get 1.0; other domains get 0.5.
K
Top K The crawler returns the top K pages with the highest overall priority scores.
eg
Example A page with PageRank 0.8 on a .edu site gets 0.7×0.8 + 0.3×1.0 = 0.86.
Graph & Scores Step 1
Start with a preset, or enter your own data
Web Graph URL → outlinks
Enter a directed web graph as a JSON dictionary. These are the pages discovered by the crawler.
PageRank Scores URL → score
Enter precomputed PageRank scores for the URLs in the graph.
Run the Crawler Priority Ranking Step 2
Top K
 
Tip: press Ctrl + Enter to run
Top — URLs to Crawl
# URL PageRank Quality Priority Score
Priority Score Comparison