Browse Browser Research
Browser Researchcapability

sitemap probe

Read a site's robots.txt + sitemap(s) and surface its URL inventory — the CHEAPEST discovery step, run BEFORE driving a page. robots.txt and sitemap.xml frequently hand you the stable data URLs directly: a `data_like_urls` list of .json/.csv/.xlsx/.xml/.pdf and /api endpoints you can `download_file` or `call_api` straight away, skipping the browser entirely. Also returns the disallow list (where a

About this entry

Observation dates are unavailable in the catalogue metadata.