Crawling and URL harvesting: gospider, gau, qsreplace

Once you know which hosts are live, the next layer is URLs: pages, endpoints, parameters and old paths. Two kinds of tool build it. Archive miners read URLs that were public in the past without touching the target. Crawlers walk the live site. You merge both, keep the parameterised URLs, and hand the noisy pile to the NotHoly chat to group by endpoint and point out the parameters worth testing.

#Mine the archives first

Archive sources never send a request to the target; they read what search engines and the Wayback Machine already stored.

waybackurls reads the Wayback Machine.

echo example.com | waybackurls | anew urls.txt

gau, short for getallurls, pulls from several archives at once.

gau --subs example.com | anew urls.txt

Both can return tens of thousands of old URLs, including paths that no longer appear in the site's navigation but still answer on the server. This is the richest passive layer and it costs the target nothing.

#Crawl the live hosts

Crawlers send requests, so run them only against in-scope hosts and mind the rate.

gospider follows links, forms and JavaScript.

gospider -S live.txt -d 2 -c 10 --other-source | anew urls.txt

hakrawler is a smaller, faster crawler that reads a list on stdin.

cat live.txt | hakrawler -d 2 | anew urls.txt

Keep depth low at first. Depth two across ten hosts already produces more than you can read in a sitting, and a deep crawl of a large app can send thousands of requests before you notice.

#Filter down to parameters

A URL with a query string is where most testing starts. qsreplace rewrites every parameter value so you can normalise and dedupe the list.

cat urls.txt | grep '=' | qsreplace FUZZ | anew params.txt

Now params.txt holds one line per unique parameter shape instead of thousands of near-duplicates. gf adds saved grep patterns for classes like redirect, ssrf and xss.

gf redirect < urls.txt | anew redirect-candidates.txt

#Let the chat group the noise

Paste a sample of params.txt into the NotHoly chat and let it cluster the endpoints.

These are URL patterns with FUZZ in each parameter.
Group them by endpoint, name the parameters that look
like a path, a URL, an ID or a redirect, and tell me
which five to test first and why.

The model clusters similar endpoints and points at the parameters worth a closer look. It is reading text, not the site, so treat the output as a reading list you confirm by hand with your own requests.

#Scope and privacy

Archive data can include URLs from hosts outside your scope or from other tenants, so filter to your roots before you crawl anything. Old URLs sometimes carry tokens or emails in the query string. Treat those as sensitive, do not replay them against the live site, and keep them out of shared notes.

#Frequently asked questions

#Why mine archives instead of only crawling?

Archives hold paths that were removed from the site but still work on the server, which a crawler of the current site will never see. The two sources overlap only partly, so you run both and merge with anew.

#How do I avoid hammering the target?

Lower gospider concurrency and depth, crawl one host at a time, and keep archive mining, which never touches the target, as the bulk of your collection. Send live requests only when you have a reason to.

With URLs and parameters mapped, widen the passive picture with certificate transparency and DNS history. Back to subdomain enumeration.