GoSpider is a Fast web spider written in Go. It has lot of features to find the subdomains, JS files, AWS details, etc.
Installation:
Install go in your environment and run the below command :
- go get -u github.com/jaeles-project/gospider
Demo

Key features
- Fast web crawling
- Brute force and parse sitemap.xml
- Parse robots.txt
- Generate and verify link from JavaScript files
- Link Finder
- Find AWS-S3 from response source
- Find subdomains from response source
- Get URLs from Wayback Machine, Common Crawl, Virus Total, Alien Vault
- Format output easy to Grep
- Support Burp input
- Crawl multiple sites in parallel
- Random mobile/web User-Agent
Usage
Basic Commands:
Run with single site
- gospider -s "https://google.com/" -o output -c 10 -d 1
Run with site list
- gospider -S sites.txt -o output -c 10 -d 1
For More details: https://github.com/jaeles-project/gospider