GoSpider is a Fast web spider written in Go. It has lot of features to find the subdomains, JS files, AWS details, etc. 


Install go in your environment and run the below command : 

  • go get -u github.com/jaeles-project/gospider


Lock image


Key features 

  • Fast web crawling 
  • Brute force and parse sitemap.xml 
  • Parse robots.txt 
  • Generate and verify link from JavaScript files 
  • Link Finder 
  • Find AWS-S3 from response source 
  • Find subdomains from response source 
  • Get URLs from Wayback Machine, Common Crawl, Virus Total, Alien Vault 
  • Format output easy to Grep 
  • Support Burp input 
  • Crawl multiple sites in parallel 
  • Random mobile/web User-Agent 


Basic Commands: 

Run with single site  

  •  gospider -s "https://google.com/" -o output -c 10 -d 1 

Run with site list  

  •  gospider -S sites.txt -o output -c 10 -d 1 

For More details: https://github.com/jaeles-project/gospider