Tool of the day

GoSpider is a Fast web spider written in Go. It has lot of features to find the subdomains, JS files, AWS details, etc. 

Installation:   

Install go in your environment and run the below command : 

• go get -u github.com/jaeles-project/gospider

• Fast web crawling 
• Brute force and parse sitemap.xml 
• Parse robots.txt 
• Generate and verify link from JavaScript files 
• Link Finder 
• Find AWS-S3 from response source 
• Find subdomains from response source 
• Get URLs from Wayback Machine, Common Crawl, Virus Total, Alien Vault 
• Format output easy to Grep 
• Support Burp input 
• Crawl multiple sites in parallel 
• Random mobile/web User-Agent 

Usage 

Basic Commands: 

Run with single site  

•  gospider -s "https://google.com/" -o output -c 10 -d 1 

Run with site list  

•  gospider -S sites.txt -o output -c 10 -d 1 

For More details: https://github.com/jaeles-project/gospider