SpiderFoot is an OSINT automation tool for reconnaissance process, written in Python 3 and GPL-licensed.



Recently, Steve Micallef released on GitHub [1] a new version (3) of SpiderFoot, with a lot of interesting enhancements.

  • Web based UI or CLI
  • Over 170 modules (see below)
  • Python 3
  • CSV/JSON/GEXF export
  • API key export/import
  • SQLite back-end for custom querying
  • Highly configurable
  • Fully documented
  • Visualisations
  • TOR integration for dark web searching
  • Dockerfile for Docker-based deployments
  • Can call other tools like DNSTwist, Whatweb and CMSeeK

According to the release notes [2]:

Out of all the targets SpiderFoot supports, those new in 3.0 since 2.12 are in bold:
- IPv4 addresses
- IPv6 addresses
- Subnets
- Hostnames/sub-domains
- Domain names
- Phone numbers
- E-mail addresses
- Usernames
- Real names
- ASNs

When targeting names and usernames, it’s important to remember to place them in quotes, e.g. "Frank Smith" and "fsmith2000". Phone numbers must be in international format, prefixed with a + followed by the country code, e.g. +15550211221.

SpiderFoot integrates with just about every OSINT data source available, utilises a range of methods for data analysis and making that data easy to navigate using an embedded web-server for providing a clean and intuitive web-based interface:

Module Description
abuse.ch Check if a host/domain, IP or netblock is malicious according to abuse.ch.
AbuseIPDB Check if a netblock or IP is malicious according to AbuseIPDB.com.
Accounts Look for possible associated accounts on nearly 200 websites like Ebay, Slashdot, reddit, etc.
AdBlock Check Check if linked pages would be blocked by AdBlock Plus.
Ahmia Search Tor 'Ahmia' search engine for mentions of the target domain.
AlienVault IP Reputation Check if an IP or netblock is malicious according to the AlienVault IP Reputation database.
AlienVault OTX Obtain information from AlienVault Open Threat Exchange (OTX)
Apility Search Apility API for IP address and domain reputation.
Archive.org Identifies historic versions of interesting files/pages from the Wayback Machine.
ARIN Queries ARIN registry for contact information.
Azure Blob Finder Search for potential Azure blobs associated with the target and attempt to list their contents.
badips.com Check if a domain or IP is malicious according to badips.com.
Bambenek C&C List Check if a host/domain or IP appears on Bambenek Consulting's C&C tracker lists.
Base64 Identify Base64-encoded strings in any content and URLs, often revealing interesting hidden information.
BGPView Obtain network information from BGPView API.
BinaryEdge Obtain information from BinaryEdge.io's Internet scanning systems about breaches, vulerabilities, torrents and passive DNS.
Bing Obtain information from bing to identify sub-domains and links.
Bing (Shared IPs) Search Bing for hosts sharing the same IP.
Binary String Extractor Attempt to identify strings in binary content.
Bitcoin Finder Identify bitcoin addresses in scraped webpages.
Blockchain Queries blockchain.info to find the balance of identified bitcoin wallet addresses.
blocklist.de Check if a netblock or IP is malicious according to blocklist.de.
BotScout Searches botscout.com's database of spam-bot IPs and e-mail addresses.
BuiltWith Query BuiltWith.com's Domain API for information about your target's web technology stack, e-mail addresses and more.
CallerName Lookup US phone number location and reputation information.
Censys Obtain information from Censys.io
CINS Army List Check if a netblock or IP is malicious according to cinsscore.com's Army List.
CIRCL.LU Obtain information from CIRCL.LU's Passive DNS and Passive SSL databases.
Citadel Engine Searches Leak-Lookup.com's database of breaches.
Cleanbrowsing.org Check if a host would be blocked by Cleanbrowsing.org DNS
CleanTalk Spam List Check if an IP is on CleanTalk.org's spam IP list.
Clearbit Check for names, addresses, domains and more based on lookups of e-mail addresses on clearbit.com.
CoinBlocker Lists Check if a host/domain or IP appears on CoinBlocker lists.
CommonCrawl Searches for URLs found through CommonCrawl.org.
Comodo Check if a host would be blocked by Comodo DNS
Company Names Identify company names in any obtained data.
Cookies Extract Cookies from HTTP headers.
Cross-Reference Identify whether other domains are associated ('Affiliates') of the target.
Certificate Transparency Gather hostnames from historical certificates in crt.sh.
Custom Threat Feed Check if a host/domain, netblock, ASN or IP is malicious according to your custom feed.
cybercrime-tracker.net Check if a host/domain or IP is malicious according to cybercrime-tracker.net.
Darksearch Search the Darksearch.io Tor search engine for mentions of the target domain.
Digital Ocean Space Finder Search for potential Digital Ocean Spaces associated with the target and attempt to list their contents.
DNS Brute-force Attempts to identify hostnames through brute-forcing common names and iterations.
DNS Common SRV Attempts to identify hostnames through common SRV.
DNS Look-aside Attempt to reverse-resolve the IP addresses next to your target to see if they are related.
DNS Raw Records Retrieves raw DNS records such as MX, TXT and others.
DNS Resolver Resolves Hosts and IP Addresses identified, also extracted from raw content.
DNS Zone Transfer Attempts to perform a full DNS zone transfer.
DroneBL Query the DroneBL database for open relays, open proxies, vulnerable servers, etc.
DuckDuckGo Query DuckDuckGo's API for descriptive information about your target.
EmailFormat Look up e-mail addresses on email-format.com.
E-Mail Identify e-mail addresses in any obtained data.
EmailRep Search EmailRep.io for email address reputation.
Errors Identify common error messages in content like SQL errors, etc.
Ethereum Finder Identify ethereum addresses in scraped webpages.
File Metadata Extracts meta data from documents and images.
Flickr Look up e-mail addresses on Flickr.
Fortiguard.com Check if an IP is malicious according to Fortiguard.com.
Fraudguard Obtain threat information from Fraudguard.io
Fringe Project Obtain network information from Fringe Project API.
F-Secure Riddler.io Obtain network information from F-Secure Riddler.io API.
FullContact Gather domain and e-mail information from fullcontact.com.
Github Identify associated public code repositories on Github.
Google Maps Identifies potential physical addresses and latitude/longitude coordinates.
Google Obtain information from the Google Custom Search API to identify sub-domains and links.
Gravatar Retrieve user information from Gravatar API.
Greynoise Obtain information from Greynoise.io's Enterprise API.
HackerOne (Unofficial) Check external vulnerability scanning/reporting service h1.nobbd.de to see if the target is listed.
HackerTarget.com Search HackerTarget.com for hosts sharing the same IP.
HaveIBeenPwned Check HaveIBeenPwned.com for hacked e-mail addresses identified in breaches.
Honeypot Checker Query the projecthoneypot.org database for entries.
Hosting Providers Find out if any IP addresses identified fall within known 3rd party hosting ranges, e.g. Amazon, Azure, etc.
hosts-file.net Malicious Hosts Check if a host/domain is malicious according to hosts-file.net Malicious Hosts.
Hunter.io Check for e-mail addresses and names on hunter.io.
Iknowwhatyoudownload.com Check iknowwhatyoudownload.com for IP addresses that have been using BitTorrent.
Instagram Gather information from Instagram profiles.
IntelligenceX Obtain information from IntelligenceX about identified IP addresses, domains, e-mail addresses and phone numbers.
Interesting Files Identifies potential files of interest, e.g. office documents, zip files.
IPInfo.io Identifies the physical location of IP addresses identified using ipinfo.io.
ipstack Identifies the physical location of IP addresses identified using ipstack.com.
Internet Storm Center Check if an IP is malicious according to SANS ISC.
Junk Files Looks for old/temporary and other similar files.
malwaredomainlist.com Check if a host/domain, IP or netblock is malicious according to malwaredomainlist.com.
malwaredomains.com Check if a host/domain is malicious according to malwaredomains.com.
MalwarePatrol Searches malwarepatrol.net's database of malicious URLs/IPs.
MetaDefender Search MetaDefender API for IP address and domain IP reputation.
Mnemonic PassiveDNS Obtain Passive DNS information from PassiveDNS.mnemonic.no.
multiproxy.org Open Proxies Check if an IP is an open proxy according to multiproxy.org' open proxy list.
MySpace Gather username and location from MySpace.com profiles.
Name Extractor Attempt to identify human names in fetched content.
NeutrinoAPI Search NeutrinoAPI for IP address info and check IP reputation.
Norton ConnectSafe Check if a host would be blocked by Norton ConnectSafe DNS
Nothink.org Check if a host/domain, netblock or IP is malicious according to Nothink.org.
numpi Lookup USA/Canada phone number location and carrier information from numpi.com.
numverify Lookup phone number location and carrier information from numverify.com.
Onion.link Search Tor 'Onion City' search engine for mentions of the target domain.
Onionsearchengine.com Search Tor onionsearchengine.com for mentions of the target domain.
Open Bug Bounty Check external vulnerability scanning/reporting service openbugbounty.org to see if the target is listed.
OpenCorporates Look up company information from OpenCorporates.
OpenDNS Check if a host would be blocked by OpenDNS DNS
OpenPhish Check if a host/domain is malicious according to OpenPhish.com.
OpenStreetMap Retrieves latitude/longitude coordinates for physical addresses from OpenStreetMap API.
Page Info Obtain information about web pages (do they take passwords, do they contain forms, etc.)
PasteBin PasteBin scraping (via Google) to identify related content.
PGP Key Look-up Look up e-mail addresses in PGP public key servers.
PhishTank Check if a host/domain is malicious according to PhishTank.
Phone Numbers Identify phone numbers in scraped webpages.
Port Scanner - TCP Scans for commonly open TCP ports on Internet-facing systems.
Psbdmp.com Check psbdmp.cc (PasteBin Dump) for potentially hacked e-mails and domains.
Pulsedive Obtain information from Pulsedive's API.
Quad9 Check if a host would be blocked by Quad9
RIPE Queries the RIPE registry (includes ARIN data) to identify netblocks and other info.
RiskIQ Obtain information from RiskIQ's (formerly PassiveTotal) Passive DNS and Passive SSL databases.
Robtex Search Robtex.com for hosts sharing the same IP.
Amazon S3 Bucket Finder Search for potential Amazon S3 buckets associated with the target and attempt to list their contents.
Scylla Gather breach data from Scylla API.
SecurityTrails Obtain Passive DNS and other information from SecurityTrails
SHODAN Obtain information from SHODAN about identified IP addresses.
Similar Domains Search various sources to identify similar looking domain names, for instance squatted domains.
Skymem Look up e-mail addresses on Skymem.
SlideShare Gather name and location from SlideShare profiles.
Social Media Profiles Tries to discover the social media profiles for human names identified.
Social Networks Identify presence on social media networks such as LinkedIn, Twitter and others.
SORBS Query the SORBS database for open relays, open proxies, vulnerable servers, etc.
SpamCop Query various spamcop databases for open relays, open proxies, vulnerable servers, etc.
Spamhaus Query the Spamhaus databases for open relays, open proxies, vulnerable servers, etc.
Spider Spidering of web-pages to extract content for searching.
SpyOnWeb Search SpyOnWeb for hosts sharing the same IP address, Google Analytics code, or Google Adsense code.
SSL Certificates Gather information about SSL certificates used by the target's HTTPS sites.
SSL Tools Gather information about SSL certificates from SSLTools.com.
Storage Stores scan results into the back-end SpiderFoot database. You will need this.
Command-line output Dumps output to standard out. Used for when a SpiderFoot scan is run via the command-line.
Strange Headers Obtain non-standard HTTP headers returned by web servers.
Talos Intelligence Check if a netblock or IP is malicious according to talosintelligence.com.
ThreatCrowd Obtain information from ThreatCrowd about identified IP addresses, domains and e-mail addresses.
ThreatExpert.com Check if a host/domain or IP is malicious according to ThreatExpert.com.
ThreatMiner Obtain information from ThreatMiner's database for passive DNS and threat intelligence.
TLD Search Search all Internet TLDs for domains with the same name as the target (this can be very slow.)
Tool - CMSeeK Identify what Content Management System (CMS) might be used.
Tool - DNSTwist Identify bit-squatting, typo and other similar domains to the target using a local DNSTwist installation.
Tool - WhatWeb Identify what software is in use on the specified website.
TORCH Search Tor 'TORCH' search engine for mentions of the target domain.
TOR Exit Nodes Check if an IP or netblock appears on the torproject.org exit node list.
TotalHash.com Check if a host/domain or IP is malicious according to TotalHash.com.
Twitter Gather name and location from Twitter profiles.
UCEPROTECT Query the UCEPROTECT databases for open relays, open proxies, vulnerable servers, etc.
URLScan.io Search URLScan.io cache for domain information.
Venmo Gather user information from Venmo API.
ViewDNS.info Reverse Whois lookups using ViewDNS.info.
VirusTotal Obtain information from VirusTotal about identified IP addresses.
VoIPBL OpenPBX IPs Check if an IP or netblock is an open PBX according to VoIPBL OpenPBX IPs.
VXVault.net Check if a domain or IP is malicious according to VXVault.net.
Watchguard Check if an IP is malicious according to Watchguard's reputationauthority.org.
Web Analytics Identify web analytics IDs in scraped webpages and DNS TXT records.
Web Framework Identify the usage of popular web frameworks like jQuery, YUI and others.
Web Server Obtain web server banners to identify versions of web servers being used.
WhatCMS Check web technology using WhatCMS.org API.
Whoisology Reverse Whois lookups using Whoisology.com.
Whois Perform a WHOIS look-up on domain names and owned netblocks.
Whoxy Reverse Whois lookups using Whoxy.com.
Wigle.net Query wigle.net to identify nearby WiFi access points.
Wikileaks Search Wikileaks for mentions of domain names and e-mail addresses.
Wikipedia Edits Identify edits to Wikipedia articles made from a given IP address or username.
XForce Exchange Obtain information from IBM X-Force Exchange
Yandex DNS Check if a host would be blocked by Yandex DNS
Zone-H Defacement Check Check if a hostname/domain appears on the zone-h.org 'special defacements' RSS feed.

SpiderFoot can be also completely controlled via the command-line:

Previously, SpiderFoot was controlled exclusively through a web interface but it’s now possible to also orchestrate scans through sf.py itself via the command-line. This means you can do things like python3 ./sf.py -m sfp_haveibeenpwned -s support@spiderfoot.net to query HaveIBeenPwned? for an e-mail address. 




References

  1. https://github.com/smicallef/spiderfoot
  2. SpiderFoot 3.0 Open Source Release