SpiderFoot 3.0: OSINT reconnaissance tool

SpiderFoot is an OSINT automation tool for reconnaissance process, written in Python 3 and GPL-licensed.

Recently, Steve Micallef released on GitHub [1] a new version (3) of SpiderFoot, with a lot of interesting enhancements.

  • Web based UI or CLI
  • Over 170 modules (see below)
  • Python 3
  • CSV/JSON/GEXF export
  • API key export/import
  • SQLite back-end for custom querying
  • Highly configurable
  • Fully documented
  • Visualisations
  • TOR integration for dark web searching
  • Dockerfile for Docker-based deployments
  • Can call other tools like DNSTwist, Whatweb and CMSeeK

According to the release notes [2]:

Out of all the targets SpiderFoot supports, those new in 3.0 since 2.12 are in bold:
– IPv4 addresses
IPv6 addresses
– Subnets
– Hostnames/sub-domains
– Domain names
Phone numbers
E-mail addresses
Usernames
Real names
ASNs

When targeting names and usernames, it’s important to remember to place them in quotes, e.g. "Frank Smith" and "fsmith2000". Phone numbers must be in international format, prefixed with a + followed by the country code, e.g. +15550211221.

SpiderFoot integrates with just about every OSINT data source available, utilises a range of methods for data analysis and making that data easy to navigate using an embedded web-server for providing a clean and intuitive web-based interface:

ModuleDescription
abuse.chCheck if a host/domain, IP or netblock is malicious according to abuse.ch.
AbuseIPDBCheck if a netblock or IP is malicious according to AbuseIPDB.com.
AccountsLook for possible associated accounts on nearly 200 websites like Ebay, Slashdot, reddit, etc.
AdBlock CheckCheck if linked pages would be blocked by AdBlock Plus.
AhmiaSearch Tor ‘Ahmia’ search engine for mentions of the target domain.
AlienVault IP ReputationCheck if an IP or netblock is malicious according to the AlienVault IP Reputation database.
AlienVault OTXObtain information from AlienVault Open Threat Exchange (OTX)
ApilitySearch Apility API for IP address and domain reputation.
Archive.orgIdentifies historic versions of interesting files/pages from the Wayback Machine.
ARINQueries ARIN registry for contact information.
Azure Blob FinderSearch for potential Azure blobs associated with the target and attempt to list their contents.
badips.comCheck if a domain or IP is malicious according to badips.com.
Bambenek C&C ListCheck if a host/domain or IP appears on Bambenek Consulting’s C&C tracker lists.
Base64Identify Base64-encoded strings in any content and URLs, often revealing interesting hidden information.
BGPViewObtain network information from BGPView API.
BinaryEdgeObtain information from BinaryEdge.io’s Internet scanning systems about breaches, vulerabilities, torrents and passive DNS.
BingObtain information from bing to identify sub-domains and links.
Bing (Shared IPs)Search Bing for hosts sharing the same IP.
Binary String ExtractorAttempt to identify strings in binary content.
Bitcoin FinderIdentify bitcoin addresses in scraped webpages.
BlockchainQueries blockchain.info to find the balance of identified bitcoin wallet addresses.
blocklist.deCheck if a netblock or IP is malicious according to blocklist.de.
BotScoutSearches botscout.com’s database of spam-bot IPs and e-mail addresses.
BuiltWithQuery BuiltWith.com’s Domain API for information about your target’s web technology stack, e-mail addresses and more.
CallerNameLookup US phone number location and reputation information.
CensysObtain information from Censys.io
CINS Army ListCheck if a netblock or IP is malicious according to cinsscore.com’s Army List.
CIRCL.LUObtain information from CIRCL.LU’s Passive DNS and Passive SSL databases.
Citadel EngineSearches Leak-Lookup.com’s database of breaches.
Cleanbrowsing.orgCheck if a host would be blocked by Cleanbrowsing.org DNS
CleanTalk Spam ListCheck if an IP is on CleanTalk.org’s spam IP list.
ClearbitCheck for names, addresses, domains and more based on lookups of e-mail addresses on clearbit.com.
CoinBlocker ListsCheck if a host/domain or IP appears on CoinBlocker lists.
CommonCrawlSearches for URLs found through CommonCrawl.org.
ComodoCheck if a host would be blocked by Comodo DNS
Company NamesIdentify company names in any obtained data.
CookiesExtract Cookies from HTTP headers.
Cross-ReferenceIdentify whether other domains are associated (‘Affiliates’) of the target.
Certificate TransparencyGather hostnames from historical certificates in crt.sh.
Custom Threat FeedCheck if a host/domain, netblock, ASN or IP is malicious according to your custom feed.
cybercrime-tracker.netCheck if a host/domain or IP is malicious according to cybercrime-tracker.net.
DarksearchSearch the Darksearch.io Tor search engine for mentions of the target domain.
Digital Ocean Space FinderSearch for potential Digital Ocean Spaces associated with the target and attempt to list their contents.
DNS Brute-forceAttempts to identify hostnames through brute-forcing common names and iterations.
DNS Common SRVAttempts to identify hostnames through common SRV.
DNS Look-asideAttempt to reverse-resolve the IP addresses next to your target to see if they are related.
DNS Raw RecordsRetrieves raw DNS records such as MX, TXT and others.
DNS ResolverResolves Hosts and IP Addresses identified, also extracted from raw content.
DNS Zone TransferAttempts to perform a full DNS zone transfer.
DroneBLQuery the DroneBL database for open relays, open proxies, vulnerable servers, etc.
DuckDuckGoQuery DuckDuckGo’s API for descriptive information about your target.
EmailFormatLook up e-mail addresses on email-format.com.
E-MailIdentify e-mail addresses in any obtained data.
EmailRepSearch EmailRep.io for email address reputation.
ErrorsIdentify common error messages in content like SQL errors, etc.
Ethereum FinderIdentify ethereum addresses in scraped webpages.
File MetadataExtracts meta data from documents and images.
FlickrLook up e-mail addresses on Flickr.
Fortiguard.comCheck if an IP is malicious according to Fortiguard.com.
FraudguardObtain threat information from Fraudguard.io
Fringe ProjectObtain network information from Fringe Project API.
F-Secure Riddler.ioObtain network information from F-Secure Riddler.io API.
FullContactGather domain and e-mail information from fullcontact.com.
GithubIdentify associated public code repositories on Github.
Google MapsIdentifies potential physical addresses and latitude/longitude coordinates.
GoogleObtain information from the Google Custom Search API to identify sub-domains and links.
GravatarRetrieve user information from Gravatar API.
GreynoiseObtain information from Greynoise.io’s Enterprise API.
HackerOne (Unofficial)Check external vulnerability scanning/reporting service h1.nobbd.de to see if the target is listed.
HackerTarget.comSearch HackerTarget.com for hosts sharing the same IP.
HaveIBeenPwnedCheck HaveIBeenPwned.com for hacked e-mail addresses identified in breaches.
Honeypot CheckerQuery the projecthoneypot.org database for entries.
Hosting ProvidersFind out if any IP addresses identified fall within known 3rd party hosting ranges, e.g. Amazon, Azure, etc.
hosts-file.net Malicious HostsCheck if a host/domain is malicious according to hosts-file.net Malicious Hosts.
Hunter.ioCheck for e-mail addresses and names on hunter.io.
Iknowwhatyoudownload.comCheck iknowwhatyoudownload.com for IP addresses that have been using BitTorrent.
InstagramGather information from Instagram profiles.
IntelligenceXObtain information from IntelligenceX about identified IP addresses, domains, e-mail addresses and phone numbers.
Interesting FilesIdentifies potential files of interest, e.g. office documents, zip files.
IPInfo.ioIdentifies the physical location of IP addresses identified using ipinfo.io.
ipstackIdentifies the physical location of IP addresses identified using ipstack.com.
Internet Storm CenterCheck if an IP is malicious according to SANS ISC.
Junk FilesLooks for old/temporary and other similar files.
malwaredomainlist.comCheck if a host/domain, IP or netblock is malicious according to malwaredomainlist.com.
malwaredomains.comCheck if a host/domain is malicious according to malwaredomains.com.
MalwarePatrolSearches malwarepatrol.net’s database of malicious URLs/IPs.
MetaDefenderSearch MetaDefender API for IP address and domain IP reputation.
Mnemonic PassiveDNSObtain Passive DNS information from PassiveDNS.mnemonic.no.
multiproxy.org Open ProxiesCheck if an IP is an open proxy according to multiproxy.org’ open proxy list.
MySpaceGather username and location from MySpace.com profiles.
Name ExtractorAttempt to identify human names in fetched content.
NeutrinoAPISearch NeutrinoAPI for IP address info and check IP reputation.
Norton ConnectSafeCheck if a host would be blocked by Norton ConnectSafe DNS
Nothink.orgCheck if a host/domain, netblock or IP is malicious according to Nothink.org.
numpiLookup USA/Canada phone number location and carrier information from numpi.com.
numverifyLookup phone number location and carrier information from numverify.com.
Onion.linkSearch Tor ‘Onion City’ search engine for mentions of the target domain.
Onionsearchengine.comSearch Tor onionsearchengine.com for mentions of the target domain.
Open Bug BountyCheck external vulnerability scanning/reporting service openbugbounty.org to see if the target is listed.
OpenCorporatesLook up company information from OpenCorporates.
OpenDNSCheck if a host would be blocked by OpenDNS DNS
OpenPhishCheck if a host/domain is malicious according to OpenPhish.com.
OpenStreetMapRetrieves latitude/longitude coordinates for physical addresses from OpenStreetMap API.
Page InfoObtain information about web pages (do they take passwords, do they contain forms, etc.)
PasteBinPasteBin scraping (via Google) to identify related content.
PGP Key Look-upLook up e-mail addresses in PGP public key servers.
PhishTankCheck if a host/domain is malicious according to PhishTank.
Phone NumbersIdentify phone numbers in scraped webpages.
Port Scanner – TCPScans for commonly open TCP ports on Internet-facing systems.
Psbdmp.comCheck psbdmp.cc (PasteBin Dump) for potentially hacked e-mails and domains.
PulsediveObtain information from Pulsedive’s API.
Quad9Check if a host would be blocked by Quad9
RIPEQueries the RIPE registry (includes ARIN data) to identify netblocks and other info.
RiskIQObtain information from RiskIQ’s (formerly PassiveTotal) Passive DNS and Passive SSL databases.
RobtexSearch Robtex.com for hosts sharing the same IP.
Amazon S3 Bucket FinderSearch for potential Amazon S3 buckets associated with the target and attempt to list their contents.
ScyllaGather breach data from Scylla API.
SecurityTrailsObtain Passive DNS and other information from SecurityTrails
SHODANObtain information from SHODAN about identified IP addresses.
Similar DomainsSearch various sources to identify similar looking domain names, for instance squatted domains.
SkymemLook up e-mail addresses on Skymem.
SlideShareGather name and location from SlideShare profiles.
Social Media ProfilesTries to discover the social media profiles for human names identified.
Social NetworksIdentify presence on social media networks such as LinkedIn, Twitter and others.
SORBSQuery the SORBS database for open relays, open proxies, vulnerable servers, etc.
SpamCopQuery various spamcop databases for open relays, open proxies, vulnerable servers, etc.
SpamhausQuery the Spamhaus databases for open relays, open proxies, vulnerable servers, etc.
SpiderSpidering of web-pages to extract content for searching.
SpyOnWebSearch SpyOnWeb for hosts sharing the same IP address, Google Analytics code, or Google Adsense code.
SSL CertificatesGather information about SSL certificates used by the target’s HTTPS sites.
SSL ToolsGather information about SSL certificates from SSLTools.com.
StorageStores scan results into the back-end SpiderFoot database. You will need this.
Command-line outputDumps output to standard out. Used for when a SpiderFoot scan is run via the command-line.
Strange HeadersObtain non-standard HTTP headers returned by web servers.
Talos IntelligenceCheck if a netblock or IP is malicious according to talosintelligence.com.
ThreatCrowdObtain information from ThreatCrowd about identified IP addresses, domains and e-mail addresses.
ThreatExpert.comCheck if a host/domain or IP is malicious according to ThreatExpert.com.
ThreatMinerObtain information from ThreatMiner’s database for passive DNS and threat intelligence.
TLD SearchSearch all Internet TLDs for domains with the same name as the target (this can be very slow.)
Tool – CMSeeKIdentify what Content Management System (CMS) might be used.
Tool – DNSTwistIdentify bit-squatting, typo and other similar domains to the target using a local DNSTwist installation.
Tool – WhatWebIdentify what software is in use on the specified website.
TORCHSearch Tor ‘TORCH’ search engine for mentions of the target domain.
TOR Exit NodesCheck if an IP or netblock appears on the torproject.org exit node list.
TotalHash.comCheck if a host/domain or IP is malicious according to TotalHash.com.
TwitterGather name and location from Twitter profiles.
UCEPROTECTQuery the UCEPROTECT databases for open relays, open proxies, vulnerable servers, etc.
URLScan.ioSearch URLScan.io cache for domain information.
VenmoGather user information from Venmo API.
ViewDNS.infoReverse Whois lookups using ViewDNS.info.
VirusTotalObtain information from VirusTotal about identified IP addresses.
VoIPBL OpenPBX IPsCheck if an IP or netblock is an open PBX according to VoIPBL OpenPBX IPs.
VXVault.netCheck if a domain or IP is malicious according to VXVault.net.
WatchguardCheck if an IP is malicious according to Watchguard’s reputationauthority.org.
Web AnalyticsIdentify web analytics IDs in scraped webpages and DNS TXT records.
Web FrameworkIdentify the usage of popular web frameworks like jQuery, YUI and others.
Web ServerObtain web server banners to identify versions of web servers being used.
WhatCMSCheck web technology using WhatCMS.org API.
WhoisologyReverse Whois lookups using Whoisology.com.
WhoisPerform a WHOIS look-up on domain names and owned netblocks.
WhoxyReverse Whois lookups using Whoxy.com.
Wigle.netQuery wigle.net to identify nearby WiFi access points.
WikileaksSearch Wikileaks for mentions of domain names and e-mail addresses.
Wikipedia EditsIdentify edits to Wikipedia articles made from a given IP address or username.
XForce ExchangeObtain information from IBM X-Force Exchange
Yandex DNSCheck if a host would be blocked by Yandex DNS
Zone-H Defacement CheckCheck if a hostname/domain appears on the zone-h.org ‘special defacements’ RSS feed.

SpiderFoot can be also completely controlled via the command-line:

Previously, SpiderFoot was controlled exclusively through a web interface but it’s now possible to also orchestrate scans through sf.py itself via the command-line. This means you can do things like python3 ./sf.py -m sfp_haveibeenpwned -s [email protected] to query HaveIBeenPwned? for an e-mail address. 


References

  1. https://github.com/smicallef/spiderfoot
  2. SpiderFoot 3.0 Open Source Release

Related posts

  1. Windows Service Accounts enumeration using Powershell
  2. Social Engineering in penetration tests: my point of view and my own custom tool
  3. TLDR #2: Cross-Site Request Forgery
  4. OWASP Amass: in-depth attack surface mapping and asset discovery
  5. Pockint: a portable OSINT Swiss Army Knife