XRay: a great network OSINT gathering tool

Really useful in the first phases of a penetration test


XRay is a tool for network OSINT gathering developed by Simone Margaritelli, useful to make initial tasks of information gathering and network mapping.

It make a bruteforce of subdomains using a wordlist and DNS requests, and for every subdomain/ip found retrieves from Shodan the open ports and from ViewDNS some historical data.
Finally, for every unique ip address, and for every open port, it launch specific banner grabbers and info collectors.

Grabbers and Collectors

  • HTTP Server, X-Powered-By and Location headers.
  • HTTP and HTTPS robots.txt disallowed entries.
  • HTTPS certificates chain.
  • HTML title tag.
  • DNS version.bind. and hostname.bind. records.
  • MySQL, SMTP, FTP, SSH, POP and IRC banners.

Anonymity and Legal Issues

The software will rely on your main DNS resolver in order to enumerate subdomains, also, several connections might be directly established from your host to the computers of the network you’re scanning in order to grab banners from open ports. Technically, you’re just connecting to public addresses with open ports (and there’s no port scanning involved, as such information is grabbed indirectly using Shodan API), but you know, someone might not like such behaviour.

If I were you, I’d find a way to proxify the whole process … #justsaying


Usage

Usage: xray -shodan-key YOUR_SHODAN_API_KEY -domain TARGET_DOMAIN
Options:
-address string
IP address to bind the web ui server to. (default "127.0.0.1")
-consumers int
Number of concurrent consumers to use for subdomain enumeration. (default 16)
-domain string
Base domain to start enumeration from.
-port int
TCP port to bind the web ui server to. (default 8080)
-preserve-domain
Do not remove subdomain from the provided domain name.
-session string
Session file name. (default "<domain-name>-xray-session.json")
-shodan-key string
Shodan API key.
-viewdns-key string
ViewDNS API key.
-wordlist string
Wordlist file to use for enumeration. (default "wordlists/default.lst")

More information and downloads

https://github.com/evilsocket/xray
https://github.com/evilsocket/xray

Google Dorks, a brief list of resources

Google hacking for fun and profit

In 2002, Johnny Long began to collect interesting Google search queries that uncovers vulnerable systems or sensitive information, and calls them “Google dorks”.

We identify with “Google Dorking” the method for finding vulnerable targets using the google dorks in order to obtain usernames and passwords, email lists, sensitive documents and website vulnerabilities.

Disclaimer!

Ethical hackers can use the “Google Dorking” to improve system security, but also a black hat could use this technique for illegal activities, including cyber terrorism, industrial espionage, and identity theft.


A brief list

The original Johnny Long’s website

Some dorks focused on SQLi vulnerabilities

Another list on Exploit-DB

SCADA and IoT

Some articles concerning Google Hacking

Some list found on github

Some list found on github

https://gist.github.com/stevenswafford/393c6ec7b5375d5e8cdc

 

Other resources? Suggestions are welcome!

Vulnerabilities and Exploits, my own list of OSINT resources

Website and mailing lists: any other suggestions would be very welcome.


Today i’m glad to share a list of OSINT sources focused on Exploits and Vulnerabilities search. Enjoy!

CVEdetails

“The ultimate security vulnerability datasource”

http://www.cvedetails.com/


CVE.mitre

Common Vulnerabilities and Exposures is a dictionary of common names for publicly known cybersecurity vulnerabilities. CVE’s common identifiers make it easier to share data across separate network security databases and tools, and provide a baseline for evaluating the coverage of an organization’s security tools.

http://www.cvedetails.com/


Full disclosure

A public, vendor-neutral forum for detailed discussion of vulnerabilities and exploitation techniques, as well as tools, papers, news, and events of interest to the community.

http://www.cvedetails.com/


See bug

Open vulnerability platform based on vulnerability and PoC/Exp sharing communities. So far, it already has 50,000+ vulnerabilities and 40,000+ PoC/Exps.

https://www.seebug.org/


CXSecurity

Free vulnerability database

http://www.cvedetails.com/


Inj3ct0r

“ Inj3ct0r is the ultimate database of exploits and vulnerabilities and a great resource for vulnerability researchers and security professionals. 
Our aim is to collect exploits from submittals and various mailing lists and concentrate them in one, easy-to-navigate database.”

http://www.cvedetails.com/


Packet Storm

Packet Storm provides around-the-clock information and tools in order to help mitigate both personal data and fiscal loss on a global scale. As new information surfaces, Packet Storm releases everything immediately through it’s RSS feeds, Twitter, and Facebook.

http://www.cvedetails.com/


Exploit-db

Archive of Exploits, Shellcode, and Security Papers.

http://www.cvedetails.com/


Vulnerability-lab

The Vulnerability Laboratory helps with the world’s first independent bug bounty hacker community. Leverage their skills and creativity to surface your critical vulnerabilities before criminals can exploit them.

http://www.cvedetails.com/


Vulndb

Yearly archive of all vulnerabilities documented in the database.

https://vuldb.com/


Vulners

Vulners.com team is the group of security experts, ethical hackers and researchers who would like to bring usability of information security content to the new level.

http://www.cvedetails.com/


Rapid7 DB

Archive of Metasploit modules for publicly known exploits, 0days, remote exploits, shellcode, and more for researches and penetration testers to review.

http://www.cvedetails.com/


NIST

National Vulnerability Database

http://www.cvedetails.com/


Security focus

“Since its inception in 1999, SecurityFocus has been a mainstay in the security community. From original news content to detailed technical papers and guest columnists, we’ve strived to be the community’s source for all things security related. SecurityFocus was formed with the idea that community needed a place to come together and share its collected wisdom and knowledge.”

http://www.cvedetails.com/


Openwall mailing list

Open Source and information security mailing list archives

http://www.cvedetails.com/


SecLists.Org Security Mailing List

“Any hacker will tell you that the latest news and exploits are not found on any web site — not even Insecure.Org.”

http://www.cvedetails.com/


Debian Security Announcements Mailing list

Security advisories about Debian packages

http://www.cvedetails.com/

OSINT Tools for Fact Checking

My own list


Another (brief) list of OSINT sources, this time focused on fact checking services.


Fact-Check

Investigate (an issue) in order to verify the facts.

‘I didn’t fact-check the assertions in the editorial’

‘reporters can’t be expected to fact-check every quotation’

Source


Here the list

  • About Urban Legends
     Debunk urban legends, fake news sites, and internet hoaxes
  • Checkdesk
    Check brings newsrooms and citizen journalists together to verify real time news anywhere in the world.
  • Citizen Desk
    Tool to help aggregate, verify and publish news reports from citizen journalists.
  • Emergent
    Emergent is a real-time rumor tracker. It’s part of a research project with the Tow Center for Digital Journalism at Columbia University that focuses on how unverified information and rumor are reported in the media. It aims to develop best practices for debunking misinformation.
  • Fact Check
    A project of the Annenberg Public Policy Center of the University of Pennsylvania
  • Full Fact
    UK’s independent factchecking charity
  • MediaBugs
    A service for correcting errors and problems in media coverage.
  • Snopes
    The oldest and largest fact-checking site on the Internet, one widely regarded by journalists, folklorists, and laypersons alike as one of the world’s essential resources.
  • Verification Junkie
    A growing directory of tools for verifying, fact checking and assessing the validity of eyewitness reports and user generated content online.
  • Verily
    Experimental web application designed to rapidly crowdsource the verification of information during humanitarian disasters.

Open Source Intelligence tools for social media: my own list

A constantly updated list of OSINT Sources

 

I continue the publication of my lists of OSINT sources, this time with a list focused on social networks public data.


Twitter


Facebook


Instagram

  • Custom Instagram Search Tools
    Wrapper page to instagram search
  • Ink361
    Instagram insights for marketers, analysts and brand leaders
  • Picodash
    Search engine to explore Instagram photos, videos, users and locations
  • Tofo.me
    Instagram Online Viewer
  • Websta
    Instagram analytics
  • Worldcam
    Find the latest Instagram photos from around the world

Pinterest

  • Pingroupie
    Find Group Boards on Pinterest with ease

Reddit


Tumblr


LinkedIn

  • FTL
    Chrome extension for finding emails of people’s profiles in linkedin

My personal list of OSINT sources: search tools

All the informations are online, you just need to know how to find them

In a previous post, we discovered the real power of OSINT sources, now let’s start to see some helpful links from my personal list.

Today the focus is on the searching tools


General purposes search engines


Specialty search engines

  • 2lingual Search
    Get Google Search Results alongside Google Cross Language Search Results.
  • Biznar
    Federated search engines that search multiple databases in real-time.
  • CiteSeerX
    Search engine focused computer literature and information science.
  • FindTheCompany
    A corporate intelligence site that uses Graphiq’s semantic technology to deliver deep insights via data-driven articles, visualizations and research tools.
  • Harmari
    Ads search engine
  • Boardreader
    Forums and Discussion Boards Search
  • Internet Archive
    Nnon-profit library of millions of free books, movies, software, music, websites
  • WorldWideScience.org
    Search on national and international scientific databases and portals
  • Zanran
    Multiple data search engine
  • Clarify
    Access to data hidden in online audio and video.

Similar sites search


Clustering search engines

  • Carrot2
    Organizes search results into topics
  • Cluuz
    Extracts important terms and images, clusters them and gives them in semantic graph and in a tag cloud
  • Yippy

Metasearch engines (aggregators)

  • Dogpile
    Combined results from a Google and Yahoo
  • Etools
    Combined results from 18 search engines.
  • Goofram
    Combined results from a Google and Wolfram Alpha
  • iZito
    Combined results from Yahoo, Microsoft Bing, YouTube, Wikipedia, Entireweb
  • Zapmeta
    Really similar to iZito
  • Metabear
    Russian and FTP search
  • Qrobe
    Combined results from multiple search engines
  • Qwant
    Anonymous metasearch engine
  • WebCrawler
    Combined results from a Google and Yahoo

Code Search

  • NerdyData
    Search for a specific piece of code into website sourcecode
  • SearchCode
    Search code snippets on major sourcecode repositories
  • Krugle
    Search code snippets on major sourcecode repositories

Image Search

Automate IP and URL analysis with Automater

Automater is a tool created to automate the OSINT analysis of IP addresses.


Given a target (URL, IP, or HASH) or a file full of targets Automater will return relevant results from sources like the following: IPvoid.com, Robtex.com, Fortiguard.com, unshorten.me, Urlvoid.com, Labs.alienvault.com, ThreatExpert.com, VxVault.net, and VirusTotal.com.

There are many output methods: –o will output to a file in the same format that is printed to screen, -c will output a csv, and –w will output an html file.


OSINT?

Open-Source INTelligence (OSINT) is intelligence collected from publicly available sources.
In the intelligence community (IC), the term “open” refers to overt, publicly available sources (as opposed to covert or clandestine sources); it is not related to open-source software or public intelligence.

(from Wikipedia)


Installation

Some python libraries are required: httplib2, requests, sys, argparse, urllib, urllib2

When the dependences are satisfied, simply use git to clone the code to your local machine:

git clone https://github.com/1aN0rmus/TekDefense-Automater.git

…or download the script from https://github.com/1aN0rmus/TekDefense-Automater/archive/master.zip


Usage

./Automater.py -h
usage: Automater.py [-h] [-o OUTPUT] [-w WEB] [-c CSV] [-d DELAY] [-s SOURCE] [ — p] target
IP, URL, and Hash Passive Analysis tool
required arguments:
target List one IP Addresses, URL or Hash to query or pass
the filename of a file containing IP Addresses, URL or
Hash to query each separated by a newline.
optional arguments:
-h, — help show this help message and exit
-o OUTPUT, — output OUTPUT This option will output the results to a file.
-w WEB, — web WEB This option will output the results to an HTML file.
-c CSV, — csv CSV This option will output the results to a CSV file.
-d DELAY, — delay DELAY This will change the delay to the inputted seconds.
Default is 2.
-s SOURCE, — source SOURCE This option will only run the target against a
specific source engine to pull associated domains.
Options are defined in the name attribute of the site
element in the XML configuration file
 — p This option tells the program to post information to
sites that allow posting. By default the program will
NOT post to sites that require a post.

More information on TekDefense website or on GitHub page.