I was looking to take on a larger project to level up my Python, and after recently doing some web scraping I was still in the mood (despite not really being a bug hunter). The idea was to find a larger well-known tool to use as reference material for coding style and structure, and I landed on Sublist3r, because everyone knows this tool (and I want to be cool like that), and after staring at the code for a few hours and testing individual modules I quickly realised it’s actually completely broken…
Okay, so what’s wrong with it? Well, half the modules are broken and don’t work, the ones that do work are lacking or obsolete, and the DNS brute force is a self-DoS. Then I saw in the comments in the issues this cool cat who’s forked Sublist3r and made Sublist3r2.
…bruh, “Sublist3r2”, half the purpose of making a tool is giving it a cool name.
Sublist3r2’s goal was to pick up where Sublist3r stopped, but really it only fixed a handful of bugs. The cool thing though was that it swapped out the brute-force module with aiodnsbrute, which is this sweet fast as f*ck asynchronous DNS brute force tool utilising the asyncio Python library.
I thought maybe this is a good opportunity to make Sublist3r3 and really fix all 130 reported issues, which seemed like a cool project and learning opportunity, but after re-writing one module I thought — this is stupid, may as well re-write the whole thing from the ground up.
…so that’s what I did. TL;DR I present:
The changes & improvements include:
- ⭐NEW MODULES⭐ AlienVault, Anubis, DuckDuckGo, Gist, Hacker Target, RapidDNS, SonarSearch, ThreatMiner, WayBack Machine
- SonarSearch utilises Rapid7’s Sonar through Crobat, but I am 95% sure the dataset is outdated (and that’s a fairly significant task to update).
- VirusTotal module now actually works, and scrapes the results better, and is keyless.
- Ask, Bing, Certificate Search, DNSDumpster all included and are basically unchanged.
- The Google module previously if detecting a captcha would just give up, now it pauses at a prompt to give you a chance to change IPs through a VPN. There’s also a few “dumb” captcha bypasses included (check comments).
- Baidu and NetCraft removed because of anti-scraping tech.
- PassiveDNS also removed and I legitimately have no idea what this module did. It pointed to a https://api.sublist3r.com/ API which is long dead (both Amass and subfinder still include this as a module… why?)
--proxyoption if you want to send it through Burp or a VPS or whatever.- The brute-force module is inspired by aiodnsbrute, it’s fast… and I’d like to say I now understand the asyncio library, but…
--recursivebrute force option (careful, not recommended for large scopes).The code from altdns has been included for a “second phase” brute force, i.e. scrape everything, brute force, combine those results and generate permutations, then brute force again.
--listand--autopilotoptions to input your scope for “set-and-forget”.Windows and Linux compatible
Python 3.10 👌
How does Snrublist3r compare to other similar tools? I could try benchmark against a half dozen other tools, but…
…so, subfinder is probably the closest comparison, here’s both run with the domain hackerone.com
This is not really a fair comparison though, since subfinder doesn’t include brute forcing and has modules that require subscriptions for API keys that I haven’t included, be that as it may it’s still a good result for an off-the-shelf tool with nothing more needed than “click go”.
That’s it, feature requests and improvements are very welcome. The end, but remember every time you git clone Snrublist3r:
Author:
Benjamin McMillan, Senior Security Consultant, Sekuro
Benjamin is a Senior Security Consultant in Sekuro's Offensive Security team. He specialises penetration testing of web applications, mobile apps, Wi-Fi, and Windows domain networks. Benjamin has delivered projects and presented outcomes and findings to key stakeholders, ranging from C-suite executives and application owners to end developers. He holds many certifications, some of which includes OSCP, CISSP, and CRTO.
Related posts:
How You Can Implement Zero Trust Even with Limited Resources
Whether it is a person, device, object, or connection, Zero Trust is the belief that we should not trust them until they prove they can be trusted.
The Patching Time Sink
Sekuro’s Customer CISO, Lee Roebig expounded on the following aspects of Zero Trust, which are fundamental to the security, and hence, the success of an organisation. Organisations
The Sekuro Talkshow with Lee Roebig (Ep.4)
In the fourth episode of the Sekuro Talkshow, Shamane Tan, Sekuro Chief Growth Officer, is joined by Lee Roebig, former Head of Information Security for Fitness and Lifestyle Group, and Sekuro’s current Customer CISO.