Quick answer: SpiderFoot automates OSINT by taking one “seed” (a domain, IP, email, username or name), querying 200+ data sources, and feeding every new finding back in as a new seed. To automate your own OSINT workflow, pick the right scan use case, add API keys, run scans from the command line with sf.py, schedule them with cron or Task Scheduler, and compare the exported CSV or JSON results over time.
Manual OSINT is slow. Checking a single domain by hand can mean visiting WHOIS, DNS tools, certificate logs, breach databases, search engines and threat feeds one by one. SpiderFoot does all of that for you in minutes. This beginner’s guide explains how SpiderFoot automation works and shows you, step by step, how to build a repeatable OSINT workflow in 2026.
What Is OSINT, and Why Automate It?
OSINT (open-source intelligence) is information collected from publicly available sources: websites, public records, DNS, social media, certificate logs, code repositories, breach dumps and more. Security teams use it to see what attackers can see. Journalists and investigators use it to verify facts.
| Manual OSINT | Automated OSINT with SpiderFoot |
|---|---|
| One lookup at a time | 200+ sources queried in parallel |
| Easy to miss connections | Every result automatically becomes a new search |
| Results spread over many tabs | One database, graph and report |
| Hard to repeat exactly | Same scan settings every time |
| Hours per target | Minutes per target |
How SpiderFoot Automation Works
SpiderFoot uses an event-driven, publish/subscribe model. Each module “listens” for certain data types and “publishes” new ones. For example:
- You enter example.com (a domain).
- A DNS module resolves it and publishes an IP address.
- Threat-intel modules see the IP and check blacklists; a geolocation module finds its location.
- A certificate module finds subdomains; each subdomain is resolved and checked again.
- Email and breach modules find addresses and check them for leaks.
This chain reaction is what makes SpiderFoot powerful. Finally, the correlation engine (37 built-in YAML rules) reviews everything and flags notable risks, such as a host listed on a malware blacklist or an email found in a breach. For the full background, read what SpiderFoot is and how it supports OSINT automation.
Step 1: Install SpiderFoot
Use our SpiderFoot install guide for Windows, macOS, Linux or Docker. On Kali Linux it is one command: sudo apt install spiderfoot.
Step 2: Choose the Right Use Case
| Use case | Automates | Touches the target? |
|---|---|---|
| Passive | Third-party lookups only | No |
| Footprint | Perimeter, subdomains, identities, infrastructure | Some modules may |
| Investigate | Malicious indicators, blacklists, reputation | Some modules may |
| All | Everything | Yes |
Beginners should start with Passive. It is fast, quiet and safe.
Step 3: Add API Keys to Unlock More Automation
Many modules work out of the box. Others need a (often free) API key. Add keys under Settings. Priority keys for beginners: VirusTotal, Shodan, SecurityTrails, Hunter.io, AbuseIPDB and AlienVault OTX. See all sources in our SpiderFoot modules and data sources guide.
Step 4: Run Automated Scans From the Command Line
The web UI is great for exploring. The CLI is better for automation because you can script it:
# Passive scan, results saved as CSV
python3 sf.py -s example.com -u passive -o csv -q > example.csv
# Only collect subdomains and email addresses
python3 sf.py -s example.com -t INTERNET_NAME,EMAILADDR -f -o json -q > example.jsonEvery flag is explained in our SpiderFoot commands cheat sheet.
Step 5: Scan Many Targets With a Simple Script
Put one target per line in targets.txt, then loop through them (Linux/macOS):
while read t; do
python3 sf.py -s "$t" -u passive -o csv -q > "results_$t.csv"
done < targets.txtOnly include targets you own or are authorised to assess.
Step 6: Schedule Scans for Continuous Monitoring
Run a weekly passive scan with cron on Linux:
0 3 * * 1 cd /opt/spiderfoot && ./venv/bin/python3 sf.py -s example.com -u passive -o csv -q > /opt/reports/example-$(date +\%F).csvOn Windows, create a task in Task Scheduler that runs python sf.py with the same options. Compare this week’s file with last week’s to spot new subdomains, new IPs, new breaches or new blacklist hits. That is continuous attack-surface monitoring on a budget.
Step 7: Review, Verify and Report
- Start with Correlations in the web UI; they rank the most important findings.
- Verify key findings manually. Automation can produce false positives.
- Export CSV for spreadsheets, JSON for SIEM or scripts, GEXF for graph tools.
- Write a short report: scope, key findings, risk level and recommended fixes.
Beginner Automation Workflows You Can Copy
Workflow 1: Personal privacy audit
Target: your email and usernames. Use case: Passive. Goal: find breaches and forgotten accounts. Action: change reused passwords and enable two-factor authentication.
Workflow 2: Small-business attack surface check
Target: your company domain. Use case: Footprint (with permission). Goal: list every subdomain and exposed service. Action: patch or shut down anything unknown.
Workflow 3: Phishing triage
Target: a suspicious domain or IP. Use case: Investigate. Goal: check reputation, age and blacklists. Action: block it and alert users if malicious.
More ideas: every target type SpiderFoot can analyze.
Limits of OSINT Automation
- False positives happen; always verify before acting.
- API rate limits can slow or stop modules.
- Some sources change or shut down, so a module may stop returning data.
- Automation is not analysis. SpiderFoot collects and links data; you still decide what it means.
Stay Legal and Ethical
Only automate scans against assets you own or have written permission to assess. Respect privacy laws such as GDPR when you store personal data, and delete results you no longer need. Learn more about keeping SpiderFoot data secure.
Frequently Asked Questions
How does SpiderFoot automate OSINT?
SpiderFoot runs 200+ modules that each listen for certain data types and publish new ones. Every new finding, like an IP or subdomain, triggers more lookups automatically, and a correlation engine highlights important results.
Can SpiderFoot run scans on a schedule?
Not by itself in the open-source version, but you can schedule command-line scans with cron on Linux or macOS, or Task Scheduler on Windows, and save each run to a file.
Is SpiderFoot automation suitable for beginners?
Yes. Beginners can start with the web interface and the Passive use case, then move to simple command-line scans and scheduling once they are comfortable.
Can SpiderFoot scan multiple targets at once?
You can run several scans from the web interface, or loop through a list of targets with a simple shell script using the sf.py command line.
What is the SpiderFoot correlation engine?
It is a set of YAML rules, 37 built in with version 4.0, that analyses scan results and flags notable findings by risk level. You can also write your own rules.
Is automated OSINT legal?
Collecting public information is generally legal, but you must follow computer misuse and privacy laws. Scan only assets you own or are authorised to assess, and handle personal data responsibly.
Conclusion
SpiderFoot turns hours of manual OSINT into a few automated minutes. Start with Passive scans, add a handful of free API keys, move to the command line, and schedule regular scans so you catch changes early. If you are just getting started, follow our SpiderFoot beginner tutorial first, then come back and automate.