+++ title = "urx" description = "Extract URLs from OSINT archives for security research. A fast, keyless Rust CLI." template = "landing.html" +++
OSINT URL Discovery

# Extract every URL a domain ever exposed.

A fast Rust CLI that pulls URLs from OSINT archives in parallel, then filters and validates them for recon.

Get started
View on GitHub
7 Sources
5 Keyless
3 Output formats
urx · recon

    $ urx dalfox.hahwul.com --providers wayback,otx,vt,urlscan --check-status

    Domains         [====================] 1/1 processed
    Wayback Machine [====================] done
    OTX             [====================] done
    VirusTotal      [====================] done
    Urlscan         [====================] done
    Filtering       [====================] 2408 URLs
    Testing URLs    [====================] 2408/2408 complete

    https://dalfox.hahwul.com/                         [200 OK]
    https://dalfox.hahwul.com/.well-known/security.txt [200 OK]
    https://dalfox.hahwul.com/docs/getting-started/    [200 OK]
    https://dalfox.hahwul.com/assets/js/main.min.js    [200 OK]
    https://dalfox.hahwul.com/allposts.html            [404 Not Found]
    ... found 2408 URLs

## Seven archives, one command.

urx queries public web archives and threat-intel feeds at the same time. Five work with no API key at all.

Keyless Wayback Machine --providers wayback The Internet Archive's CDX index. Deep historical coverage, on by default.
Keyless Common Crawl cc
Keyless OTX otx
Keyless Arquivo.pt arquivo
API key VirusTotal vt
Anonymous URLScan urlscan
API key ZoomEye zoomeye

## Built for the whole recon loop.

7 sources

### Parallel collection

Async requests fan out to every enabled provider at once, then merge and deduplicate into a single URL set. No waiting on one slow archive.

### Advanced filtering

Filter by extension, pattern, or presets like `no-images`. Control URL length and host parts.

$ urx target.com \
-e js,php \
--patterns api,v1
https://target.com/api/v1/auth.js
https://target.com/api/config.php
https://target.com/v1/users.js

### Flexible output

Plain text, JSON, or CSV. Stream to the console, a file, or a pipe.

### URL validation

Check HTTP status codes, drop dead links, and extract more URLs from live pages.

### Caching & incremental scanning

Local SQLite or remote Redis caching skips domains you already scanned. Incremental mode returns only the URLs discovered since last run.

### Built in Rust

Async runtime, parallel requests, and a small footprint for maximum throughput.

## From a domain to a clean URL set.

One command runs the full pipeline. Each stage maps to a flag you can tune.

--providers

### Collect

Query every archive in parallel and merge the results into one stream.

-e · --patterns

### Filter

Keep only the extensions, patterns, or presets you care about.

--check-status

### Validate

Probe live URLs, check HTTP status, and pull extra links from responses.

-f json · -o

### Output

Write text, JSON, or CSV to a file, or pipe it straight into your next tool.

Ready to launch

## Install urx and start collecting.

Available on Cargo, Homebrew, and as a container image. No account, no key to get going.

$ cargo install urx Cargo
$ brew install urx Homebrew
$ docker pull ghcr.io/hahwul/urx Docker
Read the docs