No description
- Rust 71.9%
- Nix 28.1%
Squibid
97a3a9d259
this version doesn't use a persistent db, and will re-scrape sites but it does respect the sites robots.txt file \o/ |
||
|---|---|---|
| src | ||
| .gitignore | ||
| Cargo.lock | ||
| Cargo.toml | ||
| flake.lock | ||
| flake.nix | ||
| README.md | ||
Small shitty web crawler writen in rust (🚀🚀🚀). All scraped pages are stored in a redis db where the key is the full url and the value is the page content.