README.md (5320B)
1 <div align="center"> 2 3 # wrds-dl 4 ### Browse and download WRDS data from your terminal 5 6 [](https://github.com/LouLouLibs/wrds-download/actions/workflows/ci.yml) 7 [](https://github.com/LouLouLibs/wrds-download/actions/workflows/ci-python.yml) 8 [](go/) 9 [](python/) 10 [](LICENSE) 11 12 <img src="./demo-wrds-download.gif" width="800" alt="Demo of the TUI browsing WRDS schemas and downloading data"> 13 14 </div> 15 16 --- 17 18 ## Features 19 20 - **Interactive TUI** — browse schemas, tables, and column metadata with keyboard navigation 21 - **CLI download** — script-friendly `download` and `info` commands for automation and HPC 22 - **Parquet & CSV** — output to compressed Parquet (ZSTD) or CSV 23 - **Dry run** — preview queries, row counts, and sample rows before downloading 24 - **Raw SQL** — run arbitrary queries when schema/table syntax isn't enough 25 - **Streaming** — server-side cursors keep memory usage low on large tables 26 - **Saved credentials** — authenticate once, connect instantly 27 28 ## Two implementations 29 30 Same CLI interface — pick whichever fits your environment: 31 32 | | [Go](go/) | [Python](python/) | 33 |---|---|---| 34 | **Install** | Pre-built binary (~19 MB) | `uv tool install` / `uv run` | 35 | **TUI browser** | Yes | No | 36 | **CLI commands** | `download`, `info` | `download`, `info` | 37 | **Dependencies** | None (static binary) | Python 3.10+, `uv` | 38 | **Best for** | Interactive exploration, offline use | HPC clusters, CI, quick installs | 39 40 > **Full documentation:** [**Go version**](go/README.md) (TUI + CLI, installation, keybindings, all flags) | [**Python version**](python/README.md) (CLI, uv setup, development) 41 42 ## Quick start 43 44 ### Go 45 46 ```sh 47 # Download binary (macOS Apple Silicon example) 48 curl -L https://github.com/louloulibs/wrds-download/releases/latest/download/wrds-dl-darwin-arm64 \ 49 -o /usr/local/bin/wrds-dl 50 chmod +x /usr/local/bin/wrds-dl 51 ``` 52 53 ### Python 54 55 ```sh 56 # Install as a uv tool 57 uv tool install wrds-dl --from ./python 58 59 # Or run directly 60 cd python && uv run wrds-dl --help 61 ``` 62 63 ## Usage 64 65 ```sh 66 # CRSP monthly stock file — prices and returns for 2020 67 wrds-dl download --schema crsp --table msf \ 68 --columns "permno,date,prc,ret,shrout" \ 69 --where "date >= '2020-01-01' AND date < '2021-01-01'" \ 70 --out crsp_msf_2020.parquet 71 72 # Inspect table metadata before downloading 73 wrds-dl info --schema crsp --table msf 74 75 # Dry run — preview query, row count, and sample rows 76 wrds-dl download --schema crsp --table msf \ 77 --where "date = '2020-01-31'" --dry-run 78 79 # Compustat fundamentals 80 wrds-dl download --schema comp --table funda \ 81 --columns "gvkey,datadate,sale,at" \ 82 --out funda_subset.parquet 83 84 # Raw SQL 85 wrds-dl download \ 86 --query "SELECT permno, date, prc FROM crsp.msf WHERE date >= '2015-01-01'" \ 87 --out crsp_msf_2015_onwards.parquet 88 89 # CSV output 90 wrds-dl download --schema crsp --table msf \ 91 --columns "permno,date,ret" --limit 1000 \ 92 --out crsp_msf_sample.csv 93 ``` 94 95 ## Authentication 96 97 WRDS uses Duo two-factor authentication. Configure credentials before using the CLI. 98 99 ### Option 1: Environment variables 100 101 ```sh 102 export PGUSER=your_username 103 export PGPASSWORD=your_password 104 ``` 105 106 Optional (defaults shown): 107 108 ```sh 109 export PGHOST=wrds-pgdata.wharton.upenn.edu 110 export PGPORT=9737 111 export PGDATABASE=wrds 112 ``` 113 114 ### Option 2: Saved credentials 115 116 Store at `~/.config/wrds-dl/credentials` (or `$XDG_CONFIG_HOME/wrds-dl/credentials`) with `0600` permissions: 117 118 ``` 119 PGUSER=your_username 120 PGPASSWORD=your_password 121 PGDATABASE=wrds 122 ``` 123 124 The Go TUI saves these automatically on first login. 125 126 ### Option 3: ~/.pgpass 127 128 Standard PostgreSQL password file: 129 130 ``` 131 wrds-pgdata.wharton.upenn.edu:9737:*:your_username:your_password 132 ``` 133 134 ## Claude Code skill 135 136 Bundled [Claude Code](https://claude.com/claude-code) skills let you download WRDS data using natural language: 137 138 ``` 139 /wrds-download CRSP daily stock data for 2020 140 ``` 141 142 Two variants available: 143 - [`claude-skill-wrds-download/`](claude-skill-wrds-download/) — uses the Go binary 144 - [`claude-skill-wrds-download-py/`](claude-skill-wrds-download-py/) — uses the Python CLI (no binary needed) 145 146 Install by copying the skill into your skills directory: 147 148 ```sh 149 cp -r claude-skill-wrds-download-py ~/.claude/skills/wrds-download 150 ``` 151 152 ## Project structure 153 154 ``` 155 wrds-download/ 156 ├── go/ # Go implementation (TUI + CLI) 157 │ ├── main.go 158 │ ├── cmd/ # download, info, tui commands 159 │ └── internal/ # db, export, tui, config modules 160 ├── python/ # Python implementation (CLI only) 161 │ ├── pyproject.toml 162 │ └── src/wrds_dl/ # cli, db, export, config modules 163 ├── claude-skill-wrds-download/ # Claude skill (Go binary) 164 ├── claude-skill-wrds-download-py/ # Claude skill (Python/uv) 165 └── .github/workflows/ # CI for both implementations 166 ``` 167 168 ## License 169 170 MIT