# Command Line Interface

Run every command from the project directory.

## Core

### Services

```bash
$ poetry run start                     # start everything
$ poetry run stop                      # stop everything
$ poetry run run_backend --start       # Valkey and Kvrocks only (--stop to stop)
$ poetry run feeders_manager --start   # all enabled feeders (--stop to stop)
$ poetry run start_website             # the website only
$ poetry run restart_website
```

`feeders_manager --start --reimport` makes every feeder reimport its whole source.

### Run an individual feeder

```bash
$ poetry run nvd_importer
$ poetry run cvelist_importer
$ poetry run github_importer
```

The full list is under `[project.scripts]` in `pyproject.toml`.

### Dumps

```bash
$ poetry run dump --feed nvd                # one source, as JSON
$ poetry run dump --correlations --index    # identifier correlations
$ poetry run euvd_dump                      # CVE ↔ EUVD mapping, as CSV
```

- `--index` adds the file to the download index. `dump --all` also writes the correlations.
- A failing auxiliary export (comments, bundles, sightings, KEV entries, correlations) is
  skipped and logged; the command then exits non-zero and names it.
- Files are replaced atomically: a failed run keeps the previous export.
- Run `poetry run dump --index` after `euvd_dump` if the index does not list it yet.

`dumps/correlations.json` maps each identifier to its directly linked identifiers:

```json
{
  "cve-2024-1234": ["ghsa-abcd-efgh-ijkl", "pysec-2024-1"],
  "ghsa-abcd-efgh-ijkl": ["cve-2024-1234"],
  "pysec-2024-1": ["cve-2024-1234"]
}
```

- Identifiers are lowercased; `GCVE-0-<year>-<n>` is merged into `cve-<year>-<n>`.
- Self-links and identifiers with no link are left out.
- Links are direct, not transitive, and not a claim of equivalence.
- **It is not guaranteed to be symmetric**: it is read from live storage. Add the
  reverse edges yourself if you need them.

Daily example:

```text
0 7 * * *  cd /path/to/vulnerability-lookup && poetry run dump --correlations --index
```

### Bootstrap from Vulnerability-Lookup dumps

With Kvrocks running, import the NDJSON files of a trusted instance into a fresh one:

```bash
$ poetry run import_dump /path/to/dumps
$ poetry run index_vulnerabilities
```

- The file name (without `.ndjson`) becomes the source name.
- Existing documents are kept; `--replace` overwrites them. Re-running is safe and
  never triggers notifications.
- Comments, bundles, sightings and KEV entries are ignored. Unknown documents are
  counted, not fatal.
- Dumps without the `vulnerability-lookup:id` and `vulnerability-lookup:score`
  annotations still import, but their documents look modified at import time.

**Dumps are a one-time bootstrap, not a sync.** Do not schedule dump downloads: an
import never updates existing documents, and full downloads cost the publisher
bandwidth. Enable the feeders afterwards, and use the API's `since=` parameter.

### Receive EUVD numbers from ENISA

EUVD numbers are issued only by ENISA's instance. Any other instance can receive them
with the `enisa_euvd` feeder, which never issues a number itself. It is off by default.

1. If this instance ever ran the former EUVD feeder, remove the numbers it issued first.
   The feeder refuses to start while that feeder's counters remain:

   ```bash
   $ poetry run python tools/purge_euvd_allocations.py          # dry run
   $ poetry run python tools/purge_euvd_allocations.py --yes
   ```

2. In `config/modules.cfg`, set `enabled = true` and the address of ENISA's instance in
   the `[feeder:enisa_euvd]` section:

   ```ini
   [feeder:enisa_euvd]
   script_name = enisa_euvd_importer
   level = INFO
   enabled = true
   url = https://address-of-enisas-instance
   ```

3. In `config/website.py`, uncomment the `euvd` entry of `SOURCES_TO_SHOW`, so EUVD is
   listed under *National & Regional Vulnerability Databases*.
4. Leave GCVE pulling off for GNA 2 (EUVD) in the admin GCVE registry. It would add a
   second, empty EUVD entry under *GCVE Numbering Authorities*.
5. Start it with the other feeders (`poetry run feeders_manager --start`), and after the
   first load run `poetry run index_vulnerabilities --source euvd`, so each record's
   status follows its CVE, and `poetry run euvd --backfill-kev`.

How it works:

- The first pass loads ENISA's export (`euvd.ndjson` and `cve-euvd-mapping.csv` under
  `<url>/dumps/`), streamed. Each line of the export says which CVE its record was
  issued for; the CSV is only used for an export written before the lines did. A load
  that stops resumes where it stopped while the export is the same file, and a line
  that cannot be read is logged and skipped, then served again by the changes feed.
- Every later pass, hourly, asks `<url>/api/euvd/changes` for the records changed after
  a cursor, in ENISA's update order. The cursor is a position in that order, not a date
  or a page number, so a record updated while the feeder pages is served again later
  rather than skipped. After the first load it starts a day before the export's newest
  record, and no later than any record it could not store: a line that could not be
  read, or a record of an older export that the CVE mapping does not list. A record the
  export says has no CVE does not move it.
- On a 429, a 5xx or a lost connection it waits `retry_after` seconds and retries, up to
  `max_retries` times; a 429's `Retry-After` is honoured up to five minutes. Any other
  4xx is not retried: the next pass asks again.
- A record is stored with only its own fields (number, reservation, publication and
  update dates). Its description, scores and products come from this instance's own CVE
  record. A record that arrives before its CVE is stored at once and completed once the
  CVE is here; a record with no CVE is not carried. A withdrawn record stays withdrawn;
  any other status follows ENISA's.
- It refuses to run where `euvd_authoritative` is true, or where `[feeder:euvd]` is
  enabled: an instance either issues EUVD numbers or receives them. The purge tool, in
  turn, refuses to run while this feeder is enabled.
- `poetry run enisa_euvd_importer --reimport` loads the whole export again on the next
  pass.

### Update the documentation

```bash
$ cd docs; poetry run make html
```

The API reference comes from `docs/_static/files/swagger.json`. Regenerate it first
with [`dump_openapi`](#regenerate-the-openapi-specification) when the API changed.

## Web service

Flask commands are prefixed with `poetry run flask --app website.app`.

### Database

```bash
$ poetry run flask --app website.app db_init
$ poetry run flask --app website.app db_backup                    # also run by `poetry run update`
$ poetry run flask --app website.app db_diagram                   # models diagram
$ poetry run flask --app website.app seed-rbac [--reset-defaults] # also run by `poetry run update`
```

`seed-rbac` adds the missing RBAC roles, permissions and policies. `--reset-defaults`
restores the built-in ones, overwriting your changes.

### User management

```bash
$ poetry run flask --app website.app create_user --login <login> --email <email> --password <password>
$ poetry run flask --app website.app create_admin --login <login> --email <email> --password <password>
$ poetry run flask --app website.app user_list
$ poetry run flask --app website.app user_delete --login <login>
$ poetry run flask --app website.app user_get_api_key --login <login>
```

### Data management

```bash
$ poetry run flask --app website.app update_warninglists
$ poetry run flask --app website.app update_gcve_registry
$ poetry run flask --app website.app sync_organizations [--login <login>]
$ poetry run python tools/update_cnw_assigners.py
$ poetry run flask --app website.app import_osi_approved_licenses
$ poetry run flask --app website.app import_languages
```

- `update_warninglists`: MISP warning lists.
- `update_gcve_registry`: local copy of the GCVE registry and the CNA partner
  catalogue. Can run in cron.
- `sync_organizations`: syncs the organizations with the GCVE registry. `--login` is
  the owner, `admin` by default.
- `update_cnw_assigners.py`: the EU CSIRT coordinated CNAs shown on the dashboard,
  from <https://github.com/enisaeu/CNW> (rows whose `CNA` column is `Yes`). Can run in
  cron, or from the admin maintenance page. A failure keeps the previous list.

### Sightings

```bash
$ poetry run flask --app website.app clean_sightings --source <source>
$ poetry run flask --app website.app clean_sightings --source '<pattern>' --regex  # PostgreSQL only
$ poetry run flask --app website.app backfill_sightings
$ poetry run flask --app website.app dedup_sightings [--yes]
$ poetry run flask --app website.app retype_sightings [--yes]
```

- `clean_sightings`: deletes the sightings of a source, or of a matching pattern.
- `backfill_sightings`: creates sightings from existing comments (*seen*), bundles
  (*seen*, one per vulnerability) and KEV entries (*exploited*), with their original
  timestamps. Safe to re-run.
- `dedup_sightings`: removes repeated sightings of the same observation (same
  vulnerability, URL or `Telegram/…` source, type, author and origin), keeping the
  oldest. Label sources like `CISA-KEV` are left alone.
- `retype_sightings`: by default retypes the Exploit-DB sightings pushed as *exploited*
  (source starting with `https://www.exploit-db.com/exploits/`) to
  *published-proof-of-concept*, deleting the duplicates. `--source-prefix`, `--from`
  and `--to` select other sightings. Safe to re-run.

Without `--yes`, `dedup_sightings` and `retype_sightings` are a dry run. Both only touch
sightings created on this instance: `--all-origins` includes synced ones (the next sync
may bring them back), `--author <login>` narrows to one user. Sync never updates a
sighting it already holds, so each instance runs them itself.

Move sightings between instances:

```bash
$ poetry run flask --app website.app export_sightings sightings.ndjson              # source instance
$ poetry run flask --app website.app import_sightings sightings.ndjson --dry-run    # target: preview
$ poetry run flask --app website.app import_sightings sightings.ndjson --author admin
```

- Both stream, so they handle volumes the browser export times out on. `-` is
  stdout/stdin (`export_sightings - | gzip`).
- The format is the one of `dumps/sightings.ndjson`; the import also reads the browser
  export.
- `--local-only` exports only the sightings created on this instance.
- The import keeps uuid, origin and timestamp, and skips sightings already present.
  Unknown authors are replaced by `--author` (`admin` by default). Invalid records are
  counted, not fatal.

### Regenerate the OpenAPI specification

```bash
$ poetry run flask --app website.app dump_openapi           # write docs/_static/files/swagger.json
$ poetry run flask --app website.app dump_openapi --check   # exit non-zero if it is out of date
```

- No running instance is needed. The output is stable, and the instance UUID is
  replaced by an example (`--example-origin` sets another).
- It refuses to write unless `user_accounts`, `cna` and `cpe_enabled` are all on, and
  names the missing ones. `--allow-partial` writes anyway.
- The `OpenAPI` workflow runs `--check` on every pull request, with every module on.
- A pre-commit hook regenerates it when you touch `website/web/api/` or the version in
  `pyproject.toml`. It skips, with a reason, on an instance that cannot produce the
  complete reference. When it regenerates, the commit stops so you can stage the file.
- A relative `--output` is taken from `VULNERABILITYLOOKUP_HOME`. The command prints
  the path used.

### Health and integrity checks

```bash
$ poetry run flask --app website.app index_health
$ poetry run euvd --integrity-scan
```

- `index_health`: reports whether the state indexes still match the corpus. A stale
  index shows no error anywhere else.
- `euvd --integrity-scan`: counts EUVD identifier breaches (several CVEs or none,
  pointer disagreements, published CVEs without EUVD) for `/metrics`. Exits non-zero on
  a breach. Run it daily. See [Metrics](metrics.md).

### Prepared statistics figures

The per-source `/api/stats/*` figures are prepared in advance, not computed per
request. A figure never prepared answers `available: false`.
`/api/stats/vulnerability/status` is always live.

Run these four commands once after installing or upgrading, then in cron:

```text
# m h dom mon dow  command
30 3 * * *  cd /path/to/vulnerability-lookup && poetry run index_vulnerabilities
30 4 * * *  cd /path/to/vulnerability-lookup && poetry run euvd --stats-scan --stats-source cvelistv5
15 * * * *  cd /path/to/vulnerability-lookup && poetry run scorecard --window-only
45 * * * *  cd /path/to/vulnerability-lookup && poetry run kev_scorecard
```

All four are idempotent.

#### Source names

A source (`--source`, `--stats-source`, `?source=`) is the name records are **stored**
under, not the feeder's section in `config/modules.cfg`. For example, `[feeder:cvelist]`
stores `cvelistv5`. List them with:

```bash
$ redis-cli -p 10002 HKEYS last_updates
```

- GNA sources are the `gna-<id>` entries, one per GNA whose records this instance pulls.
  A GNA with neither API nor dump (such as GNA 2, EUVD) never becomes a source.
- The `euvd` source holds EUVD records, which is unrelated to GNA 2. Official EUVD numbers
  are issued only by ENISA's instance, by a component that is not part of
  Vulnerability-Lookup. Any other instance holds `euvd` records only by receiving them
  from ENISA's (see [Receive EUVD numbers from ENISA](#receive-euvd-numbers-from-enisa)).

#### The reindex: counts, rankings and CVSS distributions

```bash
$ poetry run index_vulnerabilities                                  # every source, GNAs included
$ poetry run index_vulnerabilities --source cvelistv5 --source gna-1
```

- Derives the counts per year and status, the top vendors and assigners and the CVSS
  distributions, and runs the full [assigner scorecard](scorecard.md).
- Without `--source`, it covers every source in `last_updates`. Name sources to make
  the nightly run shorter.
- Run it at least once on a new, restored or upgraded instance before trusting any
  figure: a stale index looks normal but is wrong.

#### The stats scan: time to exploit and known-exploited share

```bash
$ poetry run euvd --stats-scan --stats-source cvelistv5
```

- Prepares the time to exploit, the known-exploited share and the per-month catalogue
  listings. They need both PostgreSQL (KEV entries) and Kvrocks.
- One source per run. Add `; poetry run euvd --stats-scan --stats-source gna-<id>` on
  the same cron line for more.
- Run it after the KEV catalogues are imported. It keeps the previous figures, and exits
  non-zero, when no KEV entry is exploited.
- **Always pass `--stats-source`** unless the instance holds EUVD records: it defaults to
  `euvd`.
- `euvd_seed_counters` is the same command.

#### The scorecards

```bash
$ poetry run scorecard                # assigner scorecard, whole history (also run by every reindex)
$ poetry run scorecard --window-only  # only the rolling window: cheap, hourly
$ poetry run kev_scorecard            # KEV catalogue scorecard, seconds
```

See [assigner scorecard](scorecard.md) and [catalogue scorecard](kev-scorecard.md).
Without the hourly `scorecard --window-only`, the pages show what the last reindex
prepared. The wall of shame has its own switch, `WEB_MODULES["wall_of_shame"]` (off by
default).

#### Known-exploited share

`/api/stats/kev?source=<name>` counts the sets `index:<source>:kev[:<catalog>]`:

- **EUVD** maintains its own during ingestion.
- **Other sources** get them from `euvd --stats-scan --stats-source <name>`, rebuilt on
  every run, so retracted entries drop out. Meaningful for CVE-keyed sources
  (`cvelistv5`, `nvd`, …).

The same run fills `stats:<source>:kev_listed:*` (per-period listings, for
`?granularity=year|month`) for every source, EUVD included.

Both are refused, keeping the previous values, when no KEV entry is exploited or when no
exploited CVE is a record of the source. Entries whose origin matches no catalogue count
under `unattributed`.

#### Upgrading from the EUVD-only figures

The figures used to be keyed `euvd:stats:*`; they are now `stats:<source>:*`. Run once
after upgrading:

```bash
$ poetry run euvd --migrate-stats-keys
```

It is idempotent and does nothing on an instance that never held EUVD figures. Without
it, the time-to-exploit snapshot stays `available: false`.

### Background services

```bash
$ poetry run flask --app website.app notify_users   # notifications
$ poetry run flask --app website.app sync           # synchronization
```

See [Notifications](notification.md) and [Synchronization](sync.md).
