# Performance Tuning

Settings for each layer of a production deployment: Gunicorn, Kvrocks, Valkey/Redis,
PostgreSQL, and optionally PgBouncer.

## Production Reference Architecture

The topology of the CIRCL/GCVE production instances, a good reference for high traffic:

```{mermaid}
flowchart LR
    Client(["Client<br/>HTTP/2 + TLS"])
    HA["HAProxy<br/>TLS termination<br/>L4/L7 load balancing"]
    V["Varnish<br/>HTTP reverse-proxy cache"]
    NGX_D["Nginx<br/>open-data dumps"]
    NGX_S["Nginx<br/>static-asset CDN"]
    VL["Vulnerability-Lookup<br/>Gunicorn + Flask"]
    PGB["PgBouncer<br/>connection pooling"]
    PG[("PostgreSQL")]
    KV[("Kvrocks<br/>primary storage")]
    VAL[("Valkey")]
    DUMPS[("Daily Kvrocks<br/>NDJSON dumps")]
    STATIC[("Flask /static/<br/>CSS · JS · images")]

    Client -->|HTTPS| HA
    HA -->|HTTP/2| V
    HA -->|"HTTP/2<br/>/dumps/"| NGX_D
    HA -->|"HTTP/2<br/>/static"| NGX_S
    V -->|HTTP/1.1| VL
    NGX_D --> DUMPS
    NGX_S --> STATIC
    VL -->|SQL| PGB
    PGB --> PG
    VL -->|RESP| KV
    KV -.->|replication| VAL
```

Compared to the [Nginx-only setup](#reverse-proxy-nginx), it adds HAProxy, Varnish and
two Nginx instances (dumps under `/dumps/`, static assets under `/static`).

### HAProxy (TLS termination and load balancing)

HAProxy is the only component exposed to clients. It:

- terminates TLS and speaks HTTP/2 to clients and to Varnish;
- balances across several Varnish or application instances, with health checks;
- keeps long-lived connections to the backends and absorbs connection bursts;
- can pass the client IP with the PROXY protocol (Gunicorn runs with `--proxy-protocol`).

#### Example configuration

Three backends: the cache (Varnish, the application as backup), the dumps, and the static assets.

```text
frontend web
  mode http
  bind [::]:80 accept-proxy
  bind [::]:443 ssl crt /etc/haproxy/certs/ strict-sni accept-proxy

  # clean header
  http-request del-header X-Forwarded-Proto
  http-request del-header X-Forwarded-For

  # redirect to https
  http-request redirect scheme https code 308 unless { ssl_fc }
  http-request add-header X-Forwarded-Proto https if { ssl_fc } # tell the backend we are using https

  default_backend cache
  acl vulnerability hdr(host) -i vulnerability.circl.lu
  acl vulnerability-dump path_beg /dumps/
  acl vulnerability-cdn path_beg /static/

  use_backend vulnerability-cdn if vulnerability-cdn
  use_backend vulnerability-dump if vulnerability-dump
  use_backend cache if vulnerability

backend cache
  option httpchk
  http-check send meth GET uri /api/system/configInfo hdr Host vulnerability.circl.lu
  option forwardfor
  server varnish localhost:6081 check proto h2
  server vulnerability [2001:db8::1]:10001 check backup

backend vulnerability-dump
  option httpchk
  http-check send meth GET uri / hdr Host vulnerability.circl.lu
  http-request set-path "%[path,regsub(^/dumps/,/)]"
  option forwardfor
  server vulnerability [2001:db8::2]:80 check

backend vulnerability-cdn
  option httpchk
  http-check send meth GET uri / hdr Host vulnerability.circl.lu
  option forwardfor
  server vulnerability [2001:db8::3]:80 check
```

Notes:

- The application server is a `backup` of Varnish: if Varnish fails, traffic still flows, only slower.
- `set-path "%[path,regsub(^/dumps/,/)]"` strips `/dumps/`, so the files sit at the root of that Nginx.
- The health check uses `/api/system/configInfo`.

### Varnish (HTTP reverse-proxy cache)

Varnish serves repeated GET requests from memory, without reaching Gunicorn, Kvrocks or
PostgreSQL. Vulnerability records change rarely and are read often, so even short TTLs
absorb most of the load and traffic spikes.

- Cache stable reads: `/vuln/<id>`, `/api/vulnerability/<id>`, static assets.
- Bypass the cache for POST/PUT/DELETE, authenticated requests and personal pages.
- Prefer `Cache-Control` headers set by the application over VCL rules.

#### Example configuration

A minimal VCL, as on the CIRCL instance. It passes requests with `x-api-key`, strips
every cookie except the session cookie and the SSO flow cookie (without it, SSO sign-ins
fail as expired), gzips text, and caches for 15 minutes with a 2-hour grace.

```vcl
backend default {
    .host = "2001:db8::1";
    .port = "10001";
    .first_byte_timeout = 30s;
    .between_bytes_timeout = 20s;
}

sub vcl_recv {
    # don't cache authenticated api requests
    if (req.http.x-api-key) {
        return (pass);
    }
    # remove cookie to force cache for special bootstrap content
    if (req.url ~ "^/bootstrap($|/.*)") {
        unset req.http.cookie;
    }
    # remove unwanted cookie to cache more requests; keep the session cookie and
    # the single sign-on flow cookie (docs/sso.md)
    if (req.http.Cookie) {
        set req.http.Cookie = ";" + req.http.Cookie;
        set req.http.Cookie = regsuball(req.http.Cookie, "; +", ";");
        set req.http.Cookie = regsuball(req.http.Cookie, ";(vulnerability-lookup|__Host-vl_sso_flow|vl_sso_flow)=", "; \1=");
        set req.http.Cookie = regsuball(req.http.Cookie, ";[^ ][^;]*", "");
        set req.http.Cookie = regsuball(req.http.Cookie, "^[; ]+|[; ]+$", "");

        if (req.http.Cookie == "") {
            unset req.http.Cookie;
        }
    }
}

sub vcl_backend_response {
    set beresp.ttl = 15m;
    set beresp.grace = 2h;
    # Allow browser to cache requests longer
    unset beresp.http.Cache-Control;
    set beresp.http.cache-control = "max-age=900";
    # compress text content
    if (beresp.http.content-type ~ "(application/json|text/html|image/svg|text/css|text/javascript)") {
        set beresp.do_gzip = true;
    }
    # increase cache time
    if (!beresp.http.last-modified) {
        set beresp.http.last-modified = now;
    }
}
```

### Nginx (static-file backends)

HAProxy routes `/dumps/` and `/static` to two Nginx instances, so file traffic never
reaches Varnish or Gunicorn.

- **`/dumps/`**: the daily NDJSON exports written by `bin/dump.py` (see
  [Command-Line Interface](command-line-interface.md)). Served from disk with `sendfile`
  and `autoindex on`. The `location /dumps/` block is in
  [Reverse Proxy (Nginx)](#reverse-proxy-nginx).
- **`/static`**: the application's CSS, JavaScript and images, kept apart from dynamic requests.

### Storage and database backends

- **PgBouncer → PostgreSQL**: users, comments, bundles, sightings, watchlists. See
  [PgBouncer](#pgbouncer-optional) and [PostgreSQL](#postgresql).
- **Kvrocks → Valkey**: Kvrocks (TCP `10002`, Redis protocol) replicates to Valkey for a
  hot standby and read capacity. See [Kvrocks](#kvrocks-primary-storage).
- The local cache (Valkey/Redis on a Unix socket) is not drawn. See
  [Valkey/Redis](#valkeyredis-cache-layer).

## Gunicorn (Web Server)

Set in `config/generic.json`. `bin/start_website.py` starts Gunicorn, and `bin/gunicorn_conf.py`
reads the four `website_*` settings below. A setting set to `null` keeps Gunicorn's own default.

- `website_workers`: size against the web tier's memory budget and measured load.
  Each gevent worker handles many clients; increasing workers also multiplies
  database pools and private memory. The sample's 49 workers is not a sizing rule.
- `website_worker_connections`: maximum simultaneous clients per gevent worker,
  including idle keep-alive and SSE clients (default 100).
- `website_max_requests`: recycle a worker after 1,000 requests by default; set to
  0 to disable. `website_max_requests_jitter` adds up to 100 random requests to
  stagger restarts. Recycling contains accumulated memory growth, but cannot
  protect against a single oversized request.
- `website_graceful_timeout`: allow active requests 30 seconds to drain when a
  worker exits, including recycling, and when the website is stopped or restarted:
  `stop` and `restart_website` wait for it before forcing the stop. Long exports and
  SSE connections can be interrupted; streaming clients must reconnect. Increase
  this for your workload.
- Worker silence timeout: 300 seconds. With gevent, this is not a per-request
  deadline while the worker's event loop continues sending heartbeats.

The shared `bin/gunicorn_conf.py` reads these settings for the website launcher
and the separate website Docker image. Existing instance configurations fall back
to sample values for missing keys. Gunicorn CLI options override them. Restart the
website after changing the settings.

When sizing the web tier, reserve memory for Kvrocks, Redis and PostgreSQL if they
share the container. Compare container memory and worker PSS/private memory rather
than summing RSS, which counts preloaded shared pages in each worker. Load-test
large searches and streams as well as typical requests, and monitor latency and
errors when reducing concurrency.

Search score lookups are processed in batches of at most 1,000 commands, without a
transaction. This bounds pipeline queues, not complete searches: candidate lists and
sorting still grow with the number of matches, even for a small response page.

Flags passed to Gunicorn:

```text
--worker-class gevent       # async worker for concurrent I/O
--timeout 300               # worker silence timeout
--reuse-port                # SO_REUSEPORT for load balancing across workers
--proxy-protocol            # preserve client IPs behind a reverse proxy
--preload                   # load the application once, before forking
--forwarded-allow-ips=*     # trust X-Forwarded-* from any proxy
```

## Kvrocks (Primary Storage)

Put your tuning in
`storage/kvrocks.local.conf` (copy `storage/kvrocks.conf` first). It is gitignored and
used instead of `kvrocks.conf` when present. Do not edit `kvrocks.conf` itself:
`git pull` overwrites it.

```ini
workers 8                   # I/O threads: about the number of CPU cores
maxclients 10000            # keep below the OS file descriptor limit (ulimit -n)
tcp-backlog 511             # keep net.core.somaxconn >= this value
```

RocksDB:

```ini
rocksdb.block_cache_size 4096           # MB; 25-50% of RAM on a dedicated server
rocksdb.max_open_files 8096             # -1 keeps all files open
rocksdb.write_buffer_size 64            # MB; raise for bulk imports
rocksdb.max_write_buffer_number 4
rocksdb.max_background_jobs 4           # raise on many cores for faster compaction
rocksdb.target_file_size_base 128       # MB
rocksdb.max_total_wal_size 512          # MB
```

Disk I/O:

```ini
max-io-mb 0                 # set a limit if compaction slows down requests
max-replication-mb 0
```

## Valkey/Redis (Cache Layer)

Configured in `cache/cache.conf`, on a Unix socket only:

```ini
port 0                      # TCP disabled — Unix socket only
unixsocket cache.sock
unixsocketperm 700
tcp-keepalive 300
```

Only memory needs watching: `redis-cli -s cache/cache.sock INFO memory`. Set `maxmemory`
and an eviction policy (e.g. `allkeys-lru`) to bound it.

## PostgreSQL

```ini
# Connection / memory
max_connections = 200                     # keep moderate when using PgBouncer
shared_buffers = 64GB                     # ~25% of RAM
work_mem = 64MB                           # per query memory; adjust if heavy queries
maintenance_work_mem = 4GB

# WAL / checkpoints
wal_buffers = 16MB
checkpoint_completion_target = 0.9
max_wal_size = 4GB
min_wal_size = 1GB

# Autovacuum
autovacuum = on
autovacuum_max_workers = 10
autovacuum_naptime = 10s
autovacuum_vacuum_cost_limit = 2000

# Logging
log_connections = on
log_disconnections = on
log_min_duration_statement = 2000         # log slow queries > 2s
```

## PgBouncer (Optional)

Recommended with many Gunicorn workers: it pools connections to PostgreSQL.

```ini
[databases]
vulnlookup = host=127.0.0.1 port=5432 dbname=vulnlookup

[pgbouncer]
listen_addr = 0.0.0.0
listen_port = 6432
auth_type = md5
auth_file = /etc/pgbouncer/userlist.txt
admin_users = vuln-lookup

pool_mode = transaction                     # best for web apps
default_pool_size = 200                     # number of server connections per DB
min_pool_size = 50
reserve_pool_size = 50
reserve_pool_timeout = 5.0                  # seconds to wait for reserve
max_client_conn = 5000

# Logging / stats
logfile = /var/log/pgbouncer/pgbouncer.log
log_connections = 1
log_disconnections = 1
```

- Point `DB_CONFIG_DICT` in `config/website.py` to PgBouncer (6432), not PostgreSQL (5432).
- Here `default_pool_size` equals `max_connections` (200). Keep a few connections free
  for admin access.

## SQLAlchemy Connection Pool

`SQLALCHEMY_ENGINE_OPTIONS` in `config/website.py`.

**Without PgBouncer** (default):

```python
SQLALCHEMY_ENGINE_OPTIONS = {
    "pool_size": 100,               # persistent connections in the pool
    "max_overflow": 50,             # extra connections during traffic spikes
    "pool_timeout": 30,             # seconds to wait for a connection
    "pool_recycle": 1800,           # recycle connections every 30 minutes
    "pool_pre_ping": True,          # verify connection liveness before use
}
```

:::{warning}
Each worker has its own pool: up to `pool_size × website_workers` connections
(100 × 49 = 4,900). Without PgBouncer, keep that well below `max_connections`.
:::

**With PgBouncer** (let PgBouncer manage the pool):

```python
SQLALCHEMY_ENGINE_OPTIONS = {
    "pool_size": 0,                 # no persistent connections; rely on PgBouncer
    "max_overflow": 0,
    "pool_pre_ping": True,
    "pool_timeout": 30,
    "pool_recycle": 3600,
}
```

## Logging

Lower log levels in production:

- `config/generic.json`: set `loglevel` to `WARNING` or `ERROR`
- `config/website.py`: set `LOG_LEVEL` to `WARNING`
- `storage/kvrocks.local.conf`: `log-level warning` (default)
- `config/modules.cfg`: `level = WARNING` per feeder

`config/logging.json` rotates files (1 MB, 5 backups).

## Operating System Tuning

### File Descriptors

```bash
# /etc/security/limits.conf  (or systemd unit override)
* soft nofile 65536
* hard nofile 65536
```

### Network Stack

```bash
sysctl -w net.core.somaxconn=4096
sysctl -w net.ipv4.tcp_max_syn_backlog=4096
sysctl -w net.core.netdev_max_backlog=4096
sysctl -w vm.overcommit_memory=1              # recommended for Redis/Kvrocks
```

### Transparent Huge Pages (THP)

Disable it for Kvrocks and Valkey (latency spikes):

```bash
echo never > /sys/kernel/mm/transparent_hugepage/enabled
```

## Reverse Proxy (Nginx)

A minimal configuration:

```nginx
upstream vulnlookup {
    server 127.0.0.1:10001;
    keepalive 64;
}

server {
    listen 443 ssl http2;
    server_name vulnerability.example.org;

    client_max_body_size 10m;

    location / {
        proxy_pass http://vulnlookup;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

        proxy_connect_timeout 10;
        proxy_read_timeout 300;           # match Gunicorn timeout
        proxy_send_timeout 300;
    }

    # Open-data NDJSON dumps produced daily by ``bin/dump.py``.
    # Served as static files; never proxied to Gunicorn.
    location /dumps/ {
        alias /path/to/vulnerability-lookup/dumps/;   # $VULNERABILITY_LOOKUP_HOME/dumps/

        autoindex on;                     # public directory listing
        autoindex_exact_size off;
        autoindex_localtime on;

        # Serve .ndjson with a useful content type
        types {
            application/x-ndjson  ndjson;
        }
        default_type application/octet-stream;

        sendfile on;                      # zero-copy for large dumps
        tcp_nopush on;
        gzip_static on;                   # serve pre-compressed .ndjson.gz if present

        # Dumps are regenerated daily; allow downstream caches to keep them briefly.
        expires 1h;
        add_header Cache-Control "public";

        # Only GET/HEAD make sense for a static dump tree.
        limit_except GET HEAD { deny all; }
    }
}
```

- Gunicorn runs with `--proxy-protocol`. Enable the PROXY protocol on your proxy, or rely
  on `X-Forwarded-For` and remove the flag from `bin/start_website.py`.
- The `alias` must point to the `dumps/` directory of the Vulnerability-Lookup home,
  readable by Nginx.

## Summary

| Layer | Key Settings | Tune When |
|-------|-------------|-----------|
| Gunicorn | `website_workers`, `--timeout`, `--worker-class gevent` | High concurrent users, slow responses |
| Kvrocks (`kvrocks.local.conf`) | `workers`, `maxclients`, `rocksdb.block_cache_size` | Large dataset, slow queries, high write load |
| Valkey/Redis | `maxmemory`, eviction policy | Cache memory growing unbounded |
| PostgreSQL | `max_connections`, `shared_buffers`, `work_mem` | Slow user/account queries, high concurrency |
| PgBouncer | `pool_mode`, `default_pool_size`, `max_client_conn` | Many workers exhausting PostgreSQL connections |
| SQLAlchemy | `pool_size`, `max_overflow`, `pool_pre_ping` | Connection errors, stale connections |
| OS | `nofile`, `somaxconn`, THP | Connection refused errors, latency spikes |
