# Streaming service

A Streaming service is implemented using a Publish/Subscribe (Pub/Sub) pattern.

Available channels:

- vulnerability
- comment
- bundle
- sighting

Database-level listeners automatically publish new **sightings**, **comments**, and **bundles** to their respective Pub/Sub channels.
The various feeders push incoming vulnerabilities directly to the designated vulnerability channel.

## Configuration

The streaming service can be configured using the `config/stream.json` file. An example configuration is provided below.

```bash
{
    "register_listeners": true,
    "pubsub_bp": true,
    "global_subscription": false,
    "channels": [
        "vulnerability",
        "comment",
        "bundle",
        "sighting"
    ],
    "_notes": {
        "register_listeners": "Register the listeners for comments, bundles, and sightings.",
        "pubsub_bp": "Activate or deactivate the subscription Blueprint for specific clients.",
        "channels": "The list of channels exposed by the Pub/Sub Blueprint.",
        "global_subscription": "Activate or deactivate the global subscription mechanism for server-side processing of messages."
    }
}
```

### Listeners

When database-level listeners are enabled, new comments, bundles, and sightings
are automatically streamed to the Pub/Sub service, with three rules:

- **Only published content.** A comment held for moderation is not streamed
  when it is inserted; it is streamed when it gets published. Bundles and
  sightings have no moderation step and are streamed on insert.
- **Only after the commit.** A message leaves once the transaction that wrote
  the row has committed, never for a write that was rolled back.
- **An explicit projection.** The `payload` of a message is the object as the
  REST API returns it (`GET /api/comment/<uuid>`, `/api/bundle/<uuid>`,
  `/api/sighting/<uuid>`): the author is its public UUID, timestamps are
  RFC 3339 in UTC, and internal columns such as the author's database id or
  the moderation flag never appear.

Every message is a JSON object with the same envelope:

```json
{
  "payload": {
    "uuid": "9c2b2b1e-0b8d-4a44-9a0a-6d1c6b1f2a10",
    "vulnerability_lookup_origin": "1a89b78e-f703-45f3-bb86-59eb712668be",
    "author": "5e0c6a9f-3f22-4a63-8f6e-2d3a8b7c9d01",
    "title": "Patch available",
    "description": "Fixed in 2.4.1.",
    "description_format": "markdown",
    "vulnerability": "CVE-2024-0001",
    "creation_timestamp": "2026-09-11T09:52:22.013928Z",
    "timestamp": "2026-09-11T09:52:22.013928Z",
    "related_vulnerabilities": [],
    "meta": {}
  },
  "instance_uuid": "1a89b78e-f703-45f3-bb86-59eb712668be",
  "timestamp": "2026-09-11T09:52:22.033420Z",
  "data_type": "comment",
  "event": "created",
  "uri": "https://vulnerability.circl.lu/comment/9c2b2b1e-0b8d-4a44-9a0a-6d1c6b1f2a10"
}
```

`data_type` is the channel the message was published on. `event` is
`created` for a row streamed on insert and `published` for a comment that was
moderated after its insert. `uri` is the web page of the object; it is empty
when the row was written outside a web request (by the synchronization with
a remote instance, for instance).

### Internal streaming

When `global_subscription` is set to `True`, an internal subscription mechanism is provided
for the components of the application. The internal streaming service is executed in a dedicated thread.

### HTTP streaming

The subscription endpoint is gated on the global `stream:subscribe`
permission, which the built-in `admin` role holds. The stream is a firehose
of everything written on the instance, meant for the operator's automation,
so the endpoint is deliberately not open to every confirmed account. To let
an automation account read the stream without making it an administrator,
create a global role carrying only `stream:subscribe` and assign it to that
account.

Operators should also be aware that a long-lived SSE connection needs the
reverse proxy configured for it: streaming enabled (`beresp.do_stream` in
Varnish), buffering off, and a first-byte timeout longer than the interval
between events. The endpoint emits an SSE comment line at least every 15
seconds so an idle channel still carries traffic.

Connection to the authenticated HTTP subscribing interface when (`pubsub_bp` is set to `True`):

```bash
$ curl -H "X-API-KEY: <YOUR-TOKEN>" http://127.0.0.1:10001/pubsub/subscribe/comment
```

[FediVuln](https://pypi.org/project/FediVuln) can also be used to subscribe to a channel:

```bash
$ FediVuln-Publish -t comment
```

A status will be pushed with the configured Mastodon account (see the documentation of FediVuln).
