# Price source prerequisites

Source: https://docs.settlemint.com/docs/deployment/price-source-prerequisites
The network paths, address rules, size caps, secret store, and alerts an operator sets before a price source can fetch from SFTP, FIX, Bloomberg, or LSEG.

A price source is a configured connection to an SFTP server, a FIX session, Bloomberg or LSEG that fetches prices on your behalf, so the deployment must allow that traffic. The scheduler workload, the part of the platform that runs background jobs, opens every connection to a provider. Before the first source goes live, check five things: the address rules, a fixed outbound address for Bloomberg, the paths to each provider, a production secret store, and the stale-feed alert. The size and time caps work at their defaults, and you tune them only for larger deliveries. A source for a public SFTP, FIX, or vendor host needs no address-rule change. A provider on a private network does.

The Bloomberg and LSEG requests are built from each vendor's public documentation and have not run against the live services, so test them with your own vendor account in a non-production deployment before you rely on them.

This page is for the operator who owns the Helm release. For the feeds manager who creates sources through the Platform API, use [Read and submit price feeds](https://docs.settlemint.com/docs/api-reference/guides/feeds). For what a price source is and how its checks work, use the [external price sources release note](https://docs.settlemint.com/docs/changelog/dalp-3-2/market-data).

## Settings you may change [#settings-you-may-change]

Every setting sits in the `priceSources` section of the platform configuration. In the `dalp` chart the section is `global.dalpApp.config.priceSources`. Each value is a placeholder of the form `${VAR:-default}`, so the matching `PRICE_SOURCES_` variable on the pod moves it. A literal value in the chart pins the field, and the variable then has no effect. No tenant API reads or widens these settings.

The API pods and the scheduler pod read the same section. The API pods check the host of a source when you create or edit it. The scheduler pod checks again at every connection and enforces the caps.

```yaml
# Illustrative values: allow a private proxy range and send vendor traffic through it.
global:
  dalpApp:
    config:
      priceSources:
        allowedEndpointCidrs: "10.20.0.0/16"
        forwardProxyUrl: "http://10.20.4.10:3128"
```

## The endpoint allow list [#the-endpoint-allow-list]

By default a price source can reach public hosts only. The setting `priceSources.allowedEndpointCidrs`, set by `PRICE_SOURCES_ALLOWED_ENDPOINT_CIDRS`, lists the address ranges that private providers sit in. It is empty by default. Write ranges in CIDR form, such as `10.20.0.0/16`, separated by commas or spaces. The platform refuses a value that holds anything else.

Without an entry, the platform refuses a host that is, or resolves to, any of these:

| Refused class | Examples | Can the allow list lift it |
| --- | --- | --- |
| Loopback | `127.0.0.0/8`, `::1`, `localhost` and names ending in `.localhost` | Yes for an address. The `localhost` names stay refused |
| Link-local | `169.254.0.0/16`, `fe80::/10` | Yes |
| Private | `10.0.0.0/8`, `172.16.0.0/12`, `192.168.0.0/16`, `100.64.0.0/10`, `fc00::/7`, and other reserved ranges | Yes |
| Unspecified | `0.0.0.0/8`, `::` | Yes |
| Metadata | `169.254.169.254`, `169.254.170.2`, `168.63.129.16`, `100.100.100.200`, and the names `metadata`, `metadata.google.internal`, `metadata.tencentyun.com`, `instance-data` | No. No range lifts it |
| Numeric alias | A host written as a bare number or in a short or hexadecimal form, such as `2130706433` or `0x7f.1` | No |

A range that covers a metadata address does not lift the refusal for that address. If your provider sits in `169.254.0.0/16`, the allow list admits it, and the metadata addresses inside that range stay refused.

The two places that check a host differ:

- The API checks the host as written when a feeds manager creates, edits, or activates a source. It does not resolve a name. A refusal answers 422 with [DALP-14704](https://docs.settlemint.com/docs/api-reference/errors).
- The scheduler resolves the name at every connection and checks every address the name returns. If any address is refused, the connection fails. The scheduler then connects to the address it checked, so a name that changes its answer between the check and the connection cannot reach an internal address. A host that resolves to a refused address is a configuration failure, and the delivery is recorded as `blocked_egress`.

A vendor call follows at most five redirects, and never to another origin than the vendor's own.

If your network uses DNS64, which maps an IPv4 host into `64:ff9b::/96`, add `64:ff9b::/96` to the list so a public IPv4 provider resolves. The IPv4 address embedded in such an address is still checked against the metadata ranges, the allow list, and internal ranges.

### Plaintext FIX [#plaintext-fix]

A FIX source uses TLS or plaintext. The platform accepts plaintext only for a host inside the allow list. A plaintext source that names a host outside it answers 422 with [DALP-14705](https://docs.settlemint.com/docs/api-reference/errors). Because the API checks the host as written, name the counterparty by an IP address that the allow list covers. The scheduler repeats the test on the address the host resolves to.

### Server name check for TLS [#server-name-check-for-tls]

A FIX source with TLS carries a server name. The scheduler connects to the checked address and verifies the counterparty certificate against that server name. A certificate that does not verify is a configuration failure, and the source stops after three refusals in a row. The source has a server name field and no field for a certificate authority.

## Size and time caps [#size-and-time-caps]

Every delivery is read under a cap, so the scheduler rejects an oversized file at the cap. The scheduler reads a delivery into memory, so size the scheduler pod's memory limit before you raise a cap.

| Setting | Environment variable | Default | Allowed range | What it caps |
| --- | --- | --- | --- | --- |
| `maxFileBytes` | `PRICE_SOURCES_MAX_FILE_BYTES` | 26,214,400 bytes | 1 to 1,073,741,824 | One SFTP file, and one Bloomberg or LSEG download |
| `maxRows` | `PRICE_SOURCES_MAX_ROWS` | 250,000 | 1 to 10,000,000 | Records in one delivery, and entries in one SFTP directory |
| `maxInflatedBytes` | `PRICE_SOURCES_MAX_INFLATED_BYTES` | 104,857,600 bytes | 1 to 4,294,967,295 | The size a compressed delivery may inflate to |
| `maxXmlEntities` | `PRICE_SOURCES_MAX_XML_ENTITIES` | 10,000 | 1 to 1,000,000 | Entity references in one ISO 20022 document |
| `maxXmlDepth` | `PRICE_SOURCES_MAX_XML_DEPTH` | 32 | 1 to 256 | Element nesting in one ISO 20022 document |
| `maxFixGroupEntries` | `PRICE_SOURCES_MAX_FIX_GROUP_ENTRIES` | 1,000 | 1 to 100,000 | Entries in one FIX repeating group |
| `maxFixBodyBytes` | `PRICE_SOURCES_MAX_FIX_BODY_BYTES` | 65,536 | 1 to 16,777,216 | The body of one FIX message |
| `evidenceRetentionDays` | `PRICE_SOURCES_EVIDENCE_RETENTION_DAYS` | 365 | 1 to 3,650 | Days the raw delivery is kept as evidence in object storage |
| `indexerLagBoundSeconds` | `PRICE_SOURCES_INDEXER_LAG_BOUND_SECONDS` | 300 | 1 to 86,400 | Seconds the indexer may trail the chain head before the stale-feed sweep stops judging |

The row cap, `maxRows`, counts records in one delivery. A Bloomberg or LSEG delivery counts rows times price fields against it.

What happens when a delivery passes a cap:

- A file over `maxFileBytes` settles as `failed_validation`. The read stops one byte past the cap, so the platform never holds the rest.
- A file or vendor result with more records than `maxRows` settles as `failed_validation`. An SFTP directory with more entries than `maxRows` fails the poll as a configuration failure.
- A Bloomberg or LSEG request whose mapped ISINs times requested price fields pass `maxRows` fails as a configuration failure before the scheduler sends anything.
- A compressed delivery that inflates past `maxInflatedBytes` fails validation.
- A FIX message with a repeating group over `maxFixGroupEntries` gets a session-level reject, and no part of it becomes an observation. A FIX message body over `maxFixBodyBytes` is not read as a message, and the session fails as a data failure that the platform retries.

The scheduler also enforces fixed deadlines that have no setting.

| Deadline | Value |
| --- | --- |
| Total read of a file or body | 30 seconds plus 1 second for each 64 KiB of the cap, rounded up. About 430 seconds at the default `maxFileBytes` |
| Total read of a vendor control call | 46 seconds. These calls are capped at 1 MiB |
| Vendor call, until it answers | 120 seconds |
| Vendor body with no bytes arriving | 120 seconds |
| Opening a socket | 15 seconds. An SFTP session then has 30 seconds to open and 60 seconds to list a directory |
| FIX logon | 30 seconds |
| Connection test | 30 seconds |

A file read, vendor call, or vendor body that passes its deadline fails as transient, and the platform retries with backoff.

## Bloomberg: a fixed egress address or a forward proxy [#bloomberg-a-fixed-egress-address-or-a-forward-proxy]

Bloomberg Data License accepts calls only from addresses you have registered with Bloomberg. The scheduler sends the calls to `api.bloomberg.com` over HTTPS. The address Bloomberg sees must therefore be one that does not change, and you register it in the Bloomberg Enterprise Console.

You have two ways to supply that address:

| Option | What you do | Cost |
| --- | --- | --- |
| Fixed address on the cluster | Route the scheduler pod's outbound traffic through an address that does not change, and register it with Bloomberg | You change the cluster's outbound path for the whole scheduler pod |
| Forward proxy | Set `priceSources.forwardProxyUrl` to a proxy with a fixed address, and register the proxy's address | You run a proxy. Only vendor traffic uses it |

`forwardProxyUrl`, set by `PRICE_SOURCES_FORWARD_PROXY_URL`, is empty by default, which connects directly. A value must be an `http` or `https` URL without credentials. It carries Bloomberg and LSEG traffic only. SFTP and FIX connections never use it. The proxy must forward HTTPS requests to the vendor host.

The proxy host passes the same guard as a provider host. A proxy on a private address, such as an in-cluster service, needs its range in `allowedEndpointCidrs`. The platform also resolves and checks the vendor host from the scheduler before it sends a request through the proxy, so the scheduler needs working name resolution for it.

If Bloomberg refuses the address, the platform shows it in three places:

- The delivery is recorded as `blocked_egress`. Its detail names the address Bloomberg saw, when Bloomberg's answer states it.
- The source reports a configuration failure and retries after one hour. After three refusals in a row the source stops until a feeds manager pauses and activates it.
- A connection test from the scheduler fails with the same reason and carries the address in its result.

Each Bloomberg call carries a signed token with a timestamp. If Bloomberg's clock differs from the scheduler's by more than 10 seconds, Bloomberg refuses the call. The platform then retries after five minutes and reports the skew. Keep the scheduler host clock synchronised.

## Network paths to each provider [#network-paths-to-each-provider]

Only the scheduler workload opens these connections. It is the scheduler role of the `dalp` chart, and it runs one replica by default. If you restrict outbound traffic with a network policy of your own, allow these paths for it.

| Protocol | Outbound path from the scheduler | Notes |
| --- | --- | --- |
| LSEG | HTTPS to `selectapi.datascope.lseg.com` | Direct, or through the forward proxy. The platform never follows a result location on another origin |
| Bloomberg | HTTPS to `api.bloomberg.com` | Direct, or through the forward proxy. See the Bloomberg section |
| FIX | TCP to the counterparty host and port you set on the source | The scheduler starts the session and keeps it open inside the daily hours of the source. TLS with a server name, or plaintext inside the allow list. Never through the proxy |
| SFTP | TCP to the host and port you set on the source | Read only. The platform lists directories and reads files and never writes. Never through the proxy |

Every host name must resolve from the scheduler pod. For SFTP, a feeds manager accepts the server's host key fingerprint through the API before the first poll.

## Secrets [#secrets]

A provider credential is stored through the platform's secret store, and no response, log, or error returns it. A feeds manager sets it with `PUT /api/v2/price-sources/{sourceId}/credential`. The API pods write the credential and the scheduler pod reads it, so both pods must use the same secret store.

You choose one secret store for the deployment. Set `secretManager.provider` to `local`, `aws`, `gcp`, `azure`, or `vault`, and fill the block for the provider you chose. In the `dalp` chart the setting is `global.dalpApp.config.secretManager`. The `local` provider keeps secrets in the platform's own database and fits development. For production, choose the manager your organisation already runs. [Secret-manager providers](https://docs.settlemint.com/docs/deployment/install/values-secrets#application-secret-manager-providers) describes each choice. The platform keeps each source credential in its own entry and deletes the entries when a feeds manager retires the source.

If the scheduler cannot read the store, the source fails the fetch. An unreachable store is a transient failure that retries. A store that refuses the read for a reason a retry cannot fix is a configuration failure.

## Observability [#observability]

Price source logs share the category `market-data.price-source`. Search your log store for that category, then narrow by field.

| Field | Meaning |
| --- | --- |
| `source_id` | The price source the line is about |
| `failure_class` | `transient`, `credential`, `config`, or `data`, on a warning that the adapter stopped delivering and on the error that a source stopped |
| `reason` | The failure text, or the reason a counterparty gave |
| `expires_in_days` and `warning_threshold_days` | A Bloomberg credential nearing expiry. Warnings come at 30, 7, and 1 days |
| `ref_seq_num` | The FIX message number a counterparty rejected |
| `isins` | The ISINs a FIX counterparty cannot serve |
| `chain_id` and `instrument_id` | The chain or mapping the stale-feed sweep could not judge |

The stale-feed signals are:

- The gauge `dalp_market_data_stale_price_feeds` counts mapped feeds past their staleness bound, for each `chain_id`. It has no source label. The operations case of kind `stale-feed` names the source and the cause.
- The gauge `dalp_market_data_price_feed_indexer_lag_seconds` shows how far the indexer trails the chain head. While it passes `indexerLagBoundSeconds`, the sweep makes no judgement on that chain and the first gauge keeps its last value.
- The counter `dalp_market_data_stale_sweep_pass_errors_total` counts sweep passes that could not judge every feed.

The observability chart ships two price source alert rules in its `MarketData` group. `StalePriceFeeds` fires at warning severity when the gauge stays over 0 for 15 minutes. `PriceSourceStaleSweepFailing` fires when 10 or more passes fail within 15 minutes. The rules stay dormant until you enable them, as [Alert rules are a deliberate second step](https://docs.settlemint.com/docs/deployment/install/observability-stack#alert-rules-are-a-deliberate-second-step) explains. [Observability](https://docs.settlemint.com/docs/deployment/observability) describes how the platform exports logs, metrics, and traces to that stack.

## Next steps [#next-steps]

- [Set up a source and read its error codes](https://docs.settlemint.com/docs/api-reference/guides/feeds)
- [Choose a secret-manager provider](https://docs.settlemint.com/docs/deployment/install/values-secrets)
- [Install the observability stack](https://docs.settlemint.com/docs/deployment/install/observability-stack)