Introduction

Request the data you need by POSTing to /stream and you shall receive a stream of newline-separated JSON objects that package your data by block. For example

curl https://portal.sqd.dev/datasets/solana-mainnet/stream \
  -X POST \
  -d '{
    "type": "solana",
    "fromBlock": 325000000,
    "toBlock": 325000001,
    "fields": {
      "block": {"number": true},
      "instruction": {"data": true}
    },
    "instructions": [
      {"programId": [
        "MoonCVVNZFSYkqNXP6bxHLPL6QQJiMagDL3qcqUQTrG"
      ]}
    ]
  }'

outputs

{"header":{"number":325000000},"instructions":[{"data":"XJqfG9ATWCDLLmxbNnKxcQ6KRHveSXFyrM8JwmyievueP"}]}
{"header":{"number":325000001}}

Here, we requested all records on instructions executed against the MoonCVV...Trg program, with a bare minimum of fields. One such record was found for block 325000000. There were no records of such instructions being executed in block 325000001, but the block was included anyway to indicate the range boundary.

By using this general approach, you can

• Get data on instructions, transactions, logs, balance updates and rewards.
• Ensure that you only get the data you need by using a rich palette of filters.
• Request data related to the data that matches your filters. In the example above we could request all transactions that contain matching instructions, or even all token balance updates caused by such transactions.

Solana data is updated in real time and includes unfinalized blocks, enabling latencies of 1-2 s. Just omit the toBlock field in the request and you’ll get the latest data.

​

Making your own client

Real life protocols and infra introduce a couple of complications that you need to know about to actually build a client:

1. The endpoint may terminate the stream of blocks at any point. To resume the stream the client must update the fromBlock and the optional parentBlockHash fields of its request and send the request to the API again.

   Note that that does mean that our example theoretically could have taken two requests instead of one.

   If your request specifies a toBlock you can tell that it’s done when you get either a block with this slot number or an empty HTTP 200 response.

   If the range of your request is entirely above the range of available blocks you’ll get an HTTP 204 (No Content).

2. Ensuring data correctness when streaming unfinalized blocks is somewhat complicated. If you don’t want to deal with that complexity and don’t mind that the newest available data is up to several hours old, use the alternative /finalized-stream endpoint; otherwise read on.

   Show unfinalized blocks streaming instructions

   Since the API sends the blocks before it can ascertain that they are final, it will occasionally stream blocks that end up being orphaned. The client must detect such situations and process them according to its business logic. For example

   • if the data is simply written to the memory it may be necessary to overwrite it;
   • if the data is transformed and the results are used to populate a database, it may be necessary to roll back the database state changes made due to orphan blocks and re-run the transform on final blocks, etc.

   To be able to detect and correctly process orphan blocks the client must

   A. Maintain a backlog of hash values for recent blocks (available at the .header.hash field; enable it by setting the .fields.block.hash field of the request to true). Backlog depth of ten blocks should suffice on Solana.

   B. Be ready to restart its business logic at any block in the backlog. This may involve state rollbacks.

   C. Supply the parentBlockHash request field when resuming the stream after termination.

   D. Expect HTTP 409 (Conflict) responses from the API. These occur when the server detects that the first block of the request does not exist in the current consensus / is an orphan. The response contains a list of blocks preceeding the one that was requested:

   Copy

   {
     "previousBlocks": [
       {
         "number": 21780872,
         "hash": "0xf6a96a29423093e947960fcde3cf79730eadacd389fe2ed6cd1c97deb356a12e"
       },
       {
         "number": 21780873,
         "hash": "0xcc44e9d4723600bb3078c5e0ab5df0cf7513df2e12e85f8548c5c469083b19bb"
       },
       {
         "number": 21780874,
         "hash": "0x1dce783bdb93b72af818addd1e97473d64f6e25ab512ce790a89c7f0976f6a0a"
       }
     ]
   }
   

   The client must use this response and its block hash backlog to find the latest consensus block it received. If no such block is found it can send another POST request with the earliest block from the current response and a wrong hash (e.g. the one from the backlog): an HTTP 409 response from the API will contain even earlier blocks that might match those found in the backlog. This process can be repeated until the backlog is exhausted.

   Once the client finds a non-orphaned block it should restart the business logic and resume the stream at its height.

   Some of the shared code for pipes and the Squid SDK will deal with unfinalized blocks for you under the hood.

​

Base URL

For Solana use

https://portal.sqd.dev/datasets/solana-mainnet

Other SVM networks TBA.

​

Other endpoints

• GET /metadata - dataset metadata. Includes the number of the lowest available slot.
• GET /head - slot number and hash of the highest block of the dataset.
• GET /finalized-head - same, but only for finalized blocks. In the current implementation it will be behind /head by some thousands of slots.
• POST /finalized-stream - same as the main endpoint POST /stream, but will only return finalized blocks, up to the height you can get from GET /finalized-head