Before you jump into action itâs best to know a few key facts:
In Squid SDK the indexers are called squids.
The main component of any squid is its processor - a NodeJS process that ingests data from Soldexer or another compatible API, transforms it and stores it to a data sync.
Processor fetches and processes data in batches of variable size. Each batch contains data for a range of blocks. The size of this range can be anywhere from one to several thousands of blocks. It is automatically determined by the SDK.
When itâs done processing a batch of data, the processor will send the results to one of its supported data sinks. All such write operations are normally atomic / transactional, meaning that if you terminate the indexer in the middle of processing a batch then all code executed on that batch up to that point will have no effect on the data sink.
This allows squids to be stopped and restarted at any point without any fear of data corruption. Crashes wonât corrupt data, either.
When consuming real-time data, the processor will occasionally encounter orphan blocks. When that happens itâll use a log of recent DB operations to roll back any changes to the database state made due to orphan blocks, then restart data processing from the latest known consensus block. All of this happens automatically, with the end result being that the database changes its state to the correct one. If youâre simply sending the data to your appâs frontend itâs usually nothing to be worried about, but you have to plan for that if youâre archiving the data with some external tools or are using it in decision making.
Consistency guarantees mentioned in pp. 3-5 come with some fine print: you have to use Squid SDKâs built-in tools to access your data sink, or your warranty is void. Since squid processors are regular NodeJS apps you can make any calls to external libraries and/or APIs, including writes to arbitrary data sinks. However, youâll have to manage the consistency of any such writes on your own.
Now that you know the basics letâs take a look at an actual squid processor!