The Subsquid Origin Story & Joystream Partnership

Subsquid may have first seen the light of day in the summer of 2021, but the query node framework, the engine in the Subsquid car, was around long before — at least in crypto terms.

The idea for the next-gen query node framework was conceived at Joystream, the decentralised video platform on the Polkadot network. It was officially kicked-off and developed at the Hackusama hackathon in August 2020 under the name Hydra, where it won the infrastructure track. Dmitry Zhelezhov, the technical co-Founder of Subsquid, was then working in the Joystream core-team and was responsible for developing the Hydra framework in-house, with Hydra quickly becoming crucial to Joystream’s Atlas product.

The Hydra framework attracted early interest from Subsocial and Threefold who began to use Hydra v0, the very first iteration. This support and external use made it clear that Hydra could not just survive as its own entity but thrive, and so the decision was made for Hydra to be taken outside Joystream. Subsquid was created as the new entity that would further develop and utilise the Hydra framework and build a decentralised infrastructure around it. The following, overwhelming positive response by key projects in the DotSama ecosystem was a clear indicator that there is a need for advanced data solutions that was not being met.

Joystream is now using the newest version of the Subsquid protocol and continues to build the blockchain-native, improved version of video sharing services like YouTube. An operation of this scale requires a powerful backend to match its ambitions.

Big Ambitions Mean Big Data Management

Achieving great data management involves the parsing and constant organisation of heavy amounts of data, ready for viewers to search for and consume. Individuals used to querying YouTube and getting a series of near instant results are going to be put off if their search results take longer than a few seconds to load. They may not even come back if they have a poor experience. Joystream’s vision isn’t just to have people access their content through their website — its team envisages a large number of different websites serving different audiences and websites, all sourcing data from the Joystream backend.

Joystream therefore needs a backend more powerful than those available out of the box. By design, blockchain nodes store the bare minimum of data and most of the time developers need to construct the desired outcome by stitching together traces spread across historical blockchain records. This is not just time consuming but also doesn’t lend itself to building the kind of responsive websites needed to overhaul YouTube!

Hydra Brings Instant Improvement and Scalability

Joystream decided for Hydra, knowing that it could scale with its own ambitions and growth. With rapid iterations and ever-growing complexity, Joystream’s Atlas product has become one of the most sophisticated DApps in the whole Substrate space, with Hydra helping to power it by returning up-to-date data from the Joystream blockchain. The end result of implementing Hydra has been a more responsive website with a richer user experience, while developers also benefited with reduced development cycles. This means that instead of having to redesign the backend from scratch each time a new feature is introduced to Atlas, developers simply update the Hydra-based schema and incrementally add on the functionality.

Hydra v5 Powering Joystream’s Future

Joystream has continued to use Hydra since Subsquid decoupled in May 2021, with Subsquid being equally as committed to delivering and supporting Atlas features, such as full text search at its core. With the imminent release of the much anticipated Subsquid’s Hydra v5, Joystream developers can expect radical performance improvements in many areas, allowing them to continue to improve the Joystream user experience and further expand the platform’s reach. If you like to join the Joystream community, their Discord is the best place to start.

Join our Community!

To find out more about Subsquid, join us on our Discord server and chat to one of our helpful Subsquid team members or other users. For those who want some more background information on Subsquid and what it’s all about, our website and Medium page are the perfect places to start.

Website, Twitter, Discord, Telegram, Reddit, Facebook, Insta