The State of Mainsail Part 2

What has happened over the last 12 weeks of development?

Mainsail is the name of the newly redesigned ARK Core codebase currently in development. After an announcement earlier this year to rename ARK Core and continue to build the new product in the open, we have been able to follow live GitHub commits and detailed development reports of all progress on Mainsail to date. 

In June we summarized Mainsail development over the first 6 weeks since it was announced, check it out here if you missed it https://strake.substack.com/p/the-state-of-mainsail. In this report, we will highlight another 6 weeks of progress, marking 12 weeks of Mainsail development.

This report is focusing on the technical aspects and GitHub commits of Mainsail, but if you look back over the past 6 weeks there has been a significant amount of progress across the entire ecosystem. With hundreds of commits across multiple products including releases of updated products and multiple ARK Core releases. 

We’re excited with all of the current work and releases this last 6 weeks, but let’s jump into Mainsail development itself.

The week of June 30th

During this week, the Mainsail team made progress in executing consensus-related tasks and undertaking essential P2P modifications to accommodate the impending change.

To summarize the work done:

• Implemented the addition of a store for Proposal, Prevotes, Precommits, and consensus state. Content is stored on disk using a key-value store and is cleared at each height.

• Achieved consensus bootstrapping through the use of the store state. All consensus messages from the previous height are restored and processed to restore the consensus state.

• Enhanced the functionality of the consensus and block processor to handle two structures: RoundState, which represents received Proposal, Prevotes, and Precommits for the active block round, and CommittedBlockState, which represents the committed block after receiving +2/3 commits. A valid block can be represented using either of these structures. CommittedBlockState is utilized during node synchronization, while RoundState is employed when the node is already synchronized.

• Updated the P2P functionality by removing the isAppReady plugin and introducing support for downloading CommittedBlocks. The node initiates block downloads when it detects another peer with a higher height. The PostBlock route has been removed, and nodes now only broadcast Propose, Prevote, and Precommit messages. Missing blocks and consensus messages can be downloaded if detected from the header.

• Ensured the execution of consensus handlers in a locked fashion, preventing parallel execution of two handlers to ensure data consistency.

• Introduced the concept of commitedRound, which represents the total number of rounds required to form the last committed block. This value is used to determine the correct proposer within the given validator set.

In this dev report the team said the next week would be focused on continuing their ongoing efforts to make essential changes to the consensus mechanism functionality and P2P layer. Full report here.

The week of July 7th

During this week of Mainsail development, the team reported it is on schedule and they are happy with the progress being made so far.

To summarize the work done:

• Synchronous methods were utilized wherever possible in the Consensus, ValidatorSet, and ValidatorPicker modules.

• P2P headers were incorporated into the response, with peer state being updated for every received request and response. The handling of headers was improved, including introducing a delay before performing header comparisons to allow new pre-votes and pre-commits to arrive via broadcast. Comparisons were skipped if a pending check for the given peer was already underway.

• The Downloader module was modified to allow only one concurrent job at a time. Upon completion of a download, peer headers were rechecked for any new messages that may have arrived during the process.

• Support for NetworkStart and ColdStart was removed, simplifying the P2P logic. Separate procedures for genesis boot and integration into the running network were deemed unnecessary.

• The PeerDiscoverer logic was extracted from the NetworkMonitor module, streamlining the codebase.

In this dev report the team stated the next step is to focus on further improving the downloaded logic and implementing support for concurrent downloads. Additional efforts will be dedicated to enhancing schemas and optimizing data types used within the system. Full report here.

The week of July 14th

According to the team, the development progress of Mainsail remained on track during this week.

To summarize the work done:

• Updated the third-party packages we utilize, which had any known security vulnerabilities, to their latest versions that addressed those vulnerabilities.

• AJV validation now employs a strict schema, ensuring that additional properties are not removed from the objects undergoing validation. The AdditionalProperties schema option is set to false for cases where restricted properties are encountered on the validated object.

• P2P functionality now includes a “development mode” specifically designed for local testing and development purposes. In this mode, the server responds to HTTP requests while disabling data validation and code plugins. The CORE_P2P_DEVELOPMENT_MODE_ENABLED flag can be used to enable this mode.

• P2P Reply schemas have been divided into multiple files, resulting in improved and stricter verification. These schemas have undergone testing.

This week the team reported the next steps for the upcoming week. Focusing on enhancing the download logic by incorporating support for concurrent downloads and utilizing buffers instead of hex values to reduce message size. Full report here.

The week of July 21st

During this week’s development, the team focused on improving the download logic and optimizing the codebase. Several important updates were implemented.

To summarize the work done:

• Removed the blockchain package, retaining only the essential node initialization logic. This logic was then extracted and placed into a new bootstrap package.

• To improve message efficiency, we replaced the use of hex values with buffers for serialized fields in P2P messages. This change resulted in reduced message size, contributing to better network performance.

• A new block downloader class was re-implemented to support parallel downloads from multiple nodes. This approach significantly speeds up the download process. Moreover, downloaded messages are deserialized and passed to the serializer. In case of any errors during download or response handling, messages are automatically re-downloaded from another random peer. Additionally, block downloading is now automatically initiated when a peer with a higher height is detected, ensuring the latest data is obtained.

• Introduced the validator-set-vote-weighted package, which employs the same algorithm as Core V3. This package is responsible for selecting top validators based on received votes and their vote wallet balances. Furthermore, the Mainsail structure allows for the easy binding of various validator-set strategies, such as static or newly implemented vote-weighted options. This flexibility enables the project to adapt to its specific requirements effectively.

• Implemented the validator set shuffle logic within the propose-pricker package. This logic operates independently of the validator-set calculation strategy. As a result, the given validator set is randomly shuffled every 51 blocks, promoting fairness and decentralization in the network.

The team announced the next week they aimed to enhance the download process for proposals, prevotes, and precommits, conduct a thorough review and testing of the @hapi/nes library, utilize height 0 for the genesis block, and work on improving the bootstrapping mechanism. Full report here.

The week of July 28th

This week the team reported they made significant progress on the P2P aspect and network initiation for Mainsail.

To summarize the work done:

• MaxPayload, a setting in P2P, has been removed due to inefficiencies in handling the increased traffic in Mainsail. We plan to explore alternative options for effectively limiting request payload in the future.

• Significant improvements have been made in downloading prevotes, precommits, and proposals. Nodes now concurrently retrieve missing data from peers when possible, and error handling has been enhanced.

• A new P2P class called PeerBlocker has been introduced. It maintains a list of misbehaving peers and automatically blocks their connection for a period of time. Peers may be blocked for various reasons, such as providing invalid response formats, missing or mismatched data, socket errors, or response timeouts.

• The database service has been extended to include a generator that reads committed blocks in ascending order, and this new method is utilized in the bootstrap process.

• The height of the Genesis block has been adjusted from 1 to 0, a modification made to simplify the codebase.

• In the default testnet implementation, the number of validators has been increased from 51 to 53. The decision to use 53 validators is based on the fact that 53 is a prime number.

The team said that moving forward into the next week, focus would be on ensuring peer verification is compatible with Mainsail. Additionally, improvements will be made to the P2P system, including message rebroadcasting and peer banning based on processing results. Full report here.

The week of August 4th

During this past week, the team reported making notable progress in the area of P2P optimizations, particularly focusing on rewriting the PeerVerifier module.

The specific details of the week’s progress are as follows:

• Deprecated the usage of NodeJS Buffer.slice and replaced it with Buffer.subarray to enhance performance.

• Modified the Propose, Prevote, and Precommit classes to expose a serialized property, enabling better efficiency by avoiding repeated serialization during message broadcasting.

• Updated the block timestamp representation to be in UNIX milliseconds instead of seconds. This change allows for finer tuning of the minimal block time and minimizes the delay between each block.

• Separated the signature aggregation logic from RoundState into its own Aggregator class, improving code organization and maintainability.

• Conducted cleanup in the StateStore to enhance its overall efficiency.

• The revised PeerVerifier module benefited from the block finalization process, leading to a simplified peer verification process. The acceptance of new peers now depends on the fulfillment of the following conditions:

  • Validity of the peer’s IP address.

  • Absence of blocking or pending status for the peer.

  • Presence of a valid version for the peer.

  • Matching nethashes between peers.

  • Consistency with the highest block among peers.

Moving forward, the team said their focus for the upcoming week will be on consensus optimizations, while simultaneously initiating work on improving the BlockProcessor module. Full report here.

---

As you can see, the work on Mainsail is quite continuous with 91 reported commits. This may not seem like a lot in comparison to other projects or products but remember quality of commits > than quantity of commits.

We’re looking forward to progressing closer to a live Mainsail testnet and finding out more features coming with it soon. There are a few things that stand out already over the last 12 weeks of development. One large change we are excited to see is moving from 51 to 53 validators. Take a look yourself to see some of the recent dev work and let us know if you spot something interesting.

Be sure to follow ARK on Twitter to catch any releases or news.

Check out the ARK Community Discord and Reddit for discussions, live GitHub feed, announcements and more.

This article was put together using official ARK SCIC posted blogs, anything outside of quoted text from official sources is strictly the opinion of The Strake Foundation. If you like these articles and reviews we would appreciate a vote for our ARK validator, delegate strakefoundation, to help us continue to inform and support the ARK community.