The State of Mainsail III

Part three of our ongoing ARK development summary.

In this post we will cover and attempt to quickly summarize ARK’s ongoing work on the redesigned and renamed codebase called Mainsail. Our previous two posts regarding Mainsail highlighted the first 12 weeks of development. This post will cover the 12 weeks since then. This post is lengthy and more technical but acutely summarized for easy reference and following.

Mainsail is the name of the newly redesigned ARK Core codebase currently in development. After an announcement earlier this year to rename and completely rebuild ARK Core, we have been able to follow live GitHub commits and detailed development reports of all progress on Mainsail to date. 

This post is focusing on the technical aspects and GitHub commits of Mainsail, but if you look back over the past 12 weeks there has been a significant amount of progress across the entire ecosystem. With hundreds of commits across multiple products including updated products, releases and more.

We’re excited with all of the current work and releases this last 12 weeks, but let’s jump into Mainsail development itself.

The week of August 11th, 2023

During this week the team reported that their main attention was directed toward consensus and P2P optimizations, accompanied by a series of general enhancements and resolutions.

The specific details of the week’s progress are as follows:

• The consensus handler underwent segmentation into distinct processor classes: Prevote, Precommit, Proposal, and CommittedBlock processors. Each processor assumes responsibility for activities such as buffer deserialization, data verification, signature authentication, and cross-referencing with previously received data. Subsequent to verification, the processor communicates the outcome back to the P2P layer, which possesses the capability to disqualify peers based on their verification status. Messages that are promptly authenticated are subsequently retransmitted. Notably, the node restricts the rebroadcasting activity to solely encompass messages received via broadcast, omitting those that have been downloaded.

• The transaction pool has been improved to now facilitate the broadcasting of transactions.

• The crypto generator has been updated to incorporate support for base58 addresses, a format that is used by the ARK network.

• A P2P Throttle class was extracted, introducing requisite code corrections necessary to manage heightened network traffic.

• The process of proposal serialization has been refined, contributing to improved efficiency and reliability.

• Improvements were made to the messages store, which is instrumental in restoring states during bootstrap. Notably, the store is now cleared upon block commitment to prevent data loss. It also accommodates multiple proposals from the same validator, an infrequent scenario that may arise if a block encounters non-acceptance across multiple rounds.

• Additional improvements were undertaken, encompassing refined log output, various consensus fixes, enhanced P2P verifier functionality, and precise schema adjustments. These actions collectively contribute to the ongoing enhancement of system robustness and reliability.

The report stated the next weeks primary attention would be directed toward monitoring the internal Mainsail network, enhancing network stability and identifying any potential issues. Simultaneously, efforts would be initiated to enhance the BlockProcessor module. Full report here.

The week of August 18th, 2023

During this week, the team reported there were notable advancements in Mainsail’s consensus domain, along with enhancements in the transaction pool and block verification areas.

The specific details of the week’s progress are as follows:

• Transaction pool replay was modified to use the transaction index instead of the transaction ID. This adjustment eliminates the need to deserialize the buffer in certain cases.

• Block verification has been improved. The node verifies the aggregated signature on the committed block and confirms that sufficient validators have signed the precommit message for that block.

• Consensus now reevaluates all rounds during the bootstrap phase, not exclusively the final one. A block is considered committed when over two-thirds of valid precommits and a valid proposal are received for any given round.

• Verification of lock proposals has been shifted to the proposal processor. The processor aims to promptly address invalid messages in the initial stages to prevent malicious peers, thereby banning them upon detection of invalid messages.

• The methods isMajority and isMinority have been extracted as standalone functions, as they are utilized across multiple locations.

• A new environment variable, CORE_P2P_PEER_BAN_TIME, has been introduced to establish the duration of peer bans. Peer banning is deactivated by default.

The team reported the next week would shift toward researching potential database and public API implementations. Concurrently, assessing the test network that had been established, addressing any issues uncovered during the testing phase. full report here.

The week of August 25th, 2023.

This week the team focused on efforts encompassing refining consensus-related aspects and optimizing the codebase.

The specific details of the week’s progress are as follows:

• LMDB stores have been divided into multiple files. ledger.mdb contains committed data like blocks and transactions, while consensus.mdb houses uncommitted data for restoring block rounds. This uncommitted data includes prevotes, precommits, proposals, height, and round numbers.

• To enhance readability, consensus logs now use validator usernames instead of public keys.

• A ValidatorWallet factory has been introduced in the state package. This helps in separating code and preventing direct creations in related packages. ValidatorWallet is responsible for encapsulating a wallet by the validator and offers additional methods such as getUsername, getConsensusPublicKey, and getWalletPublicKey.

• Refactoring has been done on RoundState and validator-set related code. Validator indexes are now used instead of public keys to access ValidatorWallet instances.

• Enhancements have been made to the proposal serialization with the addition of ValidRound. The serializer now supports various optional fields like uint8, uint32, bigInt, address, and more.

• Consensus code improvements include the incorporation of validRound checks.

• The Aggregator class, responsible for creating and validating aggregated signatures, has been simplified. It now provides generic aggregate and verify methods instead of specialized ones for lock proofs and committed blocks.

• Verification processes for signatures and lock proofs have been moved under the respective processor classes.

• The verification of the block generator’s public key now takes place in the ProposalProcessor rather than within the block processor. This change acknowledges that the correct block generator can only be checked for the first proposal of a block. On re-proposals of a locked block, the accurate block generator cannot be determined, but trust is placed in it due to the presence of proof with +2/3 prevotes, and validators that prevoted the initial proposal must have already verified it.

The team reported that the upcoming week’s objectives would include implementing block processor locking and committing blocks in an atomic manner. Processing and verifying Blocks, Prevotes, Precommits, and Proposals will wait while crucial state changes occur during the commit. Initial API implementation will also commence, along with an evaluation of the test network for potential issues and subsequent fixes. Full report here.

The week of September 1st, 2023.

The Mainsail team continued their efforts to refine the consensus process during this week. They also began work on API changes and addressed any issues that surfaced during this process.

The specific details of the week’s progress are as follows:

• Redundant data fields, specifically blockId and height, were removed from the IBlockCommit. This change was made due to the accessibility of this information within the block data itself, contained within IBlockCommit.

• The verification process for aggregate signatures of committed blocks was shifted from the block processor to the CommittedBlockProcessor within the consensus package. This restructuring separates block-related logic, handled by BlockProcessor, from consensus-related logic, now managed by CommittedBlockProcessor.

• The implementation of CommitLock was completed. This feature facilitates critical state changes during the commit phase, such as height adjustments, wallet updates, and validator set modifications. Verification processes dependent on the state can be executed in parallel, either preceding or following the commit stage.

• The deserialization of messages was relocated from the consensus module to the P2P component. P2P handlers now conduct basic data verification on received messages. If the received data is invalid or missing, the respective peer is blocked.

• Several minor fixes were applied to the P2P system to enhance its functionality.

• The existing API package has been renamed as api-development, reserving the API package designation exclusively for production API logic.

• The initial implementation of the public API has been initiated, encompassing blocks and transactions tables.

The team reported the next weeks objectives would include further extending the API implementation based on predefined specifications and enhancing block processing. Additionally, actively monitoring the limited testnet and promptly addressing any issues as they arise during testing. Full report here.

The week of September 8th, 2023

The Mainsail team reported this week had been actively engaged in two key areas: the initial development of the public API and addressing certain P2P-related issues that came to their attention.

The specific details of the week’s progress are as follows:

• Initial Public API Implementation: We have introduced three new packages: api-sync, api-http, and api-database. The api-sync package is designed to run within the main Mainsail process and is responsible for populating the database with fresh data. On the other hand, api-http is an implementation of a Hapi server, complete with predefined routes and controllers. This package operates as a separate process. Lastly, api-database manages database access for both api-sync and api-http, connecting to a PostgreSQL database. The current iteration of the API exposes straightforward endpoints and seeds the database with block, transaction, wallet, and validator-set round data.

• Processor and Consensus Enhancements: We’ve made several improvements and fixes related to processor and consensus functionality. Notably, we’ve added verification of block timestamps, ensuring that the time difference between blocks adheres to the duration defined in the blockTime milestone property. We’ve also addressed issues with the ProposerPicker seed calculation. Additionally, our consensus mechanism now handles discrepancies in data if a process crashes during the commit phase, ensuring that temporary messages are appropriately managed.

• Redefining State Building: Within the bootstrap package, we’ve redefined the process of state building. Custom builders, such as those for transactions, rewards, and committed rounds, have been replaced. Instead, blocks are now fully re-applied during the bootstrapping process. We’ve incorporated additional state indicators to differentiate between bootstrapping and regular block commits, which leads to less errors. During bootstrapping, certain checks and actions are skipped, such as storing blocks and transactions in the KV store and the committed transactions check.

• P2P Network Improvements: We have made significant changes to our P2P networking components. NetworkState has been eliminated, and NetworkMonitor has been replaced with a cleaner P2P service implementation. This implementation periodically checks for an adequate number of connected peers and ensures the timely delivery of messages. If a period passes without receiving any messages, random peers are reverified, and their message status is verified. Our P2P banning mechanism has also been refined; peers are now banned only when they provide invalid or inconsistent data. Timeouts and non-responsiveness, while not leading to banning, do result in peer disconnection. However, these disconnected peers have the opportunity to reconnect if they successfully pass the verification process.

The team reported the following week focus would be on expanding the API implementation and making general P2P improvements. Additionally, with plans to work on state snapshots to enable periodic saves of the blockchain state, which will improve node restart speed. Full report here.

The week of September 15th, 2023

This week, the team reported their main focus was on developing a dedicated API, which is entirely new in Mainsail, along with some improvements related to P2P.

The specific details of the week’s progress are as follows:

• P2P: Node disposes of a peer when there are request errors inside plugins. We also refactored peer discovery. When a peer is accepted, the node pings peer ports to detect which ports are open and checks their peer list. Newly discovered peers are verified, and the GetCommonBlocks endpoint is removed. We addressed issues such as app termination and the requirement of network and token flags.

• API: The API is now a standalone process. Common commands like start, restart, stop, setup, update, get status, or version are implemented within the API package. The package structure resembles that of the core. This approach allows users to install a standalone instance consisting solely of API packages. Deploying Mainsail or Mainsail API will create a folder with the same name in the default path. Each folder includes at least app.json and .env files that define plugins and parameters for the application.

• API Sync: Additional tables have been created and synced from the Mainsail process to the database. These tables include Peers, MempoolTransactions, and State. The first two tables are synced periodically, while State is updated with every commit.

• State: Wallet attributes have been refactored. Support for nested attributes like validator.username has been dropped. All attributes are now at the wallet’s root level, e.g., validatorUsername. We introduced attribute types to support attribute cloning and runtime verification. Supported attribute types include String, Number, Boolean, BigNumber, and Object. All wallet properties, except an address, are instances of these new attribute types. This feature was added to implement wallet delta comparison, support attribute cloning and verification, and add serialization for state storage. Wallet indexers were removed and will be replaced with a better implementation.

The team reported the upcoming week would consist of continued extention of the API implementations with additional options and work on refactoring of state storage. Full report here.

The week of September 22nd, 2023

Throughout this week, the team undertook several optimizations and enhancements to the blockchain state, alongside the development of a new API implementation.

The specific details of the week’s progress are as follows:

• We removed the WalletHolder class. Now, when a wallet is cloned, it maintains a reference to the original wallet. Initial attributes like balance and nonce are set to 0 upon wallet creation. The clone doesn’t duplicate these attributes but calculates them dynamically. If a requested attribute exists in the clone wallet, it’s returned; otherwise, the original wallet is checked. This approach supports multi-level cloning, allowing a wallet clone to be cloned. When committing wallet changes (forgetting and setting attributes), these changes are merged back into the original wallet. This method of implementing lightweight wallet clones ensures good performance, even for wallets with numerous registered attributes.

• We simplified WalletIndex, which now functions as a straightforward map, containing key-to-wallet mappings. We removed the backreference of wallets to keys because wallet reindexing during cloning is no longer necessary. WalletRepositoryClone now tracks index changes and applies them to the original WalletRepository upon commitment.

• WalletEvents have been removed. If necessary, they can be reimplemented differently. Users are generally interested in listening to onCommit changes in the blockchain WalletRepository.

• Wallets now contain a reference to the WalletRepository for marking themselves as dirtyWallets when changes occur. This approach eliminates the need to iterate through all wallets to filter the dirty ones, resulting in improved performance.

• WalletRepositoryClone is no longer bound to request scope but is created via a factory to support multi-level clones (original WalletRepository -> Clone WalletRepository -> CloneOfClone WalletRepository).

• Initial support for API integration tests has been introduced. These integration tests utilize a real Postgres database and are executed within the GitHub workflow.

• We’ve added search criteria and querying support to the API repositories. This means that query parameters can be included in API requests, which are then translated into SQL queries incorporating conditions like or, and, like, lessThan, and more.

In the upcoming week, the team would focus on expanding API implementations with additional features, introducing support for lists within wallets, and working on the export and import of blockchain state. Full report here.

The week of September 29th, 2023

This week, the team reported they made further improvements to the state management system and enhanced the functionality of the API.

The specific details of the week’s progress are as follows:

• We’ve introduced several new API controllers, including Transactions, Wallets, Blocks, and Node.

• We’ve refactored the state management system, utilizing attributes to store values such as height and totalRound. The attribute list can be extended to accommodate additional attributes that can be registered by packages or plugins. It’s now possible to clone the state and merge it back, paving the way for future functionality, like writing to a temporary cloned state during the export process.

• We’ve made some fixes to our P2P functionality. If a peer has not been accepted and added to the peer list, it will still be disconnected in the event of an error.

For the following week, the team reported their focus will remain on expanding the API implementation and addressing issues related to message downloads in the P2P network, particularly when another peer is multiple rounds ahead but at the same height. Additionally, working on implementing further changes to the state and its management system. Full report here.

The week of October 6th, 2023

This week the team made progress on blockchain state improvements and introduced some new API packages.

The specific details of the week’s progress are as follows:

• Transaction handlers now have a new method called getData, which is responsible for converting JSON data into the transaction data object. We introduced this method mainly to convert numeric data represented as strings into BigNumber. Amount, Fee, and Nonce are handled at the abstract transaction handler level, and concrete handlers will override this method to handle assets. For example, the multipayment handler stores payment amounts as BigNumber.

• The multipayment transaction has undergone refactoring. The amount field now holds the sum of all amounts. In V3, this field was always set to 0. We made this change to simplify the code and eliminate specific calculations for multipayment. For instance, the block amount is now calculated as the sum of all transaction amount fields. We removed the extra logic that incorrectly checked transaction type on the block level and executed custom logic to calculate the block amount.

• We implemented the blockchain controller on the HTTP API and synchronized the total supply with every commit. The value is calculated in the mainsail process.

• We added missing plugins to the HTTP API.

• A new package called api-common was introduced, containing shared logic and building blocks for different API packages. It includes a Server, abstract ServiceProvider, schemas, generic plugins, and more. API packages with specific logic use this package as a foundation and only add contextual logic in routes and controllers.

• We also implemented a new package named api-transaction-pool. This package features a single endpoint accessible on the mainsail nodes. It allows for the POSTing of transactions and attempts to add sent transactions to the mempool.

• We fixed the P2P download logic. Message downloaders can now handle multiple rounds for the same height. If another peer claims to have +1/3 prevotes for a higher round, we can skip intermediate rounds (from our round to the peer’s round) and directly download the higher round. This will initiate a new consensus round, even if we skip from round 3 to round 10, for example. Consensus ensures the safety that a block for the previous round was not committed when +1/3 prevotes exist for the current round.

• The State Service is a new manager for the state. It is responsible for creating WalletRepository and StateStore clones. Additionally, it serves as a getter for the blockchain WalletRepository and StateStore, which can no longer be obtained via identifiers. This class will also handle the import and export of the state in the future.

The team said the next step would be to continue enhancing the API’s features and work on the export state of the blockchain. Full report here.

The week of October 13th, 2023

This week, the team undertook several enhancements and expansions to the API, alongside improvements to the blockchain state.

The specific details of the week’s progress are as follows:

• The API now includes a delegates controller.

• Database migrations for the api-database are exclusively initiated from the api-sync package, rather than the api-http package.

• Hapi has been updated to version 21, alongside relevant packages such as @hapi/boom and joy.

• The synchronization procedure, responsible for storing new commit data in the database, has been moved outside the commit method. During the commit process, a new sync job is created. This alteration has reduced the duration of the commit phase, allowing more room for consensus-related processing. Additionally, the job is periodically reattempted in the event of a connection error during the sync. If the sync is not completed, and a new commit occurs, it is temporarily paused until the preceding job concludes.

• In the event of an unhandled error during the commit phase, the application is terminated. This precaution is taken due to the suspicion of state corruption, which could potentially create vulnerabilities.

• A new proposer package has been introduced to extract ProposerSelector logic that was originally part of the consensus package. Notably, ProposerSelector no longer relies on the private validatorMatrix but instead utilizes the StateStore attribute, which is exported in the snapshot.

• State snapshots are now a part of the system, encompassing the entire blockchain state, including the StateStore with registered attributes and the complete WalletRepository (comprising wallets, their attributes, and indices). These snapshots are exported in the .gz format on the data path. New environment variables have been introduced to control this feature:

  • CORE_STATE_EXPORT_DISABLED, which optionally disables exporting.

  • CORE_STATE_EXPORT_INTERVAL, which determines the export frequency (default set to 1000 blocks).

  • CORE_STATE_EXPORT_RETAIN_FILES, specifying the number of snapshots to retain, with a default value of 2.

The team reported that next steps primarily concentrate on API changes. Full report here.

The week of October 20th, 2023

During this week, the Mainsail team made progress in API integration and addressed issues found in the implementation of blockchain state management.

The specific details of the week’s progress are as follows:

• We have created a script for migrating a V3 database snapshot (including blocks, transactions, and wallets) into the Mainsail database. This script proves valuable for testing the new API with authentic data, identifying bottlenecks, and uncovering missing features.

• We’ve expanded the support for API search criteria. It now allows searches on JSONB columns with nested attributes, a crucial enhancement for various wallet-related queries and more.

• Common database indexes have been reintroduced for optimization.

• The API now correctly returns vendor fields as UTF-8 strings.

• A new wallet attribute has been introduced to store validator approval.

• Various minor issues related to recent state import/export processes have been resolved.

The team reported the following week would be focused on enhancing API test coverage and making general improvements. Additionally, work will start on dynamic validator size support. Full report here.

The week of October 27th, 2023

This week, the team made progress on API implementation and expanded its test coverage.

The specific details of the week’s progress are as follows:

• Updated all CLI commands to accommodate the new directory structure.

• Worked on enabling dynamic validator sizes, which is nearing completion. This allows for the adjustment of the number of active validators through milestones.

• The default database configuration has been set to Postgres version 16.

• Addressed missing test coverage for API controllers.

• Unique constraints have been given explicit names to enhance error readability.

• Improvements have been made to API synchronization, including the identification and resolution of issues preventing validator rounds from being recorded. This has made resynchronization more robust, reducing the need for frequent database drops.

In the upcoming week, the team reported focus will remain on testing the API implementation and its performance, alongside finalizing and testing dynamic validator size adjustments.

---

As you can see, the work on Mainsail is quite continuous with 202 reported commits over the last 12 weeks. This may not seem like a lot in comparison to other projects or products but remember quality of commits > than quantity of commits.

With Mainsail’s internal testnet running now, we are excited to learn more as the team progresses to a public testing phase.

Be sure to follow ARK on Twitter to catch any releases or news.

Follow Strake Foundation on Twitter for news, exchange listing events, announcements and more.

Check out the ARK Community Discord, Telegram and Reddit for discussions, live GitHub feed, announcements and more.

This article was put together using official ARK SCIC posted blogs, anything outside of quoted text from official sources is strictly the opinion of The Strake Foundation. If you like these articles and reviews we would appreciate a vote for our ARK validator, delegate strakefoundation, to help us continue to inform and support the ARK community.