The State of Mainsail V

Part five of our ongoing ARK development summary.

In this post we will cover and attempt to quickly summarize ARK’s ongoing work on the redesigned and renamed codebase called Mainsail. Our previous 4 posts regarding Mainsail highlighted the first 30 weeks of development. This post will cover the 12 weeks since then.

Mainsail is the name of the newly redesigned ARK Core codebase currently in development. After an announcement earlier last year to rename and completely rebuild ARK Core, we have been able to follow live GitHub commits and detailed development reports of all progress on Mainsail to date. 

This post is focusing on the technical aspects and GitHub commits of Mainsail, but if you look back over the past 12 weeks there has been a significant amount of progress across the entire ecosystem. With product updates, releases and commits across multiple repositories. We have also learned that Mainsail is almost ready for a public testnet. So lets jump right into the technical development summary.

The week of December 15th, 2023

This week, the Mainsail team worked on further improvements to the consensus and various sync performance enhancements.

The specific details of the week’s progress are as follows:

• Graceful application shutdown procedures were implemented, activating process interruption. The async dispose method is called on all registered ServiceProviders, allowing registered packages to safely close connections, and file handlers, and store data to disk. The procedure is forcefully terminated if dispose is not completed within the allotted time or if another process interruption occurs during the shutdown period.

• During node sync, blocks are now stored at the end of the download chunk, enhancing sync performance by reducing disk access frequency. Blocks are stored using a database transaction to ensure safety, while standard block processing still stores every block.

• Consensus messages (proposal, prevote, and pre-commit) for the active height are now stored on dispose, improving both sync and standard processing performance by eliminating the need to access the disk every time a new message is received or formed.

• The Crypto worker package was implemented to create a pool of node workers for executing common crypto methods off the main thread. This is used to aggregate block signatures or validator public keys and to verify signatures, with the number of workers adjustable via CORE_CRYPTO_WORKER_COUNT. Logging on workers is disabled by default but can be enabled through CORE_CRYPTO_WORKER_LOGGING_ENABLED.

• Sync performance was boosted by verifying blocks in chunks, and conducting parallel block signature verification using crypto workers. The number of active workers and CPUs significantly impacts sync performance, and chunks consist of blocks up to the end of the validator round to ensure no changes to the validator set during block processing.

• Blocks are retrieved from the state whenever possible to optimize performance and minimize unnecessary disk hits by looking up the last block in the database.

• Most packages were updated to the latest versions, including a TypeScript update to version 5.3.3, with the target build increased to ES2021. Lerna was also updated to version 8.0.0.

• Various fixes were implemented based on discoveries made on the internal testnet.

Looking ahead to next week, the teams primary focus will be on code cleanup to streamline and eliminate accumulated technical debt, along with addressing issues identified on the internal testnet. Full report here.

The week of December 22nd, 2023

This week, the team focused on optimizing the codebase to enhance readability and addressed bugs identified on the internal testnet.

The specific details of the week’s progress are as follows:

• The ‘I’ prefix has been removed from all interfaces. This modification facilitates easy differentiation between classes and interfaces. Interfaces are now imported from contract packages and appear in the code as follows: Contract.<Package>.<Interface>.

• API nodes are regularly retrieved from nodes and undergo rechecking every 10 to 20 minutes. This process ensures the maintenance of an up-to-date list of API nodes.

• The crypto-commit package has been derived from the crypto-block. The latter now exclusively contains block-related logic for serialization, de-serialization, verification, and creation. Any additional logic has been relocated to the appropriate packages.

• Improvements have been made to the ‘Dispose’ function. Consensus storage is now cleared before storing active round data, and consensus handlers await state updates before storage.

• A new ‘Timestamp’ class has been introduced. It defines the earliest timestamp for a specific message based on message height and round. BlockTime, stageTimeout, and stageTimeoutIncrease values from milestones are utilized to determine the timestamp, with an additional 500ms drift to account for potential clock differences. If a message is received too early, it indicates a consensus violation, resulting in a node being banned.

• An additional ‘round’ field has been incorporated into the block. This field represents the consensus round in which the block was created and inserted for precise timestamp verification.

• All instances of CommitBlock in the code have been renamed to Commit. Commit now refers to a structure that stores the block and proof (aggregated signatures of +2/3 validators). Looking ahead, any issues encountered on the internal testnet over the holidays will be addressed after the New Year.

This was the end of the 2023 development year as the team prepared for holidays and promised reports to resume in early January 2024.

The week of January 12th, 2024

This week, the focus was on enhancing the database layer and expanding the API with additional endpoints.

The specific details of the week’s progress are as follows:

• Merged the storage-lmdb package with the database package, integrating LMDB as the supported KV store for Mainsail.

• Introduced a generator verifier to the block verifier. The generator public key is now validated based on the previously added round parameter.

• Refactored the validator package, relocating the consensus public key to the validator.json file. This key is utilized for signing consensus messages. It is recommended to use different mnemonics for the consensus public key and wallet. The design allows users to configure a validator node with chosen mnemonics before registering the consensus public key. The node autonomously detects if the mnemonic belongs to the active validator and joins the signing process accordingly.

• Increased the MaxPeerBroadcast value to 4, which can be adjusted through the CORE_P2P_MAX_PEERS_BROADCAST environment variable. This value determines the number of randomly chosen peers for broadcasting pre-votes, pre-commits, proposals, or transactions.

• Added a new column, updated_at, to the wallets table in PostgresDB. This column represents the latest height when the wallet was updated and is used for statistical calculations in ARK Scan.

• Introduced a new endpoint, /commits/{block-height}, to the api-http. This endpoint provides a list of validators that signed the pre-commit for the specified block, allowing users to verify the block signature.

Next week, the teams announced plans for involved stress-testing the internal network and conducting scenario-based tests to assess different states. Additionally, becoming more dedicated to the documentation efforts for Mainsail. Full report here.

The week of January 19th, 2024

During this week, the team focused on general improvements and introduced a straightforward CLI tool for testing transaction types.

The specific details of the week’s progress are as follows:

• Refactored identifiers for containers, following the Identifiers.<package/namespace>.<class/instance>.<optional: factory> pattern throughout the entire project.

• Added a commit round to the api-database, distinguishing rounds in the block response: round (consensus round when the block was proposed), commitRound (consensus round when the block was committed), validatorRound (representing the validator round, e.g., round height in ARK V3).

• Refactored and improved the readability of the state and P2P packages.

• Extracted the Repository class for the state store and wallet, encompassing common logic for handling attributes. This class includes methods such as setAttribute, getAttribute, and forgetAttribute. The Repository supports nested cloning and manages data merging back to the original Repository on commit.

• Removed workerPlugins from app.json. Each ServiceProvider now includes the requiredByWorker method, indicating whether the plugin should be loaded in workers. This enhances the readability of app.json by eliminating duplicated packages.

• Integrated WalletRepository into the state. When a new block is processed, it alters the content of the state and the cloned WalletRepository. Changes are written back to the original state on commit. Multiple clones of the same state can be active concurrently in specific scenarios, such as when multiple blocks for the same height but different rounds are proposed.

• Prepared a simple mainsail-tx-tester, enabling users to send all available transactions via the CLI. This is a standalone package.

In the upcoming week, the teams plans were to include making an initial release on npmJS, addressing validator set fixes and state export improvements, while continuing our work on documentation. Full report here.

The week of January 26th, 2024

During this week, focus remained on general enhancements and bug fixes. Additionally, groundwork was completed for the NPMJS Alpha release, gearing up for both internal and public testing.

The specific details of the week’s progress are as follows:

• Enabled the allowance of 0 active validators at height 0, as the genesis block initializes the validator set. The initial validator count is established at height 1.

• Rectified the allByIndex function in WalletRepositoryClone. It now returns cloned wallets from the original repository and is primarily utilized by the ValidatorSet to obtain a list of registered validators.

• Implemented support for state exports on the ValidatorSet. Active validators are stored in the store under the activeValidators key as a string of public keys delimited by commas. Additionally, a restore method has been added to the ValidatorSet.

• Extracted the State class from the Store. The State now exclusively contains blockchain-related data and facilitates snapshot export and import. It primarily encompasses node state data, such as bootstrap status.

• Addressed an issue related to username registration errors. When overwriting a wallet username, the previously used username is removed from the index, making it available for reuse.

• Released the alpha version of Mainsail on NPMJS with the current version marked as 0.0.1-alpha.2.

• Implemented various fixes and refactorings.

Looking ahead, the teams focus was reported to be on continued writing of documentation for Mainsail and addressing fixes to enable the integration of public nodes into the existing testnet network. Full report here.

The week of February 2nd, 2024

This week the team continued polishing and adding new features to facilitate node management, as well as addressing a few issues.

The specific details of the week’s progress are as follows:

• Introducing support for alpha channel on nodes and updating schemas to accommodate tagged node versions.

• Implementation of a mechanism wherein the booting node checks for a new version. If a new version is available, it displays a message along with upgrade instructions.

• Improved detection of libjemalloc.so even in the absence of an installed GCC library, contributing to a reduction in Docker size.

• Activation of the config:forger:bip39 command.

• Set pnpm as the default package manager. Mainsail updates are performed using pnpm.

Next week, the teams focus reported to be on further documenting Mainsail and developing commands to streamline the deployment of Mainsail nodes. Full report here.

The week of February 9th, 2024

During this week, the team focused on various tasks related to Mainsail, including documentation writing, codebase cleanup and optimizations.

The specific details of the week’s progress are as follows:

• Implemented config:publish:custom CLI command, enabling users to load custom network information from online sources. Users provide links for app.json, crypto.json, and peer.json files.

• Updated and cleaned up project root dependencies.

• Continued the preparation of documentation for Mainsail.

• Worked on install scripts to facilitate the installation of Mainsail and all its dependencies.

The teams stated that the following week their focus remains on documentation writing and the preparation of essential scripts for seamless deployment of Mainsail nodes. While also addressing any bugs that may arise during testing. Full report here.

The week of February 16th, 2024

During this week, the team focused on optimizing the codebase, enhancing the install script with additional features, and continuing documentation efforts.

The specific details of the week’s progress are as follows:

• Removed the api-sync and api-database packages from the default testnet app.json. This simplifies the project’s initial setup for new users, eliminating the requirement for an active PostgreSQL instance.

• Renamed PM2 process names to mainsail and mainsail-api.

• Consolidated ENV variables into a single file, accessible for users to review all variables used in the project.

• Removed the necessity of flags for token and network, resulting in shorter and more legible commands. For instance, the command mainsail env:list --token=ark --network=testnet is now mainsail env:list. Additionally, mainsail folder paths no longer include token or network namespaces. For example, the data path has been updated from ~/.local/share/ark-core/testnet/mainsail to ~/.local/share/mainsail. This restructuring reflects the common usage of a single network by users.

• Enhanced the install script to support mainsail and mainsail-api installations alongside the PostgreSQL instance.

• Updated the p2p-iptable.sh script to support Docker setups, to be executed from the Docker host.

Next week, the team reported to be focused on finalizing documentation for public release and commencing the last preparations for the initial public testnet launch of Mainsail. Full report here.

The week of February 23rd, 2024

This week has a little more excitement behind it. It was reported that the Mainsail team has been diligently preparing all the necessary components for the upcoming initial public testnet release of the Mainsail network. Their efforts have been focused on ongoing documentation and the development of installation scripts and tools that will help with testing.

Additionally, they’ve initiated exploration into the potential integration of the Ethereum Virtual Machine (EVM), coded in the Rust programming language, into Mainsail. They’ve begun outlining the specifications and modifications required to support this integration. An early Proof of Concept for this integration has also been started.

Furthermore, a new CLI tool named Mainsail Tx Tester has been developed to facilitate easier transaction testing. This tool is now publicly available and can be accessed on GitHub .

The specific details of the week’s progress are as follows:

• The mainsail file structure underwent refactoring. A root folder named mainsail was introduced to facilitate integration with Docker. The folder structure was adjusted from ~/.local/share/mainsail to ~/.local/share/mainsail/core for core components and ~/.local/share/mainsail/api for the API.

• Docker builds have been prepared and are now available on Docker Hub for both Mainsail Core and Mainsail API .

• Mainsail Tx Tester has been configured to function with the testnet network by default. Additionally, it now includes the capability to generate a new user wallet with a generated mnemonic, public key, and address.

• An initial EVM boilerplate has been added to the repository, introducing new packages @mainsail/evm and @mainsail/crypto-transaction-evm-call.

• A new genesis block has been created using Keccak256 addresses and ECDSA public keys.

• Documentation is being drafted for inclusion on ark.dev .

Looking forward, the team reported their focus remains on preparing for the public release of the initial testnet for the Mainsail network. Concurrently, continuing advancement of the development for the EVM implementation. Additionally, they will be working on a script to detect missing or unused dependencies in packages, aiming to maintain a cleaner and more efficient codebase. Full report here.

The week of March 1st, 2024

During this week, the focus was on addressing fixes for Mainsail that were identified, as well as preparing essential components for an upcoming public testnet. Additionally, progress was made on the implementation of the EVM Proof of Concept.

The specific details of the week’s progress are as follows:

Mainsail:

• Refactored the block verification process to ensure that each handler failure provides an error message that is logged.

• Enhanced the bootstrap method to compare the genesis block from configuration with the stored genesis block from the database, ensuring compatibility between configuration and ledger data. The process halts any mismatch.

• Introduced a dependency check script to verify dependencies in both the source code and tests, reporting any missing or unused dependencies. A new CI workflow has been established to check dependencies with every pull request.

• Utilized the .jpeg version of the banner in packages to showcase images on npmjs.

Mainsail EVM:

• Introduced the evm-development package, serving as a temporary EVM bootstrap package for deploying ERC20 tokens for testing purposes.

• Included support for EVM interaction in crypto-transaction-evm-call. This enables transactions for contract calls and deployments.

• Expanded the api-common package to include support for the JSON-RPC 2.0 specification. The package now comprises a JSON-RPC processor and an additional plugin for managing server errors and returning accurate JSON-RPC error responses. Users can register custom RPC methods, and an AJV verifier is employed for JSON payload and method parameter verification.

• Created the initial skeleton implementation of Ethereum JSON-RPC in the api-evm package, leveraging the RPC building blocks provided by the api-common package.

Looking ahead, the teams focus for the next week will remain on advancing the EVM Proof of Concept implementation and finalizing all necessary components for the initial testnet launch of Mainsail. Full report here.

---

As you can see, the work on Mainsail is quite continuous with 170 reported commits over the last 12 weeks. This may not seem like a lot in comparison to other projects or products but remember quality of commits > than quantity of commits.

Of course Mainsail isn’t the only thing happening across the ARK Ecosystem. There have been new releases of other products like ARK Scan and ARK Connect, as well as ARK Core updates.

New Exchange listings continue to come in as well, but the most exciting thing by far is the fast approach of a public Mainsail testnet and an integration of a very long awaited EVM module that is being tested as well.

As you can see the public testnet is close, I can’t say the exact date, but I feel its within days to weeks now. This concludes our report, I hope you enjoyed it.

Be sure to follow ARK on Twitter to catch any releases or news.

Follow Strake Foundation on Twitter for news, exchange listing events, announcements and more.

Check out the ARK Community Discord, Telegram and Reddit for discussions, live GitHub feed, announcements and more.

This article was put together using official ARK SCIC posted blogs, anything outside of quoted text from official sources is strictly the opinion of The Strake Foundation. If you like these articles and reviews we would appreciate a vote for our ARK validator, delegate strakefoundation, to help us continue to inform and support the ARK community. If you would like to learn more about ARK and validator/delegate voting and staking, please read our Staking Guide.