The State of Mainsail IV

Part four of our ongoing ARK development summary.

In this post we will cover and attempt to quickly summarize ARK’s ongoing work on the redesigned and renamed codebase called Mainsail. Our previous 3 posts regarding Mainsail highlighted the first 24 weeks of development. This post will cover the 6 weeks since then.

Mainsail is the name of the newly redesigned ARK Core codebase currently in development. After an announcement earlier this year to rename and completely rebuild ARK Core, we have been able to follow live GitHub commits and detailed development reports of all progress on Mainsail to date. 

This post is focusing on the technical aspects and GitHub commits of Mainsail, but if you look back over the past 6 weeks there has been a significant amount of progress across the entire ecosystem. With product updates, releases and commits across multiple repositories. We have also learned that Mainsail is up and running on an internal testnet. So lets jump right into the technical development summary.

The week of November 3rd, 2023

This week, the team focused on enhancing blockchain and validation mechanics.

The specific details of the week’s progress are as follows:

• We have transitioned to using Postgres 16 Docker images.

• Node synchronization status is determined based on the height of one’s own peer and the median peer height.

• For variable string fields like id, address, and senderPublicKey, we now use varchar instead of fixed-length strings. This change allows users to utilize different cryptography methods without the need for modifying or rebuilding the database.

• If the size of the ActiveValidators is increased, an error will occur if there are insufficient active validators present. It is the responsibility of the specific network to ensure an adequate number of validators during the adjustment.

• We’ve added additional generator (the validator that forges the block) data to the block response via the API.

• The “Save” command has been replaced with “upsert” on block and transaction repositories to prevent errors when the same item is stored multiple times.

• The processing of the genesis block has been standardized to match other blocks by using the block processor. The key difference lies in the fact that the genesis block employs a value of 0 for active validators, meaning that signature checks are not conducted. However, other checks, such as transactions and nonces, proceed as usual. The genesis block is the only one allowed to create wallets with negative balances, known as the genesis wallet.

• We have completely eliminated bootstrap methods from transactions. All transaction processing and state changes now follow the same protocol, employing the block processor.

The team stated that the next weeks primary focus would be on addressing the remaining aspects of the public API and implementing further optimizations for the blockchain mechanics. Full report here.

The week of November 10th, 2023

This week marked significant strides in the teams development journey, as they dedicated efforts to finalizing the initial version of the new API. Making it align as closely to the current core version as possible.

The specific details of the week’s progress are as follows:

• Streamlining the block processor during bootstrap by omitting logs for each block and implementing a status update every 10,000 blocks.

• Implementing peer routes and controllers, enhancing network connectivity and management.

• Refining the api-common package to exclusively export plugins, while allowing concrete implementations to provide their own plugin factories and selectively register plugins.

• Ensuring a fair and balanced distribution of premine among validators through the default testnet genesis block.

• Enhancing the transaction deserializer with support for transactions signed using multisignature wallets, bolstering security and flexibility.

The team reported the upcoming weeks primary objectives are to methodically address and resolve a backlog of identified bugs and to lay the groundwork for extensive internal testnet trials, marking a pivotal step towards the initial goals for Mainsail. Full report here.

The week of November 17th, 2023

This week marked significant progress as the team focused on enhancing various aspects in preparation for thorough testing on the internal testnet.

The specific details of the week’s progress are as follows:

• The TransactionProcessor was extracted from the BlockState and BlockVerifier, taking on the responsibility of transaction verification, fee checks, transaction processing, and validator balance updates.

• The username attribute is no longer exclusive to validators. Validator registration now only necessitates a consensus public key, allowing registration without a previously registered username.

• Two new transactions were introduced in new packages: core-transaction-username-registration and core-transaction-username-resignation. The former is utilized for registering or updating wallet usernames, open to any wallet, while the latter facilitates username resignation.

• The genesis block now includes username registration for genesis validators. Mainsail logs the validator’s username if available; otherwise, it logs the validator’s public key.

• The API Database can be truncated using the CORE_API_SYNC_TRUNCATE_DATABASE environment variable. The bootstrap processor detects the Postgres database’s height and loads the appropriate snapshot based on that height.

• The Base58 checksum calculation has been corrected to adhere to the specification’s double SHA256 calculation.

• Minimum fees in schemas have been standardized to 0, and the BypassGenesis option has been removed. Validation of fees is now handled by the fee matcher. This simplification in schemas is driven by the understanding that, from a performance standpoint, setting the minimum fee to 0 or 1 has no impact, given that the check will have to go through the matcher.

• A new package, crypto-signature-schnorr-legacy, has been added to align with the Schnorr library used in V3. Mainsail’s default crypto-signature-schnorr uses Schnorr from bcrypto, while crypto-signature-schnorr-legacy uses secp256k1 from bcrypto.

The team said the next week would be focused on a few things.

• Eliminating separate transactions from the LMDB store, opting to store full blocks.

• Updating the username schema with new rules.

• Verifying fees on proposed blocks.

• Implementing additional API improvements.

• Initiating work on basic ARK Scan integration and support for Mainsail in preparation for the internal testnet.

Full report here.

The week of November 24th, 2023

This week, notable advancements were made in optimizing the database and introducing a new username schema, which is now integrated into Mainsail along with additional API features.

The specific details of the week’s progress are as follows:

• The LMDB mainsail database underwent refactoring, now storing only block commits by height. Transactions and blocks-by-id storage were removed, resulting in reduced complexity and space consumption.

• Implemented a new username schema allowing for a minimum of 1 and a maximum of 20 lowercase characters or digits. A single repetition of an underscore is permitted in the middle of the string.

• Enhanced API functionality to support transactions signed using multi-signature wallets.

• WalletRepositoryBySender, used in mempool, was fixed to accommodate changes and cloning introduced in previous PRs.

• The api-development package underwent cleanup and fixes. Basic logic (Server, Plugins, ServiceProvider) is constructed upon api-common packages, similar to other API packages.

• Addressed various crypto and API issues.

The team said that next weeks focus includes standardizing ENV variables and API options across Mainsail. Additionally, any bugs discovered in the internal testnet will be promptly addressed. Full report here.

The week of December 1st, 2023

This week, the team reported the focus was on introducing a new class responsible for managing environment variables and further improving blockchain management.

The specific details of the week’s progress are as follows:

• Implemented the Environment class to facilitate the reading of environment variables. Default values can be specified for the get methods when an env variable is set. The isTrue method is included to interpret boolean values, returning True when the env variable is set to true or 1. This adjustment enhances clarity in settings, as previously any set value was treated as true, including false and 0 values.

• Addressed apy-sync initialization by running the migration script first, followed by the utilization of boot procedures by the package.

• Reincorporated unconfirmed transactions into the Postgres database during the boot process.

• Disabled total count estimation for the peers table in api-http.

• Modified the Node to exclusively accept serialized transactions on the POST api/transaction-pool endpoint. JSON transaction support is removed, leading to skipped transaction serialization on the server side and improved performance. Transactions are deserialized and handled as provided, eliminating the need for additional serialization steps.

• Introduced schema validation on the POST api/transaction-pool endpoint, replacing the HapiAjv plugin with the built-in Joi schema validator.

• Enhanced the CommittedBlockProcessor to await consensus completion before proceeding to the next block during block synchronization. Height and rounds are increased atomically during consensus.

• Implemented Bech23m address factories and addressed various other fixes.

The team reports the next weeks focus will be on developing support for API node discovery and distribution, along with improving high payload block synchronization. As they continue testing and monitoring the internal testnet, any discovered bugs will also be promptly addressed. Full report here.

The week of December 8th, 2023

This week, the teams efforts were dedicated to enhancing the database, making additions to the API, and implementing new IP tables.

The specific details of the week’s progress are as follows:

• Fixed maxPayload values for the P2P server and client, utilizing the maximum maxTransactionBytes value for POST /transaction-pool/.

• Enabled the sharing of API nodes over the P2P, with the list accessible via the /api-nodes endpoints. Nodes can register known API nodes using the CORE_P2P_API_NODES environment variable.

• Updated the LMDB database version, implementing a new persist method to enhance download performance. Committed blocks are cached during synchronization and written to disk at the end of the processing chunk.

• Modified the IP tables script for Mainsail, allowing users to set IP table limits for P2P, public API, development API, and transaction pool API.

• Various miscellaneous fixes.

Looking ahead to the next week, the team’s focus will be on incorporating a feature for graceful node termination and working on improving synchronization speed performance. Full report here.

---

As you can see, the work on Mainsail is quite continuous with 131 reported commits over the last 6 weeks. This may not seem like a lot in comparison to other projects or products but remember quality of commits > than quantity of commits.

Of course Mainsail isn’t the only thing happening across the ARK Ecosystem. There have been new releases of other products like ARK Scan and over 22 new exchange pairs supporting ARK.

This will be our last report of the year, but we will continue to watch into the new year and revisit again in 6-12 weeks for another full report.

Be sure to follow ARK on Twitter to catch any releases or news.

Follow Strake Foundation on Twitter for news, exchange listing events, announcements and more.

Check out the ARK Community Discord, Telegram and Reddit for discussions, live GitHub feed, announcements and more.

This article was put together using official ARK SCIC posted blogs, anything outside of quoted text from official sources is strictly the opinion of The Strake Foundation. If you like these articles and reviews we would appreciate a vote for our ARK validator, delegate strakefoundation, to help us continue to inform and support the ARK community. If you would like to learn more about ARK and validator/delegate voting and staking, please read our Staking Guide.