The State of Mainsail VI

Part six of our ongoing new ARK Core development summary.

In this post we will cover and attempt to quickly summarize ARK’s ongoing work on the redesigned and renamed codebase called Mainsail. Our previous 5 posts regarding Mainsail highlighted the first 42 weeks of development. This post will cover the 16 weeks since then.

Mainsail is the name of the newly redesigned ARK Core codebase currently in development. After an announcement earlier last year to rename and completely rebuild ARK Core, we have been able to follow live GitHub commits and detailed development reports of all progress on Mainsail to date. 

This post focuses on the technical aspects and GitHub commits of Mainsail, but if you look back over the past 16 weeks there has been a significant amount of progress across the entire ecosystem. With product updates, releases and commits across multiple repositories.

Shortly after our last post, the team released the new Mainsail testnet. It is currently in public testing and anyone can jump in to help run a node and test the new Mainsail network. Just join the ARK Community Discord and ask how for all the resources and help you need.

So lets jump right into the technical development summary from where we left off last time.

The week of March 8th, 2024

This week, the team released the initial public version of the Mainsail testnet network, marking a significant step forward and a milestone in its development journey.

The specific details of the week’s progress are as follows:

Mainsail

• Fixed active validators calculation in the validator-set-vote-weighted package and corrected the vote balance calculation.

• Added support for running validators in Mainsail Docker images.

• Refactored transaction schemas: now, the default allowed amount is 0, with amount deduction and schema changes becoming responsibilities of transactions that use the amount field (e.g., transfer, multi-payment). This resolves the reported multi-payment burn issue.

• Added the env:paths:clear command to mainsail and mainsail-api, enabling the clearing of common environment paths such as data, config, temp, logs, as well as specific paths like plugins and state exports.

• Implemented a limit of one username registration per sender in the pool.

• Enhanced logging with a list of loaded validators, reporting active, stand-by, resigned, and unknown validators registered in validators.json based on the latest known state.

• Improved documentation based on feedback and reported bugs.

Mainsail EVM

• Introduced support for the initial evm_call JSON-RPC method, enabling users to call deployed EVM contracts like ERC20.

• Enhanced EVM error handling for the transact method.

• Implemented a transaction handler context providing the wallet repository and EVM instance to the transaction handler. The transaction pool and collator utilize an EVM mock that doesn’t run bytecode, while the block processor employs a real EVM instance.

Moving into the next week, the team said they’ll continue monitoring the public testnet, addressing any arising issues promptly. Additionally, working on a new crypto package demo for browsers (for integration into ARK Vault), transition from CommonJS to ESM modules (ECMAScript), address reported issues related to validator resignation and API responses, and continue on the EVM PoC implementation. Full report here.

The week of March 15th, 2024

This week, the team released a new version of the Mainsail testnet, addressing issues highlighted by testers. The updates in this new release included:

• Enforcement of a single username registration per user in the pool.

• Logging of registered validators on the local node during boot (specifically for validator nodes).

• Implementation of a new method called sizeOfIndex in the WalletRepository. This method accurately determines the index size on both WalletRepository and WalletRepositoryClone by computing the size variances between the original and clone repositories. It’s particularly useful in the validator resignation transaction, resolving resignation-related issues.

In addition to a new testnet release, the team also worked on these items:

• Mainsail has undergone a complete transformation into ES Modules (ESM). This change facilitates integration with the ARK Vault. As part of this overhaul, a new package called test-runner has been extracted.

• All packages have been updated to their latest versions. Before the migration to ES Modules (ESM), this task was challenging because some of the latest versions of third-party packages were already in ESM format.

• The transaction poll processor now returns TX indexes as numbers, replacing the previous string format. This alteration is reflected in the transaction-pool-api response.

• Numerous fixes have been implemented in mainsail-api entrypoint, API responses, and configuration generator.

Looking ahead, the teams plan is to focus on preparing the browser package for mainsail crypto, as it’s essential for ARK Vault integration. Additionally, they’ll continue monitoring the testnet network and addressing any reported issues that arise. Full report here.

The week of March 22nd, 2024

During this week, they rolled out a new version of Mainsail for testnet, incorporating several fixes for bugs reported by the community, alongside additional enhancements including:

• Transitioned fully from CommonJS to ESM (ECMAScript Modules).

• Updated nearly all dependencies to their latest versions.

• Addressed issues with wallet responses on the API.

In addition to these updates, this weeks work also encompassed:

• Implementation of several ESM-related fixes.

• Execution of mainsail and mainsail-api commands with a .js extension. This extension is also employed in the *:start commands.

• Correct loading of the package.json via the PluginManifest class.

• Resolution of issues with the test-framework build, ensuring proper output by listing the workspace project in dependencies.

• Adjustment of fees calculation on /api/fees to exclude the genesis block.

• Management of static fees through milestones, visible on the /api/node/configuration endpoint.

• Activation of the api-sync package to sync the active milestone each time a milestone is altered.

• Enhancement of transaction responses on api-http to include additional data akin to V3.

The team said the next weeks focus would remain on addressing any reported issues from community members on testnet. Additionally, aiming to enhance their testing suite and prepare scripts for functional tests on the testnet. These tests will serve as the initial batch for more targeted testing of specific scenarios that may arise on the network. Full report here.

The week of March 29th, 2024

During this week, the Mainsail team addressed reported issues from the testnet while also developing additional tests for upcoming targeted testing and implementing further optimizations and new features.

The specific details of the week’s progress are as follows:

• The install script has been fixed to accommodate the new paths naming convention.

• API nodes are now determined using URLs instead of IP & port pairs. This enables users to configure API nodes accessible via domains and concealable behind proxies.

• The transaction pool processor now accurately captures and reports transaction errors, with errors reported back on the post-transaction endpoint.

• A new package, consensus-storage, has been extracted from consensus, containing logic for storing open proposals, prevotes, and precommits.

• The database has been refactored to return commits instead of blocks. A commit is a structure that holds a block along with aggregated signatures and a list of signers.

• The transaction pool database now supports in-memory mode.

• The configuration generator now includes static fees in the milestones.

• The logic responsible for registering listeners for shutdown signals has been moved outside the application into a custom bootstrapper. This feature enables testers to skip registering listeners in the tests.

• The TSM TypeScript loader has been replaced with TSX, allowing the execution of ESM code in the tests. Consequently, rewiremock has been replaced with esmock (ESM-compatible dependency mocker), and all CommonJS-related code like the __dirname statement has been removed.

• The run hook in the test-runner has been fixed. Errors thrown in before or after hooks are caught and properly handled to result in a failed test.

• Support for functional tests has been added, including the ability to run multiple Mainsail instances in a single node process. Additional features have been incorporated to aid the testing process:

  • In-memory replacements for the database and storage are provided, along with custom configuration loaders allowing nodes to run without accessing the disk.

  • Custom P2P logic facilitates communication between instances and can be customized to simulate various scenarios such as message delay, skipped messages, targeted broadcast, and data loss.

  • A Test ProposerSelector allows tests to run with a predefined proposer, improving determinism.

  • The Test Logger logs the instance ID in the messages.

  • A Test Worker performs crypto calculations on a thread without booting subprocesses.

  • Assert and helper methods such as snoozeForBlock, assertBlockId, assertBlockRound, and assertCommitValidators are included.

  • Sandbox helpers for configuring and booting nodes are provided.

• Functional tests have been added for the transaction pool covering all transaction types, including most accept and reject scenarios.

The team stated the following weeks focus would be on preparing tests for various consensus scenarios (double forging, malicious validators, network fragmentation, etc.) and adding additional tests using multisig wallets and blocks. Full report here.

The week of April 5th, 2024

This week, the team focused on preparing a new testnet release addressing reported issues from both delegates on the testnet and their internal team. The new testnet version included the following updates:

• Excluded the genesis block from fee calculation.

• Implemented the management of static fees through milestones.

• Synchronized milestones to the API database upon any changes.

• Enhanced the milestones generator to include static fees.

• Resolved issues with minority and majority calculation.

• Introduced a new transactions/fees route.

• Enabled the allowance of 0 nonce in queries on the /wallets endpoint.

• Expanded test coverage with additional tests.

The team also worked on an updated release of ARK Scan exclusively for the new testnet. Stating their focus will remain on addressing any reported or noticed issues, while also continuing to prepare all necessary scripts and test cases for further testnet-targeted testing. Additionally, implementing checks to verify transactions included in blocks and commence internal performance testing. Full report here.

The week of April 12th, 2024

This week, the team introduced a new testnet release, version v0.0.1-alpha.15, featuring various fixes and enhancements:

• Consensus now emits events for:

  • New round

  • Accepted proposal

  • Received majority prevotes or precommits

• LMDB has undergone additional refactoring in LMDB transactions.

• The config:publish:custom command now supports the --overwrite flag, enabling updates to individual configuration files.

• JSONB attribute sorting support has been added to the public API.

• Browser support for the bls12-381 package has been implemented.

In addition to these updates, progress was also made on other tasks:

• Improved ban logic for P2P, introducing the ValidateIP plugin executed before any message deserialization.

• Added /peers/banned endpoint to the development API.

• Corrected response schema and protobuf for the postTransactions endpoint on P2P.

• Modified API responses to display the meta property followed by data for enhanced readability.

• Implemented default sorting of validators by round in the /delegates endpoint.

• Enhanced logging by including round along with height on block commit.

• Conducted tests covering various scenarios for prevotes and precommits in consensus functional tests.

• Tested rejection of invalid blocks in functional tests.

Looking ahead to next week, the team stated their focus will remain on expanding test case scenarios, particularly conducting end-to-end tests to confirm P2P functionality, and addressing fixes and performance improvements based on the results of performance tests. Full report here.

The week of April 19th, 2024

This week, Mainsail rolled out a new testnet release, version v0.0.1-alpha.16, with several fixes and improvements:

• Corrected the postTransaction schema and proto on P2P.

• API response now displays meta property before data property.

• Updated default sorting for /delegates endpoint.

• Added logging for commit rounds.

• Moved proposal signature verification to the main thread from the worker.

• Eliminated redundant block generator check.

In addition to the testnet release updates, progress was made on various other tasks:

• Implemented end-to-end (E2E) tests and optimized workflow processes.

• Conducted further performance tests to ensure optimal system functionality.

• Implemented partial proposal deserialization to enhance broadcast performance.

• Modified RoundStates to run pending processes only once.

• Added an option to encrypt validator secrets with a password.

Next week, the team stated they will continue addressing any reported issues by delegates and testers on the Mainsail testnet network. Also stating to dedicate efforts to testing Mainsail’s overall performance and developing additional tests to simulate real-life scenarios that could occur on the live network. Additionally, continuing their work on integrating the EVM into Mainsail. Full report here.

The week of April 26th, 2024

This week, the team concentrated on resolving bugs identified by the community and our internal testing, while also implementing a range of enhancements. Here’s a summary of the progress:

• We’ve optimized the proposal deserialization process into two steps. Initially, we deserialize the proposal header and conduct basic verification before broadcasting. Subsequently, we deserialize the block and other proposal data. This method significantly reduces the delay between the reception and broadcast of proposals.

• We’ve minimized the number of RoundState checks in consensus by implementing pending jobs. If an identical RoundState for the same height and round already exists, it won’t be added to the pending list.

• All emitted events now include block data, ensuring accurate dispatch events for blocks.

• We’ve upgraded pnpm-lock.json to pnpm version 9, enhancing package management within the project.

• Added support for nested key comparison in Webhooks.

Looking ahead to the next week, the primary focus should be on performance improvements, along with continued efforts towards integrating EVM within Mainsail. Full report here.

The week of May 3rd, 2024

This week, the efforts were dedicated to addressing community-reported issues and advancing the work on EVM integration. Here’s a detailed overview:

• Ensured proper restoration of webhooks upon boot, ensuring they are correctly reinstated when users restart the Mainsail core process.

• Implemented enhancements to functional tests to improve overall reliability and efficiency.

• Deferred finalizing state changes from transaction processing until commit as part of the ongoing EVM integration.

Looking ahead to next week, the team stated focus will remain on enhancing performance based on internal testing results, fixing any reported community issues and progressing with the EVM integration. Full report here.

The week of May 10th, 2024

This week, Mainsail rolled out a new testnet release, version v0.0.1-alpha.17, with several fixes and improvements:

• Support for validator secret encryption (BIP38)

• Improved performance by deserializing proposals in multiple steps

• Consensus performance improvements

• Correct dispatch of block events

• Restoration of webhooks on boot with added support for nested keys

In addition to the testnet release mentioned above, progress was made on:

• Improving the stability of functional tests

• Fixing api-sync update version and crypto configuration data during boot

• Adding the syncInterval option instead of using block time to periodically sync non-block-related data to PG

• Introducing the blockPrepareTime milestone option, which adds extra time for block preparation during long consensus cycles

• Splitting EVM transactions processing into the process and commit steps

Next week, the team stated they will be monitoring for any reported issues that may arise, as well as extracting the transaction pool into a separate process. Additionally, investigating EVM state persistence to LMDB. Full report here.

The week of May 17th, 2024

This week, the teams work focused on further improvements to the consensus and other parts of the codebase, as well as the EVM database integration. Here are the details:

• Insert transactions in chunks to PostgreSQL to support very large blocks.

• Refactor consensus to prepare a block after commit instead of waiting for a new round.

• Dispatch events for received +2/3 of prevotes or precommits only once per round.

• Support query criteria for the public API on wallet attributes.

• Add EVM database using the heed library for LMDB.

• Create a transaction-pool package similar to API, implementing CLI commands for common process actions (start, stop, run, logs, update, etc.).

• Add initial JSON-RPC implementation for transaction-pool-server and transaction-pool-client. The client will run on the main Mainsail process and communicate with the server on the transaction-pool process.

• Remove support for transaction broadcasts on P2P.

Next week, the focus should be on syncing state to the transaction-pool using state deltas, implementing the remaining endpoints on tx-pool-server (bootstrap, sync-delta, getTX, removeTX, syncPeers, getStatus), and benchmarking and improving EVM integration. Full report here.

The week of May 24th, 2024

This week, the Mainsail team added a few new endpoints to the API and continued work on the EVM. Specifically, the work done this week included:

• Added the /rounds/{round} endpoint that allows querying rounds by round number on the public API.

• Enabled validator registrations from multisignature wallets.

• Modified the state to allow the export and import of delta changes between blocks. These changes are used to sync the state between core and transaction-pool processes.

• Added the skipUnknownAttributes option to the state. This skips attributes that are not registered on the transaction-pool process, as we only need transaction-related attributes. For example, the ActiveValidators list can be skipped.

• Added additional transaction endpoints to the transaction-pool API:

  • transactions/unconfirmed -> shows paginated unconfirmed transactions.

  • transactions/unconfirmed/{transaction_id} -> returns a specific unconfirmed transaction.

  • transactions/types -> returns registered transaction types.

  • transactions/schemas -> returns registered transaction schemas.

• Made the transaction ID a mandatory field.

• Implemented client and server logic for JSON-RPC endpoints on the transaction pool:

  • get_transactions -> returns a list of transactions that can be included in the block. This considers max transactions, block size, and expired transactions.

  • commit -> syncs block, failed transactions, and state delta changes.

  • list_snapshots -> lists local snapshots on the transaction pool.

  • import_snapshot -> the core process determines which snapshot should be restored on the transaction pool.

• Refactored the EVM database persistence logic to improve performance.

• Matched the evm.mdb format to the existing LMDB layout. Database and lock files are on the same directory level.

Next week, the team stated to focus further on the EVM implementation, specifically on gas and fees for it, as well as work on implementing get_status on the transaction pool and handling communication errors. Full report here.

The week of May 31st, 2024

This week, Mainsail rolled out a new testnet release, version v0.0.1-alpha.18, with several fixes and improvements:

• Upsert Mainsail version in PG on bootstrap.

• Add blockPrepareTime to bootstrap and refactor timestamp milestones.

• Add syncInterval option to api-sync instead of relying on blockTime.

• Insert transactions in chunks on very large blocks.

• Prepare block after the commit, instead of waiting for the round to start.

• Dispatch consensus events only once per round.

• Support query criteria on wallet attributes.

• Support query round by number.

• Accept validatorRegistration from multisig wallet.

• Expose validator round votes on the /rounds/{id}/delegates route.

In addition to the testnet release mentioned above, progress was made on:

• Skip API sync logs on bootstrap.

• Implement getStatus JSON_RPC method on transaction-pool.

• Add response and request schemas for JSON_RPC actions.

• Always import to empty state on snapshot restore.

• Replace child processes with worker threads. Threads have faster boot time, don’t run extra V8 instances, and have smaller communication overheads.

• Refactor transaction pool to run in a worker thread.

Next week, the team stated they would be working on replacing JSON_RPC with IPC calls for the transaction-pool and conducting performance testing with a separated transaction pool. While also continue working on EVM implementation, focusing on algorithms for gas calculations and their implementation. Full report here.

The week of June 7th, 2024

The Mainsail team made progress this week on the general codebase and process improvements, with initial work done for the gas for EVM. This week’s work included:

• Disabled snapshot export on TX pool worker.

• Replaced JSON_RPC with IPC actions.

• Supported event emits from worker to main process.

• Fixed E2E and functional tests.

• Implemented transaction broadcaster and TX pool node verifier.

• Merged the latest changes to the EVM branch.

• Implemented initial gas fees for EVM.

Next week, the team plans to add support for logging from multiple processes and make general performance and EVM improvements. Full report here.

The week of June 14th, 2024

This week, the team worked on general improvements as well as further EVM integration. To go over work done:

• Fixed API process boot by making crypto-worker and transaction-pool optional properties in app.json.

• Awaited proposal creation at the beginning of the round to handle large blocks that might take longer to forge.

• Reduced thread blocking on block deserialization.

• Fixed bootstrap by setting processor result.

• Used ephemeral EVM instance when collating transactions, skipping transaction caching.

• Provided block context to EVM consisting of commitKey, gasLimit, timestamp, and generatorPublicKey.

• Stored TX commit receipts with fields: gasUsed, gasRefunded, success, logs, and output.

• Synced TX receipts to PG database.

Next week the team stated they’ll be focusing on performance improvements based on internal observations on testnet as well as general EVM improvements. Full report here.

The week of June 21st, 2024

This week, the team continued working on transaction pool improvements, log optimizations, and EVM integration. The work completed included:

• Fixed bootstrap by setting the genesis block after restore, as restore creates a clean state.

• Improved block creation performance by skipping the deserialization step.

• Increased P2P timeouts for consensus messages.

• Implemented a time limit on transaction collation, using 75% of the block time for collation and the remaining time for data serialization and proposal signing.

• Enhanced log readability by including the worker name and removing extra logs on the transaction pool worker.

• Added a check for block timestamps from the future in the block processor.

• Re-added transactions to the pool on boot and milestone changes.

• Re-added transactions to the pool when the sender state is invalid. State recalculation is necessary when removing a transaction from the middle of the sender stack instead of removing them in nonce order.

• Added a /receipts endpoint to obtain EVM transaction receipts.

• Fixed payload length calculation by using uint16 to represent transaction length.

• Enabled dynamic resizing of the EVM database.

• Upgraded to the latest revm version.

Next week, the team stated they’ll be working on adding webhook support for worker threads and continuing EVM integration and improvements. Aiming to also prepare a new release for the initial testnet, incorporating all the recent features, optimizations, and fixes they’ve been working on. Full report here.

---

As you can see, the work on Mainsail is quite continuous with 337 reported commits over the last 16 weeks. This may not seem like a lot in comparison to other projects or products but remember quality of commits > than quantity of commits.

One of the most exciting aspects of the current Mainsail development is the EVM integration and development in the internal testing arena. We are waiting patiently for more info on this and what direction it may be heading.

Of course Mainsail isn’t the only thing happening across the ARK Ecosystem. There have been hundreds of commits across multiple other repos for other products and multiple new releases of other products like ARK Scan, ARK Vault and ARK Connect, as well as ARK Core updates and new exchange listings.

The Mainsail testnet is currently running smooth and there is room for more node runners that wish to help in anyway they can. If you are interested at all, join the ARK community discord and check #mainsail_testnet channel.

---

Be sure to follow ARK on Twitter to catch any releases or news.

Follow Strake Foundation on Twitter for news, exchange listing events, announcements and more.

Follow ARK on CoinMarketCap for news, exchange listings, announcements, articles and more.

Check out the ARK Community Discord, Telegram and Reddit for discussions, live GitHub feed, announcements and more.

This article was put together using official ARK SCIC posted blogs, anything outside of quoted text from official sources is strictly the opinion of The Strake Foundation. If you like these articles and reviews we would appreciate a vote for our ARK validator, delegate strakefoundation, to help us continue to inform and support the ARK community. If you would like to learn more about ARK and validator/delegate voting and staking, please read our Staking Guide.