Successful [Factom Inc-015] Core Development

Was Factom Inc. Protocol Development Grant Round 2, 2019 Successful?

Have not voted

Guides Brian Deery

Authority Nodes Canonical Ledgers Canonical Ledgers

  • Total voters
  • Poll closed .
Factom Inc. will continue to provide core development to the Factom Protocol

Grant Proposal

FCT received

Factom Inc. -- 34694 FCT
Sponsor 1, Dominic Luxford -- 300 FCT
Sponsor 2, Nolan Bauer -- 300 FCT
Sponsor 3, Factomatic -- 300 FCT

Start Date
June 1st, 2019

End Date
September 1st, 2019
This update covers the 3-month protocol development and network maintenance Grant awarded to Factom Inc for the period Jun 9, 2019 - Sep 9, 2019.

The grant has 2 goals:
  1. Respond to any issues impacting the network (as priority) to ensure the network continues to operate.
  2. Refactor Code Initial Development

Network Pauses

Since the beginning of the Grant, the network has experienced one major pause on July 30th. Factom Inc. responded in a timely manner and following two reboots of the network, the problem was resolved. At the last Sponsor meeting, Steven Masley detailed initial findings on the outage, in particular that two separate (and very likely not related) problems have been observed. One of the problems occured when an Authority Node acknowledged an entry reveal before acknowledging an entry commit. This triggered a series of elections, where a second issue prevented the elections from completing successfully. Both issues are currently under investigation and we are expecting a more detailed report.

We believe that the incident was well handled and that Factom Inc. is dedicating the necessary resources to further investigate the issue. In addition to this, Factom Inc. has access to a detailed log of what was happening on the network at the time of the incident, so this should further aid the investigation.

Based on these, we deem that the first goal of this grant has so far been accomplished.

Factomd Development

The second goal is about a major ongoing refactoring of the factomd codebase. As part of this grant, Factom Inc. has compiled a document detailing the short- and longer- term goals of the refactoring. This work is a precursor to sharding.

Starting Jun. 9, two releases of factomd have been completed: Bond (v 6.3.2) and Parchment (v 6.3.3). Bond is currently running on mainnet, while Parchment has been available since July 26th and some ANOs have already upgraded to it.

Bond has been primarily a bugfix release and as such does not directly pertain to the refactoring effort. However, it bundles other improvements to the protocol, including decreased memory requirements for booting factomd (4 GB vs 8 GB in prior releases), more robust election handling and a fix for the double ACK bug, which caused a network pause following the 6.2.2 release.

The Parchment release includes 38 individual tickets and addresses several important pieces of the refactoring work, as well as numerous optimizations and stability fixes. We highlight some of the main work that went into this release below:
  • Dependent holding is a significant refactor of a central component of factomd -- the holding queue. To simplify, the holding queue is a buffer, which contains unprocessed events. Up until Parchment, the holding queue did not have an understanding of dependencies between events and was "scan-based", i.e. whenever a new event arrived, all other events had to be scanned in order to decide if any of them has to be processed. With the introduction of dependent holding, the holding queue was changed to understand the orders of operations on events in order to increase performance as well as to prepare for sharding.
  • Opportunistic entry writing is an improvement in the way entries are written to the block. Prior to Parchment, entries were written at the end of block, which resulted in delays at block boundaries. With Parchment, writing of entries is now distributed throughout the 10-minute block formation period.
  • Network stability has been improved through the identification and fixing of a new class of the Pokemon bug.
  • Syncing of the blockchain has been improved through a number of optimizations and bugfixes.
The next scheduled release is Xuan and it is actively being worked on. It will provide further improvements such as more efficient checks for missing messages, reducing network traffic, and more robust elections, among others. Currently, we have no reason to believe that this release will be delayed and it should be out before the end of the current Grant period.

Overall Impression

Our impression is that up until now, Factom Inc. has delivered on the goals & objectives outlined in their Grant proposal, despite some delays with planned releases of factomd.

Significant resources are allocated towards the development of factomd, as evidenced by the GitHub summary. In the last month, 6 people from Factom Inc. have contributed to the repository and there have been a total of 36 pull requests merged into master, with substantial additions to the codebase. With the last two releases of factomd, the performance of the network has also shown a measurable improvement, with load tests on testnet showing an increase in capacity from 3-4 entries per second (EPS) to 15 EPS.

In addition to this, the goal of network maintenance in cases of emergency has been (unfortunately) met as well.

In terms of ecosystem support, members of Factom Inc. have been regularly attending the community stand-ups and have been aiding the on-boarding of core developers from Sphereon and Factable Solutions, as well as participating in design discussions pertaining to important features to be added to factomd in the near future, such as the Live Feed API.

Thus, our impression of the performance of Factom Inc. is positive and we believe it will be consolidated with the upcoming release of Xuan.

Thank you for reading,
Nikola, Nolan & Valentin
Last edited:
Just a week ago the mainnet choked under 3 eps, the load producer had to be stopped urgently. I have a hard time matching your positive report with the reality I am seeing. If there was "measurable performance improvement", does that mean we were below 1 eps before this grant? I am afraid there is either a disconnect with the reality, or that you lost yourself in this technocratic reporting and started missing the big picture and what really matters: a network that works well. I am not saying it's all black or all white, but it just doesn't make sense to me that this report is presented overly positive given the state of the network.
Last edited:
Paul, this report is presented from us in our role as grant sponsors and it is strictly limited to a narrow interpretation of what is deliverable under the grant proposal. It is not an evaluation of what you describe as the big picture and whether we have a network that works well, as these issues are beyond the scope of the grant. We would like to work with you more closely in the future to address some of your concerns with Inc. as at the end of the day, our role is to be a liaison between the community and Inc. related to the grant. Alternatively, you could raise your concerns directly with Inc., or at a Guide meeting, or anywhere else. You could also approach Inc. and ask to be a sponsor in future grant rounds. I would personally love to work with someone with your background if Inc. is awarded a development grant again, and Factomatic and you are both sponsors.

In any case, we stand behind our opinion as expressed in the report.
I am happy to announce that the well respected and active community member @David Kuiper has accepted the role as sponsor for the remainder of the grant period of the [FACTOM-GRANT-Factom, Inc.-015] grant, ending Sep 9, 2019. He will take over from @88mph who stepped down earlier.

David has hit the ground running and joined us in our sponsor meeting this morning.

He will also serve as a sponsor for the [Factom Inc.-18] grant if it passes for the 2019-03 round covering Sep 9, 2019 through Dec 9, 2019.

The grant 018 is being voted on and is not editable, but his FCT address which will take the sponsor payout is FA2FqYZPfBeRWq7fWSFEhassT5zpMQZm8jwus3yWbzeN3PZPWybm
@Nolan Bauer will this grant be opened for success determination before the next grant round (2019 #04) reaches the Q&A phase?
The grant will be opened for success determination subject to Factom Inc's self-report, which has been delayed and is still pending (this was discussed shortly in today's sponsor meeting, which is recorded but yet to be published).

As a side note and given the recent communication in this thread and since this grant has multiple sponsors, I'm requesting that any future questions are addressed to all sponsors.

[Editing to clarify: For some reason I wasn't receiving notifications of the last few messages.]
Last edited:
@factomatic @Nolan Bauer @88mph: Will there be a final report presented by the sponsors where you describe your take on the grant?

The sponsors are in the best position to properly evaluate if the grantees effort are up to par and if the grant work has been delivered as promised.

My opinion is that whenever a grant has sponsors this report ought to provide the standing parties with this insight prior to a success vote commencing.
Final Update

This grant was officially finished on September 9, 2019. This final update contains a summary of the work completed, a timeline of major events, and our suggested score as sponsors.

The grant application is available here. There were two stated goals and success criteria:
  1. Respond to any issues impacting the network (as priority) to ensure the network continues to operate
  2. Refactor Code Initial Development
Factom Inc's overview of this grant period can be found mid-way through their September Report, starting with “September 9th also was the last day of the Factom, Inc Protocol Development Grant”. Their self-determination of this grant is a success.

It’s important to note that Parchment and Xuan were not recommended for update to the Authority Node Operators during this grant period due to issues found during deployment testing. These releases had critical fixes meant to stabilize mainnet, and it was frustrating for all involved that the releases were not able to be deployed. We want to assure the community that Factom Inc worked hard to push out these releases throughout the grant period, and these delays were due to the inherent difficulty of complex software development.

Grant Reports

These in depth reports cover the work completed for this grant period.
  1. Factom Inc Grant Update for June
  2. Factom Inc Grant Update for July
  3. Sponsor Mid-Grant Report
  4. After-Action Report of August 23 Factom Network Pause
  5. Factom Inc Grant Update for August
  6. Factom Inc Grant Final Report

Major Events
  1. June 9, 2019: Start of Grant period
  2. June 19, 2019: Release of Bond, v6.3.2
  3. July 30, 2019: Network Pause + Resolved
  4. August 22, 2019: Release of Origami (Grant Activation), v6.4.0
  5. August 23, 2019: Network Pause + Resolved, Release of Cotton (Emergency), v6.4.1
  6. September 9, 2019: End of Grant period


The following scoring rubric will be used for this grant per Doc 106:

Exceptional (9.0 - 10.0) - Successful
Overachieved (7.0 - 8.9) - Successful
Achieved (5.0 - 6.9) - Successful
Underachieved (2.0 - 4.9) - Failure
Total Failure (0.0 - 1.9) - Failure
As sponsors, we believe Factom Inc adequately completed the stated goals during this grant period and a score of 5 is appropriate, meaning “Achieved”.

We welcome additional reviews over the next 72 hours at which point we'll add the poll for final determination.

Thank you for reading,
David, Nikola, Nolan & Valentin
Per Document 106 - Grant Success Determination Process this grant may now be voted on to determine success or failure.

The full summary can be seen in the post above this one.

The following scoring rubrik will be used for this grant per Doc 106:
Exceptional (9.0 - 10.0) - Successful
Overachieved (7.0 - 8.9) - Successful
Achieved (5.0 - 6.9) - Successful
Underachieved (2.0 - 4.9) - Failure
Total Failure (0.0 - 1.9) - Failure
This poll will be open for five days.
Last edited: