Demystifying Factom Management

Recently, there has been a lot of discussion around the topic of ANO and server management. That was a stumbling block for me when I started tinkering with Factomd since the code responsible for on-boarding ANOs happens to spread across different repos. This blog aims to shine some light on the whole situation, detailing what exactly Inc. does when they “onboard” and “deboard” servers, how interested parties could launch their own custom network, and a few other maintenance utilities.

Preface

This blog is aimed at developers and other technically inclined people. I tried to keep it as simple as I could so everyone can follow along but fully detailing all the concepts like cryptographic signatures would result in a novel, not a blog post.

All encryption used in this blog is ed25519, which has a 32-byte private key and a 32-byte public key component. In most cases, they are just written as a hexadecimal string. Identity chains, also written as 32-byte hexadecimal strings, are chain id hashes. Which is which should be apparent from the context.

Overview

The genesis block needs to have a valid signature and for that purpose, Factom uses a “Bootstrap Identity” and “Boostrap Key” for the server creating the genesis block which does not exist on-chain and accepts arbitrary data. Once the genesis block is made, new nodes can be configured to use an identity chain, which lets them sign messages in their name. These identities are created on-chain and added to the identity registration chain.

To add a new node to the authority set, the message needs to be authenticated by the “Skeleton Identity”. If the Skeleton Identity doesn’t exist, it will fall back to the “Bootstrap Key”.

Tools

  • AddServerMessage: The tool to manage the authority set. You need the private skeleton key in order to use it.
  • ServerIdentity: The tool that ANOs use to create, register, and manage their server identities.
  • Sign With Ed25519: A helper utility to sign arbitrary data with a key
  • FER Entry Creator: The tool to adjust the EC rate

This blog is not a guide on how to create or manage your identity, there are existing guides for this already.

Configuring a Node/Network

All the important settings are part of factomd.conf, though not all of them are listed in the default config. Some are not configurable at all. They are split into two sections: what the node uses and what the network uses.

What the Node uses:

NameTypeDescription
IdentityChainIDchain idThe identity chain id that the node wants to use. These usually start with 888888.
LocalServerPrivKeyed25519 private keyThe private key that will sign data for the node. Should be kept very secret.
LocalServerPublicKeyed25519 public keyThe public key of LocalServerPrivKey. This key should be a valid key of the identity chain*.
* Not necessary when bootstrapping a network

Sidenote: If a node has no config file or a non-hex value for the identity chain, it will default to SHA256("FNode0") (38bab1455b7bd7e5efd15c53c777c79d0c988e9210f1da49a99d95b3a6417be9). This is the default BootstrapIdentity for LOCAL networks.

What the Network uses:

NameTypeDescription
BootstrapIdentitychain idThe identity chain id that should be automatically added during the first block to bootstrap the network.
BootstrapKeyed25519 public keyA public key that should be automatically registered with the BootstrapIdentity.
SkeletonIdentitychain idThe identity chain id that controls the management of the authority
ExchangeRateChainIdchain idThe chain that contains Factoid Exchange Rate entries
ExchangeRateAuthorityPublicKeyed25519 public keyThe public key that is allowed to sign entries to the Factoid Exchange Rate chain.

You can specify a custom BootstrapIdentity and BootstrapKey for CUSTOM networks only. Please note that the SkeletonIdentity is hardcoded and not a config setting.

With the following defaults:

NameNetworkValue
BootstrapIdentityMainNet0000000000000000000000000000000000000000000000000000000000000000
BootstrapIdentityTestNet*0000000000000000000000000000000000000000000000000000000000000000
BootstrapIdentityLocalNet38bab1455b7bd7e5efd15c53c777c79d0c988e9210f1da49a99d95b3a6417be9
BootstrapIdentityCustomNetNeeds to be provided by the config file with the setting CustomBootstrapIdentity
BootstrapKeyMainNet0426a802617848d4d16d87830fc521f4d136bb2d0c352850919c2679f189613a
BootstrapKeyTestNet*49b6edd274e7d07c94d4831eca2f073c207248bde1bf989d2183a8cebca227b7
BootstrapKeyLocalNetcc1985cdfae4e32b5a454dfda8ce5e1361558482684f3367649c3ad852c8e31a**
BootstrapKeyCustomNetNeeds to be provided by the config file with the setting CustomBootstrapKey
SkeletonIdentityMainNet8888882690706d0d45d49538e64e7c76571d9a9b331256b5b69d9fd2d7f1f14a
SkeletonIdentityTestNet*8888888888888888888888888888888888888888888888888888888888888888
SkeletonIdentityLocalNet8888888888888888888888888888888888888888888888888888888888888888
SkeletonIdentityCustomNet88888816d408cd0d7b1b28760f3371a40e98dc2e985c28410e781935954afdf3
ExchangeRateChainIdAll111111118d918a8be684e0dac725493a75862ef96d2d3f43f84b26969329bf03
ExchangeRateAuthorityPublicKeyMainNetdaf5815c2de603dbfa3e1e64f88a5cf06083307cf40da4a9b539c41832135b4a
ExchangeRateAuthorityPublicKeyTestNet1d75de249c2fc0384fb6701b30dc86b39dc72e5a47ba4f79ef250d39e21e7a4f
ExchangeRateAuthorityPublicKeyLocalNet3b6a27bcceb6a42d62a3a8d02a6f0d73653215771de243a63ac048a18b59da29***
ExchangeRateAuthorityPublicKeyCustomNetSpecified in the config, disabled otherwise
* TestNet is a deprecated network type, not the factom protocol’s testnet, which is a CUSTOM network with the id “fct_community_test”.
** The private key for this is 4c38c72fc5cdad68f13b74674d3ffb1f3d63a112710868c9b08946553448d26d
*** The private key for this is 0000000000000000000000000000000000000000000000000000000000000000

Onboarding/Deboarding

The process of onboarding is rather simple.

  1. The server operator has to create and register a valid identity
  2. The block the identity is in is completed
  3. Using the “addservermessage” tool, you type “addservermessage -host=<address of a factom node> send a <identity chain id of the server to be promoted> <network skeleton key>”

That will add the identity as an authority server. If you use “f” instead of “a”, it will onboard them directly as a federated note and increase the “fed slots” by one. To remove them, you can use “sendR” instead of “send”. You can also use this to manage the servers, by “promoting” them. You can promote a fed to an audit, or an audit to a fed. Before a node is promoted to fed, you should promote it to an audit first and make sure that it’s configured correctly and sending heartbeats.

So that’s it? It’s that simple? Pretty much, yeah.

However, let’s take a look under the hood. If you run “addservermessage” with “show”, it will print out a curl command you can execute anywhere. For example:
addservermessage show a 8888881570f89283f3a516b6e5ed240f43f5ad7cb05132378c4a006abe7c2b93 4bded4f3620f6c598d710c0f97b4d83c9743d6a4ac28d24612d6c9d1dc6d06ab
results in

curl -X POST –data ‘{“jsonrpc”: “2.0”, “id”: 0, “params”: {“message”:”1601721232ac208888881570f89283f3a516b6e5ed240f43f5ad7cb05132378c4a006abe7c2b9301a997bdaf2a8519fdc227ae8084afe77f24402275f6e3fc9a48ce0bf3760a4f756e408fd774e06042e1d15f1ccf9725480a09fb8c32a873e9354e25a00f3454c907924c2744515acbc73cbb43ef1f3ce9592fde2c554868b6fab7b9991956560b”}, “method”: “send-raw-message”}’ -H ‘content-type:text/plain;’ http://localhost:8088/v2

If you read my blog on decentralizing ANO management, this message should look familiar. The bytes “0x16” are the message type for “AddServerMsg”. “0x01721232ac20” is the timestamp. Following that is the identity we provided: 8888881570f89283f3a516b6e5ed240f43f5ad7cb05132378c4a006abe7c2b93. The next byte 0x01 specifies that it’s supposed to be an audit server. The rest is the cryptographic public key followed by the signature.

(Verify this example on the golang playground)

Changing the EC Rate

Like onboarding, the tool does most of the job for us.

  1. Make sure the factoid exchange rate chain exists (111111118d918a8be684e0dac725493a75862ef96d2d3f43f84b26969329bf03)
  2. Configure the utility and specify the private key counterpart to “ExchangeRateAuthorityPublicKey” and an EC private key
  3. Run the tool
  • Expiration Height: the height the entry becomes invalid if it was never activated
  • Activation Height: the height the price activates (needs to be +/- 6 blocks of expiration height and cannot be more than 12 blocks ahead)
  • Priority: The default priority after a price activates is 0, so this is typically always 1. If you mess up, you can use a higher priority to override a pending change
  • Factoshi per EC: the EC rate

After that, it prints out two curl commands and also calculates the assumed price of FCT so you can doublecheck it before submitting the message. The data is a simple chain entry like all other entries and the content is encoded in JSON, so there’s not much interesting under the hood; you could write these by hand and sign the JSON data. You can look at existing entries in the MainNet chain for examples of how the activation height and expiration height are used.

Example

Let’s assume we want to start a new network that we control. The first step is generating a secure key to use as our network bootstrap identity and skeleton key. We also need to pick an identity chain to use as bootstrap chain, which can be anything we want:

  • Private key: 4bded4f3620f6c598d710c0f97b4d83c9743d6a4ac28d24612d6c9d1dc6d06ab
  • Public key: a997bdaf2a8519fdc227ae8084afe77f24402275f6e3fc9a48ce0bf3760a4f75
  • Bootstrap Identity: 8888881111111111111111111111111111111111111111111111111111111111

We also need identities for our test servers but fortunately, factomd already has 100 test identities built in with known private keys that we can use. You can order factomd to create these by typing gX in the console, where X is a number between 1 and 100, which will also create the registration chain. The entries are paid for by the initial sandbox coinbase, so this wouldn’t work for a random node on MainNet.

For our example, we’ll need two additional identities:

The result of typing “g2”

I ran this test on one machine using temporary RAM-only instances of factomd (“db=Map”), using config files serverX.conf. In server0.conf, we’re setting the server to use the private key we generated above. The other two config files use the first two test identities for their local settings, but the same bootstrap identity and key.

The commands I used to start them were: (the .conf files were in $HOME/.factom/m2/)

factomd -db=Map -customnet=blogpost -blktime=60 -config=server0.conf
factomd -db=Map -customnet=blogpost -blktime=60 -config=server1.conf
factomd -db=Map -customnet=blogpost -blktime=60 -config=server2.conf

That launched a new chain, with server0 as the only federated node, creating blocks every 60 seconds.

Next, I added one server as a fed, one as an audit:

addservermessage send f 8888881570f89283f3a516b6e5ed240f43f5ad7cb05132378c4a006abe7c2b93 4bded4f3620f6c598d710c0f97b4d83c9743d6a4ac28d24612d6c9d1dc6d06ab

addservermessage send a 888888aeaac80d825ac9675cf3a6591916883bd9947e16ab752d39164d80a608 4bded4f3620f6c598d710c0f97b4d83c9743d6a4ac28d24612d6c9d1dc6d06ab

The messages were submitted and can be seen in the processlist:

The result? A shiny new fed node and audit node.

Demoting

Let’s say we want to demote a fed to an audit. Let’s run:

addservermessage send a 8888881570f89283f3a516b6e5ed240f43f5ad7cb05132378c4a006abe7c2b93 4bded4f3620f6c598d710c0f97b4d83c9743d6a4ac28d24612d6c9d1dc6d06ab

It shows up in the processlist just like we were onboarding it:

The result: two audit nodes.

Example EC Price

I wanted to change the EC price to 2 Factoshi per EC. At the time I ran the utility, the block height was 50. Before I could submit anything, I had to fund an EC address and create the FER chain:

factom-cli buyec FA2jK2HcLnRdS94dEcU27rF3meoJfpUcZPSinpb7AwQvPRY6RL1Q EC2DKSYyRcNWf7RS963VFYgMExoHRYLHVeCfQ9PGPmNzwrcmgm2r 1000

factom-cli addchain -n “FCT EC Conversion Rate Chain” -n “1950454129” EC2DKSYyRcNWf7RS963VFYgMExoHRYLHVeCfQ9PGPmNzwrcmgm2r

(The FA address is one of the sandbox coinbase addresses, the EC key is the address with a private key of all zeros)

I configured my FactomFER.conf thusly:

PaymentPrivateKey = “0000000000000000000000000000000000000000000000000000000000000000”
SigningPrivateKey = “4bded4f3620f6c598d710c0f97b4d83c9743d6a4ac28d24612d6c9d1dc6d06ab”
Version = “1.0”

FactomFER.conf

And ran the following:

After executing those commands and waiting until block 54 rolled around, the EC rate was applied.

Backdoor/Failsafe

While going pouring over the code to write this post, I stumbled across something odd: As mentioned above, there is a Skeleton Identity but it’s hardcoded and doesn’t exist anywhere on chain. If the Skeleton Identity is not found, it uses the Bootstrap key as fallback. That’s the one we used in the example and the one that currently exists in factomd.

What I wanted to know: what happens if you create that skeleton identity at a later stage? Does it start using that?

The answer is “yes, but”.

I had to recompile factomd to accept a new Skeleton Identity for CUSTOM networks, for which I used the third built-in identity of “8888887020255b631764fc0fd524dac89ae96db264c572391a3f19fcf0f8991e”. I started it up just like in the example, adding id1 and id2 as audit servers.

After creating id3, nothing happens. Factomd only loads identities that are part of the auth set, which doesn’t happen upon creation.

Next step, I added id3 as an audit server. Once again, nothing happened on the surface. However, in the background, factomd tried to add the identity into its identity manager in the place of the hardcoded skeleton identity. At that point, the original bootstrap key was no longer valid to sign messages, and hence it did not add the server. From this point on, the skeleton identity has control of the network.

After sending another “add audit server” message, this time signed with 3838383838383730323032353562363331373634666330666435323464616338 instead of 4bded4f3620f6c598d710c0f97b4d83c9743d6a4ac28d24612d6c9d1dc6d06ab, I was able to add the node into the auth set.

If the skeleton identity removes itself from the auth set, the skeleton key will revert back to the bootstrap key.

To summarise, someone who has both a) the current bootstrap key and b) the skeleton identity can backdoor the network and override the bootstrap key. I don’t know why this exists but it could be a safeguard to ensure the network can be recovered in case the bootstrap key is compromised. For this to work, Factom Inc. would have to have the identity already created but kept safe in a vault somewhere.

The skeleton identities are:

  • MainNet: 8888882690706d0d45d49538e64e7c76571d9a9b331256b5b69d9fd2d7f1f14a
  • All Custom Networks: 88888816d408cd0d7b1b28760f3371a40e98dc2e985c28410e781935954afdf3

Neither of these chains currently exists and it is not possible for someone else to create these identities (until SHA256 is broken). These keys used to be placeholders, like Local/Test, but they were specifically added January 1st, 2017, so I assume they have been created for this purpose.

Conclusion

Managing a Factom network isn’t that daunting once you know what you’re doing as long as you are aware of the pitfalls. It also highlights just how powerful the skeleton key really is. There are no checks and balances; whoever has knowledge of the key has complete control over the network’s authority set.