Factomize Core Dev Update 3

It feels good to be able to say: I finally understand how factomd and the factom protocol works.

After almost forty calendar days, I have reached a point where I’m comfortable with the internal processes of the code. It’s not a perfect mastery, that will likely take years to achieve, but it’s enough to know how all the pieces fit together, how changing one impacts the others, and most importantly of all, where to look to find out more.

To recap what I have been doing: instead of diving into the code by trying to implement something and learn by doing, I wanted to understand how everything works first and in the process create documentation. It’s a departure from the usual and I had doubts in the beginning. For that reason, I want to thank the community for the opportunity to be able to do this and the support you have shown along the way.

So what’s the verdict? Was it worth it?

I would say yes, it was worth it. At times it felt like making all those charts and documentation slowed me down but in retrospect, it helped solidify my understanding. It’s one thing to fly over a couple dozen of lines and know that everything looks good, and quite another to have to take those lines and try to describe them with words. Look at code, try to formulate it, double check to see if that makes sense, reformulate it, verify.

I will come back to this down the road and re-evaluate to see how this opinion holds up months or years from now.

One thing that surprised me was just how little of the code is about the things I thought it would be about. When working with factomd from the outside and in the community, it’s mostly about factoids, chains, and entries. The codebase is quite different. The peer to peer network is a big chunk but by far the majority of the code is about reaching consensus.

What that means specifically is sending messages between to nodes to appraise each other of incoming data in the processlists, ensuring no data is lost, verifying the data is the same, replay protection, recording the data to disk, and making sure the network stays intact. Factoids and chains are almost an afterthought to that.

What that means is that the factomd really is just a general purpose data protocol. That is something that is not readily apparent without having an in-depth understanding of the protocol. Instead of factoids and chains, you could adapt it to store virtually any kind of data in a secure blockchain format.

So what’s next?

Over the last week, I’ve been feeling that itch to start creating something tangible but I wanted to finish all the documents that I planned first. With the release of the consensus diagram and the IMsg coverage, that is now done.

At the moment, it looks like my next step will be to do some work in the P2P code. (Thank you, Brian and Clay, for all the suggestions) It will be a good time to evaluate the “real world” readiness of the things I mentioned earlier.

And now for the goods:

Consensus and messages are the two areas I spent most of my time on and in retrospect, I should have done the messages a lot sooner than I actually did (last). That’s where all the interesting stuff happens and I have to stress this part: it’s impossible to just understand one part without also understanding the other. The consensus diagram and messages are tightly interwoven and together they are the heart and blood of factomd.

I opted not to make a graphic out of messages because the format really doesn’t lend itself to one (each message is independent) and in text format, it’s also searchable. The process was roughly the same as it was for diagrams, reading the code and trying to preserve the semantics rather than the syntax. I have already talked about that at length in my previous blog, so feel free to check that out to learn more about the process.

While this is the end of what I planned to document, if there’s an area of the source code that you would like me to write about in detail in the future, please feel free to reach out to me via Discord or the Forum.