Factomatic LLC

David Chapman

Factomize
Thank you for your application. First question:

A. I like your development ideas and overall plans. I can see why you have a 20% efficiency for one node. For both Authority Nodes together, your efficiency remains 20% for each node. Eventually, the Protocol will have 65 separate entities running one Authority Node each and you will have just one. If you’re able to operate your business with the 1st Authority Node at 20%, what are your plans for the extra income of the 2nd Authority Node as you’re maintaining that 20% Efficiency across two nodes?
 
Thank you for your application. First question:

A. I like your development ideas and overall plans. I can see why you have a 20% efficiency for one node. For both Authority Nodes together, your efficiency remains 20% for each node. Eventually, the Protocol will have 65 separate entities running one Authority Node each and you will have just one. If you’re able to operate your business with the 1st Authority Node at 20%, what are your plans for the extra income of the 2nd Authority Node as you’re maintaining that 20% Efficiency across two nodes?
Hey, David! Valentin here. The extra money will go mainly towards ramping up in-house development. Since the initial grants will be hardcoded in the protocol and since in our opinion there is quite a bit of legal ground to be covered for the proper operation of the Grant Pool, we believe it will be at least a few months before grants are given out to Authority Node operators or other entities in order to develop applications on top of the protocol. In the meantime, we would be using the extra money in order to get several developers to help us in the development of the applications we talked about in our proposal. For example, we have a provisional agreement with one front-end engineer in Sofia to assist us with the UI for the Factom Explorer, if we are selected as Authority Node operators. We would be looking to add at least one back-end engineer to the team as well.

We, of course, reserve the right to change our efficiency above 20%. 20% is the minimum and we would be happy to increase it in case:
  • we end up operating 2 Authority Nodes for a long period
  • FCT price increases significantly in the coming months
  • we are able to get a grant for the development of any of the apps we have proposed

Long story short: we don't plan to be simply an Authority Node operator, but active developers of apps on top of the protocol, irregardless of our campaign outcome. The extra money from the initial operation of two Nodes, will be dedicated to in-house development.
 

David Chapman

Factomize
Thank you for your reply. I understand what you're saying and am excited about the development work you can do for the protocol. Where my concern comes in is ramping up operations with revenue that you know will go away at some point. What happens if the network is stabilized quickly and we get an influx of outstanding candidates for Authority Nodes and you lose that second node 60 days after you ramp up operations with its revenue?
 
Hi David. Nikola here. I’m happy to finally “meet” as I have been lurking in the shadows of discord and staying on top of the discussions.

Valentin and I have both pledged to contribute at least $25,000 of our personal finances to the firm to provide a buffer for such possible scenarios. This should be enough to cover the expenses on specific projects and enable us to complete them as we will be hiring developers on a per project contractual basis. Valentin himself is leading the development efforts and can be a buffer himself if we are operating at a loss.

I am traveling all day, but maybe it’s a good idea for me to send you a budget tonight? I am based on the East Coast, so we should be in the same time zone.
 

David Chapman

Factomize
Nice to speak with you as well. You're always welcome to attach more information but please do not publish anything the community can't have access to as these discussions will be opened up at the end of the process.
 

Tor Paulsen

The Factoid Authority
With your stated efficiency of 20%, at what USD price does FCT need to be for your Authority Node operation to break even on a monthly basis (with one server)?
 

David Chapman

Factomize
B. Let's say you get an Authority Node. How it performs will be easy to monitor. However, as part of your campaign you also discuss additional development work which is important since your single node Efficiency is set at 20% AND you plan to eventually file for grants. The Standing Parties will very much want to see what progress you're making since it was part of your campaign. My questions are:

1. How would you communicate with the Standing Parties. Examples would be blog, twitter, Discord, Reddit, etc.
2. What would you communicate and how often?
3. What metrics should be used to gauge your development progress and success?
 
Last edited:

Matt Osborne

Go Immutable
Exchange Working Group
Legal Working Group
Our vision of the protocol is resiliency through decentralization and wide adoption enabled by network effects.
Could you elaborate on how you see Factom's network effects playing out? Thanks
 

Niels Klomp

BI Foundation
Core Committee
Governance Working Group
Thank you for your application

NK01)
In your application you state to have 4 system administrators with a combined experience of 25 year. Could you break this down in members with their respective amount of years running production servers?
 

Niels Klomp

BI Foundation
Core Committee
Governance Working Group
NK02)
Network isolation - we will be using multiple tiers of networks (for different purposes), using firewall rules on the cloud provider level, as well as network policies within our Kubernetes cluster. Everything will be denied by default, and access is going to be granted explicitly.
We will also be leveraging Google’s Kubernetes Engine service, as we plan to orchestrate any potential
deployments using Kubernetes to ensure high availability and fault tolerance of each individual
container.
Could you elaborate on the Kubernetes setup and especially on how this would bring you the fault tolerance you are mentioning?
What if there will be no docker containers for mainnet?

Will you be using scripting or other techniques to create the kubernetes descriptors?
 

Niels Klomp

BI Foundation
Core Committee
Governance Working Group
NK03)
You are with quite a large team. You explained the deferment. What would be the minimal FCT price for your organization to run without a los while operating one node?
 

Niels Klomp

BI Foundation
Core Committee
Governance Working Group
NK04)
Like that you guys are playing with Kubernetes. Wanted to play with it as well, but somehow don't have a lot of time lately ;)

Are there any negatives you could think if when you would be running in production in the future with kubernetes?
 
With your stated efficiency of 20%, at what USD price does FCT need to be for your Authority Node operation to break even on a monthly basis (with one server)?
We estimated our break even for the operation of one Authority node at a price of $1.80-2/FCT. This factors our cloud hosting costs as well as potential and very rare cases where we might need the services of a 3rd party to temporarily maintain the service in case of unavailability of multiple team members (due to sickness or other excessive circumstances).
 
B. Let's say you get an Authority Node. How it performs will be easy to monitor. However, as part of your campaign you also discuss additional development work which is important since your single node Efficiency is set at 20% AND you plan to eventually file for grants. The Standing Parties will very much want to see what progress you're making since it was part of your campaign. My questions are:

1. How would you communicate with the Standing Parties. Examples would be blog, twitter, Discord, Reddit, etc.
2. What would you communicate and how often?
3. What metrics should be used to gauge your development progress and success?
Good question. We plan to be involved in two tracks of development: open-source applications and proprietary software for clients and industry partners. The two will require different means of communication.

For open-source software we will be adopting a very transparent development process, which is common in the space. Standing Parties (as well as anyone else) would have access to our public source code repositories as well as wiki, issue & bug tracking (e.g. using Confluence). We believe that this will be the easiest and most reliable way for anyone to track progress. For all grants, and especially for larger ones, we would be strong advocates of staged assimilation of the funds based on predefined milestones in each grant application. Communication about the successful completion of such milestones will be reported in a dedicated section on our website https://factomatic.io.

For proprietary software we would sadly be limited in the development transparency, due to IP rights, for example. Those applications will be closed source, development will be in private repositories. We would still be reporting accomplishment of bigger milestones for these applications on our application (subject to agreement with clients), but we envision that a big part of the feedback will be coming from the clients themselves.

Overall, we think that it is important to have as transparent reporting as possible and we are happy to work with the Standing Parties, the general community and our customers to define and streamline the process as much as possible.
 
Could you elaborate on how you see Factom's network effects playing out? Thanks
When we talk about the network effects of Factom, we see this playing out on two levels:

First, with a growing ecosystem of developers of applications on top of the protocol, we believe that the protocol and its use cases will be widened and receive wider adoption among multiple industries.

Second, with adoption in some industries, we see Factom quickly becoming a standard for transparency, which will be deemed as competitive advantage and will put “pressure” on other players in the given industry to adopt the protocol (“Honesty is subversive”).
 
Thank you for your application

NK01)
In your application you state to have 4 system administrators with a combined experience of 25 year. Could you break this down in members with their respective amount of years running production servers?
To clarify: we have 4 people able to manage servers with the team, but not all of them are system administrators. Below you can see the breakdown by person.

Ivan - 10 years of experience in running production servers, in most cases triple digit server fleets. My first job was at the biggest telecom in Bulgaria, and I had to manage a production farm of AIX, Linux, HP-UX and even the odd Solaris server here and there. I was unleashed on the billing HP-UX cluster on the second day of the job - and was told to fix an issue with the cluster while the rest of the team was in meetings. That was my first time using a non-Linux UNIX system, I didn’t know how to use vi (and no, vim wasn’t installed on those), and that was the first clustered system that I touched. Good times :)


Mihail - 7 years of experience in managing Windows and Linux servers. Currently, a System Administrator responsible for operations and availability of a 24/7 production data center environment to ensure maximum availability for customers. I perform system administration activities which include monitoring of system resources, system logs, and continuity of operations, managing operating systems and storage management.


Valentin - 5 years of experience in maintaining and monitoring a Debian-based Spark cluster that was used for running data analytics jobs on telecom data, processing over 1TB of data on a daily basis. I was responsible for setting up and configuring the first cluster used at Teralytics, which was later used by over 35 data scientists and data engineers (including myself) for running ad-hoc jobs, as well as for daily ingestion of telecom data. Prior to Teralytics, I was responsible for the development, deployment and monitoring of the web interface to 3Tera’s cloud computing platform for about 2.5 years. The system was used by a variety of customers, including BT.


Tito -- our security expert -- has no experience managing production servers, but has extensive experience with different Linux distributions, including CentOS, Parrot, Kali, Manjaro and will be supporting us with the server maintenance and unscheduled restarts of the system.


We are also currently interviewing three other system administrators with an average experience of 12 years to serve as additional back ups.
 
NK02)



Could you elaborate on the Kubernetes setup and especially on how this would bring you the fault tolerance you are mentioning?
What if there will be no docker containers for mainnet?

Will you be using scripting or other techniques to create the kubernetes descriptors?
I’ll start with your last question - yes, we will be, and, in fact, we already have the code fleshed out and committed to a git repo. We used this code to experiment with the sort of setup before we did the final decision to go that route, and we also did some failover testing as well (to make sure it behaves as expected and that it takes little time to restore the service).

Regarding your second question - we have actually not used the mainnet containers, we created one ourselves. We want to have a deep understanding of how things work, and make sure we have a good grasp on things, and that includes the supply chain as well.

Lastly, fault tolerance. For one, this is a clustered setup, spread across multiple availability zones. We are using a combination of techniques to ensure maximum availability, like GCP’s live migration (thus incurring no downtime when Google have to work on the underlying systems), using a lean OS for our worker nodes (allowing for a short startup/recovery time, lessens the need for frequent updates/reboots), using persistent disks (including frequent snapshots as an additional measure) to avoid having to resync everything from scratch, standby nodes to take over if necessary. The most important aspect of all - having automation perform recovery most of the time, by having made good decisions during the design phase.
 
NK04)
Like that you guys are playing with Kubernetes. Wanted to play with it as well, but somehow don't have a lot of time lately ;)

Are there any negatives you could think if when you would be running in production in the future with kubernetes?
That is a good question and we have indeed considered this scenario. I would say that the main one would be potential incompatibilities between factomd and our setup. It is quite dynamic, and there are hardly any static configs applied. We know how to deal with that, though, so it won’t cause us major problems. We are hoping to further lower the likelihood of this occurring by closely following the development of factomd and providing input, be it in the form of feedback, or pull requests.
 
NK03)
You are with quite a large team. You explained the deferment. What would be the minimal FCT price for your organization to run without a los while operating one node?
We estimated our break even for the operation of one Authority node at a price of $1.80-2/FCT. This factors our cloud hosting costs as well as potential and very rare cases where we might need the services of a 3rd party to temporarily maintain our operation in case of unavailability of multiple team members (due to sickness or other excessive circumstances).
 
Top