Factomize LLC

Please note the increasing Efficiency of our operation as the price of FCT increases. We're dedicated to operating at a very high, and hopefully increasing Efficiency so that the Standing Parties can determine what is best for the protocol via grants.
 
Last edited:
Can you provide some context? Thanks
I was hoping someone would ask about this so I can confess my sins :)

My background is online communities and some preceded MySpace and Facebook. Once social networks came around us community owners started to get a little nervous, especially since, especially back then, the profile system for community software was TERRIBLE. I founded a company that created a social network that took all the separate online communities out there and provided them with a single, centralized profile system, universal logins, content sharing, and an efficient means to move between communities. In essence, we created a decentralized Facebook with a centralized profile system where you could go join communities your friends were on or based upon your interest. I successfully raised seed capital for this. Needing to ramp up development, I contacted various VC and was invited to pitch with a few including Sequoia Capital. We met with Roelof Botha and he saw the value of the idea.

However, I also suspect he saw through me. We had a great idea, outstanding developers, and a passionate, driven, but very green CEO (me) who was unlikely to know how to take the company and product to the next level. I made every mistake in the book and made up plenty of my own so that the book had to be rewritten. Looking back now, with tweaks to the business plan and a competent CEO, as the timing was PERFECT, you should have been hearing about my company, not Facebook. The social network we created was acquired (and subsequently run into the ground) so I did ok, but I let down a lot of people. I blew it and it was an incredibly humbling, valuable learning process for me.

My current team for Factomize were all part of this company.
 
Last edited:
As one of Factom's most dedicated followers (invested right after Factom ICO), what do you think the greatest challenge the Factom protocol faces from a business standpoint is? How do you plan to contribute to addressing these challenges?
That's a difficult question as I see a host of monumental challenges ahead that pertain to business. I see regulatory, cultural, strategic, ethical, human, and competitive challenges.

How do I plan to contribute to resolving these challenges? By being a Guide and running an Authority Node, neither of which is how I truly want to spend my time. From a selfish perspective, I am doing fine financially, I have my stack of FCT, and I have a beautiful wife and two young kids. I'd like to be traveling, working in my garden, and passively investing in cryptocurrency as a hobby. But no, Factom has to come along and tug at my heart strings as a way to help make the world a bit better place. There's smarter people out there than me who would be a better Guide if they'd just step up and take on the challenge. There's people out there who could poach the amazing team I've put together for my Authority Node and do a better job. And I welcome both of those scenarios. Until then, I'm going to do my best to, "Further the Protocol".
 
Last edited:
Thanks for your application

NK01)
Nodes will be administered leveraging modern DevOps tools such as Ansible and Rundeck to easily provision nodes when needed as well as perform automated upgrades and brain-swaps while removing the “human error” factor from the equation.
Could you elaborate on the automated brainswaps? How are you going to make sure it is performed correctly for instance?
 
NK03)
With two members being able to operate the servers. How are you going to make sure these members are able to respond 24/7/365? I am not interested in the technical solution, since your application shows that. I am more interested in the human and organizational aspect
 
NK04)
Our team has managed everything from small, single-server web-hosts, to (currently) hundreds of high-performance, high-availability bare-metal servers spread out over multiple datacenters all over the world for the Ad-tech industry that process over 100 Billion transactions per day (and peak around 1.3 million transactions per second).
Could you elaborate on the high-availability part in the above statement? What do you mean with it for instance? What measures (globally) are employed?

Could you think of some techniques/experiences that could be useful in running auth-nodes? (of course not mentioning experience ;))
 
NK07)
You are talking about development as well. I do not see that displayed directly in the financial report. Could you elaborate?
Mike Miller will need to answer the other questions but I can answer this one. Development is covered by, "Brian Gunter Pay" as he'll be performing all of our development work on a contract basis/hourly rate. He will also be on call for emergent server situations whereas Mike will handle normal operations of the servers. I know Brian will have the occasional expense which would be covered by the miscellaneous category. The budget is of course very basic and doesn't go into much detail. It's simply to provide a high level understanding of how Factomize will operate at different FCT price points.
 
Last edited:
NK08)
Is see the salaries of you team members increase with FCT price. FCT price could have influence on the amount of work, but is not guaranteed to be exactly relational to it. Could you elaborate on your choices?
FCT price will not impact server administration. FCT price will have a (somewhat) direct influence on the amount of development work we do until we build up sufficient cash reserves as Brian Gunter will be paid for development on an hourly basis. I say, "Somewhat" because if we're in the middle of a project I want finished and there is insufficient funds to pay Brian, I will dip into our startup capital. I'm committed to providing tools that will be beneficial to the Factom Ecosystem. The invoice from Brian I paid yesterday would showcase that ;)

Having to try and budget while being dependent upon a volatile asset makes me twitchy. The goal is to operate lean and build up sufficient financial reserves so that I can properly budget for expenses and scale my team and operation without worrying that in three months a bear market is going to screw everything up.
 
Thanks for your application

NK01)

Could you elaborate on the automated brainswaps? How are you going to make sure it is performed correctly for instance?
From Mike Miller, Director of Operations:

Obviously it will take some work to automate the brainswaps, and testing of brainswaps will need to be performed on testnet before deploying the script for use on mainnet. Ansible is quite capable of reading variables, and setting config values based on those variables, or failing if certain conditions are not met (For example, the DBHeight is different from what the network is reporting on either node). All these checks can be made via querying the factomd API before the brainswap procedure begins. Ansible is also capable of waiting for a condition to be met (eg: waiting for a certain block to be generated) before proceeding to the next step. Logic will need to be written into the Ansible roles to verify brainswaps complete properly as well, and to fall back gracefully if they do not (if appropriate).

When this is all in place and FULLY tested, I plan on integrating the Ansible scripting into something like Rundeck so that even David could perform a brainswap and upgrade if needed by simply pushing a button.
 
"Even David?"

:mad:
NK02)

Could you elaborate on what you would be monitoring? Monitoring is a rather broad topic
From Mike Miller, Director of Operations:

As it is a broad topic, it will require broad monitoring. Zabbix monitors all the basic things “out of the box”. CPU, memory, disk, bandwidth usage etc. I’ll also be monitoring the factomd application where i can, leveraging Factom’s API calls, and any existing monitoring data (like prometheus/graphana if mainnet deployment is similar to the docker deployment/updates on testnet). Alerts and notifications will be sent to PagerDuty in the case of a catastrophic failure (or something predicting a catastrophic failure, such as the sync falling too far behind), and notifications of changes or potential problems will likely be spit out to a Slack or Discord channel that my team can glance at.
 
NK03)
With two members being able to operate the servers. How are you going to make sure these members are able to respond 24/7/365? I am not interested in the technical solution, since your application shows that. I am more interested in the human and organizational aspect
From Mike Miller, Director of Operations:

I’ve been on call 24/7/365 for the better part of 15 years at this point. It’s part of my life and It does not bother me. I am already on call for hundreds of servers, adding a dozen more is a drop in the ocean. The key to keeping your sanity while being on call like this is redundancy and consistency in the production environment. If something happens, things should be self-repairing (failing over to other servers/nodes via automation, for example), so you don’t have to get out of bed and spend valuable sleep time fixing a problem, it can wait until morning. However, if the worst happens and manual intervention is required, I’m more than willing to resolve the problem when it happens. A secondary person is basically a redundancy in case I go to a move and shut my phone off, or travel through an area with no signal.

There is no reason 2-3 people can’t handle being on call for a dozen servers. If you’re losing too much sleep because of outages something is horribly wrong and you need to re-think your approach.
 
NK04)

Could you elaborate on the high-availability part in the above statement? What do you mean with it for instance? What measures (globally) are employed?

Could you think of some techniques/experiences that could be useful in running auth-nodes? (of course not mentioning experience ;))
From Mike Miller, Director of Operations:

The infrastructure I manage employs Juniper MX routers, SRX firewalls and QFX switches (all redundant) as well as redundant F5 and A10 load balancers in each datacenter with redundant 10Gbit lines from multiple providers (GTT, Level3, Cogent as well as inter-DC peering lines).
The internal network is a mix of Infiniband (capable of up to 56Gbit) and 10Gbit ethernet. Each server has 2 connections on separate NICs bonded (LACP) to two different switches. Everything is redundant from network switches to PSUs. If one server goes out, the load Balancer marks it as offline and no one but the Operations team knows.

Anycast is employed to global load balancing in certain instances as well, to direct traffic to the nearest datacenter. (employing GTM with the F5s and GSLB for the A10s, basically the same thing, just different names). All of this allows us to have a 99.999% utpime SLA. Configuration and consistency management is handled via Ansible, as are application deployments. (I literally had to stop answering these questions halfway through to deploy a new version of our application to 183 servers. It took 16 minutes and I only had to type out one command. During the deploy process, status updates are output to a Slack channel, and the JIRA ticket is updated automatically, and closed when the deployment is completed).

Everything I’ve been able to think of at this point has pretty much been listed in our application and in the answers to the above questions. I’m sure I’ll come up with other ideas and things I want to try as things move forward and as we all gain more experience running Factom nodes. I’ve been administering Linux servers professionally for 20 years, and I still learn something new almost daily, past experience is great but the only past experience one needs is to know they need to keep learning.
 
NK05)

Could you elaborate on said VPN? Type, accounts etc
From Mike Miller, Director of Operations:

SoftLayer provides VPN Access our private network via IPSec or SSL. You can read more about it here if you desire: https://console.bluemix.net/docs/in...-started-with-virtual-private-networking-vpn- Should Softlayer’s VPN not work for our purposes, I have experience setting up OpenVPN gateways, which could easily be done on a small cloud instance. The VPN will be required to access the Auth node, as it will not have a public facing IP address. Access to the servers via SSH will also require VPN access. Users will be set up in Softlayer’s control panel, or via individual keys using OpenVPN.
 
NK06)

Could you elaborate on the firewall and network setup in the mentioned picture?
From Mike Miller, Director of Operations:

Softlayer offers a hardware firewall device that you can place in front of your network to stop any unwanted traffic from getting to your servers. Ideally all of our public traffic would run through it. If it turns out not to meet our needs, I’ll likely have to just employ iptables on each guard node. Either way, only the ports factom requires to communicate publicly will be open on public IPs. All others will be locked down and only accessible via the VPN.
 
Top