Question for Stamp-IT from Valentin Ganev

Chappie

Factomize Bot
@Stamp-IT,

@Valentin Ganev asked the following question:

We realise at current price levels a 50% efficiency definitely does not bring a hefty revenue. However, there are a number of ANOs who are operating at the same efficiency and have provided more value to the protocol in our view.

Can you elaborate if you're working on any Factom-related projects at the moment? Stamp-IT's tagline indicates so, but we are unaware of any publicly available updates regarding those.
 

Miguel Proulx

Stamp-IT
Hi @Valentin Ganev

I would like to provide you with a brief overview of what was done so far by our team for the ecosystem.

We are one of the few teams that invested in its own hardware. We do not rely on any cloud provider for our factomd nodes. This means we have to do more work more work to deploy and maintain our virtualization environment in addition to our nodes. We believe decentralisation is a key aspect of Factom protocol (and any blockchain project). The nodes we operate certainly make the Factom network more resilient.

Patrice invested a lot of time producing the initial Stamp-It Brainswap script and even more for the refactored version. Many ANOs prefer to upgrade via "method alpha" in order to keep their federated server when upgrading. We all have been impacted by human errors during brainswaps, which can be measured not only in hours of unavailability for the Factom protocol, but also in lost sleep hours during the network pauses that resulted. Our Brainswap Script allows "method alpha" upgrades in a safe way, which ensures that the protocol continues to record entries, and that ANOs can sleep at night. We did not receive any grant funds for this. All our work on the Brainswap Script was done for the benefit of the ecosystem, without asking for compensation, and we want to continue to maintain this script.

Jimmy has been advocating for Factom to multiple parties. The truth is, as we already stated in our reports, integrating blockchain to existing infrastructure is not “hot” right now. At least where we are from, blockchain is highly misunderstood and a lot of time has to be spent on educating the different potential users. This process takes time and requires a lot of money to speed up.

While not amending or writing documents, mainly due to language barriers, I have been contributing in most governance discussion. While I do not write up as eloquently in english as some of you, I think I have been active in most topics. Knowing the amount of time this requires, I am not surprised to see how many teams are more and more disconnected with it. Also, to my knowledge, we participated close to all votes.

Considering the time and money we invested, our journey in the Factom ecosystem has been far from profitable. We believe we are more than an infra ANO. Nevertheless, it seems quite a few standing parties believe it is still not enough.

For the foreseeable future, we want to stay implicated and invested in the future of Factom.

We can only do so if the standing parties support us. We ask you, how much are our efforts worth to you?

Which level of efficiency would make you comfortable giving standing to Stamp-IT?
 

Valentin Ganev

Factomatic
Thank you for the elaborate response, Miguel!

I want to start by saying that we are using your brainswap script and we find it very useful. We are also aware that you're working on a next version of the script, judging by your GitHub account. We also appreciate the efforts Jimmy has put into promoting the protocol and we take your word for this, despite the lack of tangible output that can be evaluated.

That said, we feel that you are a little short of contributions, when we compared against other ANOs operating at 50% efficiency:
  • Factoshi -- factoshi.io is a significant development effort and widely used
  • Federate This -- Off-Blocks has a lot of potential and is a significant development effort & investment. Rene is a member of the Core Committee
  • Canonical Ledgers -- In our view are doing work on FAT, which is outside the scope of their awarded grants. Sam developed a Python client for FAT and is also active in governance discussions
  • Bedrock Solutions -- Parts of MFW were developed outside the scope of the awarded grant and they have also maintained the necessary infrastructure for what we believe is a very important piece of the ecosystem. Jay has also made contributions to the setup of OpenNode outside the scope of the maintenance grants.
  • VBIF -- Nolan has made some contributions to the Python Factom client library and they have also done a lot of promotional work for the protocol, which we find is more visible than the promotional work done by Stamp-IT.
  • Factable Solutions -- Michael is very active in governance and is also making contributions to factomd
  • Consensus Networks -- Very active in governance and are developing HealthNet as part of their SBIR grant
That said, I was not aware of the fact that you have invested in your own hardware for the ANO servers. This is not a small investment and can be very useful for the purpose of decentralisation, if done right.

Would you or @Patrice Lacroix be willing to give a high-level overview of your setup, in particular:
  • are you operating both your mainnet Authority Set nodes and the follower(s) on physical machines?
  • are you operating any testnet nodes and if yes are they also running on physical machines?
  • what is the specification of the servers (how many cores, RAM, etc.)?
  • where are you storing the servers?
  • do you have any redundancy in terms of internet providers?
  • do you have any redundancy in terms of electrical supply (e.g. are you using uninterruptible power supply)?
  • are you using those servers purely for running factomd, or are they also being used for non-Factom related tasks?
 
Last edited:
Hi @Valentin Ganev,

I can certainly answer your questions:
  • Our authority nodes and followers all operate on our own physical hardware. We also have two more followers in the cloud that we could use as backups in extreme situations, but they are stopped most of the time because we have enough redundancy
  • Our two testnet nodes also operate on our own hardware (different virtual machines on the same servers)
  • We have 3 identical physical servers with 64 GB RAM, an Intel Xeon CPU with 8 physical cores (not threads) and redundant power supplies. They can easily run a mix of two factomd node (mainnet and/or testnet).
  • Our servers are located in two different canadian data centers, 200 km apart and not operated by the same entity. One of them is a certified tier 3 data center, which mean they can do maintenance of their electrical systems (generators, UPS, etc) without affecting the service. The other one also provides UPS and generators. We make sure that, when not doing maintenance, our two authority nodes run in different datacenters.
  • We do have redundant Internet connection on one physical server, but in our tests, when we spread factomd connections on two Internet connections, factomd did not handled the loss of half its connections at the same time very well... So essentially, we operate as if each node had a single Internet connection.
  • Our physical servers are connected to UPS, which are themselves connected to both the power grid and to a generator.
  • Our physical servers are 100% dedicated to running factomd nodes.
 
Last edited:

Anton Ilzheev

De Facto
Exchange Working Group
Core Committee
Website Committee
We also have two more followers in the cloud that we could use as backups in extreme situations, but they are stopped most of the time because we have enough redundancy
Hello Patrice!
So this backup servers are not synced with the network?
In case of physical failure of authority set, are you able to fast switch to backup node if it's not switched on and not synced?
Could you also elaborate "enough redundancy"?
 

Valentin Ganev

Factomatic
Anton, the way I understand it is that they have both followers running on the physical machines (which are synced) and followers in the cloud, which are turned off by default, but are getting re-synced every once in a while:

Our authority nodes and followers all operate on our own physical hardware. We also have two more followers in the cloud that we could use as backups in extreme situations, but they are stopped most of the time because we have enough redundancy
 
Anton, the way I understand it is that they have both followers running on the physical machines (which are synced) and followers in the cloud, which are turned off by default, but are getting re-synced every once in a while:
Exactly. I can also mention that the logical volumes on which our virtual machines operate are RAID1. If we were to have an SSD failure, everything would still continue to work normally.
 
Top