Ratification Doc 101 - Removal of ANO from the Authority Set for Cause

Public: Only invited members may reply

  • Viewed BI Foundation BI Foundation Bedrock Solutions Bedrock Solutions Blockrock Mining Blockrock Mining Brian Deery BuildingIM BuildingIM Canonical Ledgers Canonical Ledgers Crypto Vikings Crypto Vikings Cube3 Cube3 DBGrow DBGrow De Facto De Facto Factom Inc. Factom Inc. Factomatic Factomatic Factomize Factomize Factoshi Factoshi Federate This Federate This Go Immutable HashnStore HashnStore Julian Fletcher-Taylor LUCIAP LUCIAP LayerTech Matter of Fact Matter of Fact Multicoin Capital Multicoin Capital Niels Klomp Prestige IT Prestige IT RewardChain RewardChain Samuel Vanderwaal Stamp-IT Stamp-IT The Factoid Authority The Factoid Authority Tor Paulsen VBIF VBIF
  • Not Viewed Syncroblock Syncroblock

Should the document be ratified or amended as specified?

  • Yes

  • No


Results are only viewable after voting.

Timed Discussion

Voting will end:

Secured
#91
I don't think the Core and Code Committee (which has no mechanism to have a meeting in a timely fashion) should be involved in the immediate process. After the fact, well, hard to say about that too. I am on the committee, but if it has ever met, I don't know about it.
Do Core Developers have a mechanism to have an immediate meeting?

At least the Core and Technical Committee is somewhat decentralized. Allowing a single or couple core developers at this juncture to make such a decision is extremely centralized.
 
Secured
#92
I share Matt's concerns about the three month period talked about in 2.1.3. Perhaps, for a three month period, the bar to start the process could be raised, instead of just totally preventing anything from happening. One idea would be to require 1) that the reasons for removal involve new reasons, and not just the same reasons that were brought before. Those previous reasons could be included, but more would be necessary. That would eliminate concerns about double jeopardy. And 2) require the motion to remove be brought by at least three ANOs. That should cut down on spam.

I also trust the judgement of the guides. They should be able to recognise spam as such and ignore it.

Many ANOs, having faced expulsion and survived, will be working hard to make amends and to demonstrate to the community their commitment. This is a demonstration of sanity. Others will engage in various forms of aggression, both overt and passive. We should not have to suffer their emotional underdevelopment for three months.
 
Secured
#93
Do Core Developers have a mechanism to have an immediate meeting?

At least the Core and Technical Committee is somewhat decentralized. Allowing a single or couple core developers at this juncture to make such a decision is extremely centralized.
The Core Developers and ANOs have automated notifications and pages, and we have a network of people with contact information to mobilize within an hour at least to respond to a stall or crisis in the network.

Developers on applications around the world are authorized to take actions to keep websites up and applications up. In this, we are really no different.

Putting a committee between the developers and the network is a seriously problematic decision making structure. If you are to do this, the committee must be structured to handle such a role. We would have to:
  1. Ensure the committee can be mobilized within the hour of a crisis,
  2. Ensure everyone (or at least most) on the committee is a highly qualified developer, with experience managing blockchain networks and other high uptime applications,
  3. Ensure everyone (or at least most) on the committee is very familiar with the Factom codebase, past issue history, the consensus architecture, and all the interactions between nodes, parts of the code, message processing, etc., and
  4. Ensure the committee has enough operational context on its own so they can contribute to the process in the moment rather than just adding overhead to the process.
The goal would be to have the committee ADD to the process, not just require the developers to brief them, only to rubber stamp their suggestions anyway. Otherwise, there really is no point.

Addressing the "centralization" question, today when we respond to a stall as core developers, we pull in all the ANOs, and work with them as we make moves to address the situation. As there are 20+ ANOs, the process is already "somewhat decentralized". And we have a path for onboarding developers (to decentralize the core developers further).

But at the end of the day, a few core developers are going to be pushing buttons. That doesn't mean there are any less than several core developers and constant notifications and interactions with ANOs behind those core developers.
 
Last edited:
Secured
#96
Paul, shouldn't the committee BE the developers and a few other highly competent people? I thought that was the idea behind it.
It is my view that a committee action is a committee meeting, vote and action. Reviewing the process after the fact isn't really a problem. But being involved in an active process, not just reviewing a process, is a really unknown animal to me.
 
Secured
#98
In their more traditional sense, yes. But there's nothing saying a committee can't be involved in active processes.
Like.... how? Often I do work for committees, but I don't attribute that work to the committee, but rather I report to that committee, usually after the fact, on the general directives I've been given. I have gone to committees to brain storm solutions to problems that don't have a quick resolution, or that had a quick resolution that needed a bigger, more complete solution. But real time? Don't have any experience with development by committee in real time.

We could, as developers, just say, "Yeah, we are on the committee, therefore this work was done by the committee!" But I'm not sure that's really fair or honest.
 
Secured
#99
I should have said committee members be involved in active processes.

The somewhat unique structure we've created for "committees" within this ecosystem is one of discussion, consensus, AND action. And in a decentralized system where we want to maintain and even improve decentralization, it seems to be the best means thus far.

I see a Core Committee not just as a means of decentralization but as a means to protect the Core Developer's time and focus. Do you really want me to reach out to Clay Douglass every time I think there's an issue? Or do you want me to reach out to a committee which he is a member of (but likely not too active) where the competent people there can take my information and decide what action, if any to take. And those same competent people will know that in a highly emergent situation, their primary job is to get the heck out of the way and communicate for the Core devs when they have a chance to come up for air.

And then of course there's the defining of all the processes related to Core development. Do you really want Clay having to deal with a Bug Bounty program? A Factom Improvement Proposal? Etc? No. But you may want him to give a little input once the draft processes are developed.
 
Last edited:

Chappie

Timed Discussion Bot
Secured
AdamSLevy has made a motion to extend the discussion. If someone seconds this motion by selecting the button below, a vote on the motion will start.

A majority voting yay will pass the motion and the discussion will be extended for 72 hours. This motion will remain open until the normal discussion period ends or a motion to end the discussion is passed by a majority.
 

Chappie

Timed Discussion Bot
Secured
PaulSnow has seconded the motion to extend the discussion.

A motion is now active at the top of this thread to vote if you want to extend the discussion. A majority voting yes will pass the motion and the discussion will be extended for 72 hours. This vote will remain open until the normal discussion period ends or another motion is passed.
 
Secured
Don't the guides exist for situations that require immediate action? If we have an ANO who is deliberately damaging the protocol then maybe the core developers can ask for the guides to remove the ANO under gross misconduct or emergency removal rules?

This would maybe be a situation where the guides are able to hold a quick meeting and vote for the removal of the ANO (They would also be responsible for reaching out to the ANO to try and resolve whatever is happening first?)

There could then be a process afterward where the guides present to the ANOs what happened, why the ANO was removed and then a vote could be held to decide if the removal is to be permanent or if the ANO can be reinstated.
 
Secured
Don't the guides exist for situations that require immediate action? If we have an ANO who is deliberately damaging the protocol then maybe the core developers can ask for the guides to remove the ANO under gross misconduct or emergency removal rules?

This would maybe be a situation where the guides are able to hold a quick meeting and vote for the removal of the ANO (They would also be responsible for reaching out to the ANO to try and resolve whatever is happening first?)

There could then be a process afterward where the guides present to the ANOs what happened, why the ANO was removed and then a vote could be held to decide if the removal is to be permanent or if the ANO can be reinstated.
We think it are important to separate between suspending the ANO and suspending the Authority identities.

A entity is considered an ANO when it has been elected (participating in voting and getting the "tags" on here and discord, even if it has not yet created their Authority identities.

This discussion is about if the Authoirty identity is being misused; either by the ANO itself or by a malicious actor, so what should be done at this stage is remove the authority identity (demote it to follower), and not necessarily remove the ANO itself. This should be done by the core committee and not the guides in our opinion.
 
Secured
Don't the guides exist for situations that require immediate action? If we have an ANO who is deliberately damaging the protocol then maybe the core developers can ask for the guides to remove the ANO under gross misconduct or emergency removal rules?

This would maybe be a situation where the guides are able to hold a quick meeting and vote for the removal of the ANO (They would also be responsible for reaching out to the ANO to try and resolve whatever is happening first?)

There could then be a process afterward where the guides present to the ANOs what happened, why the ANO was removed and then a vote could be held to decide if the removal is to be permanent or if the ANO can be reinstated.
ANOs exist on chain. If the protocol is stalled, there isn't anything the guides can "do" on chain, and the vote is uninteresting. The developers and maintainers are responsible for keeping the network up and running. If an ANO's identity is being used to attack the network, and suspension is required to get the network stable, the developers have to do that.

If the removal from the authority set is to persist, then that is a decision for whomever. In the moment, it is the decision of the developers, constrained by the signatures from systems run by all of the ANOs, and the actions that can be taken as agreed to by the ANOs working on the problem.

Keep in mind, what this discussion is about is not gross misconduct outside of running the protocol. This has nothing to do with who the ANO got into an argument with, or laws they have broken, or whatever bad thing might prompt us to throw out the ANO. This is about a crisis in the protocol, where actions must be taken to keep the protocol running and to preserve the integrity of the protocol.
 
Secured
This should be done by the core committee and not the guides in our opinion.
Development and maintenance is rarely if ever done by committee. But actions taken to get the network up and running, and stable, should be reviewed and agreed upon. My suggestion is the following for one or more ANOs bringing down the network or presenting a clear danger to the network:
  1. First aid is applied by the first responders, the core developers and the ANOs, and the faulty ANO(s) is/are removed.
    1. Removal may mean just shutting down the ANO(s)'s server(s)
    2. Removal may mean swapping the nodes with the ANO(s)'s node identity(ies)
    3. Removal may mean removal of the ANO(s) from the authority set
    4. Removal may require adding code to factomd to address the issue
    5. Some combination of the above
  2. Once the network is stable, we can take one of three paths:
    1. Determine that no misconduct was involved, fix whatever the issue is, and restore the ANO(s)
    2. Determine that the ANO(s) has/have been hacked in some way, and start a longer process of fixing the ANO(s)
    3. Move for immediate removal of the ANO(s) and work through that process
    4. Move for the longer path for removal
Keep in mind, many of the Core Committee's members are core developers (as @Tor Paulsen has pointed out). That doesn't mean this process can be viewed as being executed by the Core Committee. We just don't have a mechanism for that, and I believe requiring (to continue the analogy) a hospital to provide First Aid is not appropriate. The Hospital (and the committees and political processes) kick in after the patient has be stabilized. (BTW, my list of what to do AFTER network stabilization isn't complete. There is reporting, conversations, timed discussions, and who knows what all will happen after the fact.)
 
Secured
@PaulSnow If we give core developers the discretion to temporarily remove an ANO from the authority set. How do we define who a core developer is and specifically who has the right to be able to do this?
I'd be MUCH more comfortable with the Core Committee being able to decide in an emergency session. Something like:

1. Highly emergent situation requiring an ANO to be suspended.
2. Emergency call goes out. "You have X minutes to check in"
3. The Core Committees who are present are briefed and then vote.
4. Core devs execute the necessary action.

All of this happens FAST. A speed which can be predefined so debate doesn't delay critical action.

Having a single Core dev or a couple core devs being able to make that decision is highly centralized.

As such, the document should stay as is and the Core Committee should work these processes out internally.
 
Secured
The best way I can think of to fairly resolve these conflicts without a very small minority (or just me) making the decision is to identify the outstanding issues and then create a series of polls to gauge support for the various options. Here's what I have identified, let me know if I'm missing anything.

  • Should the removal process be Private or Public?
  • Should Guides be involved in the first phase or should we eliminate them as a filter process?
  • Should the Core Committee be responsible for emergency suspension of an ANO?
  • Which of these methods should be used to limit spam (repeated attempts to remove the same ANO)?
    • Guides act solely as the filter process
    • Guides + three month limit for removal against a specific ANO (current process)
    • Require a threshold amount from Standing Parties for each removal attempt (similar to the current Guide removal process in Governance)
    • Tiered approach (first attempt has a low bar, second or subsequent attempts have a higher threshold or number of ANOs required)
    • Other
 
Secured
Paul, what specifically are you referring to?
A process to handle emergency network issues that might require removal of an ANO's identity from the Authority Set.

I'd be MUCH more comfortable with the Core Committee being able to decide in an emergency session.
So it works like this. You break your leg. You are in pain. And you are at great risk.

An EMT has made it to your side, has evaluated your situation, he has pain killers, and a gurney, and an ambulance.

So ... He calls the Hospital to get the administrator on a conference call, along with his manager, the dispatcher, the ER doc. He outlines what is going on with you, what medications he intends to apply, and some of the special circumstances, like the rattle snakes you happened to have fallen upon that are pretty upset with you. He answers questions like how many snakes, level of pain, and his suggested approach for getting you out of the street and to the hospital.

Or he dives in, pulls you out of the road (off the snakes) gets you in the ambulance, administers pain relievers and gets you stabilized and to the hospital.

Seriously, today we generally pull in 3 to 4 developers and all the ANOs when we are responding to a crisis. That's 30+ people, and actually includes most of the Core Development Committee members already. Now there are not many votes, but what the heck are we going to vote for? We have to get the network up, to date we have never had to mess with the authority set (knock on wood), but in reality we have to do what it takes to keep the network up and running. And as development goes forward, we can decentralize more aspects of the running of the network, and limit the possibility of such a crisis.

I would however like to declare this conversation off topic. Until we add something to this document about network crisis response, this is irrelevant and would be impossible to fully explain (it seems), and come to some realistic wording.
 
Secured
I would however like to declare this conversation off topic. Until we add something to this document about network crisis response, this is irrelevant and would be impossible to fully explain (it seems), and come to some realistic wording.
I agree this is off-topic for the moment and would need a dedicated thread to "network crisis response".

  • Should the removal process be Private or Public?
  • Should Guides be involved in the first phase or should we eliminate them as a filter process?
  • Should the Core Committee be responsible for emergency suspension of an ANO?
  • Which of these methods should be used to limit spam (repeated attempts to remove the same ANO)?
    • Guides act solely as the filter process
    • Guides + three month limit for removal against a specific ANO (current process)
    • Require a threshold amount from Standing Parties for each removal attempt (similar to the current Guide removal process in Governance)
    • Tiered approach (first attempt has a low bar, second or subsequent attempts have a higher threshold or number of ANOs required)
    • Other
Here are my answers:
- if we consider (as it has been stated multiple times in this discussion) that this removal needs to be fast to protect the protocol reputation and the network then I think it needs to be limited to the actors responsible for the network. So I answer "private".
- the Guides filter is nice to avoid long discussions between ANOs whereas we can be sure the motion will not pass. But I can accept to skip this process but then the ANO quorum is even more important.
- Core committee and network crisis: as stated by Paul this is a whole topic by itself.
- Limiting spam: I do not recognize the previous version of the doc. The 3 months period applied in the former version (I believe) to the ANO or other standing party making the proposal. I think it is needed and at the same time it does not prevent a bad actor which passed the first motion from being removed the day after through a new motion initiated by another ANO.
 
Secured
Paul, I’m concerned about the scenario where the EMT pulls the man with the stable fracture off the snakes causing the bone to come through the skin only to realize the rattle snakes were harmless gopher snakes anyone else on the rescue team could have identified and the video of the whole thing going viral.
Why do I love off-topic conversations so much?


In this situation you have either:
1. an EMT who knows what he's doing, and is trying to prevent something even more horrific from happening
or
2. an EMT who THINKS he knows what he's doing, doesn't, and really shouldn't be doing this job anyway.

In an emergency situation such as this, and an ANO is actively disrupting the network, it's basically the ANO or the developer. There will be repercussions, either way, if it was the ANO that did something (either intended or not), consequences will be had. The same goes for an engineer that is a little too antsy to pull the trigger.

I (and I assume many others here) work in industries where we have to make decisions instantly, as sitting around discussing what we can do will cost piles of money. If the decision is wrong, we pay the price. If the decision is VERY WRONG, we lose our jobs.

So, do we trust the Core team? I really hope so, because when it comes down to it they could do a lot more damage than an ANO, so we should also trust them to make decisions that are in the best interest of the network.
 
Secured
I agree this has moved off topic but feel we do need a network crisis response thread.

@PaulSnow at the moment the situation you've described feels like some developers have unchecked power over the network. I just want to know that those who have that power are known entities and that they have to answer to the community. You mention hospitals and doctors but if they do anything wrong they're held accountable for it and can lose their medical license.

In the case of a "bad actor" or even a genuine problem who does the developer answer to for making their decision?
 
Secured
I agree this has moved off topic but feel we do need a network crisis response thread.

@PaulSnow at the moment the situation you've described feels like some developers have unchecked power over the network. I just want to know that those who have that power are known entities and that they have to answer to the community. You mention hospitals and doctors but if they do anything wrong they're held accountable for it and can lose their medical license.

In the case of a "bad actor" or even a genuine problem who does the developer answer to for making their decision?
The core developers AND the ANOs have power over the network. Any one of them are in better positions to attack the network, certainly, but when we work on the network, we (the core developers) do so along side the ANOs.

But yeah, this does remain off topic.