Factom Live API considerations

Unrestricted Public Thread

  • Viewed Sander Postma
  • Not Viewed None
Secured
#1
1st layer

Push process list implementation
- As place of event trigger I was looking at stateConsensus.FollowerExecuteDBState around s.AddDBState. However if someone has a better idea please feel free to comment.
- The push should contain a signed process list to avoid a man in the middle attack at this point even though the underlying data is protected. (Implementers may assume that what's coming from factomd will be on the block-chain and won't double check that.)
- For the event posting to the 2nd layer app, do we only want to support a TCP socket or pipe to localhost or a cluster pod on the internal network? Or can the second layer run on a remote server? The latter would mean we need SSL support and http(s) Maybe Prometheus can also be used for this, but the dataset is much larger than just a counter.


2nd router/proxy layer service

Which type of events and gauges would want to be able to subscribe to?
- Addition of new entries
- Addition of new chains
- Progress/life-cycle updates on created entries
- Factoid/EC transactions
- Progress/life-cycle updates on transactions
- Factoid/EC address balances
- Admin block entries
- Add support for election results?
- Add support to request the values of the Prometheus counters in the result?
- Statistics like on factoshi.io and factoid explorer?


Filtering
- chain id, or better a set of chain id's.
- paying addressees
- In case of transactions, by input/output addressees
- External id filters
- Content filter
- Regex support where applicable


Result set
- We should supply both a Swagger v2 & v3 model containing enums which can be used to decode the entries, including the admin block entries. There are many existing platforms which support these models.
- Also implement support for the more flexible GraphQL


API mechanisms
Not all API mechanisms are supported in all environments, there can be firewall or proxy server restrictions the developer has no control over, so which of the following protocols would want to support? (Not all at once but in phases.)
- For all methods below you would call a "subscribe" function with set of filters
- A web hooks interface where you specify a callback URL. At the end of a valid block a JSON will be posted with the result.
- A long polling interface where the client requests the next result and waits for it. When the connection times out it retries. (Behind certain firewalls or proxy/reverse proxy servers where the implementer has no control over, this will be the only way that always works.)
- A web sockets interface where you have a active socket open. At the end of a valid block a JSON result will be pushed back.
- SSL support, optional client side certificate requirement
- Callback authentication support. At least basic and OAuth 2.0 authentication and for the latter working towards supporting more login protocols.
For instance registering a callback with a remote OAuth 2.0 server requires user interaction in a web browser
=> we'll need a web portal app for that
=> we'll need account management to give access to that web portal, so this is a big job that makes more sense for a post-release version
 
Last edited:
Secured
#3
To clarify I am reading this post correctly, the majority of the post is regarding the second layer? The filtering, result set, etc?

------
As for a man in the middle, I don't think we need to protect against a man in the middle on the factomd side. A factomd will likely only have 1 outbound pipe, so communication should be protected on the devops setup. I suppose a tls implementation would not be difficult if we want to secure the pipe even further.
If we ensure acks (signatures) are attached to each event, the consumer needs to be aware of the authority set, requiring more logic to create a second layer.
---
Code stub location:
The code stub location depends on what event we are talking about, so the exact location I think we can defer until we decide on the events we will be needing. I think a raw tcp pipe would be sufficient, and we can do tls over the tcp pipe if we need, although I don't think it is needed. There is no exchange of private data, and we want to keep the overhead of factomd minimal. I'm not adamant one way or the other.
As for the prometheus comment, I attached a document that details thoughts on using the prometheus package design pattern for our events. We would not use prometheus for the events, but their design pattern is very clean as far as registering the metrics (events) that the internals will use.
---
Extra thoughts:
- The event pipe can be thinned by not repeating itself. For example. If we are tracking the life of an entry from holding -> processlist -> Dblock. We don't need to send the data at each event, the 2nd layer can handle that with caching.
 

Attachments

Secured
#4
Okay the first time reading through I stumbled over the fact that this is made of two parts. The core implementation that sends signals (the processlist) to the LiveAPI, and the LiveAPI that takes those signals and then sends out events via various methods to the end users.

We already have some second layer solutions that provide event streams, like the js api that achieve this via polling. Have you considered just making it one package and including that in the core code, similar to how the API runs inside factomd?

Keeping these as separate applications seems like it would throw in a lot of unnecessary work (like security and upkeep between the two apps) since either one just does not work without the other.

Which type of events and gauges would want to be able to subscribe to?
I think you got all the useful ones except for the ticking of minutes / end of block (with height).

so which of the following protocols would want to support?
My wish list would be web sockets and web hooks!

I'd also like a way for core code to be able to make use of this, so perhaps also a way to send signals via go channel as an alternate to sending it to the liveapi, which would be an important stepping stone in refactoring some code.

Anyway, an event emitter like this has been on my wish list since the first time I started working with factomd, so I'm really excited this is getting made :D
 
Secured
#5
To clarify I am reading this post correctly, the majority of the post is regarding the second layer?
Yes, the goal is to minimize the impact on factomd. We (BI Foundation) will focus more on the second layer where the new functionality will be incubated. factomd just needs to get all the metrics out, nothing more.

Have you considered just making it one package and including that in the core code, similar to how the API runs inside factomd?
Yes we have, but (a more detailed explanation of the first question)
- This is brand new functionality, a separate daemon would allow a more flexible release interval not depending too much on the factomd releases. (Except for the event push functionality)
- For factomd, merging in all the outstanding feature/fix branches and getting properly tested update releases out already seems laborious enough without introducing lots of new code and their dependencies.
- We will make efforts to modularize the LiveAPI code base so once it has fully matured and after factomd has been sufficiently refactored the possibility to merge the two will be open.
 
Secured
#7
I've drafted an illustration based on the functionality where
• The API can be used by a static back-end service that always wants to listen for the same events using the same filter.
• The API can be used by front-end browser based application that wants to listen for events based who the end-user is / which options he selected at that time.
• The API can be used to subscribe to the life-cycle of a single entry or transaction until it's anchored. The subscription should self-deregister after the last event.
• The API implementer may or may not own / have access to the factomd & live API nodes.
• An attacker should not be able to hijack one or more LiveAPI nodes re-purpose them as DDOS attack servers.
• When using webhooks, the API implementer should be able to secure his requested callbacks to prevent attacks on his backend.

LiveAPI.png
 

Attachments

Secured
#8
I've drafted an illustration based on the functionality where
• The API can be used by a static back-end service that always wants to listen for the same events using the same filter.
• The API can be used by front-end browser based application that wants to listen for events based who the end-user is / which options he selected at that time.
• The API can be used to subscribe to the life-cycle of a single entry or transaction until it's anchored. The subscription should self-deregister after the last event.
• The API implementer may or may not own / have access to the factomd & live API nodes.
• An attacker should not be able to hijack one or more LiveAPI nodes re-purpose them as DDOS attack servers.
• When using webhooks, the API implementer should be able to secure his requested callbacks to prevent attacks on his backend.

View attachment 1773
After the exporting of information, the Live API Daemon looks good. I imagine the rest api might be more of a pass through in the early stages, and as we add events, you can obviously expand a lot of the data that can be exported from the daemon.

I'd like to see more information on how the connection would work between fatomd and the API daemon. I agree that having some 2 way communication would give us more flexibility in the future. Given that each factomd would likely be limited to 1 event stream, it should have external protections from any malicious actors, as it's not a public consumption thing.
The bilateral should be standardized, and I'm curious how you envision the event stream looking like inside factomd.
 
Secured
#9
I'd like to see more information on how the connection would work between fatomd and the API daemon.
I've been looking into the code and thinking about this a bit. There already is a lot of code for marshalling and transporting messages in parcels/FactomMessage and I'm looking at what could be reused instead of writing new network transport code. I'd like to explore the following idea first:
We let the LiveAPI daemon connect to the factomd as p2p peer just like any normal peer would and we get everything that NetworkOutputs normally sends/broadcasts. On top of that, allow to configure the peer as "event receiver peer" and send it extra messages about local data like pending entries/transactions and move the node to the top of the peers list to reduce delays.
If the user wants additional details like EC/FCT address balance after a transaction on his paying address we can initially call wsapi after we see a matching transaction. We could also import existing factomd packages straight into the LiveAPI project, setting up an input-only p2p node and process the incoming messages in a separate go module where it can call UnmarshalBinary() on them.
Thoughts?
 
Secured
#10
We let the LiveAPI daemon connect to the factomd as p2p peer just like any normal peer would and we get everything that NetworkOutputs normally sends/broadcasts.
I would strongly recommend against having a local node connected to the mainnet peer with another local node through a local network interface, since the node would store the local ip in the system and share it with other peers. Separating that logic via code would be hard since local development networks should treat local ips regularly. There is a possibility of determining that a node is a liveapi node through the handshake in the new p2p system but it's still a very messy solution.

Further, a single peer does not receive all messages. Broadcast messages select a random subset of connections and if the LiveAPI node is not part of that subset, it would not receive the message.

If you want to capture the raw network traffic, I would recommend embedding the code inside P2PProxy, which would let you read/copy/redirect all network traffic sent to/from a node.
 
Secured
#11
Clear, we should have a separate tcp channel for the event listener then. So I think we could use an extra send command in p2proxy after p2p.BlockFreeChannelSend() to another channel to the event listener. (I guess we won't be needing the directed messages.)
For the pending entries we can send additional messages to that same channel. We can still reuse the message marshalling code.
 
Secured
#12
We let the LiveAPI daemon connect to the factomd as p2p peer just like any normal peer would and we get everything that NetworkOutputs normally sends/broadcasts
I have an initial gut reaction against this. It sounds that would be effectively reimplementing a lot of the processing of factomd on the other side of the pipe. The idea of the live feed api is that factomd is already tracking the lifecycle of events, and could emit these with a much easier to consume stream. Currently, I am aware people have "hacked" factomd to export information they want, the goal of this is to create a standardized stream for all developers to be able to consume the data they need. We need developers to have quicker feedback for their applications, and going the p2p route sounds more like we're just pushing the problem to the next daemon, rather than solving it. We'd like the community to build potentially more than 1 "2nd layer", as a 2nd layer solution might be revolved around monitoring rather than data. Or another purpose someone comes up with, and they need the raw pipe. If we force people to reimplement a peer every time, we force the developer to:
1. Use GoLang. (Reimplementing all the marshaling and staying up to date is not practical for most projects)
2. Continually update to match any p2p changes (we change p2p more than we change api related endpoints)
3. Advanced understanding the internals of factomd on a level they can interpret acks, commits, reveals, admin block msgs, etc. These messages trigger life cycle events of an entry, identity, etc. Parsing them and implementing all these state changes is an undertaking. We hope to make these life cycles and events much simpler, such as not tracking a commit vs entry, but combining them into the same "event life cycle". (Exactly how to simplify this is tbd)

A p2p peer doesn't also know the internals of factomd, so if we want to export any internal related state, we'd have to emit custom network messages. Which isn't ideal, as the code inside factomd isn't set up to distinguish a peer from this emit peer.

Another thing about the p2p peer. I don't like creating a special peer type. I agree with everything @Who has said, and it's adding logic to the p2p side of things that does not need to be more complicated.

-------
For the wsapi calls after to get balances. That is correct, the restful api will be the endpoint for static content at the moment. It might not be the ideal solution, but once we get the live data working as we want it, we can start addressing the other pain points (such as a slow restful api).
 
Secured
#15
Ok so translate the messages into something more structured. Were you thinking JSON? Protocol buffers?
Something like that would make sense. I'm ok with sticking to the raw binary on things like entries, as they will be most of the volume and we don't want to add marshaling cost to factomd. But for many of the events, we could make the entire structure easy to parse with json or something, as they are not going to be frequent enough to be that much of a cost.

I don't think we want this pipe to make a large dent in the profile, and it's hard to anticipate what sort of data volume we should develop for, and how much we say "we'll optimize when we get there" kind of deal. We can easily do some benchmarking to compare some decisions.
 
Secured
#16
Not sure about your comment, but a loopback interface is not a real network interface of course.
Funny that 2 people get in front of me :p
As far as bandwidth is concerned sure. The current code though has no distinguishment for a different kind of peer, so we will try and share it. Who was bringing up some of the side effects on just adding a new peer with this behavior, that really isn't a peer. It would add a lot of fake peers in the network.

I think what he really signals is that we'd have to implement custom p2p logic to handle this, and that just adds complexity to the p2p package. Code wise, I'd rather this be in it's own package, or even it's own repo if you feel appropriate. The logic in factomd should be minimal so it's easy for Core Developers to add new events and emits.
 
Secured
#17
Clear, we should have a separate tcp channel for the event listener then. So I think we could use an extra send command in p2proxy after p2p.BlockFreeChannelSend() to another channel to the event listener. (I guess we won't be needing the directed messages.)
For the pending entries we can send additional messages to that same channel. We can still reuse the message marshalling code.
Had some more time to think about it and I'm not sure why you want to capture the raw network traffic for this, since that is before most error checking and validation. Things like which transactions are valid can't be determined until the processlist is being processed. If you send raw traffic to the LiveAPI, you'd have to perform your own checks of which messages have valid signatures, which transactions have valid balances, which acks are from feds, etc. You'd end up having to redundantly implement things factomd already does and things like the finished blocks are not actually sent out over the network (unless a neighbor requests it).

The best place (imo) would be to sit at the front of the processlist to watch everything going in for potential-but-not-yet-confirmed things like transactions and chains, and at the other end of the processlist to read the new block data for confirmed changes. To send the data, I'd use a push-pull method where the node just sends header information through protobuf to the LiveAPI and then the LiveAPI can request more information (like the entire block) if there are active subscriptions that actually want the data.
 
Secured
#18
I'm ok with sticking to the raw binary on things like entries, as they will be most of the volume and we don't want to add marshaling cost to factomd. But for many of the events, we could make the entire structure easy to parse with json or something, as they are not going to be frequent enough to be that much of a cost.
I would prefer to make it consistent, not one thing json and the other thing binary.
I will create an experimental setup and tests some serialization protocols to see what the impact is and how easy it is to work with them.
 
Secured
#19
As another stakeholder in the success of this feature, the Harmony team at Factom would like to express our concerns with some of the ideas being floated and also suggestions as to the direction we would like it to head.

Many people have already commented about how the listening to network traffic isn't viable, so I will leave that issue alone. I'd just like to reiterate the need for the event log to be producing messages for objects that have already been validated in the context of the block chain state (to the extent that is possible for the event type). If we have to re-implement validation in the next layer, that seems to defeat the purpose of the feature.

Additionally, we would really prefer not to have to run and maintain yet another daemon. The less moving parts the better — for everyone involved really. If we're already piping out the objects and their data (presumably to a URL), we suggest adding optional support for proper queueing services such as Google's PubSub and Amazon's SQS. A simple http request template could likely suit these needs well, and ensure that node operators aren't burdened with maintaining additional infrastructure if they'd rather pay to consume an off-the shelf SaaS offering.
 
Secured
#20
As another stakeholder in the success of this feature, the Harmony team at
Additionally, we would really prefer not to have to run and maintain yet another daemon. The less moving parts the better — for everyone involved really. If we're already piping out the objects and their data (presumably to a URL), we suggest adding optional support for proper queueing services such as Google's PubSub and Amazon's SQS. A simple http request template could likely suit these needs well, and ensure that node operators aren't burdened with maintaining additional infrastructure if they'd rather pay to consume an off-the shelf SaaS offering.
Currently, the design does require a "2nd layer", meaning a 2nd daemon will probably exist in some capacity. This solution was determined as we began to try and design the general-purpose solution for various use cases. When posed with questions such as:
- Should we only send events for entries in subscribed chains? Subscribe by extid?
- Should we send the data of the entry? Maybe the user already knows the data?
- How do we handle pubsub scaling? Will this soak the network pipe factomd is on?
- How should we handle callbacks? Raw tcp? Websockets? Do we need api authentication?
etc

It quickly came apparent we do not want to maintain the application facing system inside factomd. Instead, we realized if we export the events in a single pipe to 1 listener than a second layer solution can be built to handle whatever application logic complexities arise. The 2nd layer can also reduce the amount of data needed to export, as we don't need to repeat entry content in sequential events. It would also be beneficial to not be tied to the factomd release cadence for this, as it could be developed at a different pace.

The discussion around how the events will look, what events to include, and the format is still being discussed. We just know we'd like to keep the event system out of factomd simple. A client could directly consume this, however, to support things like pubsub and more, we'd most likely see 2nd layer solutions being built to be consumer/client facing.
 
Secured
#21
I wasn't suggesting factomd implement its own PubSub system. You guys mentioned that is going to just be a firehose of data. Rather than spewing that stream of data at a particular IP address/URL where a home-grown daemon lives, point it at a proper queue that has no maintenance overhead.

The same design would still apply: no sorting entries into buckets, no logic, no dealing with subscribers in any way, factomd just spews out everything at a single URL. And our own second layer worker processes will pick stuff off of the main queue and dispatch them into our own sub-queues to use as needed for different applications.

Does this still conflict with the general premise of the firehose? Just pointing the nozzle at a different target.

EDIT: if needed, we're prepared to throw our own man hours toward it since it's a fairly strong preference
 
Last edited:
Secured
#22
Just a small suggestion here as another stakeholder and seeing this convo go down. It might make a bit of sense to me to get several people in one or more calls to discuss. So get the heads in the same direction. Doing that through this forum is not the most optimal way as it will take weeks to get it done.

BTW the separate daemon stuff in the world of Docker and Kubernetes can be a non arg imo.
 
Secured
#25
Thats not what I was implying lol of course k8s and container orchestration are great. The home-grown daemon is the point of failure when its the onramp to all other services you run. Just unnecessary overhead that our team doesn't want to be stuck devoting man hours to maintain and debug.

We aren't discouraging you from doing that with your own products and teams and resources, or even suggesting that there can be only one option. Rather, we are saying that we want the option of pointing it at a queue rather than a daemon, that's all

edit: we can discuss more on the call
 
Last edited: