API wishlist

Secured
#3
Thanks for creating the thread @Alex.

My opinion is that your asks don't require new or modified APIs for 2 reasons:
1. You can compute those 2 values already today from the existing APIs.
2. What is the usage you expect clients to have of those? Besides a metrics application (which I understand you are building ;)) I don't see how it could be used (but feel free to contradict me). The main reason is that those values are too generic for a broad range of applications to make any use of it: the result is too generic which weakens its usefulness.

For that reason I don't see factomd being the right place to compute and store those. Of course, open to debate :)
 
Secured
#4
I suggested on Discord to have a way to retrieve the timestamp when the current block started being built as it's currently not possible to retrieve it with any of the existing APIs. Maybe we could add along other information about the current block (I know debug API provides the minute for instance, that's useful information beyond debug IMO)

And here's an answer from Paul Snow about it:
Actually the timestamp of the DBSig of vm 0 is the recorded time for the start of a block. So while there is no API to get that timestamp, it would be an easy thing to add.
 
Secured
#5
Also I'm collecting here some concerns about API naming:
  • (from schalk) entrycredit-block should maybe become entry-credit-block to remain consistent with entry-credit-balance and entry-credit-rate
 
Secured
#6
Some APIs specific to new M3 code would be also beneficial:
  • Information around elections
  • Information about rankings of authority set
  • Informations about identity authority set itself (it's in debug API, but with M3 I believe it could belong to the main API)
 
Secured
#7
Thanks for the feedback, @Luap.

How would you compute FCT and EC supply from the existing API? I went through this with Paul a while back and he didn't have any useful suggestions. The best I could come up with would be to establish how many factoshis were minted in the genesis block, then track factoshi creation/burn and EC creation/burn since then. There would be a large initial computational expenditure, but after that it would be easy enough to track.

My instinct is that we should make the cryptoeconomics of the protocol as transparent as possible. I'm building something to aid in that, but building it into the protocol itself transforms that transparency into a central tenet of the protocol rather than something which needs to be carefully extracted.

I'm not super attached to the idea though, so if others feel the same way as you then that is fine.
 
Secured
#9
Yes, I think you described correctly the process to compute those values. This is how blockchains work anyway, you rewind it and compute things on the way, so it appears perfectly natural to me. This is how the bitcoin wallet compute how much you have on a certain address for instance. The point is that you can compute and audit that value yourself which makes you sure that this value is the correct one.

And it is indeed computationally expensive which IMO is another argument that we shouldn't burden factomd with that requirement if not strictly necessary. Every addition to the API has a cost, of development, but also a performance cost at runtime (but on this only factomd developers can comment).
 
Secured
#10
@Luap Would you mind expanding on this suggestion a little:



What rankings and what information?
I believe that once M3 is at full speed (65 operators) and stable, who is fed and audit will be completely automated and the network will rank nodes to know who should be in the authority set and who should be audit vs fed. But I agree, this is probably too early for such an ask as we don't have visibility on how it will work concretely yet. Just putting it here for the record and keeping it in mind.
 

Emyrk

The Factoid Authority
Secured
#11
I'd like to be able to get:

i) Total number of spendable factoshis (i.e. not total number issued, as that would not remove those that have been burnt)
ii) Total number of spendable EC (same as above)
I agree with Luap, that these can be obtainable from the blockchain.
Factomd should not have excess api calls to maintain, applications ontop of factomd should parse/store things that are designed for the application.
 

Emyrk

The Factoid Authority
Secured
#12
As for the authority set, I agree it would be beneficial to have insight. Something as simple as listing the Identities and their status as Fed/Aud would be beneficial. There are some technical questions to be addressed, such as the processlist has a set of Fed/Audit. This set does not always match the blockchain (in the case of elections). So the API, would it be realtime status, or the blockchain status?
 
Secured
#13
Ah, good question. I don't think I know enough to give my opinion there. That said the blockchain status is probably a safe choice? And there could be a separate API for election information that could allow to use them together to get the realtime status? Just thoughts.
 
Secured
#15
New API proposal: get first entry of a chain (querying by chainid)

It's a common pattern in Factom to make the 1st entry of a chain special in the sense that it contains information about how to interpret the data in the chain. Being able to retrieve that entry immediately without having to traverse the entire chain could be beneficial (you want to access the "rules" of the chain first).
We were also discussing some other potential use such as chains that are supposed to container a single entry but that could get spammed, being able to retrieve immediately the first entry would completely defeat the spamming.
 
Secured
#16
New API proposal: get first entry of a chain (querying by chainid)

It's a common pattern in Factom to make the 1st entry of a chain special in the sense that it contains information about how to interpret the data in the chain. Being able to retrieve that entry immediately without having to traverse the entire chain could be beneficial (you want to access the "rules" of the chain first).
Second this! This is a basic requirement for applications built on Factom to be efficient, and is a use case proposed in the Factom Whitepaper:

"The first Entry in a Chain can hold a set of rules, a hash of an audit program, etc. These rules then can be understood by Applications running against Factom to ignore invalid Entries client-side." (Factom Whitepaper pg. 7)
 
Secured
#17
Related to the previous post, one thing I would very much like to see as part of the API is caching. This is partly addressed in the recent Development Grant proposed in Factom Inc., but I would like to emphasize how important increasing the performance of chain/entry retrieval is to building real world applications on Factom. Even though developers like myself want to build apps on Factom, we're coming up on read performance hindrances that are making it difficult to make them scalable and testable. In the interim, DBGrow has been using a in house caching solution to make things perform properly.
 

Emyrk

The Factoid Authority
Secured
#19
Related to the previous post, one thing I would very much like to see as part of the API is caching. This is partly addressed in the recent Development Grant proposed in Factom Inc., but I would like to emphasize how important increasing the performance of chain/entry retrieval is to building real world applications on Factom. Even though developers like myself want to build apps on Factom, we're coming up on read performance hindrances that are making it difficult to make them scalable and testable. In the interim, DBGrow has been using a in house caching solution to make things perform properly.
Ideally you would be able to select certain chains for your application, and the node/api you are talking too will cache any and all entries in that chain. Our usual philosophy is to let a layer ontop of factomd handle this, as I could see features being requested on top of this (indexing for example).
 
Secured
#20
Ideally you would be able to select certain chains for your application, and the node/api you are talking too will cache any and all entries in that chain. Our usual philosophy is to let a layer on top of factomd handle this, as I could see features being requested on top of this (indexing for example).
I would agree. I don't think it would make sense to have caching or indexing built into factomd. I think that these should be separate applications developed on top of factomd. Like the in house caching solution you have created drkatz.

This is part of what we are also working on at RewardChain - to build on top of factomd to provide a way to easily, efficiently query the blockchain.

Also - moving forward, we will eventually see sharding built into the protocol, so a single factomd client, don't contain the entire blockchain.
 
Secured
#21
@drkatz -- may I add your caching solution to a couple resource pages for others? Or is it not production ready or available for others to utilize?
Hey there! This project was started within the last month and is most definitely not production ready. The library is currently usable by others via NPM but there will be breaking changes in the near future. Our roadmap places an alpha release for the project by the end of June :)

I'm interested in the lists of resources you mentioned @DChapman, where could I find them?
 
Secured
#23
Along the lines of a request to get the first entry in a chain, it would be awesome if we could get the entry count for a chain from the API without scanning the entire thing. Will be quite useful for setting up probabilistic datastructures like Bloom filters and other applications.
 
Secured
#25
Hey @drkatz can you expand a bit on how you would use Bloom filters on Factom and to what purpose?
Bloom filters will be useful for systems built on top of Factom that need to efficiently check for duplicates. For example, a token system that requires unique nonces to prevent replay attacks when executing a transaction. When constructing the filter, it's important to know the count the filter will need to hold so the rate of false positives can be estimated.
 
Secured
#26
I would like:

1. API calls that get director block information to include information on the Anchor(s)
2. Consistent "raw" or a separate API call, to get the hex of the raw binary for the block from the start of the header to the end of the body. Currently the directory block includes raw in it's response, but not the entry block.
3. Include prevfullhash in the entry block API call as that is required to calculate the merkle root for the block.