Funded [DeFacto-001] Factom Open API (open-source REST + GraphQL API for Factom)

Status
Not open for further replies.
Secured
#1
Hello Operators & Guides!
There is a grant proposal for Factom Open API.
--
Factom Open API is a combo REST + GraphQL API Server, that connects to factomd node (locally or remotely, e.g. to Open Node) and allow developers, operators & enterprise to easily interact with the Factom blockchain.

Main features:
– Additional high-level functions
– 
Fast & easy set up of environment
– Simultaneously support of traditional REST API + modern GraphQL API
– Doesn’t require to keep blockchain locally & powerful hardware/VPS (so Open API may be installed & used even on Digital Ocean minimal $5/mo VPS)
– No separate factom-walletd instance required: all transactions are prepared and signed inside Open API, you just put EC private key in config

The main differences from the previous grant project:
– REST paths redesigned
– GraphQL module
– SDKs development included
– Works adjusted to fit into 3 months cadence
– I agreed with 2 developers (that will be hired for this project) that major part of received FCT tokens won't be liquidated

Open API presentation is attached alongside with grant proposal.


Added on 22.01.2019
If this grant will be elected, we pledged to develop an open-source WatchTower solution for immutable security logs of any linux servers and different NAS, that will use Factom Open API for writing and verifying logs on the blockchain.

Immutable logs of any linux servers will significantly increase protocol usage by different parties (server administrators, enterprise, developers and etc).


Important updates to the grant proposal, 06.02.2019
1. Grant divided on 2 sprints (grant rounds)
2. Grant amount amended to 6,980 FCT on development + 400 FCT on sponsorship
3. Jay Cheroske (Bedrock Solutions) is a sponsor of this grant with wider responsibilities — he will review and help to improve the design/arch of Open API & its features during the first sprint
4. We decided to use Golang for development of Factom Open API, so more developers may join us in the second sprint.

Changelog: I have uploaded "v2" with the indemnification clause included.
 

Attachments

Last edited:
Secured
#3
An open-source WatchTower project, that we are developing using ANO income (no grant), relies on Factom Open API.

Easy installation/usage (and possible mass adoption) & some WatchTower's functions (like reading/searching logs) completely depends on Factom Open API project.

E.g. 1 server of our client generates ~250 tamperproof logs (entries) daily.
With easy installation & configuration + open-source + low prices with using Open API (1 log = 1 EC) we may achieve that WatchTower will be used on many servers (maybe even thousands or millions servers).

We are going to release an open-source WatchTower within 1 month after Open API will be ready.

watchtower-info.png
 
Secured
#4
In the grant, you ask for 1450 hours of work to be supported within 12 weeks. This equates to 3 people working full time on the development. Can you give us some details about the developers and their experience?
 
Secured
#5
@Ben Jeater
In Nov-Dec I hired backend developer (freelance) to help me with Factom Enterprise API we recently launched.
He has about 10 years of experience and I liked his work on Enterprise API.
So I'm going to hire him again for developing Open API.

2nd backend developer has worked for me few years ago on Instashow project (https://instashow.tv), he has an experience with Yii2 & GraphQL APIs as well.
 
Secured
#6
Thank you for your proposal.

This is a generic question I ask in each grant thread.
This is currently the 3rd grant round. I consider that one of the very important criterion to select a grant (apart from its potential value) is the capacity of the grantee to deliver in time what it pledged. Therefore past grants can be used as an indicator.

If you did receive grants in previous rounds, could you please fill the following fields? This would increase transparency and help the standing parties to select grants.

- Have you, or one of your partners, previously received grants : Yes/No. If No, then you can stop here :)
- List of grants received : grant X1 from round Y1, grant X2 from round Y2...
- Status for each grant : grant X1/Still ongoing or Completed, grant X2/....
- Description of the work accomplished so far and Links supporting it : Discord Group/Factomize thread/Github/Reports/...
- Description of the residual work to be completed : XXXXXX

Thank you for your cooperation.
 
Secured
#8
@Niels Klomp I never told that we will start developing Open API from the scratch.

There are some things from Enterprise API that will be used:
— Queuing
— Entry/Chain models & input data check (chain size is hex & 64 symbols long, extIds are required for creating chains, entry size up to 10KB and etc.)
— Counting writes for API keys & using usage limits for users

I like how the queing works in E-API, but maybe we will make it better in Open API.
(We implemented @Paul Bernier's idea: if write on the blockchain is failed for any reason (factomd unavailability, chain doesn't exist and etc.), the time interval (delay) before next write attempt increments directly proportional to square of write attempts.)

This is a minor part of all works.
In our grant application we have already considered using the minor groundworks from Enterprise API and not developing it.
I consider this works are about 7-8% of total works of Open API.

Local database, subscribing to chains, search, additional endpoints, callbacks, generating & signing entries internally (without factomd-walletd), GraphQL, API user management, Web UI and many other things are needed to be developed to get a great product.
+ we will pack it all as Docker image to provide easy installation and great UX.

I think we apply for very reasonable amount of money to develop Open API.
I even say it's a very affordable price for all described works.
Comparing to other grants from this & previous rounds, the Open API project requires only 60K (and just 57K after taxation) to be delivered.

As a result community will get a tool for easy interaction with blockchain.
Look at the Chainpoint — they provide REST API out-of-the-box — why Factom provides only paid (subscription) tools for this?
It's a gap we need to cover, and we will see much more great projects based on Factom from new and existing developers.

The first project that uses Open API will be an open-source WatchTower (or how we name it at release).
 
Last edited:
Secured
#10
If someone is still not sure if the Open API grant is needed, when we have low-level libraries for factomd (we had such conversation in previous grant round), I would like to emphasise the difference:

1. High-level functions (search data in chains, queuing, callbacks, multiple API users with limits) – it exists in Open API, and doesn't exist in any low-level lib

2. REST/GraphQL APIs (any developer may use it completely independently from language — Go, js, Java, PHP, C, C++, C#, Python — and whatever you want) — that means that new developers will start using Factom, building projects and increasing usage.

3. Fast start
Open API installed within few minutes from Docker Hub. It may be installed on any server that supports Docker.
You don't need to setup factom-walletd for manage Entry Credits, that you use for writing data.
When you install Open API, fill Es private key in the config — and all entries will be signed inside Open API.
Factomd location also filled in the config — you may fill Open Node (api.factomd.net) and start writing on the blockchain instantly.

4. No need to touch crypto
The main benefit of 2-token system – you have a stable predictable price for writing entries — and you don't need to touch crypto when making entries.
If you're an Enterprise that can't buy BTC or FCT, you just go to the EC store (that already launched by De Facto & Inc) and buy ECs directly for fiat — funding the EC address that used by your Open API server.
This completely fills the ecosystem for enterprises.

If someone still thinks, that it's not time for this project, I want to say that we need Open API now.
1. Chainpoint project (Factom clone) recently updated, and they have open-source REST API out-of-the-box.
https://chainpoint.org
2. Additionally, they use the same "Load-balanced Node + REST API" combo that I suggested to realise, that looks very attractive for developers comparing to the Factom project.
https://github.com/chainpoint/chain...nt-Node-API:-How-to-Create-a-Chainpoint-Proof

So I ask all Operators for support this project (not necessary to put it on top of the voting list, but please don't lower it if you are not sure if it's needed or not).
I see (and possible you too) that this project is an extremely significant and well-timed.
 
Secured
#11
Hi Anton,

Thanks for putting that grant forward. I personally have no problem seeing that an Open API would be a great addition to the Factom ecosystem.

I have 2 rather important feedback though:
  • You have been thinking about this tool for some time now and have come up with a great list of features to make it as complete as possible. I'm thinking that it's not necessary to develop the whole thing at once in just 3 months and have to hire extra people to handle that. For instance do we really need both REST and GraphQL since day 1? Do we need the shiny dashboard UI immediately? Open API is an API interface, that UI is a bonus. So I'd be a proponent to break it down in smaller pieces and keep improving it over time (with other grants).
  • The Open API would be an important community project with great exposure to many developers, and as an open source project it needs to be top-notch in terms of quality and robustness. Similar to the on-chain voting grant or the FAT project, I'd strongly encourage you to have a partnership with at least one other party (technical ANO?) to design and build Open API. Having multiple parties collaborating will help avoid pitfalls stemming from single-minded perspective and offer a better guarantee that the software will satisfy everyone's needs.
Those 2 points would greatly improve the success rate of this project in my opinion. Without those, I cannot rank the Open API project as high as I would like for this round.
 
Secured
#12
@Paul Bernier
Big thank you for this feedback. I found your suggestions are pretty useful. Having a second view on the product from another Operator is the thing I like.

We will try to find a sponsor for this project.

Additionally, probably it makes really sense to divide this grant project into smaller parts to work in more calm rhytm with internal resources.

I will update this proposal shortly.
 
Secured
#14
We believe this is a very important enabler for easily onboarding outside developers and will support this grant, even more so if the amendments suggested by @Paul Bernier are reflected in the proposal.

I would suggest you guys look at someone not only as a sponsor, but maybe as a part-time collaborator at least for the general architecture/design stage.
 
Secured
#15
I would suggest you guys look at someone not only as a sponsor, but maybe as a part-time collaborator at least for the general architecture/design stage.
Yes, I have already thought about having a sponsor with a wider responsibilities — not just only review the product during dev stage, but someone who can look critically on our suggestions and suggest own ideas regarding design/features.
 
Secured
#16
@Paul Bernier @Samuel Vanderwaal @Alex @Matt Osborne

I divided Open API project into 2 sprints:

1) Core REST API, background fetching / subscribing to chains, local database, fast read / search chains, multiple users (API keys), queuing, signing txs inside API (without factom-walletd), counting usage, limits.
+ Dockerizing app + REST API documentation

2) GraphQL support, GraphQL subscriptions to chains, REST callbacks, Web UI for management, admin API endpoints for management Open API users (API keys) + GraphQL documentation

Works of the first sprint are amended to fit into this grant round. The budget for delivering this project amended from 11,178 FCT to 6,980 FCT for development and 400 FCT for sponsorship.

All additional features (GraphQL support, Web UI, callbacks) that we initially described were postponed to the next sprint (grant round). All core features remain, that means after this grant (Sprint 1) will be delivered, the community will receive a stable working REST API, that may be used by everyone in personal or commercial purposes out-of-the-box.

--

Jay Cheroske (Bedrock Solutions) is a sponsor of this grant with wider responsibilities — he will review and help to improve the design/arch of Open API (local database, background processes, syncing/fetching the data from the blockchain, usage & limits works and etc.) & its features during the first sprint.

De Facto already worked with Jay on the Factom Open Node, that successfully launched in Dec 2018, and we are excited of his vision on the community infrastructure projects and his experience as a developer.

After this grant will be delivered, we are going to create a workgroup to work on the second sprint, the main part of that is GraphQL support. Bedrock Solutions will be a part of this workgroup and I will be happy to see other ANOs as a part of this workgroup.

--

The amended grant proposal:
DeFacto-001-Sprint-1.pdf

@Guides could you please change the grant amount of this project in the voting docs?
 
Last edited:
Secured
#17
The effect of the Open Node project on the community is already being felt. MyFactomWallet, Enterprise Wallet, and the DeFacto Enterprise API projects (just to name a few) are using it. It's obvious that our decentralized protocol needs robust, decentralized access points. It's also obvious, given the proliferation of indexing servers and proprietary APIs, that higher-level abstractions above the basic factomd API are also needed by pretty much every dev building pretty much any app.

I agreed to join the project under the condition that we solicit input from as many interested parties as we can. We want to build high-level REST and GraphQL APIs that are what the community is looking for. We want these projects to be open collaborations, where any developer can submit their ideas and code. We are looking to leverage the experience gained, often through frustration, of devs here. The world doesn't need yet another proprietary indexing server.

The Open Node project is continuing to evolve. Bedrock is in the process of rolling out a proxy for the factomd API port that dramatically reduces the effort required to deploy a node in that system. Following that, we will be releasing our Kubernetes Helm chart, which automates the deployment of factomd node pools, complete with integrated proxy, load balancers, etc.

I mention that work because this proposal aligns with our vision of open, scaleable access to the protocol with minimal cost to deploy and maintain. To put Factom services on the Internet requires CPU, memory, and disk resources, and those costs relate directly to the amount of usage. There's only so much that can be done to reduce those costs. What can be done though is to free developers from having to write yet another complex backend, and to free admins from the burden of spinning up factomd. And, if an ANO decides they want to run servers that participate in one or more of the Open X projects, we want the time it takes to get that ANO onboarded to be minimal.
 
Secured
#21
I appreciate that you took Paul's suggestion on board. I think that was a good decision. Jay's input will be invaluable.

Am i correct that your core competency is in PHP and that you do not have much direct experience of writing Go? I understand that both Go and PHP are C-like languages, but beyond that they do not share much in common.

Generally speaking, I think Go is a much better language choice for this API. However, for this to be a scalable, extensible, production-ready API, the codebase will need to be robust. Ideally, it needs to be written to a standard that can only really come from extensive, direct experience of the language in question. This is not a comment on your skills as a dev, this is a comment on the experience of a language that is needed to write good code in that language.

Personally, I believe that this API would ultimately be much better if you wrote it after spending some time getting to know Go. I would be interested to hear your opinion on that, and the opinion of others like @Paul Bernier and @AdamSLevy?
 
Secured
#22
@Alex
I am full-stack developer with a huge experience in different languages.
The syntaxes of PHP & Go are similar and I see no issues of coding in Go (I have already deployed sample REST API that uses Postgres db — and can confirm that using Go is the great decision for making the REST API).

50% of any REST API is how it's designed & 50% is how it's coded.
You have tested our Enterprise API and, I guess, you liked how it is designed and how it works ;)

Of course, different languages have different patterns. But with a rich background, it's not a problem to explore Golang patterns and use them.

Right now I'm exploring the existent APIs written on Golang, libs that they use, how they are structured.
I'm confident in my dev skills and in ability of bringing this project to life.
Of course, the experience of community's Go devs is valuable — we will provide the Grant update with Open API design to get the community feedback on it.

--
Added

After some explorations, I found some great frameworks written on Go, which allows to build REST API.
Using existing framework as a base for Open API (as we used Yii2 framework of PHP to build Enterprise API) is a good thought.
 
Last edited:
Secured
#23
After exploring Go frameworks, I tend to use Echo framework as a base for Open API.
– It's written on Go with all its advantages (fast, lightweight, easy Docker packaging)
– It has a lot of useful functions (similar with functions that I used while developing Enterprise API)
– It's extendable – that means I can easily write up additional middleware modules to implement all functions of Open API that needed, keeping Open API codebase robust and stable
 
Secured
#24
Hi Anton, Thank you for making this application.
I am asking this question of a number of ANOs submitttng grant applications:
Assuming the grant is funded please break down what will be delivered by this grant proposal funding and what will be delivered by your ANO efficiency set at 35% over the next 3 months?
 
Secured
#27
Personally, I believe that this API would ultimately be much better if you wrote it after spending some time getting to know Go. I would be interested to hear your opinion on that, and the opinion of others like @Paul Bernier and @AdamSLevy?
From my own experience (20+ years of PHP), as dissimilar as PHP and Go are, the most important part of an API is the structure, especially for REST apis. When I started learning Go, I was surprised by how easy the Go web architecture was, since it's part of the core package, and it's pretty intuitive coming from a web background.
 
Status
Not open for further replies.