Late [Matt York-Paul Bernier-DBGrow-Canonical Ledgers-1] FAT Smart Contracts 2 - Development

Chappie

Factomize Bot
This is your grant tracking thread. Below, you will find information from your original grant.

Grant Proposal
https://factomize.com/forums/thread...ers-1-fat-smart-contracts-2-development.2291/

Team Member or Entity Forum Username
User: Matt York
FCT address: FA35Kd1Ac1aQXEHPxYTR6jNDPuwVoh6APQQGKViceQTedyE7J2vV
FCT: 200

User: Paul Bernier
FCT address: FA2kWsMnSXTV9RS2yEGkEGP2yUFrawZQVj5Hp8WqPnZUUBFCpiaH
FCT: 200

Sponsor(s)
User: David Chapman
FCT address: FA3nsSjUy5uSkqMEug8t3VcehZn5w2ciSMpgqFEEsMRwMrHoa9k3
FCT: 300

ANO / Committee
Group: DBGrow
FCT address: FA3HSuFo9Soa5ZnG82JHqyKiRi4Pw17LxPTo9AsCaFNLCGkXkgsu
FCT: 7550

Group: Canonical Ledgers
FCT address: FA2xccSAfhGm5k4tPaXF9741xkQ52drWjoJodQhpPxDxepdqasMM
FCT: 5500

Total FCT Requested
13750

Start Date
2019-09-01

Completion Date
2019-12-01

Success Criteria
Standards and documentation clearly define the solution:
Questions from the community are addressed, included in docs, and fixes are made based on feedback

Securely implement a backwards compatible contract solution on FAT:
Existing FAT balances are not affected by implementing contracts. Contracts do not open vulnerabilities

Contract solution is open, decentralized, trustless, censorship resistant:
Anyone can use the contract system and FAT. All designs and code remain open source and free to use

Clients and tooling are made compatible with the new solution:
Fat-js and the fat-cli support the new features relating to contracts

Timelines and Milestones
Create Smart Contract FAT Standard - M1 - Week 3

Implement new validation rules into FAT Daemon in prep for SC support - M1 - Week 4

Support basic contract establishment mechanics in fatd - M2 - Week 6


Expose contracts to calls via signed transactions - M2 - Week 8

Update existing documentation - M3 - Week 10

Enable SC support in fat-js for new FAT Daemon functionality & validation - M3 - Week 11

Final integration testing - 4 - Week 13

Budget
Total Project Budget:

13,750 FCT


DBGrow: 7550 FCT


Canonical Ledgers: 5500 FCT


David Chapman (Factomize, Sponsor): 300 FCT


Matt York (Factom Inc, Technical Advisor): 200 FCT


Paul Bernier (Luciap, Technical Advisor): 200 FCT
 
I'm Sponsor for this grant and can confirm that a discussion medium has been setup for efficient communication and that all recipients have received their grant funds. The first milestone for the grant is due at week 3 and 4. I'll update the community next at week 3.

Milestone 1
- Create Smart Contract FAT Standard - Week 3
- Implement new validation rules into FAT Daemon in prep for SC support - Week 4
 
Adam finished a refactor of fatd this weekend. He added a database migration framework so that users will not ever have to rebuild databases, and discovered a potential optimization for syncing a bit faster which he has also implemented. In his words,
Well actually I couldn't help myself today. I had to get this improvement on sync speed implemented while it was fresh in my brain. I now have it syncing much faster. Previously fatd could only download the next DBlock after all EBlocks in the current DBlock are processed. This was causing a bottle neck. Most EBlocks we ignore, but then we have a few chains that we track that take a lot longer and block fatd from continuing to scan the next DBlock for new FAT chains or new EBlocks on existing FAT chains. this was becoming really obvious because I am syncing the entire chain once I find it, which takes longer than just processing the single EBlock. So that meant that when fatd finds a new chain, it has to wait for the entire chain to be processed before proceeding to the next DBlock.

But now I have de-coupled scanning dblocks from processing EBlocks.

So the DBlock scan can proceed while each tracked chain processes its eblocks sequentially in its own goroutine.

So when a new FAT chain is found, a new go routine is launched responsible for just that chain. it starts off syncing the entire chain in its own thread, and then just waits for new eblocks on a channel. So this is low overhead because most of the time the goroutine will be sleeping.

its working quite nicely

I'm just fixing up some potential race issues with the server.

This will be really helpful for avoiding one smart contract execution heavy chain from blocking other chains as well.
 
Apologies for the delay in my response.

Here is an honest and detailed assessment of the current state of smart contract development in FAT under this grant.

Overall, this project is probably about 70-80% completed. A number of challenges have resulted in the development which longer than expected. However I also bear responsibility for it being so behind. Since the New Year, I have lost some steam for this project and have not put the necessary hours in to get it finished. I take full personal responsibility for that. Without excusing it, one of the contributing factors to my discouragement has simply been the fact that, at least when it comes to coding, I am effectively working alone. I expected that at this point there would be at least one other developer working with me provided by DBGrow, but their hire didn't work out. At this point though I have renewed focus on getting this over the finish line in terms of basic MVP functionality in the next two weeks.

Below I have written a high level but detailed description of all of the functionality involved with smart contracts and its state of development. There are some details that I mention below that I don't think will be completed under this grant round, but I mention them for the sake of completeness. Happy to answer any further questions.

- Contract publication data format/protocol
Contracts consist of a Wasm binary and a JSON ABI defining the callable functions for the contract. This must be published to a chain so that any FAT chain may reference the contract code via a Chain ID. This allows contracts to be shared and reused more easily.

State of development: Done and tested. This is described in the standards FAT-107, which defines a data/file storage protocol, and FAT-104, which makes specific use of FAT-107 for publishing contracts. These standards are implemented in fairly well tested packages here:
https://github.com/Factom-Asset-Tokens/factom/tree/feature/smart-contracts/fat104
https://github.com/Factom-Asset-Tokens/factom/tree/feature/smart-contracts/fat107

Additionally, because contracts are shared between FAT Chains, I needed a new contract database that would be a single place to store downloaded contracts. The low level database functions for this are implemented and tested but as I have proceeded with integration, I have realized I probably need a slightly higher level mechanism to "resolve" and validate contracts. More on this later.

- Contract delegation data format/protocol and state transition &&
- Contract call data format/protocol and state transition

A published Contract is just a public record of the code. That alone does not define the use of the contract nor instantiate any state. An address within a FAT-0 chain may delegate control of itself, including any current balance, to a published contract. From that point on, the owner of the address may no longer directly move any funds from that address, and anyone can call the functions of the contract instantiated at that address. Funds may only be moved from such addresses of the contract code allows it. A contract may include code that "self-destructs" and returns control of the address back to the owner. If the contract does not contain such functionality, the funds are effectively locked forever.

Other addresses on the same or other chains may also delegate to the same contract code, but these addresses remain entirely independent of each other. They only happen to be using the same code, like many independent running instances of the same program.

State of development: Contract delegation and calls are defined in the updated FAT-0 standard. The data structures and stateless data validation for delegation and calls is implemented here:
https://github.com/Factom-Asset-Tokens/factom/blob/feature/smart-contracts/fat0/transaction.go

The state transition code for delegation and contract calls is a work in progress and has not been committed yet. The delegation code is done. The contract call code is WIP, (~60%). No thorough testing is done yet, but this code is a fairly straight-forward stitching together of well tested components. However, after I finish my first revision of this code, it will warrant an extensive refactor at some point to improve readability and maintainability.

- Contract metering
Calling a contract requires all FATd nodes to compile and execute the contract code. Left unchecked, a malicious actor could publish, delegate to, and call a contract that consumes a lot of resources. This would effectively stall all fatd nodes on that FAT chain. Some mechanism to limit execution is required to address this. I was surprised to discover that existing Wasm runtimes do not typically implement any metering of execution. Ethereum's research into using Wasm settled on an approach that involves static code analysis and injection of gas limit checks. Fortunately, Wasmer, the wasm runtime I am using, actually includes a naive implementation of this approach to metering. Unfortunately a number of hurdles needed to be overcome to be able to take advantage of this.

For one, the options for metering were not exposed in the Wasmer Go package API. Complicating this hurdle is the fact that Wasmer is written in Rust, and so can only be used in Go projects by means of both a Rust/C and C/Go API bridge libraries. The C library for Wasmer didn't expose the metering options available in the native Rust library either. So I had my work cut out for me to get this functionality exposed in Golang.

Fortunately I wasn't the only one who was attempting this and I was able to build on the work of others in some unmerged PRs on Wasmer. In the process I also discovered a critical bug in Wasmer's metering implementation. This would have allowed well crafted contracts to exceed execution limits, potentially undetected. Perhaps unwisely, I set out to patch this bug and it turned out to require a rewrite of the metering algorithm. Fortunately I got it working. This still needs some work to be merged into upstream Wasmer but it is working for FATd for now.

The other side of this problem is gas economics. It is one thing to be able to meter a wasm runtime, but how should FATd know what contract calls get to run for how long. Gas economics is super complex and there is no current solution officially proposed. I am interested in a hash-cash type approach to contract calls that requires the caller to mine a sufficiently low transaction hash that would correspond to some deterministic amount of execution. This is just an idea though and I haven't implemented anything to this end.

State of development: The work getting metering both working and accessible from Golang was a serious hurdle but it is done now and in use. Gas economics is an open problem that I don't expect to have solved for this grant round. While I can currently meter a contract. I still have not actually wired it up fully because I have no gas mechanism at this point. I plan to stub out the gas limit determination but leave it infinite or on some fixed configurable limit for now.

- Runtime functionality for all contracts
Aside from fixing the metering issues, I would say this runtime is the most complex/challenging aspect of this project. The runtime defines a set of external functions that a Wasm binary may call or link against. Wasm code can't affect anything that the runtime doesn't provide an API to access. So in order for Smart Contracts to do anything meaningful they must be provided with a well-defined runtime API that they can call. This allows a smart contract to programmatically call "send" or "self_destruct" or query context information about the current balance and transaction. The runtime also needs to handle out of gas exceptions and revert all changes.

State of development: The spec for the available runtime functions is defined in FAT-105. Although the spec needs to be updated to match some details of the current implementation, this is completely implemented and tested in this package:
https://github.com/Factom-Asset-Tokens/fatd/tree/feature/smart-contracts/internal/runtime

The one caveat to this is that the "getStorageAt" and "setStorageAt" APIs are not implemented. These were only just added by Devon (last week) but I do not plan to immediately implement them as I need to focus on completing other things first since this was just added. Without this API, however, smart contracts are very limited in their initial functionality as they will be effectively stateless, aside from the current balance and transaction data. So it will be a priority to implement once we have contracts initially working otherwise.

- FAT SC Toolchains
A way to build Wasm smart contracts compatible with this system is necessary. Wasm can be compiled from a variety of languages. Popular and well supported languages at the moment are C/C++ and Rust.

State of Development: Currently I have a simple way to build C smart contracts that is in use in my runtime tests.

- Command line tools for working with contracts

Currently the most basic way to interact with the FAT system is through `fat-cli`. Additional commands need to be added to fat-cli to support safely publishing contracts, delegating control of an address to a contract, and calling a contract. Additional tools to help vetting and testing contracts would be super helpful as well.

State of Development: This is not yet started. I'm only going to implement the basics. Additional tools for testing smart contracts won't be developed under this grant.

- API methods for working with contracts
Of course to support fat-cli, the API needs to be expanded to help interacting with smart contracts.

State of Development: This is not yet started.
 
Hello Factom Community,

Over the last several weeks we have been working to get the standards contained in our core FAT Standards Repository in line with our evolving design and implementation of Smart contracts. Over the course of the grant the decision to create the universal FAT-107 Factom Data Store Standard has changed the landscape of how the smart contract related standards mesh together.

In short, FAT-107 allows storing arbitrarily large files on Factom in a structured and validatable way that exposes them to access within the FAT ecosystem (FAT Daemon). To summarize FAT-107 - Factom Data Store (from the standard):
  • Secure - ChainID uniquely identifies the exact data within an optional application defined Namespace.
  • Extensible - Applications may define any arbitrary metadata to attach to the data store.
  • DOS proof - The data blocks may appear in any order, on any chain, and may be interspersed with unrelated or garbage entries.
  • Efficient - Clients only need to download the first modestly sized entry in the chain to discover the total data size, and exact number of entries that they will need to download. All required Data Block Entries can be discovered and then downloaded concurrently.
  • Arbitrary size - There is no theoretical limit to the size of data that may be stored. However most clients will likely enforce sane practical limits before downloading a file.
  • Censorship resistant - Commit all entries before revealing any to ensure that
    • a data store cannot be censored.
    • Optional compression - Use gzip, zlib, or none.
This standard will be the way that WASM contract binaries are published to Factom for use in FAT. Thank you to Adam Levy for your contribution with this insanely cool and useful standard.

The standards have now been rearranged so that the FAT-104 Contract Publication Standard uses FAT-107. FAT-104 describes a specialized set of validation rules that a FAT-107 store must follow to be considered a valid and usable WASM contract on FAT. This includes:
  • Definition and validation of an Application Binary Interface (ABI) in FAT-107 metadata that lays out the available functions that are callable in the contract, their argument types, their return type
  • Defines a 100KB total size limit for WASM contract binaries
  • Defines that an unlimited number of function signatures can be defined in the ABI, but due to the structure of FAT-107 and the design of Factom this necessarily is limited to 10KB in total size in the first entry including all necessary FAT-107 fields in the content.
  • Requires the binary file buffer retrieved from FAT-107 to itself be a valid WASM binary and recommends tools for doing this validation when implemented in code.
In addition to FAT-104 and FAT-107, FAT-105 Host Interface Standard has been created to define a collection of imported functions from the host that FAT smart contracts can call to perform actions on FAT such as sending tokens, getting the entryhash of the calling transaction, getting balances at addresses, etc. This standard will be implemented by the runtime that Adam is currently creating for contracts.

Alongside these three standards, our core token standard: FAT-0 Fungible Token Standard has been modified accordingly to define methods to publish WASM contracts to an address and call them using a specialized type of FAT-0 transaction. This allows people to call a contract with an amount of tokens used for payment.

We are currently working on implementing both FAT-107 and FAT-104 in our Javascript fat-js library so that users can publish files to Factom both for use in FAT contracts and for file storage in general. You can follow along with the developments on this branch. Currently a basic implementation of FAT-107 is almost complete, with validation of the datastore to match FAT-104 coming soon after.

Adam is in the process of modifying the FAT daemon to support RPC endpoints for working with contracts for use with the cli. Those endpoints will be supported by fat-js as well as soon they are available.

Thank you for your patience as we begin close out this grant, make our deliverables available, and document them so that the community can benefit from our developments.
 
Top