paillier encryption

fundamentals

  1. fundamental theorem of arighmetic
    the fundamental theorem of arithmetic, also called the unique factorization theorem and prime factorization theorem, states that every integer greater than 1 can be represented uniquely as a product of prime numbers, up to the order of the factors wiki
  2. Euler’s totient function
    In number theory, Euler’s totient function counts the positive integers up to a given integer n that are relatively prime to n. It is written using the Greek letter phi as \( \phi (n) \), and may also be called Euler’s phi function. In other words, it is the number of integers k in the range 1 ≤ k ≤ n for which the greatest common divisor gcd(n, k) is equal to 1. The integers k of this form are sometimes referred to as totatives of n. the collection of k is denoted by \( Z_{n}^{\ast } \), and \[ \phi(n) = |Z_n^{\ast }| \]
  3. if p is prime, then \( Z_p^{\ast } = Z_p \), \( \phi(p) = p-1 \)
  4. if p is prime, for any integer r, then \( \begin{align} \tag{0.1} \phi(p^{r}) =p^{r-1}\phi(p)=p^{r-1}(p-1)\end{align} \)
  5. Euler’s totient function is a multiplicative function, meaning that if two numbers m and n are relatively prime, then \(\phi(mn) = \phi(m)\phi(n)\)
  6. Euler’s product formula, it states
    \[ \phi(n) = n \prod_{p|n}^{}(1-\frac{1}{p}) \]
    where the product is over the distinct prime numbers dividing n.
  7. Euler’s theorem
    if \(a\) and \(n\) are coprime positive integers, and \( \phi(n)\) is Euler’s totient function, then \(a\) raised to the power \(\phi(n)\) is congruent to 1 modulo n; that is
    \[a^{\phi(n)} \equiv 1 \bmod n\]
  8. according to 7, we have \( a \cdot a^{\phi(n)-1} \equiv 1 \bmod n \). then
    \[ a^{-1} = a^{\phi(n)-1} \]
  9. Fermat’s little theorem
    Fermat’s little theorem states that if p is a prime number, then for any integer a, the number
    \(a^{p}-a \) is an integer multiple of p. In the notation of modular arithmetic, this is expressed as
    \[ a^{p} \equiv a \bmod p\]
  10. Binomial theorem
    it states
    \[ y = (1+n)^{x} = \sum_{k=0}^{x}\tbinom{x}{k}n^{k} = 1 + nx + \tbinom{x}{2}n^2 + …\]
    observe that, the higher degree could be divided by \(n^2\). we have
    \[ \begin{align} \tag{0.2} (1+n)^{x} \equiv 1 + nx \bmod n^2 \end{align} \]
    therefore, \( y - 1 \equiv nx \bmod n^2 \). then we have
    \[ x \equiv \frac{y-1}{n} \bmod n \].
    In paillier, later we define \( \begin{align} \tag{0.3} L(y) = \frac{y-1}{n} \end{align} \)
    therefore
    \[ L(y \bmod n^2) \equiv x \bmod n \]

Paillier

  1. key generation
    KeyGen() -> (pk, sk)
    randomly select two big prime numbers \(p, q\). it shoud satisfy \(gcd(pq, (p-1)(q-1)) =1 \), \(p\) and \(q\) should have similar bit length. let \( n = pq \), \(\lambda = lcm(p-1, q-1)\). randomly sample \( g \in Z_{n^2}^{\ast}\). to simplify, let \( g = n+1\). we have
    \[ pk=(n,g) \]
    \[ sk = (\lambda)\]

  2. encryption
    Enc(pk, m) -> c
    randomly sample \( r \in Z_{n}^{\ast}\), then also have \( r \in Z_{n^2}^{\ast}\), cypher is calculated
    \[ \begin{align} \tag{1.1} c = g^mr^n \bmod n^2 \end{align} \]

  3. Decryption
    Dec(sk, c) -> m
    Let \(L(x) = \frac{x-1}{n} \), we have message
    \[ \begin{align} \tag{1.2} m = \frac{L(c^{\lambda} \bmod n^2)}{L(g^{\lambda} \bmod n^2)} \bmod n \end{align}\]

  4. proof of correctness
    based on Eq(1), we have \[ \begin{align} \tag{1.3} c^{\lambda} \bmod n^2 = g^{m\lambda}r^{n\lambda} \bmod n^2 \end{align}\]
    where \( r^{n\lambda} \bmod n^2 \equiv 1 \bmod n^2\), which is proved by Carmichael theorem later on. then Eq(3) becomes
    \[ \begin{align} \tag{1.4} c^{\lambda} \bmod n^2 = g^{m\lambda}\bmod n^2 \end{align}\]
    since \( g = n+1\), we have
    \[ \begin{align} \tag{1.5} c^{\lambda} \bmod n^2 = (1+n)^{m\lambda}\bmod n^2 \end{align}\]
    According to Eq(0.2), we have
    \[ \begin{align} \tag{1.6} c^{\lambda} \bmod n^2 = 1 + nm\lambda \bmod n^2 \end{align}\]
    \[ \begin{align} \tag{1.7} g^{\lambda} \bmod n^2 \equiv (1+n)^{\lambda} \bmod n^2 = 1 +\lambda n \bmod n^2 \end{align}\]
    therefore, based on definition given by Eq(0.3) we have
    \[ \begin{align} \tag{1.8} L(c^{\lambda} \bmod n^2) = \frac{c^{\lambda}-1}{n} \bmod n^2 \end{align} \]
    Substitute Eq(1.6) into Eq(1.8), we have
    \[ \begin{align} \tag{1.9} L(c^{\lambda} \bmod n^2) = m\lambda \bmod n^2 \end{align} \]
    Further, we have
    \[ \begin{align} \tag{1.10} L(g^{\lambda} \bmod n^2) = \frac{g^\lambda -1}{n} \end{align} \]
    Sub Eq(1.7) into Eq(1.10), we have
    \[ \begin{align} \tag{1.11} L(g^{\lambda} \bmod n^2) = \frac{\lambda n}{n} \equiv \lambda \bmod n^2\end{align} \]
    At last, Eq(1.2) becomes (bu sub Eq1.9 and Eq1.11)
    \[ \begin{align} m = \frac{L(c^{\lambda} \bmod n^2)}{L(g^{\lambda} \bmod n^2)} \bmod n = \frac{m \lambda}{\lambda} \equiv m \bmod n \end{align}\]
    proved!!!

  5. Carmichael theorem
    In number theory, a branch of mathematics, the Carmichael function \(λ(n)\) of a positive integer n is the smallest positive integer m such that \(a^{m}\equiv 1{\pmod {n}}\) (similar but different from Euler’s totient function). Carmichael’s λ function, the reduced totient function, and the least universal exponent function
    carmichael theorem

    let \( n = pq\), where p and q are prime numbers; \( \phi(n)\) is the Euler’s totient function. Let \(\lambda(n)\) denotes carmichael function. We have \(\phi(n)=(p-1)(q-1)\) and \( \lambda(n)=\phi(n) = (p-1)(q-1)\).

Since \( |Z_{n^2}^{\ast}| = \phi(n^2) = n \phi(n)\) (according to Eq(0.1)). Thereby, for any \( w \in Z_{n^2}^{\ast}\)
\[ \begin{align} \tag{1.12} w^{n\phi(n)} \equiv w^{n\lambda} \equiv 1 \bmod n^2 \end{align}\]

\[ \begin{align} \tag{1.13} w^{\lambda} \equiv 1 \bmod n \end{align}\]
Eq(1.13) is just Carmichael’s function

Based on Carmichael’s theorem
\[ \lambda(n^2) = lcm(\lambda(q^2),\lambda(p^2)) = lcm(\phi(q^2),\phi(p^2)) = lcm(q(q-1), p(p-1)) = pq(lcm(p-1, q-1)) = n\lambda(n) \]
therefore, we have

\[w^{\lambda(n^2)} = w ^{n\lambda} \equiv 1 \bmod n^2\]

  1. Addition homomorphic
    homomorphic addition

  2. Multiplication homomorphic
    homomorphic multiplication

references

two party ecdsa

overview

this post is my reading summary of paper Yehuda Lindell 2017: Fast secure two-party ecdsa signing. the implementation could be found in tss-lib (golang), zengo’s library (rust).

Unlike other schemes like RSA, Schnorr signatures and more, it is particularly hard to construct efficient threshold signature protocols for ECDSA as there is an inverse computaion of \( k \).

In this paper, we consider the specific case of two parties (and thus no honest majority) and construct a protocol that is approximately two orders of magnitude faster than the previous best.

Comparing ECDSA signing to EC-Schnorr signing

In both cases, the public verification key is an elliptic curve point \( Q \) and the private signing key is \( x \) such that \( Q = x \cdot G \), where \( G \) is the generator point of an EC group of order \( q \).
schnorr ecdsa comparison

Observe that Schnorr signing can be easily distributed since the private key \( x \) and the value k are both used in a linear equation. In contrast, in ECDSA signing, the equation for computing \( s \) includes \( k^{-1} \). Now, given shares \(k_1\), \(k_2\) such that \(k_1 + k_2 = k \bmod q\) .It is very difficult to compute \(k_1^{\prime}\), \(k_2^{\prime}\) such that \(k_1^{\prime} + k_2^{\prime} = k^{-1} \bmod q\)

two-party protocols for ECDSA signing use multiplicative sharing of \( x \) and of \( k \). That is, the parties hold \(x_1\), \(x_2\) such that \(x_1 \cdot x_2 = x \bmod q\), and in each signing operation they generate \(k_1\), \(k_2\) such that \(k_1 \cdot k_2 = k \bmod q\). This enables them to easily compute \(k^{-1}\) since each party can locally compute \(k_i^{\prime} = k_i^{-1} \bmod q\), and then \(k_1^{\prime}\), \(k_2^{\prime}\) are multiplicative shares of \(k^{-1}\). The parties can then use additively homomorphic encryption – specifically Paillier encryption – in order to combine their equations. For example, \(P_1\) can compute \(c_1 = Enc_{pk}(k_1^{-1} \cdot H(m))\) and \(c_2 = Enc_{pk}(k_1^{-1} \cdot x_1 \cdot r)\) . Then, using scar multiplication (denoted ⊙) and homomorphic addition (denoted ⊕), \( P_2 \) can compute \( (k_2^{-1} ⊙ c_1 ) ⊕ [( k_2^{-1} \cdot x_2) ⊙ c_2 ]\), which will be an encryption of

paillier encryption

However, proving that each party worked correctly is extremely difficult. For example, the first party must prove that the Paillier encryption includes \(k_1^{-1}\) when the second party only has \(R_1 = k_1 \cdot G\). it must prove that the Paillier encryptions are to values in the expected range, and more. This can be done, but it results in a protocol that is very expensive.

their results

[WIP]

references

MPT

trie

a trie, also called prefix tree, is a type of k-ary search tree. These keys are most often strings, with links between nodes defined not by the entire key, but by individual characters. In order to access a key, the trie is traversed depth-first, following the links between nodes, which represent each character in the key.

prefix trie
In general, the nodes of a Trie look like this:

1
[ [Ia, Ib, … I*], value]

[Ia, Ib, … I*] is the index array of the node, which takes the next character in the key as the index, and each element I* points to the corresponding child node. value represents the value

MPT

Each block of Ethereum contains three MPT trees, respectively

  • Transaction tree
  • Receipt tree
  • State tree

In the figure below are two block headers, where state root, tx root receipt root stores the roots of the three trees, and the second block shows when the data of account 175 changes (27 -> 45). Only need to store 3 nodes related to this account, and the data in the old block can still be accessed normally. (This is somewhat similar to the implementation of an immutable data structure in a functional programming language.) The detailed structure is

state reference

  • use []byte as key, other than string
  • nibble: the smallest unit of the key type (4 bit)
  • Use hashes to refer to nodes instead of memory pointers

there are two types of node: full nodes (fullNode) and short nodes (shortNode). Full nodes have 17 elements, while shortNode nodes have two elements. Their schematic expressions are as follows

1
2
fullNode: [i0, i1, i2, … i15, hash]  
shortNode: [ [k0, k1, … kn], hash ] // first element is an array

if the hash pointing to a value, it is a leaf node; if pointing another node, a non leaf node. shortNode contains extension and leaf node. full node is branch node.

mpt

Use the upper 4 bits of the first byte of the []byte value composed of nibbles as storage flag. The 0th bit stores the parity information, and the 1st bit stores the type represented by the value

hex char bits pointing to odd/even 2nd niddle padding
0 0000 node even no
1 0001 node odd yes
2 0010 value even no
3 0011 value odd yes

this encoding method is only used when accessing the database. After reading into memory, the key is directly stored in []byte type

In the trie module, there is a Database object, which you can understand as a cache layer of the underlying database. In actual use, the Trie object uses the Database as a database. However, the important function of Database is not caching, but the reference counting of node data during caching, and provides Database.Reference and Database.Dereference to reference and dereference a trie node. If the reference count of a node becomes 0, the node will be deleted from memory, so it will not be written to the real database

reference

understanding of kademlia protocol

Introduction

Kademlia, a peer-to-peer distributed hash table(DHT). Some other DHT techniques are Chord. In the Kad network, each node has a unique 160-bit ID (ETH account address is 20 bytes, which is 160 bits) <key,value> pairs are stored in nodes whose ID is ‘close’ to the key for some notion of closeness.

system description

Kad effectively treats nodes as leaves in a binary tree, with each nodes’s position determined by the shortest unique prefix of its ID. Figure 1 shows the position of a node with unique prefix 0011 in an example.

Figure 1
for any given node, the binary tree is divided into a series of successively lower subtrees that don’t contain the node.
The highest-level subtree is composed of the other half of the whole tree that does not contain itself; the next level of subtree is composed of the remaining half that does not contain itself; and so on, until the complete tree is split into n subtrees. As shown in the figure, the part contained by the dotted line is the subtree of node 0011.

if there is at least one node knwon in each subtree (in total, at least n nodes), a recursive routing algorithm can be used to reach any node within the binary tree. Figure 2 shows an example of node 0011 locating node 1110 by successively querying the best node it knows of to find contacts in lower and lower subtrees; finaly the lookup converges to the target node.

Figure 2

node distance: XOR metric

Node’s id is 160 bit. Keys are also 160 bit. The Kad algorithm uses an XOR operation to calculate the distance between nodes.
d(x,y) = x XOR y

node state

for each 0 <= i < 160, every node keeps k <IP address, UDP port, node ID> triples (as a list) for nodes of distance between 2^i and 2^i+1 from itself. it is called k-buckets.
each k-bucket is kept sorted by time last seen–least recently seen node at the head, most-recently seen at the tail.
for small values of i, the k-buckets will generally be empty (as no approriate nodes will exist).

The K value mentioned here is a system-level constant (must be an even number). It is set by the software system using Kad (for example, the Kad network used by BT download, K is set to 8).

update of k bucket

There are mainly the following ways to update the K bucket:

  • Actively collect nodes: Any node can initiate a FIND_NODE (query node) request to refresh the node information in the K-bucket.
  • Passive collection node: When receiving requests from other nodes (such as: FIND_NODE, FIND_VALUE), it will add the node ID of the other party to a certain K-bucket.
  • Detect invalid nodes: By initiating a PING request, determine whether a node in the K-bucket is online, and then clean up which offline nodes in the K-bucket.

The update principle is to insert the latest node into the tail of the queue, and not to remove the online nodes in the queue.

According to the K-bucket update principle, nodes with a long online time are more likely to remain in the K-bucket list. Therefore, by keeping the nodes with a long online time in the K-bucket, Kad will significantly increase the number of nodes in the K-bucket. it can defend against DOS attacks to a certain extent, because only when the old node fails, Kad will update the K bucket information, which avoids flooding routing information through the addition of new nodes
probability of continuous online agains onlie duration

RPC method

The Kademlia protocol includes four remote RPC operations: PING, STORE, FIND_NODE, FIND_VALUE.

  • PING probes a node to see if it is online.
  • STORE instructs a node to store a <key,value> pair for later retrieval.
  • FIND_NODE takes a 160bit ID as an argument. The receiver of this operation returns the (IP address, UDP port, Node ID) information of K nodes that it knows are closer to the target ID. The information of these nodes can be obtained from a single K-bucket, or from multiple K-buckets. In either case, the receiver will return the information of K nodes to the operation initiator.
  • The FIND_VALUE operation is similar to the FIND_NODE operation, with one exception. if the RPC receipients has received a STORE RPC for the key, it just returns the stored value.

node lookup

Kad participant must locate the k closest nodes to some given node ID. Kad employs a recursive algorithm for node lookups. It recursively send FIND_NODE requests to \alpha (out of k) closest nodes it knows of.

find a <key,value> pair

to find a <key,value> pair, a node starts by performing a lookup to find the k nodes with IDs closest to the key. However, value lookups use FIND_VLAUE rather than FIND_NODE RPCs.

node join

to join the network, a node u must have a contact to an already participating node w.

routing table

references

rust async

I/O

I/O:在计算机中指Input/Output。由于程序和运行时数据是在内存中驻留,由cpu来执行,涉及到数据交换的地方,通常是磁盘、网卡等,就需要IO接口

I/O 模型

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
@startmindmap
+ **I/O模型**
++ 同步I/O
'tag::details[]
+++_ 阻塞I/O (BIO)
+++_ 非阻塞I/O (NIO)
+++_ I/O多路复用
+++_ 信号驱动I/O
'end::details[]
++ 异步I/O
'tag::details[]
+++_ linux (AIO, io_uring)
+++_ windows (IOCP)
'end::details[]
@endmindmap

同步阻塞

  • 当用户线程发起IO请求后,会进行系统调用(system call)来让内核(Kernel)进行IO操作(系统调用是用户空间和内核空间的一个通道);
  • 此时用户线程阻塞,等待内核将数据准备好;
  • 内核将数据准备好后会将数据从内核空间拷贝到用户空间,并返回给用户线程结束阻塞。

同步非阻塞

  • 由用户线程发起IO请求,进行系统调用来让内核进行IO操作;
  • 此时如果内核没有准备好数据则会直接返回error,并不会阻塞用户线程,用户线程可以重复发起IO请求;
  • 当用户线程发起请求并且内核已经将数据准备好后,会将数据从内核空间拷贝到用户空间(这个过程是需要阻塞用户线程的),返回给用户。

同步多路复用

  • 用户线程调用select后进行系统调用(内核会监视所有select负责的socket)
  • 当用户将数据准备好后就会返回,并通知用户线程进行读取操作,此时内核将数据拷贝到用户空间并返回。此时用户线程被阻塞;

异步I/O

  • 用户线程进行aio_read,进行系统调用切换到内核;
  • 内核立即返回,并不会阻塞用户线程;
  • 内核准备好数据后会将数据从内核空间拷贝到用户空间并通知用户线程(发送信号)操作已完成。

流程图

同步blocking I/O

1
2
3
4
5
6
7
8
9
10
11
12
@startuml Test Diagram

participant "application" as app
participant "kernel" as kernel

activate app
activate kernel
app -> kernel: syscall: Read recvfrom
kernel -> kernel: wait for data (no datagram ready)
kernel -> kernel: copy datagram to user (datagram ready)
kernel -> app: return
@enduml

I/O多路复用

异步编程

the Future Trait

A Future is an asynchronous computation that can produce a value. A simplified version of the future trait might look something like this

1
2
3
4
5
6
7
8
9
10

trait SimpleFuture {
type Output;
fn poll(&mut self, wake: fn()) -> Poll<Self::Output>;
}

enum Poll<T> {
Ready(T),
Pending,
}

For example, consider the case where we want to read from a socket that may or may not have data available already.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
pub struct SocketRead<'a> {
socket: &'a Socket,
}

impl SimpleFuture for SocketRead<'_> {
type Output = Vec<u8>;

fn poll(&mut self, wake: fn()) -> Poll<Self::Output> {
if self.socket.has_data_to_read() {
// The socket has data -- read it into a buffer and return it.
Poll::Ready(self.socket.read_buf())
} else {
// The socket does not yet have data.
//
// Arrange for `wake` to be called once data is available.
// When data becomes available, `wake` will be called, and the
// user of this `Future` will know to call `poll` again and
// receive data.
self.socket.set_readable_callback(wake);
Poll::Pending
}
}
}

the real Future trait and how it is different

1
2
3
4
5
6
7
8
9
10
trait Future {
type Output;
fn poll(
// Note the change from `&mut self` to `Pin<&mut Self>`:
self: Pin<&mut Self>,
// and the change from `wake: fn()` to `cx: &mut Context<'_>`:
cx: &mut Context<'_>,
) -> Poll<Self::Output>;
}

The first change you’ll notice is that our self type is no longer &mut Self, but has changed to Pin<&mut Self>. We’ll talk more about pinning in a later section, but for now know that it allows us to create futures that are immovable. Immovable objects can store pointers between their fields, e.g. struct MyFut { a: i32, ptr_to_a: *const i32 }. Pinning is necessary to enable async/await.

Secondly, wake: fn() has changed to &mut Context<’_>. In SimpleFuture, we used a call to a function pointer (fn()) to tell the future executor that the future in question should be polled. However, since fn() is just a function pointer, it can’t store any data about which Future called wake.

In a real-world scenario, a complex application like a web server may have thousands of different connections whose wakeups should all be managed separately. The Context type solves this by providing access to a value of type Waker, which can be used to wake up a specific task.

task wakeups with Waker

Waker provides a wake() method that can be used to tell the executor that the associated task should be awoken. When wake() is called, the executor knows that the task associated with the Waker is ready to make progress, and its future should be polled again.

referencs

csdn blog

geth evm source analysis

overall

the code is under path core/vm
overview of the whole evm module evm

the core is EVM struct (in evm.go), with main function in creating or call contract. a new EVM object is created every time when processing a transaction. inside the EVM struct, the main items are Interpreter, and StateDB (for state persistence). Interpreter loops through contract call instructions.Before each instruction is executed, some checks are performed to ensure sufficient gas and stack space. actual instruction execution code is recorded in JumpTable (256 sized array of operation)

depending on the version of Ethereum, JumpTable may point to four different instruction sets: constantinopleInstructionSet, byzantiumInstructionSet, homesteadInstructionSet, frontierInstructionSet. Most of the instructions of these four sets of instruction sets are the same, but as the version is updated, the new version supports more instruction sets than the old version.

evm

The EVM object is the most important object exported by the evm module, which represents an Ethereum virtual machine

creating evm

Every time a transaction is processed, an EVM is created to execute the transaction. This is reflected in the function ApplyTransaction (core/state_processor.go)

creating contract

If the to of the transaction is empty, it means that this transaction is to create a contract, so call EVM.Create to perform related functions

  • CREATE
    1
    contractAddr = crypto.CreateAddress(caller.Address(), evm.StateDB.GetNonce(caller.Address()))
  • CREATE2
    1
    2
    codeAndHash := &codeAndHash{code: code}
    contractAddr = crypto.CreateAddress2(caller.Address(), salt.Bytes32(), codeAndHash.Hash().Bytes())
    during create contract, an object Contract is created. A Contract object contains and maintains the necessary information during the execution of the contract, such as the contract creator, the address of the contract itself, the remaining gas of the contract, the contract code and the jumpdests record of the code.

then, it invokes below method to create contract

1
2
ret, err := evm.interpreter.Run(contract, nil, false)
evm.StateDB.SetCode(address, ret)

If the operation is successful and the contract code does not exceed the length limit, call StateDB.SetCode to store the contract code in the contract account of the Ethereum state database. Of course, the storage needs to consume a certain amount of gas.

You may wonder why the stored contract code is the return code after the contract runs, not the data in the original transaction (ie Transaction.data.Payload). This is because when the contract source code is compiled into binary data, in addition to the original code of the contract, the compiler also inserts some codes to perform related functions. For creation, the compiler inserts code that executes the contract’s “constructor” (that is, the contract object’s constructor method). Therefore, when the binary compiled by the compiler is submitted to the Ethereum node to create a contract, the EVM executes this binary code, in fact, it mainly executes the constructor method of the contract, and then returns other codes of the contract, so there is a ret variable here Stored in the state database as the actual code of the contract

call contract

The EVM object has three methods to implement the call of the contract, they are:

  • EVM. Call
  • EVM. CallCode
  • EVM. DelegateCall
  • EVM.StaticCall
    The basic contract call function implemented by EVM.Call is nothing special. The following three calling methods are the differences compared with EVM.Call. So here we only introduce the particularity of the last three calling methods

EVM.CallCode & EVM.DelegateCall

The existence of EVM.CallCode and EVM.DelegateCall is to realize the characteristics of the “library” of the contract. If the code written by solidity is to be called as a library, it must be deployed on the blockchain to obtain a fixed address like a normal contract. , other contracts can call the method provided by this “library contract”. But the contract also involves some unique attributes, such as the caller of the contract, contract address, the amount of ether it owns, etc. If we directly call the code of the “library contract”, these properties must be the properties of the “library contract” itself, but this may not be what we want

as an example

1
A -> contractB - delegateCall -> libC

EVM.DelegateCall sets the caller (msg.sender) of the “library contract” (libC) to A, rather than contractB; sets the address of the “library contract” (libC) to contractB.
EVM.CallCode is similar to EVM.DelegateCall. the only difference is that EVM.CallCode only change the address of the “library contract” (libC) to contractB, without chanding the caller to A.
EVM.StaticCall is similar to EVM.Call, the only difference is that EVM.StaticCall does not allow execution of instructions that modify permanently stored data

during contract call, it first check whether it is precompiled contract. some precompiled contracts are

  • common.BytesToAddress([]byte{1}): &ecrecover{},
  • common.BytesToAddress([]byte{2}): &sha256hash{},
  • common.BytesToAddress([]byte{3}): &ripemd160hash{},
  • common.BytesToAddress([]byte{4}): &dataCopy{},

EVMInterpreter

The interpreter object EVMInterpreter is used to interpret and execute specified contract instructions. However, note that the actual instruction interpretation and execution is not really completed by the interpreter object, but by the operation object JumpTable. The interpreter object is only responsible for parsing instruction codes one by one, and then obtains the corresponding operation object, and check objects such as the stack before calling the operation.execute function that actually executre the instruction. It can also be said that the interpreter object is only responsible for the scheduling of interpretation.

execution layout

layout

intrinsic gas

The intrinsic gas for a transaction is the amount of gas that the transaction uses before any code runs. It is a constant transaction fee (currently 21000 gas) plus a fee for every byte of data supplied with the transaction (4 gas for a zero byte, 68 gas for non-zeros). These constants are all currently defined for geth in params/protocol_params.go.

gas cost

the gas cost of each instruction is stored in JumpTable.operation.dynamicGas or JumpTable.operation.constantGas. constantGas means the operation gas cost is a fixed constant. dynamicGas is a function which will return gas during runtime.

In fact, not only the interpretation and execution of the instruction itself consumes gas, but also consumes gas when using memory storage and StateDB permanent storage. For most instructions, the latter two are not used (memory & storage), but for some instructions (such as CODECOPY or SSTORE), their gasCost function will take memory and StateDB usage into account.

a method memoryGasCostis used to calculate the gas consumption of memory usage. only when the required space size exceeds the current space size, the excess part needs to consume gas.

JumpTable

jumptable is 256 sized array of operation

jump instruction

Among the instructions of the contract, there are two jump instructions (excluding CALL): JUMP and JUMPI. Their special feature is that the first instruction of the target address after the jump must be JUMPDEST

1
2
3
4
5
6
7
8
9
10
11
func opJump(pc *uint64, interpreter *EVMInterpreter, contract *Contract, memory *Memory, stack *Stack) ([]byte, error) {
pos := stack.pop()
if !contract.validJumpdest(pos) {
nop := contract.GetOp(pos.Uint64())
return nil, fmt.Errorf("invalid jump destination (%v) %v", nop, pos)
}
*pc = pos.Uint64()

interpreter.intPool.put(pos)
return nil, nil
}

A function interprets and executes the JUMP instruction. The code first fetches a value from the stack as the jump destination. This value is actually an offset relative to field 0 of the contract code. Then the code will call Contract.validJumpdest to determine whether the first instruction of this destination is JUMPDEST, if it is not, an error will occur.

To judge whether the first instruction of the destination is JUMPDEST, two points must be guaranteed: first, its value is the value of the opcode of the JUMPDEST instruction; second, it is an instruction, not ordinary data.

Let’s introduce how Contract.validJumpdest works. In addition to comparing opcode (this is very simple), Contract will also create a bit vector object (ie bitvec, bit vector). This object will analyze the contract instructions from the beginning to the end. If the byte at a certain offset of the contract belongs to ordinary data, the “bit” corresponding to the offset value in bitvec is set to 1, and if it is an instruction, it is set to 0. In Contract.validJumpdest, it is judged whether this is a normal instruction by checking whether the “bit” of the offset value of the jump destination in this bit vector object is 0

references

geth_fine_tune

If we’re a full node on mainnet without –cache specified, bump default cache allowance
ctx.Set(utils.CacheFlag.Name, strconv.Itoa(4096))

rust reflect and macro

reflect

trait object

Rust provides dynamic dispatch through a feature called trait objects. Trait objects, like &Foo or Box, are normal values that store a value of any type that implements the given trait, where the precise type can only be known at runtime. more details can be found on [references 1]

any

This module (std::any) contains the Any trait, which enables dynamic typing of any 'static type through runtime reflection. It also contains the Provider trait and accompanying API, which enable trait objects to provide data based on typed requests, an alternate form of runtime reflection.
Any itself can be used to get a TypeId

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
use std::fmt::Debug;
use std::any::Any;

fn log<T: Any + Debug>(value: &Any) {
let value_any = value as &dyn Any; // &dyn Any (a borrowed trait object), Note that &dyn Any is limited to testing whether a value is of a specified concrete type, and cannot be used to test whether a type implements a trait.

match value_any.downcast_ref::<String>() {
Some(val_str) -> {
// do with string
},
None => {
//
}
}
}

porpular crates using Any

  • oso
    The Oso Library is a batteries-included framework for building authorization in your application
  • bevy
    A refreshingly simple data-driven game engine built in Rust

macro

rust compile process

rust compile process

front end: rustc

  1. lexical analysis: Code Text -> TokenStream
  2. syntax analysis: TokenStream -> AST (abstract syntax tree)
  3. semantic analyzer:
    AST -> HIR (High-Level Intermediate Representation) -> Type HIR (static type analysis, syntactic desugar, e.g for changed to loop) -> MIR: (Mid-Level Intermediate Representation, scope, reference & borrow check)

back end: LLVM

LLVM IR -> machine code

macros in compiling

  • declarative macros: TokenStream - expand -> TokenStream
  • procedule macros: self defined AST with the help or third party crate such as syn, quote

declarative macro: macro_rules!

Declarative macros allow us to write match-like code. The match expression is a control structure that receives an expression and then matches the result of the expression with multiple patterns. Once a pattern is matched, the code associated with the pattern will be executed

1
2
3
4
5
6
7
8
9
10
11
12
#![allow(unused)]
fn main() {
match target {
match_pattern_1 => expr_1,
match_pattern_2 => {
statement1;
statement2;
expr_2
},
_ => expr_3
}
}

example 1, simplified vec!

below example use macro_rules to implement a simplified version of vec!

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
#[macro_export]
macro_rules! vec {
( $( $x:expr ),* ) => {
{
let mut temp_vec = Vec::new();
$(
temp_vec.push($x);
)*
temp_vec
}
};
}

#![allow(unused)]
fn main() {
let v: Vec<u32> = vec![1, 2, 3];
}

example 2, unless

1
2
3
4
5
6
7
8
9
10
11
12
13
macro_rules! unless {
( ($arg:expr) => $branch:tt ) => ( if !$arg {$branch};);
}

fn cmp(a:i32, b:i32) {
unless!{
(a>b) => {println!("{} < {}", a,b);}
}
}
fn main() {
cmp(1,2); /// print "1<2" as the condition is true !(a>b)
cmp(3,2); /// print nothing
}

example 3, HashMap

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
macro_rules! hashmap {
// match for "a" => 1, "b" => 2,
( $($key:expr => $value:expr,)* ) =>
{ hashmap!($($key => $value),*) }; // recuisive
// match for "a" => 1, "b" => 2
( $($key:expr => $value:expr),* ) => {
{
let mut _map = ::std::collections::HashMap::new();
$(
_map.insert($key, $value);
)*
_map
}

};
}

macro_rules! hashmap_equivalent {
( $($key:expr => $value:expr),* $(,)*) => {
{
let mut _map = ::std::collections::HashMap::new();
$(
_map.insert($key, $value);
)*
_map
}

};
}
fn main() {
let map = hashmap!{
"a" => 1,
"b" => 2, // with or without ,
};
let map_2 = hashmap_equivalent!{
"a" => 1,
"b" => 2, // with or without ,
};
}

metavariables

  • item: an Item
  • stmt: a Statement without the trailing semicolon (except for item statements that require semicolons)
  • expr: an Expression
  • ty: a Type
  • ident: an IDENTIFIER_OR_KEYWORD or RAW_IDENTIFIER
  • path: a TypePath style path
  • tt: a TokenTree (a single token or tokens in matching delimiters (), [], or {})
  • meta: an Attr, the contents of an attribute
  • lifetime: a LIFETIME_TOKEN
  • vis: a possibly empty Visibility qualifier
  • literal: matches LiteralExpression
    details to be found here

procedures macro

references

  1. rust trait object

rust frequently used crates

tokio-trace -> tracing

contexts, multi threads
causality -
structured diagnostics ( no grep)

tracing is part of tokio, tokio not requied

spans: a perido of time, entered and exited
events: singular moment in time
subscriber: collect trace

rust cargo all in one

useful cmd

1
2
cargo new ${crate_name} --lib ## create a lib crate
cargo build --verbose ## print out each rustc invocation

specifying dependencies

  • specifying dependencies from crates.io

    1
    2
    [dependencies]
    time = "0.1.12"
  • specifying dependencies from other registries

    1
    2
    [dependencies]
    some-crate = { version = "1.0", registry = "my-registry" }
  • specifying dependencies form git repositories

    1
    2
    [dependencies]
    regex = { git = "https://github.com/rust-lang/regex.git" }
  • path dependencies

    1
    2
    [dependencies]
    hello_utils = { path = "hello_utils" }
  • platform specific dependencies

    1
    2
    3
    4
    5
    [target.'cfg(unix)'.dependencies]
    openssl = "1.0.1"

    [target.'cfg(target_arch = "x86")'.dependencies]
    native-i686 = { path = "native/i686" }

    Like with Rust, the syntax here supports the not, any, and all operators to combine various cfg name/value pairs.
    If you want to know which cfg targets are available on your platform, run rustc --print=cfg from the command line.
    If you want to know which cfg targets are available for another platform, such as 64-bit Windows, run rustc --print=cfg --target=x86_64-pc-windows-msvc

  • custom target specifications

    1
    2
    3
    4
    5
    6
    [target.bar.dependencies]
    winhttp = "0.4.0"

    [target.my-special-i686-platform.dependencies]
    openssl = "1.0.1"
    native = { path = "native/i686" }
  • development dependencies
    [dev-dependencies]
    Dev-dependencies are not used when compiling a package for building, but are used for compiling tests, examples, and benchmarks.
    These dependencies are not propagated to other packages which depend on this package.

    1
    2
    [target.'cfg(unix)'.dev-dependencies]
    mio = "0.0.1"
  • build dependencies

    1
    2
    [build-dependencies]
    cc = "1.0.3"

    The build script does not have access to the dependencies listed in the dependencies or dev-dependencies section. Build dependencies will likewise not be available to the package itself unless listed under the dependencies section as well.

  • choosing features

    1
    2
    3
    4
    5
    [dependencies.awesome]
    version = "1.3.5"
    default-features = false # do not include the default features, and optionally
    # cherry-pick individual features
    features = ["secure-password", "civet"]
  • renaming dependencies in Cargo.toml
    When writing a [dependencies] section in Cargo.toml the key you write for a dependency typically matches up to the name of the crate you import from in the code. For some projects, though, you may wish to reference the crate with a different name in the code regardless of how it’s published on crates.io. For example you may wish to: Avoid the need to use foo as bar in Rust source.
    (more to be found in original book)

references

cargo book