How Zero Knowledge Proofs, Aggregation Layers, and Light Nodes Can Improve Web3 Experience and Structure?
A Deep Dive Into Web3 Infrastructural Needs For A Decentralized, Censorship-Resistant, Secure, and Private Future
In the great crypto circus, liquidations and Chapter 11 filings have become so common that people consider them almost synonymous with cryptocurrencies. But is that really the case? All of the multi-billion dollar hedge funds, firms, market makers, and trading exchanges that went bankrupt had one thing in common — opacity in their balance sheets enabled by an opaque ‘centralized’ system rather than a system of transparent ‘decentralized’ ledgers. To create truly decentralized, accessible, self-custodial, and censorship-resistant systems that can withstand attacks and fraud, we need more than a system of conventional settlement layers. This research explores the intricacies of Web3 infrastructure or middleware to identify existing challenges and future opportunities. We shed light on the utility of aggregation layers; various types of nodes; and Zero Knowledge Proofs (ZKPs) to further the mission of decentralization across the crypto ecosystem. This research is carried out and authored by 0xlol, and edited by ThePsychGuy. They can be reached on Twitter at 0xlol and ThePsychGuy respectively.
tldr;
- A robust and resilient Web3 Infrastructure is necessary to sustain decentralization, censorship resistance, transparency, and easy accessibility for crypto.
- Light Nodes or Clients make decentralization accessible to masses.
- Zero Knowledge Proofs (ZKPs) add massive value to almost every component of the Web3 infrastructure establishing it as a crucial piece of the middleware.
- ZKPs help establish a trust-minimized environment resulting into the enhancement of on-chain privacy, UX, security, and decentralization.
- Aggregation layers are one of the most important pieces of Web3 infrastructure puzzle when it comes to user and developer experience.
A Brief Introduction to Web3 Infrastructure
Web3’s infrastructure has continually evolved for the better and has the potential to generate tremendous value. It has obvious supply-and-demand drivers required to achieve sustainability and profitability. The ideal system needs to be censorship-resistant, decentralized, efficient, secure, and provide a great user experience. By Web3 infrastructure, we refer to all the components that can facilitate the workings of a decentralized ledger system. It’s a full stack required for the proper functioning of the blockchains.
In order to understand this concept better, let’s explore it in the context of Ethereum. Ethereum is a decentralized computer or a distributed network of computers, with each individual computer referred to as a node. These nodes utilize software and work together in order to verify transactions and keep track of the decentralized ledger. Even though these nodes can be said to form the backbone of the blockchain, they cannot be considered as the full stack required for its functioning. Other supporting systems that aid various functions (user interaction, data storage, etc.) of it together constitute a Web3 infrastructure.
There are three different types of nodes: full, light, and archive. A full node stores all the blockchain data up to the latest 128 blocks, validates blocks and verifies their states. An archive node has the same capabilities as that a full node, plust the ability to store historical block data dating beyond the latest 128 blocks. It makes them suitable for querying historical blockchain data. Finally, light nodes don’t record any block and instead, store the block headers. They also can’t validate blocks, but that shouldn’t make you doubt their utility! As we will in due time, they can provide ‘easy access to decentralization’ and don’t require powerful or expensive hardware.
The graphic above describes the typical anatomy of Web3 infrastructure. While the web of intricacies remains hidden from the end-user, much activity goes on to keep global decentralized ledgers in action. Even if you’re a crypto enthusiast, it’s not obvious that you understand what JSON stands for or what IPFS does. That’s because not everyone can directly interact with smart contracts connected to the EVM. It necessitates a user interface (UI) setup or a front end. The front end refers to the part of a decentralized application or system that users directly interact with. When you connect your metamask wallet to a DEX, the website or the ‘user interface’ you see is an example of the front end. When you wish to interact with the EVM, you actually interact with the front end which then relays the information via an Application Programming Interface (API) to nodes. These node providers then broadcast your transactions to the EVM. Thus, your transactions are relayed from the front end to the EVM through APIs provided by node providers, such as Infura.
To interact with a blockchain and change its state, you need to ‘sign’ a transaction with your ‘private key’. While querying data from the blockchain doesn’t require you to sign any transactions, the decentralized ledger necessitates that each change made to it is authorized by the user requesting it in the form of a transaction. Such transactions require you — the ‘signer’ — to approve a state change through your private key. It’s another crucial feature of the crypto wallets, bestowing the roles of provider as well as signer onto it since your private keys are stored on the web by wallets such as MetaMask’s applications.
Signing or authorizing every transaction on the distributed ledger requires you to pay a gas fee. The validators are rewarded with the base network fee (paid by users) to ensure that it runs with integrity. But with an increase in transaction frequency, the network gets congested. And to get your transaction processed on priority, a premium in the form of an additional gas fee has to be paid to validators. In recent years, due to significant network congestion, transaction or gas fee on Ethereum skyrocketed making the network unviable for retail users.
To mitigate the ill impacts of increasing transaction costs and time, crypto scientists have come up with Layer-2 (L2) solutions such as Polygon zkEVM. Such scaling solutions exist as an overlaying network atop the main settlement layer (in this case, Ethereum) and facilitate a batch of transactions to increase transaction throughput while lowering gas fees. We’ll shortly explore the role of Zero-Knowledge Proofs (ZKPs) which would highlight the significance of having a zkEVM infrastructure better.
Blockchains are data-generation machines, which requisites for decentralized data-storage solutions. They must stay in sync with the blockchain to maintain reliability. It must be noted that such solutions are necessary because storing data directly on the settlement layer can be very very expensive. Such solutions work by embedding the hash of the data stored in them onto the blockchain for permanent records in lieu of the actual data. Decentralized storage systems prevent important data from being controlled by centralized servers and offer incentives to users who participate in such peer-to-peer storage. Your favorite NFTs are probably stored on the blockchain in the same manner :)
Centralization, Privacy, Reliability, and Experience
The existing infrastructure for Web3 is still evolving and is not as mature as the infrastructure for Web2. Let’s investigate whether the ethos of decentralized systems — ownership, censorship resistance, and decentralization — can be sustained with the existing Web3 infrastructure. Is the current middleware competent enough to support this new paradigm?
Decentralized Enough?
Do you really own that cool monkey you bought as an NFT for 150 $ETH? Can you access and buy it from any part of the world? Suppose you sign a transaction for your NFT on the Metamask UI and it gets broadcasted to miners (proof-of-work) or validators (proof-of-stake) via Infura API. The broadcasted transaction will be queued for execution. After being put in a block by the miner or validator, the transaction undergoes validity checks and fraud screening before being executed. Upon execution, it results in a ‘state change’ on the blockchain. All irreversible transactions must change the state of the blockchain. You finally got your monkey!
Sadly, there’s a caveat here. It has been found that node providers don’t really verify the changes in the state of a blockchain once a call has been made through their interface. On top of that, they have access to all read requests, exposing transactional information before it even reaches the mempool. It exposes the user’s funds to unknown risks! Finally, node providers can censor your access to the transparent decentralized ledger and sell/leak your data to MEVors. In fact, Metamask (owned by the same company that owns Infura) has censored certain IPs based on certain geographical locations in the past.
How Reliable?
Speaking of the monkey, what did you actually buy when you paid 150 $ETH? Do you really own it or does it possess the ability to get lost in the dark lush forest? As previously discussed, storing data on the blockchain is very expensive. Usually, only a URL — generated from storing the data on platforms like IPFS or Arweave — is registered on the ledger. Opensea, one of the major NFT marketplaces, uses the aforementioned data-storage solutions. Solutions like IPFS work on a ‘content-based addressing’ in which the data is recalled directly by navigating the hash associated with it. You need to ‘pin’ the data to prevent it from getting lost. Data using content-based addressing is prone to be lost if the original file is only kept by a few specific nodes. If these nodes are corrupted, data is permanently lost. Your monkey is no longer yours to own. On top of this, the aforementioned URL generated doesn’t have any hash commitment to it. Even with permanent storage solutions like Arweave, you are at the mercy of moderators as they are responsible for filtering content before it gets registered on the blockchain.
Who Can Watch You?
Attaining privacy in Web3 is difficult since everything on the blockchains is exposed to the world. The data — or the URLs in the case of your monkey NFT — stored on decentralized storage solutions aren’t encrypted as well! Anyone can query the metadata associated with your NFT, breaching your privacy! For Web3 to succeed, it’s important that the infrastructure enables a robust layer of privacy. One of the most popular and secure traditional approaches is End-to-End (E2E) encryption, but we lack a similar arrangement in Web3.
E2E is a process of encryption where only the communicating parties have knowledge of the information being exchanged between them. Servers or intermediate components don’t have access to this information. Even if a third party manages to get access to it, they won’t be able to decrypt the communication. It is achieved by encrypting the information on the sender’s side with a key available to the sender and the receiver alone. The receiver can decrypt the information with their pair of keys. Pretty Good Privacy (PGP) and Secure Sockets Layer (SSL) are some of the popular E2E encryption methods. PGP requires a public and a private key and is usually employed to sign and encrypt emails; SSL is preferred for encrypting instant messages, given that both parties have SSL certificates. The transparency of blockchains makes it difficult to have complete E2E encryption.
User and Developer Experience Problem
User experience (UX) is going to play a big role in how successful Web3 or Crypto technology will be. A better UX means that more people will adopt the new technology. UX refers to a user’s experience (emotions, attitudes, etc.) after they have used or interacted with a particular product or service. There are multiple steps involved in interacting with a product, and the practical; experiential; affective; meaningful; and valuable aspects felt during those interactions contribute towards UX. The difficulty in understanding a product’s mechanics or estimating its utility in their life might contribute to a negative UX. It is not only with the product or service but also with the whole system involved in the delivery process.
At the same time, a developer’s experience is as important as that of the user. If the developers migrating from Web2 to Web3 don’t find it comfortable to adapt to the new environment, then it impacts productivity and innovation. Mass adoption of new technology is only possible when the users, as well as builders, feel comfortable and secure interacting with it.
For Users
How would you buy a monkey NFT if you’re new to crypto? Well, you can use a credit card on platforms like Opensea if you aren’t a resident of all the 77 countries mentioned here using moonpay. What if you are not one of the chosen (privileged) ones? Firstly, you will have to find an exchange that accepts fiat and convert it into crypto. Then, you will be buying ETH and creating a Metamask wallet; don’t forget to secure your seedphrase (you know, those 8/16/22 words which everybody asks you to memorize and never share, otherwise you can kiss your funds goodbye)! After that, you will find a way to send ETH from your exchange to your Metamask wallet. Now, you will bid on NFTs with an ever-fluctuating artificial floor (gas fee). Be cautious though, since once you sign and broadcast your transaction, you might get ‘sandwiched’ by front-running bots. When you somehow manage to execute a transaction, it will take around 120 blocks to achieve the finality, and you will finally be able to buy your monkey NFT. We hope you bought the right one and didn’t fall for a scam though, as many NFT pages get impersonated and the Metamask prompt to sign a transaction doesn’t come with Google’s “unverified security certificate” prompt. How easy and simple was that?
The design of Web3 makes it really difficult for anyone, especially beginners, to carry out operations smoothly. Honestly, we aren’t even done here. Accessing chain IDs of different chains, having knowledge of their remote procedure calls (RPCs), and being aware of different bridges; swaps; and chains isn’t an easy task. This ultimately results in a bad UX which is probably one of the worst things to happen to any new technology in terms of mass adoption.
For Developers
As developers have started migrating applications from Web2 to Web3, indexing; storing; and computing the colossal amount of data brought by them is going to be a huge task. While Web3 protocols are open source, and technically composable, actually achieving this composability in practice is much more difficult than it is in Web2.
Blockchains are state and computation machines that constantly produce data. It necessitates a robust infrastructure for indexation, node-synchronization, and backup durability. Correctly indexing data from blockchains is as important as difficult; distributed systems make it tough to retrieve data. Apart from the security and scalability of the underlying machine, such issues have long prevented blockchains from achieving instant finality (takes around ~120 blocks on Ethereum). The flexibility of storage, as well as efficient computation models, are required to circumvent the problem. The developer needs to be aware of each and every component to succeed, probably requiring a plug-and-play layer. When it comes to decentralized data storage, validity, and processing solutions — like IPFS, Arweave, Filecoin, Skynet, etc. — developers rely on centralized solutions from Amazon and Google. They find it more convenient, reliable, and easy to use. They provide mutability, while on the other hand, data on the aforementioned decentralized storage solutions are immutable, i.e., they can’t be deleted or manipulated.
The Hopeful Side: Emerging Solutions
We listed out a few key problems that are hindering Web3’s adoption as the next-gen technology. While the landscape is challenging, it would be wrong to assume that a dedicated community committed to decentralization would be deterred. New solutions and mechanisms are being developed to reduce Web3’s reliance on Web2 infrastructure; some of them require building pipelines from the ground up. Let’s have a look at these emerging and promising solutions.
Accessible Decentralization Through Light Nodes
A possible move against centralization is for individuals to set up their own nodes. However, in a world with economic inequality, not everyone can afford the 32 ETH required for setting up a node. Additionally, not everyone has the requisite knowledge to establish this infrastructure remotely. There has been a growing trend in major companies (especially small businesses) to move away from centralized control centers to cloud-based services. Developers can run their code at scale without worrying much about maintaining expensive servers.
Technologies like light clients or nodes can help achieve true decentralization, putting blockchains at the forefront of the serverless revolution. These light nodes interact with full nodes in a trust minimized manner. Since they don’t store or keep the state, they can be easily run on remote devices like a smartphone or a laptop. They can track block headers, verify inclusion proofs of state fragments sent from infrastructure providers or nodes, and verify state commitments. In this way, decentralization can be achieved with minimal storage, bandwidth, and computation costs.
Developers have also attempted to create trustless modifications to MetaMask, as can be seen in the case of Kevlar. After syncing to the latest headers of the blockchain, it will initiate a local RPC server which can be added to MetaMask or any other RPC wallet. It allows any call made by the RPC wallet to go through Merkle Inclusion proofs before being broadcasted. On the other hand, alternatives to existing node providers such as Infura have also come up; a good example is the Lavanet. As a P2P RPC network, it connects providers and dApps on the basis of service type and geography for faster processing. It focuses on censorship resistance and scalability as a whole, demeaning the need for centralized giants for connecting with Web3.
Trust Minimized Environment Through Zero Knowledge Proofs
Zero-Knowledge Proofs are a type of mathematical proof that allows someone to prove that they have knowledge of certain information without actually revealing that information. They help in establishing a ‘trust-minimal setup’ as the client need not trust the server/nodes every time the interaction occurs. The trust needs to be set up only initially where the server will send the client a commitment to the data (or its hash), which can be verified using a ZKP by the client. Subsequent inputs from the server/nodes can be taken without needing to trust the server. Such a trust-minimized setting significantly improves the security of the system. In this way, ZKPs eliminates the need to download the complete dataset just to verify a certain packet of information. This helps in the scalability and affordability of stateless clients as ZKPs save a lot of space and processing power. And finally, such zero-knowledge proofs can keep the user from interacting with a fraud or scam contract which can result in a loss of capital from the user’s wallet. Ethereum scaling Layer 2 zkEVM solutions like Polygon Hermez have been working in this direction to help improve the overall web3 landscape. Their open-source ZK Prover can be used to improve Web3 infrastructure apart from scaling Ethereum’s throughput.
Besides using ZKPs, read requests can probably be masked by using technologies like fuzzy message detection. It helps the fragments associated with the user’s state be obfuscated or muddled with a larger set of state fragments to make identification difficult, resulting in increased security.
ZKPs Enable Privacy on The Blockchains
In a traditional setting, every time a packet of information is exchanged, you need private and public keys or SSL certificates to decrypt it. The aforementioned ZKPs can be of great help here as the trust only needs to be established initially. They can improve E2E encryption by eliminating the need to query public and private keys after the first interaction. As they can easily verify certain information without revealing it, the information broadcasted to nodes by a client/device like a smartphone need not be revealed to the world. Hence, data or systems can’t be tampered with. This makes the system more efficient and secure as it is highly resistant to hacks.
Apart from using ZKPs, there are other methods to achieve E2E encryption on the blockchain. Raman Singh et. al. proposed a method where E2E encryption on the blockchain can be facilitated using certificates generated from a server for the sender and receiver (see figure below). Again, as discussed before, servers result in centralization; moreover, this approach shall require querying public and private keys every time the information is exchanged.
Nansen has recently launched Nansen Connect which provides Web3 E2E encryption to users. They claim to encrypt the private key on the user’s browser through Metamask before the information is sent. Since communication also occurs using a server, it might be affected by the same problems as discussed earlier.
ZKPs Make Blockchains Reliable
Several problems when it comes to the reliability of Web3 infrastructure, especially with decentralized storage systems can be solved using ZKPs. They can verify and establish the ownership of the data without exposing the owner or their other possessions. It establishes a layer of privacy and makes them more reliable. In case of ownership confusion or fraud because of loss, dislocation, or misrepresentation, ZKPs can help identify the correct data. In this way, data that is accurate and free of errors can be located. It is safe to say that your monkey won’t be getting lost anytime soon!
Improving The User Experience
As discussed earlier, a flawless UX is important to achieve the mass adoption of Web3. UX is a complex phenomenon that requires a combination of solutions ranging from ZKPs to other innovative solutions such as gasless transactions, ‘decentralized access control,’ seedphrase NFTs, etc. to bring about a significant improvement in how users perceive and utilize the crypto applications.
ZKPs can be a promising solution for improving Web3 UX from a security point of view. Seedphrases are an example of a sensitive Web3 component that results in bad UX. However, with ZKPs, wallets can be created without seedphrases by using a public key to generate a new wallet address. The public key is derived from a private key that is generated on the user’s device. Once trust is established through initial interaction, there is no need for a seedphrase. Therefore, with ZKPs, it becomes impossible to steal your monkey!
Apart from this, cryptography-focused solutions like Lit protocol seem promising when it comes to improving access control on the blockchain. Their ‘decentralized access control’ model gives users access to a service to only those who either hold an NFT or qualify for a particular requirement. After the requirement is met, nodes carry out a process called distributed key generation, resulting in the creation of a public/private key pair. But here’s a catch! The private key here is fragmented amongst each node involved in the process. Users can collect all these fragments to get the final key. Velas also uses a similar approach with a seedphrase NFT to create a passwordless solution. The only problem here is these fragments get stored on centralized servers like AWS, etc.
Turning our discussion towards gas and UX, ZKPs are going to make transactions cost-effective while maintaining the heightened security of Ethereum. Gas fees on Ethereum have gone up sharply after 2019 with blocks running at almost 100% capacity. By their very nature, ZKPs are a batch of transactions rolled into a single transaction, eliminating the need for discrete proofs. They help alleviate network congestion and bring down gas fees; certain With Ethereum Improvement Proposals like EIP-4844 and EIP-4488, it shall become almost free, faster, and frictionless to transact on the mainnet.
The concept of ‘gasless transactions’ put forth by protocols like Biconomy and Symbiosis Finance can also be considered as an attempt to improve the Web3 UX. With a very simplified interface, they enable users to pay for on-chain transactions in tokens of their choice. It does away with the need of understanding different chains, estimate gas, and figure out the correct RPCs. It is achieved by having versatile liquidity on multiple chains. With that being said, there are certain problems associated with this concept; price fluctuations are a major limiting factor as it makes the whole process unviable, probably sending us back to square one.
Improving The Developer Experience
Developing systems that are more composable would make life easier for developers working in Web3. Rather than having isolated solutions, there needs to be a focus on developing abstraction and aggregation platforms/layers like Spheron. On the other hand, solutions like Ceramic will enable better composability by storing the state and model of the application’s data. It will allow flexibility (mutability) with immutable data being stored on data storage platforms like IPFS or Arweave.
Also, developers well-versed in tech like SQL, and scalable; decentralized; and SQL-compatible solutions like Kwil can be of great help. However, for SQL-compatible solutions to work, data must be indexed properly. This can be achieved through solutions like Subsquid which provide APIs for such indexed data to make it easier for builders. On the other hand, protocols like Glitter make the Web3 onboarding process easier by providing indexing-as-a-service for developers. Data-validity solutions like Kyve can greatly reduce the errors and lag in initial node synchronization. Such solutions will stimulate the network’s security by facilitating new validators. In this way, developer experience in Web3 can be improved.
Apart from the solutions mentioned above, it’s important to note that the efficiency of systems has a huge role to play in the experience of agents in the system. When it comes to decentralized systems, efficiency is greatly impacted by tokenomics. Let’s take an example of a system where due to highly inflationary tokenomics, agents responsible for maintaining the network aren’t being incentivized properly. Such incentives are mostly in the form of native tokens. They make the system less secure for users and developers, and highly prone to attacks. Therefore, properly designed tokenomics, which is sustainable and ensures a healthy incentive system for every agent involved in the system, will result in a better experience for everyone.
Conclusions
The rate of development visible in the Web3 ecosystem is commendable and their commitment to their values serves as a glowing example for the path ahead. Although there is significant progress, the technology required for true decentralization is still underdeveloped, and as a result, many aspects of the crypto world are still centralized. Another problem is that the architecture required for decentralization is either not accessible (high requirements for hardware like a full node) or not user-friendly. There might be a general unwillingness among participants to help further this cause; as Moxie mentions: not everyone is going to set up their own nodes. Hence, centralized node providers are able to take advantage of this situation and censor information. Technology like light clients/nodes can help with decentralization and security on the blockchains.
The reliability of Web3 infrastructure is still low and needs to be achieved through significant improvements. It shall take a lot of effort to achieve true privacy on decentralized ledgers. Apart from that, a very important component of adoption is its ‘experience.’ We see that Web3 infrastructure provides a bad user and developer experience. Significant improvements through ZKPs, aggregation layers, etc can improve the experience.
While dealing with different aspects of the Web3 infrastructure, ZKPs seem to be stealing the show! They have the potential to increase the efficiency of light clients and nodes resulting in an efficient blockchain architecture. We also see how they can help establish a layer of security, reliability, and privacy for the ecosystem. By enabling the users to run business logic on their personal data to prove its correctness, ZKPs will enable them to have greater control over their data. Applications can scale using ZKPs as users won’t be required to establish trust multiple times. They will greatly improve the Web3 UX by drastically reducing the gas fee required to execute transactions. Also, the privacy enabled by ZKPs is regulatory compliant, which shall not create much friction with authorities and will potentially lead to the mass adoption of Web3.
Author is a Decentralized Finance (DeFi) researcher and analyst at Polygon with special emphasis on fundamentals, on-chain activity, and crypto narratives. Their other interests include studying Attention Economy, Layer 2 Solutions, and Derivative Protocols. They like to do gardening, scroll twitter, and explore different movie genres in their free time. They can be reached on twitter at 0xlol.
This research constitutes the personal views of the author formed as a result of research as well as their perspective. Polygon is completely unopinionated in its pursuit of excellence in blockchain ecosystems.
Additional Credits
Cover Picture Art: Darshan (Illustration) and 0xlol (Conceptualization)
Editing: ThePsychGuy
Meme: Shivraj
Additional Readings
Zero knowledge Proofs for Rollups & Stateless Clients
https://research.huobi.com/#/ArticleDetails?id=295
Design principles for web3. So everyone and their cat seems to be… | by Angela Ching | UX Collective
UX issues in web3 go beyond design, here is why. — Salil Naik
Web3 has a user experience problem | by Scott McDonald | May, 2022 | UX Collective
https://twitter.com/VitalikButerin/status/1479815125955715072
Infrastructure Lego: The Middleware Thesis — Deribit Insights
Blockchain middleware: The HTTP of the Web 3.0 age | by Apron Network
The web3 stack: how web3 will offer superior UX than web2 | by Polynya | Medium
Web3 Infrastructure Q1'22 Revenue Report | Messari
Web 3 Middleware — Zee Prime Capital
The Architecture of a Web 3.0 application
Moxie Marlinspike >> Blog >> My first impressions of web3
On Web3 Infrastructure — suzuha
A simple guide to the Web3 stack | Coinbase
Why calldata gas cost reduction is crucial for rollups | by Polynya | Medium
The Anatomy of MetaMask. An X-ray of Web3’s Beloved Fox | by Julia Wu | Jun, 2022
