DIDO Wiki

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== A.1 Basic Ethereum Data Store ======
[[dido:public:s_cli:05_contents:01_prt:02_basics:start| Return to DIDO CLI Background]]

===== A.1.1 The Ethereum Blockchain Data Flow =====
[[dido:public:s_cli:05_contents:03_prt:08_basic_dido_objects:start | Return to Top]]

[[dido:public:ra:xapend:xapend.a_glossary:e:ethereum|Ethereum]], as well as most DIDO Platforms, use a [[dido:public:ra:xapend:xapend.a_glossary:r:rest]] model as the basis of their [[dido:public:ra:xapend:xapend.a_glossary:a:api]]. [[dido:public:ra:xapend:xapend.a_glossary:e:ethereum_node]] (i.e.,[[dido:public:ra:xapend:xapend.a_glossary:c:client|Clients]]) formulate a request in [[dido:public:ra:xapend:xapend.a_glossary:x:xml]] or [[dido:public:ra:xapend:xapend.a_glossary:j:json]] and send it to a [[dido:public:ra:xapend:xapend.a_glossary:r:restful|RESTful API]] service that processes the request and calls the [[dido:public:ra:xapend:xapend.a_glossary:e:evm]] in the [[dido:public:ra:xapend:xapend.a_glossary:e:ethereum_node]] using a [[dido:public:ra:xapend:xapend.a_glossary:r:rpc]]. The EVM processes the request and returns any required data back to the API Service as part of the RPC. The API Service then creates an appropriate XML or JSON response and returns it to the client over [[dido:public:ra:xapend:xapend.a_glossary:h:http]] or [[dido:public:ra:xapend:xapend.a_glossary:h:https]].  

<figure>
{{  :dido:public:s_cli:05_contents:03_prt:08_basic_dido_objects:screen_shot_2021-06-12_at_4.07.17_pm.png?600  |}}
<caption>The Flow of Data from Clients to Ethereum [[dido:public:ra:xapend:xapend.a_glossary:b:blockchain|Blockchain]]</caption>
</figure>

===== A.1.2 Overview of Ethereum Blockchain Objects =====
[[dido:public:s_cli:05_contents:03_prt:08_basic_dido_objects:start | Return to Top]]

There are a few basic building blocks included by Ethereum in the infrastructure to support the Ethereum Ecosystem: 
[[dido:public:s_cli:05_contents:03_prt:08_basic_dido_objects:01_block:start | Blocks]], 
[[dido:public:s_cli:05_contents:03_prt:08_basic_dido_objects:08_transaction:start | Transactions (TX)]], 
[[dido:public:s_cli:05_contents:03_prt:08_basic_dido_objects:05_log:start | Log Entries]], 
[[dido:public:s_cli:05_contents:03_prt:08_basic_dido_objects:02_call:start | Calls]], and 
[[dido:public:s_cli:05_contents:03_prt:08_basic_dido_objects:07_trace:start | Trace]]. 

There are two objects that support the development of domain-specific applications: 
[[dido:public:s_cli:05_contents:03_prt:08_basic_dido_objects:06_token:start | Token]] and 
[[dido:public:s_cli:05_contents:03_prt:08_basic_dido_objects:03_contract:start | Contract]].

There are also a couple of generic data structures that define important concepts within the environment using [[dido:public:ra:xapend:xapend.a_glossary:j:json]] (i.e., [[dido:public:s_cli:05_contents:03_prt:08_basic_dido_objects:09_json:start | JSON Support]]:
[[dido:public:s_cli:05_contents:03_prt:08_basic_dido_objects:09_json:args | Arguments]],
[[dido:public:s_cli:05_contents:03_prt:08_basic_dido_objects:09_json:links | Links]] and
[[dido:public:s_cli:05_contents:03_prt:08_basic_dido_objects:09_json:traces | Traces]].

<figure>
{{  :dido:public:s_cli:05_contents:03_prt:08_basic_dido_objects:screen_shot_2021-06-13_at_9.41.52_am.png?700  |}}
<caption>The Ethereum Blockchain Objects</caption>
</figure>

  * <WRAP>//Keep in mind, Ethereum is the database, [[dido:public:ra:xapend:xapend.a_glossary:s:smart_contract|smart contracts]] are the data tables, and transactions from wallets are the rows in each table.//((
Andrew Hong,
Hands-On Tutorials,
__Your guide to basic SQL while learning Ethereum at the same time__,
17 April 2021,
Accessed: 11 June 2021,
[[https://towardsdatascience.com/your-guide-to-basic-sql-while-learning-ethereum-at-the-same-time-9eac17a05929]]
))
</WRAP>

===== A.1.3 Details of Ethereum Blockchain Objects =====
[[dido:public:s_cli:05_contents:03_prt:08_basic_dido_objects:start | Return to Top]]


Ethereum uses a [[dido:public:ra:xapend:xapend.a_glossary:d:dom]] [[dido:public:ra:xapend:xapend.a_glossary:d:dm]] rather than a [[dido:public:ra:xapend:xapend.a_glossary:r:relational_model]], however, the documents are shallow((
**Shallow** documents do not have a lot of nesting. Almost all of the attributes occur and immediately under the document encapsulation. As the depth of nesting increases, the mapping of the documents to a relational model becomes more difficult, especially when trying to enforce normalization rules. 
)) and lend themselves to mapping to the relational model well.

The use of Google's [[dido:public:ra:xapend:xapend.a_glossary:b:big_query|BigQuery]] was designed for analyzing data on the order of billions of rows, using a [[dido:public:ra:xapend:xapend.a_glossary:s:sql]]-like [[dido:public:ra:xapend:xapend.a_glossary:s:syntax|syntax]]. It runs on the Google Cloud Storage infrastructure and can be accessed with a REST-oriented application program interface (API).

Trying to understand and use [[dido:public:ra:xapend:xapend.a_glossary:b:big_query]] developers have been able to tab into the Ethereum [[dido:public:ra:xapend:xapend.a_glossary:d:dlt]] (i.e., [[dido:public:ra:xapend:xapend.a_glossary:b:blockchain]]) data using very close to standard SQL((
Rif Kiamil,
22 February 2021,
Medium,
Accessed: 13 June 2021,
[[https://medium.com/google-cloud/full-relational-diagram-for-ethereum-public-data-on-google-bigquery-2825fdf0fb0b]]
)).

Figure {{ref>EtlArch}} provides a high-level flow of converting Ethereum Blockchain data into Google's BigQuery and as a consequence accessible using SQL. In this workflow, an Ethereum Node is deployed within a [[dido:public:ra:xapend:xapend.a_glossary:c:container|container]] orchestrated by a Kubernetes Engine. there are two paths through the data flow: a Real-Time path and a Daily Path. 

  * In the Daily Path, a [[dido:public:ra:xapend:xapend.a_glossary:s:snapshot|snapshot]] of the blockchain data is exported once a day, converted to [[dido:public:ra:xapend:xapend.a_glossary:c:csv]]  files, and then loaded into BigQuery where the data can be accessed using the BigQuery Console. 
  * In the Real-Time Path that has a lagtime delay to prevent orphaned blocks, the data is streamed to a publish/subscribe utility that can create either 
    * An ETL Dataflow is loaded into BigQuery where data can be accessed using the BigQuery Console
    * An [[dido:public:ra:xapend:xapend.a_glossary:a:application|Application]] subscribes directly to a topic

<figure EtlArch>
{{  :dido:public:s_cli:05_contents:03_prt:08_basic_dido_objects:screen_shot_2021-06-14_at_9.51.14_am.png?700  |}}
<caption>Blockchain ETL Architecture((
Evgeny Medvedev,
Medium,
7 October 2019,
Accessed" 14 June 2021,
[[https://medium.com/google-cloud/live-ethereum-and-bitcoin-data-in-google-bigquery-and-pub-sub-765b71cd57b5]]
))</caption>
</figure>

===== A.1.4 BigQuery Data Objects =====
[[dido:public:s_cli:05_contents:03_prt:08_basic_dido_objects:start | Return to Top]]

The following is a list of Ethereum Data Objects (i.e., tables) that are available using [[dido:public:ra:xapend:xapend.a_glossary:b:big_query]] using [[dido:public:ra:xapend:xapend.a_glossary:s:sql]].

<nspages -tree -r -exclude -subns -pagesInNs -h1 -textNs="">


/**=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
/* To add a discussion page to this page, comment out the line that says 
  ~~DISCUSSION:off~~
*/
~~DISCUSSION:on|Outstanding Issues~~
~~DISCUSSION:off~~
DIDO Wiki

User Tools

Site Tools

Page Tools
