In a major step toward improving the functionality of Web3 and blockchain technology, Space and Time has launched the beta version of Python Data Jobs, a revolutionary solution designed to bridge the gap in processing long-running Python jobs. This groundbreaking development comes in response to the growing need for seamless integration of Python into data processing, analytics, and smart contract functionalities within the Web3 ecosystem.
Challenges addressed and solutions provided
If you’ve been following the developments in space and time, you may be aware of its groundbreaking achievement: the creation of the first and only Zero-Knowledge (ZK) proof for SQL. The Proof of SQL is a remarkably powerful tool that enables smart contracts to securely retrieve and process data using SQL. This cryptographic security opens up a plethora of innovative possibilities for blockchain technology.
Despite SQL’s power as a robust and nearly Turing-complete language, it doesn’t cover every conceivable business scenario. To address the need for custom business logic, especially when deploying arbitrary code becomes essential, Chainlink has ingeniously introduced a solution: Chainlink Functions. These features enable JavaScript to be executed redundantly on Chainlink nodes, achieving consensus on the output.
Consequently, smart contracts can now seamlessly access ZK proven analytics and data processing via Proof of SQL, along with efficient script execution using Functions. However, within the Web3 landscape, one category of use cases remains untouched: long-running Python tasks. Recognizing the importance of Python in enterprise and developer workflows related to space and time, the team wanted to address two key challenges.
First, it sought to enable users to effortlessly use Python to extract, transform, and load data from their existing databases in Space and Time, all without the need for manual code creation. Second, the intention was to establish a cryptographically guaranteed connection between Python tasks and smart contracts. This led to the introduction of Space and Time Python Data Jobs, now available in beta via Space and Time Studio.
Extensive functionality of Python Data Jobs
Space and Time’s Python Data Jobs embarks on a transformative journey within the Web3 landscape, introducing a paradigm shift in data processing and smart contract functionality. As we delve into the intricate capabilities of this solution, a world of seamless integration, simplified ETL processes, and enhanced security unfolds, redefining the way developers and businesses leverage Python’s potential in the blockchain ecosystem. Here’s how this innovative solution works:
1. Bringing data into space and time
Python Data Jobs streamline the data acquisition process across space and time from any off-chain source, eliminating the need for manual code creation. Earlier this year, Space and Time AI introduced SQL, a service powered by OpenAI that allows users to formulate prompts in natural language, such as “show me the top 5 wallets on Sui with the most transactions, ranked by balance.”
This prompt is then converted into an SQL query, which produces the desired result. Space and Time announced that Houston, the AI chatbot within the Space and Time Studio, can now generate simple Extract, Transform, Load (ETL) scripts. These scripts are designed to retrieve data from various sources, including Web2 databases or Web3 decentralized storage platforms. Houston handles the data preparation process and facilitates its transfer across space and time.
By creating a script that connects to databases like PostgreSQL (or alternatives like Snowflake or IPFS), Houston understands the database contents, performs necessary transformations, creates tables in SxT, and incrementally loads data from PostgreSQL into SxT, row by row. Typically, database migration is a laborious, costly and time-consuming task that requires Python expertise. Space and Time’s solution allows developers to achieve this seamlessly using natural language input in a single action.
2. Extracting data from space and time
Python Data Jobs also provides the ability to extract data from space and time, process it and send it to a smart contract. The reason this is a challenge in the Web3 space is due to the longer execution times associated with Python jobs. Consider a scenario where you have a script that needs to calculate the probability of Bitcoin (BTC) staying above $40,000 for the rest of the year.
This script needs to collect data from the markets, process it, and run a Monte Carlo simulation in Python, a process that can take about 20 seconds. When the outcome is linked to a smart contract, it becomes imperative to ensure its tamper-proof nature. While consensus-based proofing is great for quickly executing scripts, it becomes less effective for tasks that run over longer durations.
For example, if the computation is distributed redundantly across 30 nodes, node 1 can complete the task in 18 seconds, while node 5 completes in 25 seconds and node 15 takes 21 seconds. This requires the development of a new architecture to address the unique challenges associated with long-term script execution.
During the Python Data Jobs beta, Space and Time is actively making progress to achieve this by implementing Zero-Knowledge (ZK): specifically a ZK-proof for Python. Currently, the system relies on optimistic security, similar to an optimistic merge approach. When running a Python Data Job within SxT, the input, output, and the code itself are hashed and captured on a main chain.
The script is only executed once and if the outcome differs from expectations, users have the option to request a proof. SxT then cryptographically verifies what was executed. Unlike real-time proofing with redundant computation and consensus, Space and Time’s approach involves a single execution followed by hashing of all metadata. This process creates a tamper-proof audit trail, which serves as an incentive for node operators not to tamper with execution. Space and Time will share more details in the future about the ZK solution in development, which aims to improve the real-time security of Python Data Jobs.
Real-world applications and use cases
Space and Time’s Python Data Jobs go beyond theoretical possibilities and enter the practical realm with real-world applications and use cases. These scenarios showcase the versatility and adaptability of Python Data Jobs and illustrate how this innovative solution addresses complex challenges across industries and paves the way for a new era in data processing and smart contract functionality.
1. Seamless database migrations
Python Data Jobs introduces a new era of simplicity in database migrations. Users can tell Houston to generate Python scripts for complex tasks, such as loading data from platforms like Snowflake across space and time. For example, if you say “generate a Python script to transfer my Snowflake data to SxT”, Houston will request access and produce a Python script.
This script queries Snowflake, retrieves the data, differentiates the schema, and replicates it to SxT in a single Large Language Model (LLM) inference. An illustrative use case involves Truflation, which stores extensive real-time inflation data from numerous data feeds (commodities, bond rates, housing, etc.) in storage. Truflation then constructs aggregations, such as inflation indices, that can be exposed to the chain via oracles.
Python Data Jobs efficiently processes and prepares these significant data volumes for aggregation. Another use case is dClimate, which regularly performs ETL (Extract, Transform, Load) operations on weather data from multiple sources and loads that data into IPFS. Python Data Jobs can streamline this process by automating the extraction and transformation of weather data.
2. Complex calculations for DeFi
Imagine if your smart contract could perform complex off-chain calculations, such as predicting the future performance of a cryptocurrency based on various market conditions, in a secure, tamper-proof manner. Python Data Jobs allows you to integrate advanced financial models, similar to those used for predicting price movements or evaluating risk factors, into your smart contract with an optimistic security approach.
This capability allows DeFi protocols to use more advanced business logic and surpass the capabilities of Proof of SQL. For example, dYdX performs calculations for perpetual options/futures prices off-chain, as these require historical price input data and involve complex calculations that cannot be performed by on-chain smart contracts.
Python Data Jobs facilitates the execution of these calculations in a tamper-proof manner. In another example, 3Commas uses off-chain machine learning models for decision-making in DeFi/CeFi activities (such as swaps, futures, bot transactions, etc.) within a centralized computing container environment. Python Data Jobs provides a Web3-native alternative for performing these tasks.
The way forward
The beta launch of Space and Time’s Python Data Jobs marks significant progress in meeting the evolving needs of Web3 and blockchain technology. By recognizing and addressing the challenges associated with long-running Python tasks within the Web3 landscape, Space and Time has introduced a breakthrough solution that not only streamlines data processing, but also improves the functionality of smart contracts.
The extensive functionality of Python Data Jobs is a testament to the transformative potential it brings to the Web3 ecosystem. From simplifying ETL processes to ensuring the security of Python jobs within smart contracts, Space and Time’s innovative solution opens up new possibilities for developers and businesses alike.
In an effort to encourage users to explore the transformative possibilities of Python Data Jobs, Space and Time is offering free access to the service for one month. Users can start their Python Data Jobs journey with Houston in the Space and Time Studio, ushering in a new era of seamless integration between Python and Web3 technologies.