Skip to content

input

Input dataset wrapper for Context Model Python Tasks.

Provides lazy DataFrame loading from tables.

Input dataclass 

Input(table_name, spark=None)

Wrapper for input table - provides lazy DataFrame loading.

Attributes:

  • table_name (str) –

    Name of the table to read from

  • spark (Optional[SparkSession]) –

    Explicit SparkSession, or None to use the current context-local session.

Methods:

  • dataframe –

    Returns the DataFrame (lazy loaded)

  • count –

    Get row count

  • schema –

    Get DataFrame schema

  • columns –

    Get column names

  • show –

    Show sample rows

Example

input_data = Input(table_name="my_catalog.my_schema.my_table", spark=spark) df = input_data.dataframe() print(f"Row count: {input_data.count()}")

table_name instance-attribute 

table_name

spark class-attribute instance-attribute 

spark = field(default=None, repr=False)

dataframe 

dataframe()

Get the DataFrame (lazy loaded).

Each context caches its latest session/DataFrame pair. Child contexts inherit the current entry, but subsequent cache updates do not affect their parent or siblings. Cache entries are released when their Input is garbage-collected.

Returns:

  • DataFrame –

    DataFrame loaded from the table

Raises:

  • RuntimeError –

    If SparkSession is not set

count 

count()

Get row count.

Returns:

  • int –

    Number of rows in the DataFrame

schema 

schema()

Get DataFrame schema.

Returns:

  • StructType –

    StructType schema of the DataFrame

columns 

columns()

Get column names.

Returns:

  • list[str] –

    List of column names

show 

show(n=5, truncate=True)

Show sample rows.

Parameters:

  • n (int, default: 5 ) –

    Number of rows to show (default: 5)

  • truncate (bool, default: True ) –

    Whether to truncate long strings (default: True)