Main Concepts

To create your Python Extractor, you need to implement a class extending from the Celoxtractor module’s CelonisExtractor class and define two methods. That’s it! Depending on your requirements and use case, you can flexibly add additional methods that support you to get metadata and get table data, of course. That being said, get_metadata and get_data_for_table will always be the core of your Python Extractor:

# Import the required modules
from celoxtractor.extractor import CelonisExtractor

# Set up a class that extends from the Celoxtractor module's CelonisExtractor class
class MyExtractor(CelonisExtractor):

    # The get_metadata method defines the metadata for your data source, see sub-sections for examples
    def get_metadata(self):
              # TODO define your metadata

    # The get_data_for_table method fetches data from your data source, see sub-sections for examples
    def get_data_for_table(self, table_name, parameters, is_delta_load, extraction_filter):
              # TODO retrieve data for your table

    # (You only have to define the methods not execute them.)

Note that you do not have to execute those methods in your script. You only have to define their output, the Python Extractor will do all the rest (including pushing the data to the IBC).

How to use parameters

You can easily add parameters to your Python Extractor directly from the IBC to protect confidential information, such as passwords or authentication tokens, or if you are using variables in your script that can change frequently. Once defined in the IBC, you will be able to use those parameters wherever useful in your script.

Here is an example of how to define a username and password as parameters. Note that the password is defined as a confidential parameter to protect that information:

And here is how easy it is to use those parameters in your Python script. In this example, we use parameters to authenticate an API request.

 from celoxtractor.types import Table, Column
 import requests
 from requests.auth import HTTPBasicAuth

 class MyExtractor(CelonisExtractor):

     def get_metadata(self):
         response = requests.get('https://my-company.com/internal/api', auth = HTTPBasicAuth(parameters["username"], parameters["password"])).json()