Data Model - Advanced¶
In this tutorial, you will dive deeper into more advanced topics of data models, which are required to prepare your process data model for further analyses. More specifically, you will learn:
- How to create relationships between data model tables via foreign keys
- How to set up a process configuration in your data model
- How to import a name mapping file into your data model
- How to specify different data model reload options
Prerequisites¶
To follow this tutorial, you should have created a data model and should have uploaded data into it. As we continue working with the SAP Purchase-to-Pay (P2P) tables from the Data Upload tutorial, it is recommended to complete the Data Upload tutorial first. Further, it is recommended to complete the Data Export tutorial to have a basic understanding how data is retrieved from a data model via PQL.
Tutorial¶
1. Import PyCelonis and connect to Celonis API¶
from pycelonis import get_celonis
celonis = get_celonis(permissions=False)
[2024-09-05 14:14:43,437] INFO: No `base_url` given. Using environment variable 'CELONIS_URL'
[2024-09-05 14:14:43,439] INFO: No `api_token` given. Using environment variable 'CELONIS_API_TOKEN'
[2024-09-05 14:14:43,474] WARNING: KeyType is not set. Defaulted to 'APP_KEY'.
[2024-09-05 14:14:43,475] INFO: Initial connect successful! PyCelonis Version: 2.10.2
No access key provided. Please set the CELONIS_MLOPS_ACCESS_KEY environment variable.
2. Find data model tables¶
Let's start by locating the data model and the corresponding Purchase-to-Pay (P2P) tables, which we created in the Data Upload tutorial:
data_pool = celonis.data_integration.get_data_pools().find("PyCelonis Tutorial Data Pool")
data_pool
DataPool(id='b76b688a-d5f3-4fd4-9b03-41df01281bc3', name='PyCelonis Tutorial Data Pool')
data_model = data_pool.get_data_models().find("PyCelonis Tutorial Data Model")
data_model
DataModel(id='bca917ef-6777-4742-8eb9-013ee804d3e5', name='PyCelonis Tutorial Data Model', pool_id='b76b688a-d5f3-4fd4-9b03-41df01281bc3')
tables = data_model.get_tables()
tables
[ DataModelTable(id='1939a4e8-ba75-48f5-b247-f1f0560778c2', data_model_id='bca917ef-6777-4742-8eb9-013ee804d3e5', name='ACTIVITIES', alias='ACTIVITIES', data_pool_id='b76b688a-d5f3-4fd4-9b03-41df01281bc3'), DataModelTable(id='157feb92-b01d-4790-b569-36e951aaded4', data_model_id='bca917ef-6777-4742-8eb9-013ee804d3e5', name='EKPO', alias='EKPO', data_pool_id='b76b688a-d5f3-4fd4-9b03-41df01281bc3'), DataModelTable(id='096986b2-ebc4-4a81-9bbd-e7b2656336aa', data_model_id='bca917ef-6777-4742-8eb9-013ee804d3e5', name='EKKO', alias='EKKO', data_pool_id='b76b688a-d5f3-4fd4-9b03-41df01281bc3'), DataModelTable(id='4a878190-f89b-4eac-9d33-9c64e5c53493', data_model_id='bca917ef-6777-4742-8eb9-013ee804d3e5', name='LFA1', alias='LFA1', data_pool_id='b76b688a-d5f3-4fd4-9b03-41df01281bc3') ]
activities = tables.find("ACTIVITIES")
activities_columns = activities.get_columns()
ekpo = tables.find("EKPO")
ekpo_columns = ekpo.get_columns()
ekko = tables.find("EKKO")
ekko_columns = ekko.get_columns()
lfa1 = tables.find("LFA1")
lfa1_columns = lfa1.get_columns()
3. Create relationships between data model tables via foreign keys¶
As known from the Data Export tutorial, it is possible to combine columns from different data model tables inside a PQL query. These columns will be aggregated into a single result table using PQL's implicit join functionality. However, in order for this to work, we need to specify how the tables in a data model are connected. This is achieved by creating foreign key relationships between pairs of tables.
3.1 The Celonis data model¶
Tables in Celonis are organized in a snowflake schema with 1:N relationships between tables. Hereby, the activity table serves as the central fact table, around which all other tables are organized. The activity table also serves as the base table when creating the single result table during PQL's implicit grouping. Other tables, such as the case table or master data tables, are then merged via a left-outer join with the N-table on the left and the 1-table on the right side.
3.2 Create foreign key relationships¶
To create a new relationship between two tables which can be used to perform implicit joins, we have to call the create_foreign_key() method inside the data model. The method takes as input arguments:
| Name | Type | Description | Default |
|---|---|---|---|
source_table_id |
str |
ID of the source table (i.e. 1-table; right table in implicit join) | Required |
target_table_id |
str |
ID of the target table (i.e. N-table; left table in implicit join) | Required |
columns |
List[Tuple[str,str]] |
List of tuples in the format ("sourceColumn", "targetColumn") that specifies the foreign keys (i.e. over which columns the are tables connected) |
Required |
Let's start by creating a relationship between our activity table ACTIVITIES and our case table EKPO (i.e. Purchase Order Items). Hereby, EKPO (1-table) is the source and ACTIVITIES (N-table) is the target. The tables are connected via the foreign key _CASE_KEY. During an implicit join, the 1-table EKPO (right side) is then connected to the N-table ACTIVITIES (left side) via a left-outer join:
ekpo_activities_fk = data_model.create_foreign_key(
source_table_id=ekpo.id,
target_table_id=activities.id,
columns=[("_CASE_KEY", "_CASE_KEY")]
)
ekpo_activities_fk
[2024-09-05 14:14:43,572] INFO: Successfully created foreign key with id '938e5cfc-26a9-4e0a-ae86-1dfb6f4cac44'
ForeignKey(id='938e5cfc-26a9-4e0a-ae86-1dfb6f4cac44', data_model_id='bca917ef-6777-4742-8eb9-013ee804d3e5', source_table_id='157feb92-b01d-4790-b569-36e951aaded4', target_table_id='1939a4e8-ba75-48f5-b247-f1f0560778c2')
Next, we create a relationship between EKKO (i.e. Purchase Order Header) and EKPO (i.e. Purchase Order Items). Hereby, EKKO (1-table) is the source and EKPO (N-table) is the target. The tables are connected via the foreign keys EBELN and MANDT. During an implicit join, the 1-table EKKO (right side) is then connected to the N-table EKPO via a left-outer join:
ekko_ekpo_fk = data_model.create_foreign_key(
source_table_id=ekko.id,
target_table_id=ekpo.id,
columns=[("EBELN", "EBELN"), ("MANDT", "MANDT")]
)
ekko_ekpo_fk
[2024-09-05 14:14:43,584] INFO: Successfully created foreign key with id '47a0d3d4-009a-423d-9fae-af0dfe9d7952'
ForeignKey(id='47a0d3d4-009a-423d-9fae-af0dfe9d7952', data_model_id='bca917ef-6777-4742-8eb9-013ee804d3e5', source_table_id='096986b2-ebc4-4a81-9bbd-e7b2656336aa', target_table_id='157feb92-b01d-4790-b569-36e951aaded4')
Lastly, we create a relationship between EKKO (i.e. Purchase Order Header) and LFA1 (i.e. Vendor Master Data). Hereby, LFA1 (1-table) is the source and EKKO (N-table) is the target. The tables are connected via the foreign keys LIFNR and MANDT. During an implicit join, the 1-table LFA1 (right side) is then connected to the N-table EKKO (left side) via a left-outer join:
lfa1_ekko_kf = data_model.create_foreign_key(
source_table_id=lfa1.id,
target_table_id=ekko.id,
columns=[("LIFNR", "LIFNR"), ("MANDT", "MANDT")]
)
lfa1_ekko_kf
[2024-09-05 14:14:43,596] INFO: Successfully created foreign key with id '86e17b76-e537-47cc-af7b-232f4c6cd0ee'
ForeignKey(id='86e17b76-e537-47cc-af7b-232f4c6cd0ee', data_model_id='bca917ef-6777-4742-8eb9-013ee804d3e5', source_table_id='4a878190-f89b-4eac-9d33-9c64e5c53493', target_table_id='096986b2-ebc4-4a81-9bbd-e7b2656336aa')
We can verify that the newly-created foreign key relationships exist by calling the get_foreign_keys() method from the data model:
data_model.get_foreign_keys()
[ ForeignKey(id='938e5cfc-26a9-4e0a-ae86-1dfb6f4cac44', data_model_id='bca917ef-6777-4742-8eb9-013ee804d3e5', source_table_id='157feb92-b01d-4790-b569-36e951aaded4', target_table_id='1939a4e8-ba75-48f5-b247-f1f0560778c2'), ForeignKey(id='47a0d3d4-009a-423d-9fae-af0dfe9d7952', data_model_id='bca917ef-6777-4742-8eb9-013ee804d3e5', source_table_id='096986b2-ebc4-4a81-9bbd-e7b2656336aa', target_table_id='157feb92-b01d-4790-b569-36e951aaded4'), ForeignKey(id='86e17b76-e537-47cc-af7b-232f4c6cd0ee', data_model_id='bca917ef-6777-4742-8eb9-013ee804d3e5', source_table_id='4a878190-f89b-4eac-9d33-9c64e5c53493', target_table_id='096986b2-ebc4-4a81-9bbd-e7b2656336aa') ]
Important:
In order for the table relationships to be effective, we have to reload the data model. Otherwise, we will receive a No common table error when querying across columns in multiple tables.
data_model.reload()
[2024-09-05 14:14:43,622] INFO: Successfully triggered data model reload for data model with id 'bca917ef-6777-4742-8eb9-013ee804d3e5'
[2024-09-05 14:14:43,623] INFO: Wait for execution of data model reload for data model with id 'bca917ef-6777-4742-8eb9-013ee804d3e5'
3.3 Querying across multiple data model tables¶
Let's retrieve the PQL query from the Data Export tutorial, which we used to get data from the EMS. In this example, we could only query columns from a single table, i.e. ACTIVITIES, as the tables in the data model were not connected to each other:
from pycelonis.config import Config
Config.DEFAULT_DATA_MODEL = data_model
import pycelonis.pql as pql
df = pql.DataFrame({
"_CASE_KEY": activities_columns.find("_CASE_KEY"),
"ACTIVITY_EN": activities_columns.find("ACTIVITY_EN"),
"EVENTTIME": activities_columns.find("EVENTTIME"),
"_SORTING": activities_columns.find("_SORTING"),
})
df = df[df._CASE_KEY == "800000000006800001"].sort_values(by=["EVENTTIME", "_SORTING"])
result_df = df.to_pandas(limit=3)
result_df
[2024-09-05 14:14:43,700] INFO: Successfully created data export with id 'ca85ef25-e328-42b0-89a9-34268d8fb722'
[2024-09-05 14:14:43,701] INFO: Wait for execution of data export with id 'ca85ef25-e328-42b0-89a9-34268d8fb722'
[2024-09-05 14:14:43,707] INFO: Export result chunks for data export with id 'ca85ef25-e328-42b0-89a9-34268d8fb722'
| _CASE_KEY | ACTIVITY_EN | EVENTTIME | _SORTING | |
|---|---|---|---|---|
| Index | ||||
| 0 | 800000000006800001 | Create Purchase Requisition Item | 2008-12-31 07:44:05 | 0.0 |
| 1 | 800000000006800001 | Create Purchase Order Item | 2009-01-02 07:44:05 | 10.0 |
| 2 | 800000000006800001 | Print and Send Purchase Order | 2009-01-05 07:44:05 | NaN |
Since the tables in the data model are now connected via foreign key relationships, we can also add columns from other tables to the result table. Let's add the material text from the purchase order item table EKPO, the document type from the purchase order header table EKKO, and the vendor name from the vendor master data table LFA1:
df["Material Text (MAKT_MAKTX)"] = ekpo_columns.find("Material Text (MAKT_MAKTX)")
df["Document Type Text (EKKO_BSART)"] = ekko_columns.find("Document Type Text (EKKO_BSART)")
df["NAME1"] = lfa1_columns.find("NAME1")
result_df = df.to_pandas(limit=3)
result_df
[2024-09-05 14:14:43,774] INFO: Successfully created data export with id '0b8a60ea-631b-4438-8b47-759bbddcb1ce'
[2024-09-05 14:14:43,775] INFO: Wait for execution of data export with id '0b8a60ea-631b-4438-8b47-759bbddcb1ce'
[2024-09-05 14:14:43,782] INFO: Export result chunks for data export with id '0b8a60ea-631b-4438-8b47-759bbddcb1ce'
| _CASE_KEY | ACTIVITY_EN | EVENTTIME | _SORTING | Material Text (MAKT_MAKTX) | Document Type Text (EKKO_BSART) | NAME1 | |
|---|---|---|---|---|---|---|---|
| Index | |||||||
| 0 | 800000000006800001 | Create Purchase Requisition Item | 2008-12-31 07:44:05 | 0.0 | Shafting assembly | Electronic commerce | eSupplier, Inc |
| 1 | 800000000006800001 | Create Purchase Order Item | 2009-01-02 07:44:05 | 10.0 | Shafting assembly | Electronic commerce | eSupplier, Inc |
| 2 | 800000000006800001 | Print and Send Purchase Order | 2009-01-05 07:44:05 | NaN | Shafting assembly | Electronic commerce | eSupplier, Inc |
4. Set up process configuration¶
After having defined the foreign key relationships, the next step in preparing the Celonis data model for further analyses is to set up a process configuration. This involves specifying:
- Which tables in our data model are the activity and case table
- Which columns of our activity table denote the Case ID, Activity Name, Timestamp, and Sorting columns
This can be done by calling the create_process_configuration() method from our data model:
process_configuration = data_model.create_process_configuration(
activity_table_id=activities.id,
case_id_column="_CASE_KEY",
activity_column="ACTIVITY_EN",
timestamp_column="EVENTTIME",
sorting_column="_SORTING",
case_table_id=ekpo.id
)
process_configuration
[2024-09-05 14:14:43,814] INFO: Successfully created process configuration with id 'fd207b81-20da-44e9-a9b7-7015a5bfa215'
ProcessConfiguration(id='fd207b81-20da-44e9-a9b7-7015a5bfa215', data_model_id='bca917ef-6777-4742-8eb9-013ee804d3e5', activity_table_id='1939a4e8-ba75-48f5-b247-f1f0560778c2', case_table_id='157feb92-b01d-4790-b569-36e951aaded4', case_id_column='_CASE_KEY', activity_column='ACTIVITY_EN', timestamp_column='EVENTTIME', sorting_column='_SORTING')
In order for the changes to be effective, we need to reload the data model again:
data_model.reload()
[2024-09-05 14:14:43,837] INFO: Successfully triggered data model reload for data model with id 'bca917ef-6777-4742-8eb9-013ee804d3e5'
[2024-09-05 14:14:43,838] INFO: Wait for execution of data model reload for data model with id 'bca917ef-6777-4742-8eb9-013ee804d3e5'
5. Data Model Reload Advanced¶
5.1 Partial data model reload¶
A last topic in setting up our data model is to decide, how the tables should be loaded from the data pool into our data model. We need to specify whether all tables should be reloaded or only selected tables:
data_model.reload()
[2024-09-05 14:14:43,905] INFO: Successfully triggered data model reload for data model with id 'bca917ef-6777-4742-8eb9-013ee804d3e5'
[2024-09-05 14:14:43,905] INFO: Wait for execution of data model reload for data model with id 'bca917ef-6777-4742-8eb9-013ee804d3e5'
data_model.partial_reload(data_model_table_ids=[lfa1.id, ekko.id])
[2024-09-05 14:14:43,979] INFO: Successfully triggered data model reload for data model with id 'bca917ef-6777-4742-8eb9-013ee804d3e5'
[2024-09-05 14:14:43,979] INFO: Wait for execution of data model reload for data model with id 'bca917ef-6777-4742-8eb9-013ee804d3e5'
5.2. Data model reload without wait¶
Lastly, we can specify whether we want to wait for the data model reload (wait). If wait=True, the method waits until the reload is successfully completed and raises an error if the reload fails. If wait=False, the method does not wait for the reload and does not raise an error in case of a failed reload:
data_model.reload(wait=False)
[2024-09-05 14:14:44,062] INFO: Successfully triggered data model reload for data model with id 'bca917ef-6777-4742-8eb9-013ee804d3e5'
data_model.partial_reload(wait=False, data_model_table_ids=[ekko.id])
[2024-09-05 14:14:52,183] INFO: Successfully triggered data model reload for data model with id 'bca917ef-6777-4742-8eb9-013ee804d3e5'
Conclusion¶
Congratulations! You have successfully learned how to prepare your process data model for further analyses. In the next tutorial Data Upload & Export Advanced, you will dive deeper into advanced topics of data pushs/exports, such as chunking, different export/import types, and specifying custom column configurations.