xl2times package

Contents

xl2times package#

Subpackages#

Submodules#

xl2times.datatypes module#

class xl2times.datatypes.Config#

Bases: object

Encapsulates all configuration options for a run of the tool, including the mapping betwen excel tables and output tables, categories of tables, etc.

__init__(mapping_file, times_info_file, times_sets_file, veda_tags_file, veda_attr_defaults_file, regions, include_dummy_imports, case)#
Parameters:

  • mapping_file (str)

  • times_info_file (str)

  • times_sets_file (str)

  • veda_tags_file (str)

  • veda_attr_defaults_file (str)

  • regions (str)

  • include_dummy_imports (bool)

  • case (str | None)

all_attributes: set[str]#
attr_aliases: set[str]#
attr_by_index: dict[str, set[str]]#
attr_by_type: dict[str, set[str]]#
column_aliases: dict[Tag, dict[str, str]]#
column_default_value: dict[Tag, dict[str, str]]#
dd_table_order: Iterable[str]#
discard_if_empty: Iterable[Tag]#
filter_regions: set[str]#
ie_override_in_syssettings: bool = False#
include_dummy_imports: bool#
known_columns: dict[Tag, set[str]]#
lists_columns: dict[Tag, set[str]]#
produce_case: str | None#
query_columns: dict[Tag, set[str]]#
required_columns: dict[Tag, set[str]]#
row_comment_chars: dict[Tag, dict[str, list]]#
times_sets: dict[str, list[str]]#
times_xl_maps: list[TimesXlMap]#
veda_attr_defaults: dict[str, dict[str, list]]#
class xl2times.datatypes.DataModule#

Bases: str, Enum

Categorise data into modules based on the file they are coming from.

__new__(value)#
base = 'VT_*.*, BY_Trans.*'#
demand = 'SuppXLS/Demands/Dem_Alloc+Series.*, SuppXLS/Demands/ScenDem_*.*'#
classmethod determine_type(path)#
Parameters:

path (str)

Return type:

DataModule | None

lma = 'LMA*.*'#
classmethod module_name(path)#
Parameters:

path (str)

Return type:

str | None

classmethod module_type(path)#
Parameters:

path (str)

Return type:

str | None

scen = 'SuppXLS/Scen_*.*'#
sets = 'Set*.*'#
classmethod submodule(path)#
Parameters:

path (str)

Return type:

str | None

subres = 'SubRES_TMPL/SubRES_*.*'#
syssettings = 'SysSettings.*'#
trade = 'SuppXLS/Trades/ScenTrade_*.*'#
class xl2times.datatypes.EmbeddedXlTable#

Bases: object

A table object: a pandas dataframe wrapped with some metadata.

Variables:

  • tag (str) – Table tag associated with this table in the excel file used as input.

  • defaults (str | None) – Defaults for the table that are separated by a colon from the tag.

  • uc_sets (dict[str, str]) – User constrained tables are declared with tags which indicate their type and domain of coverage.

  • sheetname (str) – Name of the excel worksheet where this table was extracted from.

  • range (str) – Range of rows and columns that contained this table in the original excel worksheet.

  • filename (str) – Name of the original excel file where this table was extracted from.

  • dataframe (pandas.core.frame.DataFrame) – Pandas dataframe containing the values of the table.

__init__(tag, uc_sets, sheetname, range, filename, dataframe, defaults=None)#
Parameters:

  • tag (str)

  • uc_sets (dict[str, str])

  • sheetname (str)

  • range (str)

  • filename (str)

  • dataframe (DataFrame)

  • defaults (str | None)

Return type:

None

dataframe: DataFrame#
defaults: str | None = None#
filename: str#
range: str#
sheetname: str#
tag: str#
uc_sets: dict[str, str]#
class xl2times.datatypes.Tag#

Bases: str, Enum

Enum class to enumerate all the accepted table tags by this program.

You can see a list of all the possible tags in section 2.4 of https://iea-etsap.org/docs/Documentation_for_the_TIMES_Model-Part-IV.pdf

__new__(value)#
active_p_def = '~ACTIVEPDEF'#
book_regions_map = '~BOOKREGIONS_MAP'#
comagg = '~COMAGG'#
comemi = '~COMEMI'#
currencies = '~CURRENCIES'#
def_units = '~DEFUNITS'#
defaultyear = '~DEFAULTYEAR'#
drvr_allocation = '~DRVR_ALLOCATION'#
drvr_table = '~DRVR_TABLE'#
endyear = '~ENDYEAR'#
fi_comm = '~FI_COMM'#
fi_process = '~FI_PROCESS'#
fi_t = '~FI_T'#
classmethod has_tag(tag)#
Parameters:

tag (str)

Return type:

bool

milestoneyears = '~MILESTONEYEARS'#
series = '~SERIES'#
start_year = '~STARTYEAR'#
tfm_ava = '~TFM_AVA'#
tfm_ava_c = '~TFM_AVA-C'#
tfm_comgrp = '~TFM_COMGRP'#
tfm_csets = '~TFM_CSETS'#
tfm_dins = '~TFM_DINS'#
tfm_dins_at = '~TFM_DINS-AT'#
tfm_dins_ts = '~TFM_DINS-TS'#
tfm_dins_tsl = '~TFM_DINS-TSL'#
tfm_fill = '~TFM_FILL'#
tfm_fill_r = '~TFM_FILL-R'#
tfm_ins = '~TFM_INS'#
tfm_ins_at = '~TFM_INS-AT'#
tfm_ins_ts = '~TFM_INS-TS'#
tfm_ins_tsl = '~TFM_INS-TSL'#
tfm_ins_txt = '~TFM_INS-TXT'#
tfm_mig = '~TFM_MIG'#
tfm_psets = '~TFM_PSETS'#
tfm_topdins = '~TFM_TOPDINS'#
tfm_topins = '~TFM_TOPINS'#
tfm_upd = '~TFM_UPD'#
tfm_upd_at = '~TFM_UPD-AT'#
tfm_upd_ts = '~TFM_UPD-TS'#
time_periods = '~TIMEPERIODS'#
time_slices = '~TIMESLICES'#
uc_sets = '~UC_SETS'#
uc_t = '~UC_T'#
unitconversion = '~UNITCONVERSION'#
class xl2times.datatypes.TimesModel#

Bases: object

A class containing all the information about the processed TIMES model.

__init__(internal_regions=<factory>, all_regions=<factory>, processes=<factory>, commodities=<factory>, commodity_groups=<factory>, topology=<factory>, implied_topology=<factory>, trade=<factory>, attributes=<factory>, user_constraints=<factory>, uc_attributes=<factory>, ts_tslvl=<factory>, ts_map=<factory>, time_periods=<factory>, units=<factory>, start_year=<factory>, files=<factory>, data_modules=<factory>, custom_psets=<factory>, user_psets=<factory>, user_csets=<factory>, cases=<factory>)#
Parameters:

  • internal_regions (set[str])

  • all_regions (set[str])

  • processes (DataFrame)

  • commodities (DataFrame)

  • commodity_groups (DataFrame)

  • topology (DataFrame)

  • implied_topology (DataFrame)

  • trade (DataFrame)

  • attributes (DataFrame)

  • user_constraints (DataFrame)

  • uc_attributes (DataFrame)

  • ts_tslvl (DataFrame)

  • ts_map (DataFrame)

  • time_periods (DataFrame)

  • units (DataFrame)

  • start_year (int)

  • files (list[str])

  • data_modules (list[str])

  • custom_psets (DataFrame)

  • user_psets (DataFrame)

  • user_csets (DataFrame)

  • cases (dict[str, str])

Return type:

None

all_regions: set[str]#
attributes: DataFrame#
cases: dict[str, str]#
commodities: DataFrame#
commodity_groups: DataFrame#
custom_psets: DataFrame#
data_modules: list[str]#
property data_years: set[int]#

data_years are years for which there is data specified.

property external_regions: set[str]#
files: list[str]#
implied_topology: DataFrame#
internal_regions: set[str]#
property model_years: set[int]#

model_years is the union of past_years and the representative years of the model (milestone years).

TIMES populates MODLYEAR on its own, so this could be dropped in the future.

property past_years: set[int]#

past_years is the set of all years up to and including start year for which past investments are specified.

TIMES populates PASTYEAR on its own, so this could be dropped in the future.

processes: DataFrame#
start_year: int#
time_periods: DataFrame#
topology: DataFrame#
trade: DataFrame#
ts_map: DataFrame#
ts_tslvl: DataFrame#
uc_attributes: DataFrame#
units: DataFrame#
user_constraints: DataFrame#
user_csets: DataFrame#
user_psets: DataFrame#
property veda_cgs: dict[tuple[str, str, str], str]#

A dictionary mapping commodities to their Veda commodity groups.

class xl2times.datatypes.TimesXlMap#

Bases: object

The mapping between the TIMES excel tables used by the tool for input and the transformed tables it outputs. The mappings are defined in the times_mapping.txt and times-info.json files.

Variables:

  • times_name (str) – Name of the table in its output form.

  • times_cols (list[str]) – Name of the columns that the table will have in its output form. They will be in the header of the output csv files.

  • xl_name (str) – Tag for the Excel table used as input. You can see a list of all the possible tags in section 2.4 of https://iea-etsap.org/docs/Documentation_for_the_TIMES_Model-Part-IV.pdf

  • xl_cols (list[str]) – Columns from the Excel table used as input.

  • col_map (dict[str, str]) – A mapping from Excel column names to Times column names.

  • filter_rows (dict[str, str]) – A map from column name to value to filter rows to. If {}, all rows are outputted. E.g., {‘Attribute’: ‘COM_ELAST’}

__init__(times_name, times_cols, xl_name, xl_cols, col_map, filter_rows)#
Parameters:

  • times_name (str)

  • times_cols (list[str])

  • xl_name (str)

  • xl_cols (list[str])

  • col_map (dict[str, str])

  • filter_rows (dict[str, str])

Return type:

None

col_map: dict[str, str]#
filter_rows: dict[str, str]#
times_cols: list[str]#
times_name: str#
xl_cols: list[str]#
xl_name: str#

xl2times.dd_to_csv module#

xl2times.dd_to_csv.convert_dd_to_tabular(dd_files, output_dir, headers_by_attr)#
Parameters:

  • dd_files (list[Path])

  • output_dir (str)

  • headers_by_attr (dict[str, list[str]])

Return type:

None

xl2times.dd_to_csv.generate_headers_by_attr()#
Return type:

dict[str, list[str]]

xl2times.dd_to_csv.main(arg_list=None)#
Parameters:

arg_list (None | list[str])

xl2times.dd_to_csv.parse_parameter_values_from_file(path)#

Parse a *.dd file and extract the sets and parameters.

There are parameters and sets, and each has a slightly different format. *.dd files have data of the following form:

PARAMETER PARAM_NAME ‘ ‘/ attr_1 attr_2 …

/

SET SET_NAME / attr_1.attr_2.attr_3 …

/

Parameters:

path (Path)

Return type:

tuple[dict[str, list], dict[str, set]]

xl2times.dd_to_csv.save_data_with_headers(param_data_dict, headers_data, save_dir)#

Saves data (with headers) to the provided directory.

Note that the header and data dictionaries are assumed to be parallel dictionaries

Parameters:

  • param_data_dict (dict[str, DataFrame | list[str]]) – Dictionary containing key=param_name and val=dataframe for parameters or List[str] for sets

  • headers_data (dict[str, list[str]]) – Dictionary containing key=param_name and val=dataframes

  • save_dir (str) – Path to folder in which to save the tabular data files

Return type:

None

xl2times.excel module#

xl2times.excel.are_cells_all_empty(df, row, start_col, end_col)#

Check if all cells in a given row are empty by calling cell_is_empty() on them.

Parameters:

  • df – Dataframe object containing all values for the worksheet being evaluated

  • row (int) – Row of the dataframe to be evaluated.

  • start_col (int) – Initial column of the dataframe to be evaluated.

  • end_col (int) – Final column of the dataframe to be evaluated.

Returns:

Boolean indicating if all the cells are empty.

Return type:

bool

xl2times.excel.cell_is_empty(value)#

Check if the given cell is empty.

Parameters:

value – Cell value.

Returns:

Boolean indicating if the cells are empty.

Return type:

bool

xl2times.excel.extract_table(tag_row, tag_col, uc_sets, df, sheetname, filename)#

For each individual table tag found in a worksheet, this function aims to extract the associated table.

We recognise several types of tables:

  • Single cell tables: Tables with only one value, either below or to the right of the table tag. We interpret these as a single data item with a column name VALUE.

  • Multiple cell tables: Tables with multiple values, possibly extending accross several rows and columns. We delimitate them using empty spaces around them and the column names are determined by the values in the row immediately below the table tag

Parameters:

  • tag_row (int) – Row number for the tag designating the table to be extracted

  • tag_col (int) – Column number for the tag designating the table to be extracted

  • uc_sets (dict[str, str]) – Sets (regions and timeslices) for user constraints

  • df (DataFrame) – Dataframe object containing all values for the worksheet being evaluated

  • sheetname (str) – Name of the worksheet being evaluated

  • filename (str) – Path to the excel file being evaluated.

Return type:

Table object in the EmbeddedXlTable format.

xl2times.excel.extract_tables(filename)#

Run the extract_table function on each individual table in each worksheet of the given excel file.

Parameters:

filename (str) – Path to the excel file we will extract tables from.

Returns:

List of table objects in EmbeddedXlTable format.

Return type:

list[datatypes.EmbeddedXlTable]

xl2times.main module#

xl2times.main.compare(data, ground_truth, output_dir)#
Parameters:

  • data (dict[str, DataFrame])

  • ground_truth (dict[str, DataFrame])

  • output_dir (str)

Return type:

str

xl2times.main.dump_tables(tables, filename)#
Parameters:

  • tables (list)

  • filename (str)

Return type:

list

xl2times.main.invalidate_cache(max_age=datetime.timedelta(days=365))#

Delete any cache files older than max_age.

Parameters:

max_age (timedelta) – Maximum age of a cache file to be considered valid. Any cache files older than this are deleted.

xl2times.main.parse_args(arg_list)#

Parses command line arguments.

Parameters:

arg_list (None | list[str]) – List of command line arguments. Uses sys.argv (default argparse behaviour) if None.

Returns:

Parsed arguments.

Return type:

argparse.Namespace

xl2times.main.produce_times_tables(config, input, model)#
Parameters:
Return type:

dict[str, DataFrame]

xl2times.main.read_csv_tables(input_dir)#
Parameters:

input_dir (str)

Return type:

dict[str, DataFrame]

xl2times.main.read_xl(inputs, regions, include_dummy_imports, case=None, output_dir=None, no_cache=False, stop_after_read=False)#
Parameters:

  • inputs (list[str])

  • regions (str)

  • include_dummy_imports (bool)

  • case (str | None)

  • output_dir (str | None)

  • no_cache (bool)

  • stop_after_read (bool)

Return type:

tuple[TimesModel, Config]

xl2times.main.run(args)#

Runs the xl2times conversion.

Parameters:

args (Namespace) – Pre-parsed command line arguments

Return type:

comparison with ground-truth string if ground_truth_dir is provided, else None.

xl2times.main.strip_filename_prefix(table, prefix)#
xl2times.main.to_tables(config, model)#

Convert a TimesModel to a set of Times tables (GAMS sets/parameters).

Parameters:
Return type:

dict[str, DataFrame]

xl2times.main.write_csv_tables(tables, output_dir)#
Parameters:

  • tables (dict[str, DataFrame])

  • output_dir (str)

xl2times.main.write_dd_files(tables, config, output_dir)#
Parameters:

  • tables (dict[str, DataFrame])

  • config (Config)

  • output_dir (str)

xl2times.transforms module#

xl2times.transforms.apply_final_fixup(config, tables, model)#

Clean up remaining issues. Includes handling of aliases that could not be generalised.

Parameters:
Return type:

dict[str, DataFrame]

xl2times.transforms.apply_tag_specified_defaults(config, tables, model)#
Parameters:
Return type:

list[EmbeddedXlTable]

xl2times.transforms.apply_transform_tables(config, tables, model)#

Include data from transformation tables (excl. availability).

Parameters:
Return type:

dict[str, DataFrame]

xl2times.transforms.assign_model_attributes(config, tables, model)#

Assign model attributes to the model.

Parameters:
Return type:

dict[str, DataFrame]

xl2times.transforms.capitalise_table_values(config, tables, model)#

Ensure that all table entries are uppercase. Strip leading and trailing whitespace.

Parameters:
Return type:

list[EmbeddedXlTable]

xl2times.transforms.complete_dictionary(model)#
Parameters:

model (TimesModel)

Return type:

dict[str, DataFrame]

xl2times.transforms.complete_model_trade(config, tables, model)#

Supplement model trade with IRE flows from external regions (i.e. IMPEXP and MINRNW).

Parameters:
Return type:

dict[str, DataFrame]

xl2times.transforms.complete_processes(config, tables, model)#

Generate processes based on trade links if not defined elsewhere.

Parameters:
Return type:

dict[str, DataFrame]

xl2times.transforms.convert_aliases(config, tables, model)#

Ensure TIMES names for all attributes.

Parameters:
Return type:

dict[str, DataFrame]

xl2times.transforms.convert_com_tables(config, tables, model)#

Transform comemi and comagg tables to fi_t.

Parameters:
Return type:

list[EmbeddedXlTable]

xl2times.transforms.convert_to_string(tables)#
Parameters:

tables (dict[str, DataFrame])

Return type:

dict[str, DataFrame]

xl2times.transforms.create_model_cgs(config, tables, model)#

Create model commodity groups.

Parameters:
Return type:

dict[str, DataFrame]

xl2times.transforms.create_model_topology(config, tables, model)#

Create model topology. Drop rows with missing values in fi_t tables.

Parameters:
Return type:

list[EmbeddedXlTable]

xl2times.transforms.create_model_units(config, tables, model)#

Create a dataframe with all units in the model.

Parameters:
Return type:

dict[str, DataFrame]

xl2times.transforms.df_indexed_by_col(df, col)#

Set df index using an existing column; make index uppercase.

Parameters:

  • df (DataFrame)

  • col (str)

Return type:

DataFrame

xl2times.transforms.enforce_availability(config, tables, model)#

Include information on process availability by region in model topology, model processes, and the fi_t table.

Parameters:
Return type:

dict[str, DataFrame]

xl2times.transforms.eval_and_update(table, rows_to_update, new_value)#

Performs an inplace update of rows rows_to_update of table with new_value, which can be a update formula like *2.3.

Parameters:

  • table (DataFrame)

  • rows_to_update (Index)

  • new_value (str)

Return type:

None

xl2times.transforms.expand_rows(query_columns, lists_columns, table)#

Expand entries with commas in lists_columns into separate entries in the same column. Do this for all tables; keep entries in query_columns as lists.

Parameters:

  • query_columns (set[str]) – List of query column names.

  • lists_columns (set[str]) – List of columns that may contain comma-separated lists.

  • table (EmbeddedXlTable) – Table in EmbeddedXlTable format.

Returns:

Table in EmbeddedXlTable format with expanded comma entries.

Return type:

EmbeddedXlTable

xl2times.transforms.expand_rows_parallel(config, tables, model)#
Parameters:
Return type:

list[EmbeddedXlTable]

xl2times.transforms.explode_process_commodity_cols(config, tables, model)#

Explodes the process and commodity columns in the tables that contain them as lists after process_wildcards.

We store wildcard matches for these columns as lists and explode them late here for performance reasons - to avoid row-wise processing that would otherwise need to iterate over very long tables.

Parameters:
Return type:

dict[str, DataFrame]

xl2times.transforms.fill_in_column_defaults(config, tables, model)#

Fill-in defaults specified by column. Also populate region column if the value is dependent on the file path. A column should be existing for the defaults to be applied.

Parameters:
Returns:

List of tables in EmbeddedXlTable format with empty values filled in.

Return type:

list[EmbeddedXlTable]

xl2times.transforms.fill_in_missing_pcgs(config, tables, model)#

Fill in missing primary commodity groups in model.processes.

Expand primary commodity groups specified by a suffix. Remove indication of auxillary flows from model topology.

Parameters:
Return type:

dict[str, DataFrame]

xl2times.transforms.filter_by_pattern(df, pattern)#

Filter dataframe index by a pattern specifying which items to include and/or exclude. Return a set of corresponding items from the first (and only) column in the dataframe.

Parameters:

  • df (DataFrame)

  • pattern (str)

Return type:

set[str]

xl2times.transforms.generate_dummy_processes(config, tables, model)#

Define dummy processes and specify default cost data for them to ensure that a TIMES model can always be solved.

This covers situations when a commodity cannot be supplied by other means. Significant cost is usually associated with the activity of these processes to ensure that they are used as a last resort

Parameters:
Return type:

list[EmbeddedXlTable]

xl2times.transforms.generate_implied_topology(config, tables, model)#

Generate implied topology i.e., niether part of model.topology nor model.trade.

Parameters:
Return type:

dict[str, DataFrame]

xl2times.transforms.generate_topology_dictionary(tables, model)#
Parameters:

  • tables (dict[str, DataFrame])

  • model (TimesModel)

Return type:

dict[str, DataFrame]

xl2times.transforms.generate_uc_properties(config, tables, model)#

Generate a dataframe containing User Constraint properties.

Parameters:
Return type:

list[EmbeddedXlTable]

xl2times.transforms.get_matching_items(row, topology, item_map)#

Return a list of items that match conditions in the given row.

Parameters:

  • row (Series)

  • topology (dict[str, DataFrame])

  • item_map (dict[str, str])

Return type:

list[str] | None

Transform tradelinks to tradelinks_dins.

Parameters:
Return type:

list[EmbeddedXlTable]

xl2times.transforms.include_cgs_in_topology(config, tables, model)#

Include commodity groups in model topology.

Parameters:
Return type:

dict[str, DataFrame]

xl2times.transforms.include_tables_source(config, tables, model)#

Add a column specifying source filename to every table.

Parameters:
Return type:

list[EmbeddedXlTable]

xl2times.transforms.is_year(col_name)#

A column name is a year if it is an int >= 0.

xl2times.transforms.merge_tables(config, tables, model)#

Merge all tables in ‘tables’ with the same table tag. Return a dictionary linking each table tag with its merged table or populate TimesModel class.

Parameters:
Returns:

Dictionary associating a given table tag with its merged table.

Return type:

dict[str,DataFrame]

xl2times.transforms.normalize_column_aliases(config, tables, model)#
Parameters:
Return type:

list[EmbeddedXlTable]

xl2times.transforms.normalize_tags_columns(config, tables, model)#

Normalize (uppercase) tags and (lowercase) column names.

Parameters:
Returns:

List of tables in EmbeddedXlTable format with normalzed values.

Return type:

list[EmbeddedXlTable]

xl2times.transforms.prepare_for_querying(config, tables, model)#

Prepare tables for querying by harmonising them and filling in missing values. This includes - Adding expected columns with NA if they are missing. - Expanding year columns if they contain ranges. - Applying attribute-specific default values (incl. if specified under a different column). - Filling in missing values in year and currency columns with defaults.

Parameters:
Return type:

dict[str, DataFrame]

xl2times.transforms.process_drvr_tables(config, tables, model)#

Process DRVR_TABLE, DRVR_ALLOCATION and SERIES tables.

Parameters:
Return type:

list[EmbeddedXlTable]

xl2times.transforms.process_flexible_import_tables(config, tables, model)#

Attempt to process all flexible import tables in ‘tables’.

The processing includes:

  • Checking that the table is indeed a flexible import table. If not, return it unmodified.

  • Removing, adding and renaming columns as needed.

  • Populating index columns.

  • Handing Attribute column and Other Indexes.

See https://iea-etsap.org/docs/Documentation_for_the_TIMES_Model-Part-IV_October-2016.pdf from p16.

Parameters:
Returns:

List of tables in EmbeddedXlTable format with all FI_T processed.

Return type:

list[EmbeddedXlTable]

xl2times.transforms.process_processes(config, tables, model)#

Process processes. The steps include: - Replace custom sets with standard TIMES sets. - Fill in missing ts level values for processes.

Create model.custom_psets.

Parameters:
Return type:

list[EmbeddedXlTable]

xl2times.transforms.process_regions(config, tables, model)#

Read model regions and update model.internal_regions and model.all_regions.

Include IMPEXP and MINRNW in model.all_regions (defined by default by Veda).

Parameters:
Return type:

list[EmbeddedXlTable]

xl2times.transforms.process_time_periods(config, tables, model)#
Parameters:
Return type:

list[EmbeddedXlTable]

xl2times.transforms.process_time_slices(config, tables, model)#
Parameters:
Return type:

list[EmbeddedXlTable]

Process tradelinks and create model trade (between internal regions).

Parameters:
Return type:

list[EmbeddedXlTable]

xl2times.transforms.process_transform_availability(config, tables, model)#

Process transform availability tables. Steps include: - Removing rows with missing values in the “value” column.

Parameters:
Return type:

list[EmbeddedXlTable]

xl2times.transforms.process_transform_table_variants(config, tables, model)#

Reduces variants of TFM_INS like TFM_INS-TS to TFM_INS.

Parameters:
Return type:

list[EmbeddedXlTable]

xl2times.transforms.process_transform_tables(config, tables, model)#

Process transform tables.

Parameters:
Return type:

list[EmbeddedXlTable]

xl2times.transforms.process_user_constraint_tables(config, tables, model)#

Process all user constraint tables in ‘tables’.

The processing includes:

  • Removing, adding and renaming columns as needed.

  • Populating index columns.

  • Handing Attribute column and wildcards.

See https://iea-etsap.org/docs/Documentation_for_the_TIMES_Model-Part-IV_October-2016.pdf from p16.

Parameters:
Returns:

List of tables in EmbeddedXlTable format with all FI_T processed.

Return type:

list[EmbeddedXlTable]

xl2times.transforms.process_user_defined_sets(config, tables, model)#

Process user-defined sets.

Parameters:
Return type:

dict[str, DataFrame]

xl2times.transforms.process_wildcards(config, tables, model)#

Process wildcards in the tables.

Parameters:
Return type:

dict[str, DataFrame]

xl2times.transforms.query(table, process, commodity, attribute, region, year, limtype, val, module)#
Parameters:

  • table (DataFrame)

  • process (str | list[str] | None)

  • commodity (str | list[str] | None)

  • attribute (str | None)

  • region (str | list[str] | None)

  • year (int | list | None)

  • limtype (str | list[str] | None)

  • val (int | float | None)

  • module (str | list[str] | None)

Return type:

Index

xl2times.transforms.remove_comment_cols(table)#

Return a modified copy of ‘table’ where columns with labels starting with ‘*’ have been deleted. Assumes that any leading spaces in the original input table have been removed.

Parameters:

table (EmbeddedXlTable) – Table object in EmbeddedXlTable format.

Returns:

Table object in EmbeddedXlTable format without comment columns.

Return type:

EmbeddedXlTable

xl2times.transforms.remove_comment_rows(config, tables, model)#

Remove comment rows from all the tables.

Assumes table dataframes are not empty.

Parameters:
Return type:

list[EmbeddedXlTable]

xl2times.transforms.remove_exreg_cols(config, tables, model)#

Remove external region columns from all the tables except tradelinks.

Parameters:
Return type:

list[EmbeddedXlTable]

xl2times.transforms.remove_fill_tables(config, tables, model)#
Parameters:
Return type:

list[EmbeddedXlTable]

xl2times.transforms.remove_invalid_values(config, tables, model)#

Remove all entries of any dataframes that are considered invalid. The rules for allowing an entry can be seen in the ‘constraints’ dictionary below.

Parameters:

  • config (Config)

  • tables (dict[str, DataFrame]) – List of tables in EmbeddedXlTable format.

  • model (TimesModel)

Returns:

List of tables in EmbeddedXlTable format with disallowed entries removed.

Return type:

list[EmbeddedXlTable]

xl2times.transforms.remove_tables_with_formulas(config, tables, model)#

Return a modified copy of ‘tables’ where tables with formulas (as identified by an initial ‘=’) have deleted from the list.

Parameters:
Returns:

List of tables in EmbeddedXlTable format without any formulas.

Return type:

list[EmbeddedXlTable]

xl2times.transforms.resolve_remaining_cgs(config, tables, model)#

Resolve commodity group names in model.attributes specified as commodity type.

Supplement model.commodity_groups with resolved commodity groups.

Parameters:
Return type:

dict[str, DataFrame]

xl2times.transforms.revalidate_input_tables(config, tables, model)#

Perform further validation of input tables: - remove tables without required columns; - remove any row with missing values in any of the required columns; - add any column expected for processing downstream; - forward fill values in columns as specified in the config.

Parameters:
Return type:

list[EmbeddedXlTable]

xl2times.transforms.validate_input_tables(config, tables, model)#

Perform some basic validation (tag names are valid, no duplicate column labels), and remove empty tables (for recognized tags).

Parameters:
Return type:

list[EmbeddedXlTable]

xl2times.transforms.verify_uc_topology(config, tables, model)#

Verify if region / process / commodity in UC_T are present in the topology. Remove rows with invalid region/process or region/commodity combinations.

Parameters:
Return type:

dict[str, DataFrame]

xl2times.utils module#

xl2times.utils.apply_composite_tag(table)#

Handles table level declarations.

Declarations can be included in the table tag and will apply to all data that doesn’t have a different value for that index specified. For example, ~FI_T: DEMAND would assign DEMAND as the attribute for all values in the table that don’t have an attribute specification at the column or row level. After applying the declaration this function will return the modified table with the simplified table tag (e.g. ~FI_T).

See page 15 of https://iea-etsap.org/docs/Documentation_for_the_TIMES_Model-Part-IV.pdf for more context.

Parameters:

table (EmbeddedXlTable) – Table in EmbeddedXlTable format.

Returns:

Table in EmbeddedXlTable format with declarations applied and table tag simplified.

Return type:

datatypes.EmbeddedXlTable

xl2times.utils.compare_df_dict(df_before, df_after, sort_cols=True, context_rows=2)#

Simple function to compare two dictionaries of DataFrames.

Parameters:

  • df_before (dict[str, DataFrame]) – the first dictionary of DataFrames to compare

  • df_after (dict[str, DataFrame]) – the second dictionary of DataFrames to compare

  • sort_cols (bool) – whether to sort the columns before comparing. Set True if the column order is unimportant. (Default value = True)

  • context_rows (int) – number of rows to show around the first difference (Default value = 2)

Return type:

None

xl2times.utils.create_negative_regexp(pattern)#

Create a regular expression for negative patterns (those starting with ‘-‘).

This is the complementary function to create_regexp(), handling the excluded patterns. When a pattern contains items starting with ‘-’, those items become the patterns to exclude from matches.

Parameters:

pattern (str) – The VEDA pattern containing negative patterns (items starting with ‘-‘)

Returns:

A regular expression pattern that matches only the negative patterns.

Return type:

str

Examples

>>> create_negative_regexp("Heat*,-HeatPump,-HeatWaste")
'^HeatPump$|^HeatWaste$'  # Creates pattern matching the excluded items

>>> create_negative_regexp("-Gas*,-Oil*")
'^Gas.*$|^Oil.*$'  # Matches anything starting with 'Gas' or 'Oil'

>>> create_negative_regexp("Coal,Gas")
'^$'  # No negative patterns, matches only the empty string
xl2times.utils.create_regexp(pattern)#

Create a regular expression pattern that handles VEDA wildcards and comma-separated lists.

This function converts VEDA-style patterns into regular expressions, handling: - Wildcards: ‘*’ (multiple chars), ‘?’ or ‘_’ (single char) - Literal underscores using ‘[_]’ - Comma-separated values that match any of the items - Negative patterns (starting with ‘-’) are ignored here; use create_negative_regexp() to handle them.

Parameters:

pattern (str) – The VEDA pattern to convert.

Returns:

A regular expression pattern that matches according to VEDA rules.

Return type:

str

Examples

>>> create_regexp("Elec*")
'^Elec.*$'  # Matches 'Elec', 'Electric', 'Electricity', etc.

>>> create_regexp("Fuel?_Gen")
'^Fuel..Gen$'  # Matches 'Fuel1_Gen', 'FuelA_Gen', 'Fuel22Gen' etc.

>>> create_regexp("Tech?[_]1")
'^Tech._1$'  # Matches 'TechA_1', 'TechB_1' etc, treating '_' as literal

>>> create_regexp("Coal,Gas")
'^Coal$|^Gas$'  # Matches exactly 'Coal' or 'Gas'

>>> create_regexp("Heat*,-HeatPump")
'^Heat.*$'  # Matches anything starting with 'Heat', the negative 'HeatPump' pattern is ignored
xl2times.utils.diff_state(filename_before, filename_after, sort_cols=False)#

Diffs dataframes from two persisted state files created with save_state().

Typical usage: - Save the state from a branch with a regression at some point in the transforms: - Switch to main branch and save the state from the same point: - Diff the two states:

For example:

>>> from utils import save_state, diff_state
>>> save_state(config, tables, model, "branch.pkl.gz")
>>> save_state(config, tables, model, "main.pkl.gz")
>>> diff_state("branch.pkl.gz", "main.pkl.gz")
Parameters:

  • filename_before (str)

  • filename_after (str)

  • sort_cols (bool)

Return type:

None

xl2times.utils.explode(df, data_columns)#

Transpose the ‘data_columns’ in each row into a column of values, replicating the other columns. The name for the new column is “VALUE”.

Parameters:

  • df (DataFrame) – Dataframe to be exploded.

  • data_columns (list[str]) – Names of the columns to be exploded.

Returns:

Tuple with the exploded dataframe and a Series of the original column name for each value in each new row.

Return type:

tuple[DataFrame,pd.Series]

xl2times.utils.filter_veda_filename_patterns(files)#

Filter files by patterns recognised by Veda. This function does not verify file extensions.

Parameters:

files (list[str])

Return type:

list[str]

xl2times.utils.get_scalar(table_tag, tables)#
Parameters:
xl2times.utils.has_negative_patterns(pattern)#
Parameters:

pattern (str)

Return type:

bool

xl2times.utils.is_veda_based(files)#

Determine whether the model follows Veda file structure. This function does not verify file extensions.

Parameters:

files (list[str])

Return type:

bool

xl2times.utils.merge_columns(tables, tag, colname)#

Return a list with all the values belonging to a column ‘colname’ from a table with the given tag.

Parameters:

  • tables (list[EmbeddedXlTable]) – List of tables in EmbeddedXlTable format.

  • tag (str) – Tag name to select tables

  • colname (str) – Column name to select values.

Returns:

List of values for the given column name and tag.

Return type:

numpy.ndarray

xl2times.utils.missing_value_inherit(df, colname)#

For each None value in the specifed column of the dataframe, replace it with the last non-None value. If no previous non-None value is found leave it as it is. This function modifies the supplied dataframe and returns None.

Parameters:

  • df (DataFrame) – Dataframe to be filled in.

  • colname (str) – Name of the column to be filled in.

Returns:

None. The dataframe is filled in in place.

Return type:

None

xl2times.utils.remove_negative_patterns(pattern)#
Parameters:

pattern (str)

Return type:

str

xl2times.utils.remove_positive_patterns(pattern)#
Parameters:

pattern (str)

Return type:

str

xl2times.utils.remove_whitespace(pattern)#
Parameters:

pattern (str)

Return type:

str

xl2times.utils.round_sig(x, sig_figs)#
xl2times.utils.run_gams(times_folder, out_folder)#

Runs GAMS on the @out_folder@, which is assumed to have the output of this tool.

Requires the @gamspy-base@ package.

Parameters:

  • times_folder (str)

  • out_folder (str)

Return type:

None

xl2times.utils.save_state(config, tables, model, filename)#

Save the state from a transform step to a single pickle file.

Useful for troubleshooting regressions by diffing with state from another branch.

Parameters:

  • config (Config)

  • tables (dict[str, DataFrame])

  • model (TimesModel)

  • filename (str)

Return type:

None

xl2times.utils.set_log_level(level)#

Sets the log level, in order of priority, to the provided int level, the LOGURU_LEVEL environment variable, or WARNING by default.

E.g. os.environ[“LOGURU_LEVEL”] = “INFO” Available levels are TRACE, DEBUG, INFO, SUCCESS, WARNING, ERROR, and CRITICAL. Default is SUCCESS which is level 0, and higher levels are more verbose.

Parameters:

level (int | None)

Return type:

str

xl2times.utils.setup_logger(level, log_name='__main__', log_dir='.')#

Configure loguru.

Call this once from entrypoints to set up a new logger. In non-entrypoint modules, just use from loguru import logger directly.

Log file will be written to f”{log_dir}/{log_name}.log”

Parameters:

  • log_name (str) – Name of the log. Corresponding log file will be called {log_name}.log. (Default value = default_log_name)

  • log_dir (str) – Directory to write the log file to. Default is the current working directory.

  • level (int | None)

xl2times.utils.single_column(tables, tag, colname)#

Make sure exactly one table in ‘tables’ has the given table tag, and return the values for the given column name. If there are none or more than one raise an error.

Parameters:

  • tables (list[EmbeddedXlTable]) – List of tables in EmbeddedXlTable format.

  • tag (str) – Tag name.

  • colname (str) – Column name to return the values of.

Returns:

Values for the column in the given table.

Return type:

numpy.ndarray

xl2times.utils.single_table(tables, tag)#

Make sure exactly one table in ‘tables’ has the given table tag, and return it. If there are none or more than one raise an error.

Parameters:

  • tables (list[EmbeddedXlTable]) – List of tables in EmbeddedXlTable format.

  • tag (str) – Tag name.

Returns:

Table with the given tag in EmbeddedXlTable format.

Return type:

datatypes.EmbeddedXlTable

Module contents#

Contents