6. API reference¶
Content below is automatically produced from docstrings in the sources, and needs more work...
6.1. Core¶
pdcalc | A best-effort attempt to build computation dependency-graphs from method with dict-like objects (such as pandas), |
datamodel | |
processor | The core calculations required for transforming the Input-datamodel to the Output one. |
6.2. ExcelRunner¶
FuefitExcelRunner | Sample xlwings script |
6.3. Tests¶
cmdline_test | Check cmdline parsing and model building. |
6.4. Module: fuefit.datamodel¶
- fuefit.datamodel.base_model()[source]¶
The base model for running a WLTC experiment.
It contains some default values for the experiment (ie the default ‘full-load-curve’ for the vehicles). But note that it this model is not valid - you need to override its attributes.
:return :json_tree: with the default values for the experiment.
- fuefit.datamodel.jsonpointer_parts(jsonpointer)[source]¶
Iterates over the jsonpointer parts.
Parameters: jsonpointer (str) – a jsonpointer to resolve within document Returns: a generator over the parts of the json-pointer Author: Julian Berman, ankostis
- fuefit.datamodel.merge(a, b, path=, []list_merge_mode=<MergeMode.REPLACE: 1>, raise_struct_mismatches=False)[source]¶
‘Merges b into a.
List merge modes: REPLACE, APPEND_HEAD, APPEND_TAIL, OVERLAP_HEAD, OVERLAP_TAIL
- fuefit.datamodel.model_schema(additional_properties=False)[source]¶
The json-schema for input/output of the fuefit experiment.
:return :dict:
- fuefit.datamodel.resolve_jsonpointer(doc, jsonpointer, default=<object object at 0x7f59483905e0>)[source]¶
Resolve a jsonpointer within the referenced doc.
Parameters: - doc – the referrant document
- jsonpointer (str) – a jsonpointer to resolve within document
Returns: the resolved doc-item or raises JsonPointerException
Author: Julian Berman, ankostis
- fuefit.datamodel.set_jsonpointer(doc, jsonpointer, value, object_factory=<class 'dict'>)[source]¶
Resolve a jsonpointer within the referenced doc.
Parameters: - doc – the referrant document
- jsonpointer (str) – a jsonpointer to the node to modify
Raises: JsonPointerException (if jsonpointer empty, missing, invalid-contet)
6.5. Module: fuefit.processor¶
The core calculations required for transforming the Input-datamodel to the Output one.
Uses pandalon‘s automatic dependency extraction from calculation functions.
- fuefit.processor._robust_residualfunc(coeffs, modelfunc, X, YData, is_robust=False, robust_prcntile=None)[source]¶
A non-linear iteratively-reweighted least-squares (IRLS) residual function (objective-function) that robustly fits YData = modelfunc(X).
This method applies weights on each iteration so as to downscale any outliers and high-leverage data-points based on the ‘bisquare’ standardized adjusted residuals:[1]
rac{r}{K imes hat{sigma} imes sqrt{1 - h}}
where:
- \(r\) (vector)
- the residuals \(\hat{y} - y\)
- \(K\) (scalar)
- the robust percentile tuning constant used on each iteration to filter-out adjusted-standardized-weights above 1, expressed as the Bisquare M-estimator efficiency under Gaussian model.
- \(\hat{\sigma}\) (scalar)
- the robust estimate of the standard deviation of the residuals based on MAD[#]_ like this: \(\hat{\sigma}=1.4826 imes\operatorname{MAD}\)
- \(h\) : (vector)
- the hat vector, the diagonal of the hat matrix,[#]_ which is used to reduce the weight of high-leverage data points that are having a large effect on the least-squares fit.
param modelfunc: The modeling function that accepts the dict of coeffs param nparray X: param nparray YData: measured-data points param boolean is_robust: Whether to deleverage outlier YData. param float robust_prcntile: The K percentile of the MAD, [default: 4.68, filters-out approximately 5% of the residuals as outliers] See also
curve_fit, leastsq
[1] http://www.mathworks.com/help/stats/robustfit.html [2] https://en.wikipedia.org/wiki/Median_absolute_deviation [3] https://en.wikipedia.org/wiki/Hat_matrix
- fuefit.processor.eng_points_2_std_map(params, engine, eng_points)[source]¶
A factory of the calculation functions for reaching to the data necessary for the Fitting.
The order of the functions below not important, and the actual order of execution is calculated from their dependencies, based on the data-frame’s column access.
6.6. Module: fuefit.pdcalc¶
A best-effort attempt to build computation dependency-graphs from method with dict-like objects (such as pandas), inspired by XForms: http://lib.tkk.fi/Diss/2007/isbn9789512285662/article3.pdf
See also
Dependencies @calculation @calculation_factory [TODO] execute_funcs_map()
- class fuefit.pdcalc.Dependencies[source]¶
Bases: builtins.object
Discovers and stores the rough functions-relationships needed to produce ExecutionPlanner
The relation-tuples are “rough” in the sense that they may contain duplicates etc requiring cleanup by _consolidate_relations().
Usage:
Use the harvest_{XXX}() methods or the add_func_rel() method to gather dependencies from functions and function-facts and then use the build_plan() method to freeze them into a plan.
- add_func_rel(item, deps=None, func=None)[source]¶
Parameters: - item (str) – a dotted.var
- deps – a string or a sequence of dotted.vars
- func – a standalone func, or a funcs_factory as a 2-tuple (func, index)
- build_plan(sources, dests)[source]¶
Builds an execution-plan that when executed with execute_plan() it will produce dests from sources.
Turns any stored-relations into a the full dependencies-graph then trims it only to those ‘’dotted.varname’’ nodes reaching from sources to dests.
Parameters: - dests (sequence) – a list of ‘’dotted.varname’’s that will be the outcome of the calculations
- sources – a list of ‘’dotted.varname’’s (existent or not) that are assumed to exist when the execution wil start
Returns: the new plan that can be fed to execute_plan()
Return type: pd.Series
- classmethod from_funcs_map(funcs_map, deps=None)[source]¶
Factory method for building Dependencies by harvesting multiple functions and func_factories, at once
Parameters: - funcs_map –
a mapping or a sequence of pairs (what --> bool_or_null) with values:
- True
- when what is a funcs_factory, a function returning a sequence of functions processing the data
- False
- when what is a standalone_function, or
- None
- when what is an explicit relation 3-tuple (item, deps, func) to be fed
directly to Dependencies.add_func_rel(), where:
- item
- a ‘’dotted.varname’’ string,
- deps
- a string or a sequence of ‘’dotted.varname’’ strings, and
- func
- a standalone func or a funcs_factory as a 2-tuple (func, index)
Both items and deps are dot-separated sequence of varnames, such as ‘foo.bar’
- deps – if none, a new Dependencies instance is created to harvest relations
Returns: a new (even inherited) or an updated Dependencies instance
Important
All functions must accept exactly the same args, or else the results will be undetermined and it will most-likely scream on execute_funcs_map.
- funcs_map –
- class fuefit.pdcalc._DepFunc(func, is_funcs_factory=False, _child_index=None)[source]¶
Bases: builtins.object
A wrapper for functions explored for relations, optionally allowing them to form a hierarchy of factories and produced functions.
- It can be in 3 types:
- 0, standalone function: args given to function invocation are used immediatelt,
- 10, functions-factory: args are stored and will be used by the child-funcs returned,
- 20, child-func: created internally, and no args given when invoked, will use parent-factory’s args.
- Use factory methods to create one of the first 2 types:
- pdcalc._wrap_standalone_func()
- pdcalc._wrap_funcs_factory()
- fuefit.pdcalc._consolidate_relations(relations)[source]¶
(item1, deps, func), (item1, ...) –> {item1, (set(deps), set(funcs))}
- fuefit.pdcalc._filter_common_prefixes(deps)[source]¶
deps: not-empty set
- example::
- deps = [‘a’, ‘a.b’, ‘b.cc’, ‘a.d’, ‘b’, ‘ac’, ‘a.c’] res = _filter_common_prefixes(deps) assert res == [‘a.b’, ‘a.c’, ‘a.d’, ‘ac’, ‘b.cc’]
- fuefit.pdcalc._find_missing_input(calc_inp_nodes, graph)[source]¶
Search for tentatively missing data.
- fuefit.pdcalc._harvest_indexing(index)[source]¶
Harvest any strings, slices, etc, assuming to be DF’s indices.
- fuefit.pdcalc._harvest_mock_call(mock_call, func, deps_set, func_rels)[source]¶
Adds a 2-tuple (indep, [deps]) into indeps with all deps collected so far when it visits a __setartr__.
- fuefit.pdcalc._research_calculation_routes(graph, sources, dests)[source]¶
Find nodes reaching ‘dests’ but not ‘sources’.
sources: a list of nodes (existent or not) to search for all paths originating from them dests: a list of nodes to search for all paths leading to them them return: a 2-tuple with the graph and its nodes topologically-ordered
- fuefit.pdcalc.calculation(deps=None)[source]¶
A decorator that extracts dependencies from a function.
Parameters: deps – if None, any 2 extracted deps are added in the calculation_deps module-variable. Example:
@calculation def foo(a, r): a['bc'] = a.aa r['r1'] = a.bb @calculation def bar(a, r): r['r2'] = a.bc plan = calculation_deps.build_plan(['a.ab', 'a.bc', 'r'], ['r.r2']) planner.execute_funcs_map(plan, 1, 2)
- fuefit.pdcalc.default_arg_paths_extractor(arg_name, arg, paths)[source]¶
Dig recursively objects for indices to build the sources/dests sequences for Dependencies.build_plan()
It extracts indices recursively from maps and pandas The inner-ones paths are not added, ie df.some.key, but not df or df.some.
Note
for pandas-series their index (their infos-axis) gets appended only the 1st time (not if invoked in recursion, from DataFrame columns).
Parameters: paths (list) – where to add the extracted paths into
- fuefit.pdcalc.execute_funcs_factory(funcs_fact, dests, *args, **kwargs)[source]¶
A one-off way to run calculations from a funcions-factory (see execute_funcs_map())
- fuefit.pdcalc.execute_funcs_map(funcs_map, dests, *args, **kwargs)[source]¶
A one-off way to run calculations from a funcs_map as defined in Dependencies:from_funcs_map().
Parameters: - funcs_map – a dictionary similar to the one used by Dependencies.from_funcs_map()
- dests (seq) –
what is required as the final outcome of the execution, ie: for the func-functory:
def some_funcs(foo, bar): def func_1(): foo['a1'] = ... ... def func_n(): ... foo['a2'] = ... bar['b1'] = ... return [func_1, ..., func_n]
the dests may be:
['foo.a1', 'foo.a2', 'bar.b1']
- args – the actual args to use for invoking the functions in the map, and the names of these args would come rom the first function to be invoked (which ever that might be!).
Note
Do not use it to run the calculations repeatedly. Preffer to cache the function-relations into an intermediate Dependencies instance, instead.
- fuefit.pdcalc.tell_paths_from_named_args(named_args, arg_paths_extractor_func=<function default_arg_paths_extractor at 0x7f5942e9b378>, paths=None)[source]¶
Builds the sources or the dests sequences params of Dependencies.build_plan() from a map of function-arguments
Parameters: named_args – an ordered map {param_name --> param_value} similar to that returned by inspect.signature(func).bind(*args).arguments: BoundArguments. Use the utility name_all_func_args() to generate such a map for some function. Returns: the paths updated with all ‘’dotted.vars’’ found
6.7. Module: fuefit.excel.FuefitExcelRunner¶
6.7.1. Sample xlwings script¶
The functions included provide for running a batch of experiments in an excel-table with json-pointer paths as headers (see accompanying .xlsm).
- You can debug it by running it directly as a python script, as suggested by :
- http://docs.xlwings.org/debugging.html
<< EDIT THIS SCRIPT TO PUT YOUR EXCEL/XLWINGS PYTHON-CODE, BELOW >>
- fuefit.excel.FuefitExcelRunner.build_models(vehs_df)[source]¶
Builds all input-dataframes as Experiment classes and returns them in a list of (veh_id, exp) pairs.
Parameters: vehs_df – A dataframe indexed by veh_id, and with columns json-pointer paths into the model Returns: a list of (veh_id, model) tuples
- fuefit.excel.FuefitExcelRunner.read_input_as_df(table_ref_str)[source]¶
Expects excel-table with jsonpointer paths as Header and 1st column named id as index, like this:
id vehicle/test_mass vehicle/p_rated vehicle/gear_ratios veh_1 1500 100 [120.75, 75, 50, 43, 37, 32] veh_2 1600 80 [120.75, 75, 50, 43, 37, 32] veh_3 1200 60 [120.75, 75, 50, 43, 37, 32]
- fuefit.excel.FuefitExcelRunner.resolve_excel_ref(ref_str, default=<object object at 0x7f5948390620>)[source]¶
if ref_str is an excel-ref, it returns the referred-contents as DataFrame or a scalar, None otherwise.
Excel-ref examples:
@a1 @E5.column @some sheet_name!R1C5.TABLE @1!a1:c5.table(header=False) @3!a1:C5.horizontal(strict=True; atleast_2d=True) @sheet-1!A1.table(asarray=True){columns=['a','b']} @any%sheet^&name!A1:A6.vertical(header=True) ## NOTE: Setting Range's `header` kw and # DataFrame will parse 1st row as header
The excel-ref syntax is case-insensitive apart from the key-value pairs, which it is given in BNF-notation:
excel_ref ::= "@" [sheet "!"] cells ["." shape] ["(" range_kws ")"] ["{" df_kws "}"] sheet ::= sheet_name | sheet_index sheet_name ::= <any character> sheet_index ::= integer cells ::= cell_ref [":" cell_ref] cell_ref ::= A1_ref | RC_ref | tuple_ref A1_ref ::= <ie: "A1" or "BY34"> RC_ref ::= <ie: "R1C1" or "R24C34"> tuple_ref ::= <ie: "(1,1)" or "(24,1)", the 1st is the row> shape ::= "." ("table" | "vertical" | "horizontal") range_kws ::= kv_pairs # keywords for xlwings.Range(**kws) df_kws ::= kv_pairs # keywords for pandas.DataFrafe(**kws) kv_pairs ::= <python code for **keywords ie: "a=3.14, f = 'gg'">
Note that the “RC-notation” is not converted, so Excel may not support it (unless overridden in its options).
6.8. Module: fuefit.test.cmdline_test¶
Check cmdline parsing and model building.