graphid.util.util_misc module

graphid.util.util_misc.randn(mean=0, std=1, shape=[], a_max=None, a_min=None, rng=None)

graphid.util.util_misc.aslist(sequence)

Ensures that the sequence object is a Python list. Handles, numpy arrays, and python sequences (e.g. tuples, and iterables).

Parameters:

sequence (sequence) – a list-like object

Returns:

list_ - sequence as a Python list

Return type:

list

Example

>>> s1 = [1, 2, 3]
>>> s2 = (1, 2, 3)
>>> assert aslist(s1) is s1
>>> assert aslist(s2) is not s2
>>> aslist(np.array([[1, 2], [3, 4], [5, 6]]))
[[1, 2], [3, 4], [5, 6]]
>>> aslist(range(3))
[0, 1, 2]

class graphid.util.util_misc.classproperty(fget=None, fset=None, fdel=None, doc=None)

Bases: property

Decorates a method turning it into a classattribute

References

https://stackoverflow.com/questions/1697501/python-staticmethod-with-property

graphid.util.util_misc.estarmap(func, iter_, **kwargs)

Eager version of it.starmap from itertools

Note this is inefficient and should only be used when prototyping and debugging.

graphid.util.util_misc.delete_dict_keys(dict_, key_list)

Removes items from a dictionary inplace. Keys that do not exist are ignored.

Parameters:

• dict_ (dict) – dict like object with a __del__ attribute

• key_list (list) – list of keys that specify the items to remove

Example

>>> dict_ = {'bread': 1, 'churches': 1, 'cider': 2, 'very small rocks': 2}
>>> key_list = ['duck', 'bread', 'cider']
>>> delete_dict_keys(dict_, key_list)
>>> result = ub.urepr(dict_, nl=False)
>>> print(result)
{'churches': 1, 'very small rocks': 2}

graphid.util.util_misc.flag_None_items(list_)

graphid.util.util_misc.where(flag_list)

takes flags returns indexes of True values

graphid.util.util_misc.delete_items_by_index(list_, index_list, copy=False)

Remove items from list_ at positions specified in index_list The original list_ is preserved if copy is True

Parameters:

• list_ (list)

• index_list (list)

• copy (bool) – preserves original list if True

Example

>>> list_ = [8, 1, 8, 1, 6, 6, 3, 4, 4, 5, 6]
>>> index_list = [2, -1]
>>> result = delete_items_by_index(list_, index_list)
>>> print(result)
[8, 1, 1, 6, 6, 3, 4, 4, 5]

graphid.util.util_misc.make_index_lookup(list_, dict_factory=<class 'dict'>)

Parameters:

list_ (list) – assumed to have unique items

Returns:

mapping from item to index

Return type:

dict

Example

>>> list_ = [5, 3, 8, 2]
>>> idx2_item = make_index_lookup(list_)
>>> result = ub.urepr(idx2_item, nl=False, sort=1)
>>> assert list(ub.take(idx2_item, list_)) == list(range(len(list_)))
>>> print(result)
{2: 3, 3: 1, 5: 0, 8: 2}

graphid.util.util_misc.cprint(text, color=None)

provides some color to terminal output

Parameters:

• text (str)

• color (str)

Example0:

>>> import pygments.console
>>> msg_list = list(pygments.console.codes.keys())
>>> color_list = list(pygments.console.codes.keys())
>>> [cprint(text, color) for text, color in zip(msg_list, color_list)]

Example1:

>>> import pygments.console
>>> print('line1')
>>> cprint('line2', 'red')
>>> cprint('line3', 'blue')
>>> cprint('line4', 'magenta')
>>> cprint('line5', 'reset')
>>> cprint('line5', 'magenta')
>>> print('line6')

graphid.util.util_misc.ensure_iterable(obj)

Parameters:

obj (scalar or iterable)

Returns:

obj if it was iterable otherwise [obj]

Return type:

it3erable

Timeit:

%timeit util.ensure_iterable([1]) %timeit util.ensure_iterable(1) %timeit util.ensure_iterable(np.array(1)) %timeit util.ensure_iterable([1]) %timeit [1]

Example

>>> obj_list = [3, [3], '3', (3,), [3,4,5]]
>>> result = [ensure_iterable(obj) for obj in obj_list]
>>> result = str(result)
>>> print(result)
[[3], [3], ['3'], (3,), [3, 4, 5]]

graphid.util.util_misc.highlight_regex(str_, pat, reflags=0, color='red')

FIXME Use pygments instead

graphid.util.util_misc.regex_word(w)

graphid.util.util_misc.setdiff(list1, list2)

returns list1 elements that are not in list2. preserves order of list1

Parameters:

• list1 (list)

• list2 (list)

Returns:

new_list

Return type:

list

Example

>>> list1 = ['featweight_rowid', 'feature_rowid', 'config_rowid', 'featweight_forground_weight']
>>> list2 = [u'featweight_rowid']
>>> new_list = setdiff(list1, list2)
>>> result = ub.urepr(new_list, nl=False)
>>> print(result)
['feature_rowid', 'config_rowid', 'featweight_forground_weight']

graphid.util.util_misc.all_dict_combinations(varied_dict)

Parameters:

varied_dict (dict) – a dict with lists of possible parameter settings

Returns:

dict_list a list of dicts correpsonding to all combinations of params settings

Return type:

list

Example

>>> varied_dict = {'logdist_weight': [0.0, 1.0], 'pipeline_root': ['vsmany'], 'sv_on': [True, False, None]}
>>> dict_list = all_dict_combinations(varied_dict)
>>> result = str(ub.urepr(dict_list))
>>> print(result)
[
    {'logdist_weight': 0.0, 'pipeline_root': 'vsmany', 'sv_on': True},
    {'logdist_weight': 0.0, 'pipeline_root': 'vsmany', 'sv_on': False},
    {'logdist_weight': 0.0, 'pipeline_root': 'vsmany', 'sv_on': None},
    {'logdist_weight': 1.0, 'pipeline_root': 'vsmany', 'sv_on': True},
    {'logdist_weight': 1.0, 'pipeline_root': 'vsmany', 'sv_on': False},
    {'logdist_weight': 1.0, 'pipeline_root': 'vsmany', 'sv_on': None},
]

graphid.util.util_misc.iteritems_sorted(dict_)

change to iteritems ordered

graphid.util.util_misc.partial_order(list_, part)

graphid.util.util_misc.replace_nones(list_, repl=-1)

Recursively removes Nones in all lists and sublists and replaces them with the repl variable

Parameters:

• list_ (list)

• repl (obj) – replacement value

Returns:

list

Example

>>> list_ = [None, 0, 1, 2]
>>> repl = -1
>>> repl_list = replace_nones(list_, repl)
>>> result = str(repl_list)
>>> print(result)
[-1, 0, 1, 2]

graphid.util.util_misc.take_percentile_parts(arr, front=None, mid=None, back=None)

Take parts from front, back, or middle of a list

Example

>>> arr = list(range(20))
>>> front = 3
>>> mid = 3
>>> back = 3
>>> result = take_percentile_parts(arr, front, mid, back)
>>> print(result)
[0, 1, 2, 9, 10, 11, 17, 18, 19]

graphid.util.util_misc.snapped_slice(size, frac, n)

Creates a slice spanning n items in a list of length size at position frac.

Parameters:

• size (int) – length of the list

• frac (float) – position in the range [0, 1]

• n (int) – number of items in the slice

Returns:

slice object that best fits the criteria

Return type:

slice

SeeAlso:

take_percentile_parts

Example:

Example

>>> # DISABLE_DOCTEST
>>> print(snapped_slice(0, 0, 10))
>>> print(snapped_slice(1, 0, 10))
>>> print(snapped_slice(100, 0, 10))
>>> print(snapped_slice(9, 0, 10))
>>> print(snapped_slice(100, 1, 10))
pass

graphid.util.util_misc.get_timestamp(format_='iso', use_second=False, delta_seconds=None, isutc=False, timezone=False)

Parameters:

• format_ (str) – (tag, printable, filename, other)

• use_second (bool)

• delta_seconds (None)

Returns:

stamp

Return type:

str

Example

>>> format_ = 'printable'
>>> use_second = False
>>> delta_seconds = None
>>> stamp = get_timestamp(format_, use_second, delta_seconds)
>>> print(stamp)
>>> assert len(stamp) == len('15:43:04 2015/02/24')

graphid.util.util_misc.isect(list1, list2)

returns list1 elements that are also in list2. preserves order of list1

intersect_ordered

Parameters:

• list1 (list)

• list2 (list)

Returns:

new_list

Return type:

list

Example

>>> list1 = ['featweight_rowid', 'feature_rowid', 'config_rowid', 'featweight_forground_weight']
>>> list2 = [u'featweight_rowid']
>>> result = isect(list1, list2)
>>> print(result)
['featweight_rowid']

graphid.util.util_misc.safe_extreme(arr, op, fill=nan, finite=False, nans=True)

Applies an exterme operation to an 1d array (typically max/min) but ensures a value is always returned even in operations without identities. The default identity must be specified using the fill argument.

Parameters:

• arr (ndarray) – 1d array to take extreme of

• op (func) – vectorized operation like np.max to apply to array

• fill (float) – return type if arr has no elements (default = nan)

• finite (bool) – if True ignores non-finite values (default = False)

• nans (bool) – if False ignores nans (default = True)

graphid.util.util_misc.safe_argmax(arr, fill=nan, finite=False, nans=True)

Doctest

>>> assert safe_argmax([np.nan, np.nan], nans=False) == 0
>>> assert safe_argmax([-100, np.nan], nans=False) == 0
>>> assert safe_argmax([np.nan, -100], nans=False) == 1
>>> assert safe_argmax([-100, 0], nans=False) == 1
>>> assert np.isnan(safe_argmax([]))

graphid.util.util_misc.safe_max(arr, fill=nan, finite=False, nans=True)

Parameters:

• arr (ndarray) – 1d array to take max of

• fill (float) – return type if arr has no elements (default = nan)

• finite (bool) – if True ignores non-finite values (default = False)

• nans (bool) – if False ignores nans (default = True)

Example

>>> arrs = [[], [np.nan], [-np.inf, np.nan, np.inf], [np.inf], [np.inf, 1], [0, 1]]
>>> arrs = [np.array(arr) for arr in arrs]
>>> fill = np.nan
>>> results1 = [safe_max(arr, fill, finite=False, nans=True) for arr in arrs]
>>> results2 = [safe_max(arr, fill, finite=True, nans=True) for arr in arrs]
>>> results3 = [safe_max(arr, fill, finite=True, nans=False) for arr in arrs]
>>> results4 = [safe_max(arr, fill, finite=False, nans=False) for arr in arrs]
>>> results = [results1, results2, results3, results4]
>>> result = ('results = %s' % (ub.urepr(results, nl=1, sv=1),))
>>> print(result)
results = [
    [nan, nan, nan, inf, inf, 1],
    [nan, nan, nan, nan, 1.0, 1],
    [nan, nan, nan, nan, 1.0, 1],
    [nan, nan, inf, inf, inf, 1],
]

graphid.util.util_misc.safe_min(arr, fill=nan, finite=False, nans=True)

Example

>>> arrs = [[], [np.nan], [-np.inf, np.nan, np.inf], [np.inf], [np.inf, 1], [0, 1]]
>>> arrs = [np.array(arr) for arr in arrs]
>>> fill = np.nan
>>> results1 = [safe_min(arr, fill, finite=False, nans=True) for arr in arrs]
>>> results2 = [safe_min(arr, fill, finite=True, nans=True) for arr in arrs]
>>> results3 = [safe_min(arr, fill, finite=True, nans=False) for arr in arrs]
>>> results4 = [safe_min(arr, fill, finite=False, nans=False) for arr in arrs]
>>> results = [results1, results2, results3, results4]
>>> result = ('results = %s' % (ub.urepr(results, nl=1, sv=1),))
>>> print(result)
results = [
    [nan, nan, nan, inf, 1.0, 0],
    [nan, nan, nan, nan, 1.0, 0],
    [nan, nan, nan, nan, 1.0, 0],
    [nan, nan, -inf, inf, 1.0, 0],
]

graphid.util.util_misc.stats_dict(list_, axis=None, use_nan=False, use_sum=False, use_median=False, size=False)

Parameters:

• list_ (listlike) – values to get statistics of

• axis (int) – if list_ is ndarray then this specifies the axis

Returns:

stats: dictionary of common numpy statistics

(min, max, mean, std, nMin, nMax, shape)

Return type:

OrderedDict

Examples0:

>>> # xdoctest: +IGNORE_WHITESPACE
>>> import numpy as np
>>> axis = 0
>>> np.random.seed(0)
>>> list_ = np.random.rand(10, 2).astype(np.float32)
>>> stats = stats_dict(list_, axis, use_nan=False)
>>> result = str(ub.urepr(stats, nl=1, precision=4, with_dtype=True))
>>> print(result)
{
    'mean': np.array([0.5206, 0.6425], dtype=np.float32),
    'std': np.array([0.2854, 0.2517], dtype=np.float32),
    'max': np.array([0.9637, 0.9256], dtype=np.float32),
    'min': np.array([0.0202, 0.0871], dtype=np.float32),
    'nMin': np.array([1, 1], dtype=np.int32),
    'nMax': np.array([1, 1], dtype=np.int32),
    'shape': (10, 2),
}

Examples1:

>>> import numpy as np
>>> axis = 0
>>> rng = np.random.RandomState(0)
>>> list_ = rng.randint(0, 42, size=100).astype(np.float32)
>>> list_[4] = np.nan
>>> stats = stats_dict(list_, axis, use_nan=True)
>>> result = str(ub.urepr(stats, precision=1, sk=True))
>>> print(result)
{mean: 20.0, std: 13.2, max: 41.0, min: 0.0, nMin: 7, nMax: 3, shape: (100,), num_nan: 1,}