solid_waffle.pyirc

Back end routines for solid-waffle.

Functions

get_version

Version number of script

get_nside

Function to get array size from format codes.

get_num_slices

Gets the number of time slices in an image cube.

load_segment

Function to load an image segment.

pyIRC_percentile

Routine to get percentile cuts with a mask removed.

pyIRC_mean

Routine to get mean with a mask removed.

ref_corr

Get reference corrections from left & right pixel sets.

ref_array

Get reference corrections from left & right pixel sets for a full list of files.

ref_array_onerow

Similar to ref_array, but for one row being valid. Saves time if you only need that one row.

ref_array_block

Extracts reference pixel data in a specified range of y values.

pixel_data

Generate a 4D date cube containing information on a region of the detector.

gen_nl_cube

Routine to get nonlinearity curve.

compute_gain_corr

Gets the correction to the gain from using the full model versus the beta model.

compute_gain_corr_many

Gets the correction to the gain from using the full model versus the beta model.

compute_xc_corr

Gets the correction to the adjacent-pixel correlation from using the full model versus the beta model.

compute_xc_corr_many

Gets the correction to the adjacent-pixel correlation from using the full model versus the beta model.

gain_alphacorr

Routine to get IPC-corrected gain from properties of a difference image.

gain_alphabetacorr

Get IPC+NL-corrected gain.

basic

Basic characterization of a data cube.

corr_5x5

Extracts 5x5 correlation matrix from light and dark data.

corrstats

Routine to obtain statistical properties of a region of the detector across many time slices.

polychar

Routine to characterize of a region of the detector across many time slices.

bfe

Routines to compute the BFE coefficients.

hotpix

Selects hot pixels.

hotpix_ipc

Return IPC data from a list of hot pixels.

slidemed_percentile

Sliding median function.

get_vmin_vmax

Generates min and max range for a color bar based on inter-quartile range.

Classes

IndexDictionary

Table of indices.

Attributes

Test_SubBeta

Classes

IndexDictionary

Table of indices.

Functions

get_version()

Version number of script

get_nside(formatpars)

Function to get array size from format codes.

get_num_slices(formatpars, filename)

Gets the number of time slices in an image cube.

load_segment(filename, formatpars, xyrange, tslices, ...)

Function to load an image segment.

pyIRC_percentile(this_array, mask, perc[, disc])

Routine to get percentile cuts with a mask removed.

pyIRC_mean(this_array, mask)

Routine to get mean with a mask removed.

ref_corr(filename, formatpars, yrange, tslices, verbose)

Get reference corrections from left & right pixel sets.

ref_array(filelist, formatpars, ny, tslices, verbose)

Get reference corrections from left & right pixel sets for a full list of files.

ref_array_onerow(filelist, formatpars, iy, ny, ...)

Similar to ref_array, but for one row being valid. Saves time if you only need that one row.

ref_array_block(filelist, formatpars, yrange, tslices, ...)

Extracts reference pixel data in a specified range of y values.

pixel_data(filelist, formatpars, xyrange, tslices, ...)

Generate a 4D date cube containing information on a region of the detector.

gen_nl_cube(filelist, formatpars, timeslice, ngrid, ...)

Routine to get nonlinearity curve.

compute_gain_corr(fit_array, deriv_array, Ib, tslices, ...)

Gets the correction to the gain from using the full model versus the beta model.

compute_gain_corr_many(fit_array, deriv_array, Ib, ...)

Gets the correction to the gain from using the full model versus the beta model.

compute_xc_corr(fit_array, deriv_array, Ib, tslices, ...)

Gets the correction to the adjacent-pixel correlation from using the full model versus the beta model.

compute_xc_corr_many(fit_array, deriv_array, Ib, ...)

Gets the correction to the adjacent-pixel correlation from using the full model versus the beta model.

gain_alphacorr(graw, CH, CV, signal)

Routine to get IPC-corrected gain from properties of a difference image.

gain_alphabetacorr(graw, CH, CV, signal, frac_dslope, ...)

Get IPC+NL-corrected gain.

basic(region_cube, dark_cube, tslices, lightref, ...)

Basic characterization of a data cube.

corr_5x5(region_cube, dark_cube, tslices, lightref, ...)

Extracts 5x5 correlation matrix from light and dark data.

corrstats(lightfiles, darkfiles, formatpars, box, ...)

Routine to obtain statistical properties of a region of the detector across many time slices.

polychar(lightfiles, darkfiles, formatpars, box, ...)

Routine to characterize of a region of the detector across many time slices.

bfe(region_cube, tslices, basicinfo, ctrl_pars_bfe, ...)

Routines to compute the BFE coefficients.

hotpix(darkfiles, formatpars, tslices, pars, verbose)

Selects hot pixels.

hotpix_ipc(y, x, darkfiles, formatpars, tslices, pars, ...)

Return IPC data from a list of hot pixels.

slidemed_percentile(x, y, p[, mrange, niter, pivot])

Sliding median function.

get_vmin_vmax(mydata, qext)

Generates min and max range for a color bar based on inter-quartile range.

Module Contents

Test_SubBeta = False[source]
get_version()[source]

Version number of script

get_nside(formatpars)[source]

Function to get array size from format codes.

Parameters:

formatpars (int) – Format code.

Returns:

The array size (including reference pixels).

Return type:

int

Notes

This is 4096 for Roman, but we have codes that enable us to run the script on H2RG data.

get_num_slices(formatpars, filename)[source]

Gets the number of time slices in an image cube.

Parameters:

  • formatpars (int) – The format code.

  • filename (str) – The file name.

Returns:

The number of time slices in that file.

Return type:

int

load_segment(filename, formatpars, xyrange, tslices, verbose)[source]

Function to load an image segment.

Parameters:

  • filename (str) – Name of the source FITS file.

  • formatpars (int) – Format code.

  • xyrange (list of int) – List in the form [xmin,xmax,ymin,ymax]. Numpy convention so the first row & column are 0, and excludes xmax and ymax.

  • tslices (list of int) – Time slices to use (beginning slice is 1).

  • verbose (bool) – Whether to print a lot of outputs.

Returns:

A 3D array of the data, shape (len(tslices), ymax-ymin, xmax-xmin). All formats are in “ramp slope negative” format (digital saturation is 0).

Return type:

np.array of float

Notes

Floating-point return type was chosen instead of the native uint16 so that differences don’t underflow to 65535 minus a small number. But integers are exactly represented.

class IndexDictionary(itype)[source]

Table of indices.

These are designed for consistency with the outputs lists of certain functions.

Parameters:

itype (int) – Index type (currently only accepts 0).

N

Number of parameters.

Type:

int

Nb

Number of “basic” parameters.

Type:

int

Nbb

Number of “basic” + BFE parameters.

Type:

int

p

Number of non-linearity coefficients (polynomial order is p + 1).

Type:

int, optional

ngood

Column number for how many pixels in a sub-pixel are good.

Type:

int

graw

Column number for raw gain.

Type:

int

gacorr

Column number for IPC-corrected gain.

Type:

int

g

Column number for best estimate of the gain.

Type:

int

alphaH

Column number for horizontal IPC.

Type:

int

alphaV

Column number for vertical IPC.

Type:

int

beta

Column number for non-linearity parameter (in inverse electrons).

Type:

int

I

Column number for flat intensity in e/p/s.

Type:

int

alphaD

Column number for diagonal IPC.

Type:

int

tCH

Column number for horizontal pixel-pixel covariance.

Type:

int

tCV

Column number for vertical pixel-pixel covariance.

Type:

int

ker0

Column number for the (0,0) position in the BFE kernel.

Type:

int, optional

s

Size of BFE kernel (range on both axes is from -s to +`s`, inclusive).

Type:

int, optional

addbfe()[source]

Adds columns for BFE kernel.

addhnl()[source]

Adds columns for higher-order non-linearity kernel.

Notes

Additional columns: - If BFE is enabled:

the column numbers for the BFE kernel (dx,dy) are ker0 + dy*(2s+1) + dx.

  • If non-linearity is enabled: The column numbers for the higher-order linearity terms are that the degree d term is in column Nbb + d-2.

It is not allowed to add non-linearity and then BFE.

addbfe(s)[source]

Adds columns for the BFE kernel (the size is (2s+1, 2s+1), but flattened).

Parameters:

s (int) – Size of kernel (dx and dy range from -s to +`s`).

Return type:

None

addhnl(p)[source]

Adds columns for higher-order non-linearity coefficients (p coefs, for total degree 1+p).

Parameters:

p (int) – Number of non-linearity coefficients (starts at degree 2 since 0 and 1 are linear, so the total degree is 1+p).

Return type:

None

pyIRC_percentile(this_array, mask, perc, disc=True)[source]

Routine to get percentile cuts with a mask removed.

Parameters:

  • this_array (np.array) – The array from which we want percentiles.

  • mask (np.array of bool or int) – The mask (True or >=1 for good pixels, False or <=0 for bad). Must be the same size as this_array. (Usually they are the same shape, but this is not strictly necessary since they are flattened.)

  • perc (float) – The desired percentile (between 0 and 100).

  • disc (bool, optional) – Interpolate a percentile based on the input data being discrete?

Returns:

Masked percentile of this_array.

Return type:

float

pyIRC_mean(this_array, mask)[source]

Routine to get mean with a mask removed.

Parameters:

  • this_array (np.array) – The array from which we want percentiles.

  • mask (np.array of bool or int) – The mask (True or >=1 for good pixels, False or <=0 for bad). Must be the same size as this_array. (Usually they are the same shape, but this is not strictly necessary since they are flattened.)

Returns:

The mean of the masked array.

Return type:

float

ref_corr(filename, formatpars, yrange, tslices, verbose)[source]

Get reference corrections from left & right pixel sets.

Parameters:

  • filename (str) – The input file name.

  • formatpars (int) – The input format.

  • yrange (list of int) – The minimum and maximum row number, yrange[0]<=y<yrange[1].

  • tslices (list of int) – A list of the time slices to use in ascending order (first in the file is 1).

  • verbose (bool) – Whether to talk a lot.

Returns:

output_ref – A list of reference pixel corrections in these rows. There are 2*``ntslices``+1 entries, each a 1D numpy array (where ntslices is the length of tslices).

Return type:

list of np.array

Notes

The contents of output_ref are as follows. In pseudocode where S[j] refers to the signal in time slice ntslices[j]:

  • In all cases, the first ntslices elements are the median of the corresponding time slice, i.e., output_ref[j] is the median of S[j].

  • The next ntslices elements are the medians of the difference frames, i.e., output_ref[ntslices+j] is the median of the difference of slices S[0]-S[j].

  • If there are at least 2 slices, then the last element is the median of the double difference (D[-2]-D[-1]) - (D[0]-D[1]). Otherwise it is zero. (This is mainly used to measure curvature of the reference pixel ramp.)

Both the left and right side reference pixels are used and medianed together.

ref_array(filelist, formatpars, ny, tslices, verbose)[source]

Get reference corrections from left & right pixel sets for a full list of files.

Parameters:

  • filelist (list of str) – The list of input files to use.

  • formatpars (int) – The format parameter.

  • ny (int) – Number of groups to bin the rows for making a list of reference pixels (should be a power of 2).

  • tslices (list of int) – A list of the time slices to use in ascending order (first in the file is 1).

  • verbose (bool) – Whether to talk a lot.

Returns:

output_array – An array of the reference pixel information, including medians of differences. The shape is (num_files, ny, 2*``ntslice_use``+1), where num_files is the number of files in filelist; and ntslice_use is the number of time slices.

The indexing is that output_array[i,iy,j] is the median reference pixel in file i, in row group iy, and in combination of time slices j (see notes since there are more than ntslice_use options for j).

Return type:

np.array

Notes

The combination of time slices (last axis of output_array) is as follows. In pseudocode where S[j] refers to the signal in time slice ntslices[j]:

  • In all cases, the first ntslices elements are the median of the corresponding time slice, i.e., output_array[:,:,j] is the median of S[j].

  • The next ntslices elements are the medians of the difference frames, i.e., output_array[:,:,ntslices+j] is the median of the difference of slices S[0]-S[j].

  • If there are at least 2 slices, then the last element output_array[:,:,-1] is the median of the double difference (D[-2]-D[-1]) - (D[0]-D[1]). Otherwise it is zero. (This is mainly used to measure curvature of the reference pixel ramp.)

ref_array_onerow(filelist, formatpars, iy, ny, tslices, verbose)[source]

Similar to ref_array, but for one row being valid. Saves time if you only need that one row.

Only use this function if you are absolutely sure you don’t need the other rows!

Parameters:

  • filelist (list of str) – The list of input files to use.

  • formatpars (int) – The format parameter.

  • iy (int) – Which row group (between 0 and ny-1) needs to be good.

  • ny (int) – Number of groups to bin the rows for making a list of reference pixels (should be a power of 2).

  • tslices (list of int) – A list of the time slices to use in ascending order (first in the file is 1).

  • verbose (bool) – Whether to talk a lot.

Returns:

output_array – An array of the reference pixel information, including medians of differences.

Return type:

np.array

See also

ref_array

See for description of the output_array (with the warning that only that one row group is going to be good).

ref_array_block(filelist, formatpars, yrange, tslices, verbose)[source]

Extracts reference pixel data in a specified range of y values.

Parameters:

  • filelist (list of str) – The list of input files to use.

  • formatpars (int) – The format parameter.

  • yrange (list or tuple of int) – Which row range to extract; length 2 ( = ymin, ymax ), with usual Python convention (extracts ymin<=y<ymax).

  • tslices (list of int) – A list of the time slices to use in ascending order (first in the file is 1).

  • verbose (bool) – Whether to talk a lot.

Returns:

output_array – A 2D array, shape (num_files, 2*``ntslices``+1).

The indexing is that output_array[i,j] is the median reference pixel in file i, and in combination of time slices j (see notes since there are more than ntslice_use options for j).

Return type:

np.array

Notes

The combination of time slices (last axis of output_array) is as follows. In pseudocode where S[j] refers to the signal in time slice ntslices[j]:

  • In all cases, the first ntslices elements are the median of the corresponding time slice, i.e., output_array[:,j] is the median of S[j].

  • The next ntslices elements are the medians of the difference frames, i.e., output_array[:,ntslices+j] is the median of the difference of slices S[0]-S[j].

  • If there are at least 2 slices, then the last element output_array[:,-1] is the median of the double difference (D[-2]-D[-1]) - (D[0]-D[1]). Otherwise it is zero. (This is mainly used to measure curvature of the reference pixel ramp.)

pixel_data(filelist, formatpars, xyrange, tslices, maskinfo, verbose)[source]

Generate a 4D date cube containing information on a region of the detector.

Parameters:

  • filelist (list of str) – A list of the input file names.

  • formatpars (int) – Which input format type to use.

  • xyrange (list of int) – The rectangular region to pull, in [xmin,xmax,ymin,ymax] format. Python format, i.e., the first row and column are zero, and xmax and ymax are not included.

  • tslices (list of int) – The time slices to use (first time slice is 1).

  • maskinfo (list or tuple) – A list-like object with at least 2 elements, [cut_offset, do_mask]. Here cut_offset is the range around median to accept (default: 0.1, must be within 10% of median); and do_mask is a boolean on whether to do the masking. If we don’t have at least 2 elements, defaults to [0.1, True].

  • verbose (bool) – Whether to print lots of information.

Returns:

output_array – A 4D array. The shape is (num_files``+1, ``ntslices, dy, dx), where dy=ymax-ymin and dx=xmax-xmin are the sizes of the regions on the detector; ntslices is the number of time slices requested (length of tslice); and num_files is the number of files (length of filelist). The last slice, output_array[-1,:,:,:] is the mask (good=True).

Return type:

np.array

gen_nl_cube(filelist, formatpars, timeslice, ngrid, Ib, usemode, swi, verbose)[source]

Routine to get nonlinearity curve.

Parameters:

  • filelist (list of str) – A list of the input file names.

  • formatpars (int) – Which input format type to use.

  • timeslice (int or list of int) – Which samples to use. If a list, uses timeslice[1] through timeslice[2], assuming reset at time timeslice[0]. If an integer, does time slices 1 … timeslice, assuming reset at slice 0.

  • ngrid (list or tuple of int) – Number of cells on each axis; length 2, y first:[ny,nx] or (ny,nx).

  • Ib (variable) – Deprecated; can pass anything.

  • usemode (str) – Either 'dev' (deviation from beta fit) or 'abs' (absolute – zero of time is absolute).

  • swi (class) – Column table.

  • verbose (bool) – Whether to talk a lot.

Returns:

  • output_array (np.array) – Reference corrected signal in DN; shape = (nt, ny, nx), where nt is the number of time slices requested. This is the median within a file, and then we take the mean across files.

  • fit_array (np.array) – The polynomial fit in DN; shape = (nt, ny, nx). Would be equal to output_array if the fit is perfect.

  • deriv_array (np.array) – The derivative of the polynomial fit in DN/frame; shape = (nt, ny, nx).

  • coefs_array (np.array, optional) – The polynomial coefficients for the ramps; shape (order``+1, ``ny, nx). Order is ascending, i.e., constant term is coefs_array[0,:,:], then the linear term is coefs_array[1,:,:], etc. Only returned if usemode is 'abs'.

compute_gain_corr(fit_array, deriv_array, Ib, tslices, reset_frame)[source]

Gets the correction to the gain from using the full model versus the beta model.

Parameters:

  • fit_array (np.array of float) – Signal in DN for true curve (length tmax array, starting with frame 1).

  • deriv_array (np.array of float) – Signal rate in DN/frame (length tmax array, starting with frame 1).

  • Ib (float) – The charge per frame times beta (unitless).

  • tslices (list of int) – Length 3 list: the time slices used for the quadratic fit in determining beta.

  • reset_frame (int or float) – Reset frame (ideally int, but float would be OK if you want to represent an imperfect reset as we observe in Roman detectors).

Returns:

The ractional gain error, log( gain[full NL] / gain[est. quad.]), caused by using a beta-model for the nonlinearity curve instead of the full curve.

Return type:

float

See also

compute_gain_corr_many

Similar but for multiple superpixels at once.

compute_gain_corr_many(fit_array, deriv_array, Ib, tslices, reset_frame, is_good)[source]

Gets the correction to the gain from using the full model versus the beta model.

This version works for many superpixels.

Parameters:

  • fit_array (np.array of float) – Signal in DN for true curve. Shape (tmax, ny, nx), where the first axis corresponds to the time stamp and the others are superpixel indices. Time stamps start at frame 1.

  • deriv_array (np.array of float) – Signal rate in DN/frame, same shape as fit_array.

  • Ib (np.array of float) – The charge per frame times beta (unitless). This is a 2D array, shape (ny, nx).

  • tslices (list of int) – Length 3 list: the time slices used for the quadratic fit in determining beta.

  • reset_frame (int or float) – Reset frame (ideally int, but float would be OK if you want to represent an imperfect reset as we observe in Roman detectors).

  • is_good (np.array of int or bool) – Mask array, shape (ny, nx), True or 1 indicates good pixels, False or 0 indicates bad.

Returns:

The ractional gain error, log( gain[full NL] / gain[est. quad.]), caused by using a beta-model for the nonlinearity curve instead of the full curve. This is 2D with all the superpixels, shape (ny, nx).

Return type:

np.array of float

See also

compute_gain_corr

Similar but for only one superpixel.

compute_xc_corr(fit_array, deriv_array, Ib, tslices, reset_frame)[source]

Gets the correction to the adjacent-pixel correlation from using the full model versus the beta model.

Parameters:

  • fit_array (np.array of float) – Signal in DN for true curve (length tmax array, starting with frame 1).

  • deriv_array (np.array of float) – Signal rate in DN/frame (length tmax array, starting with frame 1).

  • Ib (float) – The charge per frame times beta (unitless).

  • tslices (list of int) – Length 3 list: the time slices used for the quadratic fit in determining beta.

  • reset_frame (int or float) – Reset frame (ideally int, but float would be OK if you want to represent an imperfect reset as we observe in Roman detectors).

Returns:

The correction to the adjacent-pixel correlation, (full correlation) / (beta model correlation) caused by using a beta-model for the nonlinearity curve instead of the full curve.

Return type:

float

See also

compute_xc_corr_many

Similar but for multiple superpixels at once.

compute_xc_corr_many(fit_array, deriv_array, Ib, tslices, reset_frame, is_good)[source]

Gets the correction to the adjacent-pixel correlation from using the full model versus the beta model.

This version works for many superpixels.

Parameters:

  • fit_array (np.array of float) – Signal in DN for true curve. Shape (tmax, ny, nx), where the first axis corresponds to the time stamp and the others are superpixel indices. Time stamps start at frame 1.

  • deriv_array (np.array of float) – Signal rate in DN/frame, same shape as fit_array.

  • Ib (np.array of float) – The charge per frame times beta (unitless). This is a 2D array, shape (ny, nx).

  • tslices (list of int) – Length 3 list: the time slices used for the quadratic fit in determining beta.

  • reset_frame (int or float) – Reset frame (ideally int, but float would be OK if you want to represent an imperfect reset as we observe in Roman detectors).

  • is_good (np.array of int or bool) – Mask array, shape (ny, nx), True or 1 indicates good pixels, False or 0 indicates bad.

Returns:

The ractional gain error, log( gain[full NL] / gain[est. quad.]), caused by using a beta-model for the nonlinearity curve instead of the full curve. This is 2D with all the superpixels, shape (ny, nx).

Return type:

np.array of float

See also

compute_xc_corr_many

Similar but for only one superpixel.

gain_alphacorr(graw, CH, CV, signal)[source]

Routine to get IPC-corrected gain from properties of a difference image.

Parameters:

  • graw (float) – Uncorrected gain (e/DN)

  • CH (float) – Horizontal correlation (DN^2)

  • CV (float) – Vertical correlation (DN^2)

  • signal (float) – Signal in this ramp (DN)

Returns:

If successful, returns a list of [gain, alphaH, alphaV] (with gain alpha-corrected and in e/DN). Returns an empty list if failed.

Return type:

list of float

gain_alphabetacorr(graw, CH, CV, signal, frac_dslope, times)[source]

Get IPC+NL-corrected gain.

Parameters:

  • graw (float) – Uncorrected gain (e/DN)

  • CH (float) – Horizontal correlation (DN^2)

  • CV (float) – Vertical correlation (DN^2)

  • signal (float) – Signal in this ramp (DN)

  • frac_dslope (float) – Signal ratio for non-linearity, S_{cd}/S_{ab}-1 (unitless).

  • times (list of int) – The time slices to use (length 4: [ta, tb, tc, td]).

Returns:

If successful, returns a list of [gain, alphaH, alphaV, beta, current] (with gain corrected and in e/DN; beta in 1/e; and current in e/frame). Returns an empty list if failed.

Return type:

list of float

Notes

This is solving the following set of equations (see Hirata’s brighter-fatter effect paper):

# in pseudocode
graw = g * [ 1 + beta I (3tb+3td-4ta) ] / [ (1-4alpha)^2 + 2alphaH^2 + 2alphaV^2 ]
CH = (2 I tad alphaH / g^2) [ 1 - 4alpha - 4 beta I td ]
CV = (2 I tad alphaV / g^2) [ 1 - 4alpha - 4 beta I td ]
signal = I tad [ 1 - beta I (ta+td) ] / g
frac_dslope = - beta I (tc+td-ta-tb)
basic(region_cube, dark_cube, tslices, lightref, darkref, ctrl_pars, verbose)[source]

Basic characterization of a data cube.

Parameters:

  • region_cube (np.array) – 4D array of the region of interest. shape: (num_files+1, nt, dy, dx), where num_files is the number of files, nt is the number of time slices, and (dy, dx) is the shape of the region on the SCA used. The last slice, region_cube[-1,:,:,:], is the mask (1 for good, 0 for bad).

  • dark_cube (np.array) – Like region_cube, but for the darks. It is optional whether there is a separate mask (it isn’t used, so it is OK if the first axis has length num_files or num_files+1).

  • tslices (list of int) – List of the time slice numbers; length nt.

  • lightref (np.array) – Reference pixel table for correcting light exposures. shape = (num_files, 2*nt+1); the way the time axis is managed is described in ref_corr().

  • darkref (np.array) – Similar to lightref, but for the dark exposures.

  • ctrl_pars (class) – Contains the control parameters as attributes (see Notes).

  • verbose (bool) – Whether to print lots of information.

Returns:

The basic calibration parameters. The return information depends on whether ctrl_pars.full_corr is True or False:

# True (default):
[number of good pixels, gain_raw, gain_acorr, gain_abcorr, aH, aV, beta, I, 0., tCH, tCV]
# False:
[number of good pixels, median, variance, tCH, tCV, tCD]

Returns the null list [] if failed.

Return type:

list

Notes

The ctrl_pars class contains the following attributes:

  • epsilon : float Fraction of data points to cut for computing correlations (default 0.01)

  • subtr_corr : bool Do mean subtraction for the IPC correlation? (default to True)

  • noise_corr : bool Do noise subtraction for the IPC correlation? (default to True)

  • reset_frame : int Reset frame (default to 0)

  • subtr_href : bool Horizontal reference pixel subtraction? (default to True)

  • full_corr : bool Which parameters to report? (default to True = standard basic pars; False = correlation data instead)

  • leadtrailSub : bool Perform lead-trail subtraction? (default to False)

  • g_ptile : float Percentile for inter-quantile range (default to 75)

This includes a test so this won’t crash if tslices[1]>=tslices[-1] but returns meaningful cross-correlation C_{abab} (everything else is nonsense in this case).

corr_5x5(region_cube, dark_cube, tslices, lightref, darkref, ctrl_pars, verbose)[source]

Extracts 5x5 correlation matrix from light and dark data.

Parameters:

  • region_cube (np.array) – 4D array of the region of interest. shape: (num_files+1, nt, dy, dx), where num_files is the number of files, nt is the number of time slices, and (dy, dx) is the shape of the region on the SCA used. The last slice, region_cube[-1,:,:,:], is the mask (1 for good, 0 for bad).

  • dark_cube (np.array) – Like region_cube, but for the darks. It is optional whether there is a separate mask (it isn’t used, so it is OK if the first axis has length num_files or num_files+1).

  • tslices (list of int) – List of the time slice numbers; length nt.

  • lightref (np.array) – Reference pixel table for correcting light exposures. shape = (num_files, 2*nt+1); the way the time axis is managed is described in ref_corr().

  • darkref (np.array) – Similar to lightref, but for the dark exposures.

  • ctrl_pars (class) – A class containing the control parameters as attributes. These are optional (but recommended); if specified, they follow the same format as in basic().

  • verbose (bool) – Whether to print lots of information.

Returns:

The return list has 5 entries. In what follows, “a”, “b”, and “d” correspond to time slices tslices[0], tslices[1], and tslices[-1]:

  • Number of good pixels

  • Accumulated signal S_{bd} (in DN; median of mean method).

  • Estimated robust variance (from inter-quantile range) of S_{ab} (DN^2).

  • Estimated robust variance (from inter-quantile range) of S_{ad} (DN^2).

  • Correlation function out to 2 pixels of S_{ad}, i.e., C_{adad}, shape = (5,5) cenetered on (0,0).

Return type:

list

corrstats(lightfiles, darkfiles, formatpars, box, tslices, sensitivity_spread_cut, ctrl_pars)[source]

Routine to obtain statistical properties of a region of the detector across many time slices.

Parameters:

  • lightfiles (list) – The list of “light” (flat field) files.

  • darkfiles (list) – The list of dark files.

  • formatpars (int) – The file format.

  • box (list of int) – The box boundaries in the form [xmin, xmax, ymin, ymax], with the usual convention that pixels are included if xmin<=x<xmax and ymin<=y<ymax.

  • tslices (list if int) – The list of time slices to use, with length at least 2. The first two entries are tmin and tmax (with tmin<=t<tmax). If the length is exactly 2, then all time slice combinations are computed (see Returns). Otherwise, the additional entries are “deltas”. For example, [4,10,1,3] means that computations are done for 4<=ti<tj<10, but only with combinations that have tj-ti equal to 1 or 3.

  • sensitivity_spread_cut (float) – What percentage response from median flat to cut for identifying good pixels (typically 0.1).

  • ctrl_pars (class) – A class containing the control parameters as attributes. These are optional (but recommended); if specified, they follow the same format as in basic().

Returns:

data – Array of return information, shape = (nt, nt, 6), where nt is the number of time slices (so data for time slice pair ti,tj is in data[ti-tmin,tj-tmin,:]). The fields on the last axis are:

  • Number of good pixels

  • Accumulated signal S_{ab} (in DN; median of mean method).

  • Variance of S_{ab} (DN^2).

  • Correlation function for horizontal pixels, C_{abab}(1,0) (DN^2).

  • Correlation function for vertical pixels, C_{abab}(0,1) (DN^2).

  • Correlation function for diagonal pixels, C_{abab}(+/-1,+/-1), averaged over the two diagonal directions (DN^2).

Return type:

np.array

polychar(lightfiles, darkfiles, formatpars, box, tslices, sensitivity_spread_cut, ctrl_pars, addInfo, swi, corrstats_data=None)[source]

Routine to characterize of a region of the detector across many time slices.

Parameters:

  • lightfiles (list) – The list of “light” (flat field) files.

  • darkfiles (list) – The list of dark files.

  • formatpars (int) – The file format.

  • box (list of int) – The box boundaries in the form [xmin, xmax, ymin, ymax], with the usual convention that pixels are included if xmin<=x<xmax and ymin<=y<ymax.

  • tslices (list if int) – The list of time slices to use, with length at least 2. The first two entries are tmin and tmax (with tmin<=t<tmax). If the length is exactly 2, then all time slice combinations are computed (see Returns). Otherwise, the additional entries are “deltas”. For example, [4,10,1,3] means that computations are done for 4<=ti<tj<10, but only with combinations that have tj-ti equal to 1 or 3.

  • sensitivity_spread_cut (float) – What percentage response from median flat to cut for identifying good pixels (typically 0.1).

  • ctrl_pars (class) – A class containing the control parameters as attributes. These are optional (but recommended); see the Notes.

  • addInfo (list) –

    Some additional information needed for IPNL corrections to the inferred gain and IPC data. The length may be 0 (null, no corrections), 2, or 3. If there is a correction, then the entries are:

    • addInfo[0] : str Either ‘bfe’ or ‘nlipc’ (which form of IPNL to assume dominant).

    • addInfo[1] : np.array BFE kernel, shape (2s+1, 2s+1), centered at 0, units in inverse electrons.

    • addInfo[2] : np.array, optional 1D array of polynomial coefficients, needed if ctrl_pars.use_allorder is True. This is in DN-based units, starting with the quadratic coefficient (unit: DN^-1).

  • swi (class) – Column table.

  • corrstats_data (np.array, optional) – If given, saved data from corrstats() (saves time if alraedy computed).

Returns:

The list entries are [isgood (1 = good, 0 = bad), gain (e/DN), alpha_H (IPC), alpha_V (IPC), beta (1/e), Intensity (e/frame), alpha_D (IPC), change in alpha from previous iteration (residual)]. Returns the empty list [] in the event of a failure.

Return type:

list

Notes

The ctrl_pars class contains the following attributes. They follow the same format as in basic(), except that use_allorder is added:

  • epsilon : float Fraction of data points to cut for computing correlations (default 0.01)

  • subtr_corr : bool Do mean subtraction for the IPC correlation? (default to True)

  • noise_corr : bool Do noise subtraction for the IPC correlation? (default to True)

  • reset_frame : int Reset frame (default to 0)

  • subtr_href : bool Horizontal reference pixel subtraction? (default to True)

  • full_corr : bool Which parameters to report? (default to True = standard basic pars; False = correlation data instead)

  • leadtrailSub : bool Perform lead-trail subtraction? (default to False)

  • g_ptile : float Percentile for inter-quantile range (default to 75)

  • use_allorder : bool Whether to use the full polynomial expansion for the non-linearity correction? (default to False)

bfe(region_cube, tslices, basicinfo, ctrl_pars_bfe, swi, verbose)[source]

Routines to compute the BFE coefficients.

Parameters:

  • region_cube (np.array) – 4D array of the region of interest. shape: (num_files+1, nt, dy, dx), where num_files is the number of files, nt is the number of time slices, and (dy, dx) is the shape of the region on the SCA used. The last slice, region_cube[-1,:,:,:], is the mask (1 for good, 0 for bad).

  • tslices (list of int) – A list of at least 4 time slices. The first two and last two (“a”, “b”, “c”, and “d”) are used for the BFE determination.

  • basicinfo (list) – Output from basic() (inclides gains, IPC, and non-linearity).

  • ctrl_pars_bfe (class) – Parameters to control BFE determination; see Notes.

  • swi (class) – Column table.

  • verbose (bool) – Whether to talk a lot.

Returns:

The BFE kernel, shape (2s+1, 2s+1), Antilogus coefficients in inverse electrons.

Return type:

np.array

Notes

The (optional) attributes in ctrl_pars_bfe are:

- ``epsilon`` : float
  Fraction of data to cut in computing correlation coefficients (default to 0.01).

- ``treset`` : int or float
  Reset frame (default to 0). Fractional values are possible.

- ``BSub`` : bool
  Perform baseline subtraction? (default to True)

- ``vis`` : bool
  Does this have visible light information (for quantum yield)? (default to False)

- ``Phi`` : np.array
  2D quantum yield + charge diffusion kernel (only used if `ctrl_pars_bfe`.vis is True).
  This is omega/(1+omega)*p2, where 1+omega is the quantum yield (omega is the probability
  of getting 2 charges from 1 photon) and p2[s+dy,s+dx] is the pairwise probability that two charges
  generated at the same point land in the pixel (dx,dy). Here the kernel has shape (2s+1, 2s+1),
  centered at zero (so it is symmetric).
hotpix(darkfiles, formatpars, tslices, pars, verbose)[source]

Selects hot pixels.

Parameters:

  • darkfiles (list of str) – A list of the filenames of the dark exposures.

  • formatpars (int) – The format code.

  • tslices (list of int) – The time slices to read (the first slice is 1).

  • pars (np.array or array-like) – Parameters controlling the hot pixel selection. These should be [Smin, Smax, stability, f_isolation] (see Notes for detailed meaning).

  • verbose (bool) – Whether to print lots of information.

Returns:

row, col – The row and column values of the selected hot pixels. The two arrays have the same length; the i``th hot pixel is at ``[row[i],col[i]].

Return type:

np.array of int

Notes

The hot pixels in the array must meet the following criteria:

  • The apparent brightness in time slices up through tslices[-1] is assessed. Pixels are required to have a dark signal in DN between Smin and Smax.

  • Repeatable to within a top-to-bottom error of stability as a fraction of the maximum signal (e.g. 0.1 for 10% repeatability).

  • Isolation: if f_isolation>0, rejects pixels with neighbors that are at least this many times as bright as this pixel itself (e.g. 0.1 for 10% isolation).

hotpix_ipc(y, x, darkfiles, formatpars, tslices, pars, verbose)[source]

Return IPC data from a list of hot pixels.

Parameters:

  • y (np.array of int) – Tables of hot pixel coordinates to use (probably selected from hotpix()).

  • x (np.array of int) – Tables of hot pixel coordinates to use (probably selected from hotpix()).

  • darkfiles (list of str) – List of dark files to use.

  • formatpars (int) – Format code for dark files.

  • tslices (list of int) – List of time slices to report.

  • pars (list, [np.array, bool]) – Parameters to control data selection. If not empty, `pars`[0] is a map of the non-linearity times gain (units: 1/DN) and `pars`[1] is a boolean for whether the non-linearity should be referenced to a median stack of initial image.

  • verbose (bool) – Whether to talk a lot.

Returns:

Data cube with the signal in DN from the hot pixels. The shape is (npix, nt, 10), where npix is the number of hot pixels (length of y and x); and nt is the number of time stamps (length of tslices). The last index indicates the position: 0 = that pixel; 1 = right; 2 = up; 3 = left; 4 = downl; 5 = upper-right; 6 = upper-left; 7 = lower-left; 8 = lower-right; 9 = background (from the next 5x5-3x3=13 pixels out).

Return type:

np.array

slidemed_percentile(x, y, p, mrange=[-1, 1], niter=64, pivot='pos')[source]

Sliding median function.

Takes in points specified by x and y arrays, and percentile p, and returns slope m such that p% of the data are below the line y = m*x.

(If all values in x are positive, this is simply a percentile of y/x. But this version is not biased when the noise causes a few values to fluctuate negative.)

Parameters:

  • x (np.array of float) – Vectors of the same length.

  • y (np.array of float) – Vectors of the same length.

  • p (float) – Desired percentile.

  • mrange (list, optional) – Slope range to search, length 2 ([mmin, mmax]).

  • niter (int, optional) – Number of bisections to perform.

  • pivot (str, optional) – The pivot is not used but is here for forward compatibility; right now it assumes that the pivot point of the distribution is at x>0 (hence ordering in the bisection). In a future release if we need to change this the functionality is there.

Returns:

The slope of the line meeting that percentile criterion.

Return type:

float

get_vmin_vmax(mydata, qext)[source]

Generates min and max range for a color bar based on inter-quartile range.

Parameters:

  • mydata (np.array) – The data to consider

  • qext (float) – Number of interquartile ranges to extend (should be >=0, with 0 corresponding to 25th through 75th percentile).

Returns:

The minimum and maximum of the scale.

Return type:

float, float