lookout.style.format.benchmarks.quality_report_noisy

Evaluate how well a given model is able to fix style mistakes randomly added in a repository.

Module Contents

lookout.style.format.benchmarks.quality_report_noisy.REPOSITORIES
lookout.style.format.benchmarks.quality_report_noisy.Misprediction
lookout.style.format.benchmarks.quality_report_noisy.train(training_dir:str, ref:ReferencePointer, output_path:str, language:str, bblfsh:str, config:Optional[Union[str, dict]], log:Optional[logging.Logger]=None)

Train a FormatModel for debugging purposes.

Parameters:
  • training_dir – Path to the directory containing the files to train from.
  • ref – Reference pointer to repository for training
  • output_path – Path to the model to write.
  • language – Language to filter on.
  • bblfsh – Address of the babelfish server.
  • config – Path to a YAML config to use during the training or json-like object with a config.
  • log – logger used to report during training.
Returns:

Trained FormatNodel.
lookout.style.format.benchmarks.quality_report_noisy.get_content_from_repo(folder:str)

Extract the content of the files given their path.

Parameters:folder – Path to the files to get the content from.
Returns:Dictionary where the key is the path to a file and its value the content of the file.
lookout.style.format.benchmarks.quality_report_noisy.get_difflib_changes(true_content:Mapping[str, str], noisy_content:Mapping[str, str])

Return the files and the first offsets that have been changed when adding random noise.

Given 2 contents of one repository (the original and its noisy version), returns the list of files that have been modified, the first offsets that have been changed.

Parameters:
  • true_content – Dictionary containing the content of the original repository.
  • noisy_content – Dictionary containing the content of the noisy version of the repository.
Returns:

The list of files where a style mistake has been added, and the mirror list of the original files, and the dictionary of firsts offsets that have been changed when adding random noise.
lookout.style.format.benchmarks.quality_report_noisy.files2vnodes(filepaths:Iterable[str], feature_extractor:FeatureExtractor, rules:Rules, client:BblfshClient)

Return the VirtualNode-s extracted from a list of files.

Parameters:
  • filepaths – List of files to get Misprediction-s and VirtualNode-s from.
  • feature_extractor – FeatureExtractor to use.
  • rules – Rules to use for prediction.
  • client – Babelfish client. Babelfish server should be started accordingly.
Returns:

List of VirtualNode-s extracted from a given list of files.
lookout.style.format.benchmarks.quality_report_noisy.files2mispreds(filepaths:Iterable[str], feature_extractor:FeatureExtractor, rules:Rules, client:BblfshClient, log:logging.Logger)

Return the model’s Misprediction-s on a list of files.

Parameters:
  • filepaths – List of files to get Misprediction-s from.
  • feature_extractor – FeatureExtractor to use.
  • rules – Rules to use for prediction.
  • client – Babelfish client. Babelfish server should be started accordingly.
  • log – Logger.
Returns:

List of Misprediction-s extracted from a given list of files.
lookout.style.format.benchmarks.quality_report_noisy.get_mispreds(y:numpy.ndarray, y_pred:numpy.ndarray, nodes:Iterable[VirtualNode], winners:numpy.ndarray)

Return the list of Misprediction-s where the labels differ.

Parameters:
  • y – Numpy 1-dimensional array of labels.
  • y_pred – Numpy 1-dimensional array of predicted labels by the model.
  • nodes – List of VirtualNode-s.
  • winners – Numpy 1-dimensional array of the winning rule indices for each sample.
Returns:

List of Misprediction-s where the labels y and y_pred differ.
lookout.style.format.benchmarks.quality_report_noisy.get_diff_mispreds(mispreds:Iterable[Misprediction], start_changes:Mapping[str, int])

Filter Misprediction-s to select those involving at least one line that has been modified.

Parameters:
  • mispreds – List of Misprediction-s to filter.
  • start_changes – Dict of first offsets that have been changed when adding random noise.
Returns:

Dictionary of the Misprediction-s located at the offset where a random mistake has been previously added.
lookout.style.format.benchmarks.quality_report_noisy.get_style_fixes(mispreds:Mapping[str, Misprediction], vnodes:Iterable[VirtualNode], true_files:Iterable[str], noisy_files:Iterable[str], feature_extractor:FeatureExtractor)

Return Misprediction-s that fix the style mistakes added.

Given a list of Misprediction-s potentially fixing a style mistake added since involving at least one line that has been modified, return the list of Mispredicitons really fixing that mistake because their prediction on the noisy files would match the ground truth labels of the original files.

Parameters:
  • mispreds – Dictionary of Misprediction-s potentially fixing a style mistake.
  • vnodes – List of VirtualNode-s extracted from the list of true_files.
  • true_files – list of files of the original repos where a style mistake has been added.
  • noisy_files – list of files from the noisy repos where a modification has been made
  • feature_extractor – FeatureExtractor used to extract features.
Returns:

List of Misprediction-s where the prediction on a noisy file matches the ground truth label of the original file i.e. Misprediction-s actually fixing the mistakes added.
lookout.style.format.benchmarks.quality_report_noisy.compute_metrics(changes_count:int, predictions_count:int, true_positive:int)

Compute precision, recall and F1-score metrics.

Parameters:
  • changes_count – Overall number of cases.
  • predictions_count – Total number of predictions made by the model.
  • true_positive – Number of positive cases predicted as positive.
Returns:

Prediction rate and precision metrics.
lookout.style.format.benchmarks.quality_report_noisy.plot_curve(repositories:Iterable[str], prediction_rates:Mapping[str, numpy.ndarray], precisions:Mapping[str, numpy.ndarray], precision_threshold:float, limit_conf_id:Mapping[str, int], path_to_figure:str)

Plot y versus x as lines and markers using matplotlib.

Parameters:
  • repositories – List of the repository names we plot the precision-recall curve.
  • prediction_rates – Dict of 1-D numpy array containing the x coordinates.
  • precisions – Dict of 1-D numpy array containing the y coordinates.
  • precision_threshold – Precision threshold tolerated by the model. Limit drawn as a red horizontal line on the figure.
  • limit_conf_id – Dict of last accepted rule indices according to the maximum confidence threshold observed.
  • path_to_figure – Path to the output figure, in png format.
lookout.style.format.benchmarks.quality_report_noisy.quality_report_noisy(bblfsh:str, language:str, confidence_threshold:float, support_threshold:int, precision_threshold:float, dir_output:str, config:Optional[dict]=None, repos:Optional[str]=None)

Generate a quality report on the artificial noisy dataset including evaluation curves.

Parameters:
  • bblfsh – Babelfish client. Babelfish server should be started accordingly.
  • language – Language to consider, others will be discarded.
  • confidence_threshold – Confidence threshold to filter relevant rules.
  • support_threshold – Support threshold to filter relevant rules.
  • precision_threshold – Precision threshold tolerated by the model. Limit drawn as a red horizontal line on the figure.
  • dir_output – Path to the output directory where to store the quality report in Markdown and the precision-recall curve in png format.
  • config – FormatAnalyzer config to use. Default one is used if not set.
  • repos – Input list of urls to the repositories to analyze. Should be strings separated by newlines. If it is None, we use the string defined at the beginning of the file.