Software Failure and Reliability Assessment Tool (SFRAT)


Description

The key to the success of all software is its reliability. The Software Failure and Reliability Assessment Tool (SFRAT) is an open source application to estimate and predict the reliability of a software system during test and operation. It allows users to answer the following questions about a software system during test:

  1. Is the software ready to release (has it achieved a specified reliability goal)?
  2. How much more time and test effort will be required to achieve a specified goal?
  3. What will be the consequences to the system’s operational reliability if not enough testing resources are available?
SFRAT runs under the R statistical programming framework and can be used on computers running Windows, Mac OS X, or Linux

If you benefit from the SFRAT in your work or research, please consider citing the following open access article to make others aware of this free and open source tool: V. Nagaraju, V. Shekar, J. Steakelum, M. Luperon, Y. Shi, and L. Fiondella, 2019. Practical software reliability engineering with the Software Failure and Reliability Assessment Tool (SFRAT). SoftwareX, 10, p.100357.

Resources

WARNING: Web instance is for demonstration only. Please do not upload sensitive data to the site
Web instance
Example failure data sets
SFRAT Github repository
User's Guide
Contributor's Guide

Automated Report generation script
Script instruction manual

Contributions

  1. Dr. Shinji Inoue from Kansai University, Kansai, Osaka, Japan translated the GUI to Japanese version. Available at:
    SFRAT - Japanese Version
  2. Dr. Hiroyuki Okamura from Hiroshima University, Hiroshima, Japan contributed 11 additional models to the tool. Available at:
    SFRAT_Okamura
  3. Aerospace Corporation contributed three models. Available at:
    SFRAT_Aerospace
  4. Barry Von Tobel from MITRE developed a script to automatically transport failure data collected from JIRA tool to SFRAT compatible data format. Available at:
    JIRA export data to SFRAT data format
NOTE: If you are interested in helping to create a translation of the tool interface and manual in your language, please contact Dr. Fiondella

Software Defect Estimation Tool (SweET)

Description

The Software Defect Estimation Tool (SweET) is an open source application to track error identification and removal efforts during the software development lifecycle. SwEET is a free and open source version of the SoftWare Error Estimation Program (SWEEP) and SweET uses Weibull software reliability growth model utilizing Expectation Conditional Maximization algorithm to ensure stability and performance of the model fitting process. SweET simplifies four models of SWEEP into three modes:

  1. Mode A: Time-based model: Estimates and tracks errors during system test and integration cycles.
  2. Mode B: Phase-based and planning aid model: Predict and track defects for multiple phases and can provide defect information before running any code, whereas the planning aid model generates an error discovery profile based on the phase based historical data to help a software prohect achieve its objectives.
  3. Mode C: Defect injection model: Allows the user to understand the probable defect injection profile and resulting efficiency and effectiveness of the verification process.
SweET runs under the Python 3.x programming framework and can be used on computers running Windows, Mac OS X, or Linux

Resources

Example data sets
SweET Github repository
User's Guide

Publications

Invited Talks

  1. Covariate Software Reliability Models and Applications, Oak Ridge National Laboratory, Oak Ridge, TN, June 8, 2020.
  2. Covariate Software Reliability Models and Applications, Virginia Tech Research Center, Arlington, VA, March 12, 2020.
  3. Continuous Integration, Software Reliability in the 21st Century Panel Discussion held at the 66th Annual Reliability and Maintainability Symposium (RAMS 2020), Palm Springs, CA, Jan 28, 2020.
  4. Towards a Guide for Software Defect Tracking, Modeling, and Analysis, NASA Software Assurance Research Program Review, Greenbelt, MD, February 6, 2020.
  5. Software Reliability and Security Modeling and Analysis from Software Defect Tracking Data, NASA Goddard Space Flight Center, Software Engineering Division, Code 580, November 19, 2019.
  6. Relationships between Machine Learning and Reliability Engineering, Air Force Institute of Technology, Dayton, OH, October 31, 2019.
  7. Software Reliability Engineering (two day course), BAE Systems, Nashua, NH, January 16-17, 2019.
  8. Software Reliability Engineering: Algorithms and Tools, Rutgers University, New Brunswick, NJ, December 20, 2018.
  9. Practical Software Reliability Modeling and Application, NASA Software Assurance Working Group Meeting, Mountain View, CA, July 31, 2018.
  10. Software Reliability Engineering: Algorithms and Tools, Indian Institute of Science, Bangalore, India, June 12, 2018.
  11. Software Reliability Engineering: Algorithms and Tools, Indian Institute of Technology Kharagpur, Karagpur, India, May 31, 2018.
  12. Software Reliability Engineering: Algorithms and Tools, Amity University, Noida, India, May 29, 2018.
  13. Tools to Assess Software Reliability and Security, MIT Lincoln Laboratory, Lexington, MA, April, 30, 2018.
  14. Software Reliability Engineering: Algorithms and Tools, Auburn University, Auburn, AL, March 27, 2018.
  15. Practical Software Reliability Modeling and Application, NASA Goddard Space Flight Center, Greenbelt, MD, March 7, 2018.
  16. Software Reliability Engineering: Algorithms and Tools, Old Dominion University, Norfolk, VA, February 15, 2018.
  17. Software Reliability Engineering: Algorithms and Tools, Virginia Commonwealth University, Richmond, VA, February 1, 2018.
  18. Software Reliability Engineering: Algorithms and Tools, Florida International University, Miami, FL, January 20, 2018.
  19. Free and Open Source Tools to Assess Software Reliability and Security, Boston Software Process Improvement Network (SPIN), Bedford, MA, November 21, 2017.
  20. Tools to Assess Software Reliability and Security, Unmanned System Common Control System (CCS) Integrated Product Team Meeting, Patuxent River, MD, December 5, 2017.
  21. ECM Algorithms & An Open Source Tool, Tottori University, Tottori, Japan, July 4, 2017.
  22. Software Reliability Engineering: Algorithms and Tools, Florida State University, Tallahassee, FL, February 27, 2017.
  23. Software Reliability Research - Assessments & Tools, Office of the Secretary of Defense, Developmental Test and Evaluation Functional Integrated Product Team Meeting, Washington, DC, Jan 11, 2017.
  24. Software Reliability Research - Assessments & Tools, Office of the Deputy Assistant Secretary of Defense for Developmental Test & Evaluation, Alexandria, VA, Oct 28, 2016.
  25. Software Reliability: Tools and Algorithms, IEEE Boston Reliability Society, Lexington, MA, Sep 14 2016.
  26. Software Reliability Engineering: Algorithms and Tools, Naval Postgraduate School, Monterey, CA, May 27 2016.
  27. Software Reliability: Tools and Models, Workshop on Rigorous Test and Evaluation for Defense, Aerospace, and National Security, Arlington, VA, April 12, 2016.
  28. Algorithms and Tools for Software Reliability Engineering, University of Maryland, Dec 2, 2015.
  29. An Open Source Software Reliability Tool and Model Fitting Algorithm, Wright State University, Oct 7, 2015.
  30. Software Reliability Engineering: Practice and Theory, UMass- Naval Undersea Warfare Center (NUWC) Lecture Series, Nov 20, 2014.

Presentations/Tutorials

  1. V. Nagaraju, C. Jayasinghe, and L. Fiondella, Covariate Software Reliability Models and Applications, Presented to WG17 Logistics, Reliability and Maintainability and WG24 Test and Evaluation (T&E) and Experimentation, at the 88th Military Operations Research Symposium (MORS 2020), New London, CT, June 2020.
  2. M. Nafreen, M. Luperon, V. Nagaraju, Y. Shi, and L. Fiondella, Connecting Software Reliability Growth Models to Software Defect Tracking, Presented to WG17 Logistics, Reliability and Maintainability and WG24 Test and Evaluation (T&E) and Experimentation, at the 88th Military Operations Research Symposium (MORS 2020), New London, CT, June 2020.
  3. A. Gula, C. Ellis, S. Bhattacharya, and L. Fiondella, Relationships between Machine Learning and Reliability Engineering, Presented to WG17 Logistics, Reliability and Maintainability and WG35 AI and Autonomous Systems, at the 88th Military Operations Research Symposium (MORS 2020), New London, CT, June 2020.
  4. M. Nafreen, M. Luperon, V. Nagaraju, Y. Shi, and L. Fiondella, Connecting Software Reliability Growth Models to Software, Presented at the Defense and Aerospace Test and Analysis (DATA) Workshop, Springfield, VA, May, 2020.
  5. A. Gula, C. Ellis, S. Bhattacharya, and L. Fiondella, Software and System Reliability Engineering for Autonomous Systems incorporating Machine Learning, Poster presented at the 66th Annual Reliability and Maintainability Symposium (RAMS 2020), Palm Springs, CA, Jan 2020.
  6. M. Nafreen, S. Bhattacharya, and L. Fiondella, Architecture-based Software Reliability incorporating Fault-Tolerant Machine Learning, Poster presented at the 66th Annual Reliability and Maintainability Symposium (RAMS 2020), Palm Springs, CA, Jan 2020.
  7. V. Nagaraju, Y. Shi, and L. Fiondella, Software Reliability and Security Assessment: Automation and Frameworks, Presented at the Defense and Aerospace Test and Analysis (DATA) Workshop, Springfield, VA, April, 2019.
  8. V. Nagaraju, L. Fiondella, Introduction to Mathematical Software Reliability Models, In Proc. 65th Annual Reliability and Maintainability Symposium (RAMS 2019), Orlando, FL, Jan 2019.
  9. V. Nagaraju, L. Fiondella, and T. Wandji, A Single Changepoint Software Reliability Growth Model with Heterogeneous Fault Detection Processes, Poster presented at the Society of Risk Analysis (SRA) Annual Meeting, Arlington, VA, December, 2017.
  10. V. Nagaraju, T. Wandji, and L. Fiondella, Free and Open Source Tools to Assess Software Reliability and Security, Presented at the National Defense Industrial Association (NDIA) Annual Systems Engineering Conference, Springfield, VA, October, 2017.
  11. V. Nagaraju, T. Wandji, and L. Fiondella, Free and Open Source Tools to Assess Software Reliability and Security, Presented at the National Defense Industrial Association (NDIA) Annual Systems Engineering Conference, Springfield, VA, October, 2017.
  12. V. Nagaraju, T. Wandji, and L. Fiondella, Software Reliability: Modeling and Tools, Tutorial presented at the 85th Military Operations Research Symposium (MORS 2017), West Point, NY, June 2017.
  13. V. Nagaraju, T. Wandji, and L. Fiondella, Software Reliability Modeling, Presented at the Science of Test Workshop, Springfield, VA, April, 2017.
  14. V. Nagaraju, L. Fiondella, T. Wandji, Introduction to Mathematical Software Reliability Models, In Proc. 63nd Annual Reliability and Maintainability Symposium (RAMS 2017), Orlando, FL, Jan 2017.
  15. V. Nagaraju, T. Wandji, and L. Fiondella, Software Failure and Reliability Assessment Tool (SFRAT): An Open Source Application for the Practitioner and Research, Presented at the Conference on Applied Statistics in Defense, Washington, DC, October, 2016.
  16. L. Fiondella, A. Nikora, and T. Wandji, Software Reliability and Security: Challenges and Crosscutting Themes, Fast abstract In Proc. International Symposium on Software Reliability Engineering, Ottawa, Canada, Nov, 2016.
  17. V. Nagaraju, T. Wandji, and L. Fiondella, An Open Source Tool to support Software Reliability Assessment, Presented to WG 17 Logistics, Reliability and Maintainability and WG 24 Test and Evaluation (T&E) at the 84rd Military Operations Research Symposium (MORS 2016), Quantico, VA, June 2016. Nominated for the 85th Barchi Prize.
  18. V. Nagaraju and L. Fiondella, Introduction to Mathematical Software Reliability Models, In Proc. 62nd Annual Reliability and Maintainability Symposium (RAMS 2016), Tucson, AZ, Jan 2016.
  19. V. Nagaraju and L. Fiondella, An Adaptive EM Algorithm for NHPP Software Reliability Models, UMassD Sigma Xi Research Exhibition, Apr 2015.
  20. L. Fiondella and T. Wandji, An Open Source Application Architecture for Software Reliability Models, In Proc. International Applied Reliability Symposium (ARS), Tucson, AZ, Jun 2015.

Acknowledgements

This material is based upon work supported by the National Science Foundation under Grant Number s (1526128 and 1749635), the Naval Air Systems Command (NAVAIR) through the Systems Engineering Research Center, under Research Task 139, and by NAVAIR under contract N00421-16-T-0373.

Work at JPL was supported under NASA Prime Contract NNN12AA01C, under Task Plan No. 81-19648.