Skip to content

Development

This document describes some of the development practices used in MLTE.

Quick Start

The best examples of how to use MLTE are contained with in the Demos. If looking to get started on development, these are the best place to start to get a feel for how the tool works. The code within the demos walks through the MLTE library SDMT process. These can be ran after making a virtual environment and installing MLTE along with the demo dependencies.

$ pyenv install 3.12
$ pyenv local 3.12
$ make venv

The other part of MLTE is the frontend and backend. These are used for visualizing the results of the SDMT process and provide a user-friendly interface to create the Negotiation Card . This can be setup manually or by using the run_environment.sh script. This will start MLTE as 2 docker containers with an empty file system based store. If the script demo/run_environment.sh is used instead, the store will be populated with the data in demo/store, which will include sample models, versions and Negotiation Cards along with any results created by running the demo notebooks. This is volume mounted so any changes made using the frontend will be available locally in that same directory. The frontend will be available at localhost:8000.

For an empty store:

bash run_environment.sh

To use the demo store:

cd demo && bash run_environment.sh

All issue tracking for MLTE is done in this Github Project [Github Project].

Once changes have been made, make qa and make test should be ran to ensure that code fits QA standards and all unit tests pass. If just making python changes, make qa-python can be used instead of make qa to avoid having to setup a Node.js environment for the frontend QA.

Setup

Python Version Support

Currently, MLTE supports Python versions between 3.10 and 3.13, both included.

If you do not have one of these versions installed, or you want to target a specific version that is not your default, pyenv can be used to manage multiple Python versions locally. Note that this is optional, and only needed if you have a not-supported default Python version. To set up a specific version of Python with pyenv:

  • Install pyenv as described in this link: https://github.com/pyenv/pyenv
  • Install the desired Python version (in this example, 3.12):
$ pyenv install 3.12
  • While inside the root repository folder, run this command to set that Python version to be used when executed in that folder:
$ pyenv local 3.12
  • You can use python --version to check if it worked.

Requirements

  • MLTE uses poetry (v 2.0.1 or higher) to handle the required runtime and development packages. You can install poetry on your system with the instructions available here: https://python-poetry.org/docs/#installation

Dev Environment Setup

You will need to set up a virtual Python environment where poetry will work, and install all dependencies there. The easiest way to do this, installing all dependencies, is to run this command:

$ make venv

If you want more control over what is being installed, you can do it manually instead. While inside the root of the repository, execute these commands (which do not install the demo dependencies):

$ python -m venv .venv
$ poetry install --with dev --all-extras

Now you are ready to start working on MLTE!

Demos

There are several demos available in the demo/ folder, as Jupyter notebooks. To run them, you need to install their dependencies first if you created the environment manually; otherwise they have already been installed for you. To install them manually. run:

$ poetry install --with demo

You can go to the Jupyter notebooks in the subfolders inside the demo/ folder and try them out in order to see how MLTE works. This assumes you are running the Jupyter notebooks from the same virtual environment that was just set up in the step above.

If you want to run the frontend UI and backend in an environment that will allow you to see the results of the artifacts created by the demos, you can run the following script, which will run them inside a container and point them to the proper store:

$ cd demo
$ bash run_environment.sh

Information relating to creating a new demo can be found here.

Project Development Commands

You can run most project commands (e.g., format sources, lint) in two ways: using the commands in the included Makefile, or running things manually. Using the Makefile works on UNIX-like systems (or anywhere make is available), and is shorter to type. Alternatively, you can run each command manually. The sections below describe how to run commands in both ways.

Also, the commands below do not assume that you have your virtual environment enabled. Calling poetry run ensures things run in the current virtual environment even if it is not activated. If you manually activate your virtual environment you can run all the commands below without the poetry run prefix.

To manually activate your environment, run:

$ source .venv/bin/activate

Make Overview

There are a couple of shorthand commands in the Makefile to run several of the below commands at the same time. The most useful ones include:

  • make qa: executes the schema generation, doc check, source sorting, formatting, linting, static type checking commands of Python and Typescript code, and sample test catalog generation.
  • make check-qa: executes the schema check, doc check, source sorting check, formatting check, linting check, static type checking commands of Python and Typescript code and sample catalog check.
  • make ci: executes the same commands as check-qa, but also runs test to execute the unit tests, cleaning caches first to better simulate execution in a CI environment.

make qa and make test should be ran before every push to ensure that the changes made adhere to the QA standards and will pass the unit tests. These two encapsulate all of the commands below that are generally applicable.

If you want to run either the frontend or backend QA/CI without setting up the environment, there are scripts available to run them within containers. These will mount your local version as a volume and run the QA/CI with the environment in the container while applying the results to your local copy.

$ make qa-python-docker
$ make ci-python-docker
$ make qa-frontend-docker
$ make ci-frontend-docker

Import Sorting

We sort all Python import code in this project with isort. You can run this locally with:

$ make isort

If you just want to check the sorting of the imports, without making any changes, you can run this:

$ make check-isort

Code that does not satisfy the sorter will be rejected from pull requests.

Source Formatting

We format all Python code in this project with the black source formatter. You can run the formatter locally with:

$ make format

If you just want to check the format, without making any changes, you can run this:

$ make check-format

Code that does not satisfy the formatter will be rejected from pull requests.

Source Linting

We lint all Python code in this project with the flake8 source linter. You can run the linter locally with:

$ make lint

Code that does not satisfy the linter will be rejected from pull requests.

Static Type Checking

We run static type-checking with mypy. You can run the type-checker locally with:

$ make typecheck

Code that does not satisfy static type-checking will be rejected from pull requests.

Doc Generation

More details in the Documentation section, but to run document generation from the make command, run this: 

$ make docs

Unit Tests

We unit test the MLTE library with the pytest package, a test runner for Python. You can run unit tests locally with:

$ make test

Unit test failures result in build failures in CI.

To test the Juypter notebooks present in the demo folders, run:

$ make demo-test

Model Schema Generation

The artifacts used by MLTE have schemas that are used to validate them. These schemas need to be updated if their internal structure (code) changes. You can do this locally with:

$ make schema

If you just want to check the schemas, without making any changes, you can run this:

$ make check-schema

Schema failures result in build failures in CI.

Front End Formatting and Linting

We format and lint all .vue, .js, and .ts files with ESLint, which can be run from the root of the repository with:

$ make lint-frontend

Front End Static Type Checking

All typescript code takes advantage of static typing. This type checking can be done by running the following command from the root of the repository:

$ make typecheck-frontend

Front End

Front end development requires Node.js. The front end was developed using v20.11.0; the latest LTS version can be found here.

To initialize the development environment for the front end, navigate to the subfolder ./mlte/frontend/nuxt-app and run:

$ npm install
$ npx gulp init

You can also run the following make command:

$ make frontend-env

Now the environment is set up and the front end can be run with the following command:

$ npm run dev

This will run the front end at http://localhost:3000. The backend can be run with a command like this one (using a file system store, in the local ./store folder):

$ mlte backend --store-uri fs://store

Continuous Integration

We utilize GitHub Actions for continuous integration.

Documentation

We build documentation with mkdocs and host documentation on ReadTheDocs. A webhook is set up in the MLTE repository to trigger an integration effect on ReadTheDocs when certain changes to the repo are made.

You can build and serve the documentation with the following command, when run from inside the docs/ folder:

$ mkdocs serve

You can preview the documentation accessing http://127.0.0.1:8000/ on your browser.

Versioning and Publishing

Versioning

We follow semantic versioning when versioning the MLTE package. We use bump2version to consistently update versions across the project. A configuration file is provided in .bumpversion.cfg at the root of the project.

Bumping the version for a new release can be accomplished with:

$ make bump-patch

or

$ make bump-minor

or

$ make bump-major

depending on whether you want to update the patch,minor or major version as appropriate for the release. NOTE: bumpversion will change all instances of the current version's text to the new version's text in the files it has been configured to do so, so if you have other text in these files which happens to match the current version's, it will be incorrectly changed. Manually inspect changes after running this tool, and discard any incorrect ones.

Publishing

We publish the MLTE package on PyPi. Ensure you have properly incremented the version for the new release, as described in Versioning.

To build the frontend and then the whole package, it is enough to execute the following command from the main repo folder. Note that you need to have the venv and node environments properly setup before this will work.

$ make build-local

To build it in a Docker environment, instead of your local environment:

$ make build-in-docker

Once the package is built, publish the package to PyPi using a PyPi API token:

$ poetry publish --username __token__ --password <TOKEN>

Docker Integration

We package the MLTE backend and frontend as a set of Docker container images. To build and run these from the source repository, run:

# From inside the docker/deployment folder
bash rebuild_and_restart.sh

This exposes the backend on the host at localhost:8080, and the frontend at localhost:8000. By default, PostgreSQL database is used in a container, and the data is mapped to the local ./pgdata folder.

You can CTRL+C to stop seeing the output in the console, but the containers will continue running. You can check the current logs at any time with:

# From inside the docker/deployment folder
bash logs.sh

Stop the containers with:

# From inside the docker/deployment folder
bash stop.sh

Contributing

To contribute to MLTE, check out our GitHub repository!