Poetry a smart way of packaging and dependency management in Python
Packaging and dependency management in python is the a most dreadful thing to most of us. Even for seasoned developers it might be cumbersome at times to create and manage all files needed in a Python project.
I was looking for a solution to overcome these problems and after spending some time on internet I found Poetry is the most suitable tool for me. From their official site this is the introductory lines about Poetry.
“Poetry is a tool for dependency management and packaging in Python. It allows you to declare the libraries your project depends on and it will manage (install/update) them for you.”
Poetry is not only packaging and dependency management tool it’s more than that you will get realize it while moving gradually toward the end of this blog. Here are my notes and experience about Poetry starting from creating project through managing dependency, build, install, writing unitetest, prebuild checks and at the end publish. Hope you will like it and I would encourage you to try it.
Poetry is developed by Sébastien Eustace and its first version 0.1.0 got released on 28-Feb-2018 and its latest current version is 1.0.5. On this date Poetry has 169 contributors and there are more that 80 releases has marked. Git Repo: https://github.com/python-poetry/poetry. Let see step by step explanation of its usage and implementation.
Step 1 : Poetry Installation
This is the command to install poetry on your system. It will support Linux/Mac/Windows all of these platform.
curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | pythonOnce it got installed type poetry — help on terminal it will show its usage, cmd arguments, Global Options and available commands that it has. Poetry CLI has a nice and simpler interface to use all the features available with it and if you closely observe this has more similarity with the RUST cargo tool which is use for same purpose.
paragkamble@Parags-MacBook-Pro ~ % poetry --help
Poetry version 1.0.5
USAGE
poetry [-h] [-q] [-v [<...>]] [-V] [--ansi] [--no-ansi] [-n] <command> [<arg1>] ... [<argN>]
ARGUMENTS
<command> The command to execute
<arg> The arguments of the command
GLOBAL OPTIONS
-h (--help) Display this help message
-q (--quiet) Do not output any message
-v (--verbose) Increase the verbosity of messages: "-v" for normal output, "-vv" for more verbose output and "-vvv"
for debug
-V (--version) Display this application version
--ansi Force ANSI output
--no-ansi Disable ANSI output
-n (--no-interaction) Do not ask any interactive question
AVAILABLE COMMANDS
about Shows information about Poetry.
add Adds a new dependency to pyproject.toml.
build Builds a package, as a tarball and a wheel by default.
cache Interact with Poetry's cache
check Checks the validity of the pyproject.toml file.
config Manages configuration settings.
debug Debug various elements of Poetry.
env Interact with Poetry's project environments.
export Exports the lock file to alternative formats.
help Display the manual of a command
init Creates a basic pyproject.toml file in the current directory.
install Installs the project dependencies.
lock Locks the project dependencies.
new Creates a new Python project at <path>.
publish Publishes a package to a remote repository.
remove Removes a package from the project dependencies.
run Runs a command in the appropriate environment.
search Searches for packages on remote repositories.
self Interact with Poetry directly.
shell Spawns a shell within the virtual environment.
show Shows information about packages.
update Update the dependencies as according to the pyproject.toml file.
version Shows the version of the project or bumps it when a valid bump rule is provided.Step 2: Create Project
To create a project (or you can say workspace) this is the command poetry new < project name >. It will create a directory structure for the project let see each of these in detail.
- README.rst : We can write a doc for the project in this file.
- mypackage : This directory will contain the all project source code and this directory name will always be the project name.
- pyproject.toml : this is a main configuration file for the project and this is the file of replacement of setup.py, requriment.txt, MANIFEST.INI etc. this is the most important file of the project we will look in detail in below section
- tests : It will having all the unittest for the project
If you have observed there is pyproject.toml file in the project Dir that is the file which we can say replacement of setup.py, requirements.txt, setup.cfg, MANIFEST.in and the newly added Pipfile.
When we create a project along with it virtualenv also get created and all the package resolution is getting handle there.
paragkamble@Parags-MacBook-Pro projects % poetry new mypackage
Created package mypackage in mypackage
paragkamble@Parags-MacBook-Pro projects % tree
.
└── mypackage
├── README.rst
├── mypackage
│ └── __init__.py
├── pyproject.toml
└── tests
├── __init__.py
└── test_mypackage.py
3 directories, 5 filespyproject.toml
This file has various section and some of them we are covering here.
paragkamble@Parags-MacBook-Pro projects % cat mypackage/pyproject.toml
[tool.poetry]
name = "mypackage"
version = "0.1.0"
description = ""
authors = ["Parag Kamble <kamble.parag@gmail.com>"]
[tool.poetry.dependencies]
python = "^3.7"
[tool.poetry.dev-dependencies]
pytest = "^5.2"
[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"- [tool.poetry] This section contain the metadata about package such as name, description, author details, etc. Most of the config values are optional unless you’re planning on publishing this package to PyPi.
- [tool.poetry.dependencies] Here is the list of dependencies for package required to download to run. We can specify the specific version of package or just put “*” to install latest version of the package. To add these dependencies we need not to edit file and add those package version and the simplest way is “poetry add <module>” this command will do it for us.
- [tool.poetry.dev-dependencies]: Dev dependencies are packages that contributing developers should download to iterate on this project. Dev dependencies are not required to run the app, and won’t be downloaded when the app is built by default. Some of the examples like pytest, black, pre-commit kind of packages comes under this section.
- [tool.poetry.scripts]: Here we can define the entry point of package so the argument must be a valid module and method name e.g. within our package my_package, you have log_revision.py, which has a method start(). Then you have to write:
[tool.poetry.scripts]
my-script = "my_package.log_revision:start"With this set, we can then launch our app via the Poetry CLI by typing “poetry run” command.
- [tool.poetry.urls]: This is a completely optional section where you can add any number of helpful links or resources that somebody downloading this package might find useful.
Step 3: Adding packages in poetry
“poetry add” command install required packages to your project and installs them. There are two types of package dependency get manage one is project related dependency means those packages required to run the project and another development related dependencies like pytest, black, pre-commit etc. we can see this difference in pyproject.toml file there are two separate section get maintain as tool.poetry.dependencies and tool.poetry.dev-dependencies.
On bellow terminal snippet I have added pendulum package to my project as this is my project required package hence it get listed in tool.poetry.dependencies section and if i want to ad this in dev dependencies I just need to add — dev option with the package add command.
paragkamble@Parags-MacBook-Pro mypackage % poetry add pendulum
Creating virtualenv mypackage-nSFcQrwE-py3.7 in /Users/paragkamble/Library/Caches/pypoetry/virtualenvs
Using version ^2.1.0 for pendulum
Updating dependencies
Resolving dependencies... (8.8s)
Writing lock file
Package operations: 14 installs, 0 updates, 0 removals
- Installing zipp (3.1.0)
- Installing importlib-metadata (1.6.0)
- Installing pyparsing (2.4.7)
- Installing six (1.14.0)
- Installing attrs (19.3.0)
- Installing more-itertools (8.2.0)
- Installing packaging (20.3)
- Installing pluggy (0.13.1)
- Installing py (1.8.1)
- Installing python-dateutil (2.8.1)
- Installing pytzdata (2019.3)
- Installing wcwidth (0.1.9)
- Installing pendulum (2.1.0)
- Installing pytest (5.4.2)There are several ways to add package like with specific version, specific git branch, tag or revision or make them point to a local directory or file.
poetry add pendulum="2.0.0"
poetry add git+https://github.com/sdispater/pendulum.git
poetry add git+https://github.com/sdispater/pendulum.git#develop
poetry add git+https://github.com/sdispater/pendulum.git#2.0.5
poetry add ./my-package/
poetry add ../my-package/dist/my-package-0.1.0.tar.gz
poetry add ../my-package/dist/my_package-0.1.0.whlpoetry.lock
This is the most important thing is placed in your poetry project, it has record of all the packages dependencies and the exact versions of them. You should commit the poetry.lock file to your project repo so that all people working on the project are locked to the same versions of dependencies.
There are commands are available to remove and show packages from your project. those are works as per their meaning so i am not covering here.
Step 4 : Writing unittests
When the project structure gets build it has a “tests” dir to manage all unitetest scripts files. It has a predefine one test ready to test the project version likewise we can add more unittest tests in to that.
paragkamble@Parags-MacBook-Pro project % pytest
==================== test session starts =============================
platform darwin -- Python 3.7.7, pytest-5.3.5, py-1.8.1, pluggy-0.13.1
sensitiveurl: .*
rootdir: /Users/paragkamble/gitprojects/testpoetry/project
plugins: html-2.0.1, variables-1.9.0, base-url-1.4.1, flaky-3.6.1, selenium-1.17.0, metadata-1.8.0
collected 1 item
tests/test_project.py . [100%]
========================== 1 passed in 0.03s =========================Step 5 : Build Validation
I am usually installing black, pre-commit, pylint kind of tools as a dev dependencies with my project so it will help me to auto format and validate the code as per the rules defined.
paragkamble@Parags-MacBook-Pro myproject % poetry run black myproject
All done! ✨ 🍰 ✨2 files left unchanged.
Step 6 : Poetry check
This is the step should run before the making build from project. This command validates the structure of the pyproject.toml file and returns a detailed report if there are any errors.
paragkamble@Parags-MacBook-Pro project % poetry check
All set!
paragkamble@Parags-MacBook-Pro project % poetry check
[UnexpectedCharError]
Unexpected character: ',' at line 9 col 17Step 7 : Poetry Build
This command will Builds the source and wheels archives and this will not includes the dev dependencies.
paragkamble@Parags-MacBook-Pro project % poetry build
Building project (0.1.0)
- Building sdist
- Built project-0.1.0.tar.gz
- Building wheel
- Built project-0.1.0-py3-none-any.whlStep 8: Publishing Project
After building the package there is a provision to publish package to the PyPI. ( Python Package Index (PyPI) is a repository of software for the Python programming language. )
paragkamble@Parags-MacBook-Pro project % poetry publish
Publishing flake8-markdown (0.1.1) to testpypi
Username:
Password:
- Uploading flake8-markdown-0.1.1.tar.gz 100%
- Uploading flake8_markdown-0.1.1-py3-none-any.whl 100%Step 9: Poetry install
“Poetry install” command reads the pyproject.toml file from the current project, resolves the dependencies, and installs them. and the exact package version to install it will get from the poetry.lock file. This ensures that everyone using the library will get the same versions of the dependencies.
paragkamble@Parags-MacBook-Pro mypackage % poetry install
Installing dependencies from lock file
No dependencies to install or update
- Installing mypackage (0.1.0)These are all 9 steps has describe the project life cycle directed by the Poetry and there are lot of other sub commands are available to make easier project management.
Wrap-up
It’s not much about it :-) , there are lots of other things about Poetry that I have not mentioned here. There are other similar tools are also available like pipenv, pyenv etc. and I also feel there is still space to improve Poetry like in dependency resolution performance and including things like C/C++ dependencies or Cython integration code.
Hope you like this blog and feel free to add your suggestion and comments below.
email: kamble.parag@gmail.com
