Линтинг — Python: Настройка окружения
У кода есть множество разных характеристик, по которым можно судить, насколько хорошо он написан. Среди них есть одна базовая, с которой начинают все разработчики — это стиль написания.
Сравните два варианта оформления кода:
## Без форматирования def find_sum(a,b): c = a+b; return c ## С форматированием def find_sum(a, b): sum = a + b return sum Второй вариант читается значительно проще. Чем больше будет кода, тем больше будет различий. Хороший стиль кодирования — базовое требование к коду в коммерческой разработке, потому что он упрощает командную разработку. В одном проекте может работать несколько десятков программистов. Важно, чтобы им было легко читать код друг друга, не спотыкаясь о неправильное форматирование.
Для отслеживания подобных ситуаций существуют линтеры — особый класс программ. Линтеры содержат большое количество правил, по которым они могут выдать рекомендации по коду. Другими словами, они подсказывают, как стоит писать код, а как — не стоит. Более того, часто линтеры расширяются плагинами под конкретные фреймворки, что позволяет отслеживать специфичные ошибки и давать рекомендации по кодированию в этих фреймворках.
В Python особой популярностью пользуется линтер flake8 . Количество правил, по которым он проверяет код, исчисляется десятками . Посмотрите на этот небольшой участок кода:
from math import sqrt def sum(a, b): c = 5 return a + b С точки зрения форматирования здесь все хорошо, а что скажет линтер flake8?
Он выдаст два предупреждения:
- F401 ‘math.sqrt’ imported but unused . Модуль импортируется, но не используется — либо он не нужен, либо в коде есть ошибка
- F841 local variable ‘c’ is assigned to but never used . Переменная не используется — либо она не нужна, либо в коде ошибка
Ссылки выше ведут на страницы конкретных правил. Там подробно объясняется, почему так код писать не нужно. Изучать правила flake8 очень полезно, они прививают хорошие практики написания кода.
Установка и настройка flake8
Линтер flake8 устанавливается как dev зависимость прямо в проект:
--group=dev flake8 После установки его можно настроить под набор ваших правил и ограничений. Для этого нужно создать файл конфигурации setup.cfg, который нужно добавить в репозиторий. Ниже пример такого файла, который используется для настройки линтера в наших практиках:
Выше настраивается максимальная сложность функций, максимальная длина строчки кода и другие параметры. Так же мы можем указать, какие файлы не нужно проверять линтером или какие правила можно игнорировать. Список всех опций вы можете изучить в документации .
И последний шаг — запуск flake8:
# Обратите внимание на точку # Это указание текущей директории и всех поддиректорий poetry run flake8 . Открыть доступ
Курсы программирования для новичков и опытных разработчиков. Начните обучение бесплатно
- 130 курсов, 2000+ часов теории
- 1000 практических заданий в браузере
- 360 000 студентов
Наши выпускники работают в компаниях:
flake8-isort 2.0.2
Use isort to check if the imports on your python files are sorted the way you expect.
Add an .isort.cfg to define how you want your imports sorted and run flake8 as you usually do.
See isort documentation for .isort.cfg available options.
Install
Install with pip:
$ pip install flake8-isort
Options
Since version 1.0 a check for an .isort.cfg file is done, since version 1.1 setup.cfg is also checked for an [isort] section.
This potentially avoids to lint a project that has no formal definition of how import should be sorted.
With either —no-isort-config command line switch, or no-isort-config flake8 configuration option it can be disabled.
Requirements
- Python 2.7, 3.5
- flake8
- isort
License
Changelog
2.0.2 (2016-11-22)
- Add flake8 classifier. [sigmavirus24]
- Require flake8 3.2.1. flake8 series 3.1.x and 3.2.0 where not reporting flake8-isort errors. [gforcada]
- Test on pypy and pypy3. [gforcada]
- Fix tests and formatting. [gforcada]
2.0.1 (2016-09-22)
- Fix standard input processing. [carljm]
2.0 (2016-09-14)
- Refactor code to handle flake8 version 3. [danpalmer]
- Require flake8 version 3.0. [gforcada]
1.3 (2016-06-20)
- Make error messages clearer. [do3cc]
- Use either pep8 or pycodestyle (new name for pep8). [Maxim Novikov]
- Fix coveralls. [gforcada]
1.2 (2016-03-05)
- Allow stdin processing, this way text editor can pass input to flake8. [mjacksonw]
1.1.1 (2016-02-16)
- Silence isort messages. [gforcada]
- Improve wording. [gforcada]
1.1 (2016-02-16)
- Check for isort configuration on setup.cfg as well. [plumdog]
1.0 (2015-12-16)
- Check for an isort configuration file. [gforcada]
0.2 (2015-09-14)
- Fix entry point. [gforcada]
0.1.post0 (2015-09-13)
- Release wheels as well. [gforcada]
0.1 (2015-09-13)
- Initial release [gforcada]
- Add all boilerplate files. [gforcada]
- Create the flake8 plugin per se. [gforcada]
flake8-isort 6.1.1
Use isort to check if the imports on your python files are sorted the way you expect.
Add an .isort.cfg to define how you want your imports sorted and run flake8 as you usually do.
See isort documentation for .isort.cfg available options.
Install
Install with pip:
$ pip install flake8-isort
Install with conda:
$ conda install -c conda-forge flake8-isort
Configuration
If using the select option from flake8 be sure to enable the I category as well, see below for the specific error codes reported by flake8-isort .
See flake8 —help for available flake8-isort options.
Error codes
Requirements
- Python 3.8, 3.9, 3.10, 3.11 and pypy3
- flake8
- isort
Relation to flake8-import-order
As an alternative to this flake8 plugin, there’s flake8-import-order that could be worth checking out. In contrast to this plugin that defers all logic to isort, the flake8-import-order comes bundled with it’s own logic.
flake8-import-order comes with a few predefined set of styles meanwhile this plugin can be customized a bit more. But the biggest difference could lie in that flake8-isort actually has the corresponding sorting engine isort that can sort the import orders of your existing python files. Meanwhile flake8-import-order has no such corresponding tool, hence big existing projects who want to adopt either would get a more automized experience choosing flake8-isort.
License
Подробности проекта
Ссылки проекта
Статистика
Метаданные
Лицензия: GNU General Public License v2 (GPLv2) (GNU GENERAL PUBLIC LICENSE Version 2, June 1991 Copyright (C) 1989, 1991 Free Software Foundation, . )
Метки flake8, imports, isort, pep8, python
Требует: Python >=3.8
Сопровождающие
Классификаторы
- Development Status
- 5 — Production/Stable
- Console
- Flake8
- Plone
- Developers
- OSI Approved :: GNU General Public License v2 (GPLv2)
- OS Independent
- Python
- Python :: 3
- Python :: 3 :: Only
- Python :: 3.8
- Python :: 3.9
- Python :: 3.10
- Python :: 3.11
- Python :: 3.12
- Python :: Implementation :: CPython
- Python :: Implementation :: PyPy
- Software Development
- Software Development :: Quality Assurance
How to use black, flake8, isort, and pre-commit framework to format Python codes
With black you can format Python code from 2.7 all the way to 3.8 (as of version 20.8b1), which makes for a great replacement for YAPF which can only format code depending on the Python version being used to run it.
My preference is using PEP 8 as my style guide, and so, 79-characters per line of code is what I use. So it’s as simple as running the following code at the root of my project and all non-compliant files will be reformatted:
$ black --line-length 79 --target-version py27 .
Let’s explain each option.
- -l or —line-length : How many characters per line to allow. [default: 88]
- -t or —target-version : Python versions that should be supported by Black’s output. [default: per-file auto-detection]
Fairly simple. Allow 79 characters per line, and use py27 as the targetted version.
isort: A Python library to sort imports.
And just as their slogan states: “isort your imports, so you don’t have to.”
$ isort --multi-line 3 --profile black --python-version 27 .
The options used are mainly to be compatible with black (see here):
- —multi-line : Multiline output (0-grid, 1-vertical, 2-hanging, 3-vert-hanging, 4-vert-grid, 5-vert-grid-grouped, 6-vert-grid-grouped-no-comma, 7-noqa, 8-vertical-hanging-indent-bracket, 9-vertical-prefix-from-module-import, 10-hanging-indent-with-parentheses).
- 3-vert-hanging
- black
- 27 for Python 2.7
But there’s still something missing. black does not care about comments or docstrings, and isort cares even less, for obvious reasons; enter flake8 .
flake8: A python tool that glues together pep8, pyflakes, mccabe, and third-party plugins to check the style and quality of some python code
Anthony Sottile (@asottile) has mentioned that he plans to drop support for Python 2.7 in future releases, maybe in version 3.9 or 4.0.
Fortunately, I can still use it for Python 2 by running the following command:
$ flake8 --max-doc-length=72 --ignore=E211,E999,F401,F821,W503
PEP 8 recommends limiting docstrings or comments to 72 characters, which is exactly what I’m using for flake8.
So let’s explain each option used.
- —max-doc-length : Maximum allowed doc line length for the entirety of this run. (Default: None)
- —ignore : Comma-separated list of errors and warnings to ignore (or skip). For example, —ignore=E4,E51,W234 . (Default: [‘E226’, ‘E123’, ‘W504’, ‘E121’, ‘W503’, ‘E126’, ‘E704’, ‘E24’])
In my case, I am using 72 as the maximum allowed characters for my docstrings, in accordance with PEP 8, and ignoring the following errors:
- E211 : whitespace before ‘(‘
- Since in Python 2 print is not a function, black adds a space between the print statement from Python 2, and the opening parenthesis
- In my case, this occurs again with the print statement where I am printing just one argument like this print arg
- I do import some modules in order to get “Intellisense” when I peek into the details in PyCharm
- In one of my libraries I am checking if an argument is a string, and in order to cover my bases with plain strings (str) and Unicode, I found that using basestring would work for all characters, including non-Latin characters
- It doesn’t like when binary operators are broken into multi-line statements
pre-commit: A framework for managing and maintaining multi-language pre-commit hooks.
Finally, let’s put it all together with pre-commit.
So in order to use flake8 you’ll have to create a .flake8 file. Mine looks like this:
[flake8] ignore = E211, E999, F401, F821, W503 max-doc-length = 72
A pyproject.toml file that in my case looks like this:
[tool.black] line-length = 79 target-version = ['py27'] [tool.isort] profile = "black" multi_line_output = 3 py_version = 27
And finally my .pre-commit-config.yaml file:
repos: - repo: https://github.com/psf/black rev: 20.8b1 hooks: - id: black - repo: https://github.com/PyCQA/isort rev: 5.7.0 hooks: - id: isort - repo: https://github.com/PyCQA/flake8 rev: 3.8.4 hooks: - id: flake8
After you’ve configured all of this for the first time, first run the install command for pre-commit and to run tests I use run with the —all-files option, just like this:
$ pre-commit install pre-commit installed at .git/hooks/pre-commit $ pre-commit run --all-files black. Passed isort. Passed flake8. Passed
So every time you try to commit something to your Git repo, all tests should be marked as Passed , otherwise, the commit will fail.
At the moment of writing this post both black and isort do support the use of pyproject.toml , something that flake8 still hasn’t been implemented unlike flake9 or FlakeHell, which I have not integrated into my workflow; I’m still using flake8 because I’ve installed it via Homebrew.
While you have the option to “ pip -install” all of these tools, currently, I decided to use Homebrew because I don’t usually check if my packages are outdated, something that Homebrew contributors actually do with each new release. See: black , flake8 , and isort , which will install python@3.9 as they all depend on it.
But if you do use pip , I recommend adding an alias for updating all of your outdated packages that should run the following command:
$ python -m pip list --outdated --format=freeze | grep -v '^\-e' | cut -d = -f 1 | xargs -n1 python -m pip install --upgrade
Automate Python workflow using pre-commits: black and flake8
Before I commit my staged Python files, black formats my code and flake8 checks my compliance to PEP8. If everything passes, the commit is made. If not, then I the perform necessary edits and commit again. Less time is spent on code formatting so I can focus more on code logic.
Code reviews are fun! They enable me to learn from others’ codes while providing an opportunity to teach what I know. However, there are still some things I wish to improve when facilitating reviews in my open-source projects:
- Less time commenting on code format, and more time discussing code logic
- Less hassle spotting format errors (“can you really see that trailing whitespace on Line 76?”)
- Stop sounding nitpicky (“Please put two blank lines between function definitions”)
If I could automate the processes above and remove the human-in-the-loop, we can focus more on code logic and implementation. Good thing, I learned about Git hooks, specifically pre-commit hooks. It enables you to automatically run a short script before commit ting. This script can be a checking tool or a formatter. If the script passes, then the commit is made, else, the commit is denied.
In this section, I’ll describe how I created a pre-commit pipeline in PySwarms using the black code formatter, flake8 checker, and the pre-commit Python framework. The entire pipeline looks like this:
Figure: Pre-commit pipeline with black and flake8 for checking my .py files
I’ll first discuss the pre-commit framework, then add components one-by-one: first is black , and then flake8 . I will show the dotfiles present in my project, so feel free to adapt them into your own!
The pre-commit Python framework
We can run shell files all we want to dictate how our pre-commit process goes, but this pre-commit framework written in Python got us covered. It even comes with a set of pre-commit hooks out of the box (batteries included!). To adopt pre-commit it into our system, we simply perform the following actions:
- Install pre-commit: pip install pre-commit
- Add pre-commit to requirements.txt (or requirements-dev.txt )
- Define .pre-commit-config.yaml with the hooks you want to include.
- Execute pre-commit install to install git hooks in your .git/ directory.
The YAML file configures the sources which the hooks will be taken from. In our case, flake8’s already been included in this framework so we just need to specify its id. On the other hand, we need to define where to source black using a few lines of code. Below is a sample .pre-commit-config.yaml file that I use in my project:
repos: - repo: https://github.com/ambv/black rev: stable hooks: - id: black language_version: python3.6 - repo: https://github.com/pre-commit/pre-commit-hooks rev: v1.2.3 hooks: - id: flake8
Update (03-04-2020) You can also add flake8 from its own repo like so:
repos: - repo: https://github.com/ambv/black rev: stable hooks: - id: black language_version: python3.6 - repo: https://gitlab.com/pycqa/flake8 rev: 3.7.9 hooks: - id: flake8
In the next section, I will discuss my code formatter ( black ) and checker ( flake8 ). As usual, I will provide my config files for each component.
Code Formatter: black
The black code formatter in Python is an opinionated tool that formats your code in the best way possible. You can check its design decisions in the repository itself. Some notable formatting decisions, in my opinion:
- Unlike in PEP8, code length is 88 characters, not 79.
- Use of double-quotes than single-quotes in strings.
- If there are many function args, each arg will be wrapped per line.
I’d rather maintain the recommended 79-character length. Good thing, they have an option to do so. I just need to configure my pyproject.toml to line-length=79 and everything is all set. Here’s my .toml file for configuring black :
[tool.black] line-length = 79 include = '\.pyi?$' exclude = ''' /( \.git | \.hg | \.mypy_cache | \.tox | \.venv | _build | buck-out | build | dist )/ '''
If you are not a fan of black, there’s always autopep8 — a formatter more faithful to PEP8. Good thing, the pre-commit framework already has a hook on this tool, so there’s no need to source from another repository.
Flake8 checker
Flake8 is a powerful tool that checks our code’s compliance with PEP8. In order for black to work nicely with flake8 (or prevent it from spewing out various errors and warnings), we need to list down some error codes to ignore. You can check my .flake8 configuration below:
[flake8] ignore = E203, E266, E501, W503, F403, F401 max-line-length = 79 max-complexity = 18 select = B,C,E,F,W,T4,B9
Results
So what we have is a pipeline that safeguards my project against wrongly-formatted code. On CONTRIBUTING page, I explicitly mentioned using pre-commits (or run flake8 and black on their code manually) before submitting a Pull Request.
Figure: Pre-commit pipeline with black and flake8 for checking my .py files
Now that we have a pre-commit framework set up with black and flake8, let’s see it in action! Here we’ll see how black formats a Python file automagically:
Figure: Short demo on pre-commit hooks
pre-commit
pre-commit is a framework for managing pre-commit hooks in Git. Oh, but what is Git Hook?
Git Hook is a script that is run automatically every time a specific event occurs in the Git repository.
In this case, the event here is the commit code. We will use the pre-commit hook to check the changes in the code of each style convention automatically before commit and integrating it into the system. If something goes wrong, the commit will fail and we’ll get the associated error messages to fix. Commit is only successful when no errors occur.
So far we have just introduced and run the above tools manually. Now it’s time to combine them to run automatically!
WORKFLOW WITH PRE-COMMIT HOOKS
overview
Workflow with pre-commit (Image taken from the end of post link)
Before implementing Git commit, I will use isort and black to format the code automatically, then use flake8 to check again with standard PEP8 (all configured by pre-commit ). The commit will be successful if there is no error. If an error occurs, we will go back and fix it where necessary and commit again. This workflow helps to reduce the time to reformat the code manually, thereby focusing more on the logic. Team working together is also easier and more efficient.
Set-up step by step
(These are the configuration files that I am using. You can customize them to suit your own style or needs!)
Create virtualvenv if needed and install pre-commit:
pip3 install pre-commit
Don’t forget to add it to requirements.txt or Pipfile after installation.
Configure pre-commit by creating a .pre-commit-config.yaml in the root directory with the following content (you can adjust to the latest version at the time of reading this article):
repos: - repo: https://github.com/pre-commit/pre-commit-hooks rev: v2.3.0 hooks: - id: check-yaml - id: end-of-file-fixer - id: trailing-whitespace - repo: https://github.com/asottile/seed-isort-config rev: v1.9.3 hooks: - id: seed-isort-config - repo: https://github.com/pre-commit/mirrors-isort rev: v4.3.21 hooks: - id: isort - repo: https://github.com/psf/black rev: 19.10b0 hooks: - id: black language_version: python3.6 - repo: https://gitlab.com/pycqa/flake8 rev: 3.8.3 hooks: - id: flake8
Configure isort by creating a .isort.cfg file in the root directory with the following content:
[settings] line_length = 79 multi_line_output = 3 include_trailing_comma = True
If you have noticed in the pre-commit configuration file, along with isort , we use add seed-isort-config to automatically add packages to known_third_party in the isort configuration (instead of doing it manually). Configure black by creating a pyproject.toml file in the root directory with the following content:
[tool.black] line-length = 79 include = '.pyi?$' exclude = ''' /( .git | .hg | .mypy_cache | .tox | .venv | _build | buck-out | build | dist )/ '''
Configure flake8 by creating .flake8 file with the following content:
[flake8] ignore = E203, E266, E501, W503, F403, F401 max-line-length = 79 max-complexity = 18 select = B,C,E,F,W,T4,B9
After configuration is complete, run the following command to complete the installation:
pre-commit install
Finally, before executing Git commit, run the following command:
pre-commit run --all-files
As you can see, installing the above workflow is very easy for the team because all the configuration files are already in the project, members just need to run the pre-commit install command.
Other useful pip packages
- pip-autoremove for removing packages and all of their dependencies.
- pylint , a static code analysis tool that looks for programming errors helps enforce a coding standard, sniffs for code smells and offers simple refactoring suggestions.