Starting a Python project¶
Read time: 10.7 minutes (1069 words)
Most students begin a project by writing code. That is definitely not how we start a significant project. We have some setup work to do first:
But first, we have to get this project started.
Step 1: Set up the project¶
Like all good projects, we need to create a home for the project, and get it under Git control. We also need to get it on GitHub so we have a safe “master” copy in case our local development machine explodes! (Believe me, I have had major failures using some unnamed vendor machines!)
Here is my start for this project:
Project Directory¶
First, we create a top-level directory for the project. I keep all active development projects under _projects in my home directory:
Todo
Review Sphinx todo directive for hints on extracting pylit blocks
This is a test block - ignore for now!
make dirs
More commands in script block
# test.py
def main():
print("hello")
if __name__ == '__main__':
main()
$ cd _projects
$ mkdir pylitproject
For the rest of the setup, we need to be working in this new directory;
$ cd pylitproject
We will be doing a lot of documenting in this project. I keep all reStructuredText files under rst. Using Sphinx (for now), we will process the .rst files into html files, placing them in a second file named docs. By doing this we get a free website where we can publish our documentation on Github. We need one magic file in the docs folder to get the website working properly;
$ mkdir -p rst/_static
$ mkdir -p docs
$ touch docs/.nojekyll
This project will be hosted on PyPi, so users can install it using pip. The project code will be placed in a Python package with the same name as the project:
$ mkdir -p pylitproject
$ touch pylitproject/__init__.py
Finally, we will be testing the code as we proceed with development. All test files will be placed in a tests package directory:
$ mkdir -p tests
$ touch tests/__init__.py
Step 2: Manage with Git¶
Now, we can initialize the project so Git can manage it, Make sure that this command is run in the top-level project directory:
$ git init
We need a .gitignore file in the top-level project directory to keep cruft out of our remote repository:
# Mac
.DS_Store
# Vim
*~
.*.sw*
#Python
__pycache__
_venv/
*.py[cod]
# PyPi
dist/
build/
pylitproject.egg-info/
# Sphinx
rst/_build
# Tox
.tox/
If you are going to use Github as a home for your project, you need a top-level README.rst file, so visitors to the project on GitHub will see some information about the project. Here is a basic start on this file:
pylitproject: (v0.0.3-dev)
##########################
:Author: Roie R. Black
:Date: May 7, 2020
:Email: roie.black@gmail.com
:Documentation: https://rblack42.github.io/pylitproject
|travis-build| |license|
This is the *PyLiT Project*, a project designed to build an efficient tool that
incorporates Dr. Donald Knuth's basic ideas for Literate Programming into a
documentation build system suitable for educational use by instructors.
Instead of generating documenting for just one version of a program, this tool
allows the developer to build a document that follows along as the project
evolves. In order to make that happen, extensions to Knuth's Literate
Programming markup are added so code can be modified to produce a new version.
Code extracted from the documentation is styled according to the developer's
tastes, and not obfuscated as Knuth originally wanted. Hopefully, this will
encourage use by developers who like to read their code directly.
The documentation files are managed in a block-structured store similar to that
used by git. This speeds up processing by eliminating builds on files already
processed and unchanged.
.. note::
At the present time, the documentation is managed by Git, not the extracted
code. That means that developers interested in a project managed by PyLiT
will need to "read the documentation", not the bare code. (Is that really a
bad thing?) I welcome ideas on how to manage extracted code in the
traditional form so it can be stored and viewed on Github.
.. |travis-build| image:: https://travis-ci.org/rblack42/pylitproject.svg?branch=master
:alt: Build badge from Travis-CI
.. |license| image:: https://img.shields.io/badge/License-BSD%203--Clause-blue.svg
:alt: BSD 3-Clause License
Step 3: Create a Github Repository¶
Now, we can get the project on Github. Assuming you have an account there, sign in and create a new empty repository with the project name. Once you get that done, return to your development machine and run a few final commands:
$ git remote add origin https://github.com/rblack42/rst_literate.git
Note
This example includes my Github username, which is rblack42. Obviously, you should use your own username here.
Finally, we “push” everything (except things we told Git to ignore in the .gitignore file) to Github.
$ git add .
$ git commit -m "initial project commit"
$ git push -u origin master
There is a pattern in this set up commands that we will follow over and over as we work on this project.
The first line tells Git to scan the entire project directory and subdirectories, looking for any changes made since the last time we “committed” changes to the project. Since we are just starting, this “add” command will add all files and directories to the Git management system stored in a hidden .git folder in your project directory.
You “commit” those changes with the second command line above. This step wraps us all of the changes and marks this version of your project code with a unique 40 character tag. Git can return your project directory to this version later if needed. Most of the time we do not worry about this tag, we can add human readable tags later.
Finally, the last command line above actually does copy your new version to Github.
Warning
The -u option is only added to this command the first time you push your project to Github. Leave it off after that.
Step 4: Add Documentation¶
In order to use Sphinx to document this project, we need to do a bit more work.
First, we need to add a configuration file to the rst directory:
# Configuration file for the Sphinx documentation builder.
#
# This file only contains a selection of the most common options. For a full
# list see the documentation:
# https://www.sphinx-doc.org/en/master/usage/configuration.html
# -- Path setup --------------------------------------------------------------
# If extensions (or modules to document with autodoc) are in another directory,
# add these directories to sys.path here. If the directory is relative to the
# documentation root, use os.path.abspath to make it absolute, like shown here.
#
import os
import sys
sys.path.insert(0, os.path.abspath('.'))
# Include todo list
todo_include_todos = True
# -- Project information -----------------------------------------------------
project = 'rst-literate'
copyright = '2020, Roie R. Black'
author = 'Roie R. Black'
# The full version, including alpha/beta/rc tags
release = '0.0.1'
# -- General configuration ---------------------------------------------------
# Add any Sphinx extension module names here, as strings. They can be
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
# ones.
extensions = [
'sphinx.ext.todo',
'sphinx_numfig',
'sphinx_ext.wordcount',
'sphinx_ext.pylit',
]
# Add any paths that contain templates here, relative to this directory.
templates_path = ['_templates']
# List of patterns, relative to source directory, that match files and
# directories to ignore when looking for source files.
# This pattern also affects html_static_path and html_extra_path.
exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store']
# -- Options for HTML output -------------------------------------------------
# The theme to use for HTML and HTML Help pages. See the documentation for
# a list of builtin themes.
#
html_theme = 'alabaster'
# Add any paths that contain custom static files (such as style sheets) here,
# relative to this directory. They are copied after the builtin static files,
# so a file named "default.css" will overwrite the builtin "default.css".
html_static_path = ['_static']
Only a few lines in this file change from project to project. Details will be explained later.
We also need a top-level reStructuredText page, named index.rst which will be the entry point to our documentation.
pylitproject: Documentation
===========================
.. toctree::
:maxdepth: 2
:caption: Contents:
:glob:
introduction/*
development-tasks
Indices and tables
==================
* :ref:`genindex`
* :ref:`modindex`
* :ref:`search`
Notice that I am using the Sphinx glob feature to include documentation files under an introduction subdirectory. (The file you are reading now is in that directory).
Run Sphinx at lease once (using the makefile shown below), so you have web pages in the docs directory. Then, sign on to your Github account and go yo your project repository page. Under the settings menu item, scroll down until you see Github Pages. Change the source selection that says None to Master branch/docs folder. Finally, push your new web pages to Github using the three step mantra we showed earlier. You should see those pages in any web browser by navigating to a URL like this:
Again, use your own Github account name here.
Step 5: Project Dependencies¶
Hardly any significant Python project is totally self-contained. Most projects “depend” on tools written by other developers. Those tools are loaded onto your system using pip. It is traditional to list all of these “dependencies” in a requirements.txt file in your project”
wheel
twine
tox
nose
coverage
flake8
python-coveralls
sphinx
sphinx-rtd-theme
sphinx_numfig
Step 6: Project Makefile¶
We will be doing a lot of different tasks in our development work. The individual commands needed for each task can be hard to remember. Therefore, we will write down all of those commands in a project Makefile, and use Make to run them for us, using a simple, easy to remember command. (See the Appendix for more on this file and the Make command):
# pylitproject Makefile
PROJECT := $(notdir $(PWD))
MK := mk
-include $(MK)/os-detect.mk
Step 7: Testing!¶
As we write code, we will test things. We will be using Test Driven Development in this project, which means we will write some test code for any new feature we add, then write the code needed to make sure that test “passes”. We will use a Make command to run tests locally, but there is a neat way we can run tests on a variety of machines, all for free. We will use Travis-CI which will run our tests for us on a remote system set up as we wish every time we push changes on Github.
To set this up, we need a control file in our project:
sudo: false
language: python
python:
- "3.7"
addons:
apt:
Packages:
- git
# command to install dependencies
install:
- 'pip install coveralls'
- 'pip install -r requirements.txt'
- 'python setup.py install'
script:
- tox -v
We will go over this file later.
Once you have this file in place, sign in to Travis-CI (you can use your Github credentials to do this), and look for your project under the account settings menu item. You need to toggle testing on for this project in the list of repositories shown. Once that is done, simply “push” changes to Github and Travis-CI will see those changes, ‘clone’ a fresh copy of your project into a new virtual machine, and run the tests you specified. If everything works, you will see the results in the badge we included on the REAdme.rst file. Check your project page on Github to see this.
That is enough setup work for now, let’s start working on code!