set your code free
releasing and maintaining an open-source Python library
- Hi! Welcome to the final talks session of PyCon
- I know it's been a long conference
- I'll at least keep this short
- If I'm lucky also keep you awake
- and send you off on a high note, inspired to release your own software!
You have code!
- You want me to use it (pip install).
- You want me to contribute to it.
- The premise of this talk is that you have some code.
- Writing that code is out of scope.
- pip install
- not web app deployment or GUI installer
- "accept my contributions without breaking your software or losing your
- So you look up the docs on how to do this...
- ...and you find lots of different projects
- all with their own documentation to read
- When you're doing something for the first time, choices kill.
- This talk presents a set of rails for setting up your first open-source
- It's not the only way, but _a_ way that will work.
- Long on opinions, short on choices.
- "From zero to awesome in 20 minutes."
- The awesome:
- public HTML docs with built-in navigation that update when you push
- tests that run on every push (and on pull requests)
- pip install ready
- welcoming to contributors.
- If you've done this before, hopefully you'll still pick up a new trick or
All the things
- Project structure.
- Choosing a license.
- Code hosting.
- Testing & CI.
The roadmap for this talk.
2 └── PyFly/
3 ├── docs/
4 ├── pyfly/
5 │ └── __init__.py
6 ├── tests/
7 ├── LICENSE.txt
8 ├── MANIFEST.in
9 ├── README.rst
10 └── setup.py
- This is the bare bones;
- We'll flesh this out and add to it as we go.
- First decision: releasing software as open source means choosing a
- I am not a lawyer, this is not legal advice.
- Your license is the conditions under which I can use your code.
- If you don't have a license, the default is "all rights reserved."
- A project without a license is not open-source, even if its on GitHub!
- BSD or MIT are unrestrictive licenses;
- All you ask from your users is that they credit you: keep your name and
the license attached to your work.
- GPL is more restrictive; requires that any work derived from yours must
also be released as GPL. If you're worried about freeloaders, you can go
this route, at the cost of having fewer users.
- Apache and MPL are reasonable choices if you know why you're choosing
1 Copyright (c) 2009-2014, Carl Meyer and contributors
2 All rights reserved.
4 Redistribution and use in source and binary forms, with or without
5 modification, are permitted provided that the following conditions are
8 * Redistributions of source code must retain the above copyright
9 notice, this list of conditions and the following disclaimer.
10 * Redistributions in binary form must reproduce the above
11 copyright notice, this list of conditions and the following
12 disclaimer in the documentation and/or other materials provided
13 with the distribution.
15 THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
16 "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
17 LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
18 A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
19 OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
20 SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
21 LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
22 DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
23 THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
24 (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
25 OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
03. Code hosting
- That's where the people are.
- We have a lot of areas to cover,
- so fortunately some are short and sweet.
- If it's not documented, it doesn't exist.
- Build it with Sphinx.
- Host it at ReadTheDocs.
- I like reading code. I will read code to find a bug, to fix a bug, or to
better understand how your library does what it does.
- But if I have to read your code to figure out how to use your thing -- I'm
gonna choose a different library, or just write it myself instead.
- Auto-generated API docs don't count, unless your API is very simple. A
long list of functions and classes with their docstrings is something I
can get from reading your code.
- How to write your docs: fortunately another easy choice.
- And so is where to host it.
- The combination of Sphinx and Read The Docs makes it so ridiculously easy
to put beautiful, usable docs online, it's a shame not to take advantage
of that by writing some!
1 $ pip install sphinx
4 $ cd docs/
6 $ sphinx-quickstart
9 Enter the root path for documentation.
10 > Root path for the documentation [.]:
- You can write some sphinx docs in about as much time as I'll spend on the
next two slides.
- pip install sphinx
- switch to the docs directory
- run 'sphinx-quickstart'
- answer some questions; the defaults will do
1 Welcome to PyFly!
5 Install **PyFly** with
6 ``pip install PyFly``.
10 Find a route::
12 import pyfly
13 route = pyfly.Route('KRAP', 'CYUL')
- If you haven't written restructuredtext, the basics are very easy.
- underlined headers
- different levels of headers
- inline formatting: strong with double asterisk
- inline code literals
- code blocks, automatically syntax highlighted in pretty much any language
- If you take five minutes and write exactly this much documentation -- a
simple usage example -- you've already made your package much more
attractive than one without docs.
- Run 'make html' to generate an HTML version of your docs
- Automatically updates the docs every time you push to the repo.
- Can build multiple different versions (by branch or tag) and provides a
version switcher to choose between them.
- Good-looking, mobile-responsive theme.
- If it's not tested, it's broken.
- If code is changing over time, and you don't have automated tests for it
(or a lot of time on your hands for testing manually), over time the
likelihood of that code being broken approaches 1.
- Tests are good for any code, but they are critical for open-source code
that is getting contributions. Finding time to handle pull requests is
hard enough, you really don't want to have to run through a bunch of
manual tests for every pull request to verify that it didn't break things.
A reasonable support matrix for a popular Django add-on library.
Could be worse: with another dependency or two it would have 3 or 4
dimensions, not just 2.
25 boxes in that matrix. Are you gonna create 25 virtualenvs and run the
tests 25 times for every pull request to your project? If not, your claim to
support all those versions is purely theoretical, and almost certainly not
Thankfully, there's a tool to help with this: ...
tox saves the day
- Creates a bunch of virtualenvs.
- Runs your tests in each of them.
2 envlist = py27,py34,pypy
5 deps = pytest
6 commands = py.test
- A very simple tox setup: just three Python versions, no dependencies.
- Originally I had just 2.7 and 3.4, but I made the mistake of letting Alex
Gaynor see my slides, and he tied me up and wouldn't let me go until I
- There's a lot more you can do here, such as adding various versions of
dependencies in various envs (to handle the matrix we just saw).
- You can look at the documentation, or I'd be happy to show you some
1 $ tox
2 GLOB sdist-make: /.../PyFly/setup.py
3 py27 create: /.../PyFly/.tox/py27
4 py27 installdeps: pytest
5 py27 inst: /.../PyFly/.tox/dist/PyFly-0.1.zip
6 py27 runtests: commands | py.test
7 ================== test session starts ====================
8 platform linux -- Python 2.7.6 -- py-1.4.20 -- pytest-2.5.2
9 collected 3 items
11 test_routes.py ...
13 ================== 3 passed in 0.02 seconds ===============
15 ... <same for py34 and pypy>...
17 __________________ summary ________________________________
18 py27: commands succeeded
19 py34: commands succeeded
20 pypy: commands succeeded
21 congratulations :)
Running your tests
all the time
You get a pull request, you open a terminal, you add the source of the PR as
a remote, you pull their branch, you run tox... wouldn't it be nice if when
you first looked at the pull request, it already told you whether the tests
passed or not?
This used to be hard. Today it is easy.
Will do this for free for public GitHub projects.
(There's also drone.io and probably others; Travis is the one I've used.)
- To set it up, just go to travis-ci.org and sign in with your GitHub
- It'll show you a list of all your public GitHub projects, and you just
pick which ones you want to enable.
- It will automatically set up a GitHub webhook for the projects you enable.
1 language: python
4 - 2.7
5 - 3.4
6 - pypy
9 - pip install pytest
12 - py.test
- The second thing we need to do is add a .travis.yml file to the repo,
so Travis knows how to run our tests.
- This example does the same thing as our earlier tox.ini.
Oh yes, you may want people to be able to install your thing!
1 from setuptools import setup
3 with open('README.rst') as fh:
4 long_description = fh.read()
9 description='Flying with Python',
11 author='Carl Meyer',
17 'Development Status :: 3 - Alpha',
18 'License :: OSI Approved :: BSD License',
19 'Programming Language :: Python',
20 'Programming Language :: Python :: 2.7',
21 'Programming Language :: Python :: 3',
22 'Programming Language :: Python :: 3.3',
23 'Programming Language :: Python :: 3.4',
recursive-include docs *.rst
When Python creates a source distribution of our package, it will include
all of our Python packages, plus any additional files we list here.
- git tag v0.1.2
- git push --tags
- First thing to do is tag your release in git
- and make sure you push that tag to GitHub
- (git doesn't push tags by default)
- python setup.py sdist
- pip install dist/PyFly-0.1.2.tar.gz
- pip install twine
- twine upload dist/PyFly-0.1.2.tar.gz
- pip install PyFly
- "sdist" == "source distribution"
- This is the most common format for distributing Python code, and it's a
good choice for pure Python (no compiled extensions).
- There is a new wheel format getting a lot of buzz, and it is very
exciting especially for Python projects with compiled extensions, but I'm
not going to cover it here; for your first project if it's pure Python
code I recommend just starting with sdist.
- python setup.py sdist will create an sdist for your package in the
- Before we upload this sdist to the package index, we want to be sure it
works. We can use pip to install directly from that sdist and test to
make sure it works correctly.
- There is a python setup.py command to upload your sdist to the package
index, but it uploads it over a non-SSL connection, so we'll instead use
the twine tool to do it securely.
- Before you can do this step, you need to go to the package index in your
web browser to create an account and claim your package name.
- Then we run twine upload to create the new version, upload its
metadata and the sdist file.
- Using --sign will sign your upload with your GPG key. This is a good
idea, but if you don't have a GPG key you can leave that out.
- Python Packaging User Guide
- #pypa IRC channel on FreeNode
- the distutils-sig mailing list
- Making life better for people using and contributing to your software.
- Valuing your users and contributors' time.
- X for breaking changes.
- Y for backwards-compatible feature additions.
- Z for bug fixes.
- One way to value your users' time is to communicate clearly about changes
that will affect them.
- With semantic versioning, you can use your version numbers to communicate
this information to them.
Keep a changelog
- Semantic versioning tells your users about the magnitude and type of
changes in a release.
- A changelog tells them exactly what has changed.
* Fix ``get_query_set`` vs ``get_queryset``
in ``PassThroughManager`` for Django <1.6.
Thanks whop, Bojan Mihelac, Daniel Shapiro,
and Matthew Schinckel for the report;
Matthew for the fix. Merge of GH-121.
* Fix ``FieldTracker`` with deferred model
attributes. Thanks Michael van Tellingen.
Merge of GH-115.
- A sample of what a changelog can look like.
- People think "oh, I'll just autogenerate it from my git commit history!"
- NOT the same as a git commit log.
- Pulls out and highlights changes that are relevant to users.
Have a CONTRIBUTING document
- Save time for both your contributors and yourself by having a
- How to get set up for development.
- How to run the tests.
- What to include in a bug report.
- Coding standards, test coverage standards...
- I'm all excited to fix a bug I've found in your project.
- I download it, follow your contributing guidelines to get set up
- All excited to run the tests... and some of them fail.
- My motivation to contribute is now gone.
- Another way to value your contributors' time is to respond quickly.
- You may not have time to deal with the issue right away, but you can at
least post a quick note thanking them for the contribution and saying
you'll get to it later.
- When people pitch in, give them props!
- In the commit message, in the changelog, in an AUTHORS file...
- Motivates people to contribute.
- Give commit access to helpful contributors!
- Not as technically important with DVCS as it used to be, but a mark of
confidence, will motivate greater involvement (and saves you work).
- Someday you'll want to hand off maintenance...
- Anytime you get a bug or pull request, even if it's irritating or someone
hasn't done their homework, it represents someone investing time and
energy in your project. Thank them for their time and energy!
- For every one person you communicate with directly in a public way, there
are 10, 20, 50, 100 people watching that communication, now or later. How
you treat contributors will affect their motivation to contribute.