Create your own Telegram bot with Django on Heroku – Part 6 – Creating the Django app

This entry is part 6 of 11 in the series Create your own Telegram bot with Django on Heroku

 

 

Django_Pony

In the previous part of this series, I tried to give you a brief yet thorough introduction to hosting your projects with Heroku.
That part was special because it was a completely optional part of this series; if you prefer to host your applications on a different platform and skipped that article, I’d like to repeat that this is completely OK and that I had shown nothing you will need for anything different but interacting with Heroku. You will hopefully notice no blank spots in the following articles. There is no need to read that article if you do not plan to use Heroku for hosting your bot. But you should be familiar enough with your hosting solution of choice to adopt the Heroku – commands I show here to an adequate setup for your hosting solution.

Today we will finally start creating our bot with Django. What we did up until now was just some kind of preparation and establishing background. In this part, we will finally start with the real stuff. 💪

Foreword

This article reflects how I created my first Django project. I had some experience with Python before already, but I never created any real Django project apart from the standard tutorial – setups, when one is trying to follow the official Django intro, for example.

When you follow this article, you will end up with a working Telegram bot which I have not found any issue with yet. But neither is it based on any best-practice on how to create Django applications nor will there be any conceptual groundbreaking insights. Maybe you could not even do worse than to follow my instructions when you are unfamiliar with Django and want to learn how it is done; maybe, I do not know better myself. But on the other hand I think I have created something not too bad since it is working as expected, but I doubt that what I’m about to show with this article will be a perfect first attempt.
Because of that, I encourage you to try to get the most out of this article, even if it isn’t perfect and nobody with fundamental Django skills would consider it to be a great start into Django, most certainly. But do not take anything in this article blindly as the one and only truth.

If you are more familiar with Django or do have better ideas with any of the shown codes or concepts, please let me know in the comments. I would really appreciate some feedback; positive or negative. I will adapt valuable advice by updating this article, accordingly.

Providing a proper environment

In this project, we will use the following:

  • Django
    • For our Telegram Bot, we will use Django 2.1.
  • Database
    • To not make the local setup too complicated, we will use a sqlite3 database for local development and testing.
    • For the production application in Heroku, we will use a PostgreSQL database, since this is a great SQL RDBMS and one of the three managed data services, which are available to all customers in Heroku.
      If you want to learn more about PostgreSQL in Heroku please read Herokus article on heroku-postgres.
  • Python
    • Since Heroku’s current default Python is 3.6.6 (as of 2018-08-24), I will also use 3.6; it’s not necessary to exactly match the patch level version.
    • If you want to use Python 3.7 for any reason (utilization of Data Classes for example), you can do so, but you will need an additional step, not explicitly mentioned later on anymore:
      To define Python 3.7 as the desired Python runtime in your project, add python-3.7.0 to a file called runtime.txt inside your project folder (echo ‘python-3.7.0’ > runtime.txt). Heroku will automatically pick that up and deploy your dynos with this runtime. More on Python runtimes on Heroku in python-runtimes article.

To match this environment, it is recommended to create a new virtualenv for it. I will show how to do this using pipenv in a minute.

Preparing your project

In the previous part of this series (which was optional), we created a new app on Heroku and initialized a simple Flask application in ~/tbot. We do not need any of that in the upcoming parts since it was meant to demonstrate Heroku’s workflow, exclusively. If you followed that article, you may leave everything in place like it is. If it isn’t used (no one is accessing its URL), it won’t consume any resources.

For this article, we will initiate a new project folder at ~/dtbot. If you prefer another location, feel free to do so, but please adopt that in the following occurrences of ~/dtbot  accordingly.

First, we need to have this empty folder created, initialize Git in it and create a fresh virtualenv using pipenv :

~ $ mkdir ~/dtbot
~ $ cd ~/dtbot

~/dtbot $ git init
Initialized empty Git repository in /home/testuser/dtbot/.git/
~/dtbot $ echo 'db.sqlite3' > .gitignore
~/dtbot $ echo '**/__pycache__/' >> .gitignore
~/dtbot $ echo '*.pyc' >> .gitignore

~/dtbot $ pipenv --python python3.6.6
Creating a virtualenv for this project…
Using /opt/python/python3.6.6/bin/python3.6.6 (3.6.6) to create virtualenv…
⠋Running virtualenv with interpreter /opt/python/python3.6.6/bin/python3.6.6
Using base prefix '/opt/python/python3.6.6'
New python executable in /home/testuser/.virtualenvs/dtbot-hT9CNosh/bin/python3.6.6
Also creating executable in /home/testuser/.virtualenvs/dtbot-hT9CNosh/bin/python
Installing setuptools, pkg_resources, pip, wheel...done.

Virtualenv location: /home/testuser/.virtualenvs/dtbot-hT9CNosh
Creating a Pipfile for this project…
~/dtbot $

Next, we will install some required Python modules:

~/dtbot $ pipenv install django telepot gunicorn
Installing django…
Collecting django
  Using cached https://files.pythonhosted.org/packages/51/1a/e0ac7886c7123a03814178d7517dc822af0fe51a72e1a6bff26153103322/Django-2.1-py3-none-any.whl

...

Updated Pipfile.lock (dcba73)!
Installing dependencies from Pipfile.lock (dcba73)…
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 13/13 — 00:00:03
To activate this project's virtualenv, run the following:
 $ pipenv shell
~/dtbot $

For those of you who are using Heroku to host your Django bot, install an additional module called django-heroku:

~/dtbot $ pipenv install django-heroku
Installing django-heroku…
...

django-heroku automatically pulls in some additional modules:

  • dj-database-url
  • psycopg2
  • whitenoise

With these modules installed, we can simplify and optimize the Django integration with Heroku; please read the project page for details. I will explain how to utilize these shortly.

⚠ Attention:
I’m using pipenv in this guide to populate my virtualenv with modules. This automatically creates a file called Pipfile.
Heroku’s buildpacks detect and build scripts are checking if a file called Pipfile or requirements.txt exists in the root of your project to detect the proper buildpack to be Python.
If you do not use pipenv to manage your environment, like plain virtualenv + pip or no virtualenv at all, you need to create a requirements.txt file manually whenever you change your module composition in your project like this:

(dtbot-hT9CNosh) ~/dtbot $ pip freeze > requirements.txt

This not only allows Heroku’s build process to identify the proper buildpack as Python, but it also tells that build process with which modules it needs to populate the hosting environment with to make your app work. Remember: Every change in your code results not only in the code to be exchanged and some WSGI server to be restarted, but a completely new environment will be created from zero. Heroku needs to know what modules to include in this build for your container; having the Pipfile or requirements.txt in place is the way you do this.

Create a new Heroku app

While you are inside your local project folder (with the freshly initialized Git repo in it), create a new app in your Heroku environment (for details on this process, please read the previous part of this series):

(dtbot-hT9CNosh) ~/dtbot $ heroku create --buildpack heroku/python
Setting buildpack to heroku/python... done
Creating app... done, ⬢ dry-tundra-61874
https://dry-tundra-61874.herokuapp.com/ | https://git.heroku.com/dry-tundra-61874.git
(dtbot-hT9CNosh) ~/dtbot $ git remote -v
heroku  https://git.heroku.com/dry-tundra-61874.git (fetch)
heroku  https://git.heroku.com/dry-tundra-61874.git (push)
(dtbot-hT9CNosh) ~/dtbot $

We will need this, soon. The –buildpack heroku/python part is optional, yet recommended, since it explicitly sets our buildpack (more on this later) to be Python. Otherwise Heroku would try to figure the proper one out by itself. This also works very well, but since we are Pythonistas, let’s keep it with 🧘 The Zen of Python 🐍:

Explicit is better than implicit.

Activate your virtualenv and initiate your project

We can activate our virtualenv now. I recommend to do it like this:

~/dtbot $ source $(pipenv --venv)/bin/activate
(dtbot-hT9CNosh) ~/dtbot $

This last command is a bit special in my personal workflow: I do not like the pipenv default way to spawn a sub-shell using pipenv shell. So I’m activating the virtualenv pipenv creates by sourcing it’s “activate”-script the classical virtualenv-way.

Now, the time has come to initiate your Django project:

(dtbot-hT9CNosh) ~/dtbot $ django-admin startproject dtbot .
(dtbot-hT9CNosh) ~/dtbot $ ls -l
total 28
-rw-rw-r-- 1 testuser testuser   241 Aug 24 15:58 Pipfile
-rw-rw-r-- 1 testuser testuser 13515 Aug 24 15:59 Pipfile.lock
drwxrwxr-x 2 testuser testuser  4096 Aug 24 16:15 dtbot
-rwxrwxr-x 1 testuser testuser   537 Aug 24 16:15 manage.py
(dtbot-hT9CNosh) ~/dtbot $ ls dtbot/
__init__.py  settings.py  urls.py  wsgi.py
(dtbot-hT9CNosh) ~/dtbot $

Let’s calibrate some settings in ~/dtbot/dtbot/settings.py to match Heroku’s recommended implementation first:

  • Add import django_heroku to the top of your settings.py file.
    Then, to the very bottom of it, add django_heroku.settings(locals()). By this, some sensitive credentials (like the database credentials) and secrets (like SECRET_KEY) and general settings (like DATABASE_URL) do not need to be stored in the settings.py file, but are configured using environment variables inside of the dyno or work out-of-the-box because Heroku provides these values in environment variables automatically.
    This adds some value to your project since you will need to configure less in files. Also, not even your co-workers will get these secrets exposed, your app will become re-usable without the need to make any change to its settings.py file, et cetera. Even if there are no co-workers, maybe there will be some in the future or you decide to make your bot’s code publically available on Github or similar; you can’t know today what will happen in the future. Because of that, adding sensitive information to a VCS like Git is not recommended generally.
    To learn about the details of this, please read Herokus django-app-configuration article.
  • Add import os to the top of settings.py. Then replace DEBUG = True by this:
    DEBUG = os.environ.get('DEBUG', None)
    if DEBUG is None:
        DEBUG = False
    else:
        if 'true' == DEBUG.lower():
            DEBUG = True
        else:
            DEBUG = False

    I will explain the logic behind this in a minute.

  • Add 127.0.0.1 to ALLOWED_HOSTS , so you can access your Django locally.

As a last initialization step, apply any migrations now; these will be applied to the default sqlite3 DB file db.sqlite3 in your project folder. Please execute this following command blindly, so far, even though I will not explain what it does or what a migration is right now, since that is too wide out of scope, right now. For now, you only need to know that this creates a file named sqlite3.db inside your Django project-root directory, which stores a complete database layout, which this command creates in there. I will pick this up and explain it more deeply in Part 8 of this series:

(dtbot-hT9CNosh) ~/dtbot $ python manage.py migrate
Operations to perform:
  Apply all migrations: admin, auth, contenttypes, sessions
Running migrations:
  Applying contenttypes.0001_initial... OK
  Applying auth.0001_initial... OK
  Applying admin.0001_initial... OK
  Applying admin.0002_logentry_remove_auto_add... OK
  Applying admin.0003_logentry_add_action_flag_choices... OK
  Applying contenttypes.0002_remove_content_type_name... OK
  Applying auth.0002_alter_permission_name_max_length... OK
  Applying auth.0003_alter_user_email_max_length... OK
  Applying auth.0004_alter_user_username_opts... OK
  Applying auth.0005_alter_user_last_login_null... OK
  Applying auth.0006_require_contenttypes_0002... OK
  Applying auth.0007_alter_validators_add_error_messages... OK
  Applying auth.0008_alter_user_username_max_length... OK
  Applying auth.0009_alter_user_last_name_max_length... OK
  Applying sessions.0001_initial... OK
(dtbot-hT9CNosh) ~/dtbot $ ls -l db.sqlite3
-rw-r--r-- 1 testuser testuser 40960 Aug 24 16:59 db.sqlite3
(dtbot-hT9CNosh) ~/dtbot $

That’s it so far – let’s test this and play around with it a bit next.

Start the build-in HTTP server and enable DEBUG

Let’s test drive our setup so far. Run your app now like this:

(dtbot-hT9CNosh) ~/dtbot $ python manage.py runserver
Performing system checks...

System check identified no issues (0 silenced).
August 24, 2018 - 15:02:58
Django version 2.1, using settings 'dtbot.settings'
Starting development server at http://127.0.0.1:8000/
Quit the server with CONTROL-C.

When you point your browser to http://127.0.0.1:8000/, you should notice two things:

  1. This should work and a web page should be shown.
  2. The content of the web page should be:
    Not_Found
    If you ever had worked with Django before, you may have expected a DEBUG – page with additional details. This is not shown since we changed DEBUG = True in the settings.py file for our environment extraction logic before. By default (when there is no DEBUG variable with a value of True assigned to it) this results in False as a security measure.
    To have the DEBUG – page shown here instead set the environment variable DEBUG to True before starting the HTTP server using python manage.py runserver like this:

    (dtbot-hT9CNosh) ~/dtbot $ export DEBUG="True"
    (dtbot-hT9CNosh) ~/dtbot $ python manage.py runserver
    Performing system checks...
    
    System check identified no issues (0 silenced).
    August 24, 2018 - 15:12:09
    Django version 2.1, using settings 'dtbot.settings'
    Starting development server at http://127.0.0.1:8000/
    Quit the server with CONTROL-C.

    Now, when you refresh your browser, the DEBUG page should be shown instead:

    django_21_debug_page

Heroku config vars

So – why did we change the way DEBUG is switched now?
In containerized environments, settings are often defined using environment variables inside the containers, since this enables the users to use reusable applications without changing any files in each container. It’s just more flexible and easier to orchestrate than config file settings, since scaling and multi-node setups are much more a standard pattern in containerization than it is in the bare metal world. Heroku is no exception to this.

You can change and list environment variables for an application in Heroku using the heroku CLI. To show all registered config vars and their value, just execute heroku config. New ones can be set or existing ones can be changed using heroku config:set:

(dtbot-hT9CNosh) ~/dtbot $ heroku config
=== dry-tundra-61874 Config Vars

(dtbot-hT9CNosh) ~/dtbot $ heroku config:set DEBUG=True
Setting DEBUG and restarting ⬢ dry-tundra-61874... done, v3
DEBUG: True
(dtbot-hT9CNosh) ~/dtbot $ heroku config
=== dry-tundra-61874 Config Vars
DEBUG: True
(dtbot-hT9CNosh) ~/dtbot $

Now, whenever you push a change to your project, it will be deployed with Debugging enabled.
🚨 Most certainly, you do not want that for a productive application 🚨! There are more advanced methods to debug a remote Django web application than to expose it’s debugging data to the world!
But for this tutorial-like application, it is a great and easy example to demonstrate how this concept is working on Heroku.
Leave this set to True; we will push our Django app to Heroku soon. Then, we will play around with this a bit and finally switch the content of the DEBUG config var to False, before we will never touch it again 😉

Prepare our Django app to be shipped 🎁

Before we can deploy our app on Heroku, we need to do two more preparations:

Add a PostgreSQL RDBMS to your Heroku app 🐘

In Heroku, this is only one command:

(dtbot-hT9CNosh) ~/dtbot $ heroku addons:add heroku-postgresql
Creating heroku-postgresql on ⬢ dry-tundra-61874... free
Database has been created and is available
 ! This database is empty. If upgrading, you can transfer
 ! data from another database with pg:copy
Created postgresql-rectangular-59399 as DATABASE_URL
Use heroku addons:docs heroku-postgresql to view documentation
(dtbot-hT9CNosh) ~/dtbot $

That’s it!
When you check your app’s config vars, you will notice that a new variable called DATABASE_URL was added to your application. Also, it now shows up in the list of addons:

(dtbot-hT9CNosh) ~/dtbot $ heroku addons

Add-on                                            Plan       Price  State  
────────────────────────────────────────────────  ─────────  ─────  ───────
heroku-postgresql (postgresql-rectangular-59399)  hobby-dev  free   created
 └─ as DATABASE

The table above shows add-ons and the attachments to the current app (dry-tundra-61874) or other apps.
  
(dtbot-hT9CNosh) ~/dtbot $ heroku config
=== dry-tundra-61874 Config Vars
DATABASE_URL: postgres://${PG_USER}:${PG_PASS}@ec2-54-83-51-78.compute-1.amazonaws.com:5432/${PG_DBNAME}
DEBUG:        True
(dtbot-hT9CNosh) ~/dtbot $

This way, your Django app is connected to your PostgreSQL database automatically, since the connection string defined in DATABASE_URL is picked up by the django_heroku module we added to our settings.py file earlier.

This way, you are done preparing a PostgreSQL database for your Heroku hosted web app with just one single command 👍
Isn’t this just awesome?
But there is even more: You can even connect to that DB by extracting the necessary parts for your application (pgAdminIII, psql, …) from that URL. If you want to use psql from your current workstation, there is even a more convenient way:

(dtbot-hT9CNosh) ~/dtbot $ heroku pg:psql
--> Connecting to postgresql-rectangular-59399
psql (10.5 (Ubuntu 10.5-1.pgdg16.04+1), server 10.4 (Ubuntu 10.4-2.pgdg14.04+1))
SSL connection (protocol: TLSv1.2, cipher: ECDHE-RSA-AES256-GCM-SHA384, bits: 256, compression: off)
Type "help" for help.

dry-tundra-61874::DATABASE=>

So, in theory, whenever you need a PostgreSQL DB for a project hosted literally anywhere, you could just create an empty Heroku app, add an heroku-postgres addon to it and start utilizing that DB.

To me, this is pure elegance!

Create a Procfile

Finally, as the last preparation step, let’s get one more detail out of the way: Create a Procfile to configure how Heroku should startup your dyno with but one line as content:

web: gunicorn dtbot.wsgi

If you want to know more about a Procfile, please read the previous part of this series or the corresponding Heroku article on the Procfile.

Ship your app! 🛳

We now should have the basics set and be ready to commit our files to the Git repository and to push it to the heroku remote to trigger a deployment.

🚨 Once more a warning 🚨: Remember that since we still have the config var DEBUG set to True, our app will be in debugging mode! We do not really have valuable data in it yet and it is also more than unlikely that in the few minutes between our deployment and the moment when we switch it to False someone with bad intention will visit our unpublished, randomly generated URL. But please decide this on your own. If you do not feel comfortable with this and do not need to see the debug page on your own environment to get an idea for the environment, better switch config var DEBUG to False before you do your push to heroku master.

Deploy your prepared Django app like this:

(dtbot-hT9CNosh) ~/dtbot $ git add .
(dtbot-hT9CNosh) ~/dtbot $ git commit -m "Init"
[master (root-commit) 3b3927c] Init
 13 files changed, 440 insertions(+)
 create mode 100644 .gitignore
 create mode 100644 Pipfile
 create mode 100644 Pipfile.lock
 create mode 100644 Procfile
 create mode 100644 dtbot/__init__.py
 create mode 100644 dtbot/settings.py
 create mode 100644 dtbot/urls.py
 create mode 100644 dtbot/wsgi.py
 create mode 100755 manage.py
(dtbot-hT9CNosh) ~/dtbot $ git push heroku master
Counting objects: 24, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (19/19), done.
Writing objects: 100% (24/24), 10.99 KiB | 0 bytes/s, done.
Total 24 (delta 3), reused 0 (delta 0)
remote: Compressing source files... done.
remote: Building source:
remote: 
remote: -----> Python app detected
remote: -----> Installing python-3.6.6
remote: -----> Installing pip
remote: -----> Installing dependencies with Pipenv 2018.5.18…
remote:        Installing dependencies from Pipfile.lock (ce9952)…
remote: -----> Installing SQLite3
remote: -----> $ python manage.py collectstatic --noinput
remote:        /app/.heroku/python/lib/python3.6/site-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: <http://initd.org/psycopg/docs/install.html#binary-install-from-pypi>.
remote:          """)
remote:        119 static files copied to '/tmp/build_eb69259fca4b3ef8687a35ccdf98d226/staticfiles', 375 post-processed.
remote: 
remote: -----> Discovering process types
remote:        Procfile declares types -> web
remote: 
remote: -----> Compressing...
remote:        Done: 64.8M
remote: -----> Launching...
remote:        Released v5
remote:        https://dry-tundra-61874.herokuapp.com/ deployed to Heroku
remote: 
remote: Verifying deploy... done.
To https://git.heroku.com/dry-tundra-61874.git
 * [new branch]      master -> master
(dtbot-hT9CNosh) ~/dtbot $

“remote: … deployed to Heroku” – exactly what we were hoping for to see! When we navigate to our apps URL (https://dry-tundra-61874.herokuapp.com/ in this example), we should now see the DEBUG page we did before when we tested locally. For convenience, you can simply execute heroku open to direct your browser to the correct URL:

heroku_django_debug_page
… let’s disable the debugging now to make that bad tummy feeling go away before we proceed.
In your shell, simply execute heroku config:set DEBUG=False :

(dtbot-hT9CNosh) ~/dtbot $ heroku config:set DEBUG=False
Setting DEBUG and restarting ⬢ dry-tundra-61874... done, v6
DEBUG: False
(dtbot-hT9CNosh) ~/dtbot $

With immediate effect, the debugging mode should be disabled when you refresh your browser:

django_on_heroku_without_debug

Phew, that happened to become quite a long article, hasn’t it?
Because of this, I will stop now and better continue in the next part.

So far, you managed to create your Django project skeleton, prepare it for Heroku and deployed it already! That is enough for one day anyway, I guess.

Outlook for the next part of the series

We just learned how to prepare a proper local environment for Django and Heroku by creating a virtualenv using pipenv. Also, we saw how to kickstart a new Django project, make changes to its config appropriate for Heroku hosting, deploy it to Heroku and how to use Herokus config vars properly to quickly change settings in your running and future deployed apps. Finally, we already had a peek into some of Herokus convenient development features and how to utilize them by the heroku CLI.

In the next article of this series, we will see some more of Herokus features which support you in your development. We also need to do some other actions to finalize our Django app in Heroku; we neither have applied our migrations to the PostgreSQL DB, nor created a superuser to access the Admin Area yet. Finally, we will create a Django app which can receive the JSON data sent by your Telegram bot and do something with it.

If you liked or disliked this article, I’d love to read that in the comments!

Enjoy coding!

Series Navigation << Go back to previous part of this series (Part 5)Jump to next part of this series (Part 7) >>